That's not how this works...that's not how any of this works.

Difficulty is driven by the amount of hashrate on the network and algorithmically strives to maintain an average blocktime of 10 minutes.

As more miners join the network, the difficulty will adjust higher in order to make it harder to mine a block.

As miners leave the network, the difficulty will adjust lower in order to make it easier to mine a block.

In both cases, all the difficulty cares about is maintaining an average block time of 10 minutes.

If the price falls and some miners become unprofitable (i.e. they are mining in the red) then they will stop mining and the difficulty will drop (fewer miners mining). The miners who remain will get a larger share of the blockreward and it all just sorts itself it.

At the end of the day it's a dynamic and self-correcting system.