Sort:  

It works exactly that way.

Also if you think that's scary, consider that your private key is actually just a number between 0 and 2^256-1 which means anyone that guesses it also has access to your funds.
Fortunately that is still something on the order of a trillion keys for every atom in the known universe. So it's a really, really large space to search...