I understand exactly what you are going through. It is just weird that trying to fit in just isn't allowed as such. My grandson is of mixed race as my son in law is black. I see the stares when they are at the shops and it is just wrong. Unfortunately it will take many years to fix this place to get over the bitterness and racism that still exists here.
I love France and envy you for what you are doing. I know you won't regret it and are moving for all the right reasons.