You are making some excellent points. Even the most abhorrent speech must be protected. I do believe that free speech is an absolutist concept. The minute you introduce relativism, free speech is negated.
Speech and thought are profoundly intertwined. The minute you restrict the former, your right to free thought is also endangered, and your freedom itself is compromised.