Additionally, the argument that "hate speech is not free speech" or "hate speech is unfree speech" is flawed. We don't have government-recognized freedom of speech because we want the freedom to talk about mundane things like the weather; we have protection of free speech because we need to talk about controversial topics like politics and religion in order to solve the problems of the day.
If we prevent people from expressing their racism (or other phobias), then they won't be free to out themselves as racists. To admit that you believe your race is (or ought to be) supreme, or to suggest genocide through speech or writing, or to display a flag that might connote racism, do not present the same clear and present danger that actions like leading a mob or actively inciting genocide do.
The argument that "you have the freedom of speech, but you don't have freedom from the consequences" is wrong as well. You do not get to decide that the consequences of someone standing in place giving a racist television interview includes being punched.
You don't get to decide that, for the same reason that the T.S.A. should not get to decide that the consequences of walking into an airport includes being forced to choose between an intimate, grope-like search (or possibly strip search, or cavity search) and an electronic scanning that might give them cancer, without either a warrant or probable cause to believe that the person is a threat. It's for the same reason that G.G. Allin should not get to decide that the consequences of walking close enough to him include being raped full of A.I.D.S. and then shit on.
That reason is simple: your body is your body, and other people's bodies belong to them, not you. You don't get to take away their freedom of choice, nor do you get to subject people to consequences that they didn't agree to or know about. It's the most important one of the first things we're taught in kindergarten to make sure we get along with each other; "don't steal, don't take, share, and don't hit". And if someone hits you, tell someone. It's just too bad that we also tell our kids that "it doesn't matter who started it".