Every flag draws boundaries and both includes and excludes. If every flag that was seen in a negative light by someone were removed from society there would be no flags at all anywhere.
So, if the Confederate flag haters want it removed from every public space then so should every flag be. Otherwise, it isn’t a symbol they wish removed, but a reality. And that reality is- the South.
And, even if the flag is removed from every place it now appears, it will do nothing to solve the problem of racism. The Flag Mad are attempting to cure cancer with a bandaid. And only a fool does that.