Jefferson obviously isn't perfect but do you know who is perfect in the eye of mainstream America, Abraham Lincoln. Lincoln was actually very racist (you can find some racist quotes of his online). He even proposed an amendment to prevent the federal government from ever abolishing slavery. Lincoln just wanted the Union restored because he wanted to keep the south as a tax base. And the reason why I am not going into the specifics of this article is because I want whoever is reading this article to do their own research about Lincoln.
In conclusion, I don't think Jefferson should be taken down without knowing what he stood. The people think Lincoln should stay, because they don't know what he stood for. And knowing is half the battle.