One genre that takes a lot of hits for diversity is Fantasy. And considering how often fantasy books can be very white (*cough*Game of Thrones*couch*) it is a just point.
A common defense is that it is a set in a time and location where there were not non-white people. This is pretty easily proven false, there is medieval art depicting black knights and noble-people. The best response, really, though is, “What you can imagine trolls and dragons…but a black person…that kills your suspension of disbelief?”
However, I have seen some extend this statement to condemn use of structures of oppression having no place in fantasy. Slavery has no need to exist in a fantasy world. But I am not sure I buy that. It is lazy to use slavery as your in for having black characters in your book. But to argue that you cannot address a fundamental flaw in humanity (our willingness to exploit each other for gain)…well, that kind of stretches credibility.
When it comes down to it, I find it harder to believe a fantasy world where people do not exploit each other than I find it hard to believe in a world with dragons and trolls. After all, the world we live in is full of people who exploit, violate and treat their fellow humans quite badly…and there are trolls.
Using the worst of humanity in your world building gives your heroes something to stand against. Of course these things can be handled badly (over reliance on rape comes to mind) but that does not mean they have zero place in fantasy worlds. Imagining a world with dragons? No problem. Imaging a world sans the ills that have plagued humanity for centuries? Pretty hard to buy into.