Some people who have been well represented and unquestioningly included in "majority culture" all their lives are starting to feel attacked by the push for diversity. Suddenly, it's not okay to just surround yourself in the workplace by people who are similar to you, or to write the stories that include people like you without including anyone else, or to basically do anything in your life without thinking about whether you've alienated someone from another group. Some of them feel it means they're being called worthless, or that they're bad for being typical and very similar to "the standard," or that people pushing for more diversity believe they have no problems and have easy lives because of their privilege. They feel compelled to both downplay the importance of diversity since the problems are invisible to them AND emphasize aspects of their lives that aren't perfect to share in the rallying cry of "I'm downtrodden too!"
In other words, instead of realizing that diversity is about other people, they still think it's about them. About shaming them, erasing them, taking away from them something that was theirs. They don't want to be asked to question why they think their spaces are only theirs, and they don't want to examine why a culture that privileges them is still hurting them (and hurting other people more).
I can think of three major reasons why diversifying workplaces and media is very important, none of which have anything to do with punishing, erasing, or stealing fire from "majority" culture.
Nobody even has to specifically TELL you that people like you don't do that; you figure it out, and you figure it out young. Add to that the fact that sometimes people DO tell you--and the fact that entering those fields against all the adversity leads to people doubting your competence implicitly and explicitly, resulting in an exhausting, demoralizing daily battle against microaggressions that more "traditional" practitioners of that field don't have to face--and the cards are stacked heavily against you succeeding or being happy in those fields. But when you see someone on a TV show, in a book, or holding a position, you get the idea that people of your description can and do do that thing. It might mean more than you think for someone who never sees people like them doing what they'd like to do. It also has the lovely little side effect of everybody else getting the idea that said people can do that thing--getting used to seeing them there and not processing them as exceptions or tokens.
So, when the critics solemnly intone that they still need people to be able to jump and touch the target regardless of it being harder for some of us, they insist that asking for accommodations would require us to lower the target and decrease expectations, resulting in overall less competence just so we can say we gave people a chance. We're not asking you to lower the target, though. We're asking you to give the shorter people access to platform shoes, stools, or trampolines if that's what we need. Whining about how that's cheating makes no sense, because we're probably already jumping farther proportionately than you are, but all you can see is that I have a stool and you don't, and you perceive that you should have a stool too or nobody should because that is fair. Well, what's "fair" about me living in a world that expects me to touch the same target tall people do but I was born with very short legs? Why is that "unfairness" something you don't want to worry about?
We learned to read in very different environments. I was always ahead of the game and expected to excel, and I had resources to push me even farther because my achievements suggested I'd "earned" it. The boy I tutored was sitting there at the beginning of his life already being regarded as remedial. If he ever did "catch up," he probably spent his whole youth being expected to try harder just to achieve that baseline, even though he had fewer resources and fewer people who thought he was worth the investment because he wasn't a huge achiever. And yet, if he ever did get up to average, people would not treat it like the overcoming of adversity it was; they'd just treat it like it was expected, and continue to give opportunities to people who'd gotten objectively farther--attributing it to talent, hard work, creativity, responsibility--and reminding less advantaged people that really, their failure to achieve is just laziness, inherent mediocrity, lack of initiative, and unreliability. What can fix this? Well, sometimes, extending an advantage to people in this situation. Like a scholarship. Or a special opportunity designed to help someone in poverty or from a traditionally less represented demographic. We shouldn't be saying we only want people who came from backgrounds that don't as often lead to success if they overcame waaaaaay more than the rest of us to get on what's perceived to be even ground.
It's not always perfect, but we need to pull away from assuming the world is a meritocracy. People in the majority culture set the goalposts, and they (without necessarily realizing it) rig the game to be more easily won by people from their backgrounds. We're asking for several routes to the goal to be established, assistance being offered to those who need it to get to the goal, and different forms of success to be acknowledged. Not to mention we really want people from the majority groups to stop regarding disadvantaged and "diverse" folks as having only "made it" because someone wanted to check a diversity box. Imagine overcoming all that nonsense and expecting to finally be equal only to be told you must have gotten in on the diversity card.
Also, people who have been treated differently from the "standard" throughout their lives, have different lived experiences, and have inherently different perspectives can disrupt echo chambers in a positive way. Too many people who think alike working together on a project can limit its effectiveness, even if we're not talking about making it applicable and effective for everyone who might be affected by it. Including diverse voices on your panels, in your decision-making groups, in your management, on your creative teams, and as your leaders will not only make more accessible and inclusive worlds for us to live in, but will improve creativity and broaden horizons.
Ask not why you should push for diversity in your field and in your life; ask whether you can afford not to!