There's actually a basic asnswer and it's that the only industry force is income and money, so the product has to sell. That's it. There's no Blackrock safety net, little in the way of government funding (and if there is, it's for safe and conservative projects) and no cabal of elites jerking eachother off over their art - it has to be profitable.
There's a few cases where that's not entirely true, sometimes "established" creators get way more leeway then they deserve, and there's obviously a level of reverence to stuff like a studio Ghibli production, but there's no real sense that you can make trash and get away with it forever.
Traditional TV is also still somewhat thriving and so the concept of advertising and ratings still drives the mentality of media production, even down to time formats, pacing, etc.
The other reason is as you say, the x for y concept. Shonen isn't a genre, it's a demographic, but that's exactly the thinking that leads to a healthy media landscape. At a definition level, the primary consideration is the audience a piece of media is more for, not the nature of the media itself or what it contains. You make something for x audience first, and THEN you consider what will be in product based on what that audience likes.
A broad generalisation, and there are lots of exceptions, but you see that this mentality hasn't really stopped persisting for decades. It works for a domestic audience whilst also creating media that is still somehow accessible to international audiences - because boys/girls are the same everywhere, and it's what SELLS.