Without being too picky, I think a better answer to this would be what the success of the Fifty Shades trilogy says about gender roles, as opposed to the texts on their own - but even then, it's a difficult question that needs picking apart a little.
It's worth bearing in mind that the trilogy bears a lot of stylistic similarities to the Twilight series - hardly a shock, considering that it famously started out as Twilight fanfiction - but the success of both series can teach us a few things.
First: that the truly twisted fantasies of Fifty Shades are the emotional ones. By any standards, Christian Grey is totally lacking in emotional maturity and is needy/jealous to the point of warranting a restraining order if he was true to life. The same, incidentally, goes for Edward Cullen to a different extent (it's debatable whether it's lesser or greater). They're potential partners who, abstracted from lived experience, might appear earnest or protective or a host of traits we tend to appreciate, but a fantasy in which a Grey-like figure hounds your every move should be just that.
Second: that the reading public has absolutely no problem conflating BDSM with sexual abuse. Enthusiastic consent to pain in sexual activity has a fairly extensive and illustrious history, and there are multiple studies that reveal that the concept that only those with mental trauma can enjoy BDSM activities is a myth. Throughout Fifty Shades, Grey is painted as an abused child who grows up to be a monster, and Anastasia only ever enjoys being hurt with an aggressive sense of guilt permeating each event. (There are, also, the multiple occasions where her consent appears to come more as a result of emotional blackmail than any informed, enthusiastic choice.)
This isn't that surprising - sex-positive BDSM culture is still kept quiet in civilized circles, and the information online (where one can only assume E. L. James whiled away her research hours) can be misleading at best. Nevertheless, it sends another disturbing message. Accepting the success of the trilogy, sexual abuse either becomes something with blurred distinctions (it isn't) or is actively glorified as a fantasy by its readers (it shouldn't be). I'd personally argue that there is the occasional healthy representation of sadomasochism peppered throughout, but they tend to alternate with constructions of abuse to the point where the entire experience is uncomfortable.
None of this touches on the media response, which seems to believe that the framing concept - naïve young girl becomes a sexual submissive to a slightly older dominant man - somehow goes against all of the achievements of the feminist movements of the last century. That's ridiculous, of course - healthy Dominant/submissive relationships thrive on (here's those words again) enthusiastic, informed consent. This means that all parties agree; the agreement isn't grudging or hesitant; and that both parties fully understand that any D/s relationship involves elements of roleplay and isn't intended to function as some grand societal comment. You get the impression from morning TV anchors that they haven't read it, though, so I'm not sure that this warrants much treatment.
I'm not sure you can polarise the reactions between (as you put it) MEN and WOMEN, as I've seen a number of people of all genders both glorify and vilify it. It might be worth considering that there's a degree of separation from the protagonist for male readers that might create the abstraction necessary to ignore the fact that it might arouse the reader, therefore exposing a few obvious truths: Fifty Shades is awfully written, clumsy trash that has no idea what it's talking about and thrives on emotionally manipulating its readers rather than engaging them intelligently. But - and it's definitely worth pointing this out - some of the more savage attacks of the novels I've seen have come from women, and I'd reinforce that the key distinguishing criteria are an awareness of literary accomplishment (which Fifty Shades is woefully deprived of) and a basic understanding of sex-positive gender politics and theory over the last twenty years (which E. L. James definitely doesn't have).
More questions on books: