I've been a happy Fractal user since 2009 and got my II the first week they came out.
I can totally relate to the frustration of being unable to unleash a cool tube amp. After my kid was born, my Two Rock, Rivera, and Fender languished in the basement.

A buddy of mine he let me try it out his Ultra and after 20 minutes I realized that it was just what I was looking for. It allowed me to get the same tone and any volume. I could use it through headphones (though the Ultra needed an outboard headphone amp), monitors, or FRFR cabs and all sounded great.
At this point, nobody's getting more than list for an AxeFx unless they're selling overseas, where they're a good bit more costly. In fact, Fractal will occasionally sell B-stock IIs for a benjamin or two less.
IMO, at this point, most folks who "go down the rabbit hole" do so out because of curiosity or OCD more than need. I only have a couple patches I use that depart significantly from the factory presets and those are pretty straightforward. Now that the AxeChange preset sharing web site is up, there's even less reason to drill down into amp geek parameters. Finally, the tone match feature makes it pretty straightforward to clone most any amp tone for which you can find an isolated guitar track.
Downsides? Some find it frustrating that presets might need to be tweaked after a firmware update. Of course, nobody's forced to keep their firmware on the latest/greatest version. You need to spend a goodly bit of money to go "all in": AxeFx, rack case, foot controller, FRFR or power amp + cab. On the other hand, a 2K amp + 1K pedalboard + $700 cab probably costs about the same and is less versatile and much less portable. The computer-based editor hasn't worked with the current version of firmware for several months but that' should be rectified in January. While the amp modeling is otherworldly, the dirt box modeling is merely good (other FX such as modulation, delay and reverb are delicious).