The memory cards are both proprietary and expensive, and you'll fill up one of the smaller 8 or 16 GB ones fast, so look for a good sale on the higher capacity models if you intend to have your Vita be primarily digital-based.
They released a newer model of the Vita that uses an LCD screen rather than OLED, but as far as I've heard it's fine. The older models come in Wi-Fi or 3G, but the 3G is only compatible with AT&T's data plans and not really worth any extra you might think to pay for it, but it doesn't affect the system in any significant way if you get one in a bundle or something.
I actually like the system quite a bit, their recent push for indie games as well as the fact that a lot of the more niche Japanese stuff ends up on Vita rather than PS3/4 makes me happy I own one.
Just when I was finally leaning towards getting a PS3 (I only have a Wii), Sony kills one of the main selling points of getting one. I never owned a PS2, and I usually like to go back and play cheap games from the previous generation.