Orlando Outfitters recommends buying a fly fishing vest that's one size larger than your normal shirt size since you'll often have on a jacket or sweater, or additional layers of clothing. But look for a vest that has a a number of pockets matched to the gear you carry, e.g. number of fly boxes, leaders, de-hookers...Check the number of d-rings for nets or other tools, and look for cloth or wide elastic attachment bands holding the rings. The wider the better.
Wading in deeper water demands a shorter vest length. And remember, today's vests also come in women's sizes.