I enjoy the xB quite a bit. I'm on my 2nd xB and had a Honda Fit for about 7-days before this last one (long story). Overall the xB was a better fit for me than the Fit. I'm 6'4, so it is difficult for me to comfortably drive most of these econoboxes. The xB offers a much more comfortable driving position IMO. Plus with me behind the wheel of the xB, my friends who are over 6-ft tall can comfortably fit in the back seat behind me...the same cannot be said for the Fit.
The Fit does get better gas mileage than the xB, but this is mainly due to the massive difference in how the transmissions are geared. The xB is geared to give it more down low oompf...so it is very easy to get up to speed, maneuver through traffic, pull out into traffic, etc. The Fit feels very lethargic around town and I found myself mashing the gas quite a bit more in the Fit to get it to go anywhere. Because of this gearing difference I find the xB to be much more fun to drive around town when compared to the Fit. Both my xB and the Fit were the 5-speed manual.
My suggestion is get out there and test drive both vehicles to decide for yourself.