it probably just "feels" like it is smoother.

if you want to get it smoothed out, then get a SAFC and tune it a touch.
You're probably a considerable amount on the rich side anyway.

beyond that, for the sake of everyone else, just leave the silencer in it since you have it.

as far as what you "should do"

get a stupid amount of heat wrap. (you could technically wrap all the way from exhaust flange to muffler tip, so you can't really have "too much") get the headers wrapped up really well. Personally, I would say continue wrapping the exhaust as far back as your budget will allow. Chances are, the aftermarket piping you have is thin wall stuff and you loose a lot of heat (and sound) through that. Order a high flow cat from Jegs or something to save the baby seals. then get some universal in line resonator so save all of our ears. Then you ought to be able to just finish the exhaust off with piping.
Should still be loud enough and still have a good tone to it, but won't piss everyone off.