FWIW i installed a silicon inlet a couple weeks ago. It was a bitch, let's start there.. I'm sure you know that by now. BUT i did do it w/o removing intake manifold, on a stock td04 2002 wrx. It has the same clearance issue as what is illustrated above, right before the turbo.

This is how I did it. I had my dad help me wedge it on. I'd work the back of the turbo, he took a wooden mallet handle and shoved it down around the turbo and we kept wedging it until it eventually fit. We also rotated using the wooden handle back and forth, i'd shove it into the silicon hose from the front of the car towards the back, with the handle on the front curved part nearest your power steering connection there. It still isn't perfectly square but it is seated well and does not loose vac. There are major sound and spool gains to be had, so keep that in mind and DONT GIVE UP!

Use rubbing alcohol (isopropyl) when trying to wedge it on, to make the silicon slippery. Don't worry about getting small amounts in the turbo, it will dry up. If you puddle it in there, wait a while (at least an hour) before starting it just to be safe.

Keep trying, it can be done. If you get flustered, take a break and come back to try again, that usually helps. Also, crank up some slipknot for motivation !