If you can go through all of the gears fine, then I would say it is the turbo. Since you are leaking oil from it, the seals are shot. The shaft is most likely moving or bent now. The fins and shaft are doing most of the grinding. Hence the lack of boost.

You may not want to drive it until you get a new turbo on it. There is a possiblility that you are sending metal fragments into the engine by driving it. you can do even more damage to the internal if that is happening.