Got them. It was a combo of my magnet being weak and the oil creating a little bit of suction...

Stronger magnet solved the problem.