If it were a clogged up cat you may not notice it at lower RPMS because it doesn't need as much flow to idle. At higher RPMS the air could be forcing it's way through there enough to not notice the rough idle as well. When you let off the gas does it make a weird noise?