You keep track of them for bearing clearance reasons. I wouldn't worry too much about matching 'em (aside from number halves) since it would appear that you're changing out the bearings anyways.