If your start point was vCenter 2.x with Update Manager 1.x managing ESX 3.x:
Upgrade vCenter to 4.x
Upgrade Update Manager to 4.x
Use Update Manager to upgrade ESX 3.x to 4.x
Upgrade vCenter to 5.x
Upgrade Update Manager to 5.x
Use Update Manager to upgrade ESX 4.x to 5.x
I don't think Update Manager 5.x supports upgrades of ESX 3.x to either 4.x or 5.x, and all the vSphere 5 upgrade notes talk about ESX 4.x to 5.x only.