Even the 32bit hotfixes and service packs contain updates for the 64bit agent folders (and vice versa) so you download the appropriate bit level for the media server. To save effort install all the current service packs and hotfixes before you attempty to install/update any remote servers as then they get the updates in one pass.
Although you should be able to use agents without updates applied even though the media server has been updated to a newer SP and HF level - we do recommend that you do a rolling agent upgrade as soon as possible - this is because we will have fixed some issues that specifically affect the remote agent in a lot of the updates contained in the SP - so although you may not think you have a problem there could be something critical that you do need. Also we don't do stability testing for mismatched remote agent to media server configurations which would mean if you run into anything unexpected, support would ask you to update the agents as well (just to be sure iit is not the mismatch causing the problem)