From my experience, the patch download only seems to happen when you choose to scan and remediate the host. What is downloaded initially is only the metadata of the patches. So even if all these are present, the downloads will not happen unless you remediate a 4.0/4.1 host in your inventory.
Regards
Girish