AFAIK, Creating an alarm for the condition specified is possible. But powering off least used VM will be difficult.
The approach would be to create an alarm, say memory max usage 90%.
As an action run a command, which can be a PowerCLI oneliner/Script which would get the least used VM and turns it off.