SQL Availability Group with non-standard listener port fails

I am trying to add a SQL AG with a non-standard port, so the add server looks like this:

AG-XX-EV-01.domain.local,51000

All four nodes are identified correctly, and the remote agent install is successful for all nodes.

After "Finish" it shows "inserting server", and then - after a while -:

The Server was unable to complete the requested operation. Possible Cause:

The directory or file was not found or couldn't be accessed.

And that's it! What could be the cause?

Anybody encountered that problem before?