0xe0009541 error

Verify if you are able to ping both ways from media server and the vCenter, ESX hosts.

If connecting via vCenter, are you using the Backup Exec Service Account (BESA) or if connecting via the ESX host, are you using the 'root' account ? If using a Windows account, have you verified it meets the requirements per http://www.symantec.com/business/support/index?page=content&id=TECH129645

Do enable AOFO in the backup job & port 902 should be open as well.

I use root via esxi host. They ping each other. The password is correct, I can browse the esxi server and its virtualmachines from be.

What is the version of the ESXi host and is the host licensed for VADP ?

If you add a vCenter and use a Windows-based account, are you successful in making the connection ?
 

I don't know what is VADP.

The esxi is version 5.

I have already backed up a vm on the same esxi. So it is not a problem of credentials.

I don't know if I use a windows-based account, but I want use esxi to backup a esxi.

I have the same error if I backup via vcenter.

Is your ESX a paid version or a free version? BE requires the vStorage API which only comes with the paid version