Found the problem.
the backup agent mixed the instance names (SQL management studio didn't have the problem - but ODBC manager could reproduce the problem).
Using cliconfg.exe (use c:\windows\system32\cliconfg.exe for 64 bit and C:\Windows\SysWOW64\cliconfg.exe for 32 bit.exe) we created aliases for both instances choosing TCP as protocol and fixed port number 1433)
This creates registry entries like this (you can also just add this to the registry directly - replace SERVERNAME with your SQL servers name and MYINSTANCE with you instance name - last set the correct IPnumbers).
In this example the SQL server (and default instance of SQL server) SERVERNAME binds to 10.0.0.226 (TCP port 1433)
and second instance of SQL server binds til 10.0.0.236 (TCP port 1433)
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSSQLServer\Client\ConnectTo]
"SERVERNAME"="DBMSSOCN,10.0.0.226,1433"
"SERVERNAME\\MYINSTANCE"="DBMSSOCN,10.0.0.236,1433"
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\MSSQLServer\Client\ConnectTo]
"SERVERNAME"="DBMSSOCN,10.0.0.226,1433"
"SERVERNAME\\MYINSTANCE"="DBMSSOCN,10.0.0.236,1433"
We restarted the Symantec backup agent after this