NDMP backups fail with code 58

[rocker65]

Windows master and media server version 7. Trying to backup a Netapp with NDMP. This backup has worked fine since I set it up. Yesterday it stopped and am getting error 58's. Which from what I can see means the system is having difficulty resolving hostname/ip. I can ping the netapp, I can telnet to it on port 10000. Backups fail with status code 58. Here is a look at my bpbrm log file. Not sure what to make of it.

Also, can't get any logging information in the */logs/ndmpagent folder. The NDMP install/config guide says it should be there for debuging information.

Facts:

Can ping host by name and IP

Can telnet host port 10000

Can NOT get a host properties on NDMP host monica

Have setup NDMP credentials with no problem

Log file here:

10:25:12.286 [1356.7512] <2> bpbrm main: INITIATING (VERBOSE = 5): version NetBackup 7.0 0
10:25:12.286 [1356.7512] <2> logparams: -backup -S xprosrv004 -c monica -ct -1 -ru root -cl NDMP-FULL -sched NDMP-FULL -bt 1301667911 -dt 0 -st 0 -b monica_1301667911 -mediasvr prosrv004 -jobid 10563 -jobgrpid 10559 -masterversion 700000 -maxfrag 1048576 -reqid -1301667847 -mt 2 -to 0 -stunit prosrv004-hcart-robot-tld-0 -rl 8 -rp 31536000 -cj 2 -D 6 -rt 8 -rn 0 -pool Incremental -use_ofb -use_otm -jm -secure 1 -kl 28 -rg other -fso -stream_count 3 -stream_number 2 -connect_options 16974338
10:25:12.286 [1356.7512] <2> verify_client: bpbrm.c.34145: *connect_opts: 16974338 16974338 0x01030202
10:25:12.286 [1356.7512] <2> read_client: ?
10:25:12.286 [1356.7512] <2> read_client: opendir() failed: monica: No such file or directory (2)
10:25:12.286 [1356.7512] <2> read_client: ?
10:25:12.286 [1356.7512] <2> read_client: opendir() failed: monica: No such file or directory (2)
10:25:12.286 [1356.7512] <2> verify_client: bpbrm.c.34211: db_getCLIENT failed for CLIENT: monica
10:25:12.286 [1356.7512] <2> verify_client: bpbrm.c.34361: db_getCLIENT failed: 227 227 0x000000e3
10:25:12.286 [1356.7512] <2> bpbrm start_bpcd_stat: bpbrm.c.20057: bpcd_client: monica
10:25:12.286 [1356.7512] <2> bpbrm start_bpcd_stat: bpbrm.c.20060: bpcd_client_hostname: monica
10:25:12.286 [1356.7512] <2> hosts_equal: Comparing hosts <monica> and <monica>
10:25:12.286 [1356.7512] <2> hosts_equal: names are the same
10:25:13.208 [1356.7512] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10061 0x0000274d
10:25:14.208 [1356.7512] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10061 0x0000274d
10:25:16.224 [1356.7512] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10061 0x0000274d
10:25:17.349 [1356.7512] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10061 0x0000274d
10:25:19.443 [1356.7512] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10061 0x0000274d
10:25:20.552 [1356.7512] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10061 0x0000274d
10:25:24.474 [1356.7512] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10061 0x0000274d
10:25:25.489 [1356.7512] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10061 0x0000274d
10:25:33.443 [1356.7512] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10061 0x0000274d
10:25:34.443 [1356.7512] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10061 0x0000274d
10:25:34.443 [1356.7512] <2> vnet_connect_to_service_or_vnetd: ../../libvlibs/vnet_vnetd.c.3584: vnet_async_connect failed: 18 0x00000012
10:25:34.443 [1356.7512] <2> nb_connect_to_vnetd_or_legacy: ../../../nb/lib/comm.c.2028: vnet_connect_to_vnetd_or_service failed: 18
10:25:34.443 [1356.7512] <2> local_bpcr_connect: nb_connect_to_vnetd_service(monica) failed: 25
10:25:34.443 [1356.7512] <2> local_bpcr_connect: Can't connect to client monica
10:25:34.443 [1356.7512] <2> ConnectToBPCD: bpcd_connect_and_verify(monica, monica) failed: 25
10:25:34.443 [1356.7512] <16> bpbrm start_bpcd_stat: Cannot connect to monica
10:25:34.474 [1356.7512] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2054: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
10:25:34.474 [1356.7512] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2068: service: bpjobd
10:25:34.489 [1356.7512] <2> job_connect: SO_KEEPALIVE set on socket 1524 for client xprosrv004
10:25:34.489 [1356.7512] <2> logconnections: BPJOBD CONNECT FROM 172.16.50.230.2323 TO 172.16.50.230.13724
10:25:34.489 [1356.7512] <2> job_authenticate_connection: ignoring VxSS authentication check for now...
10:25:34.489 [1356.7512] <2> job_connect: Connected to the host xprosrv004 contype 10 jobid <10563> socket <1524>
10:25:34.489 [1356.7512] <2> job_connect: Connected on port 2323
10:25:34.489 [1356.7512] <2> job_monitoring_exex: ACK disconnect
10:25:34.489 [1356.7512] <2> job_disconnect: Disconnected
10:25:34.489 [1356.7512] <4> db_error_add_to_file: Cannot connect to monica
10:25:34.489 [1356.7512] <2> verify_client: bpbrm.c.34455: start_bpcd_stat failed: 58 58 0x0000003a
10:25:34.489 [1356.7512] <2> verify_client: bpbrm.c.34459: *client_hostname: monica
10:25:34.489 [1356.7512] <2> verify_client: bpbrm.c.34461: *connect_opts: 16777472 16777472 0x01000100
10:25:34.489 [1356.7512] <2> bpbrm main: bpbrm.c.2758: verify_client failed: 58 58 0x0000003a
10:25:34.489 [1356.7512] <2> bpbrm kill_child_process_Ex: start
10:25:34.489 [1356.7512] <2> sendViaScript: attempting to send mail
10:25:34.489 [1356.7512] <2> sendViaScript: C:\Program Files\Veritas\NetBackup\bin\nbmail.cmd "rocke.robertson@pch.gc.ca" "Backup on monica - 58" "C:\WINDOWS\TEMP\nb_240.tmp"
10:25:41.943 [1356.7512] <2> bpbrm Exit: client backup EXIT STATUS 58: can't connect to client

Thank you for your insight.

[rocker65]

I have enabled verbose debugging on the ndmpd.log on the netapp. The Netbackup server isn't even trying to connect to the netapp, nothing is logged unless I telnet to the port.

Merci mes amis.

[Andy_Welburn · Accepted solution]

& can you successfully run: tpautoconf -verify ndmp_host

"Can NOT get a host properties on NDMP host monica" - you won't as there is no client installed on NDMP hosts.

Policy still set up as NDMP?

What changed yesterday?

Anything in "All Log Entries" report or "Problems" report?

[rocker65]

tpautoconf test's out perfectly. Neat test, didn't know that one.

Policy is NDMP.

Only thing that happened was our licenses expired. I was given new licenses, and they are all active. But the NDMP failures started before the licenses expired.

Thank you

[rocker65]

4/1/2011    11:26:57 AM    prosrv004    monica    Error    10566    Backup    backup of client monica exited with status 58 (can't connect to client)
4/1/2011    11:27:01 AM    prosrv004    monica    Info    10567    Backup    started backup job for client monica, policy NDMP-FULL, schedule NDMP-FULL on storage unit prosrv004-hcart-robot-tld-0
4/1/2011    11:27:02 AM    prosrv004    monica    Info    10567    Backup    client monica handling path /vol/vol1
4/1/2011    11:27:25 AM    xprosrv004    monica    Error    10567    General    Cannot connect to monica
4/1/2011    11:27:27 AM    prosrv004    monica    Info    10567    Backup Status    can't connect to client
4/1/2011    11:27:27 AM    prosrv004    monica    Error    10567    Backup    backup of client monica exited with status 58 (can't connect to client)
4/1/2011    11:27:27 AM    prosrv004    monica    Info    10564    Backup Status    can't connect to client
4/1/2011    11:27:27 AM    prosrv004    monica    Error    10564    Backup    backup of client monica exited with status 58 (can't connect to client)

Just the status 58 for the problems reporting as well for the NDMP host.

[rocker65]

Looks like this when I do a telnet to port 10000

Apr 01 11:36:12 EDT [ndmpd:101]: Created an NDMP server connection
Apr 01 11:36:12 EDT [ndmpd:101]: Message NDMP_NOTIFY_CONNECTION_STATUS sent
Apr 01 11:36:19 EDT [ndmpd:101]: Message NDMP_NOTIFY_CONNECTION_STATUS sent
Apr 01 11:36:19 EDT [ndmpd:101]: Ndmpd session closed successfully

So to me that means the NDMP component on the netapp is working fine. I get no information at all when the NB server tries to connect. In fact short of putting a sniffer on the net, I don't think the NB server is even assembling a packet.

[watsons]

Could it be when the license expired, the connection didn't work? Now when the new license is added, it's still trying to talk to the ndmpd daemon on the filer and didn't connect properly.

Try restarting ndmpd on the filer and reconnect.

On filer (Netapp for instance):

# ndmpd status
# ndmpd off
# ndmpd on

Else check the filer log to identify anything unusual. Btw, I assume the credential has not changed.

[rocker65]

Good morning inspector Watson

Not sure how to answer your first question except to say the ndmpd log on the netapp show zero connection attempts from the NB server. The NB server never even opens a socket as far as I can tell because there is no socket in any state showing that there was an attempt.

Credentials haven't changed

Have stopped and started ndmpd process on the netapp. But as I said, this is irelavent for now as the NB server hasn't even tried to connect, it fails.

I suspect that this error message has something to do with the socket pair not being assembled:

10:25:13.208 [1356.7512] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10061 0x0000274d
10:25:14.208 [1356.7512] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10061 0x0000274d

But..... can't seem to get an answer for that. I may have to bail here and call support.

Thank you for your input

[watsons]

That error "getsockopt SO_ERROR" looks like a hostname resolution issue, I believe you have checked whether it should be using FQDN or just short name (monica) across your setting.

# nbemmcmd -listhosts -verbose 

See how the ndmp host turns out.. and of course, what support says. :)

[rocker65]

Hi there

The system is capable of ping'ing the netapp with the short and fqdn name, and has been able to for 2 years. Nothing has changed there.

The nbemmcmd -listhosts -verbose produces:

information here about master server, will leave out. Then info for the ndmphost shows machine name which is correct, machinedescription which is empty as I don't have one configured, machineflags = 0, and machinenbutype = ndmp (2)

I opened a ticket with veritas support and listed it as a major problem. Apparently that warrants a call between 24 hours and 36 hours. Wow... Not quite as bad as Oracle support but getting up there.

[rocker65]

In case anyone else has to go through this. When my license key expired (long story), for whatever reason it set the policy type to none or null. Meaning the policy was unuseable because it had no idea what type of backup to run.

This produces an error 58. Who woulda thunked!

Hope this helps someone.

[Marianne]

Andy asked you to check policy type 4 days ago:

"Policy still set up as NDMP?"

You will agree that he deserves the Solution nod?

[rocker65]

Andy was Correct. When I looked at my policies, I saw NDMP in the name. For whatever reason I assumed that was the type.

Egg on my face syndrome.

We've been running on eval licenses for a year now. Vendor messed up. So every month I have to install new license keys.

[Marianne]

At least you will never forget to check this in future.

Welcome to the 'egg-on-face' club!