Backups up data to tape drive device terminates spontaneously with status code 150

[AndriiPushkarov]

Hello All,

Backups up data to tape drive device terminates spontaneously with status: termination requested by administrator (150)

If the data size is small, less than 1 GB, then there is a chance that the backup can go through without error

The problem appeared after moving the master server from a Windows Server 2012R2 server to another server running the Windows 2019 Server operating system. Media servers and magnetic tape library equipment remained unchanged

The problem appears when recording to tape drive devices. Writing to the disk storage unit is performed without error

Please, if possible, help determine the primary cause of the problem

Attached are examples of backup execution ending in errors:

- activity_monitor_fs_backup.txt – file backup;

- activity_monitor_vm_backup.txt – virtual machine backup

Thank you,

Andrii Pushkarov

[quebek]

Hey

To start trouble shooting you have to enable logging on master, media and client. to do so read this

https://www.veritas.com/support/en_US/doc/86063237-127664549-0/v40601501-127664549

and run mklogdir program for your OS. Than it would be wise to bump the verbosity to level let's say 3 - also on master, media and client.

https://www.veritas.com/support/en_US/doc/86063237-127664549-0/v40600920-127664549

Once this all done check in activity monitor what process was causing the issue, in your logs for FS lvl I noticed bpbkar which runs on client... read the bpbkar log to understand the issue.

If logging level of 3 is too low set it to 5 - and retry the job.

[Nicolai]

hi @AndriiPushkarov 

Check the servers event log for application crashes. From my experience status code 150 "from nowhere" is caused by one or more Netbackup application services crashing. 

Link to a similar case with "random" status code 150:

https://www.veritas.com/support/en_US/article.100018018

[AndriiPushkarov]

Thank you for your comment. But, according to this document, another case is described, different from mine. That is, the document says that it is the “client” that has been moved from the old master server to the new master server.

[AndriiPushkarov]

The logs show an attempt to back up both the file system and the virtual machine. Both results are in error. Please take a look at the given logs. Perhaps this will allow us to understand the root cause leading to the error. Large files in the archive...

[AndriiPushkarov]

Continuation...

[Nicolai]

Hi @AndriiPushkarov 

May i suggest you use vxlogview to capture a timeframe or a job id in one text file:

Technote:  https://www.veritas.com/support/en_US/article.100017099

From the logs it seems something is messing with your IP traffic, firewalls or endpoint protection running ?: 

20:28:29.642 [8452.2496]  <16> tar_tfi::processException: An Exception of type [SocketWriteException] has occured at: Module: ../TransporterRemote.cpp, Function: TransporterRemote::write[2](), Line: 474 Module: ../Packer.cpp, Function: Packer::getBuffer(), Line: 656 Module: tar_tfi::getBuffer, Function: D:/NB/10.2.0.1/src/cl/clientpc/util/tar_tfi.cpp, Line: 315 Local Address: 0.0.0.0:0 Remote Address: 0.0.0.0:0 OS Error: 10054 (An existing connection was forcibly closed by the remote host. ) Expected bytes: 131072

20:28:38.735 [8452.2496] <16> dtcp_read: TCP - failure: recv socket (144) (TCP 10058: Can't send after socket shutdown)
20:28:38.735 [8452.2496] <16> dtcp_read: TCP - failure: recv socket (988) (TCP 10058: Can't send after socket shutdown)

Not sure it's the same job - there is a 10 min time difference here - still indicating IP traffic is being interrupted:

20:38:56.872 [12112.11100] <2> set_job_details: Tfile (279979): LOG 1701715136 16 bpbrm 12112 from client 32arch04t.xxx.xxx.xxx.xx: ERR - tar file write error (14)
20:38:56.872 [12112.11100] <2> send_job_file: job ID 279979, ftype = 3 msg len = 101, msg = LOG 1701715136 16 bpbrm 12112 from client 32arch04t.xxx.xxx.xxx.xx: ERR - tar file write error (14)

If you cannot run production backups, pls consider to open a support ticket with Veritas Support. This forum is driven by end users volunteering to help each other

[AndriiPushkarov]

Thanks everyone for the recommendations.
On the master server and media servers, on network interfaces, change the “Speed& Duplex” parameter from Auto Negotiation to a constant value. Also, on media servers the parameter has been changed: "global autotuning=disabled".
After these changes the problem does not appear.