NDMP Backup Failed (media close error (87))

[Austin_1]

Netbackup 5250 Appliance Version : 3.2
Netbackup Version : 8.2

When running NDMP backup, when a certain capacity is reached, the following log occurs and an error occurs.

In the case of the backup, there was nothing wrong until the previous day, but the same error continues to occur after the error occurred. (NDMP Host configure is IP)

So far, I've tried enabling/disabling Acclerator, enabling multi-stream (one of the two paths is successful), and EEB_MSDP Patch.

Anything more to check?

[davidmoline]

Hi @Austin_1 

Does this backup tranverse a firewall or other WAN links? Given the time it is taking to walk the directory tree of the filer, it is possible something is dropping the connection between the filer and media server. 

Has the size of the backup increased marginally from when it was working, or has the network link slowed (more data/things using it)? 

This article describes the above issue and possible solutions. https://www.veritas.com/support/en_US/article.100009509

David

[Austin_1]

Hello davidmoline

No, the size of the failed backup was smaller.

I just tried again with an incremental backup and it worked.
The problem seems to only happen with full backups.

I checked the link you provided.
as a solution
echo "0" > Can /usr/openv/netbackup/NET_BUFFER_SZ only run on master servers (including media servers)?

[davidmoline]

HI @Austin_1 

The NET_BUFFER_SZ operates on media servers (remembering a master server can also operate as a media server) - so the answer to your question is yes. 

Before changing TCP settings first review what is currently set. The article I provided has a link at the bottom of the page which goes into more detail on idle timeouts. 

Out of curiousity, can you examine one of the successful FULL jobs from last week (the larger sized ones), and see how long it takes from when the "begin writing" message appears and the ndmpagent starts sending entries to bpdbm, and when that stage completes and it actually starts writing data (when it completes the directory tree walk).

David

[Austin_1]

Hi @davidmoline

Looking at the success log last week, I saw that the backup completed in about 1 hour and 10 minutes.

The current backup failure drops after about an hour, and it seems that it took about two hours to retry once more from time to time.

[davidmoline]

Hi @Austin_1 

I didn't ask how long the backup took - I can see that, what I asked was the time between when the message "begin writing" and when it completes the directory walk - for the failed backup this later time is when you get the "critical bptm" message and this was after 1 hr and 6mins. You need to examine the job details.

David

[Hamza_H]

Hi,

I suggest to enable logs to get more information regarding this

also, as david said, I think this is a tiemout issue somewhere.

based on shared detailled status, the accelerator will not work:

Jul 2, 2023 8:00:09 PM - Info ndmpagent (pid=230332) accelerator is not supported for this stream format

I suggest to disable it.

did you try to create a new policy from scratch (not copy to new) ?

[VerJD · Accepted solution]

As the NetBackup Appliance (NBA) version 3.2 and NetBackup (NB) version 8.2 has reached EOSS (end of standard support) and it also has many known performance issues (per KB 100047753), which are resolved in NBA 3.3.0.1 / NB 8.3.0.1 or higher versions ...

• Recommend going to the Download Center and upgrading to NB Appliance 3.3.0.2 MR2 (UPD160133 ) / NB version 8.3.0.2 (REL113988 ) on all of your Servers (primary & media), plus applying any Critical Hotfixes that might be available in the Download Center for that version.

In the interim, for NBA 3.2 / NB 8.2 on the primary & media servers:

1. Apply critical hotfixes for Etrack 3998268, plus the related tuning per the KB article above.
   • Note: This EEB bundle includes MSDP EEB bundle (version 25), hence If you already have the latest MSDP EEB bundle applied (version 37), then you may see the install for the lesser version fail, which is to be expected. The same goes for any other EEBs which are already installed.
2. Apply these touch files to pair with those EEBs and restart NB services to further help with performance issues:
   • Note: If larger values are already present for the first 3 below (BPSCHED_THRESHOLD, RBALLOC_KBYTES_THRESHOLD, REPORT_RAW_KBS), take note, but do not change.

     # echo 5000000 > /usr/openv/netbackup/db/config/BPSCHED_THRESHOLD
     # echo 25000000 > /usr/openv/netbackup/db/config/RBALLOC_KBYTES_THRESHOLD
     # echo 3600 > /usr/openv/netbackup/db/config/REPORT_RAW_KBS 
     # echo 0 > /usr/openv/netbackup/NET_BUFFER_SZ 
     # echo 100000 > /usr/openv/netbackup/MAX_ENTRIES_PER_ADD 
     # echo "DISABLE_WHITELISTCACHE_CLEANUP_ON_JOB_START = 1" | /usr/openv/netbackup/bin/nbsetconfig

3. Additionally, ensure that all clients are running NB version 8.1 or higher.

Lastly, if the issue persists after upgrading to supported levels, please open a new case for that version with Support and reference your previous case for historical purposes.

[Austin_1 · Accepted solution]

Hi @VerJD 

For this NDMP backup problem, after upgrading from 3.2 -> 4.1.0.1 MR5, the backup was completed normally.

Thanks to those who helped!

[X2]

Deleted the other thread (topic in non-English script) as it was a duplicate.