cancel
Showing results for 
Search instead for 
Did you mean: 

Backup failing from two media servers

ritwick_s
Level 3

NDMP and File Level Backups are failing from two media servers with error 83

Critical bptm (pid=5744) sts_get_server_prop_byname failed: error 2060001 one or more invalid arguments
Critical bptm (pid=5744) failure to open sts for storage server ldcdd2500.latticesemi.com: plug-in reports error 2060001 one or more invalid arguments

The same data domain when used from another media server is backing up without any issues.  

21 REPLIES 21

Nicolai
Moderator
Moderator
Partner    VIP   

Just a pure guess, but it could be name resolution (DNS) issues

sts_get_server_prop_byname

Storage Server get server properties by name.

Update : 

https://www.veritas.com/support/en_US/article.100007452

L_BR
Level 5

Remove the media server from Storage Server Credentials and add it back again. If it errors, stop/restart the DD service on the media server. Last resort is to uninstall the OST and reinstall it.

Nicolai the name resolution is fine as we already verified with our network guys.

Could this also happen because of license limitations, as our DD is out of any support now.

Laurence will give it a try with the steps mentioned by you.

pats_729
Level 6
Employee

@ritwick_sHello

sorry for pointing a very basic thing.. but do you have a supported DD Boost Plugin installed on the media server ?

you can verify installed plugin by running commands on media server

/usr/openv/netbackup/bin/admincmd/bpstsinfo -pi -stype DataDomain

 

@pats_729 I've attached the command output, also I can see that the OST is already installed

Hi @ritwick_s 

Assuming the output is from one of the problem media servers, it is not indicating that there is a configured Data Domain plugin. You should see an entry/block including the prefix indicating "Prefix: DataDomain:". You should also be able to see a lib file in the <installpath>\NetBackup\bin\ost-plugins directory called something like libstspiDataDomain.dll

Not sure what the picture is showing or where it comes from - I'm guessing the Add/Remove programs from the control panel. Given the installation date - has NetBackup been upgraded since then and was the plugin information "lost" during the process. 

I'd suggest reviewing the release notes for the plugin and determine how to reinstall this into NetBackup.

David

@ritwick_sas highlited by @davidmoline ADD/REMOVE program shows a plugin installed but the bpstsinfo command doesnt show a DD plugin installed.

My suggestion is.. you do a clean removal of DD Plugin and install it again. Ensure bpstsinfo lists the plugin after this.

Hope it helps

 

David sorry I was on sick leave, 

Well the Netbackup hasnt been upgraded in our environment it wont be anytime soon. We're still using 7.7.3 with no support from Veritas.

Do you think if I re-install the OST and the netbackup 7.7.3 on the problematic media server, would that fix the issue?

Hi @ritwick_s 

As I and @pats_729 have said, the plugin is not displaying in the bpstsinfo output. So I suspect re-installing the OST plugin for the Data Domain will fix the issue.  

David

I re-installed the OST, the bpstsinto is now failing

Hi @ritwick_s 

The command is only failing as the arguments are incorrect. The argument for stype is a storage server type and although listed in the reference guide does not appear a valid argument for the plugininfo option. Run the command like this:

# bpstsinfo -pi -serverprefix DataDomain
or
# bpstsinfo -pi

The first will show whether the reinstall has added the plugin correctly, the second will show all the plugins available - so look for the DataDomain plugin.

Cheers
David

David, the ost plugin is now getting listed, however it seems the communication is still not happening. I've attached the output of both the commands. Could this be also happening because our Data Domain support/license has been expired.

Hi @ritwick_s 

I don't see any plugin issue or problem yet.

You are seeing an error in the second command due to not using the correct syntax. See my previous reply for how the "-serverprefix" option should be used. 

The "bpstsinfo -pi" merely lists the plugin information on the server it is run on and does not communicate with the OST device (such as the DataDomain). 

I cannot comment on whether you will still have troubles with an expired support/license with the DataDomain itself - that would be a question for Dell. 

In the meantime you could try the one of the backup operations that cuase the initial fault on these media servers. 

David

Hi
Now that the plug-in is installed can you show us output for following command
nbdevquery -listdp -stype DataDomain -U
If you see storage server DOWN then something to look at DataDomain side (you already highlighted you have an expired license).

@pats_729 the storage server is showing as UP

Disk Pool Name : LDC_DD2500_NBU_POOL_H*******W
Disk Pool Id : LDC_DD2500_NBU_POOL_H*******W
Disk Type : DataDomain
Status : UP
Flag : Patchwork
Flag : Visible
Flag : OpenStorage
Flag : SingleStorageServer
Flag : CopyExtents
Flag : AdminUp
Flag : InternalUp
Flag : LifeCycle
Flag : CapacityMgmt
Flag : FragmentImages
Flag : Cpr
Flag : FT-Transfer
Flag : OptimizedImage
Raw Size (GB) : 121339.08
Usable Size (GB) : 121339.08
High Watermark : 99
Low Watermark : 80
Num Volumes : 1
Max IO Streams : 500
Comment : for new media server h******w
Storage Server : ldcdd2500.l******.com (UP)

@davidmoline I'll check with the vendor on this, however via one media server we're able to get the backup jobs completed successfully. 

I re-ran the job that was using this faulty media server and now the resource is able to get assigned but still the job is failing. 

07/12/2021 20:13:09 - Critical bptm (pid=6524) image open failed: error 2060046: plugin error
07/12/2021 20:13:11 - Info bptm (pid=6524) EXITING with status 84 <----------
07/12/2021 20:13:11 - Critical bptm (pid=6524) sts_close_server failed: error 2060046 plugin error
07/12/2021 20:50:41 - Error ndmpagent (pid=6580) terminated by parent process

  

I've checked the bptm logs and it seems the RPC connection is not going through

Hi @ritwick_s 

If not done already, try updating DD credentials from Media Server for DD

tpconfig -update -storage_server server_name -stype DataDomain -sts_user_id user_ID

Once credentials updated, then try a backup.

Hi @ritwick_s 

Did you install the correct version of the plugin on the media server - I see this in the log provided (I've filtered out lots of uninteresting info from the log provided):

20:13:06.988 [6524.1868] <16> 256129:bptm:6524:hqbckpnew.latticesemi.com: [197C:74C] ddcl_nfs_PLU_query_with_key(): OST secure query RPC failed with 5028 (unrecoverable rpc failure) -- RPC error 10; falling back to legacy RPC
20:13:06.988 [6524.1868] <16> 256129:bptm:6524:hqbckpnew.latticesemi.com: [197C:74C] ddcl_PLU_generate_psk_key(): Using legacy non-PSK authentication mechanism
20:13:09.675 [6524.1868] <16> 256129:bptm:6524:hqbckpnew.latticesemi.com: [197C:74C] ddp_set_file_xattr() failed, Err: 5045-Incompatible Version
20:13:09.675 [6524.1868] <16> 256129:bptm:6524:hqbckpnew.latticesemi.com: C:\Program Files\Veritas\NetBackup\bin\\ost-plugins\libstspiDataDomain.dll:stspi_create_image_do STS_EPLUGIN [DDErrNo = 5045 (Function not supported)] ddp_set_file_xattr create attr failed

Looks to me like either a credential problem as @pats_729 mentioned or a mismatch in versions between what is running on the DataDomain (ldcdd2500.latticesemi.com) and the OST plugin you installed on the media server (hqbckpnew.latticesemi.com).

David