06-29-2021 11:08 AM
NDMP and File Level Backups are failing from two media servers with error 83
Critical bptm (pid=5744) sts_get_server_prop_byname failed: error 2060001 one or more invalid arguments
Critical bptm (pid=5744) failure to open sts for storage server ldcdd2500.latticesemi.com: plug-in reports error 2060001 one or more invalid arguments
The same data domain when used from another media server is backing up without any issues.
06-30-2021 01:22 AM - edited 06-30-2021 01:23 AM
Just a pure guess, but it could be name resolution (DNS) issues
sts_get_server_prop_byname
Storage Server get server properties by name.
Update :
06-30-2021 01:28 AM
Remove the media server from Storage Server Credentials and add it back again. If it errors, stop/restart the DD service on the media server. Last resort is to uninstall the OST and reinstall it.
06-30-2021 10:32 AM
Nicolai the name resolution is fine as we already verified with our network guys.
Could this also happen because of license limitations, as our DD is out of any support now.
06-30-2021 10:32 AM
Laurence will give it a try with the steps mentioned by you.
06-30-2021 09:21 PM
@ritwick_sHello
sorry for pointing a very basic thing.. but do you have a supported DD Boost Plugin installed on the media server ?
you can verify installed plugin by running commands on media server
/usr/openv/netbackup/bin/admincmd/bpstsinfo -pi -stype DataDomain
07-01-2021 12:28 PM
@pats_729 I've attached the command output, also I can see that the OST is already installed
07-01-2021 05:35 PM - edited 07-01-2021 05:37 PM
Hi @ritwick_s
Assuming the output is from one of the problem media servers, it is not indicating that there is a configured Data Domain plugin. You should see an entry/block including the prefix indicating "Prefix: DataDomain:". You should also be able to see a lib file in the <installpath>\NetBackup\bin\ost-plugins directory called something like libstspiDataDomain.dll
Not sure what the picture is showing or where it comes from - I'm guessing the Add/Remove programs from the control panel. Given the installation date - has NetBackup been upgraded since then and was the plugin information "lost" during the process.
I'd suggest reviewing the release notes for the plugin and determine how to reinstall this into NetBackup.
David
07-01-2021 09:27 PM
@ritwick_sas highlited by @davidmoline ADD/REMOVE program shows a plugin installed but the bpstsinfo command doesnt show a DD plugin installed.
My suggestion is.. you do a clean removal of DD Plugin and install it again. Ensure bpstsinfo lists the plugin after this.
Hope it helps
07-05-2021 08:53 AM
David sorry I was on sick leave,
Well the Netbackup hasnt been upgraded in our environment it wont be anytime soon. We're still using 7.7.3 with no support from Veritas.
Do you think if I re-install the OST and the netbackup 7.7.3 on the problematic media server, would that fix the issue?
07-05-2021 02:57 PM
Hi @ritwick_s
As I and @pats_729 have said, the plugin is not displaying in the bpstsinfo output. So I suspect re-installing the OST plugin for the Data Domain will fix the issue.
David
07-06-2021 03:26 AM
I re-installed the OST, the bpstsinto is now failing
07-06-2021 03:40 PM
Hi @ritwick_s
The command is only failing as the arguments are incorrect. The argument for stype is a storage server type and although listed in the reference guide does not appear a valid argument for the plugininfo option. Run the command like this:
# bpstsinfo -pi -serverprefix DataDomain
or
# bpstsinfo -pi
The first will show whether the reinstall has added the plugin correctly, the second will show all the plugins available - so look for the DataDomain plugin.
Cheers
David
07-12-2021 07:35 PM
David, the ost plugin is now getting listed, however it seems the communication is still not happening. I've attached the output of both the commands. Could this be also happening because our Data Domain support/license has been expired.
07-12-2021 08:21 PM
Hi @ritwick_s
I don't see any plugin issue or problem yet.
You are seeing an error in the second command due to not using the correct syntax. See my previous reply for how the "-serverprefix" option should be used.
The "bpstsinfo -pi" merely lists the plugin information on the server it is run on and does not communicate with the OST device (such as the DataDomain).
I cannot comment on whether you will still have troubles with an expired support/license with the DataDomain itself - that would be a question for Dell.
In the meantime you could try the one of the backup operations that cuase the initial fault on these media servers.
David
07-12-2021 08:39 PM
07-12-2021 10:01 PM
@pats_729 the storage server is showing as UP
Disk Pool Name : LDC_DD2500_NBU_POOL_H*******W
Disk Pool Id : LDC_DD2500_NBU_POOL_H*******W
Disk Type : DataDomain
Status : UP
Flag : Patchwork
Flag : Visible
Flag : OpenStorage
Flag : SingleStorageServer
Flag : CopyExtents
Flag : AdminUp
Flag : InternalUp
Flag : LifeCycle
Flag : CapacityMgmt
Flag : FragmentImages
Flag : Cpr
Flag : FT-Transfer
Flag : OptimizedImage
Raw Size (GB) : 121339.08
Usable Size (GB) : 121339.08
High Watermark : 99
Low Watermark : 80
Num Volumes : 1
Max IO Streams : 500
Comment : for new media server h******w
Storage Server : ldcdd2500.l******.com (UP)
07-12-2021 10:05 PM
@davidmoline I'll check with the vendor on this, however via one media server we're able to get the backup jobs completed successfully.
I re-ran the job that was using this faulty media server and now the resource is able to get assigned but still the job is failing.
07/12/2021 20:13:09 - Critical bptm (pid=6524) image open failed: error 2060046: plugin error
07/12/2021 20:13:11 - Info bptm (pid=6524) EXITING with status 84 <----------
07/12/2021 20:13:11 - Critical bptm (pid=6524) sts_close_server failed: error 2060046 plugin error
07/12/2021 20:50:41 - Error ndmpagent (pid=6580) terminated by parent process
I've checked the bptm logs and it seems the RPC connection is not going through
07-12-2021 10:05 PM
Hi @ritwick_s
If not done already, try updating DD credentials from Media Server for DD
tpconfig -update -storage_server server_name -stype DataDomain -sts_user_id user_ID
Once credentials updated, then try a backup.
07-12-2021 11:04 PM
Hi @ritwick_s
Did you install the correct version of the plugin on the media server - I see this in the log provided (I've filtered out lots of uninteresting info from the log provided):
20:13:06.988 [6524.1868] <16> 256129:bptm:6524:hqbckpnew.latticesemi.com: [197C:74C] ddcl_nfs_PLU_query_with_key(): OST secure query RPC failed with 5028 (unrecoverable rpc failure) -- RPC error 10; falling back to legacy RPC
20:13:06.988 [6524.1868] <16> 256129:bptm:6524:hqbckpnew.latticesemi.com: [197C:74C] ddcl_PLU_generate_psk_key(): Using legacy non-PSK authentication mechanism
20:13:09.675 [6524.1868] <16> 256129:bptm:6524:hqbckpnew.latticesemi.com: [197C:74C] ddp_set_file_xattr() failed, Err: 5045-Incompatible Version
20:13:09.675 [6524.1868] <16> 256129:bptm:6524:hqbckpnew.latticesemi.com: C:\Program Files\Veritas\NetBackup\bin\\ost-plugins\libstspiDataDomain.dll:stspi_create_image_do STS_EPLUGIN [DDErrNo = 5045 (Function not supported)] ddp_set_file_xattr create attr failed
Looks to me like either a credential problem as @pats_729 mentioned or a mismatch in versions between what is running on the DataDomain (ldcdd2500.latticesemi.com) and the OST plugin you installed on the media server (hqbckpnew.latticesemi.com).
David