NetBackup 7.5.0.4 with DataDomain and Status 2106

[John_Deackman]

(1) Master/Media Server
Windows 2008 R2 Standard

(3) Media Servers
Windows 2008 R2 Standard

(2) DataDomain DD670's

Monday, I disabled all backup policies.  Then I shutdown NetBackup on all 4 severs.  Then I did a shutdown on 1 DataDomain so that I could physically move the unit to make room for a new DataDomain 890 unit.  After moving the 670, (I had to replace 4 CAT6 cables with new ones for length) I powered up the DAE's first and let then settle to a ready state.  Then I powered up the Head Unit.  Initially, the head Unit locked up half way into the boot.  Called DataDomain andthey had me force power off the unit and had me pull it out and reseat all the RAM, PCI Riser cards and all the Hard Drives.  Then replaced and repowered and all came up fine.

I then started NetBackup services on the Master and then all the media servers.  After everything was up, I reactivated all the policies.

This ran fine until 1:00am on Thursday.  Then, the DataDomain unit I moved started failing every job that ran on it with a status of 2106 (Status 2106: Disk storage server is down).  I had Symantec look at it (logs sent in that were requested), and they siad it had to be a DD-Boost problem, as they could not see an issue.  I then called DataDomain and they put there people on it and said it was a network problem because they couldn't ping the server from the unit. 

So then I went (tonight) and ran a network tester on the cables between the unit and the core switch.  Cables tested fine and i have normal flashing link lights...  So I did a shutdown restart on the DD head unit.  After it came back up, i could ping the server from the unit.  Kicked off a failed backup and it failed again with 2106 or a media write error(84)...

When looking up the 2106 error this morning, I found a Symantec article that said I needed to reautherize the the servers to the DD unit.  How do I do that?

Does anyone else have an idea what would be causing all this?  I now have Duplications (LCP) kicking off by the hundreds every 15 or minutes and I have to clean those all up too because they are failing with a status 84 and Image Cleanups failing with a status 83...

I want to pull my hair out, but I don't have any left to pull out!

Hope some one can help.

John

[John_Deackman]

I forgot to mention, half of my backups go to one DD and the other half to the other DD. Then when a job finishes, it Duplicates to the opposite DD for DR purposes. The second DD is running fine and backking up files fine. John

[Nicolai]

http://www.symantec.com/docs/HOWTO72916

Message: Disk storage server is down

Explanation: This error may occur when a job uses a disk storage unit whose disk group resides on a storage server that NetBackup has marked down.

Recommended Action: Verify that all media servers configured for the storage server can communicate with the storage server. The bpstsinfocommand queries the storage server periodically, so you can use thebpstsinfo log on the media server.

[Nicolai]

http://www.symantec.com/docs/TECH190904

Status 2106 "Disk volume is down"  in combination with Data Domain

[Nicolai]

How to disable/enable SLP

Activate /Inactivate SLP operations

nbstlutil inactive -lifecycle <lifecycle name>
nbstlutil inactivate -backupid <backupid>
nbstlutil active -lifecycle <lifecycle name>
nbstlutil activate -backupid <backupid>

http://www.symantec.com/docs/TECH170086

[John_Deackman]

I just received a email from Symantec after they looked through the logs and they are saying it looks to them like a DataDomain License issue. Waiting for DataDomain to connect up and verify.

[John_Deackman]

" The behavior was corrected by reauthorizing each media server to the storage server." How is this done?

[Marianne]

Try NBU GUI: Media and Device Management -> Credentials -> Storage Server

[Stumpr2]

If you use ddboost you might try disable/enable

[quebek]

I do believe you are asking for tpconfig command, here under is extract to add and update credentials for OpenStorage based devices

(Add OpenStorage credentials)
   tpconfig -add -storage_server <server name> -stype <server type>
            [-proxy <proxy server name>]
            -sts_user_id <user ID> [-password <password>]
            [-st <storage type>]
            Valid values for storage type (can be added together)
             Formatted Disk = 1 (DEFAULT)
             Raw Disk = 2
             Direct Attached = 4
             Network Attached = 8 (DEFAULT)

(Update OpenStorage credentials)
   tpconfig -update -storage_server <server name> -stype <server type>
            -sts_user_id <user ID> [-password <password>]
 

[John_Deackman]

This has been corrected.  I don't know what happen or for sure what corrected it, but it had to be a problem with the network connectivity between the 2 DataDomains. After hours of Symantec support and multiple reboots of the Master and all media servers, it finially started working.  We don't have a clue why.

Thanks for everyones time.

[Stumpr2]

When it happens again try the ddboost disable/enable first.