Problem with optimized duplication between two 5020 appliances

[ddm2]

Hello,

I configured optimized duplication between 2 5020 appliances on the same domain. When the SLP duplication runs, the optimized job fail with error 84 (write failed).

If I write backup directly on the target storage server, it works.

I added the this line to my bp.conf

RESUME_ORIG_DUP_ON_OPT_DUP_FAIL = TRUE

In this way when the optimized job fails, NetBackup try to copy data without optimization. The not optimized duplication runs successfully.

NetBackup 7.1.0.3, Applicance version 1.4.1.1

This is the Activity Monitor log of failing job:

02/14/2012 11:10:46 - begin Duplicate
02/14/2012 11:10:46 - requesting resource LCM_wcssbckmedia1-dr
02/14/2012 11:10:46 - granted resource  LCM_wcssbckmedia1-dr
02/14/2012 11:10:46 - requesting resource wcssbckmedia1-dr
02/14/2012 11:10:46 - reserving resource @aaaac
02/14/2012 11:10:46 - resource @aaaac reserved
02/14/2012 11:10:46 - granted resource  MediaID=@aaaae;DiskVolume=PureDiskVolume;DiskPool=DR-disk-pool;Path=PureDiskVolume;StorageServer=aitbck1.apss.tn.it;MediaServer=wcssbckmedia1.apss.tn.it
02/14/2012 11:10:46 - granted resource  wcssbckmedia1-dr
02/14/2012 11:10:46 - requesting resource @aaaac
02/14/2012 11:10:47 - Info Duplicate (pid=26211) Initiating optimized duplication from @aaaac to @aaaae
02/14/2012 11:10:47 - Info bpdm (pid=5980) started
02/14/2012 11:10:47 - started process bpdm (pid=5980)
02/14/2012 11:10:47 - Info bpdm (pid=5980) requesting nbjm for media
02/14/2012 11:10:47 - granted resource  MediaID=@aaaac;DiskVolume=PureDiskVolume;DiskPool=Main-Disk-Pool;Path=PureDiskVolume;StorageServer=acssbck1.apss.tn.it;MediaServer=wcssbckmedia1.apss.tn.it
02/14/2012 11:10:47 - ended process 0 (pid=-1)
02/14/2012 11:10:51 - Critical bpdm (pid=5980) sts_copy_extent failed: error 2060016 operation not supported
02/14/2012 11:10:52 - Critical bpdm (pid=5980) sts_close_handle failed: 2060022 software error
02/14/2012 11:10:52 - begin writing
02/14/2012 11:10:52 - Critical bpdm (pid=5980) sts_copy_extent failed: error 2060016 operation not supported
02/14/2012 11:10:52 - Critical bpdm (pid=5980) image copy failed: error 2060016: operation not supported
02/14/2012 11:10:53 - Error bpdm (pid=5980) cannot copy image from disk, bytesCopied = 18446744073709551615
02/14/2012 11:10:53 - Critical bpdm (pid=5980) sts_close_handle failed: 2060022 software error
02/14/2012 11:10:53 - Info wcssbckmedia1.apss.tn.it (pid=5980) StorageServer=PureDisk:aitbck1.apss.tn.it; Report=PDDO Stats for (aitbck1.apss.tn.it): scanned: 2 KB, stream rate: 0.00 MB/sec, CR sent: 0 KB, dedup: 100.0%, cache hits: 0 (0.0%)
02/14/2012 11:10:56 - Error bpduplicate (pid=26211) host wcssbckmedia1.apss.tn.it backup id wcssbckmedia1_1329213952 optimized duplication failed, media write error (84).
02/14/2012 11:10:56 - Info wcssbckmedia1.apss.tn.it (pid=5980) StorageServer=PureDisk:acssbck1.apss.tn.it; Report=PDDO Stats for (acssbck1.apss.tn.it): scanned: 1 KB, stream rate: 0.00 MB/sec, CR sent: 0 KB, dedup: 100.0%, cache hits: 0 (0.0%)
02/14/2012 11:10:56 - Error bpduplicate (pid=26211) Duplicate of backupid wcssbckmedia1_1329213952 failed, media write error (84).
02/14/2012 11:10:56 - Error bpduplicate (pid=26211) Status = no images were successfully processed.
02/14/2012 11:10:56 - end Duplicate; elapsed time 0:00:10
media write error  (84)

[Mark_Solutions]

Are both appliances on 1.4.1.1?

It is important that they are the same version

The line "error 2060016 operation not supported" is where we need to look - it indicates something in the setup is wrong

What type of backups are being duplicated?

[AAlmroth]

In the past these errors used to be due to mismatch in versions and type of PureDisk. E.g. using lower or higher versions of NBU media server/clients to MSDP/PDDO etc.

If you can send directly from media server to both, but not between the two 5020, I suggest you check whether there is a firewall blocking the required ports between the 5020s.With opt. dup, the two 5020 communicate directly, hence the network ports need to be open/routing/DNS must work.

/A

[Mark_Solutions]

Good point on the routing etc.

There is a know issue relating to eth 1 being connected and left in DHCP mode that can cause resolution issues - worth checking out - see the relase notes page 21:

http://www.symantec.com/docs/TECH172070

[Mark_Solutions]

One other thing - I have the feeling you need ports 10102 and 10082 open also

Hope this helps

[ddm2]

My server (master and media) are NBU 7.1.0.3.

Appliances have same version (1.4.1.1).

They reside on the same subnet, then no firewall in the middle.

I resolve names with /etc/hosts, and appliances ping each other fine.

On the WebGUI alerts I find:

"Authentication for user 'root' failed

Source:xxxx, Server:yyyy"

where xxxx is the Media Server, yyyy the appliance itself.

It seems an authorization problem. It's strange: when I configured the appliances I never changed default SPA credentials (root P@ssw0rd). I used this user and password when I run the Storage Server wizard, and it worked. On the other hand, if I run backups without duplication, they work!

[Mark_Solutions]

Have you upgraded the appliances since you set it all up?

I know on the 5200 that although it all seems to work the latest patches insist on new highly secure passwords being set to do anything with them.

You can log in and do all sorts of things but if you try sharing the logs directory or opening the patch upload directory it wont work until you have changed the password of the admin user.

Try resetting the passwords (it has been the admin one that causes more issues than the root one - cant remember if the 5020 uses both) and try again

Finally - are you using FQDN in the hosts file - I see everything in the logs refers to a FQDN and again the N5200 tends to revert itself to a short netbios name

Hope this helps

[ddm2]

I upgraded from 1.4.1 to 1.4.1.1, no password change.

In my host file I have all my server infrastructure (Master, medias and appliances) in the form

IP FQDN ShortName

[Mark_Solutions]

OK - that shouldn't change the password then and it does indicate that it is the root password it is looking at.

There is this tech note that relates to PureDisk having the same error:

http://www.symantec.com/docs/TECH160935

Work taking a look - failing that I guess you need to log a call with Symantec

Plus I guess the question I should have asked first .... Did it ever work?

[ddm2]

I've just checked this TECHNOTE.

My topology.ini file is not encrypted, and the SPA password is the dfault (the ssame of root user).

Ths is the first time I try to run the optimized duplication. Normal backups work.

[Mark_Solutions]

The other thing I noticed in realtion to authentication for Pure Disk is that there should only be one root broker in an environment - so the first pool has root broker and the others just have the authetication broker

I am not totally up to date on the 5020's but maybe this is also the case - let me know your setup and I will ask a colleague and get back to you

[Mark_Solutions]

Take a look at the De-Dupe Guide as well - it may help

There are restrictions such as selecting a Storage Unit Group as the destination not being permitted:

http://www.symantec.com/docs/DOC3657

Optimised duplication stating on page 30 - talks about MSDP but a lot is common with Pure Disk

#edit# from a colleague ... if this makes sense to you ... "you need to input the username and password for access to the SPA - you have to create this on the PureDisk GUI as root is disabled by default"

[ddm2]

Hello Mark,

I set up two separate appliances using the wizard, both with all services enabled? Are you sure this is uncorrect?

I configured the storage unit with a common media server with credentials for both appliances. I think this is right.

When I run the wizard, I used the root account for SPA credentials. It should be enabled, because I've created  disk pools and I can access properties of storage servers. Moreover normal backup work. Again, are you sure this is wrong?

Thanks for your effort.

[Mark_Solutions]

ddm2

I am not confident enough with this to ask you to change any of your setup without Symantec Support being involved

However - I do know that you cannot use optimised de-dupe if your target is a Storage Unit Group - it must be a specific storage unit

I am guessing as these are appliances that this does not apply to you.

I would reccomend opening a case to get their advice

I the meantime I will fire a question off to someone else and get back to you if I get a result

[ddm2]

Thanks again.

I'm not using Storage Unit Group but specifica storage unit.

I openede a case, but if you have some news please let me know.

[ddm2 · Accepted solution]

Support solved the problem:  i run these commands from all media server:

tpconfig -update -storage_server target_storage_server -stype PureDisk -sts_user_id root -password <PassWord>

pconfig -update -storage_server destination_storage_server -stype PureDisk -sts_user_id root -password <PassWord>