NDMP Backup Failure

[Madhusudana_D]

HI All,

NDMP backup is failing . We have 8 File system on EMC celerra and we are backing those 8 FS using 8 different policies.

All backups are getting getting successful except one FS backup

Update -

Netbackup Version - 6.0 MP4

OS version - Solaris 10

Celerra Model - NS702G

-I have Created the MAX_FILES_PER_ADD but backup did not work.

-Job is backing up the some data then it fails

I have attached ndmpagent & bpbrm logs

Below is the Detail status of failed job

10/09/2011 18:37:22 - requesting resource whpnbupms01-hcart3-robot-tld-0-server_2
10/09/2011 18:37:22 - requesting resource whpnbupms01.NBU_CLIENT.MAXJOBS.server_2
10/09/2011 18:37:22 - requesting resource whpnbupms01.NBU_POLICY.MAXJOBS.WHP_CSG_Group1
10/09/2011 18:37:22 - granted resource whpnbupms01.NBU_CLIENT.MAXJOBS.server_2
10/09/2011 18:37:22 - granted resource whpnbupms01.NBU_POLICY.MAXJOBS.WHP_CSG_Group1
10/09/2011 18:37:22 - granted resource 980580
10/09/2011 18:37:22 - granted resource IBM.ULT3580-TD3.003
10/09/2011 18:37:22 - granted resource whpnbupms01-hcart3-robot-tld-0-server_2
10/09/2011 18:37:23 - started process bpbrm (pid=15342)
10/09/2011 18:37:23 - connecting
10/09/2011 18:37:23 - connected; connect time: 0:00:00
10/09/2011 18:37:27 - mounting 980580
10/09/2011 18:37:57 - mounted 980580; mount time: 0:00:30
10/09/2011 18:37:57 - positioning 980580 to file 21
10/09/2011 18:39:04 - positioned 980580; position time: 0:01:07
10/09/2011 18:39:04 - begin writing
10/09/2011 19:28:48 - Error ndmpagent (pid=15348) NDMP backup failed, path = /CSG_Group1
10/09/2011 19:29:21 - Error ndmpagent (pid=15348) connection 177cf8 ndmp_message_process_one failed, status = 18 (NDMP_XDR_DECODE_ERR)
10/09/2011 19:29:21 - end writing; write time: 0:50:17
10/10/2011 00:38:56 - job 158303 was restarted as job 158334
NDMP backup failure (99)

Please help me out to resolve this issue

Thanks

Madhusudana

[Mark_Solutions]

Hi

A lot of EMC filers have a thread limit on NDMP of 4 so you may get four streams working and the others will fail.

Try limiting the jobs per client to 4 to see if this resolves it for you.

Go to the Master Server Host Properties - Client Attributes section - add the client name in here, highlight it and set its maximum streams (jobs) to be 4.

Let me know if this helps

[Will_Restore]

http://adsm.org/lists/html/Veritas-bu/2007-02/msg00094.html

Check logs on the NAS device.  Also, 6.0MP4 is very old!  Why not upgrade? 

[Madhusudana_D]

HI Mark,

Thank you for your reply...

I do see that Master Server Host Properties - Client Attributes section maximum streams set to 5.

I need some some clarification

Even though we have set maximum streams to 5, it was working fine earlier (almost for 2 years) but its giving an error now. is that necessary to change it to 4? or its failing due to some other issue

[Madhusudana_D]

Hi Wrobbin,

Could you please let me know that which log do i need to check on EMC Celerra.

[Yogesh9881]

TURNING NDMP LOGGING ON: $.server_config server_x "logsys set severity NDMP=LOG_DBG2"
$.server_config server_x "logsys set severity PAX=LOG_DBG2"

TURNING NDMP LOGGING OFF: $.server_config server_x "logsys set severity NDMP=LOG_ERR"
$.server_config server_x "logsys set severity PAX=LOG_ERR"

Using the server_log command you can redirect the server_log to a file using

server_log server_2 -a -s > /tmp/NDMP_Logs.txt

Note: If implementing any 'DEBUG' levels of NDMP logging, it is highly recommended that you setup the following command to pipe the Server Log into a file so that all debug information is capture. Otherwise the Server Log wraps too quickly and debug logging can wrap.
$server_log server_2 -f -s > s2log.log & [$jobs #fg %1 ctrl + c]

[Mark_Solutions]

Sorry for not replying earlier but ive been out on site.

Take a look at this which hopefully may help you, especiually as it has been working but has now started failing and may just indicate the growth has caused the issue:

http://www.symantec.com/docs/TECH47412

[Madhusudana_D]

Hi Mark,

I have already tried that option but no use :(

As wrobbin said, we are suspecting the issue might be from celerra. We will enable the log and check on that.

Thanks for ur input

Thanks,

Madhusudana

[Omar_Villa]

Can u create a tmp policy and include only the folder is failing? Run a test and let us know, but.is true celerras are limited in their num of streams, drives and block size.

[Madhusudana_D]

Hi Omar,

We have been running the backup in single stream.

But we are not able find the particular folder for which backup is failing. its not present in logs too.

But backup is getting successful for other FS even though size of the data is more compare to effected FS bakup.

Thanks,

Madhusudana

[Madhusudana_D]

Hi All,

Issue is resolved automatically..

We haven't made any changes in Celerra and as well as in Netbackup but issue is resolved now.

No idea what was the problem :)

Thank you all for ur help!!!!

Thanks,

Madhusudana D

[Yogesh9881]

Issue is resolved automatically.....