10-10-2011 12:56 AM
HI All,
NDMP backup is failing . We have 8 File system on EMC celerra and we are backing those 8 FS using 8 different policies.
All backups are getting getting successful except one FS backup
Update -
Netbackup Version - 6.0 MP4
OS version - Solaris 10
Celerra Model - NS702G
-I have Created the MAX_FILES_PER_ADD but backup did not work.
-Job is backing up the some data then it fails
I have attached ndmpagent & bpbrm logs
Below is the Detail status of failed job
10/09/2011 18:37:22 - requesting resource whpnbupms01-hcart3-robot-tld-0-server_2
10/09/2011 18:37:22 - requesting resource whpnbupms01.NBU_CLIENT.MAXJOBS.server_2
10/09/2011 18:37:22 - requesting resource whpnbupms01.NBU_POLICY.MAXJOBS.WHP_CSG_Group1
10/09/2011 18:37:22 - granted resource whpnbupms01.NBU_CLIENT.MAXJOBS.server_2
10/09/2011 18:37:22 - granted resource whpnbupms01.NBU_POLICY.MAXJOBS.WHP_CSG_Group1
10/09/2011 18:37:22 - granted resource 980580
10/09/2011 18:37:22 - granted resource IBM.ULT3580-TD3.003
10/09/2011 18:37:22 - granted resource whpnbupms01-hcart3-robot-tld-0-server_2
10/09/2011 18:37:23 - started process bpbrm (pid=15342)
10/09/2011 18:37:23 - connecting
10/09/2011 18:37:23 - connected; connect time: 0:00:00
10/09/2011 18:37:27 - mounting 980580
10/09/2011 18:37:57 - mounted 980580; mount time: 0:00:30
10/09/2011 18:37:57 - positioning 980580 to file 21
10/09/2011 18:39:04 - positioned 980580; position time: 0:01:07
10/09/2011 18:39:04 - begin writing
10/09/2011 19:28:48 - Error ndmpagent (pid=15348) NDMP backup failed, path = /CSG_Group1
10/09/2011 19:29:21 - Error ndmpagent (pid=15348) connection 177cf8 ndmp_message_process_one failed, status = 18 (NDMP_XDR_DECODE_ERR)
10/09/2011 19:29:21 - end writing; write time: 0:50:17
10/10/2011 00:38:56 - job 158303 was restarted as job 158334
NDMP backup failure (99)
Please help me out to resolve this issue
Thanks
Madhusudana
10-10-2011 02:39 AM
Hi
A lot of EMC filers have a thread limit on NDMP of 4 so you may get four streams working and the others will fail.
Try limiting the jobs per client to 4 to see if this resolves it for you.
Go to the Master Server Host Properties - Client Attributes section - add the client name in here, highlight it and set its maximum streams (jobs) to be 4.
Let me know if this helps
10-10-2011 07:46 AM
http://adsm.org/lists/html/Veritas-bu/2007-02/msg00094.html
Check logs on the NAS device. Also, 6.0MP4 is very old! Why not upgrade?
10-12-2011 07:15 PM
HI Mark,
Thank you for your reply...
I do see that Master Server Host Properties - Client Attributes section maximum streams set to 5.
I need some some clarification
Even though we have set maximum streams to 5, it was working fine earlier (almost for 2 years) but its giving an error now. is that necessary to change it to 4? or its failing due to some other issue
10-12-2011 07:57 PM
Hi Wrobbin,
Could you please let me know that which log do i need to check on EMC Celerra.
10-13-2011 12:36 AM
TURNING NDMP LOGGING ON: $.server_config server_x "logsys set severity NDMP=LOG_DBG2"
$.server_config server_x "logsys set severity PAX=LOG_DBG2"
TURNING NDMP LOGGING OFF: $.server_config server_x "logsys set severity NDMP=LOG_ERR"
$.server_config server_x "logsys set severity PAX=LOG_ERR"
Using the server_log command you can redirect the server_log to a file using
server_log server_2 -a -s > /tmp/NDMP_Logs.txt
Note: If implementing any 'DEBUG' levels of NDMP logging, it is highly recommended that you setup the following command to pipe the Server Log into a file so that all debug information is capture. Otherwise the Server Log wraps too quickly and debug logging can wrap.
$server_log server_2 -f -s > s2log.log & [$jobs #fg %1 ctrl + c]
10-13-2011 02:19 AM
Sorry for not replying earlier but ive been out on site.
Take a look at this which hopefully may help you, especiually as it has been working but has now started failing and may just indicate the growth has caused the issue:
10-13-2011 03:25 AM
Hi Mark,
I have already tried that option but no use :(
As wrobbin said, we are suspecting the issue might be from celerra. We will enable the log and check on that.
Thanks for ur input
Thanks,
Madhusudana
10-13-2011 05:12 AM
10-13-2011 08:59 PM
Hi Omar,
We have been running the backup in single stream.
But we are not able find the particular folder for which backup is failing. its not present in logs too.
But backup is getting successful for other FS even though size of the data is more compare to effected FS bakup.
Thanks,
Madhusudana
10-20-2011 07:36 PM
Hi All,
Issue is resolved automatically..
We haven't made any changes in Celerra and as well as in Netbackup but issue is resolved now.
No idea what was the problem :)
Thank you all for ur help!!!!
Thanks,
Madhusudana D
10-20-2011 11:27 PM
Issue is resolved automatically.....