06-09-2013 07:56 PM
Hi all,
I have a bit strange backup problem the backup running find for weekly full but not on the daily incremental. Bellow is the message from the screen.
The version all thes asam Netbackup Enterprise 7.5.0.2. Master is RedHat 6.1, Media Sever is HPUX 11 v2 and the client is Solaris10 (actully similar thing also happened to windows, hpux client going the the same media server)
Any solution?
The problem Daily Incremental Backup:
06/10/2013 10:20:41 - requesting resource ebsbck.NBU_POLICY.MAXJOBS.OSM01_FS_OS
06/10/2013 10:20:41 - granted resource ebsbck.NBU_CLIENT.MAXJOBS.osm01-bck
06/10/2013 10:20:41 - granted resource ebsbck.NBU_POLICY.MAXJOBS.OSM01_FS_OS
06/10/2013 10:20:41 - granted resource 400575
06/10/2013 10:20:41 - granted resource ESL02_Drive4
06/10/2013 10:20:41 - granted resource ESL02_EBS2_MPX
06/10/2013 10:20:41 - estimated 22594953 kbytes needed
06/10/2013 10:20:41 - Info nbjm (pid=7374) started backup (backupid=osm01-bck_1370830841) job for client osm01-bck, policy OSM01_FS_OS, schedule Daily_Incre on storage unit ESL02_EBS2_MPX
the backup then stucked for the long time
The OK Full backup
06/10/2013 10:38:04 - connected; connect time: 0:00:00
06/10/2013 10:39:09 - Info bptm (pid=24291) media id 401147 mounted on drive index 0, drivepath /dev/rmt/c17t0d0BESTnb, drivename ESL02_Drive6, copy 1
06/10/2013 10:39:09 - mounted 401147; mount time: 0:01:05
06/10/2013 10:39:09 - positioning 401147 to file 2
06/10/2013 10:39:13 - positioned 401147; position time: 0:00:04
06/10/2013 10:39:13 - begin writing...
it will finish OK
06-09-2013 08:06 PM
At the end for the Daily Incremental actually error status 40, but definitely not really because of the network, since the Weekly Full backup running from the same client and same media server running fine.
BTW I have also restarted the media server several times, but it no help for the Daily Incremental.
Regards,
Iwan Tamimi
06-09-2013 09:21 PM
Can you see if bpbrm and bptm processes are started on the media server?
Please check that these log folders exist on the media server.
Copy log files to reflect process name (e.g. bpbrm.txt) and post as File attachments.
06-09-2013 10:09 PM
If have seen this before. First try to set CLIENT_READ_TIMEOUT = 3600 in bp.conf on the media server and client. The daily backup is failing because it encounter a directory with a huge amount of files. Every files need to be eveulated for backup and this causes the timeout. The full does not evaluate - every file is in scope. Do you also see Error = 62 timer expired" in the job details ?
You can debug where the directory is using this procedure found at : http://www.symantec.com/docs/TECH143355
To view the last file / directory where the backup is hanging, do the following on the client:
1. Create the bpbkar debug log directory by running mkdir /<install_path>/netbackup/logs/bpbkar. Generated log information will be placed in this directory in a file named log.mmddyy (where mm = month, dd = date, yy = year).
2. Create a file named touch /<install_path>/netbackup/bpbkar_path_tr to enable extended ../netbackup/logs/bpbkar/* debug logging
3. Enable verbose 5 client logging through the NetBackup GUI on the master.
Under NetBackup Management>Host Properties>CLIENTS>double click client-name >properties>Logging>Global>5
Or add ' VERBOSE = 5 ' into the /usr/openv/netbackup/bp.conf file on the client server
4. No restart needed is needed after these settings are added
5. Run a backup to generate the log information
Note that the use of this "touch" file 'bpbkar_path_tr' will cause larger bpbkar logs than the usual.
Run command to start the test
# cd /usr/openv/netbackup/bin
# ./bpbkar -dt 0 -r 888 -nocont /YOUR-DATA-PATH-TO-TEST-HERE > /dev/null
If BPBKAR command stops or hangs, view the end of the /usr/openv/netbackup/logs/bpbkar/log.<date> file for the last file name and possible OS messages.
.
.
17:03:25.297 [642176] <2> bpbkar SelectFile: INF - cwd = /var
17:03:25.297 [642176] <2> bpbkar SelectFile: INF - path = file22
17:03:25.299 [642176] <4> bpbkar PrintFile: /var/file22
(PrintFile message posted = Received info OK, sent to media server storage device)
17:03:25.300 [642176] <2> bpbkar SelectFile: INF - cwd = /var
17:03:25.300 [642176] <2> bpbkar SelectFile: INF - path = file23
(**Hang** NOTE: No PrintFile message)
06-09-2013 10:39 PM
Before I posted the log. I should tell , the Daily incremental backup was running for months fine. Then suddenly happens, other backups like Weekly full. Daily full, database agent are running fine till now.
I think it is not the directory with lots of files problem since all the client have quite small amount of files (like bellow 1000) and it is not affecting one particular client, almost all clients (Windows, Linux, solaris) on that media server, and during certain times suddenly the backup running fine, then suddenly hang again. BTW this one also happened on other media server, it happened for several days then it is OK by itself. I believe it is something to do with the media server or the tape?
Regards,
Iwan Tamimi
06-09-2013 10:52 PM