11-09-2015 11:37 AM
Master server: uskssb5501 win 2003 netbackup 7.6
cleint:hpdmpf01 netbackup 7.1
Monthly and Weekly backups are failing with error 13. Only G drive backup is failing . I have got below detailed output.
11/6/2015 6:00:05 PM - Info nbjm(pid=6352) starting backup job (jobid=129544) for client hpdmpf01, policy hpdmpf01_tape, schedule Monthly-Full
11/6/2015 6:00:05 PM - Info nbjm(pid=6352) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=129544, request id:{BED4490C-4033-4CDC-8F7C-0930380C4C12})
11/6/2015 6:00:05 PM - requesting resource uskssb5501-hcart-robot-tld-0
11/6/2015 6:00:05 PM - requesting resource uskssb5501.NBU_CLIENT.MAXJOBS.hpdmpf01
11/6/2015 6:00:05 PM - requesting resource uskssb5501.NBU_POLICY.MAXJOBS.hpdmpf01_tape
11/6/2015 6:00:05 PM - granted resource uskssb5501.NBU_CLIENT.MAXJOBS.hpdmpf01
11/6/2015 6:00:05 PM - granted resource uskssb5501.NBU_POLICY.MAXJOBS.hpdmpf01_tape
11/6/2015 6:00:05 PM - granted resource 0142L4
11/6/2015 6:00:05 PM - granted resource HP.ULTRIUM4-SCSI.001
11/6/2015 6:00:05 PM - granted resource uskssb5501-hcart-robot-tld-0
11/6/2015 6:00:41 PM - estimated 3421693525 Kbytes needed
11/6/2015 6:00:41 PM - Info nbjm(pid=6352) started backup (backupid=hpdmpf01_1446854440) job for client hpdmpf01, policy hpdmpf01_tape, schedule Monthly-Full on storage unit uskssb5501-hcart-robot-tld-0
11/6/2015 6:01:00 PM - Info bpbrm(pid=4356) hpdmpf01 is the host to backup data from
11/6/2015 6:01:00 PM - Info bptm(pid=10700) using 262144 data buffer size
11/6/2015 6:01:00 PM - Info bpbrm(pid=4356) telling media manager to start backup on client
11/6/2015 6:01:00 PM - Info bptm(pid=10700) using 40 data buffers
11/6/2015 6:01:07 PM - Info bpbrm(pid=11664) sending bpsched msg: CONNECTING TO CLIENT FOR hpdmpf01_1446854440
11/6/2015 6:01:07 PM - connecting
11/6/2015 6:01:08 PM - Info bptm(pid=1952) setting receive network buffer to 1049600 bytes
11/6/2015 6:01:08 PM - Info bpbrm(pid=11664) start bpbkar32 on client
11/6/2015 6:01:08 PM - connected; connect time: 0:00:01
11/6/2015 6:01:09 PM - Info bpbkar32(pid=16260) Backup started
11/6/2015 6:01:09 PM - Info bpbrm(pid=11664) Sending the file list to the client
11/6/2015 6:01:28 PM - mounted
11/6/2015 6:01:34 PM - positioning 0142L4 to file 15
11/6/2015 6:03:25 PM - positioned 0142L4; position time: 0:01:51
11/6/2015 6:03:25 PM - begin writing
11/6/2015 11:03:28 PM - current media 0142L4 complete, requesting next resource Any
11/6/2015 11:03:28 PM - current media -- complete, awaiting next media Any Reason: Drives are in use, Media Server: uskssb5501,
Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
Volume Pool: Monthly, Storage Unit: uskssb5501-hcart-robot-tld-0, Drive Scan Host: N/A
11/6/2015 11:05:24 PM - granted resource 0021L4
11/6/2015 11:05:24 PM - granted resource HP.ULTRIUM4-SCSI.001
11/6/2015 11:05:24 PM - granted resource uskssb5501-hcart-robot-tld-0
11/6/2015 11:05:24 PM - end writing; write time: 5:01:59
11/6/2015 11:05:24 PM - mounting 0021L4
11/6/2015 11:06:42 PM - mounted; mount time: 0:01:18
11/6/2015 11:06:46 PM - positioning 0021L4 to file 1
11/6/2015 11:07:01 PM - positioned 0021L4; position time: 0:00:15
11/6/2015 11:07:01 PM - begin writing
11/7/2015 8:54:33 AM - current media 0021L4 complete, requesting next resource Any
11/7/2015 8:54:33 AM - current media -- complete, awaiting next media Any Reason: Drives are in use, Media Server: uskssb5501,
Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
Volume Pool: Monthly, Storage Unit: uskssb5501-hcart-robot-tld-0, Drive Scan Host: N/A
11/7/2015 8:55:52 AM - granted resource GXK612
11/7/2015 8:55:52 AM - granted resource HP.ULTRIUM4-SCSI.001
11/7/2015 8:55:52 AM - granted resource uskssb5501-hcart-robot-tld-0
11/7/2015 8:55:53 AM - end writing; write time: 9:48:52
11/7/2015 8:55:53 AM - mounting GXK612
11/7/2015 8:57:20 AM - mounted; mount time: 0:01:27
11/7/2015 8:57:26 AM - positioning GXK612 to file 1
11/7/2015 8:57:42 AM - positioned GXK612; position time: 0:00:16
11/7/2015 8:57:42 AM - begin writing
11/7/2015 9:25:16 AM - Warning bpbrm(pid=11664) from client hpdmpf01: WRN - can't open file: G:\SYS\SH\Component Prep\5A94465D.tmp (WIN32 5: Access is denied. )
11/7/2015 1:28:55 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\FVP Prep\IMG_0631.jpg (WIN32 13: The data is invalid. )
11/7/2015 1:28:55 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\FVP Prep\LIRs 9-1-14 to 9-11-15.pdf (WIN32 13: The data is invalid. )
11/7/2015 1:28:55 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\FVP Prep\MERGE - Schedule.docx (WIN32 13: The data is invalid. )
11/7/2015 1:28:55 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\FVP Prep\McPherson Internal Audit 2015 Schedule - V3.pdf (WIN32 13: The data is invalid. )
11/7/2015 4:06:21 PM - Warning bpbrm(pid=11664) from client hpdmpf01: WRN - can't open file: G:\SYS\SH\MANNING\Headcount\~$2015 Plan HC Summary.xlsx (WIN32 5: Access is denied. )
11/7/2015 4:24:07 PM - current media GXK612 complete, requesting next resource Any
11/7/2015 4:24:07 PM - current media -- complete, awaiting next media Any Reason: Drives are in use, Media Server: uskssb5501,
Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
Volume Pool: Monthly, Storage Unit: uskssb5501-hcart-robot-tld-0, Drive Scan Host: N/A
11/7/2015 4:25:24 PM - granted resource 0030L4
11/7/2015 4:25:24 PM - granted resource HP.ULTRIUM4-SCSI.001
11/7/2015 4:25:24 PM - granted resource uskssb5501-hcart-robot-tld-0
11/7/2015 4:25:24 PM - end writing; write time: 7:27:42
11/7/2015 4:25:24 PM - mounting 0030L4
11/7/2015 4:26:36 PM - mounted; mount time: 0:01:12
11/7/2015 4:26:42 PM - positioning 0030L4 to file 1
11/7/2015 4:26:58 PM - positioned 0030L4; position time: 0:00:16
11/7/2015 4:26:58 PM - begin writing
11/7/2015 4:51:17 PM - Warning bpbrm(pid=11664) from client hpdmpf01: WRN - can't open file: G:\SYS\SH\MST\1 - MS&T weekly update summaries\A0BD6BF2.tmp (WIN32 32: The process cannot access the file because it is being used by another process. )
11/7/2015 5:34:20 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\MST\Personal Folders\Sean Tomlinson\Archive\Back up 03 Feb 2015\Archive\Projects\Static cling in equipment prep\GPA Photographs\2014-01-09 08.36.34.jpg (WIN32 13: The data is invalid. )
11/7/2015 5:34:20 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\MST\Personal Folders\Sean Tomlinson\Archive\Back up 03 Feb 2015\Archive\Projects\Static cling in equipment prep\GPA Photographs\2014-01-09 08.40.21.jpg (WIN32 13: The data is invalid. )
11/7/2015 5:34:48 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\MST\Personal Folders\Sean Tomlinson\Archive\Back up 03 Feb 2015\Pictures\M15 and M16 lines\2014-01-02 14.04.58.jpg (WIN32 13: The data is invalid. )
11/7/2015 5:34:48 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\MST\Personal Folders\Sean Tomlinson\Archive\Back up 03 Feb 2015\Pictures\M15 and M16 lines\2014-01-02 14.05.05.jpg (WIN32 13: The data is invalid. )
11/7/2015 5:43:58 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\MST\Personal Folders\Sean Tomlinson\Back up 04 August 2015\Sean Tomlinson\Pictures\M15 and M16 lines\2014-01-02 13.55.44.jpg (WIN32 13: The data is invalid. )
11/7/2015 5:43:59 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\MST\Personal Folders\Sean Tomlinson\Back up 04 August 2015\Sean Tomlinson\Pictures\M15 and M16 lines\2014-01-02 13.56.15.jpg (WIN32 13: The data is invalid. )
11/7/2015 5:44:00 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\MST\Personal Folders\Sean Tomlinson\Back up 04 August 2015\Sean Tomlinson\Pictures\M15 and M16 lines\2014-01-02 14.00.45.jpg (WIN32 13: The data is invalid. )
11/7/2015 5:44:00 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\MST\Personal Folders\Sean Tomlinson\Back up 04 August 2015\Sean Tomlinson\Pictures\M15 and M16 lines\2014-01-02 14.01.16.jpg (WIN32 13: The data is invalid. )
11/7/2015 5:44:01 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\MST\Personal Folders\Sean Tomlinson\Back up 04 August 2015\Sean Tomlinson\Pictures\M15 and M16 lines\2014-01-02 14.03.01.jpg (WIN32 13: The data is invalid. )
11/7/2015 5:44:01 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\MST\Personal Folders\Sean Tomlinson\Back up 04 August 2015\Sean Tomlinson\Pictures\M15 and M16 lines\2014-01-02 14.03.30.jpg (WIN32 13: The data is invalid. )
11/7/2015 5:45:29 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\MST\Personal Folders\Sean Tomlinson\Back up 04 August 2015\Sean Tomlinson\Projects\Archive 2014\Static cling in equipment prep\GPA Photographs\2014-01-09 08.52.33.jpg (WIN32 13: The data is invalid. )
11/7/2015 5:45:30 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\MST\Personal Folders\Sean Tomlinson\Back up 04 August 2015\Sean Tomlinson\Projects\Archive 2014\Static cling in equipment prep\GPA Photographs\2014-01-09 08.52.44.jpg (WIN32 13: The data is invalid. )
11/7/2015 5:53:18 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\MST\Personal Folders\Sean Tomlinson\Back up 07 April 2015\Archive\Hospira Pictures 1-14\M6 and M18\2014-01-03 15.39.55.jpg (WIN32 13: The data is invalid. )
11/7/2015 5:53:18 PM - Error bpbrm(pid=11664) from client hpdmpf01: ERR - failure reading file: G:\SYS\SH\MST\Personal Folders\Sean Tomlinson\Back up 07 April 2015\Archive\Hospira Pictures 1-14\M6 and M18\2014-01-03 15.40.02.jpg (WIN32 13: The data is invalid. )
11/7/2015 7:42:01 PM - Warning bpbrm(pid=11664) from client hpdmpf01: WRN - can't open file: G:\SYS\SH\PD\EMPLOYEE FOLDERS\Norton, Cassandra\1-DAPTOMYCIN\~$Project list.xlsx (WIN32 32: The process cannot access the file because it is being used by another process. )
11/7/2015 7:56:04 PM - Warning bpbrm(pid=11664) from client hpdmpf01: WRN - can't open file: G:\SYS\SH\PD\EMPLOYEE FOLDERS\Walters, Tristan\Instrument\~$110615 Quart Calibration Calcs.xlsx (WIN32 32: The process cannot access the file because it is being used by another process. )
11/7/2015 9:33:49 PM - Warning bpbrm(pid=11664) from client hpdmpf01: WRN - can't open file: G:\SYS\SH\PREP Support\Module Requirements for T&E card CBrake.pdf (WIN32 5: Access is denied. )
11/7/2015 9:41:35 PM - Warning bpbrm(pid=11664) from client hpdmpf01: WRN - can't open file: G:\SYS\SH\Presentations\1TPDP\CNC Sign Project\0-Final Draft Deck\~$02.1WW_signs.pptx (WIN32 5: Access is denied. )
11/7/2015 9:42:06 PM - Warning bpbrm(pid=11664) from client hpdmpf01: WRN - can't open file: G:\SYS\SH\Presentations\1TPDP\CNC Sign Project\West Wing Signs\~$West Wing CNC Sign Info.xlsx (WIN32 5: Access is denied. )
11/7/2015 10:33:41 PM - Warning bpbrm(pid=11664) from client hpdmpf01: WRN - can't open file: G:\SYS\SH\Presentations\Vidmar Cabinet\~$09-14-15 locations.xlsx (WIN32 32: The process cannot access the file because it is being used by another process. )
11/7/2015 10:33:41 PM - Warning bpbrm(pid=11664) from client hpdmpf01: WRN - can't open file: G:\SYS\SH\Presentations\Vidmar Cabinet\~$10-12-15 locations.xlsx (WIN32 32: The process cannot access the file because it is being used by another process. )
11/7/2015 11:28:42 PM - Warning bpbrm(pid=11664) from client hpdmpf01: WRN - can't open file: G:\SYS\SH\QC\BQA\Validation Tracking-Reports\Remediation\Spreadsheets\BET Remedation (A-M) notes.xlsx (WIN32 5: Access is denied. )
11/8/2015 12:49:08 AM - current media 0030L4 complete, requesting next resource Any
11/8/2015 12:49:08 AM - current media -- complete, awaiting next media Any Reason: Drives are in use, Media Server: uskssb5501,
Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
Volume Pool: Monthly, Storage Unit: uskssb5501-hcart-robot-tld-0, Drive Scan Host: N/A
11/8/2015 12:50:25 AM - granted resource 0024L4
11/8/2015 12:50:25 AM - granted resource HP.ULTRIUM4-SCSI.001
11/8/2015 12:50:25 AM - granted resource uskssb5501-hcart-robot-tld-0
11/8/2015 12:50:25 AM - end writing; write time: 8:23:27
11/8/2015 12:50:25 AM - mounting 0024L4
11/8/2015 12:51:39 AM - mounted; mount time: 0:01:14
11/8/2015 12:51:45 AM - positioning 0024L4 to file 1
11/8/2015 12:52:01 AM - positioned 0024L4; position time: 0:00:16
11/8/2015 12:52:01 AM - begin writing
11/8/2015 1:28:51 AM - Warning bpbrm(pid=11664) from client hpdmpf01: WRN - can't open file: G:\SYS\SH\QC\ENVCTRL\Env Data\Data 2015\F8E96FE1.tmp (WIN32 5: Access is denied. )
11/8/2015 5:24:04 AM - Warning bpbrm(pid=11664) from client hpdmpf01: WRN - can't open file: G:\SYS\SH\QC\Investigations\Selley\Misc\~$Ampul false reject QES data.xlsx (WIN32 32: The process cannot access the file because it is being used by another process. )
11/8/2015 5:46:03 AM - Warning bpbrm(pid=11664) from client hpdmpf01: WRN - can't open file: G:\SYS\SH\QC\MQ Visual Inspection Team\Defect Alignment\~$Missing Pictures.xlsx (WIN32 32: The process cannot access the file because it is being used by another process. )
11/8/2015 7:02:36 AM - Error bpbrm(pid=11664) socket read failed, An existing connection was forcibly closed by the remote host. (10054)
11/8/2015 7:02:36 AM - Error bpbrm(pid=11664) could not send server status message
11/8/2015 7:02:36 AM - Error bptm(pid=1952) socket operation failed - 10054 (at child.c.1304)
11/8/2015 7:02:36 AM - Error bptm(pid=1952) unable to perform read from client socket, connection may have been broken
11/8/2015 7:02:37 AM - Info bpbrm(pid=4356) child done, status 13
11/8/2015 7:02:37 AM - Info bpbrm(pid=4356) sending message to media manager: STOP BACKUP hpdmpf01_1446854440
11/8/2015 7:03:19 AM - Info bpbrm(pid=4356) media manager for backup id hpdmpf01_1446854440 exited with status 42: network read failed
11/8/2015 7:03:19 AM - end writing; write time: 6:11:18
file read failed (13)
11-10-2015 01:59 AM
WAY too many errors such as these:
(WIN32 5: Access is denied. )
(WIN32 32: The process cannot access the file because it is being used by another process. )
Is this a file server getting backed up?
Which OS and patch level?
Is NBU Client Service on this client started with Domain Account with sufficient access permissions?
Can you confirm that VSS on this client is 'healthy' and has enough space to create snapshots?
Do you have 'Allow Multiple Data Streams' in Policy Attributes?
Any reason why client is on such an old NBU version? (Although it is supported, things simply work better when versions are not too far behind.)
Do you have log folders on client and media server?
bpfis and bpbkar on client (logging level of 3 should be sufficient)
bpbrm and bptm on media server (logging level of 3).
11-10-2015 12:19 PM
Thanks for your response Marriane.
Is this a file server getting backed up? Not a File server G drive backup is failing only
Which OS and patch level?Window 2003 service pack 2
Is NBU Client Service on this client started with Domain Account with sufficient access permissions? Its running with local system account
Can you confirm that VSS on this client is 'healthy' and has enough space to create snapshots? yes it has enough space
Do you have 'Allow Multiple Data Streams' in Policy Attributes? yes we do have allow multiple data streams
Any reason why client is on such an old NBU version? (Although it is supported, things simply work better when versions are not too far behind.)Yes we will upgrade the client but i am interested in fixing without upgrading it.
Do you have log folders on client and media server?
bpfis and bpbkar on client (logging level of 3 should be sufficient)enabled
bpbrm and bptm on media server (logging level of 3).Enabled the logging for these two only and global logging is on no logging status
11-10-2015 01:01 PM
A few notes:
Windows 2003 is obsolete. Need to upgrade OS to current version.
--
Regarding (WIN32 32: The process cannot access the file because it is being used by another process. )
Files beginning with ~ (such as ~$2015 Plan HC Summary.xlsx) are temporary files and can be excluded from backup (add ~* to client Exclude List)
Files ending with .tmp (such as 5A94465D.tmp) are also temporary files and can be excluded from backup (add *.tmp to client Exclude List)
--
Regarding (WIN32 13: The data is invalid. )
Many files ending with .jpg seem to be corrupt?
--
Can this server even back itself up using Windows Backup Utility? (not NetBackup)