cancel
Showing results for 
Search instead for 
Did you mean: 

Backup failing with error code 129. Very often

Mack_Disouza
Level 4

Hello Friends,

Could you please let me know what is the other solution for error code (129)

Increasing storage disk space everytime is not good. In our environment,I m new in Netbackup, I am facing this error code very often. File system gets filled on media server and DB LOG backups gets failed. Tell me what all i need to check ?

===============================================

Master Server Information :-
=================================
[root@lterhdqnbkm ~]# cat /usr/openv/netbackup/bin/version
NetBackup-RedHat2.6 7.1
[root@lterhdqnbkm ~]# uname -a
Linux lterhdqnbkm 2.6.18-128.1.14.el5 #1 SMP Mon Jun 1 15:52:58 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
[root@lterhdqnbkm ~]# hostname
lterhdqnbkm
===============================================
Media Server name :- Xterhdqnbk01.
==============================================================
root@XTERHDQNBK01 # cat /usr/openv/netbackup/bin/version
NetBackup-Solaris10 7.1

 

1 ACCEPTED SOLUTION

Accepted Solutions

Mark_Solutions
Level 6
Partner Accredited Certified

Looking at your high / low watermark screen shot it looks like all of your disk staging is in the same locations /archives/xxx

If that is the case then you may have issue as the first one may fill up the file system preventing the others from working

Unless each of those is on seperate file systems than you have a fundamental problem with your setup.

Each disk staging storage until should be on its own file system to prevent them having an affect on each other.

The one you write to may have completely emptied itself but still has no disk space due to the others

So you may need to re-structure the whole environment for it to work properly

I also see in your log that the predicted size of the backup is 0kb, indicating that the client has not been backed up before - this can add an extra overhead as NetBackup does its own cacluations on how much space to free up for an unknown backup so plenty of free space is generally needed

Let us know about where these folders are first - but if they are all on the same file system then it just wont work correctly

Hope this helps

View solution in original post

13 REPLIES 13

RamNagalla
Moderator
Moderator
Partner    VIP    Certified

hi Mack,

does it basic disk STU?

you need to investigate the reasons for DSU getting filled very offen

capacity issue 

or  

any other break down of functionlaty.

capacity issue:-

are you sending the data to DSU more then its capacity?

how much of data you are sending to disk?

disk space should be > the amount of data that you are sending to disk * Retenction period.

break down of functionlity:-

how you are doing the clean up of DSU? are you using disk staging or any other  duplication method for the DSU?

did your clean up jobs or duplication jobs failing?

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
I agree with Nagalla - you need to tell us more. Have you configured staging to tape? If so, are duplications running often enough? And running successfully? Without regular duplications to tape, disk images will be kept until expiration date is reached.

Mack_Disouza
Level 4

Hello Nagalla/Marianne,

Yes disk staging is in place because i can see __DSSU_POLICY_XTERHDQNBK01-stu_disk_PRD running

and duplication jobs run often too but duplication jobs are successful.

How can i check the capacity issues ? I am new to Netbackup. Suppose one of the file system on media server is 100% full and DB-log backups failing with error code 129.

What do i need to do in that case ? 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

What are your Low and High Water marks on the STU?

Important Disk Storage Unit requirements:

  1. Assign dedicated lun/disk volume to DSU 
    Do no allow any other folders/files on the DSU.
  2. One DSU per lun/volume
    Do not create multiple DSU's on the same lun.

The above ensure that NBU can properly manage disk space and cleanup properly when HWM is reached.  

Do you know if all disk backups are getting duplicated to tape before next backup window opens?
Important to ensure that duplications are not getting behind.

If all duplications are successful, NBU should pause backups when disk reaches High Water Mark, run cleanup before continuing with backup. This means backups wil only fail with status 129 if NBU cannot find enough duplicated backups to cleanup.

Some DSU TNs:

http://www.symantec.com/docs/TECH66149

DSSU cleanup behavior

http://www.symantec.com/docs/TECH44719

DSSU Relocation Behavior 

Create the following log folders to check/troubleshoot cleanup:
bpdm and admin

Mack_Disouza
Level 4

How can we do the file system cleaning ?

Do we have any command for this ? and what are the important things that we need to check before cleaning ?

RamNagalla
Moderator
Moderator
Partner    VIP    Certified

the only recommended method is to expire the images on the disk through Netbackup.

as you are seeing netbckup duplication is successfull, it should expire the images automatically as soon as it reaches the high water mark.

you did not say ,.if you are seeing the image cleanup job failurs in activity moniter or not...

if you are seeing the image cleanup job failurs that is the place you need to fix first..

 

comming to capacity management..

you need below numbers

how much data you are sending to disk  ?

how much data your duplication jobs( staging jobs) can  send to tape?

size of the disk storage unit?

what is the retenction of the images that you are keeping ?

 

Mack_Disouza
Level 4

DSSU and duplication job failed with error code 191.

Job :- __DSSU_POLICY_XTERHDQNBK02-stu_disk_TAP

======================================

03/16/2013 13:00:59 - Info nbjm (pid=12884) requesting NO_STORAGE_UNIT resources from RB for backup job (jobid=1297783, request id:{2A70ED9C-8E31-11E2-A945-CADA5A7BA41B})
03/16/2013 13:00:59 - requesting resource lterhdqnbkm.NBU_CLIENT.MAXJOBS.XTERHDQNBK02
03/16/2013 13:00:59 - requesting resource lterhdqnbkm.NBU_POLICY.MAXJOBS.__DSSU_POLICY_XTERHDQNBK02-stu_disk_TAP
03/16/2013 13:00:59 - granted resource  lterhdqnbkm.NBU_CLIENT.MAXJOBS.XTERHDQNBK02
03/16/2013 13:00:59 - granted resource  lterhdqnbkm.NBU_POLICY.MAXJOBS.__DSSU_POLICY_XTERHDQNBK02-stu_disk_TAP
03/16/2013 13:00:59 - estimated 0 kbytes needed
03/16/2013 13:00:59 - begin Parent Job
03/16/2013 13:00:59 - begin Disk Staging: Start Notify Script
03/16/2013 13:00:59 - Info RUNCMD (pid=20153) started
03/16/2013 13:00:59 - Info RUNCMD (pid=20153) exiting with status: 0
Operation Status: 0
03/16/2013 13:00:59 - end Disk Staging: Start Notify Script; elapsed time 0:00:00
03/16/2013 13:00:59 - begin Disk Staging: Execute Script
03/16/2013 13:01:00 - started process bpbrm (pid=8317)
03/16/2013 13:12:31 - end writing
Operation Status: 191
03/16/2013 13:12:31 - end Disk Staging: Execute Script; elapsed time 0:11:32
03/16/2013 13:12:31 - begin Disk Staging: Stop On Error
Operation Status: 0
03/16/2013 13:12:31 - end Disk Staging: Stop On Error; elapsed time 0:00:00
03/16/2013 13:12:31 - begin Disk Staging: End Notify Script
03/16/2013 13:12:31 - Info RUNCMD (pid=22917) started
03/16/2013 13:12:31 - Info RUNCMD (pid=22917) exiting with status: 0
Operation Status: 0
03/16/2013 13:12:31 - end Disk Staging: End Notify Script; elapsed time 0:00:00
Operation Status: 191
03/16/2013 13:12:31 - end Parent Job; elapsed time 0:11:32
03/16/2013 13:22:31 - Info nbjm (pid=12884) starting backup job (jobid=1297783) for client XTERHDQNBK02, policy __DSSU_POLICY_XTERHDQNBK02-stu_disk_TAP, schedule XTERHDQNBK02-stu_disk_TAP
no images were successfully processed  (191)

=======================================

Duplication job details :-

03/16/2013 13:23:04 - begin Duplicate
03/16/2013 13:23:05 - requesting resource Any
03/16/2013 13:23:06 - granted resource  MediaID=@aaaav;Path=/Archives/BWD;MediaServer=XTERHDQNBK01
03/16/2013 13:23:06 - granted resource  XTERHDQNBK01-stu_disk_BWD
03/16/2013 13:23:07 - Info bptm (pid=22141) start
03/16/2013 13:23:08 - started process bptm (pid=22141)
03/16/2013 13:23:08 - Info bptm (pid=22141) using 262144 data buffer size
03/16/2013 13:23:08 - Info bptm (pid=22141) setting receive network buffer to 524288 bytes
03/16/2013 13:23:08 - Info bptm (pid=22141) using 32 data buffers
03/16/2013 13:23:10 - Info bptm (pid=22141) start backup
03/16/2013 13:23:11 - Info bpdm (pid=11398) started
03/16/2013 13:23:11 - started process bpdm (pid=11398)
03/16/2013 13:23:11 - Info bptm (pid=22141) setting receive network buffer to 524288 bytes
03/16/2013 13:23:11 - Info bpdm (pid=11398) reading backup image
03/16/2013 13:23:11 - Info bptm (pid=22141) backup child process is pid 22217
03/16/2013 13:23:11 - Info bpdm (pid=11398) using 32 data buffers
03/16/2013 13:23:12 - Info bpdm (pid=11398) spawning a child process
03/16/2013 13:23:12 - Info bpbrm (pid=11398) child pid: 11399
03/16/2013 13:23:13 - begin reading
03/16/2013 13:23:18 - Warning bptm (pid=22141) storage unit XTERHDQNBK01-stu_disk_BWD is full: processing disk full condition
03/16/2013 13:23:38 - Error bptm (pid=22141) cannot write image to disk, attempted write of 262144 bytes, system wrote 106496
03/16/2013 13:23:39 - Info bptm (pid=22141) EXITING with status 129 <----------
03/16/2013 13:23:41 - Error bpdm (pid=11398) media manager terminated by parent process
03/16/2013 13:23:43 - Error bpdm (pid=11399) cannot write data to socket, Broken pipe
03/16/2013 13:23:43 - Error bpduplicate (pid=24129) host XTERHDQNBK02 backup id xterhdqtap_1363433719 read failed, media manager killed by signal (82).
03/16/2013 13:23:43 - Error bpduplicate (pid=24129) host XTERHDQNBK01 backupid xterhdqtap_1363433719 write failed, Disk storage unit is full (129).
03/16/2013 13:23:44 - Error bpduplicate (pid=24129) Duplicate of backupid xterhdqtap_1363433719 failed, Disk storage unit is full (129).
03/16/2013 13:23:45 - Info bptm (pid=22447) start
03/16/2013 13:23:45 - started process bptm (pid=22447)
03/16/2013 13:23:45 - requesting resource XTERHDQNBK01-stu_disk_BWD
03/16/2013 13:23:46 - Info bptm (pid=22447) using 262144 data buffer size
03/16/2013 13:23:46 - Info bptm (pid=22447) setting receive network buffer to 524288 bytes
03/16/2013 13:23:46 - Info bptm (pid=22447) using 32 data buffers
03/16/2013 13:23:46 - granted resource  MediaID=@aaaav;Path=/Archives/BWD;MediaServer=XTERHDQNBK01
03/16/2013 13:23:46 - granted resource  XTERHDQNBK01-stu_disk_BWD
03/16/2013 13:23:48 - Info bptm (pid=22447) start backup
03/16/2013 13:23:49 - Info bpdm (pid=11455) started
03/16/2013 13:23:49 - started process bpdm (pid=11455)
03/16/2013 13:23:50 - Info bptm (pid=22447) setting receive network buffer to 524288 bytes
03/16/2013 13:23:50 - Info bpdm (pid=11455) reading backup image
03/16/2013 13:23:50 - Info bptm (pid=22447) backup child process is pid 22497
03/16/2013 13:23:50 - Info bpdm (pid=11455) using 32 data buffers
03/16/2013 13:23:50 - Info bpdm (pid=11455) spawning a child process
03/16/2013 13:23:50 - Info bpbrm (pid=11455) child pid: 11458
03/16/2013 13:23:51 - begin reading
03/16/2013 13:23:57 - Warning bptm (pid=22447) storage unit XTERHDQNBK01-stu_disk_BWD is full: processing disk full condition
03/16/2013 13:24:18 - Error bptm (pid=22447) cannot write image to disk, attempted write of 262144 bytes, system wrote 106496
03/16/2013 13:24:18 - Info bptm (pid=22447) EXITING with status 129 <----------
03/16/2013 13:24:20 - Error bpdm (pid=11455) media manager terminated by parent process
03/16/2013 13:24:22 - Error bpdm (pid=11458) cannot write data to socket, Broken pipe
03/16/2013 13:24:22 - Error bpduplicate (pid=24129) host XTERHDQNBK02 backup id xterhdqtap_1363433856 read failed, media manager killed by signal (82).
03/16/2013 13:24:22 - Error bpduplicate (pid=24129) host XTERHDQNBK01 backupid xterhdqtap_1363433856 write failed, Disk storage unit is full (129).
03/16/2013 13:24:22 - Error bpduplicate (pid=24129) Duplicate of backupid xterhdqtap_1363433856 failed, Disk storage unit is full (129).
03/16/2013 13:24:23 - Info bptm (pid=22654) start
03/16/2013 13:24:23 - started process bptm (pid=22654)
03/16/2013 13:24:23 - Info bptm (pid=22654) using 262144 data buffer size
03/16/2013 13:24:23 - Info bptm (pid=22654) setting receive network buffer to 524288 bytes
03/16/2013 13:24:23 - Info bptm (pid=22654) using 32 data buffers
03/16/2013 13:24:23 - requesting resource XTERHDQNBK01-stu_disk_BWD
03/16/2013 13:24:23 - granted resource  MediaID=@aaaav;Path=/Archives/BWD;MediaServer=XTERHDQNBK01
03/16/2013 13:24:23 - granted resource  XTERHDQNBK01-stu_disk_BWD
03/16/2013 13:24:26 - Info bptm (pid=22654) start backup
03/16/2013 13:24:27 - Info bpdm (pid=11531) started
03/16/2013 13:24:27 - started process bpdm (pid=11531)
03/16/2013 13:24:27 - Info bptm (pid=22654) setting receive network buffer to 524288 bytes
03/16/2013 13:24:27 - Info bpdm (pid=11531) reading backup image
03/16/2013 13:24:27 - Info bptm (pid=22654) backup child process is pid 22656
03/16/2013 13:24:27 - Info bpdm (pid=11531) using 32 data buffers
03/16/2013 13:24:27 - Info bpdm (pid=11531) spawning a child process
03/16/2013 13:24:27 - Info bpbrm (pid=11531) child pid: 11532
03/16/2013 13:24:29 - begin reading
03/16/2013 13:24:42 - Warning bptm (pid=22654) storage unit XTERHDQNBK01-stu_disk_BWD is full: processing disk full condition
03/16/2013 13:25:05 - Error bptm (pid=22654) cannot write image to disk, attempted write of 262144 bytes, system wrote 106496
03/16/2013 13:25:05 - Info bptm (pid=22654) EXITING with status 129 <----------
03/16/2013 13:25:07 - Error bpdm (pid=11531) media manager terminated by parent process
03/16/2013 13:25:08 - Error bpdm (pid=11532) cannot write data to socket, Broken pipe
03/16/2013 13:25:08 - Error bpduplicate (pid=24129) host XTERHDQNBK02 backup id xterhdqtap_1363433993 read failed, media manager killed by signal (82).
03/16/2013 13:25:09 - Error bpduplicate (pid=24129) host XTERHDQNBK01 backupid xterhdqtap_1363433993 write failed, Disk storage unit is full (129).
03/16/2013 13:25:09 - Error bpduplicate (pid=24129) Duplicate of backupid xterhdqtap_1363433993 failed, Disk storage unit is full (129).
03/16/2013 13:25:10 - Info bptm (pid=22919) start
03/16/2013 13:25:10 - started process bptm (pid=22919)
03/16/2013 13:25:10 - requesting resource XTERHDQNBK01-stu_disk_BWD
03/16/2013 13:25:11 - Info bptm (pid=22919) using 262144 data buffer size
03/16/2013 13:25:11 - Info bptm (pid=22919) setting receive network buffer to 524288 bytes
03/16/2013 13:25:11 - Info bptm (pid=22919) using 32 data buffers
03/16/2013 13:25:11 - granted resource  MediaID=@aaaav;Path=/Archives/BWD;MediaServer=XTERHDQNBK01
03/16/2013 13:25:11 - granted resource  XTERHDQNBK01-stu_disk_BWD
03/16/2013 13:25:13 - Info bptm (pid=22919) start backup
03/16/2013 13:25:14 - Info bpdm (pid=11614) started
03/16/2013 13:25:14 - started process bpdm (pid=11614)
03/16/2013 13:25:14 - Info bptm (pid=22919) setting receive network buffer to 524288 bytes
03/16/2013 13:25:14 - Info bpdm (pid=11614) reading backup image
03/16/2013 13:25:15 - Info bptm (pid=22919) backup child process is pid 22921
03/16/2013 13:25:15 - Info bpdm (pid=11614) using 32 data buffers
03/16/2013 13:25:15 - Info bpdm (pid=11614) spawning a child process
03/16/2013 13:25:15 - Info bpbrm (pid=11614) child pid: 11615
03/16/2013 13:25:17 - begin reading
03/16/2013 13:25:21 - Warning bptm (pid=22919) storage unit XTERHDQNBK01-stu_disk_BWD is full: processing disk full condition
03/16/2013 13:25:43 - Error bptm (pid=22919) cannot write image to disk, attempted write of 262144 bytes, system wrote 106496
03/16/2013 13:25:44 - Info bptm (pid=22919) EXITING with status 129 <----------
03/16/2013 13:25:46 - Error bpdm (pid=11615) cannot write data to socket, Broken pipe
03/16/2013 13:25:46 - Error bpdm (pid=11614) media manager terminated by parent process
03/16/2013 13:25:46 - Error bpduplicate (pid=24129) host XTERHDQNBK02 backup id xterhdqtap_1363434134 read failed, media manager killed by signal (82).
03/16/2013 13:25:46 - Error bpduplicate (pid=24129) host XTERHDQNBK01 backupid xterhdqtap_1363434134 write failed, Disk storage unit is full (129).
03/16/2013 13:25:47 - Error bpduplicate (pid=24129) Duplicate of backupid xterhdqtap_1363434134 failed, Disk storage unit is full (129).
03/16/2013 13:25:47 - Info bptm (pid=23082) start
03/16/2013 13:25:48 - started process bptm (pid=23082)
03/16/2013 13:25:48 - Info bptm (pid=23082) using 262144 data buffer size
03/16/2013 13:25:48 - Info bptm (pid=23082) setting receive network buffer to 524288 bytes
03/16/2013 13:25:48 - Info bptm (pid=23082) using 32 data buffers
03/16/2013 13:25:48 - requesting resource XTERHDQNBK01-stu_disk_BWD
03/16/2013 13:25:48 - granted resource  MediaID=@aaaav;Path=/Archives/BWD;MediaServer=XTERHDQNBK01
03/16/2013 13:25:48 - granted resource  XTERHDQNBK01-stu_disk_BWD
03/16/2013 13:25:51 - Info bptm (pid=23082) start backup
03/16/2013 13:25:52 - Info bpdm (pid=11655) started
03/16/2013 13:25:52 - started process bpdm (pid=11655)
03/16/2013 13:25:52 - Info bptm (pid=23082) setting receive network buffer to 524288 bytes
03/16/2013 13:25:52 - Info bpdm (pid=11655) reading backup image
03/16/2013 13:25:52 - Info bptm (pid=23082) backup child process is pid 23109
03/16/2013 13:25:52 - Info bpdm (pid=11655) using 32 data buffers
03/16/2013 13:25:52 - Info bpdm (pid=11655) spawning a child process
03/16/2013 13:25:52 - Info bpbrm (pid=11655) child pid: 11656
03/16/2013 13:25:54 - begin reading
03/16/2013 13:25:59 - Warning bptm (pid=23082) storage unit XTERHDQNBK01-stu_disk_BWD is full: processing disk full condition
03/16/2013 13:26:20 - Error bptm (pid=23082) cannot write image to disk, attempted write of 262144 bytes, system wrote 106496
03/16/2013 13:26:21 - Info bptm (pid=23082) EXITING with status 129 <----------
03/16/2013 13:26:23 - Error bpdm (pid=11655) media manager terminated by parent process
03/16/2013 13:26:24 - Error bpdm (pid=11656) cannot write data to socket, Broken pipe
03/16/2013 13:26:24 - Error bpduplicate (pid=24129) host XTERHDQNBK02 backup id xterhdqtap_1363434280 read failed, media manager killed by signal (82).
03/16/2013 13:26:24 - Error bpduplicate (pid=24129) host XTERHDQNBK01 backupid xterhdqtap_1363434280 write failed, Disk storage unit is full (129).
03/16/2013 13:26:25 - Error bpduplicate (pid=24129) Duplicate of backupid xterhdqtap_1363434280 failed, Disk storage unit is full (129).
03/16/2013 13:26:26 - Info bptm (pid=23287) start
03/16/2013 13:26:26 - started process bptm (pid=23287)
03/16/2013 13:26:26 - requesting resource XTERHDQNBK01-stu_disk_BWD
03/16/2013 13:26:26 - Info bptm (pid=23287) using 262144 data buffer size
03/16/2013 13:26:26 - Info bptm (pid=23287) setting receive network buffer to 524288 bytes
03/16/2013 13:26:26 - Info bptm (pid=23287) using 32 data buffers
03/16/2013 13:26:26 - granted resource  MediaID=@aaaav;Path=/Archives/BWD;MediaServer=XTERHDQNBK01
03/16/2013 13:26:26 - granted resource  XTERHDQNBK01-stu_disk_BWD
03/16/2013 13:26:29 - Info bptm (pid=23287) start backup
03/16/2013 13:26:30 - Info bpdm (pid=11736) started
03/16/2013 13:26:30 - started process bpdm (pid=11736)
03/16/2013 13:26:30 - Info bptm (pid=23287) setting receive network buffer to 524288 bytes
03/16/2013 13:26:30 - Info bpdm (pid=11736) reading backup image
03/16/2013 13:26:30 - Info bptm (pid=23287) backup child process is pid 23311
03/16/2013 13:26:30 - Info bpdm (pid=11736) using 32 data buffers
03/16/2013 13:26:30 - Info bpdm (pid=11736) spawning a child process
03/16/2013 13:26:30 - Info bpbrm (pid=11736) child pid: 11737
03/16/2013 13:26:32 - begin reading
03/16/2013 13:26:36 - Warning bptm (pid=23287) storage unit XTERHDQNBK01-stu_disk_BWD is full: processing disk full condition
03/16/2013 13:26:58 - Error bptm (pid=23287) cannot write image to disk, attempted write of 262144 bytes, system wrote 106496
03/16/2013 13:26:59 - Info bptm (pid=23287) EXITING with status 129 <----------
03/16/2013 13:27:01 - Error bpdm (pid=11736) media manager terminated by parent process
03/16/2013 13:27:01 - Error bpdm (pid=11737) cannot write data to socket, Broken pipe
03/16/2013 13:27:01 - Error bpduplicate (pid=24129) host XTERHDQNBK02 backup id xterhdqtap_1363434441 read failed, media manager killed by signal (82).
03/16/2013 13:27:01 - Error bpduplicate (pid=24129) host XTERHDQNBK01 backupid xterhdqtap_1363434441 write failed, Disk storage unit is full (129).
03/16/2013 13:27:02 - Error bpduplicate (pid=24129) Duplicate of backupid xterhdqtap_1363434441 failed, Disk storage unit is full (129).
03/16/2013 13:27:02 - Info bptm (pid=23504) start
03/16/2013 13:27:03 - started process bptm (pid=23504)
03/16/2013 13:27:03 - Info bptm (pid=23504) using 262144 data buffer size
03/16/2013 13:27:03 - Info bptm (pid=23504) setting receive network buffer to 524288 bytes
03/16/2013 13:27:03 - Info bptm (pid=23504) using 32 data buffers
03/16/2013 13:27:03 - requesting resource XTERHDQNBK01-stu_disk_BWD
03/16/2013 13:27:03 - granted resource  MediaID=@aaaav;Path=/Archives/BWD;MediaServer=XTERHDQNBK01
03/16/2013 13:27:03 - granted resource  XTERHDQNBK01-stu_disk_BWD
03/16/2013 13:27:06 - Info bptm (pid=23504) start backup
03/16/2013 13:27:07 - Info bpdm (pid=11850) started
03/16/2013 13:27:07 - started process bpdm (pid=11850)
03/16/2013 13:27:07 - Info bptm (pid=23504) setting receive network buffer to 524288 bytes
03/16/2013 13:27:07 - Info bpdm (pid=11850) reading backup image
03/16/2013 13:27:07 - Info bptm (pid=23504) backup child process is pid 23550
03/16/2013 13:27:07 - Info bpdm (pid=11850) using 32 data buffers
03/16/2013 13:27:07 - Info bpdm (pid=11850) spawning a child process
03/16/2013 13:27:07 - Info bpbrm (pid=11850) child pid: 11852
03/16/2013 13:27:09 - begin reading
03/16/2013 13:27:13 - Warning bptm (pid=23504) storage unit XTERHDQNBK01-stu_disk_BWD is full: processing disk full condition
03/16/2013 13:27:34 - Error bptm (pid=23504) cannot write image to disk, attempted write of 262144 bytes, system wrote 106496
03/16/2013 13:27:35 - Info bptm (pid=23504) EXITING with status 129 <----------
03/16/2013 13:27:37 - Error bpdm (pid=11850) media manager terminated by parent process
03/16/2013 13:27:38 - Error bpdm (pid=11852) cannot write data to socket, Broken pipe
03/16/2013 13:27:38 - Error bpduplicate (pid=24129) host XTERHDQNBK02 backup id xterhdqtap_1363434548 read failed, media manager killed by signal (82).
03/16/2013 13:27:38 - Error bpduplicate (pid=24129) host XTERHDQNBK01 backupid xterhdqtap_1363434548 write failed, Disk storage unit is full (129).
03/16/2013 13:27:38 - Error bpduplicate (pid=24129) Duplicate of backupid xterhdqtap_1363434548 failed, Disk storage unit is full (129).
03/16/2013 13:27:39 - Info bptm (pid=23711) start
03/16/2013 13:27:40 - started process bptm (pid=23711)
03/16/2013 13:27:40 - Info bptm (pid=23711) using 262144 data buffer size
03/16/2013 13:27:40 - Info bptm (pid=23711) setting receive network buffer to 524288 bytes
03/16/2013 13:27:40 - Info bptm (pid=23711) using 32 data buffers
03/16/2013 13:27:40 - requesting resource XTERHDQNBK01-stu_disk_BWD
03/16/2013 13:27:40 - granted resource  MediaID=@aaaav;Path=/Archives/BWD;MediaServer=XTERHDQNBK01
03/16/2013 13:27:40 - granted resource  XTERHDQNBK01-stu_disk_BWD
03/16/2013 13:27:43 - Info bptm (pid=23711) start backup
03/16/2013 13:27:43 - Info bpdm (pid=11896) started
03/16/2013 13:27:47 - Info bptm (pid=23711) setting receive network buffer to 524288 bytes
03/16/2013 13:27:47 - Info bptm (pid=23711) backup child process is pid 23801
03/16/2013 13:27:47 - started process bpdm (pid=11896)
03/16/2013 13:27:47 - Info bpdm (pid=11896) reading backup image
03/16/2013 13:27:48 - Info bpdm (pid=11896) using 32 data buffers
03/16/2013 13:27:48 - Info bpdm (pid=11896) spawning a child process
03/16/2013 13:27:48 - Info bpbrm (pid=11896) child pid: 11904
03/16/2013 13:27:49 - begin reading
03/16/2013 13:27:53 - Warning bptm (pid=23711) storage unit XTERHDQNBK01-stu_disk_BWD is full: processing disk full condition
03/16/2013 13:28:24 - Error bptm (pid=23711) cannot write image to disk, attempted write of 262144 bytes, system wrote 106496
03/16/2013 13:28:24 - Info bptm (pid=23711) EXITING with status 129 <----------
03/16/2013 13:28:26 - Error bpdm (pid=11896) media manager terminated by parent process
03/16/2013 13:28:44 - Error bpdm (pid=11904) cannot write data to socket, Broken pipe
03/16/2013 13:28:44 - Error bpduplicate (pid=24129) host XTERHDQNBK02 backup id xterhdqtap_1363435732 read failed, media manager killed by signal (82).
03/16/2013 13:28:44 - Error bpduplicate (pid=24129) host XTERHDQNBK01 backupid xterhdqtap_1363435732 write failed, Disk storage unit is full (129).
03/16/2013 13:28:44 - Error bpduplicate (pid=24129) Duplicate of backupid xterhdqtap_1363435732 failed, Disk storage unit is full (129).
03/16/2013 13:28:44 - Error bpduplicate (pid=24129) Status = no images were successfully processed.
03/16/2013 13:28:44 - end Duplicate; elapsed time 0:05:40
no images were successfully processed  (191)

 

Mack_Disouza
Level 4

root@XTERHDQNBK02 # uname -a
SunOS XTERHDQNBK02 5.10 Generic_147440-23 sun4v sparc sun4v
root@XTERHDQNBK02 # df -k |grep /Archives/TAP
/dev/md/archds/dsk/d3 51634919 26878848 24239722    53%    /Archives/TAP

Mack_Disouza
Level 4

Attached is the High/Low water mark settings.

Mack_Disouza
Level 4

How to get the answer of below two questions ?

how much data you are sending to disk  ?

how much data your duplication jobs( staging jobs) can  send to tape?

 

 

Mack_Disouza
Level 4

Hello Nagalla

Could you please provide your inputs ?

Mark_Solutions
Level 6
Partner Accredited Certified

Looking at your high / low watermark screen shot it looks like all of your disk staging is in the same locations /archives/xxx

If that is the case then you may have issue as the first one may fill up the file system preventing the others from working

Unless each of those is on seperate file systems than you have a fundamental problem with your setup.

Each disk staging storage until should be on its own file system to prevent them having an affect on each other.

The one you write to may have completely emptied itself but still has no disk space due to the others

So you may need to re-structure the whole environment for it to work properly

I also see in your log that the predicted size of the backup is 0kb, indicating that the client has not been backed up before - this can add an extra overhead as NetBackup does its own cacluations on how much space to free up for an unknown backup so plenty of free space is generally needed

Let us know about where these folders are first - but if they are all on the same file system then it just wont work correctly

Hope this helps

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

In addition to above excellent post - please tell us why your high watermarks are so low?

The default is 98% and works well in most cases. 
Also understand that 'available space' is based on 100% of allocated disk space, not the 68% HWM.

You also showed us a duplication job that fails with status 191. Are any of your duplications going through? How often do they run? Hopefully you have enough tape drives to cater for your large amount of DSU's?

Do you have logging enabled? You need all of the following logs:
On master and media server: admin
On media server: bpbrm, bptm and bpdm.