Backup failng with error code 13

[wannawin]

Hello Team.

Master server Os is linux and NBU version 7.1.0.4

Client OS Linux, RedHat2.6 and NBU version 7.1.0

Backup of only one mount point is failing with error code 13 after writing same Kb of data everytime and rest of mount point completed successfully.

Job Details below..

05/06/2013 02:00:12 - Info nbjm (pid=23860) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=2484447, request id:{98BDD222-B61A-11E2-99E3-21AC1059C9DF})
05/06/2013 02:00:12 - requesting resource usgaub5500-DP1-LocalDisk
05/06/2013 02:00:12 - requesting resource usgaub5500.NBU_CLIENT.MAXJOBS.usgauo5003qc-brn
05/06/2013 02:00:12 - requesting resource usgaub5500.NBU_POLICY.MAXJOBS.Unix-FS
05/06/2013 02:00:12 - awaiting resource usgaub5500-DP1-LocalDisk. Maximum job count has been reached for the storage unit.
05/06/2013 02:05:08 - granted resource  usgaub5500.NBU_CLIENT.MAXJOBS.usgauo5003qc-brn
05/06/2013 02:05:08 - granted resource  usgaub5500.NBU_POLICY.MAXJOBS.Unix-FS
05/06/2013 02:05:08 - granted resource  MediaID=@aaaae;DiskVolume=/nbpool4;DiskPool=DP1-LocalDisk;Path=/nbpool4;StorageServer=usgaub5500;Med...
05/06/2013 02:05:08 - granted resource  usgaub5500-DP1-LocalDisk
05/06/2013 02:05:08 - estimated 30566 kbytes needed
05/06/2013 02:05:08 - Info nbjm (pid=23860) started backup job for client usgauo5003qc-brn, policy Unix-FS, schedule Incr on storage unit usgaub5500-DP1-LocalDisk
05/06/2013 02:05:08 - Info bpbrm (pid=22478) usgauo5003qc-brn is the host to backup data from
05/06/2013 02:05:08 - Info bpbrm (pid=22478) reading file list from client
05/06/2013 02:05:08 - started process bpbrm (pid=22478)
05/06/2013 02:05:08 - connecting
05/06/2013 02:05:09 - Info bpbrm (pid=22478) starting bpbkar on client
05/06/2013 02:05:09 - Info bpbkar (pid=17831) Backup started
05/06/2013 02:05:09 - Info bpbrm (pid=22478) bptm pid: 22489
05/06/2013 02:05:09 - Info bptm (pid=22489) start
05/06/2013 02:05:09 - Info bptm (pid=22489) using 262144 data buffer size
05/06/2013 02:05:09 - Info bptm (pid=22489) using 32 data buffers
05/06/2013 02:05:09 - Info bptm (pid=22489) start backup
05/06/2013 02:05:09 - Info bptm (pid=22489) backup child process is pid 22518
05/06/2013 02:05:09 - connected; connect time: 0:00:00
05/06/2013 02:05:09 - begin writing
05/06/2013 02:35:28 - Error bpbrm (pid=22478) socket read failed: errno = 62 - Timer expired
05/06/2013 02:35:30 - Error bptm (pid=22489) media manager terminated by parent process
05/06/2013 02:38:38 - Info bpbkar (pid=17831) done. status: 13: file read failed
05/06/2013 02:38:38 - end writing; write time: 0:33:29
05/06/2013 02:48:38 - Info nbjm (pid=23860) starting backup job (jobid=2484447) for client usgauo5003qc-brn, policy Unix-FS, schedule Incr
file read failed  (13)
 

[Nicolai]

Try to adjust CLIENT_READ_TIMEOUT to either 1800 or 3600. 

See http://www.symantec.com/docs/HOWTO33199

or use this T/N: http://www.symantec.com/docs/TECH31513 to see where the backup has hung.

Remember to create /usr/openv/netbackup/logs/bpbkar fist. Inspect the bpbkar log to see where the backup is hung. It may be a file, a file system or a directory with 1000 of files. Some of the last lines in bpbkar  should reveal the beastie.

[wannawin]

Hello All.

Please find the attached bpbkar logs.

created log file and backup initiated again and it failed..

[Mark_Solutions]

I dont see any failure at the end of that log

As Martin says it looks like a timeout as it fails after exactly 30 minutes - so you need a client connect and client read timeout setting on your media servers that are greater than that value

It sound like it is currently 30 minutes so go higher than that to see if it helps

It may be taking a long time to process the exclude list before starting the backup

[wannawin]

Hello Mark.

Client read time out is set to 7200 and client time out is set to 1800 on master server and backup is going through master server itself.

I exclude one file where it stuck every time but now it stuck on some another file

[Mark_Solutions]

It is the one on the Master that matters - that is 1800 and your job fails after 1800 seconds

Increase that one to 7200 and try again

[wannawin]

Hello Martin , its already 7200

80101937@usgaub5500 netbackup]$ cat /usr/openv/netbackup/bp.conf
SERVER = usgaub5500
SERVER = usgaub5502-brn
SERVER = usgaub9500
SERVER = usgaub5502.hospira.corp
CLIENT_NAME = usgaub5500
EMMSERVER = usgaub5500
VXDBMS_NB_DATA = /usr/openv/db/data
ALLOW_MEDIA_OVERWRITE = TAR
ALLOW_MEDIA_OVERWRITE = CPIO
ALLOW_MEDIA_OVERWRITE = ANSI
CLIENT_CONNECT_TIMEOUT = 1800
CLIENT_READ_TIMEOUT = 7200
VERBOSE = 0
 

[Mark_Solutions]

Ah - OK - then maybe it is just stuck on another file - was that bpbkar logs the whole log as it showed no failure

Can you try again with bpcd logging on the client too and bpbrm and bptm logging setup on the Master Server - then attached those files to the thread

Thanks

[wannawin]

Please find the job details..

05/07/2013 08:30:00 - Info nbjm (pid=23860) starting backup job (jobid=2490916) for client usgauo5003qc-brn, policy Unix-FS_usgauo5003qc, schedule Incr
05/07/2013 08:30:02 - Info bpbrm (pid=6772) starting bptm
05/07/2013 08:30:02 - Info bpbrm (pid=6772) Started media manager using bpcd successfully
05/07/2013 08:30:02 - estimated 0 kbytes needed
05/07/2013 08:30:02 - Info nbjm (pid=23860) started backup job for client usgauo5003qc-brn, policy Unix-FS_usgauo5003qc, schedule Incr on storage unit usgaub5500-hcart-tld0
05/07/2013 08:30:02 - started process bpbrm (pid=6772)
05/07/2013 08:30:03 - Info bpbrm (pid=6772) usgauo5003qc-brn is the host to backup data from
05/07/2013 08:30:03 - Info bpbrm (pid=6772) telling media manager to start backup on client
05/07/2013 08:30:03 - Info bptm (pid=6778) using 262144 data buffer size
05/07/2013 08:30:03 - Info bptm (pid=6778) using 32 data buffers
05/07/2013 08:30:03 - Info bptm (pid=6778) start backup
05/07/2013 08:30:03 - Info bptm (pid=6778) Waiting for mount of media id HS3158 (copy 1) on server usgaub5500.
05/07/2013 08:30:03 - Info bpbrm (pid=6772) spawning a brm child process
05/07/2013 08:30:03 - Info bpbrm (pid=6772) child pid: 6789
05/07/2013 08:30:03 - Info bpbrm (pid=6772) sending bpsched msg: CONNECTING TO CLIENT FOR usgauo5003qc-brn_1367933402
05/07/2013 08:30:03 - Info bpbrm (pid=6772) start bpbkar on client
05/07/2013 08:30:03 - Info bpbkar (pid=17226) Backup started
05/07/2013 08:30:03 - Info bpbrm (pid=6772) Sending the file list to the client
05/07/2013 08:30:03 - mounting HS3158
05/07/2013 08:30:03 - connecting
05/07/2013 08:30:03 - connected; connect time: 0:00:00
05/07/2013 08:30:54 - Info bptm (pid=6778) media id HS3158 mounted on drive index 0, drivepath /dev/nst7, drivename IBM.ULTRIUM-TD4.261, copy 1
05/07/2013 08:30:54 - mounted HS3158; mount time: 0:00:51
05/07/2013 08:30:54 - positioning HS3158 to file 3
05/07/2013 08:31:37 - positioned HS3158; position time: 0:00:43
05/07/2013 08:31:37 - begin writing
05/07/2013 10:34:09 - Error bpbrm (pid=6789) socket read failed: errno = 62 - Timer expired
05/07/2013 10:34:09 - Info bpbrm (pid=6772) sending message to media manager: STOP BACKUP usgauo5003qc-brn_1367933402
05/07/2013 10:34:21 - Info bpbrm (pid=6772) media manager for backup id usgauo5003qc-brn_1367933402 exited with status 150: termination requested by administrator
05/07/2013 10:34:21 - end writing; write time: 2:02:44
file read failed  (13)
 

[Nicolai]

You need VERBOSE = 5 to get the full bpbkar debug log. At verbose 5 bpbkar will write a line for each file it access. 

Take a look here : https://sites.google.com/site/netbackupwiki/netbackup-6-5-3/touch-files/bpbkar_path_tr

under "Example of bpbkar log with bpbkar_path_tr"

[wannawin · Accepted solution]

Hello All.

This issue has been resolved, below mentioned bpbkar processes was in hung state, not able to kill by KILL-9 at the end reboot the client server, after that backup completed successfully.

Hung Processes...

# ps -fea | grep bpbkar |egrep -v grep

root       416     1  0 May01 ?        00:00:04 bpbkar -r 604800 -ru root -dt 272615 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2459581 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367391784 -kl 28 -use_ofb

root      1695     1  0 May08 ?        00:00:14 bpbkar -r 604800 -ru root -dt 0 -to 0 -clnt usgauo5003qc-brn -class Unix-FS_usgauo5003qc -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 7200 -ckpt_time 900 -blks_per_buffer 511 -stream_count 1 -stream_number 1 -jobgrpid 2496094 -use_otm -use_ofb -b usgauo5003qc-brn_1368012493 -kl 28 -fso

root      2312     1  0 May07 ?        00:00:01 bpbkar -r 604800 -ru root -dt 776658 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2489711 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367910314 -kl 28 -use_ofb

root      3906     1  0 May03 ?        00:00:04 bpbkar -r 604800 -ru root -dt 428324 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2470396 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367564654 -kl 28 -use_ofb

root      4806     1  0 May07 ?        00:00:09 bpbkar -r 604800 -ru root -dt 0 -to 0 -clnt usgauo5003qc-brn -class Unix-FS_usgauo5003qc -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 7200 -ckpt_time 900 -blks_per_buffer 511 -stream_count 1 -stream_number 1 -jobgrpid 2490621 -use_otm -use_ofb -b usgauo5003qc-brn_1367925263 -kl 28 -fso

root      5320     1  0 03:04 ?        00:00:02 bpbkar -r 604800 -ru root -dt 946873 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 9200 -stream_count 48 -stream_number 14 -jobgrpid 2500738 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1368083204 -kl 28 -use_ofb

root      7359     1  0 May05 ?        00:00:03 bpbkar -r 604800 -ru root -dt 613387 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2480528 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367738296 -kl 28 -use_ofb

root      8402     1  0 May01 ?        00:00:05 bpbkar -r 604800 -ru root -dt 255453 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2459581 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367391784 -kl 28 -use_ofb

root      8577     1  0 02:09 ?        00:00:19 bpbkar -r 604800 -ru root -dt 0 -to 0 -clnt usgauo5003qc-brn -class Unix-FS_usgauo5003qc -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 9200 -ckpt_time 900 -blks_per_buffer 511 -stream_count 1 -stream_number 1 -jobgrpid 2500415 -use_otm -use_ofb -b usgauo5003qc-brn_1368079869 -kl 28 -fso

root     10416     1  0 Apr29 ?        00:00:03 bpbkar -r 604800 -ru root -dt 88375 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2448717 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367219030 -kl 28 -use_ofb

root     10628     1  0 May01 ?        00:00:01 bpbkar -r 604800 -ru root -dt 258993 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2459581 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367391784 -kl 28 -use_ofb

root     13913     1  0 May01 ?        00:00:01 bpbkar -r 604800 -ru root -dt 261678 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2459581 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367391784 -kl 28 -use_ofb

root     14188     1  0 May08 ?        00:00:05 bpbkar -r 604800 -ru root -dt 860373 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 7200 -stream_count 48 -stream_number 14 -jobgrpid 2495312 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367996703 -kl 28 -use_ofb

root     14951     1  0 May07 ?        00:00:01 bpbkar -r 604800 -ru root -dt 779608 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2489711 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367910314 -kl 28 -use_ofb

root     16539     1  0 May08 ?        00:00:04 bpbkar -r 604800 -ru root -dt 869147 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 7200 -stream_count 48 -stream_number 14 -jobgrpid 2495312 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367996703 -kl 28 -use_ofb

root     17183     1  0 05:49 ?        00:00:02 bpbkar -r 604800 -ru root -dt 956759 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 9200 -stream_count 48 -stream_number 14 -jobgrpid 2500738 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1368083204 -kl 28 -use_ofb

root     17226     1  0 May07 ?        00:00:11 bpbkar -r 604800 -ru root -dt 0 -to 0 -clnt usgauo5003qc-brn -class Unix-FS_usgauo5003qc -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 7200 -ckpt_time 900 -blks_per_buffer 511 -stream_count 1 -stream_number 1 -jobgrpid 2490915 -use_otm -use_ofb -b usgauo5003qc-brn_1367933402 -kl 28 -fso

root     17302     1  0 May02 ?        00:00:01 bpbkar -r 604800 -ru root -dt 346868 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2464898 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367478175 -kl 28 -use_ofb

root     17831     1  0 May06 ?        00:00:04 bpbkar -r 604800 -ru root -dt 687577 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2484177 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367823908 -kl 28 -use_ofb

root     18184     1  0 May06 ?        00:00:01 bpbkar -r 604800 -ru root -dt 690188 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2484177 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367823908 -kl 28 -use_ofb

root     19290     1  0 May08 ?        00:00:17 bpbkar -r 604800 -ru root -dt 0 -to 0 -clnt usgauo5003qc-brn -class Unix-FS_usgauo5003qc -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 7200 -ckpt_time 900 -blks_per_buffer 511 -stream_count 1 -stream_number 1 -jobgrpid 2496094 -use_otm -use_ofb -b usgauo5003qc-brn_1368012493 -kl 28 -fso

root     19322     1  0 May06 ?        00:00:01 bpbkar -r 604800 -ru root -dt 693676 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2484177 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367823908 -kl 28 -use_ofb

root     20153     1  0 Apr29 ?        00:00:01 bpbkar -r 604800 -ru root -dt 91213 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2448717 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367219030 -kl 28 -use_ofb

root     20555     1  0 May05 ?        00:00:01 bpbkar -r 604800 -ru root -dt 606834 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2480528 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367738296 -kl 28 -use_ofb

root     21066     1  0 Apr30 ?        00:00:01 bpbkar -r 604800 -ru root -dt 174887 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2454024 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367305364 -kl 28 -use_ofb

root     21151     1  0 May04 ?        00:00:16 bpbkar -r 604800 -ru root -dt 0 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Full -st FULL -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2475786 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367662360 -kl 28 -use_ofb

root     21579     1  0 May07 ?        00:00:15 bpbkar -r 604800 -ru root -dt 0 -to 0 -clnt usgauo5003qc-brn -class Unix-FS_usgauo5003qc -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 3600 -ckpt_time 900 -blks_per_buffer 511 -stream_count 1 -stream_number 1 -jobgrpid 2490462 -use_otm -use_ofb -b usgauo5003qc-brn_1367919624 -kl 28 -fso

root     22874     1  0 May02 ?        00:00:01 bpbkar -r 604800 -ru root -dt 344430 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2464898 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367478175 -kl 28 -use_ofb

root     23094     1  0 07:47 ?        00:00:15 bpbkar -r 604800 -ru root -dt 0 -to 0 -clnt usgauo5003qc-brn -class Unix-FS_usgauo5003qc -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 9200 -ckpt_time 900 -blks_per_buffer 511 -stream_count 1 -stream_number 1 -jobgrpid 2500415 -use_otm -use_ofb -b usgauo5003qc-brn_1368079869 -kl 28 -fso

root     23496     1  0 May05 ?        00:00:01 bpbkar -r 604800 -ru root -dt 604409 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2480528 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367738296 -kl 28 -use_ofb

root     23675     1  0 May02 ?        00:00:05 bpbkar -r 604800 -ru root -dt 341844 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2464898 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367478175 -kl 28 -use_ofb

root     23988     1  0 May03 ?        00:00:01 bpbkar -r 604800 -ru root -dt 433170 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2470396 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367564654 -kl 28 -use_ofb

root     27476     1  0 Apr29 ?        00:00:04 bpbkar -r 604800 -ru root -dt 82699 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2448717 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367219030 -kl 28 -use_ofb

root     27918     1  0 May07 ?        00:00:11 bpbkar -r 604800 -ru root -dt 0 -to 0 -clnt usgauo5003qc-brn -class Unix-FS_usgauo5003qc -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 3600 -ckpt_time 900 -blks_per_buffer 511 -stream_count 1 -stream_number 1 -jobgrpid 2490462 -use_otm -use_ofb -b usgauo5003qc-brn_1367919624 -kl 28 -fso

root     28590     1  0 May08 ?        00:00:13 bpbkar -r 604800 -ru root -dt 0 -to 0 -clnt usgauo5003qc-brn -class Unix-FS_usgauo5003qc -sched Full -st FULL -bpstart_to 300 -bpend_to 300 -read_to 7200 -ckpt_time 900 -blks_per_buffer 511 -stream_count 1 -stream_number 1 -jobgrpid 2496505 -use_otm -use_ofb -b usgauo5003qc-brn_1368026640 -kl 28 -fso

root     28921     1  0 May05 ?        00:00:04 bpbkar -r 604800 -ru root -dt 601966 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2480528 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367738296 -kl 28 -use_ofb

root     29184     1  0 Apr30 ?        00:00:01 bpbkar -r 604800 -ru root -dt 172451 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2454024 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367305364 -kl 28 -use_ofb

root     30851     1  0 Apr30 ?        00:00:05 bpbkar -r 604800 -ru root -dt 169034 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2454024 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367305364 -kl 28 -use_ofb

root     31307     1  0 May03 ?        00:00:01 bpbkar -r 604800 -ru root -dt 430751 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2470396 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367564654 -kl 28 -use_ofb

root     31463     1  0 May07 ?        00:00:13 bpbkar -r 604800 -ru root -dt 0 -to 0 -clnt usgauo5003qc-brn -class Unix-FS_usgauo5003qc -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 7200 -ckpt_time 900 -blks_per_buffer 511 -stream_count 1 -stream_number 1 -jobgrpid 2490915 -use_otm -use_ofb -b usgauo5003qc-brn_1367933402 -kl 28 -fso

root     31717     1  0 May07 ?        00:00:04 bpbkar -r 604800 -ru root -dt 773984 -to 0 -clnt usgauo5003qc-brn -class Unix-FS -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1800 -stream_count 48 -stream_number 14 -jobgrpid 2489711 -ckpt_time 900 -blks_per_buffer 512 -use_otm -fso -b usgauo5003qc-brn_1367910314 -kl 28 -use_ofb

root     32162     1  0 04:58 ?        00:00:17 bpbkar -r 604800 -ru root -dt 0 -to 0 -clnt usgauo5003qc-brn -class Unix-FS_usgauo5003qc -sched Incr -st INCR -bpstart_to 300 -bpend_to 300 -read_to 9200 -ckpt_time 900 -blks_per_buffer 511 -stream_count 1 -stream_number 1 -jobgrpid 2500415 -use_otm -use_ofb -b usgauo5003qc-brn_1368079869 -kl 28 -fso