the backup failed to back up the requested files(6)

[aquib]

Hi,

We are facing problem our one server backup not start and rest of the server backup complete successfully.

ERROR:

Feb 11, 2014 3:45:01 PM - requesting resource jsawecd-hcart2-robot-tld-0

Feb 11, 2014 3:45:01 PM - requesting resource jsawnetbackup.NBU_CLIENT.MAXJOBS.jsawecd

Feb 11, 2014 3:45:01 PM - requesting resource jsawnetbackup.NBU_POLICY.MAXJOBS.SAP-JSAWECD-FULL

Feb 11, 2014 3:45:01 PM - Waiting for scan drive stop IBM.ULT3580-TD5.000, Media server: jsawecd

Feb 11, 2014 3:45:01 PM - granted resource  jsawnetbackup.NBU_CLIENT.MAXJOBS.jsawecd

Feb 11, 2014 3:45:01 PM - granted resource  jsawnetbackup.NBU_POLICY.MAXJOBS.SAP-JSAWECD-FULL

Feb 11, 2014 3:45:01 PM - granted resource  0057L5

Feb 11, 2014 3:45:01 PM - granted resource  IBM.ULT3580-TD5.000

Feb 11, 2014 3:45:01 PM - granted resource  jsawecd-hcart2-robot-tld-0

Feb 11, 2014 3:45:01 PM - estimated 0 kbytes needed

Feb 11, 2014 3:45:02 PM - started process bpbrm (pid=422082)

Feb 11, 2014 3:45:02 PM - connecting

Feb 11, 2014 3:45:14 PM - connected; connect time: 0:00:00

Feb 11, 2014 3:45:28 PM - mounting 0057L5

Feb 11, 2014 3:45:33 PM - Error bpbrm (pid=422082) from client jsawecd: ERR - Lock file /usr/openv/netbackup/logs/user_ops/dbext/logs/vxbsa.1392113694.929958.files.1.lock is unlocked. Abort backup.

Feb 11, 2014 3:45:33 PM - Error bptm (pid=458994) media manager terminated during mount of media id 0057L5, possible media mount timeout

Feb 11, 2014 3:45:35 PM - Error bptm (pid=458994) media manager terminated by parent process

Feb 11, 2014 3:45:38 PM - end writing

the backup failed to back up the requested files  (6)

[aquib]

please resolve

[Mark_Solutions]

From the policy name this sounds like a SAP backup - take a look at this tech note which sounds very much like your issue:

http://www.symantec.com/docs/TECH58234

Hope this helps

[aquib]

Hi, Mark

In the server /openv/netbackup/logs/user_ops/sap is not found. I checked on both client server. One server compelet backup sucessfully but second not, 10 days before it is working fine.

[aquib]

Hi, Mark

In the server /openv/netbackup/logs/user_ops/sap is not found. I checked on both client server. One server compelet backup sucessfully but second not, 10 days before it is working fine.

[Mark_Solutions]

Your log in the original post shows it to actually be :

/usr/openv/netbackup/logs/user_ops/dbext/logs/

So check that location rather than the /sap location

[aquib]

Please see the logs file output:

Backup started Tue Feb 11 15:44:54 2014

15:44:58 Initiating backup

15:45:02 INF - Starting bpbrm

15:45:06 INF - Data socket = jsawecd.IPC:/tmp/vnet-30118392113704914501000000000-uwAcya;44d52d0ccf699eedda7b4279f9bed974;10;300

15:45:06 INF - Name socket = jsawecd.IPC:/tmp/vnet-46370392113705314539000000000-u_3mia;29541bc945a3c04b8e57252e37fe9e10;10;300

15:45:07 INF - Job id = 52526

15:45:07 INF - Backup id = jsawecd_1392113701

15:45:07 INF - Backup time = 1392113701

15:45:07 INF - Policy name = SAP-JSAWECD-FULL

15:45:07 INF - Snapshot = 0

15:45:08 INF - Frozen image = 0

15:45:08 INF - Backup copy = 0

15:45:08 INF - Master server = jsawnetbackup

15:45:08 INF - Media server = jsawecd

15:45:09 INF - Multiplexing = 0

15:45:09 INF - New data socket = jsawecd.IPC:/tmp/vnet-58990392113704514490000000000-tLaoUa;6b7f5cdb2851ddc25948a36bd1f9ac4d;10;300

15:45:09 INF - Use shared memory = 1

15:45:09 INF - Compression = 0

15:45:09 INF - Encrypt = 0

15:45:10 INF - Keep logs = 4

15:45:10 INF - Client read timeout = 300

15:45:10 INF - Media mount timeout = 0

15:45:27 INF - Data buffer size = 65536

15:45:28 INF - Waiting for mount of media id 0057L5 on server jsawecd for writing

15:45:29 INF - Using Media ID 0057L5 for Backup ID jsawecd_1392113701

15:45:29 /

15:45:29 /PROXY_BACKUP

15:45:29 INF - Processing /oracle/ECD/sapdata1/sr3_13/sr3.data13

15:45:33 ERR - Lock file /usr/openv/netbackup/logs/user_ops/dbext/logs/vxbsa.1392113694.929958.files.1.lock is unlocked. Abort backup.

15:45:39 INF - Server status = 6

15:45:40 INF - Backup by oraecd on client jsawecd using policy SAP-JSAWECD-FULL, sched FULL-Application-Backup:the backup failed to back up the requested files

[aquib]

/usr/openv/netbackup/logs/user_ops/dbext/logs/vxbsa.1392113694.929958.files.1.lock is unlocked. Abort backup.

[Mark_Solutions]

So refer to the tech note i linked earlier - but the path in question is not as per the tech note.

So do this:

$ chgrp dba /usr/openv/netbackup/logs/user_ops/dbext
$ chmod g+rwx /usr/openv/netbackup/logs/user_ops/dbext
 

[aquib]

It is allready full permission!!

# ls -ltr

total 8

drwxr-x--x    4 root     system          256 Apr 05 2011  root

drwxrwxrwx    4 oraecd   dba             256 May 04 2011  dbext

drwxrwxrwx    2 root     bin             256 Dec 07 2012  nbjlogs

drwxrwxrwx    2 oraecd   dba            4096 Feb 11 15:45 sap

[Mark_Solutions]

What is the logs directory structure?

Just wondering about the nbjlogs which has only root access?

[aquib]

Mark,

I am sharing you my other server logs directory structure, It is complete the backup succssfully.

# ls -ltr

total 8

drwxrwxrwx    4 oraecp   dba             256 May 03 2011  dbext

drwxr-x--x    4 root     system          256 May 04 2011  root

drwxrwxrwx    2 root     bin             256 Apr 12 2012  nbjlogs

drwxrwxrwx    2 oraecp   dba            4096 Feb 11 07:04 sap

[Mark_Solutions]

Ok then perhaps it is another issue - we would need more logs to establish it - but you may be able to identify it yourself..

Please take a look through these which can all create the same error in the job log but need other logs to pin it down:

http://www.symantec.com/docs/TECH52446
http://www.symantec.com/docs/TECH63036
http://www.symantec.com/docs/TECH58234

[Marianne]

Please show us output of:

# ls -l /usr/openv/netbackup/logs/user_ops/dbext/logs/vxbsa.1392113694.929958.files.1.lock 
# ls -l /usr/openv/netbackup/logs/user_ops/dbext/logs

[aquib]

Hi Marianne,

Please find the output:

# ls -l /usr/openv/netbackup/logs/user_ops/dbext/logs/vxbsa.1392113694.929958.files.1.lock

-rw-------    1 oraecd   dba              18 Feb 11 15:44 /usr/openv/netbackup/logs/user_ops/dbext/logs/vxbsa.1392113694.929958.files.1.lock

# ls -l /usr/openv/netbackup/logs/user_ops/dbext/logs

total 360

-rw-r--r--    1 oraecd   dba           10850 Feb 11 13:26 vxbsa.1392105384.942080.files.1

-rw-------    1 oraecd   dba              18 Feb 11 13:26 vxbsa.1392105384.942080.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 13:27 vxbsa.1392105384.942080.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 13:30 vxbsa.1392105621.938116.files.1

-rw-------    1 oraecd   dba              18 Feb 11 13:30 vxbsa.1392105621.938116.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 13:31 vxbsa.1392105621.938116.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 13:52 vxbsa.1392106972.942182.files.1

-rw-------    1 oraecd   dba              18 Feb 11 13:52 vxbsa.1392106972.942182.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 13:53 vxbsa.1392106972.942182.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 15:01 vxbsa.1392111087.864458.files.1

-rw-------    1 oraecd   dba              18 Feb 11 15:01 vxbsa.1392111087.864458.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 15:02 vxbsa.1392111087.864458.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 15:10 vxbsa.1392111654.938164.files.1

-rw-------    1 oraecd   dba              18 Feb 11 15:10 vxbsa.1392111654.938164.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 15:11 vxbsa.1392111654.938164.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 15:20 vxbsa.1392112227.962720.files.1

-rw-------    1 oraecd   dba              18 Feb 11 15:20 vxbsa.1392112227.962720.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 15:23 vxbsa.1392112227.962720.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 15:29 vxbsa.1392112773.421928.files.1

-rw-------    1 oraecd   dba              18 Feb 11 15:29 vxbsa.1392112773.421928.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 15:30 vxbsa.1392112773.421928.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 15:44 vxbsa.1392113694.929958.files.1

-rw-------    1 oraecd   dba              18 Feb 11 15:44 vxbsa.1392113694.929958.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 15:45 vxbsa.1392113694.929958.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 18:02 vxbsa.1392121930.356396.files.1

-rw-------    1 oraecd   dba              18 Feb 11 18:02 vxbsa.1392121930.356396.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 18:02 vxbsa.1392121930.356396.prog.pcb_std

[aquib]

Hi Marianne,

please find the output:

# ls -l /usr/openv/netbackup/logs/user_ops/dbext/logs/vxbsa.1392113694.929958.files.1.lock

-rw-------    1 oraecd   dba              18 Feb 11 15:44 /usr/openv/netbackup/logs/user_ops/dbext/logs/vxbsa.1392113694.929958.files.1.lock

# ls -l /usr/openv/netbackup/logs/user_ops/dbext/logs

total 360

-rw-r--r--    1 oraecd   dba           10850 Feb 11 13:26 vxbsa.1392105384.942080.files.1

-rw-------    1 oraecd   dba              18 Feb 11 13:26 vxbsa.1392105384.942080.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 13:27 vxbsa.1392105384.942080.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 13:30 vxbsa.1392105621.938116.files.1

-rw-------    1 oraecd   dba              18 Feb 11 13:30 vxbsa.1392105621.938116.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 13:31 vxbsa.1392105621.938116.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 13:52 vxbsa.1392106972.942182.files.1

-rw-------    1 oraecd   dba              18 Feb 11 13:52 vxbsa.1392106972.942182.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 13:53 vxbsa.1392106972.942182.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 15:01 vxbsa.1392111087.864458.files.1

-rw-------    1 oraecd   dba              18 Feb 11 15:01 vxbsa.1392111087.864458.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 15:02 vxbsa.1392111087.864458.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 15:10 vxbsa.1392111654.938164.files.1

-rw-------    1 oraecd   dba              18 Feb 11 15:10 vxbsa.1392111654.938164.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 15:11 vxbsa.1392111654.938164.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 15:20 vxbsa.1392112227.962720.files.1

-rw-------    1 oraecd   dba              18 Feb 11 15:20 vxbsa.1392112227.962720.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 15:23 vxbsa.1392112227.962720.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 15:29 vxbsa.1392112773.421928.files.1

-rw-------    1 oraecd   dba              18 Feb 11 15:29 vxbsa.1392112773.421928.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 15:30 vxbsa.1392112773.421928.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 15:44 vxbsa.1392113694.929958.files.1

-rw-------    1 oraecd   dba              18 Feb 11 15:44 vxbsa.1392113694.929958.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 15:45 vxbsa.1392113694.929958.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 18:02 vxbsa.1392121930.356396.files.1

-rw-------    1 oraecd   dba              18 Feb 11 18:02 vxbsa.1392121930.356396.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 18:02 vxbsa.1392121930.356396.prog.pcb_std

[aquib]

# ls -l /usr/openv/netbackup/logs/user_ops/dbext/logs/vxbsa.1392113694.929958.files.1.lock

-rw-------    1 oraecd   dba              18 Feb 11 15:44 /usr/openv/netbackup/logs/user_ops/dbext/logs/vxbsa.1392113694.929958.files.1.lock

# ls -l /usr/openv/netbackup/logs/user_ops/dbext/logs

total 360

-rw-r--r--    1 oraecd   dba           10850 Feb 11 13:26 vxbsa.1392105384.942080.files.1

-rw-------    1 oraecd   dba              18 Feb 11 13:26 vxbsa.1392105384.942080.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 13:27 vxbsa.1392105384.942080.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 13:30 vxbsa.1392105621.938116.files.1

-rw-------    1 oraecd   dba              18 Feb 11 13:30 vxbsa.1392105621.938116.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 13:31 vxbsa.1392105621.938116.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 13:52 vxbsa.1392106972.942182.files.1

-rw-------    1 oraecd   dba              18 Feb 11 13:52 vxbsa.1392106972.942182.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 13:53 vxbsa.1392106972.942182.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 15:01 vxbsa.1392111087.864458.files.1

-rw-------    1 oraecd   dba              18 Feb 11 15:01 vxbsa.1392111087.864458.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 15:02 vxbsa.1392111087.864458.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 15:10 vxbsa.1392111654.938164.files.1

-rw-------    1 oraecd   dba              18 Feb 11 15:10 vxbsa.1392111654.938164.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 15:11 vxbsa.1392111654.938164.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 15:20 vxbsa.1392112227.962720.files.1

-rw-------    1 oraecd   dba              18 Feb 11 15:20 vxbsa.1392112227.962720.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 15:23 vxbsa.1392112227.962720.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 15:29 vxbsa.1392112773.421928.files.1

-rw-------    1 oraecd   dba              18 Feb 11 15:29 vxbsa.1392112773.421928.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 15:30 vxbsa.1392112773.421928.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 15:44 vxbsa.1392113694.929958.files.1

-rw-------    1 oraecd   dba              18 Feb 11 15:44 vxbsa.1392113694.929958.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 15:45 vxbsa.1392113694.929958.prog.pcb_std

-rw-r--r--    1 oraecd   dba           10850 Feb 11 18:02 vxbsa.1392121930.356396.files.1

-rw-------    1 oraecd   dba              18 Feb 11 18:02 vxbsa.1392121930.356396.files.1.lock

-rw-r--r--    1 oraecd   dba            1678 Feb 11 18:02 vxbsa.1392121930.356396.prog.pcb_std

[aquib]

Hi, 

Please resolve this issue I cant under stand whathappen on ths server. 

[Marianne]

Have you gone through the TechNotes in Mark's post?

Here is another one that seems related:

http://www.symantec.com/docs/TECH170245

[aquib]

hi,

Find the /usr/openv/netbackup/logs/bphdb logs!!!

09:30:04.331 [725222] <2> logparams: -sb -rdbms sap -S jsawnetbackup -to 300 -c SAP-JSAWECD-CINC -s CINC_Backup -clnt jsawecd -CINC -kl 4 -b jsawecd_1392091200 -jobid 52489 -shm

09:30:04.339 [725222] <4> bphdb: Module: libncf, Release: 7.0.1 , Build: 00/00/0000 00:00:00 CDT (00000000)

09:30:04.339 [725222] <4> bphdb: System: AIX, jsawecd, 1, 6, 00C649B24C00, {5}

09:30:04.340 [725222] <4> bphdb: Current code page: ICU: ISO-8859-1, Max. character width: 1, Substitution character: 0x1A [.], Unicode substitution character: 0xfffd [ÿý]

09:30:04.340 [725222] <4> bphdb: Running as user '0', primary group '0'

09:30:04.340 [725222] <4> bphdb: Environment variable _=/usr/openv/netbackup/bin/bpcd

09:30:04.340 [725222] <4> bphdb: Environment variable LANG=en_US

09:30:04.340 [725222] <4> bphdb: Environment variable LOGIN=root

09:30:04.340 [725222] <4> bphdb: Environment variable PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java14_64/jre/bin:/usr/java14_64/bin:/usr/bin

09:30:04.340 [725222] <4> bphdb: Environment variable EXTENDED_HISTORY=ON

09:30:04.340 [725222] <4> bphdb: Environment variable LC__FASTMSG=true

09:30:04.340 [725222] <4> bphdb: Environment variable LOGNAME=root

09:30:04.340 [725222] <4> bphdb: Environment variable MAIL=/usr/spool/mail/root

09:30:04.340 [725222] <4> bphdb: Environment variable LOCPATH=/usr/lib/nls/loc

09:30:04.340 [725222] <4> bphdb: Environment variable USER=root

09:30:04.340 [725222] <4> bphdb: Environment variable AUTHSTATE=compat

09:30:04.340 [725222] <4> bphdb: Environment variable SHELL=/usr/bin/ksh

09:30:04.340 [725222] <4> bphdb: Environment variable ODMDIR=/etc/objrepos

09:30:04.340 [725222] <4> bphdb: Environment variable HOME=/

09:30:04.340 [725222] <4> bphdb: Environment variable TERM=ansi

09:30:04.340 [725222] <4> bphdb: Environment variable MAILMSG=[YOU HAVE NEW MAIL]

09:30:04.340 [725222] <4> bphdb: Environment variable PWD=/

09:30:04.340 [725222] <4> bphdb: Environment variable TZ=IST-05:30:00

09:30:04.340 [725222] <4> bphdb: Environment variable A__z=! LOGNAME

09:30:04.340 [725222] <4> bphdb: Environment variable IS_NETBACKUP_DAEMON=YES

09:30:04.340 [725222] <4> bphdb: Environment variable NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat

09:30:04.340 [725222] <4> bphdb: INF - setenv SAP_POLICY=SAP-JSAWECD-CINC

09:30:04.340 [725222] <4> bphdb: INF - setenv SAP_SNC_SCHED=CINC_Backup

09:30:04.340 [725222] <4> bphdb: INF - setenv SAP_SERVER=jsawnetbackup

09:30:04.340 [725222] <4> bphdb: INF - setenv SAP_CLIENT=jsawecd

09:30:04.340 [725222] <4> bphdb: INF - setenv SAP_FULL=0

09:30:04.340 [725222] <4> bphdb: INF - setenv SAP_INCR=0

09:30:04.340 [725222] <4> bphdb: INF - setenv SAP_CINC=1

09:30:04.340 [725222] <4> bphdb: INF - setenv SAP_USER_INITIATED=0

09:30:04.340 [725222] <4> bphdb: INF - setenv SAP_SCHEDULED=1

09:30:04.340 [725222] <4> bphdb: INF - BACKUP START

09:30:04.510 [725222] <4> bphdb: INF - CONTINUE BACKUP message received.

09:30:04.510 [725222] <4> bphdb: INF - Processing /usr/openv/netbackup/NBU_Scripts/SAP_Online_INCR_Backup

09:30:04.512 [725222] <4> bphdb: INF - Waiting for the child status.

09:30:04.523 [323638] <4> bphdb: INF - Child executing /usr/openv/netbackup/NBU_Scripts/SAP_Online_INCR_Backup

09:30:08.512 [725222] <16> bphdb: ERR - Script exited with status = 3 <valid archive image produced, but no files deleted due to non-fatal problems>

09:30:08.527 [725222] <16> bphdb: ERR - bphdb exit status = 6: the backup failed to back up the requested files

09:30:08.527 [725222] <4> bphdb: INF - EXIT STATUS 6: the backup failed to back up the requested files

09:40:14.435 [811034] <2> logparams: -sb -rdbms sap -S jsawnetbackup -to 300 -c SAP-JSAWECD-CINC -s CINC_Backup -clnt jsawecd -CINC -kl 4 -b jsawecd_1392091810 -jobid 52489 -shm

09:40:14.443 [811034] <4> bphdb: Module: libncf, Release: 7.0.1 , Build: 00/00/0000 00:00:00 CDT (00000000)

09:40:14.443 [811034] <4> bphdb: System: AIX, jsawecd, 1, 6, 00C649B24C00, {5}

09:40:14.444 [811034] <4> bphdb: Current code page: ICU: ISO-8859-1, Max. character width: 1, Substitution character: 0x1A [.], Unicode substitution character: 0xfffd [ÿý]

"log.021114" [Read only] 630 lines, 50194 characters