NDMP backups error: with file read failed (13)

[Ivy_Yang]

hi,

my backup environment:

NBU master: NBU7.5.0.3 running on linux(CentOS release 5.7 (Final))

we backup a CIFS share from EMC VNX storage and write the data to EMC datadomain.

backup policy name:lasvnx_regonline_PCI

but the backup always ends with file read failed  (13)....

is there anybody who can help point me the right direction for troublhooting?

thanks!

2013-5-16 18:43:41 - Info nbjm (pid=1926) starting backup job (jobid=1233605) for client lasvnxpss01.active.tan, policy lasvnx_regonline_PCI, schedule Bi_weekly_full
2013-5-16 18:43:42 - Info bpbrm (pid=17246) lasvnxpss01.active.tan is the host to backup data from
2013-5-16 18:43:42 - Info bpbrm (pid=17246) reading file list from client
2013-5-16 18:43:42 - Info bpbrm (pid=17246) starting ndmpagent on client
2013-5-16 18:43:42 - Info ndmpagent (pid=17248) Backup started
2013-5-16 18:43:42 - Info bpbrm (pid=17246) bptm pid: 17249
2013-5-16 18:43:42 - Info bptm (pid=17249) start
2013-5-16 18:43:42 - Info nbjm (pid=1926) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=1233605, request id:{34372872-BE93-11E2-8A04-5E1C72428ECD})
2013-5-16 18:43:42 - requesting resource lx0034nbumed01_ndmp_2_dd860_rdsu01
2013-5-16 18:43:42 - requesting resource lx0034nbumast.NBU_CLIENT.MAXJOBS.lasvnxpss01.active.tan
2013-5-16 18:43:42 - requesting resource lx0034nbumast.NBU_POLICY.MAXJOBS.lasvnx_regonline_PCI
2013-5-16 18:43:42 - granted resource  lx0034nbumast.NBU_CLIENT.MAXJOBS.lasvnxpss01.active.tan
2013-5-16 18:43:42 - granted resource  lx0034nbumast.NBU_POLICY.MAXJOBS.lasvnx_regonline_PCI
2013-5-16 18:43:42 - granted resource  MediaID=@aaaa0;Path=/dd670-uswc02/backup/non_pci/repl_wcdc/lx0034nbumast_ndmp_2;MediaServer=lx0034nb...
2013-5-16 18:43:42 - granted resource  lx0034nbumed01_ndmp_2_dd860_rdsu01
2013-5-16 18:43:42 - estimated 0 kbytes needed
2013-5-16 18:43:42 - Info nbjm (pid=1926) started backup (backupid=lasvnxpss01.active.tan_1368755022) job for client lasvnxpss01.active.tan, policy lasvnx_regonline_PCI, schedule Bi_weekly_full on storage unit lx0034nbumed01_ndmp_2_dd860_rdsu01
2013-5-16 18:43:42 - started process bpbrm (pid=17246)
2013-5-16 18:43:42 - connecting
2013-5-16 18:43:42 - connected; connect time: 0:00:00
2013-5-16 18:43:43 - Info bptm (pid=17249) using 30 data buffers
2013-5-16 18:43:43 - Info bptm (pid=17249) using 262144 data buffer size
2013-5-16 18:43:44 - Info bptm (pid=17249) start backup
2013-5-16 18:43:44 - begin writing
2013-5-16 18:44:39 - Info ndmpagent (pid=17248) 0 entries sent to bpdbm
2013-5-16 18:59:16 - Info ndmpagent (pid=17248) lasvnxpss01.active.tan: Unable to open /root_vdm_5/PCI_rol_bkup_dump/PCI_rol_bkup_dump/VSQL9/Tlogs/RegOnline/Regonline20130515_194502.trn to read. Stale handle .
2013-5-16 19:25:44 - Info ndmpagent (pid=17248) lasvnxpss01.active.tan: Unable to open /root_vdm_5/PCI_rol_bkup_dump/PCI_rol_bkup_dump/VSQL10/Tlogs/Earth/Earth20130515_195500.trn to read. Stale handle .
2013-5-16 19:25:44 - Info ndmpagent (pid=17248) lasvnxpss01.active.tan: Unable to open /root_vdm_5/PCI_rol_bkup_dump/PCI_rol_bkup_dump/VSQL10/Tlogs/Earth/Earth20130515_200500.trn to read. Stale handle .
2013-5-16 19:26:47 - Info ndmpagent (pid=17248) lasvnxpss01.active.tan: Unable to open /root_vdm_5/PCI_rol_bkup_dump/PCI_rol_bkup_dump/VSQL10/Tlogs/Earth/Earth20130515_201500.trn to read. Stale handle .
2013-5-16 19:26:47 - Info ndmpagent (pid=17248) lasvnxpss01.active.tan: Unable to open /root_vdm_5/PCI_rol_bkup_dump/PCI_rol_bkup_dump/VSQL10/Tlogs/Earth/Earth20130515_202500.trn to read. Stale handle .
2013-5-16 19:31:18 - Info ndmpagent (pid=17248) lasvnxpss01.active.tan: server_archive: emctar vol 1, 1643 files, 0 bytes read, 218952158475 bytes written
2013-5-16 19:31:19 - Info ndmpagent (pid=17248) NDMP backup successfully completed, path = /root_vdm_5/PCI_rol_bkup_dump
2013-5-16 19:31:19 - Error bpbrm (pid=17246) db_FLISTsend failed: file read failed (13)
2013-5-16 19:31:20 - Info ndmpagent (pid=0) done
2013-5-16 19:31:41 - Error bptm (pid=17249) media manager terminated by parent process
2013-5-16 19:31:41 - Info ndmpagent (pid=0) done. status: 13: file read failed
2013-5-16 19:31:41 - end writing; write time: 0:47:57
file read failed  (13)
 

[Marianne]

Stale NFS handle:

Unable to open /root_vdm_5/...... to read. Stale handle .

You need to troubleshoot at OS level to find reason for stale NFS handle.

Stort term solution is to remount the NFS mount.

[Ivy_Yang]

actually it is CIFS share

[Marianne]

The path in the error message is a UNIX path. 
CIFS share is normally specified as UNC path.

Error also refers to ndmpagent which seems that your policy may be an NDMP policy type?

Please show us your policy config:

bppllist <policy-name> -U

[Ivy_Yang]

yes it is NDMP policy

[root@lx0034nbumast ~]# bppllist  lasvnx_regonline_PCI -U
------------------------------------------------------------

Policy Name:       lasvnx_regonline_PCI

  Policy Type:         NDMP
  Active:              yes
  Effective date:      10/11/2012 14:34:59
  Mult. Data Streams:  no
  Client Encrypt:      no
  Checkpoint:          no
  Policy Priority:     0
  Max Jobs/Policy:     Unlimited
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           lx0034nbumed01_ndmp_2_dd860_rdsu01
  Volume Pool:         NetBackup
  Server Group:        *ANY*
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    no
  Application Discovery:      no
  Discovery Lifetime:      0 seconds
ASC Application and attributes: (none defined)

  Granular Restore Info:  no
  Ignore Client Direct:  no
Enable Metadata Indexing:  no
Index server name:  NULL
  Use Accelerator:  no
  HW/OS/Client:  NDMP          NDMP          lasvnxpss01.active.tan

  Include:  /root_vdm_5/PCI_rol_bkup_dump

  Schedule:              Bi_weekly_full
    Type:                Full Backup
    Maximum MPX:         1
    Synthetic:           0
    Checksum Change Detection: 0
    PFI Recovery:        0
    Retention Level:     4 (2 months)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required)
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Calendar sched: Enabled
      Allowed to retry after run day
      SPECIFIC DATE 0 - 10/27/2012
      Saturday, Week 1
      Saturday, Week 3
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:     0
    Daily Windows:
          Sunday     03:00:00  -->  Sunday     23:00:00
          Monday     03:00:00  -->  Monday     23:00:00
          Tuesday    03:00:00  -->  Tuesday    23:00:00
          Wednesday  03:00:00  -->  Wednesday  23:00:00
          Thursday   03:00:00  -->  Thursday   23:00:00
          Friday     03:00:00  -->  Friday     23:00:00
          Saturday   03:00:00  -->  Saturday   23:00:00

  Schedule:              Diff_inc
    Type:                Differential Incremental Backup
    Maximum MPX:         1
    Synthetic:           0
    Checksum Change Detection: 0
    PFI Recovery:        0
    Retention Level:     3 (1 month)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required)
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Calendar sched: Enabled
      Allowed to retry after run day
      Sunday, Week 1
      Monday, Week 1
      Tuesday, Week 1
      Wednesday, Week 1
      Thursday, Week 1
      Friday, Week 1
      Sunday, Week 2
      Monday, Week 2
      Tuesday, Week 2
      Wednesday, Week 2
      Thursday, Week 2
      Friday, Week 2
      Saturday, Week 2
      Sunday, Week 3
      Monday, Week 3
      Tuesday, Week 3
      Wednesday, Week 3
      Thursday, Week 3
      Friday, Week 3
      Sunday, Week 4
      Monday, Week 4
      Tuesday, Week 4
      Wednesday, Week 4
      Thursday, Week 4
      Friday, Week 4
      Saturday, Week 4
      Sunday, Week 5
      Monday, Week 5
      Tuesday, Week 5
      Wednesday, Week 5
      Thursday, Week 5
      Friday, Week 5
      Saturday, Week 5
      EXCLUDE DATE 0 - 10/27/2012
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:     0
    Daily Windows:
          Sunday     03:00:00  -->  Sunday     18:00:00
          Monday     03:00:00  -->  Monday     18:00:00
          Tuesday    03:00:00  -->  Tuesday    18:00:00
          Wednesday  03:00:00  -->  Wednesday  18:00:00
          Thursday   03:00:00  -->  Thursday   18:00:00
          Friday     03:00:00  -->  Friday     18:00:00
          Saturday   03:00:00  -->  Saturday   18:00:00

[Marianne]

Seems you need to find out on the NAS filer what happened to the files that fail with the error:

 Unable to open <filename> to read. Stale handle .

[Ivy_Yang]

it is finally resolved via snapshot on Storage side.

and I backup thesnapshot filesystem instead of the original CIFS share.

emc181207 has the right anwser.