Error 99 NDMP Backup on EMC Celerra

[Moley]

Hi,

I am having an issue backing up a filesystem on my NAS Cellera. I keep getting error 99.

The filesystem is approx 2.2T in size. This job was an incremental with the following params.

SET TYPE=tar
SET DIRECT=Y
SET HIST=Y
SET SNAPSHOT=Y
/fastbackups-dr

The strange thing is these options work for a policy of a different name. i.e The only difference is the policy name. We use a differnent policy name for offsite backups but all the options are the same.

12/17/2008 23:17:14 - Error ndmpagent (pid=24788) NDMP_LOG_ERROR 0 Backup is aborted.
12/17/2008 23:17:16 - Error ndmpagent (pid=24788) NDMP backup failed, path = /fastbackups-dr
12/17/2008 23:17:17 - Error bptm (pid=24789) none of the NDMP backups for client drpf0018 completed successfully
12/17/2008 23:17:17 - Error bptm (pid=24789) media manager terminated by parent process
12/17/2008 23:18:20 - end writing; write time: 0:07:11
NDMP backup failure (99)

Log from the datamover:

2008-12-17 23:11:09: NDMP: 4: Session 518 (thread ndmp518) < Backup type: tar >
2008-12-17 23:11:09: NDMP: 4: Session 518 (thread ndmp518) Name: TYPE Value: tar
2008-12-17 23:11:09: NDMP: 4: Session 518 (thread ndmp518) Name: FILESYSTEM Value: /fastbackups-dr
2008-12-17 23:11:09: NDMP: 4: Session 518 (thread ndmp518) Name: PREFIX Value: /fastbackups-dr
2008-12-17 23:11:09: NDMP: 4: Session 518 (thread ndmp518) Name: LEVEL Value: 1
2008-12-17 23:11:09: NDMP: 4: Session 518 (thread ndmp518) Name: HIST Value: Y
2008-12-17 23:11:09: NDMP: 4: Session 518 (thread ndmp518) Name: UPDATE Value: y
2008-12-17 23:11:09: NDMP: 4: Session 518 (thread ndmp518) Name: SNAPSHOT Value: Y
2008-12-17 23:11:09: NDMP: 4: Session 518 (thread ndmp518) tape drive: c80t0l2, CALLER_ADDRESS 0xfd4f44.
2008-12-17 23:11:09: NDMP: 4: Thread ndmp518 uses a set of Pax threads with handle 0.
2008-12-17 23:11:09: NDMP: 6: Thread ndmp518 filterDialect could not be set - prereqs not met [pax_cmd_thread::setFilterDialect]
2008-12-17 23:16:46: NDMP: 4: Thread PlayC9 Backup of /fastbackups-dr aborted due to file system umount
2008-12-17 23:16:46: NDMP: 4: Thread bkup052 Backup was aborted
2008-12-17 23:16:46: NDMP: 4: < LOG type: 3, msg_id: 0, entry: Backup was aborted, hasAssociatedMsg: 0, associatedMsgSeq: 0 >
2008-12-17 23:16:46: NDMP: 4: Got stat buffer with data pointer=NULL at 1370, abort=1
2008-12-17 23:16:46: NDMP: 4: Thread nasa00 Backup root directory: /fastbackups-dr
2008-12-17 23:16:46: NDMP: 4: Thread nasa00 write count 14492556288
2008-12-17 23:16:46: NDMP: 4: Thread nasa00 time used 336 sec
2008-12-17 23:16:46: NDMP: 4: Thread nasa00 write rate 42067 KB/Sec
2008-12-17 23:16:46: NDMP: 4: Thread nasa00 read rate 0 KB/Sec
2008-12-17 23:16:46: NDMP: 4: Thread nasa00 average file size: 21051KB
2008-12-17 23:16:46: NDMP: 4: Thread nasa00 dir  or   0  size file processed: 572
2008-12-17 23:16:46: NDMP: 4: Thread nasa00   1B -- 8KB  size file processed: 0
2008-12-17 23:16:46: NDMP: 4: Thread nasa00  8KB -- 16KB size file processed: 0
2008-12-17 23:16:46: NDMP: 4: Thread nasa00 16KB -- 32KB size file processed: 1
2008-12-17 23:16:46: NDMP: 4: Thread nasa00 32KB -- 64KB size file processed: 0
2008-12-17 23:16:46: NDMP: 4: Thread nasa00 64KB -- 1MB  size file processed: 33
2008-12-17 23:16:46: NDMP: 4: Thread nasa00  1MB -- 32MB size file processed: 26
2008-12-17 23:16:46: NDMP: 4: Thread nasa00 32MB -- 1GB  size file processed: 39
2008-12-17 23:16:46: NDMP: 4: Thread nasa00    1G more   size file processed: 3
2008-12-17 23:16:46: NDMP: 6: Thread nasa00 server_archive: emctar vol 1, 674 files, 0 bytes read, 14492556288 bytes written
2008-12-17 23:16:46: NDMP: 4: Got stat buffer with data pointer=NULL at 1370, abort=1
2008-12-17 23:16:46: NDMP: 4: Thread nasa00 cifs_ea new count: 3390368, delete count: 3390368, freeListCount: 383
2008-12-17 23:16:46: NDMP: 3: Thread nasa00 Backup is aborted.
2008-12-17 23:16:46: NDMP: 3: < LOG type: 2, msg_id: 0, entry: Backup is aborted., hasAssociatedMsg: 0, associatedMsgSeq: 0 >

Regards,

Paul

[Darren_Dunham]

99 is a generic NDMP failure status. Anything that causes the NDMP host to abort will likely generate a 99 on the NBU side.

---

     2008-12-17 23:16:46: NDMP: 4: Thread PlayC9 Backup of /fastbackups-dr aborted due to file system umount

---

That entry looks pretty serious.  Could /fastbackups-dr have been unmounted during the backup?  It looks like that caused the abort on the Celerra side, which then informed NBU that it was stopping the backup.

Either way it looks to me like a Celerra issue, not a NBU or NBU config issue.

-- 

Darren

[Moley]

Thanks for the reply.

Not sure why I would get that umount message. Its really strange as this filesystem backs up OK in a different policy with the same options. I ran it after this failed and it ran with no issues.

I shall be taking this up with EMC.

Regards,

Paul

[Srikanth_Gubbal]

hi,

you can also check this.

Document ID: 316263
http://support.veritas.com/docs/316263 E-Mail this document to a colleague
STATUS CODE 99: ndmp_log_error <sequence number> creating a checkpoint on a volume.

---

Details:
Overview:
Message in the error report or job details indicate "NDMP_LOG_ERROR <sequence number> creating checkpoint". This is followed by 'NDMP backup path failed" ,
and "media manager terminated by parent process". The job then exits with a status 99.

Troubleshooting:
NDMP backups for a Sun NAS client can fail if the filesystem being backed up is 90 percent full or greater. The following document explains the 90 percent threshold with the checkpoints.
http://docs.sun.com/source/819-4284-11/Admin_11_Maintenance.html

Resolution:
Free up space accordingly, or increase the size of the volume.

[Moley · Accepted solution]

Hi,

Sorry guys this was my bad.

I had SET SNAPSHOT=y instead of SET SNAPSURE=y

Regards,

Paul

-------------------------------------------------

Hi,

I am having an issue backing up a filesystem on my NAS Cellera. I keep getting error 99.

The filesystem is approx 2.2T in size. This job was an incremental with the following params.

SET TYPE=tar
SET DIRECT=Y
SET HIST=Y
SET SNAPSHOT=Y
/fastbackups-dr

[Omar_Villa]

Is there enough free space on your NAS? and what version of netbackup are u running?

let us know.

regards