cancel
Showing results for 
Search instead for 
Did you mean: 

NDMP error 99

panthony
Level 4
Partner

Hi,

Since few days, I have some trouble on my NDMP backups. My backup selection is shared between many streams. All full backup are OK, but some of incr backup fails with status 99. Apparently, the level 0 dump is post-dated, and I have this error : "Aborting the incremental dump of level (1) because the snapshot specified for this backup is older than the one used in the previous dump."

 

14/02/2016 22:36:37 - connecting

14/02/2016 22:36:37 - connected; connect time: 00:00:00

14/02/2016 22:36:38 - mounting CAD126

14/02/2016 22:38:15 - Info bptm(pid=17364) media id CAD126 mounted on drive index 2, drivepath nrst1a, drivename HP.ULTRIUM5-SCSI.002, copy 1

14/02/2016 22:38:15 - mounted; mount time: 00:01:37

14/02/2016 22:38:20 - Info ndmpagent(pid=18976) bstou103: SCSI: TAPE READ: short read for nrst1a    

14/02/2016 22:38:21 - positioning CAD126 to file 44

14/02/2016 22:39:35 - Info ndmpagent(pid=18976) bstou103: DUMP: creating "/vol/volpoles/../snapshot_for_backup.3112" snapshot.       

14/02/2016 22:39:35 - positioned CAD126; position time: 00:01:14

14/02/2016 22:39:35 - begin writing

14/02/2016 22:39:37 - Info ndmpagent(pid=18976) bstou103: DUMP: Using subtree dump       

14/02/2016 22:39:51 - Info ndmpagent(pid=18976) bstou103: DUMP: Date of this level 1 dump snapshot: Sun Feb 14 22:40:46 2016.

14/02/2016 22:39:52 - Info ndmpagent(pid=18976) bstou103: DUMP: Date of last level 0 dump: Tue Jan 19 04:14:07 2038.

14/02/2016 22:39:52 - Error ndmpagent(pid=18976) bstou103: DUMP: Aborting the incremental dump of level (1) because the snapshot specified for this backup is older than the one used in the previous dump.

14/02/2016 22:39:52 - Error ndmpagent(pid=18976) bstou103: DUMP: Dump failed to initialize.      

14/02/2016 22:39:53 - Error ndmpagent(pid=18976) bstou103: DUMP: DUMP IS ABORTED       

14/02/2016 22:39:54 - Info ndmpagent(pid=18976) bstou103: DUMP: Deleting "/vol/volpoles/../snapshot_for_backup.3112" snapshot.       

14/02/2016 22:39:56 - Error ndmpagent(pid=18976) bstou103: DATA: Operation terminated: EVENT: INTERNAL ERROR (for /vol/volpoles/qtpoles/PRP/SERIS)   

14/02/2016 22:39:56 - Error ndmpagent(pid=18976) NDMP backup failed, path = /vol/volpoles/qtpoles/PRP/SERIS      

14/02/2016 22:39:57 - Error bptm(pid=17364) none of the NDMP backups for client bstou103 completed successfully  

14/02/2016 22:40:23 - Info bptm(pid=17364) EXITING with status 99 <----------       

14/02/2016 22:40:24 - end writing; write time: 00:00:49

NDMP backup failure(99)

 

 

Any idea ? My version is 7.5.0.4

Thanks !

1 ACCEPTED SOLUTION

Accepted Solutions

cruisen
Level 6
Partner Accredited

OK ,

I saw that veritas has published a technote in  reference to this post:

https://www.veritas.com/support/en_US/article.000108612

Best regards,

Cruisen

 

 

 

View solution in original post

22 REPLIES 22

sdo
Moderator
Moderator
Partner    VIP    Certified

Are you using NetBackup Replication Director?

Exact NetBackup version and path level please?

Are you trying to backup a NetApp volume which has a SnapMirror relationship?  If yes, please show the SnapMirror details from both sides.

panthony
Level 4
Partner

Hi sdo,

I'm not using Netbckup Replication Director.

My version of Netbackup is 7.5.0.4. What do you mean by "path level" ? The location of the snapshot created ?

Indeed, the NetApp volume has a mirror to another volume. And, before, this was the mirror who was backed up, and everything was fine. But now, we backup the source volume, and the mirror has now the same errors.

 

Here is what I tested : 

-a full backup of a path and then an incremental of the same path : error 99 during the incremental backup

-a full backup of a directory in this path and then an incremental : ok

 

The only thing I see is the volumetry : all my directories until about 350G are backed up, and all the directories which falls have more than 777Go.

 

sdo
Moderator
Moderator
Partner    VIP    Certified

Sorry I meant "patch level", but you've covered that with v7.5.0.4.

I think your problem is that... this is what you used to do:

source --> snapmirror --> target --> backup         i.e. the backup was always based on the latest snapshot

...now you are doing:

source +-> backup
       |
       +-> snapmirror --> target

My understanding is that NetApp snapmirror is effectively a "pull architecture", i.e. the internal management of snapshots are maintained frrom what we view as the target, but is really the orchestrator.  i.e. NetApp is refusing to let you effectively fork the snapshots of the source.

On the face of it I can't see why you can't do what you want to do, because the snapmirror position appears to be maintained at both sides.  But from the logs, it's not NetBackup complaining, it is the NetApp complaining that it is being asked to do something that it cannot comply with.

I think you're going to have to go to NetApp and see if what you want to do is a supported topology/configuration.

panthony
Level 4
Partner

I will see that, but I don't understand why this is OK for many directories (the smallers) and not for the biggers...

sdo
Moderator
Moderator
Partner    VIP    Certified

In which case, I'm wrong about the above, and an NDMP based snapshot can be taken from either side when an existing snapmirror relationship is in place.

AFAIK, snapmirror relationships can only be established at the volume or LUN layers.   But I'm probably wrong on this too, and maybe they can now be established at the Qtree or folder level.   Either way there's something within the NetApp blocking/rejecting the request... and so... you've guessed it... it's probably time to open a case with NetApp Support?

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
Backup level is determined by /etc/dumpdates on the NetApp filer. Did anyone at any point change the date on the filer? With backup done during this period? You need to find out from NetApp if this file can be modified by hand to change the year to 2016.

sdo
Moderator
Moderator
Partner    VIP    Certified

Well spotted Marianne - I missed that:

Date of last level 0 dump: Tue Jan 19 04:14:07 2038 

The above could be a corrupt date.

2  
31  
2,147,483,648   i.e. 2^31
-1   minus 1
2,147,483,647   max seconds
01/01/1970 00:00:00   start of time
19/01/2038 03:14:07   max time

...plus an hour for daylight saving :)

Will_Restore
Level 6

That timestamp is Netbackup "infinity". 

Hope to be retired by then; hope my pension checks don't stop. cheeky

 

sdo
Moderator
Moderator
Partner    VIP    Certified

Will - you've made me have a thought...

...maybe the NDMP level 1 backup of that area is failing because the NetApp has never recorded a level 0 as having occured for some paths - i.e it is as if the client (the NetApp) lives in two policies, and policy A which does a level 0 has less paths in it, than say policy B which does a level 1.   So, when policy B tries to take a level 1 of a path, then the NetApp knows that there has never been a preceding level 0 of that path and so refuses to co-operate.

What else might cause this?   A level 0 from so long ago and/or very infrequent runs of level 0 followed the 'appearance/creation' of new Qtrees/shares/folders/files, and then an attempt of a level 1 of these new structures?  Maybe such a sceanrio makes the backups barf too.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
Please try to get hold of /etc/dumpdates on the filer. See this TN : NDMP backups - What triggers an Incremental versus a Full backup ? http://www.veritas.com/docs/000009728

panthony
Level 4
Partner

So here is the content of the /etc/dumpdates

 

/vol/vol0/      0 Sat Feb 13 12:01:24 2016

/vol/vol0/      1 Mon Feb 15 20:01:27 2016

/vol/vol0/      2 Tue Feb 16 20:01:27 2016

/vol/vol0/      3 Wed Feb 10 20:01:23 2016

/vol/vol0/      4 Thu Feb 11 20:01:28 2016

/vol/vol0/      5 Mon Aug  3 20:00:19 2015

/vol/vol0/      6 Tue Aug  4 20:00:25 2015

/vol/vol0/      7 Wed Aug  5 20:00:27 2015

/vol/vol0/      8 Thu Aug  6 20:00:23 2015

/vol/new_root/  0 Sat Jan 23 12:01:33 2016

/vol/new_root/  1 Mon Jan 25 20:01:34 2016

/vol/new_root/  2 Tue Jan 19 20:01:28 2016

/vol/new_root/  3 Wed Jan 20 20:01:25 2016

/vol/new_root/  4 Thu Jan 21 20:01:30 2016

/vol/volpoles/qtpoles/PDS       1 Fri Jan 24 08:37:01 2014

/vol/volpoles/qtpoles/PDS       0 Fri Jan 24 08:37:01 2014

/vol/volpoles/qtpoles/PRP/SESURE        0 Tue Jan 19 04:14:07 2038

/vol/volpoles/qtpoles/PRP/SDE   0 Tue Jan 19 04:14:07 2038

/vol/vol_divers/LAROUSSE        0 Fri Feb 12 16:04:48 2016

/vol/vol_divers/ROBERT  0 Fri Feb 12 16:04:48 2016

/vol/volpoles/qtpoles/PRP/SERIS 0 Tue Jan 19 04:14:07 2038

/vol/vol_services/      0 Fri Feb 12 18:06:50 2016

/vol/vol_pst/   0 Sat Feb  6 10:17:27 2016

/vol/vol_divers/LAROUSSE        1 Fri Feb 12 16:04:48 2016

/vol/vol_divers/ROBERT  1 Fri Feb 12 16:04:48 2016

/vol/vol_pst/   1 Sun Feb 14 22:06:08 2016

/vol/vol_services/      1 Fri Feb 12 18:06:50 2016

/vol/vol_metiers/       0 Sat May 22 14:26:02 2021

/vol/vol_utilisateurs/  0 Tue Jan 19 04:14:07 2038

/vol/voldirections/qtdirections/DAFCJ   0 Wed Feb  3 09:11:49 2016

/vol/voldirections/qtdirections/DRH     0 Fri Feb 12 11:42:00 2016

/vol/voldirections/qtdirections/DSPSI   0 Fri Feb 12 17:34:31 2016

/vol/volpoles/qtpoles/PSN       0 Tue Jan 19 04:14:07 2038

/vol/voldirections/qtdirections/DAFCJ   1 Wed Feb  3 09:11:49 2016

/vol/voldirections/qtdirections/DRH     1 Fri Feb 12 11:42:00 2016

/vol/voldirections/qtdirections/DSPSI   1 Fri Feb 12 17:35:06 2016

/vol/volpoles/qtpoles/PDS       2 Fri Jan 24 08:37:01 2014

/vol/vol_divers/LAROUSSE        2 Fri Feb  5 13:39:13 2016

/vol/vol_divers/ROBERT  2 Fri Feb  5 13:39:13 2016

/vol/voldirections/qtdirections/DAFCJ   2 Wed Feb  3 09:11:49 2016

/vol/voldirections/qtdirections/DRH     2 Tue Feb  9 16:47:45 2016

/vol/vol_services/      2 Tue Feb  9 15:22:42 2016

/vol/voldirections/qtdirections/DSPSI   2 Tue Feb  9 18:39:04 2016

/vol/vol_pst/   2 Tue Feb  9 22:09:58 2016

/vol/volpoles/qtpoles/PDS       3 Fri Jan 24 08:37:01 2014

/vol/vol_divers/LAROUSSE        3 Fri Feb  5 13:39:13 2016

/vol/vol_divers/ROBERT  3 Fri Feb  5 13:39:13 2016

/vol/vol_services/      3 Wed Feb 10 10:23:10 2016

/vol/voldirections/qtdirections/DAFCJ   3 Wed Feb  3 09:11:49 2016

/vol/voldirections/qtdirections/DRH     3 Tue Feb  9 16:47:45 2016

/vol/voldirections/qtdirections/DSPSI   3 Wed Feb 10 10:59:43 2016

/vol/vol_pst/   3 Wed Feb 10 10:49:05 2016

/vol/volpoles/qtpoles/PRP/SESURE/LERCM/SORA     0 Thu Oct  3 10:55:07 2013

/vol/volpoles/qtpoles/PRP/SESURE/LERCM/SORA     1 Fri Dec 11 16:49:22 2015

/vol/volpoles/qtpoles/PRP/SESURE/LERCM  0 Tue Jan 19 04:14:07 2038

 

 

There is actually some lines with the years 2038 and 2021. I edited one line to see if the incr backup success, and for the moment everything is ok.

Now, the question is why some dates are non valid...

But the problem is apparently on the NetApp side.

Thanks for the help !

Andy_Welburn
Level 6

I'd be more tempted to re-do a full backup for them which should overwrite their entries with correct dates etc.

 

All the incorrect ones are referencing full (level 0) backups and any subsequent differentials will reference those as a baseline - I'd be a little concerned that, if you changed the date manually to some "made up" date in the past, the incrementals may be baselining from the wrong date & therefore some data could be missed during subsequent incremental backups.

panthony
Level 4
Partner

Indeed, this not work...

My first incremental backup is OK, but it is post-dated too. So, my 2d incremental backup is NOK because the level 1 dump has a 2038 date.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Have you logged a call with NetApp about this?

You do understand that this is outside the control of NetBackup?

panthony
Level 4
Partner

I've opened a case with NetApp and Veritas.

I know there is no chance that this is a NBU problem, but it's a big coincidence I think that for some dumps (initiated by the filer, right ?), the date equals the "infinite" retention.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

It is not an issue with retention.

Simply the date on which the dump is done.

cruisen
Level 6
Partner Accredited

Hello,

Is there any update on this ???

Best regards,

Cruisen

 

Will_Restore
Level 6

I am curious too what NetApp says about that dumpdate.  In the meantime, found this:

"Data ONTAP denotes time as a signed 32-bit integer that is interpreted as the number of seconds since 1 January 1970, 00 hours 00 minutes 00 seconds (GMT). This interpretation imposes an upper limit of 03 hours 14 minutes 07 seconds on 19 January 2038 (GMT)."

https://library.netapp.com/ecmdocs/ECMP1196889/html/GUID-35986F6A-45FD-4847-A0E1-06FDEDAE2152.html

 

cruisen
Level 6
Partner Accredited

I got what Netbackup has to say.

Screenshot from 2016-04-26 17-07-14.png