02-16-2016 02:41 AM
Hi,
Since few days, I have some trouble on my NDMP backups. My backup selection is shared between many streams. All full backup are OK, but some of incr backup fails with status 99. Apparently, the level 0 dump is post-dated, and I have this error : "Aborting the incremental dump of level (1) because the snapshot specified for this backup is older than the one used in the previous dump."
14/02/2016 22:36:37 - connecting
14/02/2016 22:36:37 - connected; connect time: 00:00:00
14/02/2016 22:36:38 - mounting CAD126
14/02/2016 22:38:15 - Info bptm(pid=17364) media id CAD126 mounted on drive index 2, drivepath nrst1a, drivename HP.ULTRIUM5-SCSI.002, copy 1
14/02/2016 22:38:15 - mounted; mount time: 00:01:37
14/02/2016 22:38:20 - Info ndmpagent(pid=18976) bstou103: SCSI: TAPE READ: short read for nrst1a
14/02/2016 22:38:21 - positioning CAD126 to file 44
14/02/2016 22:39:35 - Info ndmpagent(pid=18976) bstou103: DUMP: creating "/vol/volpoles/../snapshot_
14/02/2016 22:39:35 - positioned CAD126; position time: 00:01:14
14/02/2016 22:39:35 - begin writing
14/02/2016 22:39:37 - Info ndmpagent(pid=18976) bstou103: DUMP: Using subtree dump
14/02/2016 22:39:51 - Info ndmpagent(pid=18976) bstou103: DUMP: Date of this level 1 dump snapshot: Sun Feb 14 22:40:46 2016.
14/02/2016 22:39:52 - Info ndmpagent(pid=18976) bstou103: DUMP: Date of last level 0 dump: Tue Jan 19 04:14:07 2038.
14/02/2016 22:39:52 - Error ndmpagent(pid=18976) bstou103: DUMP: Aborting the incremental dump of level (1) because the snapshot specified for this backup is older than the one used in the previous dump.
14/02/2016 22:39:52 - Error ndmpagent(pid=18976) bstou103: DUMP: Dump failed to initialize.
14/02/2016 22:39:53 - Error ndmpagent(pid=18976) bstou103: DUMP: DUMP IS ABORTED
14/02/2016 22:39:54 - Info ndmpagent(pid=18976) bstou103: DUMP: Deleting "/vol/volpoles/../snapshot_
14/02/2016 22:39:56 - Error ndmpagent(pid=18976) bstou103: DATA: Operation terminated: EVENT: INTERNAL ERROR (for /vol/volpoles/qtpoles/PRP/
14/02/2016 22:39:56 - Error ndmpagent(pid=18976) NDMP backup failed, path = /vol/volpoles/qtpoles/PRP/
14/02/2016 22:39:57 - Error bptm(pid=17364) none of the NDMP backups for client bstou103 completed successfully
14/02/2016 22:40:23 - Info bptm(pid=17364) EXITING with status 99 <----------
14/02/2016 22:40:24 - end writing; write time: 00:00:49
NDMP backup failure(99)
Any idea ? My version is 7.5.0.4
Thanks !
Solved! Go to Solution.
06-14-2016 06:27 AM
OK ,
I saw that veritas has published a technote in reference to this post:
https://www.veritas.com/support/en_US/article.000108612
Best regards,
Cruisen
02-16-2016 08:03 AM
Are you using NetBackup Replication Director?
Exact NetBackup version and path level please?
Are you trying to backup a NetApp volume which has a SnapMirror relationship? If yes, please show the SnapMirror details from both sides.
02-16-2016 08:38 AM
Hi sdo,
I'm not using Netbckup Replication Director.
My version of Netbackup is 7.5.0.4. What do you mean by "path level" ? The location of the snapshot created ?
Indeed, the NetApp volume has a mirror to another volume. And, before, this was the mirror who was backed up, and everything was fine. But now, we backup the source volume, and the mirror has now the same errors.
Here is what I tested :
-a full backup of a path and then an incremental of the same path : error 99 during the incremental backup
-a full backup of a directory in this path and then an incremental : ok
The only thing I see is the volumetry : all my directories until about 350G are backed up, and all the directories which falls have more than 777Go.
02-16-2016 08:48 AM
Sorry I meant "patch level", but you've covered that with v7.5.0.4.
I think your problem is that... this is what you used to do:
source --> snapmirror --> target --> backup i.e. the backup was always based on the latest snapshot
...now you are doing:
source +-> backup | +-> snapmirror --> target
My understanding is that NetApp snapmirror is effectively a "pull architecture", i.e. the internal management of snapshots are maintained frrom what we view as the target, but is really the orchestrator. i.e. NetApp is refusing to let you effectively fork the snapshots of the source.
On the face of it I can't see why you can't do what you want to do, because the snapmirror position appears to be maintained at both sides. But from the logs, it's not NetBackup complaining, it is the NetApp complaining that it is being asked to do something that it cannot comply with.
I think you're going to have to go to NetApp and see if what you want to do is a supported topology/configuration.
02-16-2016 09:05 AM
I will see that, but I don't understand why this is OK for many directories (the smallers) and not for the biggers...
02-16-2016 09:12 AM
In which case, I'm wrong about the above, and an NDMP based snapshot can be taken from either side when an existing snapmirror relationship is in place.
AFAIK, snapmirror relationships can only be established at the volume or LUN layers. But I'm probably wrong on this too, and maybe they can now be established at the Qtree or folder level. Either way there's something within the NetApp blocking/rejecting the request... and so... you've guessed it... it's probably time to open a case with NetApp Support?
02-16-2016 09:22 AM
02-16-2016 09:35 AM
Well spotted Marianne - I missed that:
Date of last level 0 dump: Tue Jan 19 04:14:07 2038
The above could be a corrupt date.
2 | |
31 | |
2,147,483,648 | i.e. 2^31 |
-1 | minus 1 |
2,147,483,647 | max seconds |
01/01/1970 00:00:00 | start of time |
19/01/2038 03:14:07 | max time |
...plus an hour for daylight saving :)
02-16-2016 12:21 PM
That timestamp is Netbackup "infinity".
Hope to be retired by then; hope my pension checks don't stop.
02-16-2016 12:40 PM
Will - you've made me have a thought...
...maybe the NDMP level 1 backup of that area is failing because the NetApp has never recorded a level 0 as having occured for some paths - i.e it is as if the client (the NetApp) lives in two policies, and policy A which does a level 0 has less paths in it, than say policy B which does a level 1. So, when policy B tries to take a level 1 of a path, then the NetApp knows that there has never been a preceding level 0 of that path and so refuses to co-operate.
What else might cause this? A level 0 from so long ago and/or very infrequent runs of level 0 followed the 'appearance/creation' of new Qtrees/shares/folders/files, and then an attempt of a level 1 of these new structures? Maybe such a sceanrio makes the backups barf too.
02-16-2016 10:15 PM
02-18-2016 02:12 AM
So here is the content of the /etc/dumpdates
/vol/vol0/ 0 Sat Feb 13 12:01:24 2016
/vol/vol0/ 1 Mon Feb 15 20:01:27 2016
/vol/vol0/ 2 Tue Feb 16 20:01:27 2016
/vol/vol0/ 3 Wed Feb 10 20:01:23 2016
/vol/vol0/ 4 Thu Feb 11 20:01:28 2016
/vol/vol0/ 5 Mon Aug 3 20:00:19 2015
/vol/vol0/ 6 Tue Aug 4 20:00:25 2015
/vol/vol0/ 7 Wed Aug 5 20:00:27 2015
/vol/vol0/ 8 Thu Aug 6 20:00:23 2015
/vol/new_root/ 0 Sat Jan 23 12:01:33 2016
/vol/new_root/ 1 Mon Jan 25 20:01:34 2016
/vol/new_root/ 2 Tue Jan 19 20:01:28 2016
/vol/new_root/ 3 Wed Jan 20 20:01:25 2016
/vol/new_root/ 4 Thu Jan 21 20:01:30 2016
/vol/volpoles/qtpoles/PDS
/vol/volpoles/qtpoles/PDS
/vol/volpoles/qtpoles/PRP/
/vol/volpoles/qtpoles/PRP/SDE
/vol/vol_divers/LAROUSSE
/vol/vol_divers/ROBERT 0 Fri Feb 12 16:04:48 2016
/vol/volpoles/qtpoles/PRP/
/vol/vol_services/ 0 Fri Feb 12 18:06:50 2016
/vol/vol_pst/ 0 Sat Feb 6 10:17:27 2016
/vol/vol_divers/LAROUSSE
/vol/vol_divers/ROBERT 1 Fri Feb 12 16:04:48 2016
/vol/vol_pst/ 1 Sun Feb 14 22:06:08 2016
/vol/vol_services/ 1 Fri Feb 12 18:06:50 2016
/vol/vol_metiers/ 0 Sat May 22 14:26:02 2021
/vol/vol_utilisateurs/ 0 Tue Jan 19 04:14:07 2038
/vol/voldirections/
/vol/voldirections/
/vol/voldirections/
/vol/volpoles/qtpoles/PSN
/vol/voldirections/
/vol/voldirections/
/vol/voldirections/
/vol/volpoles/qtpoles/PDS
/vol/vol_divers/LAROUSSE
/vol/vol_divers/ROBERT 2 Fri Feb 5 13:39:13 2016
/vol/voldirections/
/vol/voldirections/
/vol/vol_services/ 2 Tue Feb 9 15:22:42 2016
/vol/voldirections/
/vol/vol_pst/ 2 Tue Feb 9 22:09:58 2016
/vol/volpoles/qtpoles/PDS
/vol/vol_divers/LAROUSSE
/vol/vol_divers/ROBERT 3 Fri Feb 5 13:39:13 2016
/vol/vol_services/ 3 Wed Feb 10 10:23:10 2016
/vol/voldirections/
/vol/voldirections/
/vol/voldirections/
/vol/vol_pst/ 3 Wed Feb 10 10:49:05 2016
/vol/volpoles/qtpoles/PRP/
/vol/volpoles/qtpoles/PRP/
/vol/volpoles/qtpoles/PRP/
There is actually some lines with the years 2038 and 2021. I edited one line to see if the incr backup success, and for the moment everything is ok.
Now, the question is why some dates are non valid...
But the problem is apparently on the NetApp side.
Thanks for the help !
02-18-2016 02:48 AM
I'd be more tempted to re-do a full backup for them which should overwrite their entries with correct dates etc.
All the incorrect ones are referencing full (level 0) backups and any subsequent differentials will reference those as a baseline - I'd be a little concerned that, if you changed the date manually to some "made up" date in the past, the incrementals may be baselining from the wrong date & therefore some data could be missed during subsequent incremental backups.
02-18-2016 05:41 AM
Indeed, this not work...
My first incremental backup is OK, but it is post-dated too. So, my 2d incremental backup is NOK because the level 1 dump has a 2038 date.
02-18-2016 05:48 AM
Have you logged a call with NetApp about this?
You do understand that this is outside the control of NetBackup?
02-18-2016 06:37 AM
I've opened a case with NetApp and Veritas.
I know there is no chance that this is a NBU problem, but it's a big coincidence I think that for some dumps (initiated by the filer, right ?), the date equals the "infinite" retention.
02-18-2016 07:16 AM
03-22-2016 08:17 AM
Hello,
Is there any update on this ???
Best regards,
Cruisen
03-22-2016 10:19 AM
I am curious too what NetApp says about that dumpdate. In the meantime, found this:
"Data ONTAP denotes time as a signed 32-bit integer that is interpreted as the number of seconds since 1 January 1970, 00 hours 00 minutes 00 seconds (GMT). This interpretation imposes an upper limit of 03 hours 14 minutes 07 seconds on 19 January 2038 (GMT)."
https://library.netapp.com/ecmdocs/ECMP1196889/html/GUID-35986F6A-45FD-4847-A0E1-06FDEDAE2152.html
04-26-2016 08:09 AM
I got what Netbackup has to say.