Netbackup 7.5 image files integrated into the relational database

[RLeon]

Hi all,

The following statements keep appearing in all the Netbackup 7.5 guides:

"Beginning with the NetBackup 7.5 release, the images files are now stored in the NetBackup relational database.
The images files contain the metadata that describes the backups."

"Starting with NetBackup 7.5, all backup image metadata is stored in the NetBackup relational database (NBDB).
Previous versions stored this data in either the NBDB or the flat ASCII files (image header files),
depending on the source of the image information.
The following describes the advantages of this change:
 - Eliminates the consistency issues for any data that previously existed in multiple databases.
 - Improves the product search performance, especially in large catalogs.
 - Improves the performance of restores, policy scheduling, and image cleanups."

The change makes perfect sense, but I am looking for some clarification for the following.

I understand that the so called "image database" is this directory structure that stores all the .f files for all the clients.
It is the "largest part of the NetBackup catalog".
Basically, it stores the metadata for every single file for every single backup, ever. So it is understandably big.
 - Install_path\VERITAS\Netbackup\db\images

And then there is the directory that stores the "actual" SQL Anywhere database files (contains the all important NBDB.db):
 - Install_path\VERITAS\NetbackupDB\data

I have a feeling that someone is going to point out that my understandings are totally off.
Regardless, the following is what I'm trying to find out.

In one of our environments, the image database directory with its thousands of .f files is around 400GB.
All the database files from the NetbackupDB\data directory combined don't even come close to that size.

I'm curious about what exactly do they mean when they say version 7.5 stores all image files in the NetBackup relational database (NBDB).
Does that mean that all .f files from the db\images directory will disappear, and the NBDB.db in the NetbackupDB\data directory would grow over 400GB?

Well, to find out, I setup a little testing environment with 7.5 installed.
And I don't see that happening.
Backups are still generating .f files in the Install_path\VERITAS\Netbackup\db\images directory for each client. Some are huge (broken down into multiple .f pieces).
So then, "what" exactly are now stored in the NetBackup relational database in version 7.5, that previous versions don't?
I ask because the .f image metadata files are definately still separated and are still being created in their same old db\images directory.

I hope my question makes sense.

Thanks all,

RLeon
 

[mph999]

I understand that the so called "image database" is this directory structure that stores all the .f files for all the clients.
It is the "largest part of the NetBackup catalog".
Basically, it stores the metadata for every single file for every single backup, ever. So it is understandably big.
 - Install_path\VERITAS\Netbackup\db\images

100% correct

The EMM SQL DB in NetbackupDB\data is much smaller - this contains many things including the media DB, the volDB, device config, many config sttings and so on.

At 7.5 it is the header files that go into the EMM SQL DB, not the .f file.

The image DB (before 7.5) contains .

The image header file (<policy name>_<ctime>_FULL (or INCR or UBAK) )

This contains details of where the image is stored (and a few other bits).

The .f file (of the same name, but .f on the end) contains a list of what files were backed up, and correctly, is the big bit.

It is the header file that goes into EMM, the .f file stays where it is.

If the .f file exceeds (I think 2MB) the contents are broken into mutiple files in the catsore dir, and the .f file is then 72 bytes in size.  This makes use of 'b trees' which is more efficient to search when the file gets big.

Martin

[mph999 · Accepted solution]

To finish the story ..

If you run nbdb_unload <some dir> the database is unloaded, and the dir will contain many .dat files, these are the DB tables.  There is also a reload.sql files that contains what is in which DB table.

It depends on the DB schema, so this 'will' change on upgrades, but on my system :

 783  DBM_MAIN     DBM_Image

 787  DBM_MAIN     DBM_ImageFragment

 783.date is the image data, and 787.dat us the Fragment data.

 Taking  a quick look :

 787.dat

  1: "ImageFragmentKey"               unsigned bigint    = '123'

  2: ,"ImageKey"                      unsigned bigint    = '126'

  3: ,"ImageCopyKey"                  unsigned bigint    = '115'

  4: ,"CopyNumber"                    integer            = '1'

  5: ,"FragmentNumber"                integer            = '1'

  6: ,"ResumeCount"                   integer            = '0'

  7: ,"MediaID"                       varchar(1024)      = 'R0TP01'

  8: ,"MediaServerKey"                unsigned int       = '1000011'

  9: ,"StorageUnitType"               integer            = '2'

 10: ,"StuSubType"                    smallint           = '1'

 11: ,"FragmentState"                 smallint           = '1'

 12: ,"FragmentSize"                  bigint             = '32768'

 13: ,"FragmentID"                    varchar(4096)      = 'R0TP01'

 14: ,"Density"                       integer            = '12'

 15: ,"FileNum"                       integer            = '1'

 16: ,"BlockSize"                     integer            = '65536'

 17: ,"Offset"                        integer            = '2'

 18: ,"MediaDate"                     bigint             = '1337635216'

 19: ,"DeviceWrittenOn"               integer            = '0'

 20: ,"FFlags"                        integer            = '0'

 21: ,"MediaDescription"              varchar(1024)      = ''

 22: ,"FragmentCheckpoint"            smallint           = '0'

 23: ,"MediaSequenceNum"              integer            = '0'

 24: ,"MediaExtents"                  varchar(4096)      = ''

 25: ,"SnapshotClientMountHost"       varchar(1024)      = ''

 26: ,"CreatedDateTime"               timestamp          = '2012-05-21 21:21:33.242832'

27: ,"LastModifiedDateTime"          timestamp          = '2012-05-22 09:34:39.255306'

783.dat 

1: "ImageKey"                       unsigned bigint    = '126'

 2: ,"MasterServerKey"               unsigned int       = '1000002'

 3: ,"PreviousBackupImageKey"        unsigned bigint    = 

 4: ,"ParentImageKey"                unsigned bigint    = 

 5: ,"CatArcImageKey"                unsigned bigint    = ,'womble

 6: ,"FullBackupImageKey"            unsigned bigint    = '1'

 7: ,"CSScheduleKey"                 unsigned int       = 'robot_0'

 8: ,"ClientMachineName"             varchar(1024)      = '1337635216'

 9: ,"ClientKey"                     unsigned bigint    = '0'

10: ,"PolicyName"                    varchar(1024)      = '0'

11: ,"BackupTime"                    bigint             = '0'

12: ,"ViewID"                        unsigned bigint    = '10'

13: ,"ScheduleType"                  integer            = '0'

14: ,"ClientType"                    integer            = '0'

15: ,"ProtocolVersion"               integer            = '0'

16: ,"StartTime"                     bigint             = '32'

17: ,"EndTime"                       bigint             = '1'

18: ,"SnapTime"                      bigint             = '1'

19: ,"KiloBytes"                     bigint             = '0'

20: ,"NumberFragments"               integer            = '9'

21: ,"NumberCopies"                  integer            = '1'

22: ,"NumberSnapshotDevices"         integer            = '0'

23: ,"ImageVersion"                  integer            = '0'

24: ,"RetentionLevel"                integer            = '0'

25: ,"Compression"                   smallint           = '0'

26: ,"Encryption"                    smallint           = '0'

27: ,"FilesFileCompressed"           smallint           = '0'

28: ,"Mpx"                           smallint           = '1'

29: ,"TirInfo"                       integer            = '0'

30: ,"TirExpiration"                 bigint             = '77'

31: ,"PrimaryCopy"                   integer            = '1338844816'

32: ,"ImageType"                     integer            = '6'

33: ,"ElapsedTime"                   integer            = '0'

34: ,"Expiration"                    bigint             = '0'

35: ,"NumberFiles"                   integer            = '0'

36: ,"ExtSecInfo"                    integer            = '0'

37: ,"RequestPID"                    integer            = '0'

38: ,"IndividualFileRestoreFromRaw"  integer            = '0'

39: ,"ImageDumpLevel"                integer            = '0'

40: ,"FileSystemOnly"                integer            = '0'

41: ,"PrevBlkIncrTime"               bigint             = '0'

42: ,"BlkIncrFullTime"               bigint             = '0'

43: ,"StreamNumber"                  integer            = '787'

44: ,"BackupCopy"                    integer            = '0'

45: ,"BackupStatus"                  integer            = '0'

46: ,"JobID"                         integer            = '0'

47: ,"NumResumes"                    integer            = '0'

48: ,"ResumeExpiration"              bigint             = 'root'

49: ,"PfiType"                       integer            = 'Full'

50: ,"ImageAttribute"                integer            = ''

51: ,"Creator"                       varchar(1024)      = ''

52: ,"ScheduleName"                  varchar(1024)      = 'robot_0_1337635216_FULL.f'

53: ,"ProxyClient"                   varchar(1024)      = '1312'

54: ,"KeywordPhrase"                 varchar(1024)      = ''

55: ,"FilesFile"                     varchar(1024)      = ''

56: ,"FilesFileSize"                 bigint             = '0'

57: ,"SoftwareVersion"               varchar(128)       = '0'

58: ,"ObjectDescriptor"              varchar(1024)      = '0'

59: ,"EstimatedKiloBytes"            bigint             = '1337635216'

60: ,"VmType"                        integer            = '1337635216'

61: ,"IsSynthetic"                   smallint           = '12'

62: ,"RunTime"                       bigint             = '1'

63: ,"LastBirthTime"                 bigint             = ''

64: ,"ClientCharacterSet"            integer            = 0x0000000000000000000000000000000

65: ,"BmrInfo"                       integer            = ''

66: ,"OriginMasterServer"            varchar(1024)      = '0'

67: ,"OriginMasterServerID"          binary(16)         = '0'

68: ,"StorageServiceName"            varchar(1024)      = '0'

69: ,"StorageServiceState"           integer            = ''

70: ,"StorageServiceIndexOffset"     integer            = '0'

71: ,"TimeInProcess"                 bigint             = '0'

72: ,"ClassificationID"              varchar(64)        = '0'

73: ,"StorageServiceVersionNumber"   integer            = '0'

74: ,"RequiredExpirationDate"        bigint             = '0'

75: ,"ImportFromReplicaTime"         bigint             = '0'

76: ,"IrEnabled"                     integer            = '0'

77: ,"IsNotQuiesced"                 smallint           = '0'

78: ,"IsNotSnapdupable"              smallint           = '0'

79: ,"IsNotScheduled"                smallint           = '0'

80: ,"HasDependees"                  smallint           = '0'

81: ,"HasDependents"                 smallint           = '0'

82: ,"ValidationCount"               integer            = '0'

83: ,"ExpirationCount"               integer            = '2012-05-21 21:20:17.182266'

84: ,"IndexingStatus"                smallint           = '2012-05-21 21:21:33.661940'

These outputs I have run through a perl script to make them easier to read, and they are just showing the details of one image/ fragement.

I've only had a quick look, but this looks like the tables that contain the 'old' image header file data.

There is a command:  cat_export that will 'copy' the details from the SQL database back into flat files.  It makes no changes, but just recreates the header files, for technical support use.

Martin

[RLeon]

Everything makes sense now. Thanks for the detailed explanation Martin.

RLeon