06-07-2021 06:30 AM
I am using bpduplicate for regular image duplication to tape. There are about 400 images duplicated by one bpduplicate command. All images are duplicated to one LTO tape.
I expect there will be one mount and one dismout during the duplication. But the tape is dismounted and mounted about 14 times during single duplication. I want to eliminate these repetitive dismounts/mounts because they are causing job runtime prolongation and excessive drive and tape wear.
Has anyone solved it?
Solved! Go to Solution.
07-08-2021 02:32 AM
Support technician provided the solution:
Checked any allocations for the tape drives using nbrbutil -dump command
Found allocations and we have cleared the allocations using nbrbutil -resetall
Drive cleaning has been performed from the library end.
Cleared the cleaning comments using https://www.veritas.com/support/en_US/article.100046625
Ran the duplication job and unmount operation is working as expected now
RESOLUTION SUMMARY:
After clearing the allocations and performing the drive cleaning, the unmount operations was working as expected.
06-07-2021 07:24 AM
Hi @LubosDvorak
In order to understand the mount/dismount, you will have to figure out what causes the tape to be dismounted in the first place. There must be a reason what Netbackup think the tape needs to be dismounted. Netbackup is by default lazy and won't dismount unless there is a reason. Same reason Netbackup may choose to continue with backup jobs of lower priority, because higher priority jobs requires change of tape.
Is the data source disk ?
Is the destination retention and volume group the same ?
How do you start the bpduplicate command - one command or multiple commands ?
06-07-2021 07:27 AM
Hi @Nicolai ,
Data source is disk. Destination retention and volume group is the same. bpduplicate is started by one command.
Tape library is not used by any other process or application.
06-08-2021 12:51 AM
Hi @LubosDvorak
Not sure what causes the mount/dismount operation.
@mph999, @StoneRam-Simon Are you able to help @LubosDvorak ?
06-08-2021 01:29 AM
I would probably spend a bit of time feeling my way arront to work this one out, but based on previous experiences I would be wanting to look at what else is taking place around the time of the unload of the media..
Is is a case that the rehydration of data is taking too long, and the tape is being unloaded while it is waiting?
Is it a case of other jobs kicking in that want to use the same drive?
I'd be looking to see what if any impact the "Media unmount delay" value is having on this? is there any correlation between the value set for this and the frequency of the unloads you are seeing?
One other thing that popped up in my search...
https://www.veritas.com/support/en_US/article.100047764?profileURL=https%3A%2F%2Fvam-profile.veritas...
So could it be that the drive needs cleaning???
06-08-2021 03:54 PM
Hi @LubosDvorak
Can you share the job details from the activity monitor) for the duplication job (best to save to a file and attach) - this may provide some initial clues to the dismounts.
Thanks
David
06-09-2021 02:23 AM
Hi @davidmoline
Please find attached the duplication job details and a screenshot of the mounts/unmouts from the tape library.
Thanks,
Lubos
06-09-2021 03:28 AM
hey
One thing that comes to my mind... Maybe you have another job writing to tape and it is having higher priority?? Also I see you did limit number of partial volumes in this pool - can you try to bump it up? or if I am right lower the other jobs priority so the SLP in between images duplication will not 'on hold'....
06-09-2021 04:18 PM
HI @LubosDvorak
Have a read of the following article https://www.veritas.com/support/en_US/article.100020652 and see if the NetBackup volume pool has a maximum partially full volume setting other than 0 - this may be the cause of the constant dismounts.
If it is set to a value above 0, try changing to zero and and see what happens. If it is already zero, then not sure.
David
06-11-2021 12:49 AM
Hi @davidmoline ,
Setting maximum partially full volume to zero lead to mounting/dismounting of two tapes instead of one but total number of mounts/dismounts has not decreased.
Thanks,
Lubos
06-11-2021 12:53 AM
Hi@quebek ,
There is no other process writing to/reading from tape. Just one single bpduplicate CLI command is running.
Thanks,
Lubos
06-11-2021 08:48 AM
@LubosDvorak What is the media unmount delay for your master? Mine is set to 720 seconds and looks like NetBackup does wait long for unmounting the current tape so that it can use the same tape for similar job (like same retention, volume pool, etc.)
Hosts -> Master Server -> Your_Master -> Properties -> Media -> Media unmount delay:
06-14-2021 01:18 AM
@Tape_Archived
Media Unmount Delay changed 180 -> 720 secs, no impact on mounts/dismounts.
06-14-2021 03:07 AM
Did you verify that the tape drives do NOT need cleaning? Are there any messages on the media server in its various logs?
The "errors" file may also contain specific Tape Alert information (this file is on the media server)
Unix/Linux: /usr/openv/netbackup/db/media/errors
Windows: install_path\veritas\netbackup\db\media\errors
06-15-2021 04:34 AM
I believe this is not "needs cleaning" issue as there is no "drive needs cleaning" status in the library, there are no drive needs cleaning records in the /usr/openv/netbackup/logs/bptm/*.log files and the are no errors in /usr/openv/netbackup/db/media/errors
However thank you very much for focusing my attention on the btpm log files. I have found that the tape was unmounted always when the volume of processed images had exceeded 100 GB.
The only parameter having value of 100 GB is currently Maximum size per A.I.R. replication job (SLP.MAX_SIZE_PER_BACKUP_REPLICATION_JOB). I will increase it and see what happens.
If you have another idea please let me know.
Thank you.
06-15-2021 07:06 AM
Since 7.6 default Maximum size per duplication job is 100 GB
You can look to increase it here...
Host Properties > Master Server > SLP Parameters
The reasons why the tape is then un-mounted before another set of images is processed, would come down:-
The device for something else.
Maximum IO streams being reached on the disk...
or maybe..
SLP windows being set that mean another set doesn't get run straight away...
Some more info on tuning SLP for DIsk-to-Tape can be found here:-
https://www.veritas.com/support/en_US/article.100014136
06-16-2021 12:37 AM
Increasing Maximum size per A.I.R. replication job value had no effect.
Maximum size per duplication job was already set to 250 GB many months ago.
These values are obviously not used by the single bpduplicate command I am using (bpduplicate -hoursago 11 -cn 3 -sl Full -rl 3 -dstunit vdc3bck1-hcart2-robot-tld-1 -dp NetBackup -altreadhost vdc3bck1.company.com). SLP is not being used.
Going to open support ticket.
06-17-2021 07:09 AM
Hi @LubosDvorak
Please let us know the result from the support ticket, other may have the same issues like you.
Best Regards
Nicolai
06-18-2021 09:04 AM
@LubosDvorak
First I apologise for not re-reading your first post that stated you were using bpduplicate command...
Having looked at the command line options that you are passing, I do have a few other questions..
You are looking to duplicate ALL backup images that match the following selection criteria:-
Written in the last 11 hours.
They are all from schedules called "Full"
You are using the 3rd copy of any image as the source for the duplication
All data will be read by vdc3bck1 and sent over the network to the host where you are running the bpduplicate command.
Your setting the following parameters for the new copy:-
it will be using retention level 3
it will be put in the NetBackup pool
it will use the vdc3bck1-hcart2-robot-tld-1 storage unit
The bpduplicate job will run on the server where you run the command.. So given you are using the above storage unit and altreadhost I would assume you are running this on vdc3bck1
I would want to check the logs for the bpduplciate job in <instpath>/netbackup/logs/user_ops
I would also look to use "-p" option to preview what is being selected,
bpduplicate -hoursago 11 -cn 3 -sl Full -PM
I would then look to see if you have an conflicts in terms if the backupids being shown also being on the same media that you think you will be writing too... (You can't have two copies on the same tape, so it would end up changing media).
07-08-2021 02:32 AM
Support technician provided the solution:
Checked any allocations for the tape drives using nbrbutil -dump command
Found allocations and we have cleared the allocations using nbrbutil -resetall
Drive cleaning has been performed from the library end.
Cleared the cleaning comments using https://www.veritas.com/support/en_US/article.100046625
Ran the duplication job and unmount operation is working as expected now
RESOLUTION SUMMARY:
After clearing the allocations and performing the drive cleaning, the unmount operations was working as expected.