cancel
Showing results for 
Search instead for 
Did you mean: 

Many tape dismounts during single duplication job

LubosDvorak
Level 4

I am using bpduplicate for regular image duplication to tape. There are about 400 images duplicated by one bpduplicate command. All images are duplicated to one LTO tape.

I expect there will be one mount and one dismout during the duplication. But the tape is dismounted and mounted about 14 times during single duplication. I want to eliminate these repetitive dismounts/mounts because they are causing job runtime prolongation and excessive drive and tape wear.

Has anyone solved it?

 

1 ACCEPTED SOLUTION

Accepted Solutions

Support technician provided the solution:

Checked any allocations for the tape drives using nbrbutil -dump command
Found allocations and we have cleared the allocations using nbrbutil -resetall
Drive cleaning has been performed from the library end.
Cleared the cleaning comments using https://www.veritas.com/support/en_US/article.100046625
Ran the duplication job and unmount operation is working as expected now

RESOLUTION SUMMARY:
After clearing the allocations and performing the drive cleaning, the unmount operations was working as expected.

View solution in original post

19 REPLIES 19

Nicolai
Moderator
Moderator
Partner    VIP   

Hi @LubosDvorak 

In order to understand the mount/dismount, you will have to figure out what causes the tape to be dismounted in the first place. There must be a reason what Netbackup think the tape needs to be dismounted. Netbackup is by default lazy and won't dismount unless there is a reason. Same reason Netbackup may choose to continue with backup jobs of lower priority, because higher priority jobs requires change of tape. 

Is the data source disk ?

Is the destination retention and volume group the same ?

How do you start the bpduplicate command - one command or multiple commands ?

Hi @Nicolai ,

Data source is disk. Destination retention and volume group is the same. bpduplicate is started by one command.

Tape library is not used by any other process or application.

Nicolai
Moderator
Moderator
Partner    VIP   

Hi @LubosDvorak 

Not sure what causes the mount/dismount operation.

@mph999@StoneRam-Simon   Are you able to help @LubosDvorak  ?

 

StoneRam-Simon
Level 6
Partner    VIP    Accredited Certified

I would probably spend a bit of time feeling my way arront to work this one out, but based on previous experiences I would be wanting to look at what else is taking place around the time of the unload of the media..

Is is a case that the rehydration of data is taking too long, and the tape is being unloaded while it is waiting?
Is it a case of other jobs kicking in that want to use the same drive?

I'd be looking to see what if any impact the "Media unmount delay" value is having on this?  is there any correlation between the value set for this and the frequency of the unloads you are seeing?

One other thing that popped up in my search...
https://www.veritas.com/support/en_US/article.100047764?profileURL=https%3A%2F%2Fvam-profile.veritas...
So could it be that the drive needs cleaning???

 

Hi @LubosDvorak 

Can you share the job details from the activity monitor) for the duplication job (best to save to a file and attach) - this may provide some initial clues to the dismounts.

Thanks
David

Hi @davidmoline 

Please find attached the duplication job details and a screenshot of the mounts/unmouts from the tape library.

Thanks,

Lubos

quebek
Moderator
Moderator
   VIP    Certified

hey

One thing that comes to my mind... Maybe you have another job writing to tape and it is having higher priority?? Also I see you did limit number of partial volumes in this pool - can you try to bump it up? or if I am right lower the other jobs priority so the SLP in between images duplication will not 'on hold'....

HI @LubosDvorak 

Have a read of the following article https://www.veritas.com/support/en_US/article.100020652 and see if the NetBackup volume pool has a maximum partially full volume setting other than 0 - this may be the cause of the constant dismounts.

If it is set to a value above 0, try changing to zero and and see what happens. If it is already zero, then not sure.

David

Hi @davidmoline ,

Setting maximum partially full volume to zero lead to mounting/dismounting of two tapes instead of one but total number of mounts/dismounts has not decreased.

Thanks,

Lubos

Hi@quebek ,

There is no other process writing to/reading from tape. Just one single bpduplicate CLI command is running.

Thanks,

Lubos

Tape_Archived
Moderator
Moderator
   VIP   

@LubosDvorak What is the media unmount delay for your master? Mine is set to 720 seconds and looks like NetBackup does wait long for unmounting the current tape so that it can use the same tape for similar job (like same retention, volume pool, etc.)

Hosts -> Master Server -> Your_Master -> Properties -> Media -> Media unmount delay:

https://vox.veritas.com/t5/NetBackup/Could-I-ask-why-quot-Media-Unmount-Delay-quot-is-needed/td-p/48...

@Tape_Archived 
Media Unmount Delay changed 180 -> 720 secs, no impact on mounts/dismounts.

StoneRam-Simon
Level 6
Partner    VIP    Accredited Certified

Did you verify that the tape drives do NOT need cleaning?  Are there any messages on the media server in its various logs? 

The "errors" file may also contain specific Tape Alert information (this file is on the media server)

Unix/Linux: /usr/openv/netbackup/db/media/errors
Windows: install_path\veritas\netbackup\db\media\errors

@StoneRam-Simon 

I believe this is not "needs cleaning" issue as there is no "drive needs cleaning" status in the library, there are no drive needs cleaning records in the /usr/openv/netbackup/logs/bptm/*.log files and the are no errors in /usr/openv/netbackup/db/media/errors

However thank you very much for focusing my attention on the btpm log files. I have found that the tape was unmounted always when the volume of processed images had exceeded 100 GB.
The only parameter having value of 100 GB is currently Maximum size per A.I.R. replication job (SLP.MAX_SIZE_PER_BACKUP_REPLICATION_JOB). I will increase it and see what happens.

If you have another idea please let me know.
Thank you.

StoneRam-Simon
Level 6
Partner    VIP    Accredited Certified

Since 7.6 default Maximum size per duplication job is 100 GB
You can look to increase it here...
Host Properties > Master Server > SLP Parameters

The reasons why the tape is then un-mounted before another set of images is processed, would come down:-

The device for something else.

Maximum IO streams being reached on the disk...  
or maybe..
SLP windows being set that mean another set doesn't get run straight away...

Some more info on tuning SLP for DIsk-to-Tape can be found here:-

https://www.veritas.com/support/en_US/article.100014136

 

@StoneRam-Simon 

Increasing Maximum size per A.I.R. replication job value had no effect.

Maximum size per duplication job was already set to 250 GB many months ago.

These values are obviously not used by the single bpduplicate command I am using (bpduplicate -hoursago 11 -cn 3 -sl Full -rl 3 -dstunit vdc3bck1-hcart2-robot-tld-1 -dp NetBackup -altreadhost vdc3bck1.company.com). SLP is not being used.

Going to open support ticket.

Nicolai
Moderator
Moderator
Partner    VIP   

Hi @LubosDvorak 

Please let us know the result from the support ticket, other may have the same issues like you.

Best Regards

Nicolai

StoneRam-Simon
Level 6
Partner    VIP    Accredited Certified

@LubosDvorak 

First I apologise for not re-reading your first post that stated you were using bpduplicate command...
Having looked at the command line options that you are passing, I do have a few other questions..

You are looking to duplicate ALL backup images that match the following selection criteria:-

Written in the last 11 hours.
They are all from schedules called "Full"
You are using the 3rd copy of any image as the source for the duplication
All data will be read by vdc3bck1 and sent over the network to the host where you are running the bpduplicate command.

Your setting the following parameters for the new copy:-
it will be using retention level 3
it will be put in the NetBackup pool
it will use the vdc3bck1-hcart2-robot-tld-1 storage unit


The bpduplicate job will run on the server where you run the command..  So given you are using the above storage unit and altreadhost I would assume you are running this on vdc3bck1

I would want to check the logs for the bpduplciate job in <instpath>/netbackup/logs/user_ops

I would also look to use "-p" option  to preview what is being selected,

bpduplicate -hoursago 11 -cn 3 -sl Full  -PM 

I would then look to see if you have an conflicts in terms if the backupids being shown also being on the same media that you think you will be writing too...  (You can't have two copies on the same tape, so it would end up changing media).

 

Support technician provided the solution:

Checked any allocations for the tape drives using nbrbutil -dump command
Found allocations and we have cleared the allocations using nbrbutil -resetall
Drive cleaning has been performed from the library end.
Cleared the cleaning comments using https://www.veritas.com/support/en_US/article.100046625
Ran the duplication job and unmount operation is working as expected now

RESOLUTION SUMMARY:
After clearing the allocations and performing the drive cleaning, the unmount operations was working as expected.