Tape Drive access attributes and control between NetBackup and Tar command

[Genesisclimber]

I have a drive dedicated for NetBackup scheduled jobs. However, several users have requested to utilize the drive for "tar" command backup.
The servers are running Red Hat Enterprise Linux 5.2 and utilizing NetBackup 7.0.

I did some tests to check the feasibility of having a drive shared for these two backup activities and encountered several situations that I would like to check here.
Since NetBackup was installed as root, all the policies created for the scheduled jobs would fire off as expected. The drive in question has the following setup:
crw------- /dev/nst0

Since these new users are not part of the privileged group, and neither do I want them to have root privileges, I created a non-privilege user account (e.g. taradmin).
Although they have permission for "tar" and "mt" command, they are unable to read/write to the tapes since they do not have permission so I updated as follow:-

crw----rw- /dev/nst0

This is when I noted two observation:-
a) Since this is a shared drive, NetBackup scheduled jobs will fire on a  regular basis and utilizes the drive. I noticed that after the job is done, the permission is resetted to default.
i.e. crw------- /dev/nst0

This means that the taradmin users are not able to utilize the drive any more unless this access is rectified. The question is: Why is the access attributes getting resetted after every NetBackup jobs (be it scheduled or manual)? Is there any way for me to prevent the access attributes from getting reset?

b) Since this is a shared drive as mentioned, I also tested the possibility of drive contention, so i did two tests as follow:-
Test #1: Trigger a NetBackup Job and shortly after while the job is still running, trigger a tar backup
Observation: Tar command fails returning the error :"Cannot open: Device or resource busy".
This is acceptable since the drive is being used, tar is effectively blocked.

Test #2: trigger a Tar backup and shortly after, fire a NetBackup job while the tar backup is still running
Observation: The NetBackup will try to mount the correct tape to the drive based on the policy that was triggered, and eventually the tar backup will fail reporting: "Cannot write: Input/output error". The NetBackup job will then complete the mount and start the backup.
My question:
- If the tar backup is running, wouldn't the drive be busy and would disallow NetBackup from taking control? Why was NetBackup able to take control of the drive when it is obviously busy i.e. write opretaion from the tar backup would be running? Is there any way to make NetBackup aware that the drive is in use and would queue/fail the backup job instead of taking control of the drive?

What are the solution or alternatives to overcome my concerns above?

Thanks in advance.

[Nicolai · Accepted solution]

You must reserve a tape drive before you can use it for other purposes as Netbackup scan the drives from time to time. I think the netbackup command tpreq command can do the job. It allow mounting a tape for non Netbackup use.  The command require root privileges, but creating a SUDO job should solve that issue.

You use command "tpunmount" to dismount the tape drive.

NAME
       tpreq--request a tape volume for mounting and assign a file name to the drive

SYNOPSIS
       /usr/openv/volmgr/bin/tpreq -m media_id [-a accessmode] [-d density] [-p poolname] [-f] filename

DESCRIPTION
       This  command  initiates  a mount request for a tape volume on a removable media device. The information that
       you specify with this command identifies and registers the specified file as a  logical  identifier  for  the
       mount request with Media Manager. It also manages access to the volume.

       Media  Manager  automatically  mounts  the  media  if  it is in a robotic drive. Otherwise, an operator mount
       request appears in the Device Monitor window. tpreq does not complete normally in the case of a mount request
       for  a  robotic drive, if operator intervention is required. These requests also appear in the Device Monitor
       window.

       When the operation is complete, use tpunmount to unmount the volume and remove the file name from the  direcâ
       tory in which the file was created.

       When  a  tpreq  command  is  initiated, a call is made to the script drive_mount_notify immediately after the
       media is successfully placed in a pre-selected drive. This script allows user special handling to occur  now.
       Control  is then returned to tpreq to resume processing. The script is only called from the tpreq command for
       the drives that are in robots  and  is  not  valid  for  stand-alone  drives.  This  script  resides  in  the
       /usr/openv/volmgr/bin/goodies   directory.   To   use   this  script,  activate  it  and  copy  it  into  the
       /usr/openv/volmgr/bin directory; usage information is documented within the script.

       The following applies only to NetBackup Enterprise Server:

       If you request optical disk densities (odiskwm or odiskwo), tpreq acts differently than with sequential  tape
       devices.  The  logical  file  name  is a link to the data partition of the disk device. By default, it is the
       character device. tpformat labels optical platters with the volume-header partition being the label  and  the
       data partition being the rest of the disk.

       You must have root privileges to run this command.

Example from the command manual:

Example 1
The following command creates a file named tape1 in the current working
directory. It links the file to the drive that contains the volume that has media ID
of JLR01. The access mode for the tape file is set to write, and a 1/4-inch
cartridge drive is assigned.

# tpreq -f tape1 -m jlr01 -a w -d qscsi

[sdo]

Not sure why the device protection mask gets reset.

As for tar not being able to interrupt NBU, but NBU being able to interrupt tar - i should think this is down to NBU's use of persistent SCSI reservations, and that tar probably doesn't.

[Genesisclimber]

I'm aware of tpreq and tpumount. In fact, the group of users did express their desire to utilize these commands for mount/unmount puposes. However, I was not keen to allow this for the following reasons:-

1. Since their entire process is meant to be automated, they need to know the media id of the newly inserted tape. These would mean opening up more netbackup commands for a non-netbackup backup jobs e.g. vmadd, vmdelete. Since the tape they would use for their backup would always be manually inserted to the mail slot or cartridge access port, I had decided it would be better for an administrator to oversee this using the operator console or Web admin interface of the tape library to mount the tape to the drive before their automated backup procedures kick off. The removal of netbackup commands for their use would safeguard my tapes in the library and avoid any error in deleting/renaming/mounting incorrect tape media on their end. Its either I block the command, or allow the command use, which effectively means they can do everything the command allow them to, even if they issued the arguments incorrectly.

2. Since their backup utilizes tar. utilizing netbackup commands to label the media id and mounting, although facilitate their activity, it could create problems and affect my own backup operations if an incorrect media was mounted.

Nonetheless, I would try out tpreq and tpunmount, and see if the issues I have is overcome.

[Nicolai]

You could try to setting the drive path's in the DOWN state to see if it changes anything (bit I doubt it). But seen from a NBU administrative point of view, it something I would avoid.

[Genesisclimber]

I tried using tpreq and tpunmount and here is what I observe:-

1. Ran tar backup and then started a NetBackup backup job. The latter was queued since the drive is in use. The use of tpreq made NetBackup aware that the drive is currently mounted (it is stated in Activity Monitor). The Netbackup job started after issuing the tpunmount, which released the drive.

2. Access privileged still needed to be set for non-root to read/write. After tpreq and backup job executed, tpunmount was run (releasing the drive and again, reflected in Activity Monitor) and the access privileged resetted as before. As sdw303 mention, it could be due to Netbackup use of perisstent SCSI reservations - something I am not comfortable of altering since I do not want my jobs to be affected in any way.

As for putting the drive to DOWN state, as you said, it is something to avoid even if solve the issues.

So in short, using the tar command to perform backup does not 'lock' the drive and in NetBackup perspective, it is still free for it to use (and it WILL use it from what I've seen in my test).

Seems like the only solution is to provide a dedicated drive for the other users for their tar backup operations, since sharing a drive for NetBackup and non-NetBackup backup activities raises the issues I've seen.

[Marianne]

Any reason why these backups have to be done outside of NBU?
Why not create a separate volume pool for these users and let them use bpbackup to start their own backups?
You have the Rolls Royce of backup products with some users still stuck with the ox wagon??

[Genesisclimber]

Well, its business politics... to make a long story short, it's due to cost and responsibility, especially when things go wrong. -_-

1. One good way is to obtain the license for Access Management - which would allow to define various user which ar limited to specific operations and access (as NBU_User). Obviously, this will incur additional cost.

2. Using the auth.conf to allow the user to access the Admin console, but limited to Backup/Restore and Activity Monitor. I tried this and it seems sufficient enough and I don't see any implications and issues that could arise so far, since they are limited in options in the GUI. A User-Backup Policy will need to be pre-created for their backup use. However, I did not have the time and chance to scrutinize this in depth so I'm not sure if there would be impact on my own scheduled backups (or settings). No additional cost but I'm not comfortable with this yet (and based on what I've read, limiting via auth.conf isn't really a sure deal for peace of mind).

Allowing them to trigger backup jobs via CLI isn't a bad idea, but the concerns I have is that some commands are utility commands and upon firing, they could do more harm than good. Some commands require root privilege (but can be overcome with Sudo) but similarly to the first post, any error on their end with the arguments may impact my setup. Last but not least, I've noted that most (if not all) netbackup commands access attributes allow any user to run them. However, certain commands blocked non-root user at their command level e.g. bplabel. This comes to another discrepancy I noted in the Command Manual, which state that some commands require root privilege (e.g. tpreq, tpunmount) BUT I was able to run these as non-privilege user... if tpreq can be issued by non-root user, and if they issued the media ID incorrectly, I may lose my backup data. How can I guarantee that when this users issue this command, it will NOT affect any of my operations i.e. wrong arguments supplied. In fact, even if they needed Sudo to run this command, I still can't control what they use as their arguments.

Another reason is the server load. Using NetBackup to fire backup/restore jobs will consume a significant amount of CPU load %. Tar on the other hand, utilizes far less load as a whole (w/o compression). More desirable for me since the server is expected to be constantly loaded with activities. Allowing a 3rd or more NetBackup job will overload the server and would most likely slow it to a crawl. Again, increasing the server specs may help but this will go back to cost... -_-

I'm trying to justify the cost of some of the options above so that I can do away with the tar and keep everything within NetBackup but till then, I will still be trying to find a way to 'maintain' and 'enforce' the access attributes for the shared drive. Open to any suggestions.

[Nicolai]

Can you separate the user from the master/media  server ?. Netbackup clients's don't have the "dangerous" commands. Both bpbackup & bprestore takre care of permissions for non-root users as well.

[Genesisclimber]

Well, was wondering about that too.

I foresee I may not be able to set this up on my current test environment since I'm using an Autoloader via SCSI-2. In my production environment, it should be possible since these devices are connected via a SAN switch. As such, I'll have to iron out any concerns before actually trying on the production environment.

Will keep this thread updated after I try this out.
Thanks.

[Marianne]

No need to have any interaction with library.
User on client issues 'bpbackup [-p policy] [-s schedule] -f listfile' -
Master server receives backup request, finds the correct policy, schedule, assigns recources (which includes robotic tape mount via robot control host).

To restore, user issues command 'bprestore filename' -
Restore request is received by master, resources are assigned - including robotic tape mount of the appropriate media id.