SQL Server backup Error - status 96

[Nam_Sym]

Hi All,

I need help with SQL Server backups.

I have the following log from the Client:

OPERATION BACKUP
DATABASE "Zambia"
SQLHOST "Prepaidsvr1-Ndl"
NBSERVER "BACKUP-SVR-PR.ZESCOCORP.ZESCOAD.ZESCO.CO.ZM"
MAXTRANSFERSIZE 6
BLOCKSIZE 7
NUMBUFS 2
ENDOPER TRUE

INF - BACKUP STARTED USING
Microsoft SQL Server 2008 R2 (RTM) - 10.50.1600.1 (X64)
Apr  2 2010 15:48:46
Copyright (c) Microsoft Corporation
Enterprise Edition (64-bit) on Windows NT 6.2 <X64> (Build 9200: )
Batch = C:\Program Files\Veritas\NetBackup\DbExt\MsSql\ecash_daily_bkup.bch, Op# = 1
INF - Using backup image Prepaidsvr1-Ndl.MSSQL7.PREPAIDSVR1-NDL.db.Zambia.~.7.001of001.20140927094824..C
INF - backup database "Zambia" to VIRTUAL_DEVICE='VNBU0-3100-6416-1411804118' with  stats = 10, blocksize = 65536, maxtransfersize = 4194304, buffercount = 2
INF - Number of stripes: 1, Number of buffers per stripe 2.
INF - Created VDI object for SQL Server instance <Prepaidsvr1-Ndl>. Connection timeout is <300> seconds.
Backup started Sat Sep 27 09:49:24 2014

09:50:39 Initiating backup
09:50:58 INF - Server status = 96
09:51:10 INF - Backup by Administrator on client prepaidsvr1-ndl.zescocorp.zescoad.zesco.co.zm using policy E-cash_daily_bkups, sched Default-Application-Backup:unable to allocate new media for backup, storage unit has none available
ERR - Error in VxBSACreateObject: 3.
    CONTINUATION: - System detected error, operation aborted.
ERR - Error in GetCommand: 0x80770004.
DBMS MSG - ODBC return code <-1>, SQL State <37000>, SQL Message <3202><[Microsoft][ODBC SQL Server Driver][SQL Server]Write on "VNBU0-3100-6416-1411804118" failed: 995(The I/O operation has been aborted because of either a thread exit or an application request.)>.
DBMS MSG - SQL Message <3271><[Microsoft][ODBC SQL Server Driver][SQL Server]A nonrecoverable I/O error occurred on file "VNBU0-3100-6416-1411804118:" 995(The I/O operation has been aborted because of either a thread exit or an application request.).>
    CONTINUATION: - An abort request is preventing anything except termination actions.
DBMS MSG - SQL Message <3013><[Microsoft][ODBC SQL Server Driver][SQL Server]BACKUP DATABASE is terminating abnormally.>
ERR - Error found executing <backup database "Zambia" to VIRTUAL_DEVICE='VNBU0-3100-6416-1411804118' with  stats = 10, blocksize = 65536, maxtransfersize = 4194304, buffercount = 2>.
ERR - Error in VDS->Close: 0x80770004.
    CONTINUATION: - An abort request is preventing anything except termination actions.
INF - OPERATION #1 of batch C:\Program Files\Veritas\NetBackup\DbExt\MsSql\ecash_daily_bkup.bch FAILED with STATUS 1 (0 is normal). Elapsed time = 333(333) seconds.
INF - Results of executing <C:\Program Files\Veritas\NetBackup\DbExt\MsSql\ecash_daily_bkup.bch>:
<0> operations succeeded. <1> operations failed.

[sri_vani]

1)Is it a new configuration?did it run sucessfully any time?

2)Do u have enough scratches because could see 96 status code

3)Are u initiating the backup from master server?(BACKUP-SVR-PR.ZESCOCORP.ZESCOAD.ZESCO.CO.ZM)

 Initiate backups either from the master server or directly at the SQL console, and launch all restore operations directly from the SQL console.  

4)verify name resolution ...(dns)

[Mark_Solutions]

That log just says 2 things .. it is manually initiated by the user "Administrator" and there are no tapes available.

So 3 possible problems ..

1. the storage unit defined it its policy has no free tapes in the specified volume pool (or none with the same retention as the others .. remembering that the retention of a SQL job comes from the default application schedule and not the automatic one)

2. Administrator does not have rights to access the databases that are being backed up

3. The GUI / Console used was not opened using the "Run as administrator" option if UAC is enabled

To be honest i can only see it being 1. from the above

#edit#

re-read your post and certain that the media is the only certain issue here

[Nam_Sym]

Hi Sri,

Yes the configuration is new and  the SQL server backup has never been successful.

Scratch tapes are available.

The backup is being initiated from the master server.

Name resolution is working fine

[Nam_Sym]

Hi Mark,

My observation is that when the job runs the FULL backup schedule gets assigned media, while the DEFAULT APPLICATION schedule is not getting any media assigned.

I have done everything right but i'm not sure how to get round this one.

Thanks

[Marianne]

Please post all relevant info about media and the policy - this will help us to troubleshoot the status 96.

available_media
(Command is in ..\netbackup\bin\goodies)

bppllist <policy-name> -U
(Command is in ..\netbackup\bin\admincmd)

[Nam_Sym]

AVAILABLE MEDIA

media   media   robot   robot   robot   side/   ret    size     status
 ID     type    type      #     slot    face    level  KBytes
----------------------------------------------------------------------------
CatalogBackup pool

0013L4  DLT      TLD      0       13     -       -     -        AVAILABLE
0016L4  DLT      TLD      0       14     -       -     -        AVAILABLE

DataStore pool

E-cash_daily_bkups pool

0007L4  DLT      TLD      0       20     -       -     -        AVAILABLE
0018L4  DLT      TLD      0       19     -       -     -        AVAILABLE

NetBackup pool

0006L4  DLT      TLD      0       24     -       -     -        AVAILABLE
0008L4  DLT      TLD      0       16     -       -     -        AVAILABLE
0011L4  DLT      TLD      0       21     -       -     -        AVAILABLE
0012L4  DLT      TLD      0       17     -       -     -        AVAILABLE
0014L4  DLT      TLD      0       22     -       -     -        AVAILABLE
0015L4  DLT      TLD      0       18     -       -     -        AVAILABLE
0017L4  DLT      TLD      0       23     -       -     -        AVAILABLE
0019L4  DLT      TLD      0       15     -       -     -        AVAILABLE

None pool

Scratch pool

A00004  DLT      TLD      0       5      -       -     -        AVAILABLE
A00005  DLT      TLD      0       6      -       -     -        AVAILABLE
A00006  DLT      TLD      0       7      -       -     -        AVAILABLE
A00007  DLT      TLD      0       8      -       -     -        AVAILABLE
A00008  DLT      TLD      0       9      -       -     -        AVAILABLE
A00009  DLT      TLD      0       10     -       -     -        AVAILABLE
A00010  DLT      TLD      0       11     -       -     -        AVAILABLE
A00011  DLT      TLD      0       12     -       -     -        AVAILABLE
A00012  DLT      NONE     -       -      -       -     -        AVAILABLE
A00025  DLT      TLD      0       26     -       -     -        AVAILABLE
A00026  DLT      TLD      0       27     -       -     -        AVAILABLE
A00027  DLT      TLD      0       28     -       -     -        AVAILABLE
A00028  DLT      TLD      0       29     -       -     -        AVAILABLE
A00029  DLT      TLD      0       30     -       -     -        AVAILABLE
A00041  DLT      TLD      0       42     -       -     -        AVAILABLE
A00042  DLT      TLD      0       43     -       -     -        AVAILABLE
A00043  DLT      TLD      0       44     -       -     -        AVAILABLE
A00044  DLT      TLD      0       45     -       -     -        AVAILABLE
A00045  DLT      TLD      0       46     -       -     -        AVAILABLE
A00046  DLT      TLD      0       47     -       -     -        AVAILABLE
A00047  DLT      TLD      0       48     -       -     -        AVAILABLE
A00049  DLT      TLD      0       4      -       -     -        AVAILABLE
A00050  DLT      TLD      0       25     -       -     -        AVAILABLE
A00051  DLT      TLD      0       31     -       -     -        AVAILABLE
A00052  DLT      TLD      0       32     -       -     -        AVAILABLE
A00053  DLT      TLD      0       33     -       -     -        AVAILABLE
A00054  DLT      TLD      0       34     -       -     -        AVAILABLE
A00055  DLT      TLD      0       35     -       -     -        AVAILABLE
A00056  DLT      TLD      0       36     -       -     -        AVAILABLE
A00057  DLT      TLD      0       37     -       -     -        AVAILABLE
A00058  DLT      TLD      0       38     -       -     -        AVAILABLE
A00059  DLT      TLD      0       39     -       -     -        AVAILABLE
A00060  DLT      TLD      0       40     -       -     -        AVAILABLE
A00061  DLT      TLD      0       41     -       -     -        AVAILABLE

Test pool

A00000  DLT      TLD      0       1      -       -     -        AVAILABLE
A00001  DLT      TLD      0       2      -       -     -        AVAILABLE

ZESCODATA pool

A00048  DLT      TLD      0       3      -       -     -        AVAILABLE

[Nam_Sym]

bppllist <policy-name> -U

Policy Name:       E-cash_daily_bkups

  Policy Type:         MS-SQL-Server
  Active:              yes
  Effective date:      09/27/2014 09:46:41
  Mult. Data Streams:  no
  Client Encrypt:      no
  Checkpoint:          no
  Policy Priority:     0
  Max Jobs/Policy:     Unlimited
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           (specific storage unit not required)
  Volume Pool:         E-cash_daily_bkups
  Server Group:        *ANY*
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    no
  Application Discovery:      no
  Discovery Lifetime:      28800 seconds
ASC Application and attributes: (none defined)

  Granular Restore Info:  no
  Ignore Client Direct:  no
Enable Metadata Indexing:  no
Index server name:  NULL
  Use Accelerator:  no
  HW/OS/Client:  Windows-x64   Windows2008   prepaidsvr1-ndl.zescocorp.zescoad.z
esco.co.zm

  Include:  C:\Program Files\Veritas\NetBackup\DbExt\MsSql\ecash_daily_bkup.bch

  Schedule:              Full
    Type:                Automatic Backup
    Frequency:           every 1 day
    Synthetic:           0
    Checksum Change Detection: 0
    PFI Recovery:        0
    Retention Level:     9 (infinity)
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:     0
    Daily Windows:
          Sunday     18:00:00  -->  Monday     05:00:00
          Monday     18:00:00  -->  Tuesday    05:00:00
          Tuesday    18:00:00  -->  Wednesday  05:00:00
          Wednesday  18:00:00  -->  Thursday   05:00:00
          Thursday   18:00:00  -->  Friday     05:00:00
          Friday     18:00:00  -->  Saturday   05:00:00
          Saturday   18:00:00  -->  Sunday     05:00:00

  Schedule:              Default-Application-Backup
    Type:                Application Backup
    Maximum MPX:         1
    Synthetic:           0
    Checksum Change Detection: 0
    PFI Recovery:        0
    Retention Level:     9 (infinity)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required)
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:     0
    Daily Windows:
          Sunday     00:00:00  -->  Sunday     24:00:00
          Monday     00:00:00  -->  Monday     24:00:00
          Tuesday    00:00:00  -->  Tuesday    24:00:00
          Wednesday  00:00:00  -->  Wednesday  24:00:00
          Thursday   00:00:00  -->  Thursday   24:00:00
          Friday     00:00:00  -->  Friday     24:00:00
          Saturday   00:00:00  -->  Saturday   24:00:00

[Marianne]

I was hoping to see a storage unit name (that includes robot number and density) in the policy or in the schedule, but no sto. ge yniy is selected:   Residence:           (specific storage unit not required) So, next thing to check is storage unit type/density and robot number. All media is configured as DLT and in robot 0, which means that there needs to be a Storage Unit for robot 0 with DLT density. Plus DLT tape drives in robot 0 in device config. If there is a mismatch in any of these densities, it will cause status 96. Please check Storage Unit config and device config and tell us what you see.

[Nam_Sym]

Hi Marianne,

I have checked the storage unit type/density....

It seems the robot has 4 STUs with different densities as follows:

- hcart

- hcart2

-hcart3

-qscsi

While media density is DLT.

so how do we correct this.

Thanks

[Mark_Solutions]

If that was the case then none of your backups would work .. not just an SQL one.

Maybe if we could see a couple outputs it would help us decide what needs doing.

Please output these to files and attach to the thread where possible

bppllist -allpolicies -U

bpstulist -U -g

[Marianne]

Why so many different types of STU? 

What type of tape drives do you have in the robot?

And what type of tapes are these? 

We will be able to tell you how to fix it if we know the actual drive type and media type.

For example, if you have LTO4 tape drives in robot, there should also be LTO4 tapes in the robot. The default density/type for LTO4 is hcart. So, drive, media and STU type must all be hcart for backups to work.

Show us device config and contents of robot:

tpconfig -l

vmcheckxxx -rt tld -rn 0

Commands are in ...\volmgr\bin

[Mark_Solutions]

The outputs i asked for would be useful . to clarify i was at a customers site over the last couple of days and they have 2 robots which were named simialr to robot-tld-hcart-0 and robot-tld-hcart3-1 .. but both were actually hcart3 despite their names .. if we can see my and Mariannes outputs it would be great

[Marianne]

I thought  this looked familiar....

We've been through this exact same situation just over a month ago:  http://www.symantec.com/connect/forums/emm-status-no-media-available 

I thought you understood back then about  the need for same densities?

[Nam_Sym]

Hi Marianne,

Yes, you are right. This confusion was cause by the enginner who installed the tape library and insists that the robot should have 4 different densities.

Anyway, having inherited this my task is to correct what is not working..

OUTPUT

tpconfig -l

Type     Num Index  Type DrNum Status  Comment    Name                  Path

robot      0    -    TLD    -       -  -          -                     {7,0,4,1
}
  drive    -    0  qscsi    1      UP  -          HP.ULTRIUM6-SCSI.000  {7,0,4,0
}
  drive    -    1  hcart    2      UP  -          HP.ULTRIUM6-SCSI.001  {7,0,5,0
}
  drive    -    2 hcart2    3      UP  -          HP.ULTRIUM6-SCSI.002  {7,0,2,0
}
  drive    -    3 hcart3    4      UP  -          HP.ULTRIUM6-SCSI.003  {7,0,3,0

vmcheckxxx -rt tld -rn 0

Slot    Tape  Barcode           Media ID Barcode        Mismatch Detected
====    ====  =============     ======== =============  =================
   1     Yes  -none-            A00000   -none-
   2     Yes  -none-            A00001   -none-
   3     Yes  -none-            A00048   -none-
   4     Yes  -none-            A00049   -none-
   5     Yes  -none-            A00004   -none-
   6     Yes  -none-            A00005   -none-
   7     Yes  -none-            A00006   -none-
   8     Yes  -none-            A00007   -none-
   9     Yes  -none-            A00008   -none-
  10     Yes  -none-            A00009   -none-
  11     Yes  -none-            A00010   -none-
  12     Yes  -none-            A00011   -none-
  13     Yes  000013L4          0013L4   000013L4
  14     Yes  000016L4          0016L4   000016L4
  15     Yes  000019L4          0019L4   000019L4
  16     Yes  000008L4          0008L4   000008L4
  17     Yes  000012L4          0012L4   000012L4
  18     Yes  000015L4          0015L4   000015L4
  19     Yes  000018L4          0018L4   000018L4
  20     Yes  000007L4          0007L4   000007L4
  21     Yes  000011L4          0011L4   000011L4
  22     Yes  000014L4          0014L4   000014L4
  23     Yes  000017L4          0017L4   000017L4
  24     Yes  000006L4          0006L4   000006L4
  25     Yes  -none-            A00050   -none-
  26     Yes  -none-            A00025   -none-
  27     Yes  -none-            A00026   -none-
  28     Yes  -none-            A00027   -none-
  29     Yes  -none-            A00028   -none-
  30     Yes  -none-            A00029   -none-
  31     Yes  -none-            A00051   -none-
  32     Yes  -none-            A00052   -none-
  33     Yes  -none-            A00053   -none-
  34     Yes  -none-            A00054   -none-
  35     Yes  -none-            A00055   -none-
  36     Yes  -none-            A00056   -none-
  37     Yes  -none-            A00057   -none-
  38     Yes  -none-            A00058   -none-
  39     Yes  -none-            A00059   -none-
  40     Yes  -none-            A00060   -none-
  41     Yes  -none-            A00061   -none-
  42     Yes  -none-            A00041   -none-
  43     Yes  -none-            A00042   -none-
  44     Yes  -none-            A00043   -none-
  45     Yes  -none-            A00044   -none-
  46     Yes  -none-            A00045   -none-
  47     Yes  -none-            A00046   -none-
  48     Yes  -none-            A00047   -none-

[Nam_Sym]

Sorry ommitted this info:

Tape drive: HP Storage Works msl 4048

Tapes: Ultrium LTO6

[Mark_Solutions]

WOW!! Never seen anything like that!

Probably easiest to just delete all drives and the robot and storage unit then add it all back in - as you do check the properties of the robot and drives and set all to hcart3 (which i would expect them to be anyway)

Next you need to make all your media (I am assuming it is all LTO5 or LTO6) to also be hcart3.

To do this use the following command (from the volmgr\bin directory) substituting your master server name and using the media id of each tape that you have .. it should be easy to do a script to run through all of your media for you to do this

vmchange -h masterservername -new_mt hcart3 -m ABC123

[sksujeet]

By seeing the result it looks like this is test environment. When was the last time you had the successfull backup in this environment ?

All the drive and tapes have different density and there is no way you can run even a single backup right now. Multiple density drive needed whey you have old tapes which needs to be restored. If that is the case make one of the drive density same as old tape density but not all of them.What is your criteria to have different density for every tape drive?

[Marianne · Accepted solution]

Have you been able to do ANY backups in this environment?

Seems you have been battling since August:

 EMM status: No media is available

 Managing barcodes 

Have another look at my last post on 19 Aug in your other forum discussion :

hcart drive type needs hcart (NOT DLT) media and hcart STU.
hcart2 drive type needs hcart2 (NOT DLT) media and hcart2 STU. 

qscsi - 1/4 Inch drive type needs qscsi - 1/4 Inch  (NOT DLT) media and qscsi - 1/4 Inch  STU.

So, you have DLT media and qscsi - 1/4 Inch Storage Unit.
The densities do not match.
That is why you are getting status 96 error.

PLEASE get barcode labels for the media and ensure that densities match:
drive type, media type, STU type.

You also seem to have added LTO4 labels on the tapes (L4 suffix). 
What type of media is actually in the robot?

LTO6 drive can read LTO4, LTO5 and LTO6 media, but can only write LTO5 and LTO6 media.

So, not a good idea to have LTO4 labels on the tapes. 

PLEASE go through advice in this and your previous 2 posts again and start over....

If you have LTO6 drives and LTO6 media, your drives, media and Storage Unit should all be hcart3.
 

[Nam_Sym]

Hi Marianne and Mark,

I'm grateful for all your help and suggestions, my backup finally worked.

Thanks