Forum Discussion

Steve_Law's avatar
Steve_Law
Level 4
9 years ago
Solved

SQL cluster backup error: no storage units available for use(213)

Hi,

I'm having trouble with backups on a new SQL cluster. Here are the details of my setup: 

Two physical nodes configured as media servers: swt003 and swt004
These are just straight installs of NBU 7.5.0.6, client has browser and client name set to the server name
We use Data Domain via the DDBoost plugin, registered as the server name e.g: nbdevconfig -creatests -stype DataDomain -storage_server basdd002.dcsit.net -media_server swt003

The clustering itself:
Virtual server name: SWD005
SQL cluster name:  DISQLPROD
The actual SQL instance name is also DISQLPROD, so for users to connect via SQL from a remote machine they would use DISQLPROD\DISQLPROD.

Both media servers and the master have hosts entries for the physical nodes, the virtual cluster SWD005 and the sql cluster DISQLPROD. I've double-checked and they are all correct and consistent. 

When I login to the virtual cluster swd005 and run the Netbackup SQL Client (I am logged on with a domain account that is a sysadmin of the bisqlprod database and the sql account I use "netbackup" is a sysadmin), I cannot connect to the SWD005 default instance. I can only connect to the DISQLPRO\DISQLPROD instance. So I then use that to create a "Backup All" script. 

Given that i can only access the cluster as DISQLPROD I created a "DISQLPROD" app_cluster object and added the hostnames of the two physical nodes as members. 

I then created a new storage unit called NLDD002_DFS_DISQLPROD in our usual way for Data Domain and added bisqlprod as the media server.

The SQL backup job has DISQLPROD as the client, runs the script from c:\program files\veritas etc and writes directly to the storage unit created above with DISQLPROD (the app_cluster object) as the media server. 

But the backup fails immediately:

25/05/2016 17:51:40 - Info nbjm(pid=4620) starting backup job (jobid=3419675) for client bisqlprod, policy DFS_DBSQL_DISQLPROD_TEST, schedule Daily_Full  
25/05/2016 17:51:40 - Info nbjm(pid=4620) requesting MEDIA_SERVER_WITH_ATTRIBUTES resources from RB for backup job (jobid=3419675, request id:{BCEC00FA-DE57-4140-83D3-A19EB73C52DE})  
25/05/2016 17:51:40 - requesting resource NLDD002_DFS_DISQLPROD
25/05/2016 17:51:40 - requesting resource swm009.corpadds.net.NBU_CLIENT.MAXJOBS.bisqlprod
25/05/2016 17:51:40 - requesting resource swm009.corpadds.net.NBU_POLICY.MAXJOBS.DFS_DBSQL_DISQLPROD_TEST
25/05/2016 17:51:41 - Error nbjm(pid=4620) NBU status: 213, EMM status: Storage units are not available   
no storage units available for use(213)

 

What's going on here, any suggestions? I thought from the documentation that I was supposed to be able to address the database using the virtual cluster name swd005 and the default instance. I am using the SQL instance name instead, but it works ok - you can RDP to the virtual cluster server using DISQLPROD as the server name, just as you can map a unc path as e.g. \\disqlprod\c$. 

As the app_cluster name is set to "disqlprod" and that's what's associated with the storage unit, I assume that something is missing that chains that app_cluster to the actual physical nodes. But the correct hosts are members of the app-cluster group:

 

E:\Program Files\Veritas\NetBackup\bin\admincmd>nbemmcmd -listhosts -list_servers_in_app_cluster -clustername disqlprod
NBEMMCMD, Version: 7.5.0.6
The following hosts were found:
media            swt003
media            swt004
Command completed successfully.

 

Any help gratefully received :-)

 

7.5
Backup and Recovery
NetBackup
Windows Server (2003-2008)
  • Steve_Law's avatar
    Steve_Law
    9 years ago

    Ooh that worked! 
    On the storage unit I didn't select "use any available media server", I just selected both physical nodes. The backup succeeded and used swt004, which is currently the active cluster node:

    26/05/2016 12:11:28 - Info nbjm(pid=4620) starting backup job (jobid=3426184) for client swd005, policy DFS_FS_SWD005_TEST, schedule Daily_Full  
    26/05/2016 12:11:28 - estimated 0 Kbytes needed
    26/05/2016 12:11:28 - Info nbjm(pid=4620) started backup (backupid=swb005_1464261088) job for client swd005, policy DFS_FS_SWD005_TEST, schedule Daily_Full on storage unit NLDD002_DFS_SWD005
    26/05/2016 12:11:30 - started process bpbrm (12508)
    26/05/2016 12:11:31 - Info bpbrm(pid=12508) swd005 is the host to backup data from     
    26/05/2016 12:11:31 - Info bpbrm(pid=12508) reading file list from client        
    26/05/2016 12:11:31 - Info bpbrm(pid=12508) starting bpbkar32 on client         
    26/05/2016 12:11:31 - connecting
    26/05/2016 12:11:31 - connected; connect time: 00:00:00
    26/05/2016 12:11:34 - Info bpbkar32(pid=2312) Backup started           
    26/05/2016 12:11:34 - Info bptm(pid=7300) start            
    26/05/2016 12:11:43 - Info bptm(pid=7300) using 1048576 data buffer size        
    26/05/2016 12:11:43 - Info bptm(pid=7300) setting receive network buffer to 4195328 bytes      
    26/05/2016 12:11:43 - Info bptm(pid=7300) using 512 data buffers         
    26/05/2016 12:11:45 - Info bpbkar32(pid=2312) change journal NOT enabled for <E:\DaimlerBI Badfiles>      
    26/05/2016 12:11:46 - Info bptm(pid=7300) start backup           
    26/05/2016 12:11:47 - begin writing
    26/05/2016 12:12:09 - Info bptm(pid=7300) waited for full buffer 232 times, delayed 1383 times    
    26/05/2016 12:12:09 - Info bpbkar32(pid=2312) bpbkar waited 0 times for empty buffer, delayed 0 times.   
    26/05/2016 12:12:10 - Info bptm(pid=7300) EXITING with status 0 <----------        
    26/05/2016 12:12:10 - Info bpbrm(pid=12508) validating image for client swd005        
    26/05/2016 12:12:11 - end writing; write time: 00:00:24
    26/05/2016 12:12:12 - Info bpbkar32(pid=2312) done. status: 0: the requested operation was successfully completed    
    the requested operation was successfully completed(0)  

    I guess I need to fail the cluster over and confirm that when swt003 is the active node it is also automatically selected to perform the backup. 

     

9 Replies

Sort By
Replies have been turned off for this discussion
  • Michal_Mikulik1's avatar
    Michal_Mikulik1
    Moderator
    9 years ago

    Hello,

    here are several tips:

    - if OS used on cluster nodes is Windows 2012 or higher, I recommend to use higher NBU version than 7.5.0.6. Just mere client was supported with this NBU on this OS. Read compatibility matrixes carefully here, also for SQL version support

    - to simplify troubleshooting, first try filesystem backups of a file on shared disk resource, then proceed to SQL backup

    - hovewer I am not sure that virtual storage units can be used with Open Storage, PureDisk etc. types of Storage Servers. Maybe yes, but in Storage Server properties, you must allow also access of virtual media server to the Storage Server. Not personally experienced with that.

    Michal

  • Michal_Mikulik1's avatar
    Michal_Mikulik1
    Moderator
    9 years ago

    Hello,

    not sure you have read the posts carefully. There was no word about "7.5.0.6 with DataDomain" support/unsupport. It was about support related to OS only.

    Actually there won't be any matrix confirming "app_cluster with DataDomain". There will be only support statement for app_cluster feature and OST Storage Servers based storage units. But it is not explicitly in HA Guide, so yes you should address the support about this.

    However when it has worked previously, than the problem is maybe related to name resolution etc. I recommend to try file based backup from virtual resources first.

    Regards

    Michal

  • Marianne's avatar
    Marianne
    Level 6
    9 years ago
    I agree - you need to be on 7.6 or higher to use W2012 as media server. It seems that NBU does not support app_cluster with Advanced disk or OST storage. I have tried in the past to encourage a fellow Connect member to get his Support call escalated to backline or engineering, but we never heard anything back. A good friend at Veritas Support asked around internally at the time and was told that the problem exists with adding a Storage Server for the virtual hostname. It seems only physical hostname can be used as Storage server. All I can suggest is to firstly upgrade to a supported NBU version (you may want to go straight to 7.7.x as all previous versions will reach EOSL by Feb 2017) and then log a call with Support about Storage Server for app_cluster.
  • Steve_Law's avatar
    Steve_Law
    Level 4
    9 years ago

    Hi, I'm surprised to hear you say that this is not supported on 7.5.0.6 with Data Domain. The servers in question are Win2008 R2, when this environment was in development a few weeks ago I tested cluster backups and it worked like a dream - but since then the servers and cluster objects have been renamed (to replace pre-existing Win2003 servers with the same name). On that occasion everything was textbook: I was able to backup SQL using the virtual server name and default SQL instance, rather than the SQL cluster name and DISQLPROD instance. It continued to work when I failed over etc. Are you sure that 7.5.0.6 doesn't support app_cluster on Win2008R2 with Data Domain? Should I check with Veritas support? 

      

  • Steve_Law's avatar
    Steve_Law
    Level 4
    9 years ago

    Hi Michal,

    I get the same result with a simple file-based backup. I've tried with both the virtual server name swd005 and the sql cluster name disqlprod - exactly the same result. Physical node backups of swt003 and swt004 work fine, but then they're using their own storage units, not those associated with the 'virtual' media server(s).

    The posts above said that Win2012 as a media server isn't supported on 7.5.0.6. But this cluster is Win2008R2. Marianne wrote that it "seems that NBU does not support app_cluster with Advanced disk or OST storage". Data Domain is OST storage. 

    I've run bpclntcmd -hn and bpclntcmd -ip from the master and the physical cluster nodes against all the elements involved - swt003, swt004, disqlprod and swd005. They all check out. Both virtual server names disqlprod and swd005 have been added to the master's Servers list. 

     

     

     



     

  • Steve_Law's avatar
    Steve_Law
    Level 4
    9 years ago

    There's an article here - https://www.veritas.com/support/en_US/article.000013231  "Cannot create a storage unit that uses OpenStorage technology (OST) with a Virtual Media Server as app_cluster."

     

    Problem
    When attempting to configure a storage unit for that will be used for a Virtual application cluster the Virtual host name may not be present as a valid media server. 

    Cause
    In this configuration the physical nodes in the cluster will have additional software from the OST vendor that is called the plug-in.  This software allows the communications between the physical host and the OST device to communicate and allow the backups, restores and duplications to be performed to that storage server.  The issue that causes this problem is the  plug-in application along with the OST credentials are configured on the physical node not the Virtual host.  Then within NetBackup only the physical node names are present when attempting to create the storage unit for the Virtual host.

    Workaround:
    When creating the storage unit please make sure that the physical nodes in the cluster are listed in the storage unit.   For the configuration of the storage unit make sure the option to use any available media server is then selected.  In this configuration the behavior will work the same as if the storage unit was of a basic disk storage unit with the virtual host name.  In the policy that has been setup to backup the virtual host please verify that the client specified is the virtual host name and the storage unit is the new storage unit that was just created.  When a backup runs NetBackup will use the node that is actively running the application for the backup.  If the virtual application then fails over to the other node then the next time the policy runs it will use that node to perform the backup.

     

    This seems to be saying that the storage unit to be used for the virtual cluster name or sql cluster name should have both physical node media servers enabled along with the option "use any available media server" The app_cluster objects I've created are available to be added to the storage unit, but I will try it with just the swt003 and swt004 added. 

     

     

  • Steve_Law's avatar
    Steve_Law
    Level 4
    9 years ago

    Ooh that worked! 
    On the storage unit I didn't select "use any available media server", I just selected both physical nodes. The backup succeeded and used swt004, which is currently the active cluster node:

    26/05/2016 12:11:28 - Info nbjm(pid=4620) starting backup job (jobid=3426184) for client swd005, policy DFS_FS_SWD005_TEST, schedule Daily_Full  
    26/05/2016 12:11:28 - estimated 0 Kbytes needed
    26/05/2016 12:11:28 - Info nbjm(pid=4620) started backup (backupid=swb005_1464261088) job for client swd005, policy DFS_FS_SWD005_TEST, schedule Daily_Full on storage unit NLDD002_DFS_SWD005
    26/05/2016 12:11:30 - started process bpbrm (12508)
    26/05/2016 12:11:31 - Info bpbrm(pid=12508) swd005 is the host to backup data from     
    26/05/2016 12:11:31 - Info bpbrm(pid=12508) reading file list from client        
    26/05/2016 12:11:31 - Info bpbrm(pid=12508) starting bpbkar32 on client         
    26/05/2016 12:11:31 - connecting
    26/05/2016 12:11:31 - connected; connect time: 00:00:00
    26/05/2016 12:11:34 - Info bpbkar32(pid=2312) Backup started           
    26/05/2016 12:11:34 - Info bptm(pid=7300) start            
    26/05/2016 12:11:43 - Info bptm(pid=7300) using 1048576 data buffer size        
    26/05/2016 12:11:43 - Info bptm(pid=7300) setting receive network buffer to 4195328 bytes      
    26/05/2016 12:11:43 - Info bptm(pid=7300) using 512 data buffers         
    26/05/2016 12:11:45 - Info bpbkar32(pid=2312) change journal NOT enabled for <E:\DaimlerBI Badfiles>      
    26/05/2016 12:11:46 - Info bptm(pid=7300) start backup           
    26/05/2016 12:11:47 - begin writing
    26/05/2016 12:12:09 - Info bptm(pid=7300) waited for full buffer 232 times, delayed 1383 times    
    26/05/2016 12:12:09 - Info bpbkar32(pid=2312) bpbkar waited 0 times for empty buffer, delayed 0 times.   
    26/05/2016 12:12:10 - Info bptm(pid=7300) EXITING with status 0 <----------        
    26/05/2016 12:12:10 - Info bpbrm(pid=12508) validating image for client swd005        
    26/05/2016 12:12:11 - end writing; write time: 00:00:24
    26/05/2016 12:12:12 - Info bpbkar32(pid=2312) done. status: 0: the requested operation was successfully completed    
    the requested operation was successfully completed(0)  

    I guess I need to fail the cluster over and confirm that when swt003 is the active node it is also automatically selected to perform the backup.