cancel
Showing results forΒ 
Search instead forΒ 
Did you mean:Β 

Remote SQL backup processes kicking off, but hanging, or "running" and not reporting in NB Console

agardner
Level 3

Hello Symantec Forums!

I am a datacenter operator for a major manufacturing customer of Symantec, and am running into an issue.

I am attemping to automate the kicking-off of missed SQL backups via Powershell. My basic question is this:

Why do some backup processes kicked off remotely with "dbbackex.exe" simply sit there and apparently do nothing? And why do others apparently run (see below) but don't register a backup in NB console?

Our SQL DBA's created a stored procedure that reports on backups that do not complete successfully in a 24 hour period. That report is used by the script to run "dbbackex.exe" against each server and back up everything that was missed. It uses this call with appropriate credentials:

$global:dbbackp -f `"d:\netbackup_adhoc.bch`"

(where $global:dbbackp represents the registry-extracted installation path for Netbackup on each box)

It does this in two ways:

  1. If a standard netbackup.bch file exists in a defined location, it makes a copy of certain properties of that file and inserts them sequentially into another .bch file, one after another, like so:
OPERATION BACKUP
DATABASE "FAST_Query_Service_Application_DB_e451b6"
SQLHOST "XXX1"
SQLINSTANCE "Q2A"
NBSERVER "YYY"
STRIPES 2
MAXTRANSFERSIZE 6
BLOCKSIZE 7
POLICY sql-np
NUMBUFS 2
ENDOPER TRUE
 
 
OPERATION BACKUP
DATABASE "FAST_Search_Service_Application_DB_3b0d8"
SQLHOST "XXX1"
SQLINSTANCE "Q2A"
NBSERVER "YYY"
STRIPES 2
MAXTRANSFERSIZE 6
BLOCKSIZE 7
POLICY sql-np
NUMBUFS 2
ENDOPER TRUE

      2. If the standard .bch file does not exist in the standard location, I have it write an arbitrary .bch file, omitting the policy and a couple others. Vanilla. Like so:

OPERATION BACKUP
DATABASE $ALL
SQLHOST "XXX1"
SQLINSTANCE "Q2A"
NBSERVER "YYY1"
ENDOPER TRUE
 

Here is the end output running the script:

Process for backing up XXX1 was successfully started. (Successful Completion) (09/27/2013 08:46:08)
Process for backing up XXX2 was successfully started. (Successful Completion) (09/27/2013 08:46:08)
Process for backing up XXX3 was successfully started. (Successful Completion) (09/27/2013 08:46:08)
Process for backing up XXX4 was successfully started. (Successful Completion) (09/27/2013 08:46:08)
Process for backing up XXX5 was successfully started. (Successful Completion) (09/27/2013 08:46:08)
 
Of these 5 that register the process opened successfully, only two actually ran backups as registered in NB Console, one that had a standard BCH file, one that had the arbitrary. The other three are a mix. The question is why not all of them?
 
 
Here are screenshots of the process on a server that did NOT run the backup in console that had the arbitrary file my script created:
 
 
ProcessDetailFirstScreenArbBCH_0.PNG
ThreadsArbBCH.PNG
 
 
Here is one that DID have the arbitrary file that did not run in the console, but appeared to run on the server:
 
 
 
ProcessExplorerFileExists.PNG
 
ThreadsFileExists.PNG
 
 
I have attached quite a few pictures that also provide information. I hope the naming convention is clear enough.
 
Please let me know what other data you may need. Thanks in advance!
 
 
8 REPLIES 8

Mark_Solutions
Level 6
Partner Accredited Certified

One possible cause taken from the Admin Guide:

Note: If you use dbbackex through a client-based scheduler, specify the -np option
to ensure that a message box is not generated. However, before you use the
scheduler with dbbackex consider the following. Try the dbbackex syntax on the
console WITHOUT the -np option. Do this to test for the successful completion
of the batch file that you have created for your operation.

So maybe some need to -np option as there may be a pop-up somewhere that you cannot see in your session?

agardner
Level 3

I figured that might be the case as well and have tried both. The prompt basically notifies of success, and the issue is not resolved. In most cases I can see the process kick off in Process Explorer per the screenshots attached to the post, but it just sits there.

Just opened a technical ticket directly with support, too, just in case.

Mark_Solutions
Level 6
Partner Accredited Certified

I do see a status 23 in the activity monitor - that is a network issue of some sort - could it be that the SQL servers have multiple interfaces?

Have you tried adding the -s YYY to the command to fix the name of the Master Server?

 

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

It seems some scripts have NBSERVER "YYY1"  and others have NBSERVER "YYY" .

This entry must reflect the master server name.

Enable logs for troubleshooting - on client: dbclient
on master: bprd (NBU needs to be restarted to enable logging).

dbclient log will tell us if comms with master server was initiated, to which IP address and on which port (normally 1556).

bprd log on master server will tell us if connection attempt from client was received and if master was able to resolve IP address to client name that exists in a policy and so on...

agardner
Level 3

Just so everyone is aware, I had Symantec get back to me and sit on a WebEx to troubleshoot. Very helpful. Turns out the FQDN was identical between two of the three servers that were not running, and the third didn't have a short DNS name. Not sure if that would cause it - not having a short DNS name - but I'm on a roll. Thanks for everybody's comments so far.

agardner
Level 3

Hi Marianne, this was indeed one of the troubleshooting steps the tech took with me and it made it pretty clear. See the response below.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Reverse name lookup is key.

The connection request from the client IP address needs to be resolved by the master.

So, on master, 'bpclntcmp -ip <client-ip>' needs to resolve to the same hostname that is in the policy Client tab.  If client IP is resolved by master as FQDN, the name in the policy must match to the same FQDN name, not shortname. Or vice versa.

Mark_Solutions
Level 6
Partner Accredited Certified

Yes - it can depend on how the Master was specified when the client was installed but also how it was specified in the script

So it is well worth having all clients and servers be able to resolve short and FQDN to on the safe side - but if nothing else keep them consistent - so if HKLM\Software\Veritas\netbackup\currentversion\config\ server key is short make sure the script is short and also that the short name can be resolved

Use hosts files if you need to