SQL backup of BES

[RCBrown_DKH]

I have a SQL backup of Blackberry Enterprise Server that is giving me a status code 12.  The version of SQL is Server 2005 and I am using NetBackup 7.1.  I can backup the individual pieces:  BESMgmt, BMSStore, master, mdsis, model and msdb.  When doing a full backup of this instance i receive a status code 12 error:

An open of a file failed.
A disk storage unit tries to write to or create a directory on the root device of the NetBackup server or media server. In this case, the Activity Monitor job details log contains the message "not permitted to root device." By default the absolute path or specified directory for a disk storage unit cannot be on the root file system (or system disk). You must explicitly enable them to be there when the storage unit is created.

I have verified that the rights to the logs directory (C:\Program Files\VERITAS\NetBackup\logs\user_ops\mssql\logs) are correct.  Not sure where else to look.  Any help would be great.  Thanks.

[Marianne]

The problem seems to be with the Storage Unit selected in the policy.

Double-check storage unit properties, then check on media server that the STU path is on a separate filesystem/mount point.

[RCBrown_DKH]

Please forgive me, I am new to NetBackup as I started a new job and took over someone else's mess.  How do I check the STU and Policy?

[RCBrown_DKH]

Here is what the job details box says:

5/4/2012 12:02:12 AM - Error bprd(pid=29412) Unable to write progress log </C/ProgramÀ Files/Veritas/NetBackup/Logs/user_ops/dbext/logs/12048.0.1336104130> on client SERVER123. Policy=Servers.MSSQL Sched=NONE 
5/4/2012 12:02:12 AM - Error bprd(pid=29412) CLIENT SERVER123  POLICY Servers.MSSQL  SCHED NONE  EXIT STATUS 12 (file open failed)

[Marianne]

Okay - so it seems we can ignore the reference to disk storage unit in your opening post.
The error looks like a comms problem between master and client. Backup is written to a media server, but bprd on the master needs to update the progress log on the client.

We need to see what is different between the individual db job and the job for the full instance.

Is this the parent job failing? Are any child jobs starting?
If you don't mind, please share everything in Job Details.

Which OS on master server? If I look at how the path is presented in Job details, this is probably a Unix/Linux master?

Do you have different policies for the individual pieces and the full instances? If so, please post policy config for both. On master, run:
bppllist <policy-name> -U
(path is /usr/openv/netbackup/bin/admincmd on Unix/Linux master)

Please also share backup script(s) used on the client for individual piece and full instance.
Ensure dbclient log folder exist on the client - we might need it depending on what we see in above output and script(s).

Also ensure that bprd log folder exists on the master. If not, create the folder and restart NBU.
Please post bprd log after a failure - rename log to bprd.txt and post as attachment.

[RCBrown_DKH]

Let me look into the questions you asked me and i will get back to you.  thank you.

[RCBrown_DKH]

The OS on the Master server is Windows Server 2008 R2.

The job details are as follows:

5/10/2012 11:47:02 PM - Error bprd(pid=30148) Unable to write progress log </C/ProgramÀ Files/Veritas/NetBackup/Logs/user_ops/dbext/logs/12188.0.1336708019> on client EARTH4. Policy=Servers.MSSQL Sched=NONE 
5/10/2012 11:47:02 PM - Error bprd(pid=30148) CLIENT EARTH4  POLICY Servers.MSSQL  SCHED NONE  EXIT STATUS 12 (file open failed)

I do not have different policies for the individual pieces.  Only one policy for the full instance.

The backup script is as follows:

OPERATION BACKUP
DATABASE $ALL
SQLHOST "EARTH4"
SQLINSTANCE "BLACKBERRY"
NBSERVER "puck1.daykimball.org"
BROWSECLIENT "EARTH4"
MAXTRANSFERSIZE 6
BLOCKSIZE 7
NUMBUFS 2
ENDOPER TRUE

DBCLIENT folder does exist on client.

BPRD folder does exist on master.

I hope this helps.  Let me know if I forgot to give you something.  Thank you.

[Marianne]

I can see a failure for EARTH4 shortly after midnight, but not the start of the job - probably started before midnight that will be logged in previous log?

I would like to see logs (bprd and dbclient) for the following:

1. A successful backup of individual piece. Please tell us how the backup is intiated since you are saying that there is no script/policy for individual piece.
2. Backup attempt for $all that fails. Please tell how this

It is fine to let successful backup run, then try the one that fails.

Post bprd and dbclient that should contain info for both successful and failed backup.
Tell us what time each one was started. (Just makes reading logs easier).

I have found references to both EARTH4 and earth4 in bprd log.
How is client name specified in policy? How does master resolve client's IP address to hostname?

[RCBrown_DKH]

I kicked off a backup of the individual pieces manually from on the client from the Backup Microsoft SQL Server Object application.  I have a dbclient folder, but i don't seem to have a bprd folder in the logs directory on the client. 

I started the individual backups at 9:36am on 5/14 and all completed sucessfully.  At 9:41am I started a whole backup which failed. 

There is a SQL policy for the complete backup of Earth4, which includes all the individual pieces.  In the policy the serer is referenced as earth4.  Does the capitalization make big difference?  We try to make sure all references are to the fully qualified domain name.

I also noticed this when looking at the logs. 

5/14/2012 9:42:22 AM - Error bprd(pid=26524) Unable to write progress log </C/ProgramÀ Files/Veritas/NetBackup/Logs/user_ops/dbext/logs/10328.0.1337002940> on client earth4. Policy=Servers.MSSQL Sched=NONE 
5/14/2012 9:42:22 AM - Error bprd(pid=26524) CLIENT earth4  POLICY Servers.MSSQL  SCHED NONE  EXIT STATUS 12 (file open failed)

What is the extra character between 'Program' and 'Files'????

Attached is the dbclient log.

[Marianne]

Can we get bprd log on the master for the same date as the dbclient posted above (14 May)? bprd (NBU Request Service) only runs on the master and will therefore only log on the master.

The difference in backup behaviour is how job is initiated. When you select individual 'pieces' in the NBU for SQL GUI, the job is initiated on the client as the user that you have logged on as. If you select the .bch script in the GUI, the backup should be successful as well.

If backup is initiated from the Master, the job will run as the user starting the NetBackup Client Service.

As a test, please change the NBU Client Service to the same logon account as when you manually run the backups and try to kick off the job from the master again. If this works, please change the NBU Client Service to the same logon account used for SQL services. This logon account should also have local admin rights in order to write logs.

At this point in time, my 'gut feel' is permissions. Host name usage might also cause an issue, but let us get user/logon permissions right first.

I don't believe the 'funny character' in the path is causing the issue. It has to do with the way in which NBU interprets paths in a 'Unix' format. I have often seen that without causing issues.

Please also check if the log file mentioned below (C:\Program Files\Veritas\NetBackup\Logs\user_ops\dbext\logs\10328.0.1337002940) was actually created and written to:

09:42:20.307 [1772.10328] <4> serverResponse: initial client_read_timeout = <900>
09:42:20.307 [1772.10328] <4> readCommMessages: Entering readCommMessages
09:57:21.466 [1772.10328] <16> readCommFile: ERR - timed out after 900 seconds while reading from C:\Program Files\Veritas\NetBackup\Logs\user_ops\dbext\logs\10328.0.1337002940
09:57:21.466 [1772.10328] <32> serverResponse: ERR - could not read from comm file <C:\Program Files\Veritas\NetBackup\Logs\user_ops\dbext\logs\10328.0.1337002940>
09:57:21.466 [1772.10328] <16> CreateNewImage: ERR - serverResponse() failed

Example of similar issue that was resolved by changing logon account:

http://www.symantec.com/connect/forums/netbackup-655-status-code-12#comment-3552381

[RCBrown_DKH]

I will test the rights like you suggested.  I have attached the bprd file and confirmed that the 10328.0.1337002940 log was created.

[Marianne · Accepted solution]

Don't worry about permissions.

Master servers seems to resolve Client hostname and IP incorrectly.

With the manual backup started on Client, it receives connection from IP 170.163.44.174.
This IP address is resolved to hostname olsrv2.daykimball.org.
Client is also sending NBU client name of 'earth4'.

09:36:44.082 [29376.30640] <2> logconnections: BPRD ACCEPT FROM 170.163.44.174.4816 TO 10.10.1.100.1556 fd = 800

09:36:44.082 [29376.30640] <2> connected_peer: Connection from host olsrv2.daykimball.org, 170.163.44.174, on non-reserved port 4816

09:39:38.055 [31848.31356] <2> process_request: clnt_bp_conf_name = earth4
09:39:38.055 [31848.31356] <2> process_request: target_clientname = earth4
09:39:38.055 [31848.31356] <2> process_request: target_hostname = olsrv2.daykimball.org

This is why client initiated backup works.
Why would this be?

Are you using hosts files for hostname lookup or DNS?

Backup from master started over here:

09:41:45.055 [28900.32052] <2> process_request: Before bkarfiles
09:41:45.055 [28900.32052] <2> bkarfiles:    backup_or_archive = 1
09:41:45.055 [28900.32052] <2> bkarfiles:    immediate_or_userdirected = 0
09:41:45.055 [28900.32052] <2> bkarfiles:    client = earth4
09:41:45.055 [28900.32052] <2> bkarfiles:    client_hostname = olsrv2.daykimball.org
09:41:45.055 [28900.32052] <2> bkarfiles:    requesting_user = nbuadmin
09:41:45.055 [28900.32052] <2> bkarfiles:    requesting_group = nbuadmin
09:41:45.055 [28900.32052] <2> bkarfiles:    progress_file = /C/ProgramÀ Files/Veritas/NetBackup/Logs/user_ops/mssql/logs/0514112094135-1772-3020-000-000-prg
09:41:45.055 [28900.32052] <2> bkarfiles:    policy = Servers.MSSQL
....
09:41:45.055 [28900.32052] <2> ConnectToBPCD: db_getCLIENT(olsrv2.daykimball.org) failed: 227

Since backup is initiated on Master server, it needs to connect to earth4 to update log. Master server cannot connect to earth4.

09:42:22.152 [26524.32744] <2> set_job_details: Tfile (196574): LOG 1337002942 16 bprd 26524 Unable to write progress log </C/ProgramÀ Files/Veritas/NetBackup/Logs/user_ops/dbext/logs/10328.0.1337002940> on client earth4. Policy=Servers.MSSQL Sched=NONE

09:42:22.152 [26524.32744] <2> job_monitoring_exex: ACK disconnect
09:42:22.152 [26524.32744] <2> job_disconnect: Disconnected
09:42:22.152 [26524.32744] <16> bkarfiles: Unable to write progress log </C/ProgramÀ Files/Veritas/NetBackup/Logs/user_ops/dbext/logs/10328.0.1337002940> on client earth4. Policy=Servers.MSSQL Sched=NONE

Please double-check forward and reverse lookup of client. Check DNS as well as hosts files.

Once you have found and corrected the problem, please clear host cache on master server before attempting another backup:

bpclntcmd -clear_host_cache

[RCBrown_DKH]

We use DNS.  Let me look into this and i will get back to you. 

[Marianne]

Please check as follows on the client:
Run the following from cmd (in C:\Program Files\Veritas\NetBackup\bin) :
bpclntcmd -self

On master:
nslookup earth4
nslookup olsrv2.daykimball.org
nslookup 170.163.44.174

[RCBrown_DKH]

On Client:

C:\Program Files\VERITAS\NetBackup\bin>bpclntcmd -self
gethostname() returned: earth4
host earth4: EARTH4.daykimball.org at 170.163.44.174
aliases:     EARTH4.daykimball.org     earth4     170.163.44.174

___________________________________________

On Master:

C:\Users\nbuadmin>nslookup earth4
Server:  phobos.daykimball.org
Address:  170.163.44.8

Name:    earth4.daykimball.org
Address:  170.163.44.174

_____________________________________________

C:\Users\nbuadmin>nslookup olsrv2.daykimball.org
Server:  phobos.daykimball.org
Address:  170.163.44.8

Name:    olsrv2.daykimball.org
Address:  10.12.4.47

______________________________________________

C:\Users\nbuadmin>nslookup 170.163.44.174
Server:  phobos.daykimball.org
Address:  170.163.44.8

Name:    olsrv2.daykimball.org
Address:  170.163.44.174

[Marianne]

Do you see the problem? Reverse lookup of earth4's IP address resolves olsrv2.daykimball.org instead of earth4.

C:\Users\nbuadmin>nslookup earth4
Name:    earth4.daykimball.org
Address:  170.163.44.174

C:\Users\nbuadmin>nslookup 170.163.44.174
Name:    olsrv2.daykimball.org
Address:  170.163.44.174

[RCBrown_DKH]

Yes i do see it.  I'm working on finding that issue in DNS right now.  Get back to you shorly.

[RCBrown_DKH]

All fixed!  I tested the backup and it worked!!!  I deleted the wrong IP address from DNS.  Looks like that address was added in manually, not by me.  Thank you so much for your help.

[Marianne]

Awesome!!

Glad we could eventually find the problem!

I originally gave that large bprd log one look and my heart sank... It took a couple of hours to find the courage!

Resolving a 2-week old headache like this one is what keeps me going!