SQL Database Backup Fails...!

Question

Hi,

MS SQL database transanction&nbsp;logs backup&nbsp;is failing with error code &nbsp;"2" and database backup is failing with different error&nbsp;codes &nbsp;25, 41,54 and 58.&nbsp;

Connection to client from the master and media server is sometimes fine and sometimes not.&nbsp; I increased client read time out from 300 sec to 2000 also. But it is still failing.... :-(

NB Client verison 6.0 MP4 and NBU Master version is 6.0 MP6

The following error log on user_ops\MsSql\Logs:

ERR - Error in VxBSACreateObject: 3.
&nbsp;&nbsp;&nbsp; CONTINUATION: - System detected error, operation aborted.
ERR - Error in VDS-&gt;Close: 0x80770004.
&nbsp;&nbsp;&nbsp; CONTINUATION: - An abort request is preventing anything except termination actions.
INF - OPERATION #1 of batch nb60_sql_logs.bch FAILED with STATUS 1 (0 is normal). Elapsed time = 2021(0) seconds.

The following error log on dbclient:
&nbsp;
02:22:31.408 [456.3084] &lt;4&gt; serverResponse: entering serverResponse.
02:22:31.408 [456.3084] &lt;4&gt; serverResponse: initial client_read_timeout = &lt;2000&gt;
02:22:31.408 [456.3084] &lt;4&gt; readCommMessages: Entering readCommMessages
02:22:41.408 [456.3084] &lt;4&gt; serverResponse: 02:22:35 Initiating backup
02:22:41.408 [456.3084] &lt;4&gt; readCommMessages: Entering readCommMessages
02:56:06.382 [456.3084] &lt;16&gt; readCommFile: ERR - timed out after 2000 seconds while reading from C:\Program Files\VERITAS\NetBackup\Logs\user_ops\mssql\logs\0903109022230-456-5704-000-000-prg
02:56:06.382 [456.3084] &lt;32&gt; serverResponse: ERR - could not read from comm file &lt;C:\Program Files\VERITAS\NetBackup\Logs\user_ops\mssql\logs\0903109022230-456-5704-000-000-prg&gt;
02:56:06.382 [456.3084] &lt;16&gt; CreateNewImage: ERR - serverResponse() failed
02:56:06.382 [456.3084] &lt;4&gt; closeApi: entering closeApi.
02:56:06.382 [456.3084] &lt;4&gt; closeApi: INF - EXIT STATUS 6: the backup failed to back up the requested files

07:33:31.591 [5596.624] &lt;16&gt; CDBbackrec::FreeDeviceSet(): ERR - Error in VDS-&gt;Close: 0x80770004.
07:33:31.591 [5596.624] &lt;1&gt; CDBbackrec::FreeDeviceSet():&nbsp;&nbsp;&nbsp;&nbsp; CONTINUATION: - An abort request is preventing anything except termination actions.
07:33:31.591 [5596.624] &lt;4&gt; Dbbackrec::Perform: INF - OPERATION #4 of batch nb60_sql_incr.bch FAILED with STATUS 1 (0 is normal). Elapsed time = 2642(0) seconds.
07:33:33.591 [5596.5012] &lt;4&gt; CDBbackmain::dbbackup: INF - Results of executing &lt;nb60_sql_incr.bch&gt;:&nbsp; &lt;0&gt; operations succeeded. &lt;4&gt; operations failed.

Can you please anyone&nbsp;provide me your valuable suggestion to resolve the issue...?

Shiv Sankar.G

omar_villa · Accepted Answer

run:

from client:
bpclntcmd -pn
bpclntcmd -self
bpclntcmd -hn &lt;master server&gt;
bpclntcmd -hn &lt;media server&gt;
bpclntcmd -ip &lt;master IP&gt;
bpclntcmd -ip &lt;media IP&gt;

from master:
traceroute &lt;client name&gt;
ping -s -i &lt;nic&gt; -p 13782 &lt;client name&gt; 65000 4
bpgetconfig -M &lt;client name&gt; -x
bptestbpcd -client &lt;client name&gt; -verbose

from media:
traceroute &lt;client name&gt;
ping -s -i &lt;nic&gt; -p 13782 &lt;client name&gt; 65000 4
bpgetconfig -M &lt;client name&gt; -x
bptestbpcd -client &lt;client name&gt; -verbose

Paste this data and we can see if is a network issue or not.
Regards.

frschind · Answer

Have you checked the host name resolution in both directions?&nbsp;Reverse lookup?

I've heard of issues like yours when using multiple channels - in these cases everything went fine when using only one channel to backup.

dragnet · Answer

You might want to make sure the SQL system account executing the bch file has access permissions the C:\Program Files\VERITAS\NetBackup\Logs\user_ops\mssql directory.

shiv_sankar · Answer

Hi Frerk,

Win2K3 SQL Database server is having only one ethernet card is connected.

Checked host name to ip resolution in the master, media servers and client server. Everything is working fone. But it is still failing.

Is there any parameters needs to be changed to resolve the issue..?

Pls suggest me...!

nbutse · Answer

Omar is on the right track with the networking troubleshooting, I suspect that your SQL&nbsp;server's IP&nbsp;addresses are not resolvable by the master server and media server.&nbsp; Ensure all the master and media server can resolve any configured IP&nbsp;addresses on the SQL&nbsp;server back to host names.

-Sean

shiv_sankar · Answer

Hi Omar,

Thanks for suggestion...!

Pls tell me that how to run the command "ping -s -i &lt;nic&gt; -p 13782 &lt;client name&gt; 65000 4" .

Pls find the remaing command o/p's from client, master and media server.

Pls suggest me on this...! Note: Local drive backup is completing successfully for the same server.

From Client Side o/p:

C:\Program Files\VERITAS\NetBackup\bin&gt;bpclntcmd -pn
expecting response from server mst2
pepwds00041.corp.com pepwds00041 156.81.238.206 2503
C:\Program Files\VERITAS\NetBackup\bin&gt;bpclntcmd -self
gethostname() returned: pepwds00041
host pepwds00041: pepwds00041.corp.com at 156.81.238.206 (0xceee519c)
checkhname: aliases:
C:\Program Files\VERITAS\NetBackup\bin&gt;bpclntcmd -hn mst2
host mst2: mst2 at 156.81.208.30 (0x1ed0519c)
checkhname: aliases:
C:\Program Files\VERITAS\NetBackup\bin&gt;bpclntcmd -hn media1
host media1: media1.corp.com at 156.81.204.222 (0xdecc519c)
checkhname: aliases:
C:\Program Files\VERITAS\NetBackup\bin&gt;bpclntcmd -hn media2
host media2: media2.corp.com at 156.81.204.223 (0xdfcc519c)
checkhname: aliases:
C:\Program Files\VERITAS\NetBackup\bin&gt;bpclntcmd -hn media4
host media4: media4.corp.com at 156.81.204.225 (0xe1cc519c)
checkhname: aliases:
C:\Program Files\VERITAS\NetBackup\bin&gt;bpclntcmd -ip 156.81.208.30
checkhaddr: host&nbsp;&nbsp; : mst2: mst2 at 156.81.208.30 (0x1ed0519c)
checkhaddr: aliases:&nbsp;&nbsp;&nbsp;&nbsp; plnbmst2.corp.com
C:\Program Files\VERITAS\NetBackup\bin&gt;bpclntcmd -ip 156.81.204.222
checkhaddr: host&nbsp;&nbsp; : media1: media1.corp.com at 156.81.204.222 (0xde
cc519c)
checkhaddr: aliases:
C:\Program Files\VERITAS\NetBackup\bin&gt;bpclntcmd -ip 156.81.204.223
checkhaddr: host&nbsp;&nbsp; : media2: media2.corp.com at 156.81.204.223 (0xdf
cc519c)
checkhaddr: aliases:
C:\Program Files\VERITAS\NetBackup\bin&gt;bpclntcmd -ip 156.81.204.225
checkhaddr: host&nbsp;&nbsp; : media4: media4.corp.com at 156.81.204.225 (0xe1
cc519c)
checkhaddr: aliases:

Master Server Output:

traceroute pepwds00041
traceroute: Warning: Multiple interfaces found; using 156.81.208.30 @ lan900:1
traceroute to pepwds00041 (156.81.238.206), 30 hops max, 40 byte packets
&nbsp;1&nbsp; 156.81.208.3 (156.81.208.3)&nbsp; 0.432 ms&nbsp; 0.235 ms&nbsp; 0.225 ms
&nbsp;2&nbsp; pepwds00041.corp.com (156.81.238.206)&nbsp; 0.164 ms&nbsp; 0.142 ms&nbsp; 0.145 ms
# cd /usr/openv/netbackup/bin/admincmd
# ./bpgetconfig -M pepwds00041 -x
SERVER = mst2
SERVER = mst2.corp.com
SERVER = media1
SERVER = media2
SERVER = media3
SERVER = media4
BPBACKUP_POLICY = any
BPBACKUP_SCHED = any
BPARCHIVE_POLICY = any
BPARCHIVE_SCHED = any
CLIENT_NAME = pepwds00041
VERBOSE = 5
PEM_VERBOSE = -2
JM_VERBOSE = -2
RB_VERBOSE = -2
RESTORE_RETRIES = 0
CLIENT_READ_TIMEOUT = 2000
LIST_FILES_TIMEOUT = 300
DISALLOW_SERVER_FILE_WRITES = NO
BUSY_FILE_DIRECTORY
BUSY_FILE_NOTIFY_USER
ALLOW_NON_RESERVED_PORTS = YES
CRYPT_OPTION = ALLOWED
CRYPT_STRENGTH = DES_40
Debug_Database = 0
KEEP_LOGS_DAYS = 3
RANDOM_PORTS = YES
CLIENT_RESERVED_PORT_WINDOW = 512 1023
CLIENT_PORT_WINDOW = 0 0
Time_Overlap = 60
Buffer_Size = 32
Use_Archive_Bit = YES
Perform_Default_Search = YES
Accumulate_Files = NO
User_Directed_Timeout = 60
Announce_DHCP_Interval = 0
Clear_Archive_Bit_Incr_Wait = 300
Exclude = *.LDF
Exclude = *.MDF
Exclude = C:\Marimba\*
Exclude = C:\Program Files\Symantec AntiVirus\*
Exclude = C:\Program Files\VERITAS\NetBackup\bin\*.lock
Exclude = C:\Program Files\VERITAS\NetBackup\bin\bprd.d\*.lock
Exclude = C:\Program Files\VERITAS\NetBackup\bin\bpsched.d\*.lock
Exclude = C:\Program Files\VERITAS\Volmgr\misc\*
Exclude = D:\SQLServer\MSSQL$SQLSHNMOM1P\Data
Exclude = C:\RECYCLER\
Browser = pepwds00041
Clients = pepwds00041.corp.com
Port_BPCD = 13782
Port_BPRD = 13720
Debug_Flush = 0
Debug_Heap = 0
Debug_Level = 2
Debug_TCP = 0
VSP_CacheVols
VSP_Cache_Size_Init = 40
VSP_Cache_Size_Max = 60
Error_Messages_Max = 10
Policy_Type = 13
Client_Version = 500
Case_Sensitive_Exclude_List = NO
Compress_Use = 0
Compress_Buffer_Size = 62
Compress_Minimum_Size = 512
Cluster_Database_Force_Restore = NO
ChangeJournalUse = NO
GENERATE_ENGLISH_LOGS = NO
EXCHANGE_ENABLE_SIS = NO
VERSIONINFO = "WindowsNET" "5" "nt" "NetBackup" "6.0" 600000
VSP_Auto_Size = YES
# ./bptestbpcd -client pepwds00041 -verbose
0 0 2
156.81.208.30:1001 -&gt; 156.81.238.206:13782
156.81.208.30:693 &lt;- 156.81.238.206:790
PEERNAME = mst2
HOSTNAME = pepwds00041
CLIENT_NAME = pepwds00041
VERSION = 0x06000404
PLATFORM = nt
156.81.208.30:987 &lt;- 156.81.238.206:514

Media Server1 Output:

# traceroute pepwds00041
traceroute to pepwds00041 (156.81.238.206), 30 hops max, 40 byte packets
&nbsp;1&nbsp; 156.81.204.12 (156.81.204.12)&nbsp; 0.377 ms&nbsp; 0.230 ms&nbsp; 0.243 ms
&nbsp;2&nbsp; pepwds00041 (156.81.238.206)&nbsp; 0.235 ms&nbsp; 0.191 ms&nbsp; 0.142 ms
# cd /usr/openv/netbackup/bin/admincmd
# ./bpgetconfig -M pepwds00041 -x
SERVER = mst2
SERVER = mst2.corp.com
SERVER = media1
SERVER = media2
SERVER = media3
SERVER = media4
BPBACKUP_POLICY = any
BPBACKUP_SCHED = any
BPARCHIVE_POLICY = any
BPARCHIVE_SCHED = any
CLIENT_NAME = pepwds00041
VERBOSE = 5
PEM_VERBOSE = -2
JM_VERBOSE = -2
RB_VERBOSE = -2
RESTORE_RETRIES = 0
CLIENT_READ_TIMEOUT = 2000
LIST_FILES_TIMEOUT = 300
DISALLOW_SERVER_FILE_WRITES = NO
BUSY_FILE_DIRECTORY
BUSY_FILE_NOTIFY_USER
ALLOW_NON_RESERVED_PORTS = YES
CRYPT_OPTION = ALLOWED
CRYPT_STRENGTH = DES_40
Debug_Database = 0
KEEP_LOGS_DAYS = 3
RANDOM_PORTS = YES
CLIENT_RESERVED_PORT_WINDOW = 512 1023
CLIENT_PORT_WINDOW = 0 0
Time_Overlap = 60
Buffer_Size = 32
Use_Archive_Bit = YES
Perform_Default_Search = YES
Accumulate_Files = NO
User_Directed_Timeout = 60
Announce_DHCP_Interval = 0
Clear_Archive_Bit_Incr_Wait = 300
Exclude = *.LDF
Exclude = *.MDF
Exclude = C:\Marimba\*
Exclude = C:\Program Files\Symantec AntiVirus\*
Exclude = C:\Program Files\VERITAS\NetBackup\bin\*.lock
Exclude = C:\Program Files\VERITAS\NetBackup\bin\bprd.d\*.lock
Exclude = C:\Program Files\VERITAS\NetBackup\bin\bpsched.d\*.lock
Exclude = C:\Program Files\VERITAS\Volmgr\misc\*
Exclude = D:\SQLServer\MSSQL$SQLSHNMOM1P\Data
Exclude = C:\RECYCLER\
Browser = pepwds00041
Clients = pepwds00041.corp.com
Port_BPCD = 13782
Port_BPRD = 13720
Debug_Flush = 0
Debug_Heap = 0
Debug_Level = 2
Debug_TCP = 0
VSP_CacheVols
VSP_Cache_Size_Init = 40
VSP_Cache_Size_Max = 60
Error_Messages_Max = 10
Policy_Type = 13
Client_Version = 500
Case_Sensitive_Exclude_List = NO
Compress_Use = 0
Compress_Buffer_Size = 62
Compress_Minimum_Size = 512
Cluster_Database_Force_Restore = NO
ChangeJournalUse = NO
GENERATE_ENGLISH_LOGS = NO
EXCHANGE_ENABLE_SIS = NO
VERSIONINFO = "WindowsNET" "5" "nt" "NetBackup" "6.0" 600000
VSP_Auto_Size = YES
# ./bptestbpcd -client pepwds00041 -verbose
0 0 2
156.81.204.222:856 -&gt; 156.81.238.206:13782
156.81.204.222:636 &lt;- 156.81.238.206:893
PEERNAME = media1.corp.com
HOSTNAME = pepwds00041
CLIENT_NAME = pepwds00041
VERSION = 0x06000404
PLATFORM = nt
156.81.204.222:823 &lt;- 156.81.238.206:873

Media Server2 Output:

# traceroute pepwds00041
traceroute to pepwds00041 (156.81.238.206), 30 hops max, 40 byte packets
&nbsp;1&nbsp; 156.81.204.12 (156.81.204.12)&nbsp; 0.415 ms&nbsp; 0.241 ms&nbsp; 0.239 ms
&nbsp;2&nbsp; pepwds00041 (156.81.238.206)&nbsp; 0.223 ms&nbsp; 0.179 ms&nbsp; 0.174 ms
# cd /usr/openv/netbackup/bin/admincmd
# ./bpgetconfig -M pepwds00041 -x
SERVER = mst2
SERVER = mst2.corp.com
SERVER = media1
SERVER = media2
SERVER = media3
SERVER = media4
BPBACKUP_POLICY = any
BPBACKUP_SCHED = any
BPARCHIVE_POLICY = any
BPARCHIVE_SCHED = any
CLIENT_NAME = pepwds00041
VERBOSE = 5
PEM_VERBOSE = -2
JM_VERBOSE = -2
RB_VERBOSE = -2
RESTORE_RETRIES = 0
CLIENT_READ_TIMEOUT = 2000
LIST_FILES_TIMEOUT = 300
DISALLOW_SERVER_FILE_WRITES = NO
BUSY_FILE_DIRECTORY
BUSY_FILE_NOTIFY_USER
ALLOW_NON_RESERVED_PORTS = YES
CRYPT_OPTION = ALLOWED
CRYPT_STRENGTH = DES_40
Debug_Database = 0
KEEP_LOGS_DAYS = 3
RANDOM_PORTS = YES
CLIENT_RESERVED_PORT_WINDOW = 512 1023
CLIENT_PORT_WINDOW = 0 0
Time_Overlap = 60
Buffer_Size = 32
Use_Archive_Bit = YES
Perform_Default_Search = YES
Accumulate_Files = NO
User_Directed_Timeout = 60
Announce_DHCP_Interval = 0
Clear_Archive_Bit_Incr_Wait = 300
Exclude = *.LDF
Exclude = *.MDF
Exclude = C:\Marimba\*
Exclude = C:\Program Files\Symantec AntiVirus\*
Exclude = C:\Program Files\VERITAS\NetBackup\bin\*.lock
Exclude = C:\Program Files\VERITAS\NetBackup\bin\bprd.d\*.lock
Exclude = C:\Program Files\VERITAS\NetBackup\bin\bpsched.d\*.lock
Exclude = C:\Program Files\VERITAS\Volmgr\misc\*
Exclude = D:\SQLServer\MSSQL$SQLSHNMOM1P\Data
Exclude = C:\RECYCLER\
Browser = pepwds00041
Clients = pepwds00041.corp.com
Port_BPCD = 13782
Port_BPRD = 13720
Debug_Flush = 0
Debug_Heap = 0
Debug_Level = 2
Debug_TCP = 0
VSP_CacheVols
VSP_Cache_Size_Init = 40
VSP_Cache_Size_Max = 60
Error_Messages_Max = 10
Policy_Type = 13
Client_Version = 500
Case_Sensitive_Exclude_List = NO
Compress_Use = 0
Compress_Buffer_Size = 62
Compress_Minimum_Size = 512
Cluster_Database_Force_Restore = NO
ChangeJournalUse = NO
GENERATE_ENGLISH_LOGS = NO
EXCHANGE_ENABLE_SIS = NO
VERSIONINFO = "WindowsNET" "5" "nt" "NetBackup" "6.0" 600000
VSP_Auto_Size = YES
# ./bptestbpcd -client pepwds00041 -verbose
0 0 2
156.81.204.223:756 -&gt; 156.81.238.206:13782
156.81.204.223:560 &lt;- 156.81.238.206:733
PEERNAME = media2.corp.com
HOSTNAME = pepwds00041
CLIENT_NAME = pepwds00041
VERSION = 0x06000404
PLATFORM = nt
156.81.204.223:752 &lt;- 156.81.238.206:767

Media Server3 Output:

# traceroute pepwds00041
traceroute to pepwds00041 (156.81.238.206), 30 hops max, 40 byte packets
&nbsp;1&nbsp; 156.81.204.12 (156.81.204.12)&nbsp; 0.381 ms&nbsp; 0.223 ms&nbsp; 0.219 ms
&nbsp;2&nbsp; pepwds00041 (156.81.238.206)&nbsp; 0.211 ms&nbsp; 0.163 ms&nbsp; 0.169 ms
# cd /usr/openv/netbackup/bin/admincmd
# ./bpgetconfig -M pepwds00041 -x
SERVER = mst2
SERVER = mst2.corp.com
SERVER = media1
SERVER = media2
SERVER = media3
SERVER = media4
BPBACKUP_POLICY = any
BPBACKUP_SCHED = any
BPARCHIVE_POLICY = any
BPARCHIVE_SCHED = any
CLIENT_NAME = pepwds00041
VERBOSE = 5
PEM_VERBOSE = -2
JM_VERBOSE = -2
RB_VERBOSE = -2
RESTORE_RETRIES = 0
CLIENT_READ_TIMEOUT = 2000
LIST_FILES_TIMEOUT = 300
DISALLOW_SERVER_FILE_WRITES = NO
BUSY_FILE_DIRECTORY
BUSY_FILE_NOTIFY_USER
ALLOW_NON_RESERVED_PORTS = YES
CRYPT_OPTION = ALLOWED
CRYPT_STRENGTH = DES_40
Debug_Database = 0
KEEP_LOGS_DAYS = 3
RANDOM_PORTS = YES
CLIENT_RESERVED_PORT_WINDOW = 512 1023
CLIENT_PORT_WINDOW = 0 0
Time_Overlap = 60
Buffer_Size = 32
Use_Archive_Bit = YES
Perform_Default_Search = YES
Accumulate_Files = NO
User_Directed_Timeout = 60
Announce_DHCP_Interval = 0
Clear_Archive_Bit_Incr_Wait = 300
Exclude = *.LDF
Exclude = *.MDF
Exclude = C:\Marimba\*
Exclude = C:\Program Files\Symantec AntiVirus\*
Exclude = C:\Program Files\VERITAS\NetBackup\bin\*.lock
Exclude = C:\Program Files\VERITAS\NetBackup\bin\bprd.d\*.lock
Exclude = C:\Program Files\VERITAS\NetBackup\bin\bpsched.d\*.lock
Exclude = C:\Program Files\VERITAS\Volmgr\misc\*
Exclude = D:\SQLServer\MSSQL$SQLSHNMOM1P\Data
Exclude = C:\RECYCLER\
Browser = pepwds00041
Clients = pepwds00041.corp.com
Port_BPCD = 13782
Port_BPRD = 13720
Debug_Flush = 0
Debug_Heap = 0
Debug_Level = 2
Debug_TCP = 0
VSP_CacheVols
VSP_Cache_Size_Init = 40
VSP_Cache_Size_Max = 60
Error_Messages_Max = 10
Policy_Type = 13
Client_Version = 500
Case_Sensitive_Exclude_List = NO
Compress_Use = 0
Compress_Buffer_Size = 62
Compress_Minimum_Size = 512
Cluster_Database_Force_Restore = NO
ChangeJournalUse = NO
GENERATE_ENGLISH_LOGS = NO
EXCHANGE_ENABLE_SIS = NO
VERSIONINFO = "WindowsNET" "5" "nt" "NetBackup" "6.0" 600000
VSP_Auto_Size = YES
# ./bptestbpcd -client pepwds00041 -verbose
0 0 2
156.81.204.225:887 -&gt; 156.81.238.206:13782
156.81.204.225:659 &lt;- 156.81.238.206:935
PEERNAME = media4.corp.com
HOSTNAME = pepwds00041
CLIENT_NAME = pepwds00041
VERSION = 0x06000404
PLATFORM = nt
156.81.204.225:530 &lt;- 156.81.238.206:659

&nbsp;

Forum Discussion

SQL Database Backup Fails...!

11 Replies

Related Content

Oracle database restore from vmware type backup

Oracle database restore fail

Failed to query database

Backup Fails for oracle database

parent job fails for database backup

Recent Discussions

command: bperror

MS-SharePoint policy restore error (2804) .

How to restore a backup

How to configure RBAC

10 years old netbackup appliance database service down, ssl certification out date