Is there anything special about backing up Windows FTP servers via NetBackup? One FTP server is a Windows 2003 server and the other is a Windows 2008 server and they both reside within our DMZ domain. They are using the basic Windows FTP service, nothing special or fancy. I am able to FTP from the outside and via command line.
In the past couple of weeks, the Activity Monitor has been showing Errors 13 (file read error) and 59 (access to the client was not allowed) for both servers upon doing a differential and full backup's.
For both FTP servers, I have the NetBackup domain account configured to run the NetBackup client services on both boxes. The password and account are both valid.
I know that this is not a firewall or network ACL blockage as NetBackup is successfully backing up other web servers and domain controllers that reside within our DMZ domain with no problem.
Both servers are fully patched and up to date with vendor patches and service packs as is the NetBackup server.
Here is the NetBackup server configuration:
Windows Server 2003
NetBackup v6.5.6 (same for the FTP clients on the servers)
NetBackup server contains both master and media server.
Intel Xeon CPU
3.99GB of RAM
I’ve consulted the Status Code documentation that Symantec offers, but to no avail have any of those helped me in this situation.
If anyone has come across a similar situation as this, please help! J I look forward to any comments/suggestions/advice/criticism.
Thanks to all. I appreciate it.
access to the client was not allowed
The master or the media server tries to access the client, but the client does not recognize the server as a valid server.
Please ensure that bpcd log exists on clients. After next status 59, check log for incoming request from media server. See if client can resolve media server IP address to hostname that corresponds with a valid SERVER entry in client's config.
Status 13 could be a network error:
file read failed
A read of a file or socket failed.
The possible causes include as follows:
* A network communication problem has occurred on the master server, media server, or one of the clients.
* A socket read failure caused by a network problem or a problem with the process that writes to the socket.
Do the following, as appropriate:
* Check the NetBackup Problems report for clues on where and why the problem occurred.
* Check that network communication is working properly.
See "Resolving network communication problems" in the Troubleshooting Guide.
**** I do realise that I've copied the above from the Troubleshooting Guide, but network errors are not easy to pinpoint. We need more info - either error in Problems Report or client's bpbkar log.
Can you check if both the Windows cilents permit access from the media server?
To be specific, a "Server" entry of the media server in your client host properties, or in registry if you know how to check that.
It's very common if we use a storage unit group for client backup. Let say we used to have 2 media servers in the group to backup these 2 Windows clients, then one day we added in a new media server, but we forgot to grant access for this media server to access those clients.
Thank you for your reply back. I've checked everything per your suggestions and do not see any network communication problems...funny thing is, I deleted the policy for the FTP servers then re-created them exactly the way they were setup before I deleted them. Differentials ran no problem the two nights before, but what is weird is that the differentials ran successfully for both FTP servers, then about 3 hours later NB attempted to do another differential backup and then the error 13 showed up for both FTP clients...I don't know why if NB had two successful differential backup's for both FTP servers, why would it attempt to do another differential backup within the same backup window?
Watsons - Thank you for your reply as well. Both Windows clients have full connectivity between the master server and from the master server back to the clients. Ping is good, all that. The master/media server (because they are both on the same server) are correctly identified within both of the Windows FTP registries.
1. Can you please provide the following output from the master server.
2. On the Client, please confirm if you have the master server entries in the 'hosts' file.
Here is the following information from the master server:
|The following hosts were found:|
|Command completed successfully.|
|JobID||Type||State||Status||Policy||Schedule||Client||Dest Media Srv||Active||PID||FATPipe|
I've omitted all successful backup's from this query as they don't need any analyzation. The error's 13 and 24 are the most common ones that are happening to the FTP servers, which reside in our DMZ domain. As you can see, the error 13 is also happening on our Exchange servers, which reside within the internal LAN, and this policy is just for backing up their local drives (C:\, E:\ etc etc).
Unfortunately, there were no error 59's throughout the Full backup's that were attempted/completed over this weekend. Just 24's and 13's.
Also, I've checked on all of the clients that are getting these errors, and they already have the master NB server hostname contained within their host files.
Attached is the query for bpcd...
Also,...I've set Verbose logging to 5 within the registry of the master NB server HKLM\Software\Veritas\NetBackup\CurrentVersion\Config, so do I do the same thing to enable logging for the bpbkar? Do I have to create a bpbkar folder within <install path>\Veritas\NetBackup\Logs (like I did to get the text file for bpcd) and then try and re-run the policy to get the information submitted via text file for the bpbkar?
Log directories do no exist by default - they need to be created to enable logging.
Create the folders on the client under <install-path>\veritas\netbackup\logs.
You should find a 'mklogdir.bat' in the logs folder. If you run this, it will create all possible log directories (most of them never needed on a client). I prefer to create log directories manually.
I have followed the instructions from the below TN, yet after I re-run the backup, no output is being generated in the bpbkar creted directory...the bpcd text file is growing however, from the failed jobs lol:
On Windows clients, the following steps are used:
1. Create the \\<install_path>\NetBackup\logs\bpbkar directory. Generated log information will be placed in this directory in a file named mmddyy.log (where mm = month, dd = date, yy = year).
2. Create an empty file called \\<install_path>\NetBackup\bpbkar_path_tr (ensure that the file has no extension and is named as defined)
3. Enable logging for the desired system by setting the 'Global logging level' to 5(maximum). This can be found in the Logging tab of the Host Properties | Clients | Properties section of the NetBackup Administration Console.
4. Restart the 'NetBackup Client Service' in the Control Panel Services applet (usually contained within Administrative Tools)
5. Run a backup to generate the log information
Where was the the bpcd log collected from - the client or the master/media server?
The bpcd log posted above seems to be coming from the media server - you can see that bpbrm (backup and restore manager) process is started :
bpcd main: fork cmd = /usr/openv/netbackup/bin/bpbrm bpbrm -backup -S monorail -c bugs -ct 13 -ru root -cl DMZ1-Domain-Controllers -sched Full
This bpbrm process on the media server will initiate connection with client 'bugs'.
The client's bpcd log will only be useful after status 59 is seen.
Please also confirm that bpbkar log directory was created on the client (bugs).
"I've set Verbose logging to 5 within the registry of the master NB server "
Logging level on the CLIENT needs to be increased - as per the extract from the TN that you've posted above:
In Host Properties, select Clients, select problematic client (bugs), right-click -> Properties. Under Logging, increase logging level to 5.
here is the bpcd output from the client (bugs):
12:58:00.937 [5200.5268] <16> bpcd main: Server access denied
12:59:29.296 [5020.3212] <2> bpcd main: offset to GMT 28800
12:59:29.296 [5020.3212] <2> bpcd main: Got socket for input 612
12:59:29.296 [5020.3212] <2> logconnections: BPCD ACCEPT FROM 220.127.116.11.4267 TO 18.104.22.168.13724
12:59:29.296 [5020.3212] <2> bpcd main: setup_sockopts complete
12:59:29.312 [5020.3212] <2> bpcd peer_hostname: Connection from host monorail.dmc.pa.mil (22.214.171.124) port 4267
12:59:29.312 [5020.3212] <2> bpcd valid_server: comparing monorail and monorail.dmc.pa.mil
12:59:36.176 [5020.3212] <2> hosts_equal: gethostbyname failed for monorail: The requested name is valid, but no data of the r (0)
12:59:36.176 [5020.3212] <4> bpcd valid_server: monorail.dmc.pa.mil is not a master server
12:59:36.176 [5020.3212] <16> bpcd valid_server: monorail.dmc.pa.mil is not a media server either
12:59:36.176 [5020.3212] <2> bpcd main: output socket port number = 1
12:59:36.176 [5020.3212] <2> bpcd main: Duplicated vnetd socket on stderr
12:59:36.176 [5020.3212] <2> bpcd main: <---- NetBackup 6.5 0 ------------initiated
12:59:36.176 [5020.3212] <2> bpcd main: VERBOSE = 0
12:59:36.176 [5020.3212] <2> bpcd exit_bpcd: exit status 46 ----------->exiting
12:59:36.176 [5020.3212] <4> bpcd exit_bpcd: FTL - BPCD EXIT STATUS 46
Also, I cannot increase the logging level (In Host Properties, select Clients, select problematic client (bugs), right-click -> Properties. Under Logging, increase logging level to 5.) because when I double-click the client within the master server, I get an acess denied:
Add FQDN for master to client's SERVER entries. Use BAR GUI on client to do this.
Another option is to add alias for short and FQDN in client's hosts file.
You can see in bpcd log that the client resolves master's IP address as FQDN:
Connection from host monorail.dmc.pa.mil
Client only has SERVER entry for master's short name:
bpcd valid_server: comparing monorail and monorail.dmc.pa.mil
bpcd valid_server: monorail.dmc.pa.mil is not a master server
bpcd valid_server: monorail.dmc.pa.mil is not a media server either
I've added the FQDN via the BAR GUI on the client's SERVER list and am running a manual full backup. So far, so good! I see the two DC's with written percentages, so it looks like this was the fix even though I had the FQDN of the NB server within the client's host files...I'll let you know how this goes once the backup's have completed.
Thanks a million for your help on fixing the error 13's for the FTP servers. There are also error 13's happening on our Windows Server 2003 that have Exchange 2003 installed on them. The bpcd directory is there but the information pertaining to the error 13 that the Exchange servers are seeing are completely different than that seen on the FTP servers. Could you please help me diagnose what the problem is with the attached error 13 information since this forum still pertains to general error 13 errors?
All NB is backing up on these Exchange servers are ALL_LOCAL_DRIVES and System States...still, they result in the error 13...I'm not seeing any DNS related issues within the bpcd output.
The error 13 for the Exchange servers only seem to disappear temporarily when I delete then re-create the policy to backup ALL_LOCAL_DRIVES, then they bomb out again with the 13.
Any help is greatly appreciated. And many thanks to all who have contributed to helping me find a solution to these bothersome error 13's especially Marianne.
The bpcd log in the last post has no errors - initial 'handshake' between media server and client is successful. Status 13 is caused by a problem during the data transfer process - either a 'file read error' or 'socket read error'.
Please run a Problems Report for each client producing status 13 to determine which of the above is causing the problem. In most cases, the problem is network-related.
Please check bpbkar log on client as well as bptm and bpbrm logs on media server for errors.
Sometimes resolving status 13 is as easy as increase the Client Read Timeout on the media server (which seems to be your master server).
I guess I'm going to close this even though the error 13's are still happening...I've done everything I can on here to remediate this issue, yet it continues to happen (only to the Exchange 2003) servers now. Any other ideas before I close this out?
Sorry this has taken so long for me to update. What happened after this post was that I ended up re-creating the policies for the Exchange servers and the nightly differential backup's for the Exchange servers local drives and mailboxes started working fine; however, the error 13's started showing up again as of last week.
Attached are the three error logs you've requested. The bpbkar log for one of the Exchange servers that is getting the error 13 failure in NBU, and the two bptm and bpbrm log files for the master NBU server.
If you have a chance, please take a look at them and let me know what you think as this error is completely perplexing to me. As stated previously, I've researched the error 13's from this forum and on other sites, but nothing is helping to resolve this ongoing issue with the three Exchange servers (this is happening to some of the FTP servers here as well, but maybe if we can pin down exactly what is happening to the Exchange servers, it might be the same resolution for the FTP servers).
Thanks much to any and all who may have any suggestions. I appreciate it.