Hi guys,I´ve been experiencing some issues with my Netbackup 7.7.3. (yes, we are planning to upgrade) I have 8 appliances 2.7.3.A few days ago I had to perform a factory reset on one of my appliances, after this I´m seeing a lot of error 54 during 12 pm and 0:00 am (weird ah?). Not only with this appliance but with all of them.Before the factory reset, I had no problems.In order to have the jobs running normally, I have to shut down the services in this appliance. During the rest of the backup window, I have no problem even with this appliance.I hope I was clear.

Hi AndresV While I think Marianne has the most sensible suggestion and you should start planning an upgrade, I'm curious about what might be the cause of the problem you are seeing. I struggle to see why an issue on the one appliance would cause widespread issues as you describe and only at those specific times. Can you answer some questions please? Are the backups that are failing all SQL backups - and are they transaction log backups? The connection setup for this type of backup is somewhat different to a standard backup). For the job details you provided, I see it was using the media server (and disk pool) from the server ecbpprnb3. Is this the problem server? Are the other backups that fail at the same time using this media server in any way (load balancing, disk pool etc)? If you remove the problem appliance from backups, how do you manage the lack of the media server and associated disk pool? Is this a manual process to change the storage or what? Although I cant see why this might be time related are you sure you restored all network routing correctly to the appliance? When you performed the factory reset, did you retain the backup data on the appliance? Can the data on that appliance be dsicarded or recreated? - a possible solution might be to reimage the appliance and start from scratch. Have you re-installed all EEBs that were installed on the appliance (and are installed on the other appliances)? What was the reason for performing the factory reset in the first place? To me it seems like one of the house keeping tasks the appliance runs is causing the issue, but why it affects all backups at that time eludes me for now. I don't have ready access to an appliance to see what might be running at that time (but I'll fire up a lab when I have a spare moment and see what I can see). David

Hi AndresV So a status 54 is a timeout connecting to client issue. When you performed the factory reset, did you retain the network settings? Even if so were there any bp.conf paramters that you may have had modified (in particular the timeout related ones) that were not restored after the reset? Next - how are these errors appearing (not so clear to me)? Are they backup issues (status 54) with random clients that are using this media server which occur at 0:00 and 12:00? At what stage are the backups? Have you verifiedd that you have restored ALL the required SERVER entries into the media server bp.conf? Does the appliance behave correctly during the rest of the time? Lots of possibilities - I think for us to help, we need more information. David

Hi AndresV Thanks for your answers - but of course this leads to more questions.... So if I understand correctly from your replies, the failed jobs use media server load balancing to write to the STU on the problem appliance - can you confirm this please?. What is the nature of all the data that is backed up by these differential backups every 2 hours? (SQL, File, is it compressed or encrypted) Besides the 2 hourly differential backups what else happens around these problem times in NetBackup (other backup jobs, tape duplications)? Okay - so tell us a bit more about the configuration of the problem appliance - is it the same as was originally delivered or has it been upgraded subsequently (officially or otherwise)? How much storage and how much RAM is in the appliance? Is all or most of the storage utilised as MSDP? For the backups that are running does anything different happen at the source around the problem times (extra processing leading to more data for instance to be backed up)? Have you checked that there are no underlying hardware issues on this problem appliance (RAID set issues for instance - CLISH: Support->Test Hardware)? When you modify the policies to avoid using the problem appliance do you change everything to use a single different STU? Have you just tried changing the policy to the different STU without shutting down NBU on the problem appliance? Does it cause backup failures? Can you provide details of a job that has failed on a different appliance also please? Assuming you can manage the data on this appliance (discarding or duplicating elsewhere if needed), an interrim option could be to reimage the appliance (rather than factory reset) using the USB stick that should have been provided. I still think upgrading to a current supported version of NetBackup is your best bet (but certainly more painful than fixing a single appliance). David

Hi David,I appreciate your answer. I didn't retain the network settings because I had a ifconfig backup, /etc/hosts backup and bp.conf backup.We run differential jobs every 2 hours but the jobs which have to run 12 pm and 0:00 am start failing with error 54, not only jobs on this appliance but with all appliances. The next differential jobs run fine on every applianceThe rest of the day the appliances work fine including the one which was reset.Yes, I checked the entries in the bp.conf file and in the host properties -> media server -> servers tab

AndresV Can you please share all text in Job Details of a status 54 job? It is okay if you want to replace server names with generic names, e.g. Master, MediaAppl1, Client1, etc.

Error 54 every 12 hours

11 Replies

davidmoline
Level 6
5 years ago
Hi AndresV

So a status 54 is a timeout connecting to client issue. When you performed the factory reset, did you retain the network settings? Even if so were there any bp.conf paramters that you may have had modified (in particular the timeout related ones) that were not restored after the reset?

Next - how are these errors appearing (not so clear to me)? Are they backup issues (status 54) with random clients that are using this media server which occur at 0:00 and 12:00? At what stage are the backups?

Have you verifiedd that you have restored ALL the required SERVER entries into the media server bp.conf?

Does the appliance behave correctly during the rest of the time?

Lots of possibilities - I think for us to help, we need more information.

David
- AndresV
  Level 6
  5 years ago
  Hi David,
  I appreciate your answer. I didn't retain the network settings because I had a ifconfig backup, /etc/hosts backup and bp.conf backup.
  We run differential jobs every 2 hours but the jobs which have to run 12 pm and 0:00 am start failing with error 54, not only jobs on this appliance but with all appliances. The next differential jobs run fine on every appliance
  The rest of the day the appliances work fine including the one which was reset.
  Yes, I checked the entries in the bp.conf file and in the host properties -> media server -> servers tab
  - Marianne
    Level 6
    5 years ago
    AndresV
    
    Can you please share all text in Job Details of a status 54 job?
    
    It is okay if you want to replace server names with generic names, e.g. Master, MediaAppl1, Client1, etc.

Forum Discussion

Error 54 every 12 hours

11 Replies

Related Content

IT Analytics - Backup Success Rate Percentage past 24 hours

Exchange snapshot takes 2 hours

Last 24 hours backup report

Slow restore, 410GB 12 hours.

Amout of data backed up by Media Servers per hour for last 24 hours