Failure to logon to a server using backup exec

[Decatur5717]

Failure to logon to a server using backup exec

This morning I get into work and find that 3 (out of 22 full backups) backups have failed. They all express the same message. This is the 3^rd weekend in a row that this has happened (strangely enough it doesn’t fail often during the week with incremental)

Job ended: Saturday, May 11, 2013 at 4:04:26 AM Completed status: Failed Final error: 0xe0009b61 - Backup Exec could not log on to the server with the logon account specified for it. The logon account does not have valid credentials. Ensure that the user name and password are correct in the logon account. Final error category: Security Errors

Job ended: Sunday, May 12, 2013 at 10:04:24 AM Completed status: Failed Final error: 0xe0009b61 - Backup Exec could not log on to the server with the logon account specified for it. The logon account does not have valid credentials. Ensure that the user name and password are correct in the logon account. Final error category: Security Errors

Job ended: Sunday, May 12, 2013 at 4:04:24 PM Completed status: Failed Final error: 0xe0009b61 - Backup Exec could not log on to the server with the logon account specified for it. The logon account does not have valid credentials. Ensure that the user name and password are correct in the logon account. Final error category: Security Errors

These errors make zero sense to me as they cannot possibly be due to an account error. Just as a quick check and so you can understand what all is going on here is a list of everything

Backup server
-Server 2008 R2
-Backup Exec 2012 rev 1798
-Dell TL Tape Library
-Connected to domain

Environment
-22 servers are scheduled for full backups over the weekend
-same group of servers do Incremental Monday – Friday
-All servers (except the backup server connected to the Tape drive) are running VMware 5
 

Permissions
-All servers are connected to the domain
-All servers are linked to their own security group in AD (Administrators -> ServerLocalAdmins)
-This security group is linked to the Masters Admin group (ServerLocalAdmins -> MasterAdmin)
-This group contains the user account that Backup Exec uses for logons

My Troubleshooting

Monday April 29
-Noticed backup failures (Same servers plus 4 others)
-Triple checked to make sure the permission structure is valid – Looks good
-Tested the Credentials within the job - Success for all
-Logged into the server using the Backup Exec account – Looks good
-Reran the Full backup – Success
-Left it along as I figured it was a false positive

Monday May 6
-Noticed backup failures (Same servers minus the 4 from previous weekend)
-Checked the permission structure again – Looks good
-Tested the Credentials within the job - Success for all
-Reran the Full backup – Success
-Checked for Backup Exec updates – Currently running the newest
-Logged into servers and checked the BE Agent – Looks good
-Checked local Event viewer – No related errors (had a few printer errors related to RDP, so no issue)
-Removed the server from BE and re-setup – Done
-Reran the Full Backup – Success

Monday May 13
-Noticed the backup failures (Same servers)
-Posted this

And of course this problem has to happen to 3 of my most important servers. Any ideas how I can resolve?

[CraigV]

...did the account perhaps lock out during the backup?

Thanks!

[Decatur5717]

I sure wish that was the case .... but no. Here is the next backup for the server following the first error (using the same account)

 

Byte count          : 29,126,430,591 bytes

Job rate            : 1,224.77 MB/Min (Byte count of all backup sets divided by Elapsed time for all backup sets)
Files               : 47,701

Directories         : 11,845

Skipped files       : 0

Corrupt files       : 0

Files in use        : 0
Original start time : Saturday, May 04, 2013 5:00:00 AM

Job started         : Saturday, May 11, 2013 5:00:00 AM

Job ended           : Saturday, May 11, 2013 5:32:43 AM

Elapsed time        : 00:32:43

And according to group policy ..... after 5 failures it locks the account

Just looking through the error reports ..... I don't see anything that says the account failed to logon

 

[lmosla]

Were there any maintenance jobs that might have disrupted the backups?

[Decatur5717]

Such as tape cleaning? Then no. But I am currently looking into other related scheduled tasks that may use these servers at the same time, so far I can't find any. At this point the only thing I can find that is different about these servers vs the others is the date they were created. I have 5 servers that have been created sense I updated the BE software a few months ago. Of the 5 servers 3 are failing.

On a side note, about 2 months ago (due to another error where BE's agents simply failed, hence all servers failed) I completed removed all traces of BE and reinstalled. These 5 servers were added to BE on April 10th and I've been messing with the jobs and scheduling until April 25th ...ish. The first backup that included these servers failed along with 2 others ... hence the April 29th report of "Noticed backup failures (Same servers plus 4 others)"

What throws me off is the next weekend only 3 of those newly created servers fail

I'll keep looking but for the moment I am leaning towards possible connection issues with the servers, either because something else is running or that a port is being used that is needed for this operation..... who knows

[CraigV]

...have you checked to make sure that any AV installed isn't perhaps causing issues with BE?

THanks!

[Decatur5717]

Looked into that as well and embarassing as it is I haven't installed it yet :(

[pkh]

The default time for BEDB maintenance is 4 a.m.  Make sure that your jobs does not clash with this time.  If necessary, click on the BE button ---> Configuration and Settings ---> Backup Exec Settings ---> Database Maintenance and change the timing.

[Decatur5717]

but this still doesn't explain the 10AM and 4PM errors

[Decatur5717]

Logging in this morning I don't see any of these problems, I have other backup issues but they relate to a change in the file structure. I guess I'll have to wait until next weekend to see if these problems appear again ..... for now, seems its been resolved (though I did not make any changes)