Scheduled jobs are not starting

I've been working on a problem for a little over a month and have managed to even stump the fine folks at Symantec so far, so I though I would reach out to all of you and see if I can get this fixed.

Back in March, my scheduled jobs suddenly just stopped working. They just don't start. If I restart my servers, or services, the jobs will queue up and run fine for approximately 24 hours, then not run again. I have approximately 150 policies that will show up if I type in:

nbpemreq -predict -date (next 24 hours)

If I do the same command the next day, there will only be 4 policies that show up. These policies have worked seemlessly throughout the problem, which doesn't make troubleshooting any easier. The policies that work are wide ranging, from a flat file backup, Exchange, and RMAN's. Some write to a Data Domain, some write direct to tape. New backup policies behave in the same manner. Manually starting a job works fine.

Master - W2K8 running NBU 7.5.0.3

Clients - vary between Windows and Redhat (both fail)

Any suggestions would be greatly appreciated. I will post any log files that you need.

Windows Server (2003-2008)

Dan_Giberson
12 years ago
Ok....I think we have a fix...I hope. It turns out there was a wildcard (*) in a policy that caused NBPEM to try and calculate over 700,000 files into new streams which caused it to constantly hang. As such, I have now gone through and removed all wildcards from my policies.

I will continue to monitor this through the weekend. Thanks everyone for all the suggestions.

35 Replies

LucSkywalker195
Level 4
12 years ago
Do you see anything in your Windows system and application event logs that might indicate something going on with the server? Did you make or introduce any changes or patches last March?
Dan_Giberson
Level 5
12 years ago
Nope, I looked at that too.....the only work I was doing was troubleshooting SLP replication.
Marianne
Level 6
12 years ago
What type of scheduling are you using? Calendar or Frequency?

If Calendar - do the schedules span midnight?

Were any backups kicked of manually after backup window closed?
Dan_Giberson
Level 5
12 years ago
I only use Calendar based scheduling in my environment. Some of these jobs will span midnight, but not all. I can manually kick a backup off at any time and it will run perfectly fine.
Marianne
Level 6
12 years ago
The problem with manual backups is that it will affect scheduled backups. If a daily has run during the day, another backup will not be started that night.
LucSkywalker195
Level 4
12 years ago
When's the last time you rebooted your master?
Jaykullar
Level 5
12 years ago
I have see this previously, however in 7.1.0.4.

Can you pick a policy that does not auto run, delete your schedules from it. Then run nbpemreq -updatepolicies. The re-create your schedules in the same policy and again run nbpemreq -updatepolicies.

Then run nbpemreq -predict_all -date **/**/**

Let us know how that figures.
Dan_Giberson
Level 5
12 years ago
Luc, I have rebooted my master several times in the last couple of weeks. If I reboot, everything appears to work fine for the first 24 hours, then it halts again.

Jaykullar, tried that with the same result. The only jobs in the predict list are the one's that have never stopped working, but good suggestion. If I create a new policy, it's the same as well. Even if I copy from one of the policies that still work, it fails....
mph999
Level 6
12 years ago
" ... and have managed to even stump the fine folks at Symantec "

Yikes, this will be fun then ...

I doubt the type of job is relevant, can't see why pem would care.

Anyhows ...

in the pem log, for a given client/ policy you will see lines like this :

[PolicyClientTask::cancel] scheduling for policy <policy name>, client <client name>, has been abandoned because of image expiration, will recalculate

then, a bit later you should see :

[PolicyClientTask::run] policy <policy name>, client <client name>, schedule <schedule name> will be submitted for execution at <date /time>

For a given policy/ client do you find that at some point, the "submitted for execution" lines suddenly stop ?

Cheers,

M
Jaykullar
Level 5
12 years ago
Sounds like a bit of problem there bud.

Have Symantec recommened MP5 at all?

I know this sounds silly, but are you using Java or Admin Console? Have you tried creating policies in both?