Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- VOX
- Data Protection
- Backup Exec
- DST Issues - I have some answers!
Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
DST Issues - I have some answers!
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-30-2006 10:29 AM
* Our servers operate on Alaska Time and we follow DST
Initial observed behavior:
* Backup jobs scheduled to run the night of Oct 28th/morning of Oct 29th ran successfully and on reasonable schedules.
* Some backup jobs appeared to maintain their DST-relative windows (i.e. jobs scheduled for 6:45 AM ADT were properly rescheduled for 6:45 AM AST).
* Other backup jobs did not have their DST-relative windows adjusted (i.e. jobs scheduled for 10:00 PM ADT were now scheduled for 11:00 PM ADT, and jobs scheduled for 11:20 PM ADT were now scheduled for 12:20 AM ADT).
Behavior observed when I tried to fix things on Sunday, Oct 29th, between 6 AM and 2 PM:
* I attempted to reschedule the jobs, and observed that jobs that were properly rescheduled for 6:45 AM AST showed the proper schedule and behaved properly under all attempts at manipulation.
* Jobs that did not get rescheduled properly displayed schedules when I went to edit them that were off by an hour. If I reset the schedule to the proper time and submitted the job, the time showed in the Job Monitor as being off by an hour!
* If I then opened these jobs, the schedule showed as being off by an hour, and resubmitting the job without touching the schedule caused the job to adjust forward yet another hour!
* Through quick experimentation I was able to determine that the critical time was around 3 PM. Jobs with windows starting at 2:00 PM and ending at 2:10 PM showed up in Job Monitor properly. Jobs with windows starting at 3:00 PM and ending at 3:10 PM showed up in Job Monitor as being scheduled for 4:00 PM.
* The above described behavior was independent of the schedule start time, whether the job was due to run Sunday night, Monday night, or even Friday night!
* The 3:00 PM time does have significance, however. 3:00 PM AST (4:00 PM ADT) is 00:00 UTC!
* I rescheduled all of the late night jobs using times shifted by an hour - i.e. to schedule a job for 10:00 PM, I entered the schedule start window as starting at 9:00 PM.
* At this point, I did not know whether the jobs would actually start at 9:00 PM or whether they would run at 10:00 PM (that is to say, I didn't know what to trust).
* I set my alarm clock for critical time points in the night to watch the jobs (hey, I was tired, so I napped in the computer room and got up periodically to check on the servers via VPN). The jobs ran based on the time displayed in the Job Monitor (i.e. 10 PM), not the times entered when creating the jobs (i.e. 9 PM)!
Behavior observed Monday, Oct 30th, around 6 AM:
* Out of curiosity, I opened the jobs. The jobs showed 10:00 PM to 10:10 PM as the window. However, when I resubmitted the jobs, the times no longer jumped!
Current Theory:
The job execution engine is fine. What shows in Job Monitor is when the job will actually run. The job
scheduling engine has a problem. Somehow it's mixing the local time zone with UTC something like this:
* Take window time in local time zone (i.e. 10:00 PM). Subtract the UTC offset for the local time zone (i.e. our offset is -0900 when we are on AST, so subtract -0900, which is the same as adding 0900). This results in 7:00 AM UTC.
* Mixin the current date. That is to say, create the complete time stamp Oct 29 7:00 AM UTC.
* Convert that back into local time. Oct 29 7:00 AM UTC in AT is Oct 28 11:00 PM ADT because it is before 3 AM on Oct 29th in the local time zone, thus the offset is only -0800.
* Adjust the timestamp to the current date - i.e Oct 29 11:00 PM AST. Convert that back into UTC (since that is how things are evidently stored in the database). That is, Oct 29 8:00 AM.
* Schedule based on that.
I may be overcomplicating this. But somehow, the system is screwing things up for certain time windows when the manipulation happens on the day of the DST change if the window times are the next day in UTC. It doesn't screw them up in the window times are the same day in UTC. The same logic must be being used to adjust the schedules for the jobs when the system sees DST change as is used for creating the jobs, thus some jobs get adjusted properly (i.e. 6:45 AM ADT -> 6:45 AM AST), but others do not (10:00 PM ADT -> 11:00 PM AST).
Upshot:
* Until Symantec/Veritas fixes this (which doesn't seem to be high on their list - see http://seer.support.veritas.com/docs/280521.htm), plan on working the Sunday of any DST change.
* Inspect all of your jobs and their scheduled times. Plan on iterative changes to the times until what shows in Job Monitor is correct. You may need to enter jobs with times one hour ahead or behind the desired run time. In some situations, you may have to wait until Monday to make some changes because of limitations in the code. Adjusting the schedule in effect date doesn't trick the system into fixing the bug. Sorry!
* Since we don't know whether Symantec/Veritas rolled their own code or relies on the OS, plan on working both the new 2007 DST dates and the 2007 DST dates that would have been in effect if Congress hadn't decided to earn the ire of every IT person by adjusting the DST schedules!!!!
* Don't be surprised if you see different UI behavior on Sunday and Monday following a DST change.
--Toby Ovod-Everett
Initial observed behavior:
* Backup jobs scheduled to run the night of Oct 28th/morning of Oct 29th ran successfully and on reasonable schedules.
* Some backup jobs appeared to maintain their DST-relative windows (i.e. jobs scheduled for 6:45 AM ADT were properly rescheduled for 6:45 AM AST).
* Other backup jobs did not have their DST-relative windows adjusted (i.e. jobs scheduled for 10:00 PM ADT were now scheduled for 11:00 PM ADT, and jobs scheduled for 11:20 PM ADT were now scheduled for 12:20 AM ADT).
Behavior observed when I tried to fix things on Sunday, Oct 29th, between 6 AM and 2 PM:
* I attempted to reschedule the jobs, and observed that jobs that were properly rescheduled for 6:45 AM AST showed the proper schedule and behaved properly under all attempts at manipulation.
* Jobs that did not get rescheduled properly displayed schedules when I went to edit them that were off by an hour. If I reset the schedule to the proper time and submitted the job, the time showed in the Job Monitor as being off by an hour!
* If I then opened these jobs, the schedule showed as being off by an hour, and resubmitting the job without touching the schedule caused the job to adjust forward yet another hour!
* Through quick experimentation I was able to determine that the critical time was around 3 PM. Jobs with windows starting at 2:00 PM and ending at 2:10 PM showed up in Job Monitor properly. Jobs with windows starting at 3:00 PM and ending at 3:10 PM showed up in Job Monitor as being scheduled for 4:00 PM.
* The above described behavior was independent of the schedule start time, whether the job was due to run Sunday night, Monday night, or even Friday night!
* The 3:00 PM time does have significance, however. 3:00 PM AST (4:00 PM ADT) is 00:00 UTC!
* I rescheduled all of the late night jobs using times shifted by an hour - i.e. to schedule a job for 10:00 PM, I entered the schedule start window as starting at 9:00 PM.
* At this point, I did not know whether the jobs would actually start at 9:00 PM or whether they would run at 10:00 PM (that is to say, I didn't know what to trust).
* I set my alarm clock for critical time points in the night to watch the jobs (hey, I was tired, so I napped in the computer room and got up periodically to check on the servers via VPN). The jobs ran based on the time displayed in the Job Monitor (i.e. 10 PM), not the times entered when creating the jobs (i.e. 9 PM)!
Behavior observed Monday, Oct 30th, around 6 AM:
* Out of curiosity, I opened the jobs. The jobs showed 10:00 PM to 10:10 PM as the window. However, when I resubmitted the jobs, the times no longer jumped!
Current Theory:
The job execution engine is fine. What shows in Job Monitor is when the job will actually run. The job
scheduling engine has a problem. Somehow it's mixing the local time zone with UTC something like this:
* Take window time in local time zone (i.e. 10:00 PM). Subtract the UTC offset for the local time zone (i.e. our offset is -0900 when we are on AST, so subtract -0900, which is the same as adding 0900). This results in 7:00 AM UTC.
* Mixin the current date. That is to say, create the complete time stamp Oct 29 7:00 AM UTC.
* Convert that back into local time. Oct 29 7:00 AM UTC in AT is Oct 28 11:00 PM ADT because it is before 3 AM on Oct 29th in the local time zone, thus the offset is only -0800.
* Adjust the timestamp to the current date - i.e Oct 29 11:00 PM AST. Convert that back into UTC (since that is how things are evidently stored in the database). That is, Oct 29 8:00 AM.
* Schedule based on that.
I may be overcomplicating this. But somehow, the system is screwing things up for certain time windows when the manipulation happens on the day of the DST change if the window times are the next day in UTC. It doesn't screw them up in the window times are the same day in UTC. The same logic must be being used to adjust the schedules for the jobs when the system sees DST change as is used for creating the jobs, thus some jobs get adjusted properly (i.e. 6:45 AM ADT -> 6:45 AM AST), but others do not (10:00 PM ADT -> 11:00 PM AST).
Upshot:
* Until Symantec/Veritas fixes this (which doesn't seem to be high on their list - see http://seer.support.veritas.com/docs/280521.htm), plan on working the Sunday of any DST change.
* Inspect all of your jobs and their scheduled times. Plan on iterative changes to the times until what shows in Job Monitor is correct. You may need to enter jobs with times one hour ahead or behind the desired run time. In some situations, you may have to wait until Monday to make some changes because of limitations in the code. Adjusting the schedule in effect date doesn't trick the system into fixing the bug. Sorry!
* Since we don't know whether Symantec/Veritas rolled their own code or relies on the OS, plan on working both the new 2007 DST dates and the 2007 DST dates that would have been in effect if Congress hadn't decided to earn the ire of every IT person by adjusting the DST schedules!!!!
* Don't be surprised if you see different UI behavior on Sunday and Monday following a DST change.
--Toby Ovod-Everett
Labels:
0 REPLIES 0
Related Content
- Issue with Veritas System Recovery 22 and Windows Server, see my Description in System Recovery
- Discrepancy in Charged TB in NetBackup Deployment Utility in NetBackup
- Alta Recovery Vault - Which Documents Should I Use??? in NetBackup
- Veritas System Recovery Manager 22 issue in System Recovery
- Need Troubleshooting Advice for Alta Recovery Vault? Look NO Further! in NetBackup
