02-18-2009 08:41 AM
My Exchange 2007 policy only fails on the weekly backup. The daily and the monthly run successfully.
I either have the job fail because the server fails over to the 2nd cluster node or i get the following error
for each stream:
2/18/2009 9:52:06 AM - granted resource 0319L3
2/18/2009 9:52:06 AM - granted resource IBM.ULTRIUM-TD3.000
2/18/2009 9:52:06 AM - granted resource nvl-media1-hcart3-robot-tld-1
2/18/2009 9:52:11 AM - estimated 0 kbytes needed
2/18/2009 9:53:02 AM - connecting
2/18/2009 9:53:01 AM - Error bpbrm(pid=3656) from client nvl-ms9: ERR - object not found for file system backup: Microsoft Exchange Mailboxes:\[q-t]*
2/18/2009 9:53:02 AM - Error bpbrm(pid=3656) could not send server status message
2/18/2009 9:53:03 AM - connected; connect time: 00:00:01
2/18/2009 9:53:03 AM - begin writing
2/18/2009 9:53:21 AM - end writing; write time: 00:00:18
invalid filelist specification(69)
The policy has the mailboxes broken into the following streams:
NEW STREAM
Microsoft Exchange Mailboxes:\[a-e]*
NEW STREAM
Microsoft Exchange Mailboxes:\[f-l]*
NEW STREAM
Microsoft Exchange Mailboxes:\[m-p]*
NEW STREAM
Microsoft Exchange Mailboxes:\[q-t]*
NEW STREAM
Microsoft Exchange Mailboxes:\[u-z]*
NEW STREAM
Microsoft Exchange Mailboxes:\[_]*
I have media multiplexing set at 10. I'm not sure if something is wrong with how I have the streams set up. If the streams are not set up properly they could be putting a heavy load on the cluster to make it fail over. Any Ideas?
02-18-2009 12:26 PM
My hunch is you do not have the rights to see the mailboxes on the Exchange server.
Did you follow page 2 of this document on creating a service account to run the Netbackup client service on the Exchange server:
http://seer.entsupport.symantec.com/docs/292587.htm
If so, can you log in with the service account user and run Netbackup Archive Restore and see if you can enumerate the mailboxes there?
Have you also installed the Exchange MAPI client from Microsoft? Mapi is no longer part of Exchange 2007 and is required for Mailbox backups and restores.
Regards,
Benjamin Schmaus
02-18-2009 01:33 PM
Thanks for your response!
Yes we have a specific backup account that is a domain admin, and an Exchange Organization Admin. Its used to start the netbackup client service on both of the cluster nodes. I opened the backup/archive/restore client with "run as" with that account and made sure I could click on the mail store and see all the mailboxes. I did this for each cluster node.
The IMAPI client is installed also. Exchange IMAPI client 1.2.1 v6.5.8022.0
I looked up the bprm(pid=3656) in the bprm log on the media server and this is what I found (sorry for the long insert):
09:53:00.991 [3656.4972] <2> MNG: backup_cmd = /usr/openv/netbackup/bin/bpbkar bpbkar32 -r 3024000 -ru root -dt 0 -to 0 -clnt nvl-ms9 -class Exchange_2007 -sched Weekly_Mailboxes_2007 -st FULL -bpstart_to 500 -bpend_to 500 -read_to 18000 -blks_per_buffer 127 -stream_count 5 -stream_number 4 -jobgrpid 232258 -use_otm -use_ofb -b nvl-ms9_1234968731 -kl 7 -fso -ct 16
09:53:00.991 [3656.4972] <2> bpbrm handle_backup: forking client backup
09:53:00.991 [3656.4972] <2> bpbrm send_bpsched_connected_msg: sending bpsched msg: CONNECTED TO CLIENT FOR nvl-ms9_1234968731
09:53:01.101 [3656.4972] <2> get_readline_type: Returning BPBRM_READ_INFORM_WHEN_DONE
09:53:01.319 [3656.4972] <2> bpbrm read_backup_start: from client nvl-ms9: read client start message
09:53:01.319 [3656.4972] <2> bpbrm write_continue_backup: wrote CONTINUE BACKUP on COMM_SOCK <1648>
09:53:01.319 [3656.4972] <2> write_file_names: buffering file name 'Microsoft Exchange Mailboxes:\[q-t]*' for output
09:53:01.319 [3656.4972] <2> write_file_names: successfully wrote buffer to COMM_SOCK
09:53:01.319 [3656.4972] <2> bpbrm write_filelist: wrote CONTINUE on COMM_SOCK
09:53:01.507 [3656.4972] <2> bpbrm handle_backup: ESTIMATE -1 -1 nvl-ms9_1234968731
09:53:01.507 [3656.4972] <16> bpbrm handle_backup: from client nvl-ms9: ERR - object not found for file system backup: Microsoft Exchange Mailboxes:\[q-t]*
09:53:01.507 [3656.4972] <2> vnet_cached_gethostbyname: vnet_hosts.c.307: found host in cache: nvl-bk2
09:53:01.507 [3656.4972] <2> vnet_cached_gethostbyname: vnet_hosts.c.307: found host in cache: nvl-bk2
09:53:01.507 [3656.4972] <2> vnet_async_connect: vnet_vnetd.c.4019: connect in progress: 0 0x00000000
09:53:01.522 [3656.4972] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2043: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
09:53:01.522 [3656.4972] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2057: service: bpjobd
09:53:01.694 [3656.4972] <2> vnet_async_connect: vnet_vnetd.c.4205: in progress connect: 0 0x00000000
09:53:01.694 [3656.4972] <2> vnet_async_connect: vnet_vnetd.c.4208: connect: async CONNECT FROM 10.0.5.92.3581 TO 10.0.5.83.13724 fd = 1624
09:53:01.694 [3656.4972] <2> logconnections: BPJOBD CONNECT FROM 10.0.5.92.3581 TO 10.0.5.83.13724
09:53:01.694 [3656.4972] <2> vauth_authentication_required: vauth_comm.c.749: no methods for address: no authentication required
09:53:01.694 [3656.4972] <2> vauth_connector: vauth_comm.c.182: no methods for address: no authentication required
09:53:01.694 [3656.4972] <2> job_authenticate_connection: no authentication required
09:53:01.694 [3656.4972] <2> job_authenticate_connection: ignoring VxSS authentication check for now...
09:53:01.694 [3656.4972] <2> job_connect: Connected to the host nvl-bk2 contype 10 jobid <232262> socket <1624>
09:53:01.694 [3656.4972] <2> job_connect: Connected on port 3581
09:53:01.694 [3656.4972] <2> set_job_details: Sending Tfile jobid (232262)
09:53:01.694 [3656.4972] <2> set_job_details: LOG 1234968781 16 bpbrm 3656 from client nvl-ms9: ERR - object not found for file system backup: Microsoft Exchange Mailboxes:\[q-t]*
09:53:01.694 [3656.4972] <2> set_job_details: Done
09:53:01.913 [3656.4972] <2> job_monitoring_exex: ACK disconnect
09:53:01.913 [3656.4972] <2> job_disconnect: Disconnected
09:53:01.913 [3656.4972] <2> vnet_cached_gethostbyname: vnet_hosts.c.307: found host in cache: nvl-bk2
09:53:01.913 [3656.4972] <2> vnet_cached_gethostbyname: vnet_hosts.c.307: found host in cache: nvl-bk2
09:53:01.913 [3656.4972] <2> vnet_async_connect: vnet_vnetd.c.4019: connect in progress: 0 0x00000000
09:53:01.929 [3656.4972] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2043: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
09:53:01.929 [3656.4972] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2057: service: bpdbm
09:53:02.132 [3656.4972] <2> vnet_async_connect: vnet_vnetd.c.4205: in progress connect: 0 0x00000000
09:53:02.132 [3656.4972] <2> vnet_async_connect: vnet_vnetd.c.4208: connect: async CONNECT FROM 10.0.5.92.3582 TO 10.0.5.83.13724 fd = 1624
09:53:02.132 [3656.4972] <2> logconnections: BPDBM CONNECT FROM 10.0.5.92.3582 TO 10.0.5.83.13724
09:53:02.132 [3656.4972] <2> vauth_authentication_required: vauth_comm.c.749: no methods for address: no authentication required
09:53:02.132 [3656.4972] <2> vauth_connector: vauth_comm.c.182: no methods for address: no authentication required
09:53:02.132 [3656.4972] <2> check_authentication: no authentication required
09:53:02.132 [3656.4972] <2> vnet_check_vxss_client_magic_with_info: vnet_vxss_helper.c.770: Ignoring VxSS authentication: 2 0x00000002
09:53:02.132 [3656.4972] <2> vnet_check_vxss_client_magic_with_info: vnet_vxss_helper.c.926: Not using VxSS authentication: 2 0x00000002
09:53:02.851 [3656.4972] <2> bpbrm handle_backup: client nvl-ms9 EXIT STATUS = 69: invalid filelist specification
09:53:02.851 [3656.4972] <2> inform_client_of_status: INF - Server status = 69
09:53:02.851 [3656.4972] <2> put_long: (11) network write() error: An established connection was aborted by the software in your host machine. ; socket = 1648
09:53:02.851 [3656.4972] <16> inform_client_of_status: could not send server status message
09:53:02.851 [3656.4972] <2> vnet_cached_gethostbyname: vnet_hosts.c.307: found host in cache: nvl-bk2
09:53:02.851 [3656.4972] <2> vnet_cached_gethostbyname: vnet_hosts.c.307: found host in cache: nvl-bk2
09:53:02.851 [3656.4972] <2> vnet_async_connect: vnet_vnetd.c.4019: connect in progress: 0 0x00000000
09:53:02.866 [3656.4972] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2043: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
09:53:02.866 [3656.4972] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2057: service: bpjobd
09:53:02.897 [3656.4972] <2> vnet_async_connect: vnet_vnetd.c.4205: in progress connect: 0 0x00000000
09:53:02.897 [3656.4972] <2> vnet_async_connect: vnet_vnetd.c.4208: connect: async CONNECT FROM 10.0.5.92.3584 TO 10.0.5.83.13724 fd = 1620
09:53:02.897 [3656.4972] <2> logconnections: BPJOBD CONNECT FROM 10.0.5.92.3584 TO 10.0.5.83.13724
09:53:02.897 [3656.4972] <2> vauth_authentication_required: vauth_comm.c.749: no methods for address: no authentication required
09:53:02.897 [3656.4972] <2> vauth_connector: vauth_comm.c.182: no methods for address: no authentication required
09:53:02.897 [3656.4972] <2> job_authenticate_connection: no authentication required
09:53:02.897 [3656.4972] <2> job_authenticate_connection: ignoring VxSS authentication check for now...
09:53:02.897 [3656.4972] <2> job_connect: Connected to the host nvl-bk2 contype 10 jobid <232262> socket <1620>
09:53:02.897 [3656.4972] <2> job_connect: Connected on port 3584
09:53:02.897 [3656.4972] <2> set_job_details: Sending Tfile jobid (232262)
09:53:02.897 [3656.4972] <2> set_job_details: LOG 1234968782 16 bpbrm 3656 could not send server status message
02-20-2009 11:31 AM
What is the difference between your daily/monthly schedules and your weekly? Are they incrementals, cumulatives or fulls? Mailbox vs information store?
You mentioned this is a cluster, CCR cluster or a shared storage backend cluster? Doing circular logging? If CCR are you using snapshots and backing up passive node?
I noticed in the bpbrm log it only has errors for the mailboxes q-t. Do you see the same errors for the other streams?
Depending on your answers above, do the following:
Create the following log on the both nodes of the cluster C:\Program Files\VERITAS\Netbackup\logs\bpbkar
-Optional: Increase the logging level on the client by opening up Netbackup Archive Restore: File-> Netbackup Client Properties ->Troubleshooting. Set General to 2 and Verbose to 5. Restart Netbackup Client Service
Run your job and then paste the log that gets created.
Regards,
Benjamin Schmaus
02-20-2009 12:30 PM
The cluster is a shared storage backend cluster. I'm not sure about the logging.
I ran this command to get the policy info for the Mailbox policy and the Info Store policy:
bppllist <exchange_policy_name> -U > c:\exchangepolicy.txt
------------------------------------------------------------
Policy Name: Exchange_Information_Stores
Policy Type: MS-Exchange-Server
Active: yes
Effective date: 02/02/2009 11:10:52
Mult. Data Streams: no
Client Encrypt: no
Checkpoint: no
Policy Priority: 0
Max Jobs/Policy: Unlimited
Disaster Recovery: 0
Collect BMR info: no
Residence: nvl-media1-hcart3-robot-tld-1
Volume Pool: NetBackup
Server Group: *ANY*
Keyword: (none specified)
Data Classification: Gold
Residence is Storage Lifecycle Policy: no
Granular Restore Info: no
HW/OS/Client: PC-X64 Windows2003 nvl-ms9
Include: NEW_STREAM
Microsoft Information Store:\
Schedule: Monthly_Info_Store
Type: Automatic Backup
Maximum MPX: 1
Synthetic: 0
PFI Recovery: 0
Retention Level: 8 (1 year)
Number Copies: 1
Fail on Error: 0
Residence: (specific storage unit not required)
Volume Pool: Vault_1_Year
Server Group: (same as specified for policy)
Calendar sched: Enabled
Monday, Week 2
Residence is Storage Lifecycle Policy: 0
Daily Windows:
Monday 14:00:00 --> Tuesday 03:00:00
Schedule: Weekly_Info_Store
Type: Automatic Backup
Maximum MPX: 1
Synthetic: 0
PFI Recovery: 0
Retention Level: 4 (5 weeks)
Number Copies: 1
Fail on Error: 0
Residence: (specific storage unit not required)
Volume Pool: Weekly_Backup
Server Group: (same as specified for policy)
Calendar sched: Enabled
Monday, Week 1
Monday, Week 3
Monday, Week 4
Monday, Week 5
Residence is Storage Lifecycle Policy: 0
Daily Windows:
Monday 14:00:00 --> Tuesday 03:00:00
Schedule: Daily_Info_Stores
Type: Differential Incremental Backup
Maximum MPX: 1
Synthetic: 0
PFI Recovery: 0
Retention Level: 3 (8 days)
Number Copies: 1
Fail on Error: 0
Residence: (specific storage unit not required)
Volume Pool: Daily
Server Group: (same as specified for policy)
Calendar sched: Enabled
Residence is Storage Lifecycle Policy: 0
Daily Windows:
Sunday 00:00:00 --> Sunday 06:00:00
Tuesday 14:00:00 --> Wednesday 06:00:00
Wednesday 14:00:00 --> Thursday 06:00:00
Thursday 14:00:00 --> Friday 06:00:00
Friday 14:00:00 --> Saturday 06:00:00
Saturday 14:00:00 --> Saturday 24:00:00
------------------------------------------------------------
Policy Name: Exchange_2007
Policy Type: MS-Exchange-Server
Active: yes
Effective date: 05/21/2003 09:34:48
Mult. Data Streams: yes
Client Encrypt: no
Checkpoint: no
Policy Priority: 1
Max Jobs/Policy: Unlimited
Disaster Recovery: 0
Collect BMR info: no
Residence: nvl-media1-hcart3-robot-tld-1
Volume Pool: Daily
Server Group: *ANY*
Keyword: (none specified)
Data Classification: Gold
Residence is Storage Lifecycle Policy: no
Granular Restore Info: no
HW/OS/Client: PC-X64 Windows2003 nvl-ms9
Include: NEW_STREAM
Microsoft Exchange Mailboxes:\[_-e]*
NEW_STREAM
Microsoft Exchange Mailboxes:\[f-l]*
NEW_STREAM
Microsoft Exchange Mailboxes:\[m-p]*
NEW_STREAM
Microsoft Exchange Mailboxes:\[q-t]*
NEW_STREAM
Microsoft Exchange Mailboxes:\[u-z]*
Schedule: Monthly_Mailboxes_2007
Type: Automatic Backup
Maximum MPX: 10
Synthetic: 0
PFI Recovery: 0
Retention Level: 8 (1 year)
Number Copies: 1
Fail on Error: 0
Residence: nvl-media1-hcart3-robot-tld-1
Volume Pool: Vault_1_Year
Server Group: (same as specified for policy)
Calendar sched: Enabled
Tuesday, Week 1
Residence is Storage Lifecycle Policy: 0
Daily Windows:
Tuesday 18:00:00 --> Wednesday 07:00:00
Schedule: User
Type: Application Backup
Maximum MPX: 1
Synthetic: 0
PFI Recovery: 0
Retention Level: 8 (1 year)
Number Copies: 1
Fail on Error: 0
Residence: (specific storage unit not required)
Volume Pool: (same as policy volume pool)
Server Group: (same as specified for policy)
Residence is Storage Lifecycle Policy: 0
Daily Windows:
Sunday 00:00:00 --> Saturday 23:50:00
Schedule: Weekly_Mailboxes_2007
Type: Automatic Backup
Maximum MPX: 10
Synthetic: 0
PFI Recovery: 0
Retention Level: 4 (5 weeks)
Number Copies: 1
Fail on Error: 0
Residence: nvl-media1-hcart3-robot-tld-1
Volume Pool: Weekly_Backup
Server Group: (same as specified for policy)
Calendar sched: Enabled
Allowed to retry after run day
Tuesday, Week 2
Tuesday, Week 3
Tuesday, Week 4
Tuesday, Week 5
Residence is Storage Lifecycle Policy: 0
Daily Windows:
Tuesday 18:00:00 --> Wednesday 07:10:00
Schedule: Daily_Mailboxes_2007
Type: Differential Incremental Backup
Maximum MPX: 10
Synthetic: 0
PFI Recovery: 0
Retention Level: 3 (8 days)
Number Copies: 1
Fail on Error: 0
Residence: (specific storage unit not required)
Volume Pool: (same as policy volume pool)
Server Group: (same as specified for policy)
Calendar sched: Enabled
Monday, Week 1
Wednesday, Week 1
Thursday, Week 1
Friday, Week 1
Saturday, Week 1
Monday, Week 2
Wednesday, Week 2
Thursday, Week 2
Friday, Week 2
Saturday, Week 2
Monday, Week 3
Wednesday, Week 3
Thursday, Week 3
Friday, Week 3
Saturday, Week 3
Monday, Week 4
Wednesday, Week 4
Thursday, Week 4
Friday, Week 4
Saturday, Week 4
Monday, Week 5
Wednesday, Week 5
Thursday, Week 5
Friday, Week 5
Saturday, Week 5
Residence is Storage Lifecycle Policy: 0
Daily Windows:
Monday 18:00:00 --> Tuesday 06:00:00
Wednesday 18:00:00 --> Thursday 06:00:00
Thursday 18:00:00 --> Friday 06:00:00
Friday 18:00:00 --> Saturday 06:00:00
Saturday 18:00:00 --> Sunday 06:00:00
----------------------------------
I am running another job right now with the bpkar log turned on on the client. I will post it below.
02-25-2009 10:57 AM
So your information store backups work, but your mailbox backups do not?
For mailbox backups you need to have the Netbackup client service run as n account that has rights to peer into the mailboxes. You also need the MAPI client installed. I posted a technote on this board in one of the Exchange 2007 threads.
02-25-2009 11:05 AM
(from an above post...)
Yes we have a specific backup account that is a domain admin, and an Exchange Organization Admin. Its used to start the netbackup client service on both of the cluster nodes. I opened the backup/archive/restore client with "run as" with that account and made sure I could click on the mail store and see all the mailboxes. I did this for each cluster node.
02-25-2009 06:16 PM