Snapshot Preparation Failed , status 156
Guys, help me in resolving the issue.
Environment:
NBU Master version 7.1.0.4 :: on VCS 5.0 :: 2 nodes :: RHEL5_x86_ 2.6
NBU Media servers verion :: 6.5.6 :: RHEL5 ::Serving as alternate client for snapshot backups.
Snapshot Method :: Legacy Shadowimage for Hitachi USPV -- Hitachi AMS
Client OS: RHEL5
VXVM 5.0mp3 on master, media and clients
Policy vaildation fails with below error:
There was an error found with snapshot configuration on clients specified in this policy.
Policy: MBS2_store5 -- client: xxxx-xxx-xxx --Snapshot could not be created with ShadowImage method - (Status 223).
Tried running the backup anyway resulting in 156.
Bpplinfo:
Policy Type: FlashBackup (20)
Active: yes
Effective: 01/05/2010 14:44:01
Follow NFS Mounts: no
Cross Mount Points: no
Client Compress: no
Collect TIR info: no
Policy Priority: 0
Ext Security Info: no
File Restore Raw: yes
Client Encrypt: no
Max Jobs/Policy: Unlimited
Mult. Data Stream: yes
Block Level Incremental: no
Perform Snapshot Backup: yes
Snapshot Method: ShadowImage
Snapshot Method Arguments: sync_at_start=0,keep_fi=0,timeout=240,async_resync=0
Perform Offhost Backup: yes
Backup Copy: 0
Use Data Mover: no
Data Mover Type: -1
Use Alternate Client: yes
Alternate Client Name: xxxxxxxxxxxx
Use Virtual Machine: 0
Hyper-V Server: (none)
Enable Instant Recovery: no
Collect BMR Info: no
Max Frag Size: 0 MB
Checkpoint Restart: no
Residence: xxxxxxxxxxxx-hcart-robot-tld-2
Volume Pool: MbsPool
Server Group: *ANY*
Data Classification: -
Residence is storage lifecycle policy: no
Granular Restore: no
Generation: 4
Ignore Client Direct: no
Exchange Source attributes: no
Application Discovery: no
Discovery Lifetime: 0 seconds
Recent changes: A new set of Luns were presented to both client (parent volume) and media servers(clone volume).
Backup process: Involves suspending mirror (Primary to clone), snapshot creation of volume (rdsk), importing VXVMdiskgroup _clone .
Bpfis on client :
7:12:53.278 [11154] <2> parse_ssm_parms: Found prompt Sync I/O delay in milliseconds (UNIX only)
17:12:53.278 [11154] <2> onlfi_vfms_logf: INF - vfm_configure_fi_one: configuring VxFI credentials
17:12:53.285 [11154] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
17:12:53.285 [11154] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bprd
17:12:53.326 [11154] <2> logconnections: BPRD CONNECT FROM 10.4.20.144.47911 TO 10.4.22.59.13724
17:12:53.449 [11154] <2> get_long: (2) premature end of file (byte 1)
17:12:53.449 [11154] <2> bprd_read_text_file: get_string() failed, Input/output error (5), premature end of file encountered
17:12:53.449 [11154] <2> onlfi_vfms_logf: INF - vfm_configure_fi_one: read credentials file failed
17:12:53.455 [11154] <2> onlfi_vfms_logf: INF - TimeFinder FIM $Revision: 1.54.8.4 $
17:12:53.457 [11154] <4> bpfis: INF - FIS_ID=dsmdc-mail-mbs3_1393974591
17:12:53.477 [11154] <4> onlfi_fim_dev_info: INF - Preparing freeze of /dev/vx/rdsk/mbs3store5dg/store5 using snapshot method ShadowImage.
17:12:58.038 [11154] <32> onlfi_fim_dev_info: FTL - VfMS error 11; see following messages:
17:12:58.038 [11154] <32> onlfi_fim_dev_info: FTL - Fatal method error was reported
17:12:58.038 [11154] <32> onlfi_fim_dev_info: FTL - vfm_freeze: method: ShadowImage, type: FIM, function: ShadowImage_freeze
17:12:58.038 [11154] <32> onlfi_fim_dev_info: FTL - VfMS method error 3; see following message:
17:12:58.038 [11154] <32> onlfi_fim_dev_info: FTL - set_devinfo: Could not find targ info for device /dev/sdad
17:12:58.038 [11154] <4> onlfi_thaw: INF - Thawing /dev/vx/rdsk/mbs3store5dg/store5 using snapshot method ShadowImage.
17:12:58.039 [11154] <8> do_thaw: WRN - VfMS error 2; see following messages:
17:12:58.039 [11154] <8> do_thaw: WRN - Feature is not supported
17:12:58.039 [11154] <8> do_thaw: WRN - vfm_fi_ctl: Function is not supported for this method
17:12:58.039 [11154] <4> onlfi_thaw: INF - do_thaw return value: 0
17:12:58.040 [11154] <16> bpfis: FTL - snapshot preparation failed, status 156
17:12:58.040 [11154] <4> onlfi_thaw: INF - Thawing /dev/vx/rdsk/mbs3store5dg/store5 using snapshot method ShadowImage.
17:12:58.040 [11154] <8> onlfi_thaw: WRN - /dev/vx/rdsk/mbs3store5dg/store5 is not frozen
17:12:58.040 [11154] <8> bpfis: WRN - snapshot delete returned status 20
17:12:58.046 [11154] <4> bpfis: INF - EXIT STATUS 156: snapshot error encountered
17:13:00.065 [11431] <2> logparams: bpfis delete -nbu -id dsmdc-mail-mbs3_1393974591 -bpstart_to 300 -bpend_to 300 -clnt dsmdc-mail-mbs3 -S dsmdc-bkp-mst-cls
17:13:00.068 [11431] <4> bpfis: INF - BACKUP START 11431
17:13:00.068 [11431] <32> fis_rebuild_from_db: FTL - cannot open /usr/openv/netbackup/online_util/fi_cntl/bpfis.fim.dsmdc-mail-mbs3_1393974591.0
17:13:00.068 [11431] <4> bpfis: INF - EXIT STATUS 0: the requested operation was successfully completed
BPBRM on Media server
17:12:53.016 [11845] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bpcd
17:12:53.056 [11845] <2> logconnections: BPCD CONNECT FROM 10.4.22.55.38142 TO 10.4.20.144.13724
17:12:53.056 [11845] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.182: msg: VNETD CONNECT FROM 10.4.22.55.33078 TO 10.4.20.144.13724 fd = 7
17:12:53.064 [11845] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.535: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
17:12:53.105 [11845] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.552: ipc_string: /tmp/vnet-11151393974773103770000000000-BgmzUC
17:12:53.187 [11845] <2> bpcr_get_hostname_rqst: Server hostname length = 26
17:12:53.187 [11845] <2> bpbrm start_bpcd_stat: DATA_SOCK from bpcr = 6
17:12:53.187 [11845] <2> bpbrm start_bpcd_stat: NAME_SOCK from bpcr = 7
17:12:53.188 [11845] <2> pfi_start_client: command = /usr/openv/netbackup/bin/bpfis bpfis create -nbu -owner NBU -rhost dsmdc-bkp-med1 -id dsmdc-mail-mbs3_1393974591 -bpstart_to 300 -bpend_to 300 -b
ackup_copy 0 -ru root -pt 20 -clnt dsmdc-mail-mbs3 -rg root -fim ShadowImage:sync_at_start=0,timeout=240 -class MBS3_store5 -sched Full-Snap -fso -S dsmdc-bkp-mst-cls
17:12:53.188 [11845] <2> pfi_start_client: received bpcd success message
17:12:53.253 [11845] <2> pfi_start_client: read start message from dsmdc-mail-mbs3, msg=<INF - BACKUP START 11154>
17:12:53.253 [11845] <2> pfi_start_client: client_pid=11154
17:12:53.253 [11845] <2> write_file_names: buffering file name 'NEW_STREAM' for output
17:12:53.253 [11845] <2> write_file_names: buffering file name '/dev/vx/rdsk/mbs3store5dg/store5' for output
17:12:53.253 [11845] <2> write_file_names: successfully wrote buffer to COMM_SOCK
17:12:53.253 [11845] <2> bpbrm write_filelist: wrote CONTINUE on COMM_SOCK
17:12:53.457 [11845] <2> get_bpfis_msg: bpfis buff: INF - FIS_ID=dsmdc-mail-mbs3_1393974591
17:12:53.480 [11845] <2> get_bpfis_msg: bpfis buff:
17:12:58.039 [11845] <2> get_bpfis_msg: bpfis buff:
17:12:58.040 [11845] <2> get_bpfis_msg: bpfis buff:
17:12:58.040 [11845] <2> get_bpfis_msg: bpfis buff: FTL - snapshot preparation failed, status 156
17:12:58.041 [11845] <32> get_bpfis_msg: from client dsmdc-mail-mbs3: FTL - snapshot preparation failed, status 156
17:12:58.043 [11845] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
17:12:58.043 [11845] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bpdbm
17:12:58.085 [11845] <2> logconnections: BPDBM CONNECT FROM 10.4.22.55.34826 TO 10.4.22.59.13724
17:12:58.171 [11845] <2> get_bpfis_msg: bpfis buff: WRN - /dev/vx/rdsk/mbs3store5dg/store5 is not frozen
17:12:58.171 [11845] <8> get_bpfis_msg: from client dsmdc-mail-mbs3: WRN - /dev/vx/rdsk/mbs3store5dg/store5 is not frozen
17:12:58.173 [11845] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
17:12:58.173 [11845] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bpdbm
17:12:58.215 [11845] <2> logconnections: BPDBM CONNECT FROM 10.4.22.55.47037 TO 10.4.22.59.13724
17:12:58.300 [11845] <2> bpbrm kill_child_process: start
17:12:58.300 [11845] <2> bpbrm Exit: attempting to send mail to root on dsmdc-mail-mbs3
17:12:58.300 [11845] <2> hosts_equal: Comparing hosts <dsmdc-mail-mbs3> and <dsmdc-mail-mbs3>
17:12:58.300 [11845] <2> hosts_equal: names are the same
17:12:59.849 [11854] <2> decode_fim_attr: decode_fim_attr: fim=ShadowImage:alternate_backup_client=dsmdc-bkp-med1,sync_at_start=0,keep_fi=0,timeout=240,async_resync=0, client version=650000
17:12:59.849 [11854] <2> decode_fim_attr: decode_fim_attr: mapped fim: ShadowImage:sync_at_start=0,timeout=240, rfi_backup 1, keep_fi 0
17:12:59.849 [11854] <2> bpbrm Exit: backup_status: 156
17:12:59.849 [11854] <2> bpbrm start_bpcd_stat: bpbrm.c.19676: bpcd_client: dsmdc-mail-mbs3
17:12:59.849 [11854] <2> bpbrm start_bpcd_stat: bpbrm.c.19679: bpcd_client_hostname: dsmdc-mail-mbs3
17:12:59.849 [11854] <2> hosts_equal: Comparing hosts <dsmdc-mail-mbs3> and <dsmdc-mail-mbs3>
17:12:59.849 [11854] <2> hosts_equal: names are the same
17:12:59.868 [11854] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
17:12:59.868 [11854] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bpcd
17:12:59.909 [11854] <2> logconnections: BPCD CONNECT FROM 10.4.22.55.54700 TO 10.4.20.144.13724
17:12:59.909 [11854] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.182: msg: VNETD CONNECT FROM 10.4.22.55.41407 TO 10.4.20.144.13724 fd = 6
17:12:59.926 [11854] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.535: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
17:12:59.968 [11854] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.552: ipc_string: /tmp/vnet-11429393974779967209000000000-iQ0xmU
17:13:00.051 [11854] <2> bpcr_get_hostname_rqst: Server hostname length = 26
17:13:00.051 [11854] <2> bpbrm start_bpcd_stat: DATA_SOCK from bpcr = 4
17:13:00.051 [11854] <2> bpbrm start_bpcd_stat: NAME_SOCK from bpcr = 6
17:13:00.052 [11854] <2> pfi_start_client: command = /usr/openv/netbackup/bin/bpfis bpfis delete -nbu -id dsmdc-mail-mbs3_1393974591 -bpstart_to 300 -bpend_to 300 -clnt dsmdc-mail-mbs3 -S dsmdc-bkp-
mst-cls
17:13:00.052 [11854] <2> pfi_start_client: received bpcd success message
17:13:00.068 [11854] <2> pfi_start_client: read start message from dsmdc-mail-mbs3, msg=<INF - BACKUP START 11431>
17:13:00.068 [11854] <2> pfi_start_client: client_pid=11431
17:13:00.068 [11854] <2> get_bpfis_msg: bpfis buff: FTL - cannot open /usr/openv/netbackup/online_util/fi_cntl/bpfis.fim.dsmdc-mail-mbs3_1393974591.0
17:13:00.068 [11854] <32> get_bpfis_msg: from client dsmdc-mail-mbs3: FTL - cannot open /usr/openv/netbackup/online_util/fi_cntl/bpfis.fim.dsmdc-mail-mbs3_1393974591.0
No. For array snapshot methods, there is a piece of software that is installed on the client and the media server. This software contains all the utilities and commands that needs to be used to perform checks, snapshots, break-off of mirror, etc.
These utilities normally have a username and password that needs to be supplied during initial NBU config (and when anything has changed). Detailed steps for each supported array type is in the Snapshot Client manual.As I've said above, I am not familiar with HDS tools, but I have assisted with similar configs with EMC and HP array clones/mirrors.
The Snapshot Client manual contains steps such as:
Steps to be performed by storage admin.
Steps to be performed by NBU admin.Please go through steps in the manual as if it is a new config to verify setup.