Forum Discussion

Corruptcache's avatar
Corruptcache
Level 3
11 years ago

Snapshot Preparation Failed , status 156

Guys, help me in resolving the issue.

Environment:

NBU Master version 7.1.0.4 ::  on VCS 5.0 :: 2 nodes :: RHEL5_x86_ 2.6

NBU Media servers verion :: 6.5.6 :: RHEL5 ::Serving as alternate client for snapshot backups.

Snapshot Method :: Legacy Shadowimage for Hitachi USPV -- Hitachi AMS

Client OS: RHEL5

VXVM  5.0mp3 on master, media  and clients

Policy vaildation fails with below error:

There was an error found with snapshot configuration on clients specified in this policy.

Policy: MBS2_store5 -- client: xxxx-xxx-xxx --Snapshot could not be created with ShadowImage method - (Status 223).

Tried running the backup anyway resulting in 156.

Bpplinfo:

Policy Type:            FlashBackup (20)
Active:                 yes
Effective:              01/05/2010 14:44:01
Follow NFS Mounts:      no
Cross Mount Points:     no
Client Compress:        no
Collect TIR info:       no
Policy Priority:        0
Ext Security Info:      no
File Restore Raw:       yes
Client Encrypt:         no
Max Jobs/Policy:        Unlimited
Mult. Data Stream:      yes
Block Level Incremental:        no
Perform Snapshot Backup:        yes
Snapshot Method:                ShadowImage
Snapshot Method Arguments:      sync_at_start=0,keep_fi=0,timeout=240,async_resync=0
Perform Offhost Backup:         yes
Backup Copy:                    0
Use Data Mover:                 no
Data Mover Type:                -1
Use Alternate Client:           yes
Alternate Client Name:          xxxxxxxxxxxx
Use Virtual Machine:            0
Hyper-V Server:         (none)
Enable Instant Recovery:        no
Collect BMR Info:       no
Max Frag Size:          0 MB
Checkpoint Restart:     no
Residence:              xxxxxxxxxxxx-hcart-robot-tld-2
Volume Pool:            MbsPool
Server Group:           *ANY*
Data Classification:            -
Residence is storage lifecycle policy:  no
Granular Restore:       no
Generation:     4
Ignore Client Direct:   no
Exchange Source attributes:              no
Application Discovery:  no
Discovery Lifetime:     0 seconds

Recent changes: A new set of  Luns were presented to both client (parent volume) and media servers(clone volume). 

Backup process: Involves suspending mirror (Primary to clone), snapshot creation of volume (rdsk), importing VXVMdiskgroup _clone .

Bpfis on client :

7:12:53.278 [11154] <2> parse_ssm_parms: Found prompt Sync I/O delay in milliseconds (UNIX only)
17:12:53.278 [11154] <2> onlfi_vfms_logf: INF - vfm_configure_fi_one: configuring VxFI credentials
17:12:53.285 [11154] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
17:12:53.285 [11154] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bprd
17:12:53.326 [11154] <2> logconnections: BPRD CONNECT FROM 10.4.20.144.47911 TO 10.4.22.59.13724
17:12:53.449 [11154] <2> get_long: (2) premature end of file (byte 1)
17:12:53.449 [11154] <2> bprd_read_text_file: get_string() failed, Input/output error (5), premature end of file encountered
17:12:53.449 [11154] <2> onlfi_vfms_logf: INF - vfm_configure_fi_one: read credentials file failed
17:12:53.455 [11154] <2> onlfi_vfms_logf: INF - TimeFinder FIM $Revision: 1.54.8.4 $
17:12:53.457 [11154] <4> bpfis: INF - FIS_ID=dsmdc-mail-mbs3_1393974591
17:12:53.477 [11154] <4> onlfi_fim_dev_info: INF - Preparing freeze of /dev/vx/rdsk/mbs3store5dg/store5 using snapshot method ShadowImage.
17:12:58.038 [11154] <32> onlfi_fim_dev_info: FTL - VfMS error 11; see following messages:
17:12:58.038 [11154] <32> onlfi_fim_dev_info: FTL - Fatal method error was reported
17:12:58.038 [11154] <32> onlfi_fim_dev_info: FTL - vfm_freeze: method: ShadowImage, type: FIM, function: ShadowImage_freeze
17:12:58.038 [11154] <32> onlfi_fim_dev_info: FTL - VfMS method error 3; see following message:
17:12:58.038 [11154] <32> onlfi_fim_dev_info: FTL - set_devinfo: Could not find targ info for device /dev/sdad
17:12:58.038 [11154] <4> onlfi_thaw: INF - Thawing /dev/vx/rdsk/mbs3store5dg/store5 using snapshot method ShadowImage.
17:12:58.039 [11154] <8> do_thaw: WRN - VfMS error 2; see following messages:
17:12:58.039 [11154] <8> do_thaw: WRN - Feature is not supported
17:12:58.039 [11154] <8> do_thaw: WRN - vfm_fi_ctl: Function is not supported for this method
17:12:58.039 [11154] <4> onlfi_thaw: INF - do_thaw return value: 0
17:12:58.040 [11154] <16> bpfis: FTL - snapshot preparation failed, status 156
17:12:58.040 [11154] <4> onlfi_thaw: INF - Thawing /dev/vx/rdsk/mbs3store5dg/store5 using snapshot method ShadowImage.
17:12:58.040 [11154] <8> onlfi_thaw: WRN - /dev/vx/rdsk/mbs3store5dg/store5 is not frozen
17:12:58.040 [11154] <8> bpfis: WRN - snapshot delete returned status 20
17:12:58.046 [11154] <4> bpfis: INF - EXIT STATUS 156: snapshot error encountered
17:13:00.065 [11431] <2> logparams: bpfis delete -nbu -id dsmdc-mail-mbs3_1393974591 -bpstart_to 300 -bpend_to 300 -clnt dsmdc-mail-mbs3 -S dsmdc-bkp-mst-cls
17:13:00.068 [11431] <4> bpfis: INF - BACKUP START 11431
17:13:00.068 [11431] <32> fis_rebuild_from_db: FTL - cannot open /usr/openv/netbackup/online_util/fi_cntl/bpfis.fim.dsmdc-mail-mbs3_1393974591.0
17:13:00.068 [11431] <4> bpfis: INF - EXIT STATUS 0: the requested operation was successfully completed

BPBRM on Media server

17:12:53.016 [11845] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bpcd
17:12:53.056 [11845] <2> logconnections: BPCD CONNECT FROM 10.4.22.55.38142 TO 10.4.20.144.13724
17:12:53.056 [11845] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.182: msg: VNETD CONNECT FROM 10.4.22.55.33078 TO 10.4.20.144.13724 fd = 7
17:12:53.064 [11845] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.535: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
17:12:53.105 [11845] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.552: ipc_string: /tmp/vnet-11151393974773103770000000000-BgmzUC
17:12:53.187 [11845] <2> bpcr_get_hostname_rqst: Server hostname length = 26
17:12:53.187 [11845] <2> bpbrm start_bpcd_stat: DATA_SOCK from bpcr = 6
17:12:53.187 [11845] <2> bpbrm start_bpcd_stat: NAME_SOCK from bpcr = 7
17:12:53.188 [11845] <2> pfi_start_client: command = /usr/openv/netbackup/bin/bpfis bpfis create -nbu -owner NBU -rhost dsmdc-bkp-med1 -id dsmdc-mail-mbs3_1393974591 -bpstart_to 300 -bpend_to 300 -b
ackup_copy 0 -ru root -pt 20 -clnt dsmdc-mail-mbs3 -rg root -fim ShadowImage:sync_at_start=0,timeout=240 -class MBS3_store5 -sched Full-Snap -fso -S dsmdc-bkp-mst-cls
17:12:53.188 [11845] <2> pfi_start_client: received bpcd success message
17:12:53.253 [11845] <2> pfi_start_client: read start message from dsmdc-mail-mbs3, msg=<INF - BACKUP START 11154>
17:12:53.253 [11845] <2> pfi_start_client: client_pid=11154
17:12:53.253 [11845] <2> write_file_names: buffering file name 'NEW_STREAM' for output
17:12:53.253 [11845] <2> write_file_names: buffering file name '/dev/vx/rdsk/mbs3store5dg/store5' for output
17:12:53.253 [11845] <2> write_file_names: successfully wrote buffer to COMM_SOCK
17:12:53.253 [11845] <2> bpbrm write_filelist: wrote CONTINUE on COMM_SOCK
17:12:53.457 [11845] <2> get_bpfis_msg: bpfis buff: INF - FIS_ID=dsmdc-mail-mbs3_1393974591
17:12:53.480 [11845] <2> get_bpfis_msg: bpfis buff:
17:12:58.039 [11845] <2> get_bpfis_msg: bpfis buff:
17:12:58.040 [11845] <2> get_bpfis_msg: bpfis buff:
17:12:58.040 [11845] <2> get_bpfis_msg: bpfis buff: FTL - snapshot preparation failed, status 156
17:12:58.041 [11845] <32> get_bpfis_msg: from client dsmdc-mail-mbs3: FTL - snapshot preparation failed, status 156
17:12:58.043 [11845] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
17:12:58.043 [11845] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bpdbm
17:12:58.085 [11845] <2> logconnections: BPDBM CONNECT FROM 10.4.22.55.34826 TO 10.4.22.59.13724
17:12:58.171 [11845] <2> get_bpfis_msg: bpfis buff: WRN - /dev/vx/rdsk/mbs3store5dg/store5 is not frozen
17:12:58.171 [11845] <8> get_bpfis_msg: from client dsmdc-mail-mbs3: WRN - /dev/vx/rdsk/mbs3store5dg/store5 is not frozen
17:12:58.173 [11845] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
17:12:58.173 [11845] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bpdbm
17:12:58.215 [11845] <2> logconnections: BPDBM CONNECT FROM 10.4.22.55.47037 TO 10.4.22.59.13724
17:12:58.300 [11845] <2> bpbrm kill_child_process: start
17:12:58.300 [11845] <2> bpbrm Exit: attempting to send mail to root on dsmdc-mail-mbs3
17:12:58.300 [11845] <2> hosts_equal: Comparing hosts <dsmdc-mail-mbs3> and <dsmdc-mail-mbs3>
17:12:58.300 [11845] <2> hosts_equal: names are the same

17:12:59.849 [11854] <2> decode_fim_attr: decode_fim_attr: fim=ShadowImage:alternate_backup_client=dsmdc-bkp-med1,sync_at_start=0,keep_fi=0,timeout=240,async_resync=0, client version=650000
17:12:59.849 [11854] <2> decode_fim_attr: decode_fim_attr: mapped fim: ShadowImage:sync_at_start=0,timeout=240, rfi_backup 1, keep_fi 0
17:12:59.849 [11854] <2> bpbrm Exit: backup_status: 156
17:12:59.849 [11854] <2> bpbrm start_bpcd_stat: bpbrm.c.19676: bpcd_client: dsmdc-mail-mbs3
17:12:59.849 [11854] <2> bpbrm start_bpcd_stat: bpbrm.c.19679: bpcd_client_hostname: dsmdc-mail-mbs3
17:12:59.849 [11854] <2> hosts_equal: Comparing hosts <dsmdc-mail-mbs3> and <dsmdc-mail-mbs3>
17:12:59.849 [11854] <2> hosts_equal: names are the same
17:12:59.868 [11854] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
17:12:59.868 [11854] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bpcd
17:12:59.909 [11854] <2> logconnections: BPCD CONNECT FROM 10.4.22.55.54700 TO 10.4.20.144.13724
17:12:59.909 [11854] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.182: msg: VNETD CONNECT FROM 10.4.22.55.41407 TO 10.4.20.144.13724 fd = 6
17:12:59.926 [11854] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.535: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
17:12:59.968 [11854] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.552: ipc_string: /tmp/vnet-11429393974779967209000000000-iQ0xmU
17:13:00.051 [11854] <2> bpcr_get_hostname_rqst: Server hostname length = 26
17:13:00.051 [11854] <2> bpbrm start_bpcd_stat: DATA_SOCK from bpcr = 4
17:13:00.051 [11854] <2> bpbrm start_bpcd_stat: NAME_SOCK from bpcr = 6
17:13:00.052 [11854] <2> pfi_start_client: command = /usr/openv/netbackup/bin/bpfis bpfis delete -nbu -id dsmdc-mail-mbs3_1393974591 -bpstart_to 300 -bpend_to 300 -clnt dsmdc-mail-mbs3 -S dsmdc-bkp-
mst-cls
17:13:00.052 [11854] <2> pfi_start_client: received bpcd success message
17:13:00.068 [11854] <2> pfi_start_client: read start message from dsmdc-mail-mbs3, msg=<INF - BACKUP START 11431>
17:13:00.068 [11854] <2> pfi_start_client: client_pid=11431
17:13:00.068 [11854] <2> get_bpfis_msg: bpfis buff: FTL - cannot open /usr/openv/netbackup/online_util/fi_cntl/bpfis.fim.dsmdc-mail-mbs3_1393974591.0
17:13:00.068 [11854] <32> get_bpfis_msg: from client dsmdc-mail-mbs3: FTL - cannot open /usr/openv/netbackup/online_util/fi_cntl/bpfis.fim.dsmdc-mail-mbs3_1393974591.0

 

 

 

 

 

 

7.1.x and Earlier
Agents
Backup and Recovery
Basics
Configuring
Error messages
Evaluating
Linux
NetBackup
Rules
  • Marianne's avatar
    Marianne
    11 years ago

    No. For array snapshot methods, there is a piece of software that is installed on the client and the media server. This software contains all the utilities and commands that needs to be used to perform checks, snapshots, break-off of mirror, etc. 
    These utilities normally have a username and password that needs to be supplied during initial NBU config (and when anything has changed). Detailed steps for each supported array type is in the Snapshot Client manual.

    As I've said above, I am not familiar with HDS tools, but I have assisted with similar configs with EMC and HP array clones/mirrors.

    The Snapshot Client manual contains steps such as:
    Steps to be performed by storage admin.
    Steps to be performed by NBU admin.

    Please go through steps in the manual as if it is a new config to verify setup.

  • Marianne's avatar
    Marianne
    Level 6
    11 years ago

    You forgot to tell us NBU version on the client?

    Any reason why media sever is still on NBU 6.5? 
    Are you aware of the fact that NBU 6.x ran out of support in 2012?
    Why not upgrade? This will at least enable you to log a Support call should our efforts to assist fail.

    Have you tested the process using the HDS tools?

    I see this in bpfis log:

    vfm_configure_fi_one: read credentials file failed

    and what you have also seen.

     set_devinfo: Could not find targ info for device /dev/sdad

    So, my suggestion is to manually test the process that NBU needs to automate.

    And to upgrade media server and client....

  • Corruptcache's avatar
    Corruptcache
    Level 3
    11 years ago

    Marianne ,

    Clients are running NBU 6.5.6 , Upgrade to 7.5 is scheduled in july this year.

    The HDS process involving pairsplit, resync and VXVM import, deport were tested on one of the clients and completed successfully, so were the automated  backups with no issues on current versions.  

    I have few questions before manual testing of process, disk group which gets imported by netbackup are usually named dgname_clone , while manual "vxdg import dgname" imports  the disk group with out  _clone , can this cause an error. I have seen 156 error if a disk group is already imported on few occasions.

    Correct me if i am wrong,

    from : Could not find targ info for device /dev/sdad.

    sdad is the parent device on client which has a clone device mapped onto media server, is could not find taget info is stating that it cannot establish a communication with clone ?

    or

    can it be an issue with internal pathing of lun on primary host/client .

    reason being is that the media servers were rebooted , clone luns were discovered, were been validated fstyp as vxfs , imported and test mounted sucessfully.

     

     

     

     

  • Marianne's avatar
    Marianne
    Level 6
    11 years ago

    You will know what belongs where and what the device relationships are...
    Have you double-checked mount points and device names after the reboot?

    And commands such as vxdisk list?

    I see 'could not find targ info..' coming after the credentials error. Have you double-checked credentials provided in NBU?

    In all honesty, I am not familiar with the HDS cloning and snapshot procedures, but when I need to assist with such a config, I make sure the relative resources with the know-how are available and can assist with manual testing.

    After testing manually, undo the process again to deport and re-establish mirror on the client.
    Only test with NBU if the clones/mirrors are in sync.

    Any reason why you cannot use VxVM FlashSnap for the entire process? 
    At least a familiar process from beginning to end.

     

     

  • Corruptcache's avatar
    Corruptcache
    Level 3
    11 years ago

    Providing credentials referring to bp.conf on client , media server or master ?

  • Marianne's avatar
    Marianne
    Level 6
    11 years ago

    No. For array snapshot methods, there is a piece of software that is installed on the client and the media server. This software contains all the utilities and commands that needs to be used to perform checks, snapshots, break-off of mirror, etc. 
    These utilities normally have a username and password that needs to be supplied during initial NBU config (and when anything has changed). Detailed steps for each supported array type is in the Snapshot Client manual.

    As I've said above, I am not familiar with HDS tools, but I have assisted with similar configs with EMC and HP array clones/mirrors.

    The Snapshot Client manual contains steps such as:
    Steps to be performed by storage admin.
    Steps to be performed by NBU admin.

    Please go through steps in the manual as if it is a new config to verify setup.

  • Corruptcache's avatar
    Corruptcache
    Level 3
    11 years ago

    Marianne ,

    nbfirescan  from SNP client manual revealed the reason for the error i was facing.

    nbfirescan wasnt able to recognise the Luns which are supposed to be backed up, reason beiing is that those voulmes were configured with diabled path with a flag of STD on them, these luns were recovered from hosts, reconfigured with enable status with PVOL attribute (vxdisk list ) .

    nbfirescan was now able to detect + No issues with backups.

    Thank you again for directing in right path.