Forum Discussion

bibelo's avatar
bibelo
Level 2
9 years ago

I/O error on vxfs volume

Hello everyone,

Please I need help regarding a volume I cannot mount. It is 2 Symmetrix EMC SAN meta (emcpower), forming one dg (supposedly clonedg), divided into vol01 and vol02. Each day, the metas are cloned from the principal server and mounted, but yesterday we had this issue :

/dev/vx/dsk/clonedg/vol01  460G  379G   77G  84% /clone/hydra/oracle

df: `/clone/hydra/ora11g': Input/output error

(for /dev/vx/dsk/clonedg/vol02)

Note that vol01 works perfectly, but not 02.

Last time it happened, it did a umount -l and waited for the next day for the clone and it worked.

 

So I umounted it again and then tried to clone and remount, but this time, it does not work.

I then tried to deport the dg, in the hope to import it, but no luck :

# vxdg -C import clonedg
VxVM vxdg ERROR V-5-1-10978 Disk group clonedg: import failed:
No valid disk found containing disk group

# vxdg -f import clonedg
VxVM vxdg ERROR V-5-1-10978 Disk group clonedg: import failed:
No valid disk found containing disk group

I also did :

vxdctl initdmp
vxdisk scandisks

 

and now I have this :

# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
cciss/c0d0   auto:none       -            -            online invalid
emcpowera    auto:cdsdisk    -            -            error
emcpowera    auto            -            -            error
emcpowera    auto            -            -            error
emcpowera    auto            -            -            error
emcpowera    auto            -            -            error
emcpowera    auto            -            -            error
emcpowerb    auto:cdsdisk    -            -            error
emcpowerb    auto            -            -            error
emcpowerb    auto            -            -            error
emcpowerb    auto            -            -            error
emcpowerb    auto            -            -            error
emcpowerb    auto            -            -            error
emcpowerb    auto            -            -            error
emcpowerc    auto            -            -            error
emcpowerc    auto            -            -            error
emcpowerc    auto:cdsdisk    -            -            error
emcpowerc    auto            -            -            error
emcpowerc    auto            -            -            error
emcpowerc    auto            -            -            error
emcpowerd    auto:cdsdisk    -            -            online udid_mismatch
emcpowerd    auto            -            -            error
emcpowere    auto:cdsdisk    -            -            online udid_mismatch
emcpowere    auto            -            -            error
emcpowerf    auto            -            -            error
emcpowerf    auto:cdsdisk    -            -            online
emcpowerf    auto            -            -            error
emcpowerf    auto            -            -            error
emcpowerf    auto            -            -            error
emcpowerf    auto            -            -            error
emcpowerg    auto:LVM        -            -            LVM

 

Note that many volume are in error, and they are duplicated. 

 

From here, I don't know what to do, since I'm not expert in vxfs.

 

so any help would be appreciated!

 

Thank you.

  • I forgot to mention that since the dg is not recognized and cannot be imported, now I cannot even mount vol01.

  • What are the outputs of

    # vxdmpadm listctlr all

    # vxdisk list emcpowera

    # vxdmpadm listenclosure all

    # modinfo |grep -i vx

     

    G

     

  • Hello G. Thank you for your answer.

     

    I am writing you from a phone connected to my workstation via TViewer...

     

    Here is the result of the commands :

     

    vxdmpadm listctlr all
    CTLR-NAME       ENCLR-TYPE      STATE      ENCLR-NAME
    =====================================================
    c360            OTHER_DISKS     ENABLED      other_disks
    emcp            PP_EMC          ENABLED      pp_emc0
    [root@miranda log]#
    
    vxdisk list emcpowera
    Device:    emcpowera
    devicetag: emcpowera
    type:      auto
    flags:     error private autoconfig
    pubpaths:  block=/dev/vx/dmp/emcpowera char=/dev/vx/rdmp/emcpowera
    guid:      {37e4d0f2-fff2-11e0-9100-459b072bef83}
    udid:      EMC%5FSYMMETRIX%5F000290103833%5F334D5000
    site:      -
    errno:     Duplicate device
    Multipathing information:
    numpaths:   1
    emcpowera       state=enabled
    [root@miranda log]#
    
    modinfo vxfs
    filename:       /lib/modules/2.6.18-274.el5/veritas/vxfs/vxfs.ko
    version:        5.1.00.00
    description:    Veritas File System     Send bug reports to support@veritas.com
    author:         Symantec Corporation
    license:        Proprietary
    srcversion:     5A5F73DB92E11B69B7574C9
    depends:
    vermagic:       2.6.18-8.el5 SMP mod_unload gcc-4.1
    parm:           vxfs_ninode:Maximum number of vxfs inodes (int)
    parm:           vx_ifree_timelag:Seconds before scanning inode free list (long)
    parm:           vxfs_mbuf:Maximum memory used for vxfs buffer cache (charp)
    parm:           vx_dexh_sz:Large directory hashing threshold (uint)
    parm:           vxfs_maxdiospace:Maximum memory wired down for direct IO (charp)
    parm:           vxfs_mindiospace:Minimum memory wired down for direct IO (charp)
    parm:           vxfs_pmem:Physical memory to use as input for tuning curves (charp)
    parm:           vxfs_icachechunk:Maximum number of pages for each memory chunk used for VxFS inodes (int)
    parm:           vx_fiostats_tunable:Memory used for file I/O stats (int)
    parm:           vx_fiostats_winterval:FCL winterval for file I/O stats (int)
    parm:           vxfs_start_node_id:Highest node ID - override for low memory scanning (int)
    parm:           vxfs_naio_threads_max:Maximum number of threads for native AIO (uint)
    parm:           vxfs_naio_threads_min:Minimum number of threads for native AIO (uint)
    parm:           vxfs_naio_plug_enable:Throttle native AIO on memory usage (uint)
    parm:           vx_maxlink:Maximum number of hardlinks (uint)
    parm:           vxfs_sendfile_chunk:Number of pages to fetch for large sendfile transfers (uint)
    parm:           vxfs_sendfile_limit:Maximum number of processes in a sendfile main loop (uint)
    parm:           vxfs_sendfile_nfs:Setting to non-zero changes VxFS behaviour with sendfile from NFS (uint)
    parm:           vxfs_io_proxy_vxvm:Include VxVM devices in IO hand-off decisions (int)
    parm:           vxfs_io_proxy_level:Hand-off IO to worker threads for `raw` block devices when less than this number of bytes free on kernel stack (int)
    parm:           vxfs_io_proxy_pools:Number of IO pools for hand-off. (int)
    parm:           vxfs_io_proxy_time:If an IO has been waiting on an IO queue for this number of clock ticks, then create a new IO hand-off thread (int)
    parm:           vxfs_io_proxy_max:Maximum number of threads per IO handoff queue (int)
    
    # vxdmpadm listenclosure all
    
    vxdmpadm: Cannot allocate memory

    The last command actually hanged for quite a bit.

  • Hello G,

     

    I finally solved the problem :

    1) rebooted the server

    2) 

    vxdctl initdmp

    vxdisk scandisks

    3) 

    vxconfigrestore -p

    vxconfigrestore -c

    4) mount the FS

    The config looks clean, hope it'll stay that way :)