05-17-2012 06:00 AM
Hi,
Dedup and Advanced Disks are lost after one PSU of disk shelf failed, backups stopped. Could you find what is the root cause of problem and what can we do to not repeat problem?
Note: After I relazied problem, I halted all disk shelfs and controller, after reboot it continue to work.
I found problem starting time from /var/log/messages
May 15 05:07:49 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:49 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 9401599128
May 15 05:07:49 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:49 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 3210993496
May 15 05:07:53 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:53 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 4157743456
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 37802501408
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 4793729440
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 38083998622
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 76167526144
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 2499793520
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 9136886608
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 1226160
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 69946701648
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 428528
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 428464
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 428400
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 428336
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 76167561088
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 42882368008
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 13527093248
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 207401808
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 210330384
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 4158071712
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 3085736416
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 8671234304
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000
May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 37762134960
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: megasas: RESET -4302543 cmd=12 retries=0
May 15 05:07:59 bhmdpbmedp1 kernel: megaraid_sas: HBA reset handler invoked without an internal reset condition.
May 15 05:07:59 bhmdpbmedp1 kernel: megaraid_sas: no more pending commands remain after reset handling.
May 15 05:07:59 bhmdpbmedp1 kernel: megasas: reset successful
May 15 05:07:59 bhmdpbmedp1 kernel:
May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: megasas: RESET -4302543 cmd=12 retries=0
May 15 05:07:59 bhmdpbmedp1 kernel: megaraid_sas: HBA reset handler invoked without an internal reset condition.
May 15 05:07:59 bhmdpbmedp1 kernel: megaraid_sas: no more pending commands remain after reset handling.
May 15 05:07:59 bhmdpbmedp1 kernel: megasas: reset successful
May 15 05:07:59 bhmdpbmedp1 kernel:
May 15 05:07:59 bhmdpbmedp1 kernel: megasas_aen_polling[1]: scanning ...
May 15 05:07:59 bhmdpbmedp1 kernel: Synchronizing SCSI cache for disk sdb:
May 15 05:07:59 bhmdpbmedp1 kernel: megasas_aen_polling[1]: scanning ...
May 15 05:07:59 bhmdpbmedp1 kernel: megasas_aen_polling[1]: scanning ...
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 vxvm:vxconfigd: V-5-1-7909 Detached disk disk_2
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-112 disabled path 8/0x10 belonging to the dmpnode 201/0x20 due to path failure
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-111 disabled dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c59113e00 on device 0xc900020 to DMP
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 74085274704: Uncorrectable write error<4>VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 3210927448: Uncorrectable read error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 71252961696: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 1 mesg 038: V-2-38: vx_dataioerr - /dev/vx/dsk/nbuapp/pdvol file system file data write error in dev/block 0/4453441178
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 9401533080: Uncorrectable read error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 4157677408: Uncorrectable read error
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 2 mesg 037: V-2-37: vx_metaioerr - vx_bufiodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/1072927824
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 4793663392: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c59ebf600 on device 0xc900020 to DMP<4>vxfs: msgcnt 3 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/500793248
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 75882109528: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 37802435360: Uncorrectable read error<3> 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c59113000 on device 0xc900020 to DMP
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-1 block 1076400: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810656c3ee00 on device 0xc900020 to DMP
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-1 block 1112976: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c59113c00 on device 0xc900020 to DMP
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-2 block 25584: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c5a28ce00 on device 0xc900020 to DMP
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-2 block 44272: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810656c3ec00 on device 0xc900020 to DMP<4>VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 38083932574: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-2 block 438800: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c58883200 on device 0xc900020 to DMP<4>VxVM vxio V-5-0-1266 Subdisk disk_2-1 block 51840: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-2 block 494672: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 2499727472: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 4 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/2501824624
May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 5 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/2501824624
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c59d41a00 on device 0xc900020 to DMP<4>VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 9136820560: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 6 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/548983120
May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 7 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/548983120
May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 8 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/548983120
May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 9 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/548983120
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 1160112: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-1 block 16: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c59d41800 on device 0xc900020 to DMP
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 4157677408: Uncorrectable read error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff81064cd45000 on device 0xc900020 to DMP
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 37802435360: Uncorrectable read error<4>VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 69946635600: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 10 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/nbuapp/pdvol file system super-block write error
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 11 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/1229256016
May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 12 mesg 037: V-2-37: vx_metaioerr - vx_delbuf_flush - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/74087423808
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 362480: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 362416: Uncorrectable write error<4>vxfs: msgcnt 13 mesg 031: V-2-31: vx_disable - /dev/vx/dsk/nbuapp/pdvol file system disabled
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c59d41c00 on device 0xc900020 to DMP
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 38083932578: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 362352: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 362288: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-1 block 86784: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 42882301960: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 13527027200: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 207335760: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 210264336: Uncorrectable write error<3> 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 14 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/212361488
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff8106580af200 on device 0xc900020 to DMP<4>VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 4158005664: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 15 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/4160102816
May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 16 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/4160102816
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 3085670368: Uncorrectable read error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: 1:2:0:0: rejecting I/O to offline device
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 8671168256: Uncorrectable write error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff8106580af200 on device 0xc900020 to DMP<4>vxfs: msgcnt 17 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/83330816
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 37762068912: Uncorrectable read error
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20
May 15 05:08:09 bhmdpbmedp1 kernel:
May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 37762068944: U
05-18-2012 01:47 PM
Interesting. Most of the POC's I do I actually pull a power supply to show the 2N redundancy of the power and never seen the box do this. Do you have a trouble ticket for this?