cancel
Showing results for 
Search instead for 
Did you mean: 

Dedup and Advanced Disks are lost after one PSU of disk shelf failed

ayazici
Level 2

Hi,

Dedup and Advanced Disks are lost after one PSU of disk shelf failed, backups stopped. Could you find what is the root cause of problem and what can we do to not repeat problem?

Note: After I relazied problem, I halted all disk shelfs and controller, after reboot it continue to work.

I found problem starting time from /var/log/messages

 

May 15 05:07:49 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:49 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 9401599128

May 15 05:07:49 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:49 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 3210993496

May 15 05:07:53 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:53 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 4157743456

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 37802501408

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 4793729440

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 38083998622

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 76167526144

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 2499793520

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 9136886608

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 1226160

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 69946701648

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 428528

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 428464

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 428400

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 428336

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 76167561088

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 42882368008

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 13527093248

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 207401808

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 210330384

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 4158071712

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 3085736416

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 8671234304

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: SCSI error: return code = 0x00040000

May 15 05:07:59 bhmdpbmedp1 kernel: end_request: I/O error, dev sdb, sector 37762134960

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: megasas: RESET -4302543 cmd=12 retries=0

May 15 05:07:59 bhmdpbmedp1 kernel: megaraid_sas: HBA reset handler invoked without an internal reset condition.

May 15 05:07:59 bhmdpbmedp1 kernel: megaraid_sas: no more pending commands remain after reset handling.

May 15 05:07:59 bhmdpbmedp1 kernel: megasas: reset successful

May 15 05:07:59 bhmdpbmedp1 kernel:

May 15 05:07:59 bhmdpbmedp1 kernel: sd 1:2:0:0: megasas: RESET -4302543 cmd=12 retries=0

May 15 05:07:59 bhmdpbmedp1 kernel: megaraid_sas: HBA reset handler invoked without an internal reset condition.

May 15 05:07:59 bhmdpbmedp1 kernel: megaraid_sas: no more pending commands remain after reset handling.

May 15 05:07:59 bhmdpbmedp1 kernel: megasas: reset successful

May 15 05:07:59 bhmdpbmedp1 kernel:

May 15 05:07:59 bhmdpbmedp1 kernel: megasas_aen_polling[1]: scanning ...

May 15 05:07:59 bhmdpbmedp1 kernel: Synchronizing SCSI cache for disk sdb:

May 15 05:07:59 bhmdpbmedp1 kernel: megasas_aen_polling[1]: scanning ...

May 15 05:07:59 bhmdpbmedp1 kernel: megasas_aen_polling[1]: scanning ...

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 vxvm:vxconfigd: V-5-1-7909 Detached disk disk_2

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-112 disabled path 8/0x10 belonging to the dmpnode 201/0x20 due to path failure

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-111 disabled dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c59113e00 on device 0xc900020 to DMP

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 74085274704: Uncorrectable write error<4>VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 3210927448: Uncorrectable read error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 71252961696: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 1 mesg 038: V-2-38: vx_dataioerr - /dev/vx/dsk/nbuapp/pdvol file system file data write error in dev/block 0/4453441178

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 9401533080: Uncorrectable read error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 4157677408: Uncorrectable read error

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 2 mesg 037: V-2-37: vx_metaioerr - vx_bufiodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/1072927824

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 4793663392: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c59ebf600 on device 0xc900020 to DMP<4>vxfs: msgcnt 3 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/500793248

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 75882109528: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 37802435360: Uncorrectable read error<3> 1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c59113000 on device 0xc900020 to DMP

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-1 block 1076400: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810656c3ee00 on device 0xc900020 to DMP

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-1 block 1112976: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c59113c00 on device 0xc900020 to DMP

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-2 block 25584: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c5a28ce00 on device 0xc900020 to DMP

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-2 block 44272: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810656c3ec00 on device 0xc900020 to DMP<4>VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 38083932574: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-2 block 438800: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c58883200 on device 0xc900020 to DMP<4>VxVM vxio V-5-0-1266 Subdisk disk_2-1 block 51840: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-2 block 494672: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 2499727472: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 4 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/2501824624

May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 5 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/2501824624

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c59d41a00 on device 0xc900020 to DMP<4>VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 9136820560: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 6 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/548983120

May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 7 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/548983120

May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 8 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/548983120

May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 9 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/548983120

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 1160112: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-1 block 16: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c59d41800 on device 0xc900020 to DMP

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 4157677408: Uncorrectable read error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff81064cd45000 on device 0xc900020 to DMP

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 37802435360: Uncorrectable read error<4>VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 69946635600: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 10 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/nbuapp/pdvol file system super-block write error

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 11 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/1229256016

May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 12 mesg 037: V-2-37: vx_metaioerr - vx_delbuf_flush - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/74087423808

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 362480: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 362416: Uncorrectable write error<4>vxfs: msgcnt 13 mesg 031: V-2-31: vx_disable - /dev/vx/dsk/nbuapp/pdvol file system disabled

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff810c59d41c00 on device 0xc900020 to DMP

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 38083932578: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 362352: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 362288: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-1 block 86784: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 42882301960: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 13527027200: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 207335760: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 210264336: Uncorrectable write error<3> 1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 14 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/212361488

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff8106580af200 on device 0xc900020 to DMP<4>VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 4158005664: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 15 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/4160102816

May 15 05:08:09 bhmdpbmedp1 kernel: vxfs: msgcnt 16 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/4160102816

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 3085670368: Uncorrectable read error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel:  1:2:0:0: rejecting I/O to offline device

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 8671168256: Uncorrectable write error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-3-1285 voldmp_errbuf_sio_start: Failed to flush the error buffer ffff8106580af200 on device 0xc900020 to DMP<4>vxfs: msgcnt 17 mesg 037: V-2-37: vx_metaioerr - vx_inode_iodone - /dev/vx/dsk/nbuapp/pdvol file system meta data write error in dev/block 0/83330816

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 37762068912: Uncorrectable read error

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 201/0x20

May 15 05:08:09 bhmdpbmedp1 kernel:

May 15 05:08:09 bhmdpbmedp1 kernel: VxVM vxio V-5-0-1266 Subdisk disk_2-01 block 37762068944: U

 



 

1 REPLY 1

Chad_Wansing2
Level 5
Employee Accredited Certified

Interesting.  Most of the POC's I do I actually pull a power supply to show the 2N redundancy of the power and never seen the box do this.  Do you have a trouble ticket for this?