Disk Volume Pool Down (2074)
Friends,
given up on fixing disk pool volume down issue after spending whole morning.. this is the typical one..
nbdevquery and nbdevconfig works successfully.. you can inventory the disk pool from.. you can see the storage configuration properties under credentials tab.. disk pool stays alive less than 10secs and it goes down.. here is the spoold logs from that path.
gsrugnbapp01:/dedup_data/log/spad/gsrugnbapp01.groupservices.co.uk/spoold/spad
number of errors being reported of connection reset by peer.. i can bet my brand new BMW 325D on this.. the name resolution/nslookup and everything working okay.. no problem.. this problem arose over the weeked due where the media server (built in storage) went offline in the middle of the backup due to unexpected power outage.. looking at the logs below its suggesting connection reset by peer; REceive session shutdown
any suggestion will highly be appreciated.
February 25 13:49:19 INFO [140449875887872]: Task Type : spad
February 25 13:49:19 INFO [140449875887872]: Started at : Mon Feb 25 13:49:19 2013
February 25 13:49:19 INFO [140449875887872]: Task ID : 0
February 25 13:49:19 INFO [140449875887872]: Thread ID : 140449875887872
February 25 13:49:19 INFO [140449875887872]: Session ID : 1563389238
February 25 13:49:19 INFO [140449875887872]: Data Selection : 1
February 25 13:49:19 INFO [140449875887872]: Agent Name : spoold
February 25 13:49:19 INFO [140449875887872]: Agent Address : gsrugnbapp01.groupservices.co.uk:39478
February 25 13:49:19 INFO [140449875887872]: OS Family : Linux-x86_64
February 25 13:49:19 INFO [140449875887872]: Software Version : 7.0003.0012.091
February 25 13:49:19 INFO [140449875887872]: Protocol Version : 6.6.1
February 25 13:49:19 INFO [140449875887872]: Crypto engine : server
February 25 13:49:19 INFO [140449875887872]: Data Encryption : enabled
February 25 13:49:19 INFO [140449875887872]: Data Compression : enabled
February 25 13:49:19 INFO [140449875887872]: Accelerated Access: disabled
February 25 13:49:19 INFO [140449875887872]: task 0 [thread 140449875887872] for gsrugnbapp01.groupservices.co.uk:39478: spad handler running
February 25 13:49:48 INFO [140449875887872]: newEvent Adding new event. type:4
February 25 13:50:14 INFO [140449875887872]: Task Type : spad
February 25 13:50:14 INFO [140449875887872]: Started at : Mon Feb 25 13:50:14 2013
February 25 13:50:14 INFO [140449875887872]: Task ID : 0
February 25 13:50:14 INFO [140449875887872]: Thread ID : 140449875887872
February 25 13:50:14 INFO [140449875887872]: Session ID : 1393531371
February 25 13:50:14 INFO [140449875887872]: Data Selection : 1
February 25 13:50:14 INFO [140449875887872]: Agent Name : spoold
February 25 13:50:14 INFO [140449875887872]: Agent Address : gsrugnbapp01.groupservices.co.uk:39494
February 25 13:50:14 INFO [140449875887872]: OS Family : Linux-x86_64
February 25 13:50:14 INFO [140449875887872]: Software Version : 7.0003.0012.091
February 25 13:50:14 INFO [140449875887872]: Protocol Version : 6.6.1
February 25 13:50:14 INFO [140449875887872]: Crypto engine : server
February 25 13:50:14 INFO [140449875887872]: Data Encryption : enabled
February 25 13:50:14 INFO [140449875887872]: Data Compression : enabled
February 25 13:50:14 INFO [140449875887872]: Accelerated Access: disabled
February 25 13:50:14 INFO [140449875887872]: task 1 [thread 140449875887872] for gsrugnbapp01.groupservices.co.uk:39494: spad handler running
February 25 13:50:32 INFO [140449875887872]: newEvent Adding new event. type:1
February 25 13:50:46 INFO [140449875887872]: newEvent Adding new event. type:1
February 25 13:53:08 INFO [140449875887872]: newEvent Adding new event. type:1
February 25 13:53:08 ERR [140449875887872]: 25000: NetSelect: invalid pipe
February 25 13:53:08 ERR [140449875887872]: 25036: binaryMessageRead:3398, Could not receive binary message: expected 4 bytes got 0 bytes. Cause: connection reset by peer
February 25 13:53:08 ERR [140449875887872]: 25036: binaryMessageRead:3398, Could not receive binary message: expected 4 bytes got -1 bytes. Cause: connection reset by peer
February 25 13:53:08 INFO [140449875887872]: task 1 [thread 140449875887872] for gsrugnbapp01.groupservices.co.uk:39494: spad handler returns
February 25 13:53:08 INFO [140449875887872]: task 0 [thread 140449875887872] for gsrugnbapp01.groupservices.co.uk:39478: spad handler returns
February 25 13:53:08 INFO [140449875887872]: Receive session shutdown
February 25 13:53:08 INFO [140449875887872]: Receive session shutdown
February 25 13:53:08 INFO [140449875887872]: Uptime : 173.54 sec
February 25 13:53:08 INFO [140449875887872]: Uptime : 228.61 sec
February 25 13:53:08 INFO [140449875887872]: Messages Received : 19
February 25 13:53:08 INFO [140449875887872]: Messages Received : 8
February 25 13:53:08 INFO [140449875887872]: Bytes Received : 3996
February 25 13:53:08 INFO [140449875887872]: Bytes Received : 2104
February 25 13:53:08 INFO [140449875887872]: Seconds spent receiving : 0.00
February 25 13:53:08 INFO [140449875887872]: Seconds spent receiving : 0.00
February 25 13:53:08 INFO [140449875887872]: Messages Sent : 19
February 25 13:53:08 INFO [140449875887872]: Messages Sent : 8
February 25 13:53:08 INFO [140449875887872]: Bytes Sent : 1820
February 25 13:53:08 INFO [140449875887872]: Bytes Sent : 744
February 25 13:53:08 INFO [140449875887872]: Seconds spent sending : 0.00
February 25 13:53:08 INFO [140449875887872]: Seconds spent sending : 0.00
February 25 13:53:08 INFO [140449875887872]: 0.00 MB sent in 173.54 sec
February 25 13:53:08 INFO [140449875887872]: 0.00 MB sent in 228.61 sec
February 25 13:53:08 INFO [140449875887872]: spad completed at: Mon Feb 25 13:53:07 2013
February 25 13:53:08 INFO [140449875887872]: spad completed at: Mon Feb 25 13:53:07 2013
February 25 13:53:43 INFO [140449875887872]: Task Type : spad
February 25 13:53:43 INFO [140449875887872]: Started at : Mon Feb 25 13:53:43 2013
February 25 13:53:43 INFO [140449875887872]: Task ID : 0
February 25 13:53:43 INFO [140449875887872]: Thread ID : 140449875887872
February 25 13:53:43 INFO [140449875887872]: Session ID : 291392991
February 25 13:53:43 INFO [140449875887872]: Data Selection : 1
February 25 13:53:43 INFO [140449875887872]: Agent Name : spoold
February 25 13:53:43 INFO [140449875887872]: Agent Address : gsrugnbapp01.groupservices.co.uk:39541
February 25 13:53:43 INFO [140449875887872]: OS Family : Linux-x86_64
February 25 13:53:43 INFO [140449875887872]: Software Version : 7.0003.0012.091
February 25 13:53:43 INFO [140449875887872]: Protocol Version : 6.6.1
February 25 13:53:43 INFO [140449875887872]: Crypto engine : server
February 25 13:53:43 INFO [140449875887872]: Data Encryption : enabled
February 25 13:53:43 INFO [140449875887872]: Data Compression : enabled
February 25 13:53:43 INFO [140449875887872]: Accelerated Access: disabled
February 25 13:53:43 INFO [140449875887872]: task 0 [thread 140449875887872] for gsrugnbapp01.groupservices.co.uk:39541: spad handler running
February 25 13:54:17 INFO [140449875887872]: newEvent Adding new event. type:4
February 25 13:54:45 INFO [140449875887872]: Task Type : spad
February 25 13:54:45 INFO [140449875887872]: Started at : Mon Feb 25 13:54:45 2013
February 25 13:54:45 INFO [140449875887872]: Task ID : 0
February 25 13:54:45 INFO [140449875887872]: Thread ID : 140449875887872
February 25 13:54:45 INFO [140449875887872]: Session ID : 1820742354
February 25 13:54:45 INFO [140449875887872]: Data Selection : 1
February 25 13:54:45 INFO [140449875887872]: Agent Name : spoold
February 25 13:54:45 INFO [140449875887872]: Agent Address : gsrugnbapp01.groupservices.co.uk:39553
February 25 13:54:45 INFO [140449875887872]: OS Family : Linux-x86_64
February 25 13:54:45 INFO [140449875887872]: Software Version : 7.0003.0012.091
February 25 13:54:45 INFO [140449875887872]: Protocol Version : 6.6.1
February 25 13:54:45 INFO [140449875887872]: Crypto engine : server
February 25 13:54:45 INFO [140449875887872]: Data Encryption : enabled
February 25 13:54:45 INFO [140449875887872]: Data Compression : enabled
February 25 13:54:45 INFO [140449875887872]: Accelerated Access: disabled
February 25 13:54:45 INFO [140449875887872]: task 9 [thread 140449875887872] for gsrugnbapp01.groupservices.co.uk:39553: spad handler running
February 25 13:56:23 INFO [140449875887872]: newEvent Adding new event. type:1
February 25 13:57:42 ERR [140449875887872]: 25036: binaryMessageRead:3398, Could not receive binary message: expected 4 bytes got 0 bytes. Cause: connection reset by peer
February 25 13:57:42 INFO [140449875887872]: task 0 [thread 140449875887872] for gsrugnbapp01.groupservices.co.uk:39541: spad handler returns
February 25 13:57:42 INFO [140449875887872]: Receive session shutdown
February 25 13:57:42 INFO [140449875887872]: Uptime : 238.56 sec
February 25 13:57:42 INFO [140449875887872]: Messages Received : 6
Applied the EEB last night only on to effected media servers (3 of 22) and all stable..
didnt apply the EEB on rest of them as they have not broken yet.. fingers crossed!!
Solution:
Bug: http://www.symantec.com/business/support/index?page=content&id=TECH199067
Fix: http://www.symantec.com/business/support/index?page=content&id=TECH199269
