05-18-2015 03:02 AM
Friends,
I have got strange problem going where one of my media server keep on rebooting mostly over the weekend.
OS SuSe Linux 11 SP2
NBU 7.6.0.3
Physical box HP DL180 G6
Its an onsite appliance that backs up client locally and replicate (AIR) to central location.
/var/log/messages doesn't contain much information..
I have ran the HP diagnostic to see if there are any problem with hardware componenets but all came out.
I have upgraded the FW and number of drivers and due to upgrade BIOS but i dont think abnormal reboot has anything to do with BIOS.
Has anyone seen this problem. any input/ideas will highly be appreicated otherwise i would have no choice except to rebuilt the server.
Solved! Go to Solution.
05-19-2015 02:35 AM
Following the updates of BIOS, Firmware and various drivers the firmware error appears to have been disappeared...
05-18-2015 03:16 AM
.... /var/log/messages doesn't contain much information......
Curious to know what is logged in messages shortly before reboot.
05-18-2015 03:19 AM
thanks marianne..
May 16 22:03:39 gstamnbapp01 syslog-ng[1552]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=554', processed='center(received)=524', processed='desti
nation(messages)=518', processed='destination(mailinfo)=2', processed='destination(mailwarn)=0', processed='destination(localmessages)=3', processed='destination(newserr)=0', processed='destination(mailerr)=
0', processed='destination(netmgm)=0', processed='destination(warn)=11', processed='destination(console)=7', processed='destination(null)=2', processed='destination(mail)=2', processed='destination(xconsole)
=7', processed='destination(firewall)=0', processed='destination(acpid)=2', processed='destination(newscrit)=0', processed='destination(newsnotice)=0', processed='source(src)=524'
May 16 22:06:43 gstamnbapp01 syslog-ng[1558]: syslog-ng starting up; version='2.0.9'
May 16 22:06:43 gstamnbapp01 firmware.sh[1586]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:43 gstamnbapp01 firmware.sh[1592]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:43 gstamnbapp01 firmware.sh[1603]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1614]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1625]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1636]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1652]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1669]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1682]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1695]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1708]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1721]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1734]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1747]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1760]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1773]: Cannot find firmware file 'intel-ucode/06-2c-02'
May 16 22:06:45 gstamnbapp01 ifup: lo
May 16 22:06:45 gstamnbapp01 ifup: lo
May 16 22:06:45 gstamnbapp01 ifup: IP address: 127.0.0.1/8
May 16 22:06:45 gstamnbapp01 ifup:
May 16 22:06:45 gstamnbapp01 ifup:
May 16 22:06:45 gstamnbapp01 ifup: IP address: 127.0.0.2/8
May 16 22:06:45 gstamnbapp01 ifup:
May 16 22:06:45 gstamnbapp01 ifup: eth0 device: Intel Corporation 82576 Gigabit Network Conne
May 16 22:06:45 gstamnbapp01 ifup: eth0
May 16 22:06:45 gstamnbapp01 ifup: IP address: 10.8.20.65/24
May 16 22:06:45 gstamnbapp01 ifup:
May 16 22:06:45 gstamnbapp01 SuSEfirewall2: SuSEfirewall2 not active
May 16 22:06:46 gstamnbapp01 ifup: eth1 device: Intel Corporation 82576 Gigabit Network Conne
05-18-2015 04:13 AM
what does /va/logs/messages say ?
May be a hardware related issue. Is the load during weekends larger than normal ?
Did you have a look at the ILO console log ?
05-18-2015 05:19 AM
i can see the above logs repeating in /var/log messages -- cannot find firmware file and then system restart as you can see above.
well, with regards to backups they are dedup/optimized synthetic and replication (NBU AIR)
so its a not a backup load or anything..
i have googled the above error SuSe says install the following package which is alreayd installed
https://www.novell.com/support/kb/doc.php?id=7010705
HP says ignore this message but thats for DL380 G7.. mine is DL180 G6
http://h20564.www2.hp.com/hpsc/doc/public/display?docId=emr_na-c03697510
Found this thread where people talking about temprature and fans -
http://h30499.www3.hp.com/t5/ProLiant-Servers-ML-DL-SL/HP-ProLiant-DL180-G6-Server-restart-by-itself/td-p/4777782#.VVnHlflViko
no errors in system event logs on the iLO.
Ran HP diagnostic tools no hardware error found..
Only thing is left is to update the BIOS .. :S
05-18-2015 06:32 AM
BIOS upgrades can do all sort of wonders.
If you open a support ticket with HP they will insist on having the latest firmware revision loaded, so not a bad thing to do.
05-18-2015 06:41 AM
i know thats why am bit reluctant to do the BIOS.
support ticket with HP is already opened and they advised to update BIOS, firmware and whole lot of drivers..
i have updated the firmware but other drivers are asking missing dependencies..
==========================================================================================================================
gstamnbapp01:/export # rpm -ivh hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64.rpm
warning: hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64.rpm: Header V3 RSA/SHA1 signature: NOKEY, key ID 5ce2d476
error: Failed dependencies:
kernel(xen:kernel_timer) = 423e9e985483e429 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:drivers_pci_pci) = 85638a4879aa8d22 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:mm_slab) = 845e11a139410068 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:net_ethernet_eth) = 406b7cb5cf4b32c3 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:arch_x86_kernel_pci-dma) = 94e621eac16acd1c is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:net_core_dev) = 3a6bf572c39cfef2 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:kernel_panic) = 836b6bbe1d0c6009 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:drivers_hwmon_hwmon) = c3b63051fffc528b is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:drivers_pci_msi) = d69d27819ce5716f is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:net_core_netpoll) = b93abb3bba079a91 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:drivers_base_power_runtime) = 0514c1712f4a5433 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:mm_page_alloc) = 915b9881f339020c is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:init_main) = f547105fd821a308 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:arch_x86_kernel_x8664_ksyms_64) = 2af1e5a3b31ce371 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:kernel_module) = 0a94a4e092761946 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:vmlinux) = 67c649353f6f4284 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:kernel_printk) = d1982403e0a2e113 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:net_core_skbuff) = 4c9f9431a9f5f113 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:net_core_sock) = 4a8b3ca6d3876f61 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:drivers_base_power_wakeup) = 6c968d103ae8c86e is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:kernel_params) = 7af15e23aeb0c96a is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:mm_vmalloc) = 49ca512fe4d78100 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:arch_x86_mm_ioremap) = 41e494602b96f119 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:drivers_pci_pci-driver) = 4886485b5e358022 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:arch_x86_kernel_cpu_common) = 6387e2748a649d07 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:lib_bitmap) = 1bd5f25878fbbd41 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:drivers_base_dd) = 6902639516c11d0c is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:kernel_rcutree) = bb9ce712db2a1247 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:drivers_pci_iov) = 53b8b35afc7e5a03 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:drivers_base_core) = a062475ae36e4c3c is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:drivers_pci_search) = a1315fafd5354c6e is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:drivers_pci_pcie_aer_aerdriver) = 5e0a9b219cee4519 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:net_sched_sch_generic) = 54f52bb0ded5130f is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:drivers_pci_access) = 141ba8f9f4b107fe is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:net_core_rtnetlink) = 394ca6becda42c3e is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:drivers_char_random) = 0262f5133940a8f2 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:kernel_time_jiffies) = 0a1119a9d9ca3d67 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:mm_nobootmem) = 2a5b8099b0d91697 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:kernel_workqueue) = 935a39c08f5e6a71 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:mm_swap) = a17f9792b744588a is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:kernel_cpu) = aa72f1964aa3a83a is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:kernel_capability) = d4d1b3113d734c41 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:kernel_irq_manage) = 7749acddf1ec7248 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:arch_x86_mm_physaddr) = 8a91ed3165f0c415 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:mm_memory) = a1b99c042a439d3b is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:net_core_utils) = 9e8577ad79f87179 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
kernel(xen:arch_x86_mm_hypervisor) = 8eefd097daaf1260 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
gstamnbapp01:/export #
================================================================================================================================
gstamnbapp01:/export # rpm -ivh cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64.rpm
warning: cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64.rpm: Header V3 RSA/SHA1 signature: NOKEY, key ID 5ce2d476
error: Failed dependencies:
kernel(xen:fs_seq_file) = 805652a43af70950 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:mm_slab) = 845e11a139410068 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:arch_x86_kernel_pci-dma) = 94e621eac16acd1c is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:kernel_panic) = 836b6bbe1d0c6009 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:drivers_pci_msi) = d69d27819ce5716f is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:drivers_xen_core_spinlock) = 2bf87a8209d08c72 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:drivers_base_core) = a062475ae36e4c3c is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:fs_proc_proc) = e8f18e6c37526c48 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:arch_x86_kernel_x8664_ksyms_64) = 2af1e5a3b31ce371 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:block_blk-core) = cccfc96d00e04e81 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:vmlinux) = 67c649353f6f4284 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:kernel_module) = 0a94a4e092761946 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:kernel_printk) = d1982403e0a2e113 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:block_blk-softirq) = 0fc741941da0952e is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:drivers_scsi_scsi_mod) = 211ec24fadce0701 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:kernel_params) = 7af15e23aeb0c96a is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:lib_scatterlist) = 3b20213c9dc3874a is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:arch_x86_mm_ioremap) = 41e494602b96f119 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:mm_page_alloc) = 915b9881f339020c is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:drivers_pci_pci-driver) = 4886485b5e358022 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:drivers_base_dd) = 6902639516c11d0c is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:block_blk-merge) = 0534acb8280d9c5f is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:kernel_spinlock) = 325eda3ccd7facf4 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:kernel_sched) = 5eb007fdef6134ee is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:kernel_mutex) = 2efeeae89371ff9e is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:block_blk-settings) = abc34e60742ed184 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:drivers_pci_access) = 141ba8f9f4b107fe is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:block_genhd) = 43ef350547039e77 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:arch_x86_kernel_setup_percpu) = 1e35c1127e2872e6 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:block_scsi_ioctl) = 2c2dbca314161b0b is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:block_scsi_ioctl) = 2c4e5cc85471c421 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:drivers_pci_pci) = 85638a4879aa8d22 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:drivers_xen_core_evtchn) = 8ebfb63c2796a54d is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:kernel_compat) = d2cd44da4237bdb2 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:init_main) = f547105fd821a308 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:kernel_capability) = d4d1b3113d734c41 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:kernel_timer) = 423e9e985483e429 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:kernel_irq_manage) = 7749acddf1ec7248 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:drivers_base_bus) = a43ddf048951ea5d is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:arch_x86_mm_physaddr) = 8a91ed3165f0c415 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:mm_memory) = a1b99c042a439d3b is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
kernel(xen:arch_x86_mm_hypervisor) = 8eefd097daaf1260 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
================================================================================================================================
05-18-2015 08:47 AM
right, i have updated all drivers, firmware and BIOS :| Fingers crossed!
HP to replace system board..
05-19-2015 02:35 AM
Following the updates of BIOS, Firmware and various drivers the firmware error appears to have been disappeared...
05-19-2015 04:35 AM