cancel
Showing results for 
Search instead for 
Did you mean: 

NBU MSDP Media Server kept on rebooting intermittently.

Spartacus81
Level 6
Partner Accredited

Friends,

I have got strange problem going where one of my media server keep on rebooting mostly over the weekend.

OS SuSe Linux 11 SP2

NBU 7.6.0.3

Physical box HP DL180 G6

Its an onsite appliance that backs up client locally and replicate (AIR) to central location. 

/var/log/messages doesn't contain much information..

I have ran the HP diagnostic to see if there are any problem with hardware componenets but all came out. 

I have upgraded the FW and number of drivers and due to upgrade BIOS but i dont think abnormal reboot has anything to do with BIOS. 

Has anyone seen this problem. any input/ideas will highly be appreicated otherwise i would have no choice except to rebuilt the server. 

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Spartacus81
Level 6
Partner Accredited

Following the updates of BIOS, Firmware and various drivers the firmware error appears to have been disappeared... 

View solution in original post

9 REPLIES 9

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

.... /var/log/messages doesn't contain much information...... 

Curious to know what is logged in messages shortly before reboot.

Spartacus81
Level 6
Partner Accredited

thanks marianne..

May 16 22:03:39 gstamnbapp01 syslog-ng[1552]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=554', processed='center(received)=524', processed='desti
nation(messages)=518', processed='destination(mailinfo)=2', processed='destination(mailwarn)=0', processed='destination(localmessages)=3', processed='destination(newserr)=0', processed='destination(mailerr)=
0', processed='destination(netmgm)=0', processed='destination(warn)=11', processed='destination(console)=7', processed='destination(null)=2', processed='destination(mail)=2', processed='destination(xconsole)
=7', processed='destination(firewall)=0', processed='destination(acpid)=2', processed='destination(newscrit)=0', processed='destination(newsnotice)=0', processed='source(src)=524'
May 16 22:06:43 gstamnbapp01 syslog-ng[1558]: syslog-ng starting up; version='2.0.9'
May 16 22:06:43 gstamnbapp01 firmware.sh[1586]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:43 gstamnbapp01 firmware.sh[1592]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:43 gstamnbapp01 firmware.sh[1603]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1614]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1625]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1636]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1652]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1669]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1682]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1695]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1708]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1721]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1734]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1747]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1760]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:44 gstamnbapp01 firmware.sh[1773]: Cannot find  firmware file 'intel-ucode/06-2c-02'
May 16 22:06:45 gstamnbapp01 ifup:     lo
May 16 22:06:45 gstamnbapp01 ifup:     lo
May 16 22:06:45 gstamnbapp01 ifup: IP address: 127.0.0.1/8
May 16 22:06:45 gstamnbapp01 ifup:
May 16 22:06:45 gstamnbapp01 ifup:
May 16 22:06:45 gstamnbapp01 ifup: IP address: 127.0.0.2/8
May 16 22:06:45 gstamnbapp01 ifup:
May 16 22:06:45 gstamnbapp01 ifup:     eth0      device: Intel Corporation 82576 Gigabit Network Conne
May 16 22:06:45 gstamnbapp01 ifup:     eth0
May 16 22:06:45 gstamnbapp01 ifup: IP address: 10.8.20.65/24
May 16 22:06:45 gstamnbapp01 ifup:
May 16 22:06:45 gstamnbapp01 SuSEfirewall2: SuSEfirewall2 not active
May 16 22:06:46 gstamnbapp01 ifup:     eth1      device: Intel Corporation 82576 Gigabit Network Conne

 

Nicolai
Moderator
Moderator
Partner    VIP   

what does /va/logs/messages say ?

May be a hardware related issue. Is the load during weekends larger than normal ?

Did you have a look at the ILO console log ? 

 

Spartacus81
Level 6
Partner Accredited

i can see the above logs repeating in /var/log messages -- cannot find firmware file and then system restart as you can see above.

well, with regards to backups they are dedup/optimized synthetic and replication (NBU AIR)

so its a not a backup load or anything.. 

i have googled the above error SuSe says install the following package which is alreayd installed

https://www.novell.com/support/kb/doc.php?id=7010705

HP says ignore this message but thats for DL380 G7.. mine is DL180 G6

http://h20564.www2.hp.com/hpsc/doc/public/display?docId=emr_na-c03697510

Found this thread where people talking about temprature and fans - 

http://h30499.www3.hp.com/t5/ProLiant-Servers-ML-DL-SL/HP-ProLiant-DL180-G6-Server-restart-by-itself/td-p/4777782#.VVnHlflViko

no errors in system event logs on the iLO.

Ran HP diagnostic tools no hardware error found.. 

Only thing is left is to update the BIOS .. :S

 

Nicolai
Moderator
Moderator
Partner    VIP   

BIOS upgrades can do all sort of wonders. 

If you open a support ticket with HP they will insist on having the latest firmware revision loaded, so not a bad thing to do.

Spartacus81
Level 6
Partner Accredited

i know thats why am bit reluctant to do the BIOS. 

support ticket with HP is already opened and they advised to update BIOS, firmware and whole lot of drivers..

i have updated the firmware but other drivers are asking missing dependencies..

 

==========================================================================================================================
gstamnbapp01:/export # rpm -ivh hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64.rpm
warning: hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64.rpm: Header V3 RSA/SHA1 signature: NOKEY, key ID 5ce2d476
error: Failed dependencies:
        kernel(xen:kernel_timer) = 423e9e985483e429 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:drivers_pci_pci) = 85638a4879aa8d22 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:mm_slab) = 845e11a139410068 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:net_ethernet_eth) = 406b7cb5cf4b32c3 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:arch_x86_kernel_pci-dma) = 94e621eac16acd1c is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:net_core_dev) = 3a6bf572c39cfef2 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:kernel_panic) = 836b6bbe1d0c6009 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:drivers_hwmon_hwmon) = c3b63051fffc528b is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:drivers_pci_msi) = d69d27819ce5716f is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:net_core_netpoll) = b93abb3bba079a91 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:drivers_base_power_runtime) = 0514c1712f4a5433 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:mm_page_alloc) = 915b9881f339020c is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:init_main) = f547105fd821a308 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:arch_x86_kernel_x8664_ksyms_64) = 2af1e5a3b31ce371 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:kernel_module) = 0a94a4e092761946 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:vmlinux) = 67c649353f6f4284 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:kernel_printk) = d1982403e0a2e113 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:net_core_skbuff) = 4c9f9431a9f5f113 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:net_core_sock) = 4a8b3ca6d3876f61 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:drivers_base_power_wakeup) = 6c968d103ae8c86e is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:kernel_params) = 7af15e23aeb0c96a is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:mm_vmalloc) = 49ca512fe4d78100 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:arch_x86_mm_ioremap) = 41e494602b96f119 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:drivers_pci_pci-driver) = 4886485b5e358022 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:arch_x86_kernel_cpu_common) = 6387e2748a649d07 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:lib_bitmap) = 1bd5f25878fbbd41 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:drivers_base_dd) = 6902639516c11d0c is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:kernel_rcutree) = bb9ce712db2a1247 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:drivers_pci_iov) = 53b8b35afc7e5a03 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:drivers_base_core) = a062475ae36e4c3c is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:drivers_pci_search) = a1315fafd5354c6e is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:drivers_pci_pcie_aer_aerdriver) = 5e0a9b219cee4519 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:net_sched_sch_generic) = 54f52bb0ded5130f is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:drivers_pci_access) = 141ba8f9f4b107fe is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:net_core_rtnetlink) = 394ca6becda42c3e is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:drivers_char_random) = 0262f5133940a8f2 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:kernel_time_jiffies) = 0a1119a9d9ca3d67 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:mm_nobootmem) = 2a5b8099b0d91697 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:kernel_workqueue) = 935a39c08f5e6a71 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:mm_swap) = a17f9792b744588a is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:kernel_cpu) = aa72f1964aa3a83a is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:kernel_capability) = d4d1b3113d734c41 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:kernel_irq_manage) = 7749acddf1ec7248 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:arch_x86_mm_physaddr) = 8a91ed3165f0c415 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:mm_memory) = a1b99c042a439d3b is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:net_core_utils) = 9e8577ad79f87179 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
        kernel(xen:arch_x86_mm_hypervisor) = 8eefd097daaf1260 is needed by hp-igb-kmp-xen-5.2.15_3.0.13_0.27-2.sles11sp2.x86_64
gstamnbapp01:/export #
================================================================================================================================


gstamnbapp01:/export # rpm -ivh cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64.rpm
warning: cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64.rpm: Header V3 RSA/SHA1 signature: NOKEY, key ID 5ce2d476
error: Failed dependencies:
        kernel(xen:fs_seq_file) = 805652a43af70950 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:mm_slab) = 845e11a139410068 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:arch_x86_kernel_pci-dma) = 94e621eac16acd1c is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:kernel_panic) = 836b6bbe1d0c6009 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:drivers_pci_msi) = d69d27819ce5716f is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:drivers_xen_core_spinlock) = 2bf87a8209d08c72 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:drivers_base_core) = a062475ae36e4c3c is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:fs_proc_proc) = e8f18e6c37526c48 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:arch_x86_kernel_x8664_ksyms_64) = 2af1e5a3b31ce371 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:block_blk-core) = cccfc96d00e04e81 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:vmlinux) = 67c649353f6f4284 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:kernel_module) = 0a94a4e092761946 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:kernel_printk) = d1982403e0a2e113 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:block_blk-softirq) = 0fc741941da0952e is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:drivers_scsi_scsi_mod) = 211ec24fadce0701 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:kernel_params) = 7af15e23aeb0c96a is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:lib_scatterlist) = 3b20213c9dc3874a is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:arch_x86_mm_ioremap) = 41e494602b96f119 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:mm_page_alloc) = 915b9881f339020c is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:drivers_pci_pci-driver) = 4886485b5e358022 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:drivers_base_dd) = 6902639516c11d0c is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:block_blk-merge) = 0534acb8280d9c5f is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:kernel_spinlock) = 325eda3ccd7facf4 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:kernel_sched) = 5eb007fdef6134ee is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:kernel_mutex) = 2efeeae89371ff9e is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:block_blk-settings) = abc34e60742ed184 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:drivers_pci_access) = 141ba8f9f4b107fe is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:block_genhd) = 43ef350547039e77 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:arch_x86_kernel_setup_percpu) = 1e35c1127e2872e6 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:block_scsi_ioctl) = 2c2dbca314161b0b is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:block_scsi_ioctl) = 2c4e5cc85471c421 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:drivers_pci_pci) = 85638a4879aa8d22 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:drivers_xen_core_evtchn) = 8ebfb63c2796a54d is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:kernel_compat) = d2cd44da4237bdb2 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:init_main) = f547105fd821a308 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:kernel_capability) = d4d1b3113d734c41 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:kernel_timer) = 423e9e985483e429 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:kernel_irq_manage) = 7749acddf1ec7248 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:drivers_base_bus) = a43ddf048951ea5d is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:arch_x86_mm_physaddr) = 8a91ed3165f0c415 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:mm_memory) = a1b99c042a439d3b is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
        kernel(xen:arch_x86_mm_hypervisor) = 8eefd097daaf1260 is needed by cciss-kmp-xen-4.6.28-24.sles11sp2.x86_64
================================================================================================================================

Spartacus81
Level 6
Partner Accredited

right, i have updated all drivers, firmware and BIOS :| Fingers crossed!

HP to replace system board.. 

 

Spartacus81
Level 6
Partner Accredited

Following the updates of BIOS, Firmware and various drivers the firmware error appears to have been disappeared... 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
Issues like these is excellent justification for NBU Appliance...