Forum Discussion

Alexis_Jeldrez's avatar
Alexis_Jeldrez
Level 6
7 years ago

Troubleshooting MSDP with Disk Pool regularly going down

Master is the Storage Server: NetBackup 8.0 with Windows 2008 R2 SP1 Disk used for MSDP disk pool: External storage connected to the Windows server through a dedicated HBA. Uses a SAN switch and Mic...
8
Windows Server 2008 R2
RiaanBadenhorst's avatar
RiaanBadenhorst
Level 6

Hi,

You can try configuring / modifying these touch files. The issue is not on the actual media server, or the storage. It is the master server that can't communicate or receive the status from the media server across the LAN. When it can't get an update if believes the MSDP is down and them marks it in the EMM.

 

https://www.veritas.com/support/en_US/article.100007548

Alexis_Jeldrez's avatar
Alexis_Jeldrez
Level 6
7 years ago

Sorry for not replying before, I couldn't check the platform until today.

Thanks for the information, I haven't tried the CR_STATS_TIMER = 300 before in the pd.conf file. I'll check what happens now.

I had already implemented the touch files and used the 1800 seconds configuration. The result was that the monitoring was delayed for 30 minutes before declaring the disk pool down (before it just waited for couple of minutes).

  • Alexis_Jeldrez's avatar
    Alexis_Jeldrez
    Level 6
    6 years ago

    Thanks for replying.

    RiaanBadenhorst

    I tried everything in https://www.veritas.com/support/en_US/article.100007548 including CR_STATS_TIMER = 300 in the pd.conf file and rebooted the OS: problem persisted. I'm reverting the changes (deleting the three DPS_PROXY files). I checked the nbrmms and dps logs with vxviewlogs and spoold.log & spad.log but they just stop logging when the Disk Volume goes down.

    Marianne

    Most of the conversation in that thread is about a shared platform; I have physical access to the one I'm troubleshooting and I can confirm it's 100% physical and dedicated. RAM does't seem to be a problem, it never ever goes beyond 60% of use.

    As for the speed of the storage, I tried https://www.veritas.com/support/en_US/article.000095782 and got these results:

    • Write speed: 239.2 MB/sec
    • Read with a 64k buffer: 139.4 MB/sec
    • Read with a 1024k buffer: 249.2 MB/sec
    • Read with a 4096k buffer: 300.6 MB/sec

    The contentrouter.cfg reads the following, which seems okay:

     

    ; This parameter determines the data store read buffer size, in Bytes. The default value
; is 33554432 (32MB)
; @restart
; @validate [0-9]+
ReadBufferSize=4194304

     

    I tried testing the reading speed with a buffer of 8MB but got an error, which is strange because I should have 15GB of RAM in standby.

     

    E:\test>nbperfchk -i E:\TEST\file.test -bs 8192k -o NUL
     800 MB @ 266.7 MB/sec,      792 MB @ 264.0 MB/sec
    1728 MB @ 288.0 MB/sec,      928 MB @ 309.3 MB/sec
input: Invalid access to memory location.

     

    I'm still checking https://www.veritas.com/support/en_US/article.100037977 , I've already confirmed:

    • Windows 2008 R2 has Service Pack 1 installed
    • Block Size of the NTFS filesystem is 64K
    • Windows disk indexing disabled
    • MSDP path is excluded from antivirus
    • NetBackup policies don't use compression nor encryption

    I'll keep trying with the other stuff. On the meantime, I'll just have to reduce again the number of duplications from MSDP to tape in the weekends.

    • sdo's avatar
      sdo
      Moderator
      6 years ago

      Check your SAN switch ports for the ports that are zoned to provide connectivity between NetBackup servers and storage (any tape and disk).

      i.e. there will be three sets of ports to zero the counters on and then check next time the problem happens, or even to check every day.

      set 1 - the SAN switch ports facing the NetBackup servers

      set 2 - the SAN switch ports facing the disk storage (for NetBackup)

      set 3 - the SAN switch ports facing the tape storage (for NetBackup)

      .

      Check the light levels too of every SAN switch port too.  If your signal levels are less than -12dB (less than minus twelve) then that is borderline and suspect.

      Of course also look for errors counts on all ports.

      And look for "LLI" (lost link indicator) counts which hints of reboots.