Highlighted

NBU commands running very slow on the solaris media server .

i have a solaris  10 media server  , running NBU 7.5  .tHe issue with the media server is the o/p for any command is taking lomng time to genreate the o/p . even vmoprcmd -d wil generate the o/p after a minute or 2 .

this server is used to run NDMP backups .

6 Replies
Highlighted

If you're sharing tape drives

If you're sharing tape drives among multiple media servers, "vmoprcmd -d" would usually take a while which is expected.

How about this command: nbemmcmd -listhosts -verbose 

This will be a better test to see if EMM database query commands are running fast or slow. Query to EMM database is also applicable to commands like bpimagelist, bpmedialist etc.

Highlighted

Firstly, check the name

Firstly, check the name resolution - make sure all servers in the environment can correctly and quickly forward and reverse lookup to server hostnames that they communicate with. nbdna -phase 2 can be run from both the master and the media server to check lookups. The location of the utput is given when the command completes Is this a master /media, or just a media ? Do you get the same issue on any other server. When did this problem start ? What has been changed ? Are all commands slow - try a few different ones to be sure. A good command to run from the media server is nbemmcmd -listhosts Are backups slow on this media server - I don't mean slow when running, but are they slow to 'start' - that is when doing the bits before the data transfer actually starts. Is there any difference between a backup of the media server itself and one of the media servers clients.
Highlighted

this command took 3 minutes

this command took 3 minutes rto generate the o/p

 

bash-3.00# nbemmcmd -listhosts -verbose
NBEMMCMD, Version: 7.5.0.5
The following hosts were found:
williams
        ClusterName = ""
        MachineName = "williams"
        FQName = "williams-nic3.cisco.com"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0xf7
        MachineNbuType = media (1)
        MachineState = active for tape and disk jobs (14)
        MasterServerName = "rtp7nbu1"
        NetBackupVersion = 7.5.0.5 (750500)
        OperatingSystem = solaris (2)
        ScanAbility = 5
cobb
        ClusterName = ""
        MachineName = "cobb"
        FQName = "cobb.cisco.com"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0xf7
        MachineNbuType = media (1)
        MachineState = active for tape and disk jobs (14)
        MasterServerName = "rtp7nbu1"
        NetBackupVersion = 7.5.0.5 (750500)
        OperatingSystem = solaris (2)
        ScanAbility = 5
rtp7nbu1
        ClusterName = ""
        MachineName = "rtp7nbu1"
        FQName = "rtp7nbu1-nic1.cisco.com"
        GlobalDriveSeed = "VEND:#.Smiley TongueROD:#.:IDX"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0xf7
        MachineNbuType = master (3)
        MachineState = active for tape and disk jobs (14)
        NetBackupVersion = 7.5.0.5 (750500)
        OperatingSystem = solaris (2)
        ScanAbility = 5
rtp1-netapp04b
        MachineName = "rtp1-netapp04b"
        FQName = "rtp1-netapp04b.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
rtp1-netapp04a
        MachineName = "rtp1-netapp04a"
        FQName = "rtp1-netapp04a.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
rtp1-emc06b
        MachineName = "rtp1-emc06b"
        FQName = "rtp1-emc06b.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
rtp1-emc06a
        MachineName = "rtp1-emc06a"
        FQName = "rtp1-emc06a.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
sjc-filer37a
        MachineName = "sjc-filer37a"
        FQName = "sjc-filer37a.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
rtp7-netapp01b
        MachineName = "rtp7-netapp01b"
        FQName = "rtp7-netapp01b.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
rtp7-netapp01a
        MachineName = "rtp7-netapp01a"
        FQName = "rtp7-netapp01a.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
sjc-filer51a
        MachineName = "sjc-filer51a"
        FQName = "sjc-filer51a.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
rtp7-netapp03b
        MachineName = "rtp7-netapp03b"
        FQName = "rtp7-netapp03b.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
rtp7-netapp02a
        MachineName = "rtp7-netapp02a"
        FQName = "rtp7-netapp02a.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
rtp7-netapp03a
        MachineName = "rtp7-netapp03a"
        FQName = "rtp7-netapp03a.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
rtp7-netapp02b
        MachineName = "rtp7-netapp02b"
        FQName = "rtp7-netapp02b.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
rtp7-emc01c
        MachineName = "rtp7-emc01c"
        FQName = "rtp7-emc01c.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
rtp7-emc01b
        MachineName = "rtp7-emc01b"
        FQName = "rtp7-emc01b.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
rtp7-emc01a
        MachineName = "rtp7-emc01a"
        FQName = "rtp7-emc01a.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
rtp7-emc03b
        MachineName = "rtp7-emc03b"
        FQName = "rtp7-emc03b.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
rtp7-emc03a


        MachineName = "rtp7-emc03a"
        FQName = "rtp7-emc03a.cisco.com"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
rtp7nbu1
        MachineName = "rtp7nbu1"
        FQName = "rtp7nbu1-nic1.cisco.com"
        MachineDescription = ""
        MachineNbuType = server (6)
Command completed successfully.
bash-3.00#
 

Highlighted

This is just a media server 

This is just a media server  running NDMP backups , i have rebooted the master server to see if this makes a difference , as it was online for 529 days . but did not help .

The backups have not very slow to start with , and end end up woth 196 .

Highlighted

df -k

df -k o/p

 

 

Filesystem            kbytes    used   avail capacity  Mounted on
/dev/vx/dsk/bootdg/rootvol
                     16525754 7745048 8615449    48%    /
/devices                   0       0       0     0%    /devices
ctfs                       0       0       0     0%    /system/contract
proc                       0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
swap                 60030520    1408 60029112     1%    /etc/svc/volatile
objfs                      0       0       0     0%    /system/object
sharefs                    0       0       0     0%    /etc/dfs/sharetab
/platform/SUNW,T5240/lib/libc_psr/libc_psr_hwcap2.so.1
                     16525754 7745048 8615449    48%    /platform/sun4v/lib/libc_psr.so.1
/platform/SUNW,T5240/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1
                     16525754 7745048 8615449    48%    /platform/sun4v/lib/sparcv9/libc_psr.so.1
fd                         0       0       0     0%    /dev/fd
/dev/vx/dsk/bootdg/var
                     41310066  977748 39919218     3%    /var
swap                 60029160      48 60029112     1%    /tmp
swap                 60029136      24 60029112     1%    /var/run
swap                 60029112       0 60029112     0%    /dev/vx/dmp
swap                 60029112       0 60029112     0%    /dev/vx/rdmp
/dev/vx/dsk/bootdg/crash
                     14120715 7911298 6068210    57%    /var/crash
              

Could this be a memory issue ?

 

                                                

Highlighted

OK, so nbemmcmd output was

OK, so nbemmcmd output was also slow. If you test the same command from master server, or other media server, would it be the same?

If yes, then it's probably the issue of host connection delay, on your master server try to run this:

/usr/openv/netbackup/bin/support/nbdna

It will generate some files to report on any host connection issue, if you have problem reading the result, consider logging a Support case to have them analyze for you.