Forum Discussion

sdo's avatar
sdo
Moderator
10 years ago

Duplication, performance of "bpdm.exe"...

Env: NetBackup v7.6.0.1 on RHEL 6.4.  Master/media server (single node).  Fairly new build, about 3 weeks old.

Issue: Duplication to tape (over FC) from MSDP disk pool (itself on SAN/FC storage) seems slow.

One small MSDP disk pool of c. 10TB, which has several weeks of backups now within it, and is about 55% occupied.  Currently duplicating a set of full backups (circa 300 backup images, totalling about 7TB of backup data) via two 'bpduplicate' commands.  The backups being duplicated, are the first set of full backups ever ingested by this environment).

No other backups, replications or duplications are running.

No legacy log folders exist.  VxUL logging was the typical usual default of Debug=1 and Diagnostic=6.

Nothing scary seen in /var/log/messages.

I'm seeing about 50 MB/s to each of two tape drives.

Also seeing two "bpdm.exe" processes regularly hitting 99%/100% of CPU, and a total of 25% of machine CPU used.  RHEL reports server has 8 CPU cores.

NetBackup KMS is configured, and the duplications are writing to an "ENCR_" volume pool for KMS.

NetBackup application daemons were restarted recently (a few days ago).

ulimit and semaphores checked and are ok.

CPU(s) appears to be...  two physical CPU, each quad core, each single thread.

[user@server ~]# cat /proc/cpuinfo | egrep "MHz|physical id|core id|^$"
cpu MHz         : 1200.000
physical id     : 0
core id         : 0

cpu MHz         : 1800.000
physical id     : 0
core id         : 1

cpu MHz         : 1200.000
physical id     : 0
core id         : 2

cpu MHz         : 1800.000
physical id     : 0
core id         : 3

cpu MHz         : 1200.000
physical id     : 1
core id         : 0

cpu MHz         : 1200.000
physical id     : 1
core id         : 1

cpu MHz         : 1800.000
physical id     : 1
core id         : 2

cpu MHz         : 1200.000
physical id     : 1
core id         : 3

(I think the CPUs must be 1.2 GHz with boost to 1.8GHz, because, for each CPU two cores show at 1.2 GHz, and two cores show at 1.8 GHz).

I wasn't involved in the environment sizing, design, build or implementation - so I've no idea (yet) what the server tin actually is.

I suspect that the fact that each "bpdm.exe" maxes out a CPU core is causing the limitation of each duplication to tape to run at around 50MB/s.

I would have expected "bpdm.exe" to simply be a data mover, receiving data from spoold, and forwarding to bptm, and I wouldn't have expected to see each process consuming 100% of a CPU core.

Can anyone offer any reason as to why bpdm maxes out a CPU core?

Thanks in advance.

7.6
Backup and Recovery
Linux
NetBackup
Performance
Troubleshooting
  • mph999's avatar
    mph999
    10 years ago

    As a real simple test, backup some data to a basic disk STU, then duplicate that, how fast is it ?

  • Nicolai's avatar
    Nicolai
    10 years ago

    1: We had a in-house Linux admin to look at our setup. And this was his advice

    2: You can see if hyper-threading is enabled in the BIOS. You enable/disable in BIOS as well.

    Regarding the use of swap space set 

    vm.swappiness = 1 in /etc/sysctl.conf

    Explained here : https://lonesysadmin.net/2013/12/11/adjust-vm-swappiness-avoid-unneeded-disk-io/

     

  • Nicolai's avatar
    Nicolai
    10 years ago

    I would increase NUMBER_DATA_BUFFERS to 128 or 256

    MSDP encryption is documented in the deduplication guide

    http://www.symantec.com/docs/DOC6466

    Page 34. As I read it, its Netbackup that does the encryption/decryption. But Netbackup may very well uses library call to other modules in the OS. The manual say "Deduplication uses the Blowfish algorithm for encryption"

  • Nicolai's avatar
    Nicolai
    10 years ago

    strace can do it - but you need kernel skills in decoding the output.

    http://en.wikipedia.org/wiki/Strace

15 Replies

Sort By
  • revarooo's avatar
    revarooo
    Level 6
    10 years ago
    No idea. Have you tried enabling bpdm? For a dedupe duplication 50mb/s to tape is not terrible considering the rehydration overhead.
  • Nicolai's avatar
    Nicolai
    Moderator
    10 years ago

    Try disabling hyper-threading. Single threaded process will not benefit from hyber-threading. actual you may end up with a worse performance because to single threaded process will fight for resources from the same core.

    Besides that the its the quality of disk under the msdp pool that decides what re-hydration speed you will get.

  • sdo's avatar
    sdo
    Moderator
    10 years ago

    Hey thanks Nicolai.

    1) What (in your experience) brings you to suggest that?

    2) How do I determine whether Hyper-Threading is active?  Then I can go to O/S admin team, and see whether it's something that can be easily enabled/disabled so that I can perform some testing either way.

     

     

  • sdo's avatar
    sdo
    Moderator
    10 years ago

    On Friday/Saturday, I noticed that 4GB of swap space was consumed by the O/S, and that this server has 23GB of RAM - but today, on the now quiet system, zero swap space is currently in use.

    IMO, a NetBackup master/media should not be making use of any swap space.

    My rough calculations are:

        RHEL O/S                       6 GB

        NetBackup application    8 GB

        NetBackup buffers          2 GB   (although not configured at the moment)

        MSDP (1.5 * 10TB)       15 GB

    ...so, possibly, this system should have at least 31GB of RAM - which might go some way to explain why I saw 4GB of swap space used during busy periods.

     

    I don't know enough about RHEL O/S to know whether recalling lots of data from O/S swap space consumes CPU or where that CPU time/cost is charged too - so I don't know whether the use of swap space was somehow causing CPU cycles to be consumed within the bpdm processes when I saw them hitting 100%.

     

    Anway, I grabbed the bptm data consumer wait and delay times (from the activity monitor job log for the duplication jobs) and there's many hours lost for bptm waiting for full buffer, but unfortunately, I didn't have logging enabled at the time, so I don't have the bpdm data producer waits/delays for empty buffer - but I did some further duplication testing (with logging enabled) on a small-ish (15GB) backup image and bpdm was spending some time waiting too - but during the duplication test (from MSDP to tape) I didn't see the bpdm process go above an average of 35% of a CPU core.

     

    The LTO6 tape media (2.5TB native) which did reach full, both show as circa 3.5 TB occupied - so the tape drive heads are definitely able to SCSI T10 compress the incoming data - which says to me that the tape drives are not receiving 'encrypted or compressed' data to write.

    I had previously enabled MSDP encryption at rest (in the pd.conf file), so I'm now wondering if maybe something about the mixture of NetBackup and RHEL and the Intel CPU model/type inside the server tin is altogether somehow not very good at decrypting the MSDP de-duped data blocks read from MSDP pool.

    Does anyone know where the 'encryption/decryption' of NetBackup MSDP is performed?  I mean, is encryption/decryption performed by:

    a) software local to NetBackup MSDP, i.e. do Symantec use their own software routines.

    b) or does NetBackup call a RHEL O/S library routine to perform encryption/decryption

    c) or does NetBackup (and/or the O/S library call) make use of some extended hardware instructions within the Intel CPU itself?

    d) or maybe, it's option c) first, and if the required CPU hardware instructions for encryption/decryption are not present within the CPU model, then option b) or a) is performed.

    Which then makes me want to ask, how do I actually determine where NetBackup MSDP encryption/decryption is peformed?

    Thanks.

     

  • mph999's avatar
    mph999
    Level 6
    10 years ago

    As a real simple test, backup some data to a basic disk STU, then duplicate that, how fast is it ?

  • sdo's avatar
    sdo
    Moderator
    10 years ago
    yes - more testing is needed - just did a quick test of duplicating from MSDP to DSU on same 10TB LUN in RHEL, and it achieved 154MB/s - I'll try to find time to do a wider variety of storage unit to storage unit duplication test, with enough bpdm logging - and maybe some buffer tuning testing too, after I've base lined the basics.
  • Nicolai's avatar
    Nicolai
    Moderator
    10 years ago

    1: We had a in-house Linux admin to look at our setup. And this was his advice

    2: You can see if hyper-threading is enabled in the BIOS. You enable/disable in BIOS as well.

    Regarding the use of swap space set 

    vm.swappiness = 1 in /etc/sysctl.conf

    Explained here : https://lonesysadmin.net/2013/12/11/adjust-vm-swappiness-avoid-unneeded-disk-io/

     

  • sdo's avatar
    sdo
    Moderator
    10 years ago

    Thanks for the swappiness thing Nicolai.  Not changed that yet, but shared info with sys admin.

    Thanks Martin.  I did various tests between DSU, MSDP, LTO6 - to and from - with around 175GB of data, and have more than doubled MSDP to LTO6 duplication performance simply by setting size_data_buffers to 262144, and number_data_buffers to 64.

    I'll have to wait for the next big batch of duplications to see whether it has any real world benefit.

    The RHEL question re Hyper-Threading intefering with a single threaded process - is probably not for here...

    ...so, I think my remaining NetBackup question is:

    1) How do I determine where MSDP encryption/decryption is actually taking place?  In CPU hardware, or via RHEL module, or via NetBackup software?

    Thanks.

  • Haniwa's avatar
    Haniwa
    Level 4
    10 years ago

    What do you find to be the best method to measure duplication speed, especially when dealing with a large batch of images ?

    Ken W
     

  • Nicolai's avatar
    Nicolai
    Moderator
    10 years ago

    I would increase NUMBER_DATA_BUFFERS to 128 or 256

    MSDP encryption is documented in the deduplication guide

    http://www.symantec.com/docs/DOC6466

    Page 34. As I read it, its Netbackup that does the encryption/decryption. But Netbackup may very well uses library call to other modules in the OS. The manual say "Deduplication uses the Blowfish algorithm for encryption"