NetBackup client-side deduplication and Multiple policies

[yobole]

NetBackup client-side deduplication and multiple policies 

I found this article 

http://www.symantec.com/business/support/index?page=content&id=HOWTO13907

And especially 

A NetBackup client using change journal support must belong to only one policy. To use one policy avoids the confusion that multiple backup settings causes. Multiple backup settings can cause conflicted update sequence number (USN) information in the permanent record “ 

 We wanted to setup two policies to backup a Netbackup client with multiple volume. Put all the smaller volumes in one policies and the larger one in one policies as the larger one taking forever to  backup … is this possible ??

[Mark_Solutions]

Based on that tech note i guess the way to proceed it to not use the change journal (set via Host Properties of the client where you would have it set to use the archive bit)

That may be different if you are using Accelerator though as it will use the chnage journal if it is available

All of these however are dependant on whether the client itself actually has the change journal actually enabled

In view of the advice and what you want it would probably be best for you to have your multiple policies, incremental backups set to use the archive bit, use accelerator too but have the change jounal not enabled on the client itself

[yobole]

we do have client side dedupe , Change journal and accelerator all configured

[Marianne]

There is no need to separate volumes into different policies.

You can add them in one policy and break them up into different streams.

Enable 'Allow Multiple data streams' in policy attribute and divide streams as per this example:

NEW_STREAM
small-volume1
small-volume2
small-volume3
...etc
NEW_STREAM
big-volume1/folder1
NEW_STREAM
big-volume1/folder2
NEW_STREAM
big-volume1/folder3
...etc

[yobole]

This looks like what I need I have  Enabled  "'Allow Multiple data streams' in policy attribute " but not sure what you man below . We just only Enabled "'Allow Multiple data streams' in policy attribute "

NEW_STREAM
small-volume1
small-volume2
small-volume3
...etc
NEW_STREAM
big-volume1/folder1
NEW_STREAM
big-volume1/folder2
NEW_STREAM
big-volume1/folder3

[Marianne]

I thought the example would be self-explanatory....

The example groups a number of small volumes into a single stream.

With NEW_STREAM directive you can add a big volume into a single stream, or you can break it down into smaller 'chunks' or streams. 
You will need a good knowledge of the filesystem layout (directory structure) in order to perform this breakdown.

[yobole]

Thanks but its looks like multiple streams from the same volume is not recommended

http://www.symantec.com/business/support/index?page=content&id=HOWTO34492

[yobole]

Having multiple stream from the large volume will be the best solution but then

http://www.symantec.com/business/support/index?page=content&id=HOWTO34492

Somebody out there must be backing up 6TB of data on 1 voulme with Netbackup :)

[Marianne]

Correct. Multiple streams is not recommended where volumes are on a single disk/device (e.g. 1 physical hard drive that is partitioned.)

Large volumes on a SAN is NEVER on a single disk.

Raid configurations consist of multiple physical disks that are suited for multiple read streams.

[Mark_Solutions]

I do have customers with that sort of data and when lots of very small files are involved a lot of streams do help even when they are on the same volume

As you are using accelerator the track log takes care of things so it should not really be a great issue

However, If you do change your policy you will need to do a forced re-scan (real full backup) first to recreate the track log information for the new paths

You may also need to increase the WorkerThreads value in the /disk/etc/puredisk/contentrouter.cfg file to get the de-dupe to cope (default is 64 so it will need to go up to 128 or 256 - it should be 128 anyway when using accelerator)

[yobole]

Thank you for all your help . yes the large 6TB is on the SAN as an RDM

[yobole]

whats is the maximum number of stream that can be configured per network client. The volume has 20 shares and was going to created 30 streams ... Hope this will be possible . and then am sure netbackup will handle it in chunks

[Mark_Solutions]

I dont remember there being a limit as such - the number of streams will be down to the storage unit you aim it at

The max number of jobs per policy is 999 and per Sorage unit is 256

However - trial and error is the way here - you may want to start with your 30 streams but limit the jobs per policy to 10 and see how it goes

You can decrease / increase it later until you find the best performance for the client - doing it in this way will not affect the track log

#edit# the other limit is the I/O per disk pool which is 65535 - so your 30 is fine but i would still start at 10 and work up or down according to performance when you actually run it

[Marianne · Accepted solution]

The maximum max jobs per client is 99. 

This is a theoretical limit. I have never seen any client being able to handle this amount of streams.

I would personally start with 4, then try 6, then 8. Carefully monitor cpu and memory usage on the client as well as backup throughput after each increase.  Depending on resources, you will get to a point where jobs will be queued or where performance deteriorates. Stick to the value where you got best performance.

Remember that Max jobs per Client is a global setting.
You can customize per client in Host Properties -> Master -> Client Attributes.
Add/select client name to customize.