How to optimaly utilize if i have 20 drives in ndmp backup

[Zailar]

Hello Expert, Directory backup selection and multistream backup
/level1a/level2a/level3a
/level1a/level2a/level3b
/level1a/level2a/level3c
/level1a/level2a/level3d
/level1a/level2a/level3e
/level1a/level2a/level3f
/level1a/level2a/level3g
/level1a/level2a/level3h

I have tries backup selection with
/level1a/level2a/level3a
/level1a/level2a/level3b
/level1a/level2a/level3c
/level1a/level2a/level3d
/level1a/level2a/level3e
/level1a/level2a/level3f
/level1a/level2a/level3g
/level1a/level2a/level3h/level4a
/level1a/level2a/level3h/level4b
/level1a/level2a/level3h/level4c
/level1a/level2a/level3h/level4d
/level1a/level2a/level3h/level4e
/level1a/level2a/level3h/level4f
/level1a/level2a/level3h/level4g
/level1a/level2a/level3h/level4h
/level1a/level2a/level3h/level4i
/level1a/level2a/level3h/level4j
/level1a/level2a/level3h/level4k
/level1a/level2a/level3h/level4l
/level1a/level2a/level3h/level4m

but sometime error code 23 (random in directory /level1a/level2a/level3h/level4m or /level1a/level2a/level3h/level4a) How to optimaly utilize if i have 20 drives in ndmp backup Or have idea for speed up baackup in ndmp?

Thank You Zainal

*** Inserted line breaks for readability...  M. ***

[quebek]

hey

To leverage multiple drives - add to this selection list directive 

NEW_STREAM

as first entry, than after few streams (which should be occupying one tape driver) add yet another NEW_STREAM 

and so on...

Hope this help. What is that filer? Some have max of 4 concurrent jobs...

You can read about this directive here https://www.veritas.com/support/en_US/article.100026337

and if you will google you will find more.

[Nicolai]

From my experience, that's a couple of years old by now. You should be careful not to start to many NDMP data streams at the same time, as the data movers internally in the filer also has a capacity. The manual for the filer should state how may data stream you can start at the same time.

[Zailar]

Hi Quebek,

Thank You

For directory backup selection list bellow and attribute multistream , just only use 8 drives

/level1a/level2a/level3a
/level1a/level2a/level3b
/level1a/level2a/level3c
/level1a/level2a/level3d
/level1a/level2a/level3e
/level1a/level2a/level3f
/level1a/level2a/level3g
/level1a/level2a/level3h

For directory backup selection list bellow and attribute multistream, use 20 drives

but sometime error code 23 (random in directory /level1a/level2a/level3h/level4m or /level1a/level2a/level3h/level4a)

/level1a/level2a/level3a
/level1a/level2a/level3b
/level1a/level2a/level3c
/level1a/level2a/level3d
/level1a/level2a/level3e
/level1a/level2a/level3f
/level1a/level2a/level3g
/level1a/level2a/level3h/level4a
/level1a/level2a/level3h/level4b
/level1a/level2a/level3h/level4c
/level1a/level2a/level3h/level4d
/level1a/level2a/level3h/level4e
/level1a/level2a/level3h/level4f
/level1a/level2a/level3h/level4g
/level1a/level2a/level3h/level4h
/level1a/level2a/level3h/level4i
/level1a/level2a/level3h/level4j
/level1a/level2a/level3h/level4k
/level1a/level2a/level3h/level4l
/level1a/level2a/level3h/level4m

Your suggetion mean like this?  just only use 4 drives

NEW_STREAM
/level1a/level2a/level3a
/level1a/level2a/level3b
NEW_STREAM
/level1a/level2a/level3c
/level1a/level2a/level3d
NEW_STREAM
/level1a/level2a/level3e
/level1a/level2a/level3f
NEW_STREAM
/level1a/level2a/level3g
/level1a/level2a/level3h

use Isilon filers and 4 A100

Regards

Zainal

[Zailar]

Hi Nicolai,

You are right,

I was tried changes backup selection more than 10, for better backup result time.

Regards

Zainal

[Marianne]

@Zailar 

I agree with @Nicolai 

You need to check your Isilon documentation for NDMP limitations.

[Zailar]

Hello Marianne,

About NDMP limitations, just only backup speed.

To the point question ,

If backup selection list bellow and attribute multistream :

/level1a/level2a/level3a
/level1a/level2a/level3b
/level1a/level2a/level3c
/level1a/level2a/level3d
/level1a/level2a/level3e
/level1a/level2a/level3f
/level1a/level2a/level3g
/level1a/level2a/level3h

What parameter we can use, for running backup with 20 drives?. 

Or the backup selection must be list bellow for running backup with 20 drive:

/level1a/level2a/level3a
/level1a/level2a/level3b
/level1a/level2a/level3c
/level1a/level2a/level3d
/level1a/level2a/level3e
/level1a/level2a/level3f
/level1a/level2a/level3g
/level1a/level2a/level3h/level4a
/level1a/level2a/level3h/level4b
/level1a/level2a/level3h/level4c
/level1a/level2a/level3h/level4d
/level1a/level2a/level3h/level4e
/level1a/level2a/level3h/level4f
/level1a/level2a/level3h/level4g
/level1a/level2a/level3h/level4h
/level1a/level2a/level3h/level4i
/level1a/level2a/level3h/level4j
/level1a/level2a/level3h/level4k
/level1a/level2a/level3h/level4l
/level1a/level2a/level3h/level4m

Regards

Zainal

[Marianne]

@Zailar 

You cannot go higher than the Isilon NDMP limitations and recommendations.

Please see this link:

http://doc.isilon.com/onefs/8.1.1/help/en-us/ifs_r_ndmp_performance_recommendations.html

[Zailar]

Thank you Marianne for your advice.

At the moment we ignore the Isilon NDMP limitation.

We have more than 150 directory (backup selection),  not too bad and running well with 20 drives. But the last backup run just only with 10 drives, because the size is big.

We tries break the directory big size like this

/level1a/level2a/level3h/level4a
/level1a/level2a/level3h/level4b
/level1a/level2a/level3h/level4c
/level1a/level2a/level3h/level4d
/level1a/level2a/level3h/level4e
/level1a/level2a/level3h/level4f
/level1a/level2a/level3h/level4g
/level1a/level2a/level3h/level4h
/level1a/level2a/level3h/level4i
/level1a/level2a/level3h/level4j
/level1a/level2a/level3h/level4k
/level1a/level2a/level3h/level4l
/level1a/level2a/level3h/level4m

Sometime backup error 23 the file not found.

In linux/unix client server, we can grab file or directory with command ls -l /level1a/level2a/level3h as backup selection

Or my question not have answer?.

[Marianne]

@Zailar 

At the moment we ignore the Isilon NDMP limitation.

If you choose to ignore the vendor's performance recommendations, you are on your own.

Good luck!

[Genericus]

Please be aware - there is an issue with Isilon TCP/IP and NetBackup - watch out for multiple tries on incremental backups

Backup will complete successfully update the listing then fail and rerun successfully and backup nothing.

Summary

After review, engineering has confirmed from the Network trace investigation that this issue is observed due to rare DMA slowness with reading from the received buffer (1-2% failure jobs). At the same time, the RST flag is sent by the Isilon node due to fast socket termination by the NDMP daemon without confirmation that all data was received by the DMA client exactly at the same slowness time on the DMA side.

To improve this situation from the Isilon NDMP side, engineering is looking at the following NDMP changes:

1. Сall shutdown() before close()

The Dev team has already started this improvement in the Isilon NDMP code. Also, Engineering is working on a repro of this problem to have a lab where the code improvement can be validated. The current estimation is that the fix will take up to 4 weeks, assuming no issues. ( It has been 5 weeks now and no updated ETA... )