This is to share with you the Long pending issue which we faced in our environment for 6 months . Our Duplication Jobs started running slow and we started seeing backlog of Duplication jobs building up on daily basis which went up to 500 TB.
Earlier Setup :-
Duplication Jobs Window - 24x7
Replication Jobs Window - 24x7
Storage Unit groups Used for Disk Pool Storage units and Tape Storage units.
Netbackup Appliance 5230 ( 184.108.40.206 )- 1 Master Appliance , 1 Media Appliance in Primary Site , 1 in Remote site for DR.
Netbackup Version - 220.127.116.11.
We started seeing Backlog increase slowly which began to increase to huge size over the period of few months. Backups also were running slow and Replication also went slow. Overall Appliances Performance became very slow.
Case raised with Symantec and worked with Backline To resolve this Performance issue.
Did Data Collects , Logs collected and Sent to Symantec multiple Times.
1. Replaced predictive disk failures for 2 disks in Master appliance on two occasions.
2. Upgraded Netbackup patch to 18.104.22.168
3. Changed the NUMBER_DATA_BUFFERS and NUMBER_DATA_BUFFERS_DISK to 32 to avoid long delays.
4. Broke the Stu Group and Started using Netbackup Disk Storage Unit and Tape Storage Unit for Backup and Duplication respectively.
5. Added Alternate read server in SLP as same as the one used for backup to overcome Data travelling from appliance to other for duplication to tape.
6. /usr/openv/lib/ost-plugins/pd.conf changed from PREFETCH_SIZE = 33554432 to PREFETCH_SIZE = 67108864
7. /disk/etc/puredisk/contentrouter.cfg changed from PrefetchThreadNum=8 to PrefetchThreadNum=16
8. Changed the Max duplication to 5 TB and Min to 500 GB
9. Changed the Replication window and Duplication window such that they dont Overlap.
10. Upgraded the Appliances BIOS and RAID Software
11. Upgraded the Netbackup software to 22.214.171.124 Version
Now After the Upgrade to 126.96.36.199 Version , The performance has improved Drastically and we are seeing Backlog decreasing rapidly with 50TB per day using 10 Tape drives.
Replication performance has also increased and Backups are completing well within time.
Cheers and thanks to Symantec Support and all others who helped us in this issue.