MSDP crcontrol seem to hang, how to see if MSDP is busy internally ?

crcontrol --getmode

This will tell you if all of the components are actually up and working or not

Failoing that you need to go through the logs spoold and storaged to see if they are indicating any corruption

If you do get corruption then raise a case with Support who may be able to supply the recoverCR tool that can put it all right (takes a while to run - aboyt 1 hour per TB of disk space)

Hope this helps

Hello Michael,

Did you ever get a response from crcontrol? 

When the NetBackup Deduplication Engine (spoold) starts, it reads the file segment fingerprints from the deduplication pool. This can take a while to do. As you probably noticed, store operations) are disabled while the cache is incomplete.

Already have case 05507853 at Symantec Support

No response from crcontrol yet, but can see that spad is reading though very little very slowly

If I remember correctly it took a couple of days last time before it responded

Seems to me that spad/spoold is not really utilizing the amount of hardware that is recommended for deduplication especially for the big volumes like ours on 50 TB

Hello Michael,

I have reached out to the tech who has the case. It looks like this could be related to a bug with Accelerator. 

Hope not, as we are not using accelerator on any policies as far as I know

Have just upgraded to 7.5.0.6 because of the improvements for the deduplication.

Unfortunately there is seemly not a fix for our issue included in the 7.5.0.6 patch, but hope the spad lock improvement will migate the issue.

Michael - are all your disks and BBU battery OK?

Either of these having an issue can cause a real hit on the appliances - if it is a 5230 the BBU won't be an issue as it doesnt have a battery - but a disk issue slows things right down

You could do the accelerator tuning even though you are not using it - it may well help anyway - edit the

/disk/etc/puredisk/contentrouter.cfg and make the WorkerThreads entry 128 (default is 64)

Also note that some setting can get re-set during an upgrade so worth checking through any changes you have made - size and number data buffers, client connect and read timeouts, DPS_PROXYDEFAULTRECVTMO (should be 800)

If you have had issue also check that you dont have any high logging setup - especially the ost-plugin as it causes a huge log to be created and takes a lot of resources (unless Symantec have asked for these of course - though a pd-plugin with a value of 10 will be too large a log to ever transfer anywhere!!

As always try and hack your queue down to as small a value as possible to keep things running better - if there was a big delay running this is may have just grown too big in the first place.

Just a few ideas ....

We are using physical media servers not appliance and the machines an disk systems has been tested throughly. As I understand the mails between support and my collegaue we still have the hang issue some times on SLP between the dedup stores

I had the same problem with you some time ago. In fact we sill have it.

We found that our problem was that the disk subsystem is running at 30 to 40 MB/sec read or write speed. We had test the system at the installation phase and all was working OK. We do not know why the disks are slowed down and we will investigate it. The system isan IBM server with redhat 6.3.
One more evidence is that the start process to load the hashes to memory taking ages to finish.

I'm not saying that this is your problem, but I suggest you to test again the disk subsystem speed, just in case.

The disk system was tested again with very good results (>200MB/sec), looked a little more on the issue as we had a hang again friday afternoon. It is very strange, neither CPU, disk or Memory seems to be highly utilized when this happes. My best guess is some kind of deadlock inside the postgress database.

Not sure i fi have mentioned it before .. but do you do any VMware backups and if so do you have multiple paths configured

It may be more relevant to Stephanos as he is *nix based but i guess there is always the possibility of a semaphore memory leak for Windows too?

It really is not obvious when this happens - everything just grinds to a halt and all you can do is re-boot

For unix you can run the following to see if you have any leaks:

for semid in `ipcs -s | awk '/^0x/ {if ($1=="0x00000000") print $2}'`; do ipcs -s -i $semid; done | egrep -v "se               maphore|uid|mode|nsems|otime|ctime|semnum" | awk '{if ($5 != "0" && $5 != "") print $5}' | uniq | xargs ps -p