06-28-2012 02:45 AM
Hi all,
I need assitance because after a full file system of my MSDP server I've got this message in the storaged.log:
Solved! Go to Solution.
06-28-2012 04:03 AM
Provide output of below syntax
--crcontrol queueinfo
Also perform below steps to release storage space
Check compaction status
--crcontrol compactstate
Turn Compaction ON
--crcontrol compacton
Start Compaction
--crcontrol compactstart
Check Process queue info and process the queue if pending
--crcontrol processqueueinfo
output should be no for busy and pending if is yes, execute below syntax and check . do it till queue is turned to NO
--crcontrol processqueue
06-28-2012 03:30 AM
OS?
error says no space. Verify MSDP storage have atleast 10% free space left if not get it done first.
06-28-2012 03:46 AM
OS
SLES 11 SP1 (Media Server + MSDP)
Some additional info about space allocation:
DF
nbme3:~ # df -h /nbustage1
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg01-lvol1
4.9T 4.6T 76G 99% /nbustage1
CRCONTROL
nbme3:~ # crcontrol --dsstat
************ Data Store statistics ************
Data storage Raw Size Used Avail Use%
4.8T 4.6T 4.6T 0.0M 100%
Number of containers : 19739
Average container size : 246359744 bytes (234.95MB)
Space allocated for containers : 4862894990815 bytes (4.42TB)
Space used within containers : 4846993964232 bytes (4.41TB)
Space available within containers: 15901026583 bytes (14.81GB)
Space needs compaction : 193103422413 bytes (179.84GB)
Reserved space : 299208470528 bytes (278.66GB)
Reserved space percentage : 5.6%
Records marked for compaction : 4166145
Active records : 86007957
Total records : 90174102
Use "--dsstat 1" to get more accurate statistics
For your information yesterday the volume was completley full.
So I copied the database folder to a different mount point and I create a symbolik link in the original location (in my environment is /nbustage1).
thks
mic
06-28-2012 04:03 AM
Provide output of below syntax
--crcontrol queueinfo
Also perform below steps to release storage space
Check compaction status
--crcontrol compactstate
Turn Compaction ON
--crcontrol compacton
Start Compaction
--crcontrol compactstart
Check Process queue info and process the queue if pending
--crcontrol processqueueinfo
output should be no for busy and pending if is yes, execute below syntax and check . do it till queue is turned to NO
--crcontrol processqueue
06-28-2012 04:50 AM
nbme3:~ # crcontrol --queueinfo
total queue size : 621840462
creation date of oldest tlog : Tue Jun 26 02:50:59 2012
nbme3:~ # crcontrol --compactstate
Data store compaction: ON, DeleteSpaceThreshold: 30%, CompactLBound: 4MB
Compaction busy: No
nbme3:~ # crcontrol --compacton
Data store compaction turned on
nbme3:~ # crcontrol --compactstart
Data store compaction started
nbme3:~ # crcontrol --processqueueinfo
Busy : no
Pending: no
nbme3:~ # crcontrol --processqueueinfo
Busy : no
Pending: no
nbme3:~ #
06-28-2012 05:00 AM
now check with dsstat 1 . does it show reduction in storage space used by MSDP
06-28-2012 07:12 AM
Unfortunately no reduction.
nbme3:/nbustage1/log/spoold # crcontrol --dsstat 1
************ Data Store statistics ************
Data storage Raw Size Used Avail Use%
4.8T 4.6T 4.6T 0.0M 100%
Number of containers : 19739
Average container size : 237690565 bytes (226.68MB)
Space allocated for containers : 4691774081947 bytes (4.27TB)
Space used within containers : 4664571389123 bytes (4.24TB)
Space available within containers: 27202692824 bytes (25.33GB)
Space needs compaction : 164865344893 bytes (153.54GB)
Reserved space : 299208474624 bytes (278.66GB)
Reserved space percentage : 5.6%
Records marked for compaction : 3637979
Active records : 82596678
Total records : 86234657
06-28-2012 07:25 AM
could you perform reboot of MSDP server. if still doesn't work .time to call Support on this. we had similar cases where reboot did freed space to perform MSDP maintenance but if not then support is the best choice to get assistance
06-28-2012 08:01 AM
some space was freed. Is it a good news?
Thanks
crcontrol --dsstat
************ Data Store statistics ************
Data storage Raw Size Used Avail Use%
4.8T 4.6T 4.4T 139.6G 97%
Number of containers : 19739
Average container size : 236416977 bytes (225.46MB)
Space allocated for containers : 4666634712274 bytes (4.24TB)
Space used within containers : 4639130276099 bytes (4.22TB)
Space available within containers: 27504436175 bytes (25.62GB)
Space needs compaction : 139424231869 bytes (129.85GB)
Reserved space : 295743315968 bytes (275.43GB)
Reserved space percentage : 5.6%
Records marked for compaction : 2996688
Active records : 82596678
Total records : 85593366
Use "--dsstat 1" to get more accurate statistics
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg01-lvol1
4.9T 4.4T 260G 95% /nbustage1
06-28-2012 07:10 PM
as you have got free space. now start processqueue , start compaction and check if you get further reduction
this is time consuming process. so it might not reflect immediately so perform it time interval
06-29-2012 11:12 AM
At this moment the situation on the MSDP we wrote about yesterday, nbme3, is ok.
The only problem is that also my other MSDP server has the same problems.
I will apply the to-do-list above and I will update you.
0. Got first out of space error.
1. Moved the databases dir to a different file system.
2. Created a symlink to new location.
3. Got a second out-of-space error with scary message in storaged.log.
4. Exectued ervy command you told me:
4.1 crcontrol --compacton
4.2 crcontrol --compactstart
4.3 crcontrol --processqueueinfo
5. Verified erything it's working fine.
6. Resetted the default *SpaceThreshold in contentrouter.cfg. (yes I know I shouldn't have to touch this but unfortunately I did it).
7. Removed the reserved blocks with tune2fs on the msdp file system.
At this moment everything is ok.
Many many thanks.
Michele
LOG
All my MSDP/PDDE logs don't show any error.
DF
06-29-2012 10:26 PM
If issue is resolved. Kindly mark the post which had helped
06-30-2012 06:04 AM
Hi Speedfreak,
The problem was solved, I marked the post.
But I'm worried about the fragility of my Netbackup, based on MSDP, architecture.
I can't get any evidence of this CRITICAL ERRORS in my Netbackup Administration Console and only the deep log analysis got me to the critical error in storaged.log.
I Netbackup I saw only some 84 and 2074 erorr, that my MSDP media server was active and the disk pool was full but active.
What do you think about this management problems?
Cheers
michele
06-30-2012 01:22 PM
There are some cons as of now. I'm sure NBU team is working on this. However i believe we can use Alert facility in Opscenter to alert us upon threshold is reached. I shall check it and get back to you on this. You may also suggest an idea under Connect section to improve NBU and it's feature/components etc.
07-01-2012 07:49 AM
Review the <msdp path>\log\spoold\spoold.log file. Search for the word COMPACT to confirm it is running properly and to tell you when it has completed.
this would report if any issues with MSDP compaction
07-03-2012 12:01 AM
Hi
I've found the COMPACT word with detailed info in spoold.log, it seems to work fine.
But I'm still having troubles with one of ths MSDP media server: out of space.
I'm working to activate a monitor of the free space to check it hourly.
I will update you.
cheers
michele
07-10-2012 05:07 PM
Hi All,
Can you Please help me with regards to my current issue about creating MSDP Dedupe Storage on Linux Media Server.
Thank You
07-10-2012 08:52 PM
please create new thread.... this multi issue thread would confuse others.. from searching solution on their isssue similar to urs.
also brief about ur issue including your environment details with OS and all