cancel
Showing results for 
Search instead for 
Did you mean: 

Netbackup 7.1.0.4 - MSDP storaged CRITICAL ERROR

Michele_Ghiglio
Level 3

Hi all,

I need assitance because after a full file system of my MSDP server I've got this message in the storaged.log:

 

Reason     : ERROR:  could not extend relation 1663/16387/270426: No space left on device
HINT:  Check free disk space.
June 28 01:26:33 ERR [140021850629904]: 25004: Could not process spool entry 3018209: unknown error
June 28 01:26:33 ERR [140021850629904]: 25004: Could not process tlog entries: unknown error
June 28 01:26:49 WARNING [140021850629904]: 25000: Transaction log /nbustage1/queue/sorted-498964-499079.tlog failed: Could not process tlog entries: unknown error
 Transaction will be retried.
June 28 01:26:49 ERR [140021850629904]: 25004: Queue processing failed five times in a row. Queue processing will be disabled and the CR will no longer accept new backup data. Content router has been totally shut down. Please contact support immediately!
 
Thanks in advance

 

1 ACCEPTED SOLUTION

Accepted Solutions

V4
Level 6
Partner Accredited

Provide output of below syntax

--crcontrol queueinfo

Also perform below steps to release storage space

Check compaction status

--crcontrol compactstate

Turn Compaction ON

--crcontrol compacton

Start Compaction

--crcontrol compactstart

Check Process queue info and process the queue if pending

--crcontrol processqueueinfo

output should be no for busy and pending if is yes, execute below syntax and check . do it till queue is turned to NO

--crcontrol processqueue

 

 

 

 

View solution in original post

17 REPLIES 17

V4
Level 6
Partner Accredited

OS?

error says no space. Verify MSDP storage have atleast 10% free space left  if not get it done first.

Michele_Ghiglio
Level 3

OS
SLES 11 SP1 (Media Server + MSDP)

Some additional info about space allocation:

DF
nbme3:~ # df -h /nbustage1
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg01-lvol1
                      4.9T  4.6T   76G  99% /nbustage1

CRCONTROL
nbme3:~ # crcontrol --dsstat

************ Data Store statistics ************
Data storage      Raw    Size   Used   Avail  Use%
                   4.8T   4.6T   4.6T   0.0M 100%

Number of containers             : 19739
Average container size           : 246359744 bytes (234.95MB)
Space allocated for containers   : 4862894990815 bytes (4.42TB)
Space used within containers     : 4846993964232 bytes (4.41TB)
Space available within containers: 15901026583 bytes (14.81GB)
Space needs compaction           : 193103422413 bytes (179.84GB)
Reserved space                   : 299208470528 bytes (278.66GB)
Reserved space percentage        : 5.6%
Records marked for compaction    : 4166145
Active records                   : 86007957
Total records                    : 90174102

Use "--dsstat 1" to get more accurate statistics

For your information yesterday the volume was completley full.

So I copied the database folder to a different mount point and I create a symbolik link in the original location (in my environment is /nbustage1).

thks

mic

 

 

V4
Level 6
Partner Accredited

Provide output of below syntax

--crcontrol queueinfo

Also perform below steps to release storage space

Check compaction status

--crcontrol compactstate

Turn Compaction ON

--crcontrol compacton

Start Compaction

--crcontrol compactstart

Check Process queue info and process the queue if pending

--crcontrol processqueueinfo

output should be no for busy and pending if is yes, execute below syntax and check . do it till queue is turned to NO

--crcontrol processqueue

 

 

 

 

Michele_Ghiglio
Level 3

nbme3:~ # crcontrol --queueinfo
total queue size : 621840462
creation date of oldest tlog : Tue Jun 26 02:50:59 2012

nbme3:~ # crcontrol --compactstate
Data store compaction: ON, DeleteSpaceThreshold: 30%, CompactLBound: 4MB
Compaction busy: No
nbme3:~ # crcontrol --compacton
Data store compaction turned on
nbme3:~ # crcontrol --compactstart
Data store compaction started
nbme3:~ # crcontrol --processqueueinfo
Busy   : no
Pending: no
nbme3:~ # crcontrol --processqueueinfo
Busy   : no
Pending: no
nbme3:~ #

 

V4
Level 6
Partner Accredited

now check with dsstat 1 . does it show reduction in storage space used by MSDP

Michele_Ghiglio
Level 3

Unfortunately no reduction.

nbme3:/nbustage1/log/spoold # crcontrol --dsstat 1

************ Data Store statistics ************
Data storage      Raw    Size   Used   Avail  Use%
                   4.8T   4.6T   4.6T   0.0M 100%

Number of containers             : 19739
Average container size           : 237690565 bytes (226.68MB)
Space allocated for containers   : 4691774081947 bytes (4.27TB)
Space used within containers     : 4664571389123 bytes (4.24TB)
Space available within containers: 27202692824 bytes (25.33GB)
Space needs compaction           : 164865344893 bytes (153.54GB)
Reserved space                   : 299208474624 bytes (278.66GB)
Reserved space percentage        : 5.6%
Records marked for compaction    : 3637979
Active records                   : 82596678
Total records                    : 86234657

V4
Level 6
Partner Accredited

could you perform reboot of MSDP server. if still doesn't work .time to call Support on this. we had  similar cases where reboot did freed space to perform MSDP maintenance but if not then support is  the best choice to get assistance

Michele_Ghiglio
Level 3

some space was freed. Is it a good news?

Thanks

 

crcontrol --dsstat

************ Data Store statistics ************
Data storage      Raw    Size   Used   Avail  Use%
                   4.8T   4.6T   4.4T 139.6G  97%

Number of containers             : 19739
Average container size           : 236416977 bytes (225.46MB)
Space allocated for containers   : 4666634712274 bytes (4.24TB)
Space used within containers     : 4639130276099 bytes (4.22TB)
Space available within containers: 27504436175 bytes (25.62GB)
Space needs compaction           : 139424231869 bytes (129.85GB)
Reserved space                   : 295743315968 bytes (275.43GB)
Reserved space percentage        : 5.6%
Records marked for compaction    : 2996688
Active records                   : 82596678
Total records                    : 85593366

Use "--dsstat 1" to get more accurate statistics

df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg01-lvol1
                      4.9T  4.4T  260G  95% /nbustage1
 

V4
Level 6
Partner Accredited

as you have got free space. now start processqueue , start compaction and check if you get further reduction

this is time consuming process. so it might not reflect immediately so perform it time interval

Michele_Ghiglio
Level 3

At this moment the situation on the MSDP we wrote about yesterday, nbme3, is ok.


The only problem is that also my other MSDP server has the same problems.
I will apply the to-do-list above and I will update you.

Above I recap every action I did:

0. Got first out of space error.
1. Moved the databases dir to a different file system.
2. Created a symlink to new location.
3. Got a second out-of-space error with scary message in storaged.log.
4. Exectued ervy command you told me:
 4.1 crcontrol --compacton
 4.2 crcontrol --compactstart
 4.3 crcontrol --
processqueueinfo
5. Verified erything it's working fine.
6. Resetted the default *SpaceThreshold in contentrouter.cfg. (yes I know I shouldn't have to touch this but unfortunately I did it).
7. Removed the reserved blocks with tune2fs on the msdp file system.

At this moment everything is ok.

Many many thanks.

Michele

 

LOG

All my MSDP/PDDE logs don't show any error.

DF


nbme3:~ # df -h
Size  Used Avail Use% Mounted on
4.9T  4.4T  528G  90% /nbustage1
70G   40G   27G  60% /usr
 
CRCONTROL
nbme3:~ # crcontrol --dsstat
 
************ Data Store statistics ************
Data storage      Raw    Size   Used   Avail  Use%
                   4.8T   4.3T   4.2T  95.9G  98%
 
Number of containers             : 19739
Average container size           : 236257672 bytes (225.31MB)
Space allocated for containers   : 4663490193095 bytes (4.24TB)
Space used within containers     : 4639130276099 bytes (4.22TB)
Space available within containers: 24359916996 bytes (22.69GB)
Space needs compaction           : 140331074177 bytes (130.69GB)
Reserved space                   : 627816472576 bytes (584.70GB)
Reserved space percentage        : 11.8%
Records marked for compaction    : 3004528
Active records                   : 82588838
Total records                    : 85593366
 
Use "--dsstat 1" to get more accurate statistics
 
Many thanks again

 

V4
Level 6
Partner Accredited

If issue is resolved. Kindly mark the post which had helped

Michele_Ghiglio
Level 3

Hi Speedfreak,

The problem was solved, I marked the post.

But I'm worried about the fragility of my Netbackup, based on MSDP, architecture.

I can't get any evidence of this CRITICAL ERRORS in my Netbackup Administration Console and only the deep log analysis got me to the critical error in storaged.log.

I Netbackup I saw only some 84 and 2074 erorr, that my MSDP media server was active and the disk pool was full but active.

What do you think about this management problems?

Cheers

michele 

V4
Level 6
Partner Accredited

There are some cons as of now. I'm sure NBU team is working on this. However i believe we can use Alert facility in Opscenter to alert us upon threshold is reached.  I shall check it and get back to you on this. You may also suggest an idea under Connect section to improve NBU and it's feature/components etc.

V4
Level 6
Partner Accredited

Review the <msdp path>\log\spoold\spoold.log file.  Search for the word COMPACT to confirm it is running properly and to tell you when it has completed.

this would report if any issues with MSDP compaction

Michele_Ghiglio
Level 3

Hi 

I've found the COMPACT word with detailed info in spoold.log, it seems to work fine.

But I'm still having troubles with one of ths MSDP media server: out of space.

I'm working to activate a monitor of the free space to check it hourly.

 

I will update you.

cheers

michele

 

Ner0
Level 3
Partner Accredited

Hi All,

 

Can you Please help me with regards to my current issue about creating MSDP Dedupe Storage on Linux Media Server.

 

Thank You

V4
Level 6
Partner Accredited

please create new thread.... this multi issue thread would confuse others.. from searching solution on their isssue similar to urs.

 

also brief about ur issue including your environment details with  OS and all