cancel
Showing results for 
Search instead for 
Did you mean: 

Database server is down, check for low disk space(4005006)

Plusnet
Level 2

Hi,

I am very new to the Netbackup appliance so please bear with me

I was adding new backups to the device, following documentation from someone who no longer works for us. I misconfigured a backup so that it was backing up to something called diskstage (a BasicDisk) which subsequently filled up.

I noticed this once random backups started failing, fixed it and expired the images I did not need using bpexpdate.

However, this did not free any space (the cleanup jobs were failing but i do not have access to the error any more) and now, when I try and do anything I get an error something along the lines of:

database system error - Database server is down, check for low disk space(4005006)

Now, I am pretty sure from reading many posts here, that the database has shut itself down to preserve data integrity. Everything says "free up some space and start it again"

What I can't find is any documentation on how you free up disk space or similar, thus allowing me to start the database again and delete whatever is filling the disk up using the conventional tools

Can you point me in the direction of how you access the filesystem directly on the appliance? Is there a way of doing it via the CLI? Maybe a way of mounting it up somewhere else?

I just want to compress enough logs and such that I can start the database again, at while point the automatic management should clear it down.

Josh

1 ACCEPTED SOLUTION

Accepted Solutions

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Image Cleanup has probably cleaned up the expired images. 

You can try to stop and restart NBU with:

netbackup stop
(wait for all processes to stop)
netbackup start
(wait for all processes to start)
Check that NBU is running: 
bpps -x

View solution in original post

8 REPLIES 8

andrew_mcc1
Level 6
   VIP   

Something strange here, NetBackup Appliances don't have BasicDisk, they support MSDP and AdvancedDisk for normal backups. You may want to open a call with Veritas if there is limited NetBackup or Appliance knowledge available.

Good luck, Andrew

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@Plusnet 

Do you have another media server in the environment?
Maybe this is where the BasicDisk is configured.

Please open the properties for the Storage Unit and check the config.
What is the name of the media server?
What is the path for the storage?

I am also curious about how exactly you expired the images. 
Can you tell us?

The error that you are seeing now sounds like the root filesystem might be full.
I totally agree with @andrew_mcc1  - please log a Support call with Veritas for assistance with cleanup of logs. 

 

Hi

I cannot see another media server. When I go through the Java console and look at 'Media and Device Management' -> 'Devices' -> 'Media Servers' I see only the one device (which is also listed as a master server)

I've attached a screenshot of the appropriate disk from our other netbackup device that is still working. The problem one is the same but was 99% full (and I can no longer see it due to the database being down)

The command I used for expiring the images was this (the backupid differing of course). They were set to expire in 2 weeks or so and I wanted to drop that to immediately:

bpexpdate -backupid webmaildb06.d.int.plus.net_1553201425 -d 0 -force

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Apologies, I misunderstood :)  I thought you said "I did not need using bpexpdate.
Reading the entire sentence now again makes perfect sence.... :) 

Is 'netbackup02' the Appliance and the only master/media server?

This type of STU is dangerous - especially with 'directory can exist on root filesystem'. 
There is also no staging configure that will remove duplicated images when HWM is is reached.

If your root filesystem is full, NBU will shutdown EMM to prevent database corruption.
Without EMM running, you cannot expire images. 

Please contact Support for assistance with cleanup... 

Sorry, mis-typed that

I know it is set up wrong and I will raise a piece of work to fix the disk being used to move it off the root disk. I had nothing to do with the original build of the platform so my knowledge is limited.

I have actually just found maintenance mode in the shell, and having looked at this the disk is NOT full (or not full any more at least). this disk is the right size so I am pretty sure it is the right disk

maintenance-!> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root
90G 39G 47G 46% /

So do I just need to start the database process again? Is there something you can run that just validates which process are running and starts them if they are not?

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Image Cleanup has probably cleaned up the expired images. 

You can try to stop and restart NBU with:

netbackup stop
(wait for all processes to stop)
netbackup start
(wait for all processes to start)
Check that NBU is running: 
bpps -x

Worked it out, I just needed to run this from the CLI

netbackup start

The images must have been aged out of that directory as I hoped, but not quick enough to stop the database from killing itself to avoid corruption.

Thanks very much for you assistance, and I will definately look to get that disk moved to some other storage!

andrew_mcc1
Level 6
   VIP   

I know it is set up wrong and I will raise a piece of work to fix the disk being used to move it off the root disk

Well as Appliances only support AdvancedDisk and MSDP you can't just move that Storage Unit elsewhere on the Appliance (at least not with Veritas support). It is normal to use MSDP on Appliances, do you know any reason why a BasicDisk Storage Unit was created? Is MSDP also full? If so you will probably have to reduce backup retentions or frequencies etc. or acquire more storage (which for an Appliance needs to be from Veritas).

Andrew