/Storage filled up on MBE

[Lance_the_Boil]

Hi,

I'm running Puredisk 6.6

I have 3 nodes (SPA, CR and MBE).

The /Storage partition on the MBE has filled despite it being scaled sufficiently. When I look I can see that the vast bulk of the data is in /Storage/tmp/quarantine - it has 172GB out of the 200GB of data.

The data is in files that start as bulkInsertPO* and prexxxxxxxxx.tmp files

I have an outstanding critical error logged with Symantec but they are deciding to not answer the phone or ring me back.

Can anyone here identify if any of these files can be deleted without causing any issues with the data.

I am happy to lose any temp data from backup jobs that have not fully completed, but cannot lose data from previous full backups.

Thanks,

Tim.

[S_Williamson]

Hi

Follow this to free up some space

http://www.symantec.com/business/support/index?page=content&id=TECH66776&key=52672&actp=LIST

Else anything in /Storage/tmp can generally be deleted. The BulkInserts are usually from importing jobs, if there is 1 particular file that is really large might be worth just deleting it. If you do delete anything just make sure there is no active jobs running else you might break one of those. Always terminate gracefully.

Checking one of my PD SPA's in /Storage/tmp/quarantine is 168MB (8TB Storage Pool)

total 171608
-rw-r--r-- 1 root     root        619563 Apr 14 23:28 bulkInsertPO1723345902236611781
-rw-r--r-- 1 root     root       4993418 Apr 15 07:30 bulkInsertPO4987775422992872416
-rw-r--r-- 1 root     root     114501930 Apr 14 23:28 bulkInsertPO5549656096251953483
-rw-r--r-- 1 root     root      55122805 Apr 15 07:30 bulkInsertPO6662917363342972757
-rwxrwxr-x 1 www-data www-data    390613 Mar 29 12:00 bulkInsertPO7705782156767067342
-rwxrwxr-x 1 www-data www-data         0 Mar 29 12:00 errsplit1305306962531784366
-rw-r--r-- 1 root     root             0 Apr 15 07:30 errsplit2574260245056201682
-rwxrwxr-x 1 www-data www-data         0 Mar 29 12:01 errsplit3404573549386764255
-rw-r--r-- 1 root     root             0 Apr 14 23:28 errsplit868707632069837191
-rwxrwxr-x 1 www-data www-data     45056 Oct 27  2010 pre4885860012418562252.tmp
-rwxrwxr-x 1 www-data www-data     16384 Mar 15 06:53 pre704065592789438229.tmp

Simon

[Lance_the_Boil]

Hi,

I have removed all data from /Storage/tmp/quarantine (moved to an archive drive just in case) so have removed the issues with space.

However my backups no longer function, nor do any data removal jobs.

They all fail on the MBE stage.

The error is a routing error and always to a file called 647.recommended. No matter what the job is

[2011-May-05 08:23:28 BST]Error: 33 : Route table /Storage/var/rt/647.recommended contains no routes.

[2011-May-05 08:23:28 BST]Error: 33 : Router initialization failed: Route table /Storage/var/rt/647.recommended contains no routes.

[2011-May-05 08:23:28 BST]Error: 33 : Initialization failed: specified route table does not exist, is not readable or contains no entries

And don't even get me started on how I have a call logged with Symantec as critical and 48 hours later not a squeak. I actually spnt over an hour on hold after one of their people rang me in response to my boss moaning at them and I was promised to speak to the next available engineer. That's why I'm asking these question that should go to support.

Any help would be greatly appreciated as the support is now so poor I don't expect any Symantec help with this.

Thanks,

Tim.

[Lance_the_Boil]

it seems that 647.recommended was blank. I have done the folloiwng to replace it

cat 647.current > 647.recommended

and will post here what happens next.

[Lance_the_Boil]

it seems that 647.recommended was blank. I have done the folloiwng to replace it

cat 647.current > 647.recommended

and will post here what happens next.

[S_Williamson]

Hi

I havent seen this issue before but looking at all the files in that folder for me, they are both the same, none of them are empty

-rwxrwxr-x 1 www-data www-data 14 Oct 12  2009 1.current
-rwxrwxr-x 1 www-data www-data 14 Oct 12  2009 1.recommended
-rwxrwxr-x 1 www-data www-data 40 Apr 20 18:48 2100.current
-rwxrwxr-x 1 www-data www-data 40 Apr 20 18:48 2100.recommended
-rwxrwxr-x 1 www-data www-data 42 Nov 12  2009 362_1310.current
-rwxrwxr-x 1 www-data www-data 42 Nov 12  2009 362_1310.recommended
-rwxrwxr-x 1 www-data www-data 42 Nov 14  2009 362_1452.current
-rwxrwxr-x 1 www-data www-data 42 Nov 14  2009 362_1452.recommended
-rwxrwxr-x 1 www-data www-data 42 Nov 21  2009 362_2056.current
-rwxrwxr-x 1 www-data www-data 42 Nov 21  2009 362_2056.recommended
-rwxrwxr-x 1 www-data www-data 42 Nov 28  2009 362_2709.current
-rwxrwxr-x 1 www-data www-data 42 Nov 28  2009 362_2709.recommended
 

[Lance_the_Boil]

After a few days Syamntec support eventually decided that they would help me with my 2-hour response critical incident.

They told me that the Server DB Maintenance workflow will clear out the /Storage/tmp directory.

I must admit I'm not convinced about that but as deleting all the files in quaratine seemed to fix the problem all seems well.

I got an apology from the support team leader who has promised me that the support issues for Puredisk will be resolved soon as they are recruiting more staff.

Thanks for your suggestions.

[f25 · Accepted solution]

Hi!

Please log on as "root" and go to reports. Check if MBE is 90% full in "Capacity Report" (or something like that).

The MBE needs not only the disk space but also is limited by number of file entires. 400M is max when PureDisk becomes dead.

To make PD alive (only if you have reached the 90% of MBE utilisation and are unable to do anything): [extreeme and emergency solution's summary, data loss will occur]: try to delete minor data selection (i.e. one C: drive), run maintenance jobs. Stop not using archive bit.

It may be good to try to install MBE on other nodes and reroute it gaining more total MBE capacity.

Do not run jobs not using archive bit. Avoid running manual backup jobs (that will always not use the archive bit).

In case of disk capacity issues: all stuff under /Storage/tmp older than 2-3 days may "go bananas" ;) [logs and history may be lost but files should be restorable]

BR, Michał