File read error (13)/Database server is down while logging in Netbackup

[imashu]

Hi,

While i try logging in netbackup it was giving me error File read error (13) and further by logging in through putty it says that Database server is down.

I searched the internet and found some useful technotes whic suggested that this may be due to space issue on the EMM server, which eventually was correct.

We found that the space on our master server was 100% full. We created some space and were able to login into the java console and putty session.

Our environment is Netbackup 6.5.6 and Master/EMM server is on HP-UX 11.23.

We are facing this issue regularly since 2-3 days. Later with the help of OS teams we found out that /opt/netbackup is getting alomost full everyday.

It seems /temp is trying to write in /opt and eventually makes it full.

It was approx 80% usage in the morning and now it's full after sometime it again reduces.

Below is the backup that is taking very large space as high as 30GB. 

/opt/openv/netbackup/db/images/bze1opt02

 bze1opt02 is a windows server and is backed up through a flashbackup policy. I believe we can't delete this.

find /opt -size +1000000 -exec ls -l {} \;
-rw------- 1 root sys 1190805192 Dec 21 08:38 /opt/openv/netbackup/db/images/bze1opt02/1338000000/catstore/Flashbackup_bze1opt02_E_1338827449_FULL.f_imgRecord0
-rw------- 1 root sys 1246685256 Dec 21 08:32 /opt/openv/netbackup/db/images/bze1opt02/1346000000/catstore/Flashbackup_bze1opt02_E_1346769076_FULL.f_imgRecord0
-rw------- 1 root sys 1222475832 Dec 21 08:34 /opt/openv/netbackup/db/images/bze1opt02/1344000000/catstore/Flashbackup_bze1opt02_E_1344270646_FULL.f_imgRecord0
-rw------- 1 root sys 1269301032 Dec 21 08:30 /opt/openv/netbackup/db/images/bze1opt02/1349000000/catstore/Flashbackup_bze1opt02_E_1349178255_FULL.f_imgRecord0
-rw------- 1 root sys 1129565520 Dec 21 08:20 /opt/openv/netbackup/db/images/bze1opt02/1328000000/catstore/Flashbackup_bze1opt02_E_1328549457_FULL.f_imgRecord0
-rw-rw-rw- 1 root sys 730595328 Dec 21 09:06 /opt/openv/netbackup/db/images/bze1opt02/1330000000/catstore/Flashbackup_bze1opt02_E_1330968645_FULL.f_imgRecord0
-rw------- 1 root sys 1206493992 Dec 21 08:36 /opt/openv/netbackup/db/images/bze1opt02/1341000000/catstore/Flashbackup_bze1opt02_E_1341246653_FULL.f_imgRecord0
-rw------- 1 root sys 1280256696 Dec 21 08:28 /opt/openv/netbackup/db/images/bze1opt02/1350000000/catstore/Flashbackup_bze1opt02_E_1350932431_FULL.f_imgRecord0
-rw------- 1 root sys 1298672640 Dec 21 08:14 /opt/openv/netbackup/db/images/bze1opt02/1354000000/catstore/Flashbackup_bze1opt02_E_1354571083_FULL.f_imgRecord0
-rw------- 1 root sys 1287043992 Dec 21 08:24 /opt/openv/netbackup/db/images/bze1opt02/1352000000/catstore/Flashbackup_bze1opt02_E_1352219015_FULL.f_imgRecord0
-rw------- 1 root sys 1289529864 Dec 21 08:22 /opt/openv/netbackup/db/images/bze1opt02/1352000000/catstore/Flashbackup_bze1opt02_E_1352757837_FULL.f_imgRecord0
-rw------- 1 root sys 1301518728 Dec 11 06:03 /opt/openv/netbackup/db/images/bze1opt02/1355000000/catstore/Flashbackup_bze1opt02_E_1355160760_FULL.f_imgRecord0
-rw------- 1 root sys 1304507520 Dec 18 06:07 /opt/openv/netbackup/db/images/bze1opt02/1355000000/catstore/Flashbackup_bze1opt02_E_1355765445_FULL.f_imgRecord0
-rw------- 1 root sys 1283349024 Dec 21 08:26 /opt/openv/netbackup/db/images/bze1opt02/1351000000/catstore/Flashbackup_bze1opt02_E_1351531853_FULL.f_imgRecord0
-rw------- 1 root sys 1292672808 Dec 21 08:19 /opt/openv/netbackup/db/images/bze1opt02/1353000000/catstore/Flashbackup_bze1opt02_E_1353347682_FULL.f_imgRecord0
-rw------- 1 root sys 1294851600 Dec 21 08:16 /opt/openv/netbackup/db/images/bze1opt02/1353000000/catstore/Flashbackup_bze1opt02_E_1353691846_FULL.f_imgRecord0

 Could you please let me know what is the issue behind this and how can we resolve this.

[Marianne · Accepted solution]

It is perfectly normal for images database to take up lots of space.

The reason why we implement Flashbackup is because we have clients with millions of small files.
Although Flashbackup does a backup of a single raw filesystem, each path, each filename needs to be cataloged. 
Daily Full backups and long retention periods will increase space requirements in images database.

You need to confirm backup requirements: full vs incremental and retention periods.

If this cannot be changed, you need to increase disk space on the master.

AND PLEASE upgrade your environment! NBU 6.x reached EOSL on 3 Oct 2012...

[imashu]

Thanks Marianne!!

Our full backup backs up 1100Gb of data while on a daily basis 300GB of data is being backed up by this Flashbackup policy.

Earlier this used to work since 10-11 months but don't know why this is giving issues.

It seems /temp is trying to write in /opt and eventually makes it full.

I have manually started Image cleanup job which is taking too long to get completed.

Can we do some settings at Netbackup or OS side to resolve this as increasing disk space is the last option for us because the data had been unchnaged since long.