Cannot lstat Updown

[rookie11]

HI 

master NBU , media server windows 2003 version7.1

2 clients in 2 different policy , having multistreaming on 8 streams running from each client ,7 runs fine  but 1 from each shows error in logs.

client 1 solaris 5.7  

error :

 ERR - Cannot lstat Updown.class. Errno = 5: I/O error

 ERR - Cannot lstat UpdownN.class. Errno = 5: I/O error

 ERR - Cannot lstat ReqERateN.class. Errno = 5: I/O error

linux 2.4 ,client 2

error :

ERR - Cannot lstat Version-2971745. Errno = 5: Input/output error

ERR - Cannot lstat text-plain. Errno = 5: Input/output error

ERR - Cannot lstat application-msword. Errno = 5: Input/output error

ERR - Cannot lstat application-pdf. Errno = 5: Input/output error

ERR - Cannot lstat application-vnd.illustrator. Errno = 5: Input/output error

ERR - Cannot lstat application-octet-stream. Errno = 5: Input/output error

[Marianne]

Can you run ls -l from within the directory where the filename(s) resides?

See http://www.symantec.com/docs/TECH38434

[mph999]

Excellent post from Marianne, I agree, most likely a filesystem issue. What is odd, is that it is on two separate clients, at the same time. Also, it is on two separate os. Considering these facts, it is difficult to pin this down, until you consider the fact that lstat is an operating system function, not nbu The common factor is /u10, so I think the most likely cause is some issue With the filesystem. Martin

[rookie11]

multistreaming is on in client1 only stream name client1_d8 shows status 1. compression is also on in policy. below is bpbkar from client log

18:09:49.980 [17462] <4> bpbkar: INF - BACKUP START

18:09:49.980 [17462] <4> bpbkar: INF - Estimate:-1 -1

18:09:50.018 [17462] <4> bpbkar: INF - Processing /oracle/client1_d8

18:09:55.152 [17500] <4> bpbkar main: real locales <C>

18:09:55.153 [17500] <4> bpbkar main: standardized locales - lc_messages <C> lc_ctype <C> lc_time <C> lc_collate <C> lc_numeric <C>

18:09:55.172 [17500] <2> logparams: bpbkar32 -r 3628800 -ru root -dt 86704 -to 0 -clnt client1_d8-class Biz_Ux -sched Differential-Inc -st INCR -bpstart_to 300 -bpend_to 300 -read_to 500 -stream_count 9 -stream_number 9 -jobgrpid 150090 -ckpt_time 900 -blks_per_buffer 512 -tir -use_otm -fso -Z -b client1_d8_1327014428 -kl 35 -use_ofb

18:09:55.186 [17500] <4> bpbkar: INF - Setting network send buffer size to 32032 bytes

18:09:55.187 [17500] <4> bpbkar: INF - setenv KEYWORD=NONE

18:09:55.187 [17500] <4> bpbkar: INF - setenv STREAM_PID=17500

18:09:55.187 [17500] <4> bpbkar: INF - setenv STREAM_NUMBER=9

18:09:55.187 [17500] <4> bpbkar: INF - setenv STREAM_COUNT=9

18:09:55.188 [17500] <4> bpbkar: INF - setenv STREAMS=0

18:09:55.188 [17500] <4> bpbkar: INF - setenv BPSTART_TIMEOUT=300

18:09:55.188 [17500] <4> bpbkar: INF - setenv BPEND_TIMEOUT=300

18:09:55.188 [17500] <4> bpbkar: INF - setenv RESTARTED=0

18:09:55.188 [17500] <4> bpbkar: INF - setenv BACKUPID= client1_d8_1327014428

18:09:55.188 [17500] <4> bpbkar: INF - setenv UNIXBACKUPTIME=1327014428

18:09:55.189 [17500] <4> bpbkar: INF - setenv BACKUPTIME=Thu Jan 19 18:07:08 2012

18:09:55.191 [17500] <4> bpbkar: INF - BACKUP START

18:09:55.191 [17500] <4> bpbkar: INF - Estimate:-1 -1

18:09:55.228 [17500] <4> bpbkar: INF - Processing /oracle/ client1_d9

18:09:55.390 [17500] <4> bpbkar: INF - Excluded /oracle/ client1_d9/oradata/ECOM/TEMP1.dbf.gz by exclude_list entry /oracle/ client1_d9/oradata/ECOM/*

18:09:55.391 [17500] <4> bpbkar: INF - Excluded /oracle/ client1_d9/oradata/ECOM/users02.dbf.gz by exclude_list entry /oracle/ client1_d9/oradata/ECOM/*

18:09:55.391 [17500] <4> bpbkar: INF - Client completed sending data for backup

18:09:55.395 [17500] <4> bpbkar: INF - bpbkar exit normal

18:09:55.395 [17500] <4> bpbkar: INF - EXIT STATUS 0: the requested operation was successfully completed

18:09:55.398 [17500] <4> bpbkar: INF - setenv FINISHED=1

18:10:01.444 [17462] <16> bpbkar: ERR - Cannot lstat Updown.class. Errno = 5: I/O error

18:10:01.445 [17462] <16> bpbkar: ERR - Cannot lstat UpdownN.class. Errno = 5: I/O error

18:10:01.445 [17462] <16> bpbkar: ERR - Cannot lstat ReqERateN.class. Errno = 5: I/O error

18:10:25.960 [17462] <4> bpbkar: INF - Excluded /oracle/client1_d8/lost+found by exclude_list entry lost+found

18:10:25.975 [17462] <4> bpbkar: INF - Excluded /oracle/client1_d8/oradata/ECOM/control03.ctl.gz by exclude_list entry /oracle/client1_d8/oradata/ECOM/*

18:10:25.976 [17462] <4> bpbkar: INF - Excluded /oracle/client1_d8/oradata/ECOM/redo_ECOM_02b.log.gz by exclude_list entry /oracle/client1_d8/oradata/ECOM/*

18:10:25.976 [17462] <4> bpbkar: INF - Excluded /oracle/client1_d8/oradata/ECOM/redo_ECOM_03.log.gz by exclude_list entry /oracle/client1_d8/oradata/ECOM/*

18:10:25.976 [17462] <4> bpbkar: INF - Excluded /oracle/client1_d8/oradata/ECOM/tools01.dbf.gz by exclude_list entry /oracle/client1_d8/oradata/ECOM/*

18:10:25.977 [17462] <4> bpbkar: INF - Excluded /oracle/client1_d8/oradata/ECOM/tools02.dbf.gz by exclude_list entry /oracle/client1_d8/oradata/ECOM/*

18:10:25.977 [17462] <4> bpbkar: INF - Excluded /oracle/client1_d8/oradata/ECOM/users01.dbf.gz by exclude_list entry /oracle/client1_d8/oradata/ECOM/*

18:10:25.978 [17462] <4> bpbkar: INF - Client completed sending data for backup

18:10:25.984 [17462] <16> bpbkar: ERR - bpbkar exit status = 1: the requested operation was partially successful

[Mark_Solutions]

I see that your backup does not actually fail, just ends with Status 1, partailly successful.

The tech note Marianne listed covers pretty much the same error and indicates that it is a file system issue with advice on dealing with it.

As Martin says it is strange that you have 2 systems with the same issue unless there is shared mounted disk somewhere between them or you have had a power outage recently and there two had their file systems damaged?

[Marianne]

Have you tried 'ls -l' on the client on these specific files yet?

[rookie11]

stream name on client 1 : /oracle/client1_d8. on 2nd client stream name is /u10.

@marianne did u mean ls -l inside partition /oracle/client_d8 or u mean ls -l <filename>. sorry i didn't get u.

on which file ur asking to run ls -l  ???????????

[mph999]

Do ls -l on the files themselves ...

18:10:01.444 [17462] <16> bpbkar: ERR - Cannot lstat Updown.class. Errno = 5: I/O error

18:10:01.445 [17462] <16> bpbkar: ERR - Cannot lstat UpdownN.class. Errno = 5: I/O error

18:10:01.445 [17462] <16> bpbkar: ERR - Cannot lstat ReqERateN.class. Errno = 5: I/O error

eg

ls -l  ReqERateN.class

Regards,

martin

[Marianne]

Or run 'ls -l' on the directory where the files reside.

If you Google 'cannot lstat file' you will find instances where files have characters that cannot be interpreted correctly by a Unix shell, such as \ , ? , etc.

[rookie11]

few steps i have take are pasted on the log file.

please help

[Will_Restore]

Solaris 7 ?? 

What version NetBackup client? 

[rookie11]

client 1 solaris 8 NBU 6.5

[Marianne]

Do you have Compression selected in the policy attributes?

[rookie11]

yes compression is ON in the policy

[Marianne]

So, when you do bpbkar on the client without compression, files can be read successfully.

Is there any particular reason why you are doing client side compression?

I have never seen any real benefit for this option other than clients on a very slow (10BaseT network).

Client side compression is extremely memory intensive and in most cases slows down backups significantly.

Seems we now know what is causing the 'cannot lstat' problem.

You may have discovered a bug with the particular version of NBU, but you will probably be told that it is fixed in a later version that is not supported on the clients' old OS...

Easiest will be to deselect compression in the policy and see if backup goes through without the 'lstat' errors.

[rookie11]

even when compression is off , getting the same error cannot lstat

[Mark_Solutions]

Have you done a disk check on these yet?

[rookie11]

yes unix SA has done disk check infact they replaced the disk but still  netbackup shows same error

[Yasuhisa_Ishika]

Error of lstat call is fundamentaly caused by filesystem error.
You also have to check and correct filesystem. Run fsck for this FS.

[Will_Restore · Accepted solution]

Typically, the lstat or lstat64 message followed by an "Errno = 5: I/O error" indicates a file system or hardware error. Another potential cause is when a file is removed during the backup.

 URL http://www.symantec.com/docs/TECH38434