Sis common store files may be getting locked during backup

[Pamela_S1]

Hi everyone.

I have a master-media server with netbackup 7.5.0.6 and a client on the same version. Both are windows 2008 r2, but the client is a storage server.

The client has a W:\ drive that presents a LUN from a NetAPP. On this W drive, we have a folder called "usuarios" with about 660 Gb and 1272856 files (full backup on weekends), and has Single Instance Storage enable, so in the backup we get all the "SIS Common Store" folder even if not included on the policy backup selections. The backup selections of the policy is:

W:\Usuarios

Couple of times the client became unavailable during backup, and after a memory dump and analysis, the diagnostic they have given me is:

Symantec Net Backup process bpbkar32.exe has the oldest waiting thread blocked on this file:

\Usuarios\username\nameofthefile.doc

 sisss.sys is involved in the blockage but seems was only passing the request to NTFS. So this driver might not be responsible

I have looked for some similar issues, haven´t found any, neither on the documentation. Anybody having similar issues or knows if this is really netbackup related (i´m guessing it is more a problem of the sisss.sys causing an error), but not so sure how to start analyzing this.

Thanks a lot,

Pamela.

[SymTerry]

I would like to get more information on this. Would it be possable to post the WINDBG analysis? If not would you be able to open a case with support?

This doesn't look like a direct NetBackup issue, but I would be glad to get this reviewed.  

[Yasuhisa_Ishika]

I believe bpbkar32.exe ***is blocked by*** any other component works at lower level. bpbkar32.exe itself runs in user-land so it is hard for bpbkar32. to bring the system outage. Please montior the CPU usage. If CPU time is spend in system, there might be something wrong in kernel and drivers.

sisss.sys is not only passing I/O to NTFS but managing deduplied files. sisss.sys is more responsible than user applications(bpbkar32.exe in this case) in case of I/O operation. Please review sisss.sys side.

[Pamela_S1]

Hi everyone,

Yes we had a case with support, but the diagnostic, which made sense, was:

Did not find any errors on OS level that may point to a NetBackup Client service (bpbkar) internal issue. According to the bpbkar log all backups were completed successfully, no problems were reported.

We reopened the case and have some tests to do with vssadmin and snapshots.

After some analysis, i think i have a idea of that is going on. On the W volume  we have something like this

• w:\users  (660 GB)
• w:\departments (1 TB)

And we have 2 policies, 1 for users and one for departments. The first policy is simple as "w:\users", no multistreaming, so when it is running, it is sequential. It gets all folder contents and then backups SIS Common store files even tough is not specified on the backup  selection.

But the second policy is multistreamed and has 1 stream per department, aprox (22 streams) and for each stream also backups the SIS Common Store files. So we have on a "single backup" many repetitions of the SIS Common Store. I understand that with 2 policies of the same volume, the SIS Common store could be repetead twice, but if for each stream it backups the SIS Common Store once, there is a really good chance that this is what could cause trouble and overload the sisss.sys because this backups take almost 24 hrs.

So now, i tried to exclude the SIS Common Store to backup it up on a separated policy, but it is not working. Any help on this will be really appreciated!

Thanks,

Pamela.

[Yasuhisa_Ishika · Accepted solution]

I have no experience with SIS so I'm not sure it is really true, but I think it is nature of SIS. Contents of SIS volume is backed in SIS Common store, so even if you don't explicitly include SIS Common Store, backup is redirected to SIS Vommon Store, and it will backed up.

http://msdn.microsoft.com/en-us/library/windows/desktop/aa362664(v=vs.85).aspx

Adding SIS Common Store into Exclude List may have no effect. Please consider to reduce Maximin jobs per policy or reorganize the backup streams to reduce the system load raised by sisss.sys.