cancel
Showing results for 
Search instead for 
Did you mean: 

"Access is denied" from backup software when backing up NetApp placeholders

Sam_Wood
Level 4
NetApp filer, OnTap 7.0.2
EV 6 SP3
ArcServe or NTBackup

Archiving runs fine, we get placeholder creation on the NetApp filer after backup of the vault store partition as expected. Can retrieve files by opening placeholders.

However, when we try to backup the placeholders on the filer, we get "access is denied" errors.

No AV running against the filer, and nothing is logged on the EV server (nothing in event logs, no FSA reports in the reports folder that match the time of the backup run). All other files backup as normal.

From NTBackup:

Backup Status
Operation: Backup
Active backup destination: File
Media name: "Backup.bkf created 12/02/2007 at 9:07 a.m."

Volume shadow copy creation: Attempt 1.
Backup of "Z: \\nasprd01\vol_t_snapshot$"
Backup set #1 on media #1
Backup description: "Set created 12/02/2007 at 9:07 a.m."
Media name: "Backup.bkf created 12/02/2007 at 9:07 a.m."

Backup Type: Normal

Backup started on 12/02/2007 at 9:07 a.m..
Warning: Unable to open "Z:\hourly.0\xx\Information Systems\Inwards\xxWeb Content\xxTrade Folder 2002\Cover_Folder_?2.pdf" - skipped.
Reason: Access is denied.


Warning: Unable to open "Z:\hourly.0\xx\Information Systems\Inwards\xxWeb Content\xxTrade Folder 2002\Inside_Folder_?2.pdf" - skipped.
Reason: Access is denied.

Any one seen this before? Only thing I can think of is to recommend an upgrade to OnTap 7.2.x
12 REPLIES 12

Micah_Wyenn
Level 6
Sam,
An upgrade sounds like a good idea, but i don't know if that'll actually fix you. What account is the backup job running under (since it's a netapp it should need some sort of auth info)? Maybe the account isn't strong enough to grab those files?

micah

Sam_Wood
Level 4
Just checked with the customer; he can't upgrade to 7.2 because that'll break the Win2k servers connecting to the filer with iSCSI.

I'll check the permissions level on the account, but other files in the same dir get backed up with no errors - EV placeholders should have the same permissions.

Micah_Wyenn
Level 6
You'd think...but check it. Weirder shiznit has happened.

micah

Sam_Wood
Level 4
Checked permissions. Backup service account should be fine (and can open placeholders when logged into a workstation).

More symptoms:
NTBackup throws "access is denied" on placeholders when backing up the volume
NTBackup throws "access is denied" on placeholders when backing up a snapshot of the volume
ArcServe throws "access is denied" on placeholders when backing up the volume
ArcServe throws "access is denied" on placeholders when backing up a snapshot of the volume
The backup user account can open placeholders from the volume
The backup user account throws "access is denied" when opening placeholders from a snapshot of the volume

Tests:
We're able to back up the vault stores and all other files on the filer without any issues.
All users, and the account used to run the backups, can open the placeholders from the volume without issue.
We can back up placeholders on the Windows file server without issue.
Both ArcServe and NTBackup do the same thing.
No permissions errors (or errors of any kind) are logged by EV

I'm thinking this is something to do with the filer, not with us. Thoughts?

Micah_Wyenn
Level 6
Hmmmm....if you go to URL's, does it exhibit the same problem? I'm thinking it's in the way the netapp is talking to the placeholder service. Don't ask me what, I'm sorta guessing at this point.

micah

Sam_Wood
Level 4
Yeah, so am I. I've contacted the professional services company that's working with us on the NetApp side; we'll see what logs etc they can give me.

Pretty sure the placeholders themselves are fine - anyone can open up the archived files from the file system (including the backup service account). I'm discounting the errors opening placeholders from the snapshot image, I think that's a different issue.

Kopfjager
Level 5
Employee Certified
If the backup software is actually trying to recall the messages while backing them up, then I think you might be hitting the max recall limit which after 20 files in 10 seconds (default) will block recalls.

The question is whether they are recalling or not. You can check the IIS logs at the time of the backup and see if recalls were made at that time. I'm not sure if the NetAppPH service on the EV server generates a error message the same way the normal PH service does when this occurs.

Just a thought

Sam_Wood
Level 4
Thanks Paul. The NetApp engineer has said pretty much the same thing - if the backup agent was trying to recall the file to back it up, the filer would throw a permissions error because the snap-shot is set to read-only.

We're continuing with some testing (EVFSABackupmode, just to see if we get different errors from the backup agent).

Sam_Wood
Level 4
Edited dtrace of both PH services during a backup. It looks like the backup software is definitely recalling files.

There's a number of files like this one that say they're not a placeholder. Does that mean fpolicy on the filer is querying EV for each file to see if it has been archived during a read?

6022:13:46.542 (EvNetAppPHSvr)<7144>EV:LCArchiveSvr::CheckWorkerThreads (Exit) |Success |
6122:14:01.715 (EvNetAppPHSvr)<2740>EV:MFiler: NASPRD01.xxx.com requested the Fpolicy server status - returning OK
6222:14:11.465 (EvNetAppPHSvr)<2740>EV:LOpenFileWorkItem::OpenFileWorkItem (Entry) |
6322:14:11.465 (EvNetAppPHSvr)<880>EV:HFlatten::Flatten
6422:14:11.465 (EvNetAppPHSvr)<880>EV:MCFolderPermSyncWrapper SynchronizeEveryoneSID is 1
6522:14:11.465 (EvNetAppPHSvr)<880>EV:HCFolderPermSyncWrapper::SynchroniseFolder - Expand local groups:0
6622:14:11.465 (EvNetAppPHSvr)<2740>EV:LOpenFileWorkItem::OpenFileWorkItem (Exit) |Success |
6722:14:11.465 (EvNetAppPHSvr)<2740>EV:LCQueue::EnQueue (Entry) |
6822:14:11.465 (EvNetAppPHSvr)<2740>EV:MCQueue::EnQueue item enQueued
6922:14:11.465 (EvNetAppPHSvr)<7240>EV:MCQueue::DeQueue item deQueued, currently 1 items on the queue
7022:14:11.465 (EvNetAppPHSvr)<2740>EV:LCQueue::EnQueue (Exit) |Success |
7122:14:11.465 (EvNetAppPHSvr)<7240>EV:LCFileLock::Lock (Entry) |
7222:14:11.465 (EvNetAppPHSvr)<7240>EV:MThe exclusive FileLock was granted for the file: \\?\UNC\NASPRD01.xxx.COM\ONTAP_ADMIN$\VOL\VOL_T\xxx\INFORMATION SYSTEMS\INWARDS\ENZA WEB CONTENT\ENZA TRADE FOLDER 2002\INSIDE_FOLDER_?2.PDF
7322:14:11.465 (EvNetAppPHSvr)<7240>EV:LCFileLock::Lock (Exit) |Success |
7422:14:11.465 (EvNetAppPHSvr)<7240>EV:LOpenFileWorkItem::Process (Entry) |
7522:14:11.480 (EvNetAppPHSvr)<7240>EV:M The file \\NASPRD01\xxx\Information Systems\Inwards\ENZA Web Content\ENZA Trade Folder 2002\Inside_Folder_?2.pdf is not an Enterprise Vault placeholder and will be ignored
7622:14:11.543 (EvNetAppPHSvr)<7240>EV:LOpenFileWorkItem::Process (Exit) |Success |
7722:14:11.543 (EvNetAppPHSvr)<7240>EV:LCFileLock::~CFileLock (Entry) |
7822:14:11.543 (EvNetAppPHSvr)<7240>EV:MThe exclusive FileLock was released for the file: \\?\UNC\NASPRD01.xxx.COM\ONTAP_ADMIN$\VOL\VOL_T\xxx\INFORMATION SYSTEMS\INWARDS\ENZA WEB CONTENT\ENZA TRADE FOLDER 2002\INSIDE_FOLDER_?2.PDF
7922:14:11.543 (EvNetAppPHSvr)<7240>EV:LCFileLock::~CFileLock (Exit) |Success |
8022:14:11.543 (EvNetAppPHSvr)<7240>EV:LCQueue::DeQueue (Entry) |
8122:14:16.543 (EvNetAppPHSvr)<7256>EV:MPinging Filer: NASPRD01.xxx.com to check its still alive
8222:14:16.543 (EvNetAppPHSvr)<7256>EV:MSuccessful ping of connection to filer: NASPRD01.xxx.com. Resetting retry count

This is a placeholder:

10922:14:16.543 (EvNetAppPHSvr)<7144>EV:LCArchiveSvr::CheckWorkerThreads (Exit) |Success |
11022:14:17.465 (EvNetAppPHSvr)<2740>EV:LOpenFileWorkItem::OpenFileWorkItem (Entry) |
11122:14:17.465 (EvNetAppPHSvr)<880>EV:HFlatten::Flatten
11222:14:17.465 (EvNetAppPHSvr)<880>EV:MCFolderPermSyncWrapper SynchronizeEveryoneSID is 1
11322:14:17.465 (EvNetAppPHSvr)<880>EV:HCFolderPermSyncWrapper::SynchroniseFolder - Expand local groups:0
11422:14:17.465 (EvNetAppPHSvr)<2740>EV:LOpenFileWorkItem::OpenFileWorkItem (Exit) |Success |
11522:14:17.465 (EvNetAppPHSvr)<2740>EV:LCQueue::EnQueue (Entry) |
11622:14:17.465 (EvNetAppPHSvr)<2740>EV:MCQueue::EnQueue item enQueued
11722:14:17.465 (EvNetAppPHSvr)<7244>EV:MCQueue::DeQueue item deQueued, currently 1 items on the queue
11822:14:17.465 (EvNetAppPHSvr)<2740>EV:LCQueue::EnQueue (Exit) |Success |
11922:14:17.465 (EvNetAppPHSvr)<7244>EV:LCFileLock::Lock (Entry) |
12022:14:17.465 (EvNetAppPHSvr)<7244>EV:MThe exclusive FileLock was granted for the file: \\?\UNC\NASPRD01.xxx.COM\ONTAP_ADMIN$\VOL\VOL_T\xxx\INFORMATION SYSTEMS\INWARDS\ENZA WEB CONTENT\ENZA TRADE FOLDER 2002\INSIDE_FOLDER_?2.PDF
12122:14:17.465 (EvNetAppPHSvr)<7244>EV:LCFileLock::Lock (Exit) |Success |
12222:14:17.465 (EvNetAppPHSvr)<7244>EV:LOpenFileWorkItem::Process (Entry) |
12322:14:17.465 (EvNetAppPHSvr)<7244>EV:M The file \\NASPRD01\xxx\Information Systems\Inwards\ENZA Web Content\ENZA Trade Folder 2002\Inside_Folder_?2.pdf is an Enterprise Vault placeholder and will recalled
12422:14:17.465 (EvNetAppPHSvr)<7244>EV:LWorkItem::ExtractplaceholderData (Entry) |
12522:14:17.481 (EvNetAppPHSvr)<7244>EV:LWorkItem::ExtractplaceholderData (Exit) |Success |
12622:14:17.481 (EvNetAppPHSvr)<7244>EV:LOpenFileWorkItem::DownloadFile (Entry) |
12722:14:17.481 (EvNetAppPHSvr)<7244>EV:LOpenFileRequest::GetDefaultWebAppURL (Entry) |
12822:14:17.481 (EvNetAppPHSvr)<7244>EV:HGetDefaultWebAppURL() expireCache: 79 Url: http://tgevserver1.xxx.com/EnterpriseVault/download.asp?VaultID=1A91F27D106C5264A83E9088F49C7C0691110000TGEVSITE.xxx.com&SavesetId=537062000000000~200201160112030000~0~B85F8F4EE9FA48358F0FE25AA4A89F8&Request=NativeItem&EVHeaders=1
12922:14:17.481 (EvNetAppPHSvr)<7244>EV:LOpenFileRequest::GetDefaultWebAppURL (Exit) |
13022:14:17.496 (EvNetAppPHSvr)<9672>EV:HConnectForSegment:http://tgevserver1.xxx.com/EnterpriseVault/download.asp?VaultID=1A91F27D106C5264A83E9088F49C7C0691110000TGEVSITE.xxx.com&SavesetId=537062000000000~200201160112030000~0~B85F8F4EE9FA48358F0FE25AA4A89F8&Request=NativeItem&EVHeaders=1&Segment=0
13122:14:17.731 (EvNetAppPHSvr)<9672>EV:HBC 1
13222:14:17.731 (EvNetAppPHSvr)<9672>EV:Hdwcode 200
13322:14:17.731 (EvNetAppPHSvr)<9672>EV:HBT 1
13422:14:17.731 (EvNetAppPHSvr)<9672>EV:HFile size:535392
13522:14:17.746 (EvNetAppPHSvr)<9672>EV:L Creating unnamed hash from 535392 byte source...
13622:14:17.746 (EvNetAppPHSvr)<9672>EV:HSegment:1, read:535392
13722:14:17.746 (EvNetAppPHSvr)<880>EV:LCFolderPermSyncWrapper::ResetFileDates (Entry) |
13822:14:17.746 (EvNetAppPHSvr)<880>EV:LCFolderPermSyncWrapper::_ResetFileDates (Entry) |
13922:14:17.746 (EvNetAppPHSvr)<880>EV:MCFolderPermSyncWrapper::_ResetFileDates (Exit) |Success |
14022:14:17.746 (EvNetAppPHSvr)<880>EV:MCFolderPermSyncWrapper::ResetFileDates (Exit) |Success |
14122:14:17.746 (EvNetAppPHSvr)<7244>EV:LOpenFileWorkItem::DownloadFile (Exit) |Success |
14222:14:17.809 (EvNetAppPHSvr)<7244>EV:LOpenFileWorkItem::Process (Exit) |Success |
14322:14:17.809 (EvNetAppPHSvr)<7244>EV:LCFileLock::~CFileLock (Entry) |
14422:14:17.809 (EvNetAppPHSvr)<7244>EV:MThe exclusive FileLock was released for the file: \\?\UNC\NASPRD01.xxx.COM\ONTAP_ADMIN$\VOL\VOL_T\xxx\INFORMATION SYSTEMS\INWARDS\ENZA WEB CONTENT\ENZA TRADE FOLDER 2002\INSIDE_FOLDER_?2.PDF
14522:14:17.809 (EvNetAppPHSvr)<7244>EV:LCFileLock::~CFileLock (Exit) |Success |
14622:14:17.809 (EvNetAppPHSvr)<7244>EV:LCQueue::DeQueue (Entry) |

Any other sections of this dtrace you'd like to see? I'm still waiting on logs from the client for a number of other situations (and backup logs to check the specific palceholders that are throwing errors).

Sam_Wood
Level 4
Just got done talking to support (love those guys), and the issue is resolved.

Basically, we had to have the backup agent service account in a local group on the NetApp filer called �Enterprise Vault Backup Operators,� as well as putting FSA into backup mode.

The documentation is pretty unclear on this point (pg 113 and 114 of the EV6 SP3 Admin Guide), but the support rep explained this is a must-do for backup apps that don't support Offline files very well.

Apparently, we wouldn't have had this issue if we'd been using NetBackup ;)

Thanks for the help guys.

Kopfjager
Level 5
Employee Certified
Well it looks like it is recalling files so your probably hitting the recall limit.

Do those ones that are referring to "not a placeholder" occur right after the recalls?

Also, can you post the log section that shows that happening? It should not parse every file in that log, just ones it thinks are placeholders, but i recall a issue where valid placeholders were giving that message and want to see if this is it.

Sam_Wood
Level 4
Hi Paul, the issue is resolved, but I can email you the full log if you're still curious.