SharePoint backups

[NathanNieman]

I will start with WHAT A PAIN IN THE BUTT>>!!!!!  Why is this so hard.

Ok I got a backup to work a week ago.  I started the backup it ran, and backup the files.  So I waited and it ran a day later and failed, and since they again has been failing.  I have had this issue from day 1...

Server

2008 RS

Netbackup 7.5.0.6

Sharepoing front end

2008 RS server

Running sharepoint 2010

Backup version on server is 7.5.0.6

Here is what I am getting when trying to back up this server.

11/5/2013 6:54:55 AM - Info nbjm(pid=5152) starting backup job (jobid=49870) for client uns4025-38, policy uns4025-38, schedule Diff  

11/5/2013 6:54:56 AM - estimated 0 Kbytes needed

11/5/2013 6:54:56 AM - Info nbjm(pid=5152) started backup (backupid=uns4025-38_1383652495) job for client uns4025-38, policy uns4025-38, schedule Diff on storage unit uns40-88-hcart3-robot-tld-2

11/5/2013 6:54:56 AM - started process bpbrm (6504)

11/5/2013 6:54:57 AM - Info bpbrm(pid=6504) uns4025-38 is the host to backup data from     

11/5/2013 6:54:57 AM - Info bpbrm(pid=6504) reading file list from client        

11/5/2013 6:54:57 AM - connecting

11/5/2013 6:55:01 AM - Info bpbrm(pid=6504) starting bpbkar32 on client         

11/5/2013 6:55:01 AM - connected; connect time: 00:00:04

11/5/2013 6:55:05 AM - Info bpbkar32(pid=0) Backup started           

11/5/2013 6:55:05 AM - Info bptm(pid=6920) start            

11/5/2013 6:55:05 AM - Info bptm(pid=6920) using 65536 data buffer size        

11/5/2013 6:55:05 AM - Info bptm(pid=6920) setting receive network buffer to 263168 bytes      

11/5/2013 6:55:05 AM - Info bptm(pid=6920) using 64 data buffers         

11/5/2013 6:55:05 AM - Info bptm(pid=6920) start backup           

11/5/2013 6:55:05 AM - Info bptm(pid=6920) backup child process is pid 7236.3588       

11/5/2013 6:55:05 AM - Info bptm(pid=6920) Waiting for mount of media id 1417L3 (copy 1) on server uns40-88. 

11/5/2013 6:55:05 AM - Info bptm(pid=7236) start            

11/5/2013 6:55:05 AM - Error bptm(pid=7236) socket operation failed - 10054 (at child.c.1296)      

11/5/2013 6:55:05 AM - mounting 1417L3

11/5/2013 6:55:06 AM - Error bpbrm(pid=6504) could not send server status message       

11/5/2013 6:55:08 AM - Info bpbkar32(pid=0) done. status: -536836945: invalid error number       

11/5/2013 6:55:08 AM - end writing

invalid error number(-536836945)

If I am on the front end of SharePoint I open backup, Archive, and restore and I can see the Microsoft SharePoint resources to backup.

when I click on restore I can see 2 fulls "26th, and 19th, and one diff which I ran on the 18th"

This should run every day.

What am I missing.  I have 4 other servers I need to get this working on, but heck his is a pain in the butt so far.

any help would be great.

[RamNagalla]

Status = none of the files in the file list exist.

does it still have the same status..?

if yes please post the output of bppllist <Policyname> -L

[NathanNieman]

Here is the output of BPPLIST

C:\Program Files\Veritas\NetBackup\bin\admincmd>bppllist uns4025-38 -L

Policy Name:       uns4025-38

Options:           0x0

template:          FALSE

audit_reason:         ?

Names:             (none)

Policy Type:       MS-SharePoint (8)

Active:            yes

Effective date:    06/27/2013 10:37:44

Mult. Data Stream: no

Perform Snapshot Backup:   no

Snapshot Method:           (none)

Snapshot Method Arguments: (none)

Perform Offhost Backup:    no

Backup Copy:               0

Use Data Mover:            no

Data Mover Type:           2

Use Alternate Client:      no

Alternate Client Name:     (none)

Use Virtual Machine:      0

Hyper-V Server Name:     (none)

Enable Instant Recovery:   no

Policy Priority:   0

Max Jobs/Policy:   1

Disaster Recovery: 0

Collect BMR Info:  no

Keyword:           Legal Sharepoint Backup Non GRT

Data Classification:       -

Residence is Storage Lifecycle Policy:    yes

Client Encrypt:    no

Checkpoint:        no

Residence:         IBM_VTL_Storage_Life_Policy

Volume Pool:       NetBackup

Server Group:      *ANY*

Granular Restore Info:  no

Exchange Source attributes:              no

Exchange DAG Preferred Server: (none defined)

Application Discovery:      no

Discovery Lifetime:      28800 seconds

ASC Application and attributes: (none defined)

Generation:      18

Ignore Client Direct:  no

Enable Metadata Indexing:  no

Index server name:  NULL

Use Accelerator:  no

Client/HW/OS/Pri/DMI:  uns4025-38 Windows-x64 Windows2008 0 0 0 0 ?

Include:           Microsoft SharePoint Resources:\

Schedule:              Full

  Type:                FULL SSharePoint Portal (0)

  Frequency:           7 day(s) (604800 seconds)

  Maximum MPX:         1

  Synthetic:           0

  Checksum Change Detection: 0

  PFI Recovery:        0

  Retention Level:     0 (1 week)

  u-wind/o/d:          0 0

  Incr Type:           DELTA (0)

  Alt Read Host:       (none defined)

  Max Frag Size:       0 MB

  Number Copies:       1

  Fail on Error:       0

  Residence:           (specific storage unit not required)

  Volume Pool:         (same as policy volume pool)

  Server Group:        (same as specified for policy)

  Residence is Storage Lifecycle Policy:         0

  Schedule indexing:     0

  Daily Windows:

   Day         Open       Close       W-Open     W-Close

   Sunday      000:00:00  000:00:00

   Monday      000:00:00  000:00:00

   Tuesday     000:00:00  000:00:00

   Wednesday   000:00:00  000:00:00

   Thursday    000:00:00  000:00:00

   Friday      000:00:00  000:00:00

   Saturday    015:00:00  055:00:00   159:00:00  199:00:00 031:00:00

Schedule:              Diff

  Type:                INCR (1)

  Frequency:           7 day(s) (604800 seconds)

  Maximum MPX:         1

  Synthetic:           0

  Checksum Change Detection: 0

  PFI Recovery:        0

  Retention Level:     0 (1 week)

  u-wind/o/d:          0 0

  Incr Type:           DELTA (0)

  Alt Read Host:       (none defined)

  Max Frag Size:       0 MB

  Number Copies:       1

  Fail on Error:       0

  Residence:           (specific storage unit not required)

  Volume Pool:         (same as policy volume pool)

  Server Group:        (same as specified for policy)

  Residence is Storage Lifecycle Policy:         0

  Schedule indexing:     0

  Daily Windows:

   Day         Open       Close       W-Open     W-Close

   Sunday      000:00:00  000:00:00

   Monday      022:00:00  031:00:00   046:00:00  055:00:00

   Tuesday     022:00:00  031:00:00   070:00:00  079:00:00

   Wednesday   022:00:00  031:00:00   094:00:00  103:00:00

   Thursday    022:00:00  031:00:00   118:00:00  127:00:00

   Friday      000:00:00  000:00:00

   Saturday    000:00:00  000:00:00

[Will_Restore]

Status = none of the files in the file list exist.

Policy = uns4025-38

Schedule = Diff

Try running a Full backup first.  I know you said it was working but it seems to have 'lost its head'.

[NathanNieman]

Will give that a try now.

[NathanNieman]

Ok full just failed  here is what it said

11/6/2013 2:19:48 PM - Info nbjm(pid=7216) starting backup job (jobid=50902) for client uns4025-38, policy uns4025-38, schedule Full  

11/6/2013 2:19:49 PM - estimated 484854 Kbytes needed

11/6/2013 2:19:49 PM - Info nbjm(pid=7216) started backup (backupid=uns4025-38_1383765588) job for client uns4025-38, policy uns4025-38, schedule Full on storage unit uns40-88-hcart3-robot-tld-2

11/6/2013 2:19:49 PM - started process bpbrm (18892)

11/6/2013 2:19:50 PM - Info bpbrm(pid=18892) uns4025-38 is the host to backup data from     

11/6/2013 2:19:50 PM - Info bpbrm(pid=18892) reading file list from client        

11/6/2013 2:19:50 PM - connecting

11/6/2013 2:19:55 PM - Info bpbrm(pid=18892) starting bpbkar32 on client         

11/6/2013 2:19:55 PM - connected; connect time: 00:00:05

11/6/2013 2:20:03 PM - Info bpbkar32(pid=10200) Backup started           

11/6/2013 2:20:03 PM - Info bptm(pid=18828) start            

11/6/2013 2:20:03 PM - Info bptm(pid=18828) using 65536 data buffer size        

11/6/2013 2:20:03 PM - Info bptm(pid=18828) setting receive network buffer to 263168 bytes      

11/6/2013 2:20:03 PM - Info bptm(pid=18828) using 64 data buffers         

11/6/2013 2:20:03 PM - Info bptm(pid=18828) start backup           

11/6/2013 2:20:03 PM - Info bptm(pid=18828) backup child process is pid 8268.6880       

11/6/2013 2:20:03 PM - Info bptm(pid=18828) Waiting for mount of media id 1430L3 (copy 1) on server uns40-88. 

11/6/2013 2:20:03 PM - Warning bpbrm(pid=18892) from client uns4025-38: WRN - object not found for file system backup: Microsoft SharePoint Resources:\ConfigurationV4-DB (UNS40-04/SharePoint_Config_UNS4025-38)

11/6/2013 2:20:03 PM - Warning bpbrm(pid=18892) from client uns4025-38: WRN - object not found for file system backup: Microsoft SharePoint Resources:\Microsoft SharePoint Foundation Web Application\SharePoint - 3000 - Legal Workflow\Content-DB 1 (UNS40-04/WSS_Content_Legal_Workflow)

11/6/2013 2:20:03 PM - Warning bpbrm(pid=18892) from client uns4025-38: WRN - object not found for file system backup: Microsoft SharePoint Resources:\Microsoft SharePoint Foundation Web Application\SharePoint - 46141 - Legal Workflow V2\Content-DB 1 (UNS40-04/WSS_Content_46141_LegalWorkflowV2)

11/6/2013 2:20:03 PM - Info bptm(pid=8268) start            

11/6/2013 2:20:03 PM - Warning bpbrm(pid=18892) from client uns4025-38: WRN - object not found for file system backup: Microsoft SharePoint Resources:\Microsoft SharePoint Foundation Web Application\SharePoint - 80\Content-DB 1 (UNS40-04/WSS_Content_UNS4025-38)

11/6/2013 2:20:03 PM - Warning bpbrm(pid=18892) from client uns4025-38: WRN - object not found for file system backup: Microsoft SharePoint Resources:\Services\SPUserCodeV4/Metadata (UNS40-04/7fc958c1-6666-459f-91a5-9adc4917673e)

11/6/2013 2:20:03 PM - Warning bpbrm(pid=18892) from client uns4025-38: WRN - object not found for file system backup: Microsoft SharePoint Resources:\Shared Services\Shared Services Applications\Security Token Service Application/Metadata (UNS40-04/b91d6635-8460-46bc-bde1-97332e75b072)

11/6/2013 2:20:03 PM - Warning bpbrm(pid=18892) from client uns4025-38: WRN - object not found for file system backup: Microsoft SharePoint Resources:\Shared Services\Shared Services Applications\Business Data Connectivity Service\Services-DB 1 (UNS40-04/Bdc_Service_DB_6f7a186be5064530a952d4a5d14e714e)

11/6/2013 2:20:03 PM - Warning bpbrm(pid=18892) from client uns4025-38: WRN - object not found for file system backup: Microsoft SharePoint Resources:\Shared Services\Shared Services Proxies\Business Data Connectivity Service/Proxy (UNS40-04/4feea249-69ce-45d9-b2b7-21e2260acc0f)

11/6/2013 2:20:03 PM - Warning bpbrm(pid=18892) from client uns4025-38: WRN - object not found for file system backup: Microsoft SharePoint Resources:\WSS_Administration\WebApplication\Content-DB 1 (UNS40-04/SharePoint_AdminContent_047fa7f0-0a12-4cb9-b405-7c3e2f508803)

11/6/2013 2:20:03 PM - mounting 1430L3

11/6/2013 2:20:04 PM - Error bpbrm(pid=18892) could not send server status message       

11/6/2013 2:20:06 PM - Info bpbkar32(pid=10200) done. status: 71: none of the files in the file list exist 

11/6/2013 2:20:06 PM - end writing

none of the files in the file list exist(71)

but if I log into the server i see this so that data is there.  What the chuck...!!

[Will_Restore]

Interesting it lists the objects and then says they're not there.  Seem like some permissions got lost.  Recheck this, from the SharePoint Checklist:

Under Host Properties - Clients on the Master server, under the Windows Client - SharePoint section, make sure that the Domain Account from step 2 is specified there, with the syntax as follows:  DOMAIN\Username  (not just the Username).  Do this for every client/server member of the SharePoint farm.

[NathanNieman]

All is correct.

uns4025-38, uns40-72 =  front end Sharepoint servers

uns40-40 =  SQL server

[NathanNieman]

if you look a the backup, archive, reestore screen shot you will see it has ran 3 times.  Between the first on, and this first post I didn't do anything.  Just strange it worked 3 times, and that is it.  

[sri_vani]

can you plz check these points:

Ref link:https://www-secure.symantec.com/connect/forums/netbackup-75-unable-backup-sharepoint-2010-dr-or-grt

SharePoint Configuration Check List

1.  NetBackup Client version should be the same as the Master Server, and should be installed on all servers in the SharePoint farm, even SQL backend.   For SharePoint 2010 SP 1 or GRT you must be on 7.1.0.4 or higher.

2.  The NetBackup Client service should be started by a Domain Account.  (MUST be in format“domain\user”)
      (additional services are needed for GRT, please see GRT Section)

3.  The domain account in Step 2 must have the following privileges and permissions:
  a.  "Replace a process level token" and "Debug Programs" (Administrator Tools - Local Security Policy - Local Policies - User Rights Assignment) for all servers in the SharePoint farm, backend SQL included.
  b.  Local Administrator rights on all servers in the SharePoint farm.
  c.  Within SharePoint Central Administrator, the Domain Account is specified as a SharePoint Farm
Administrator
  d.  The System Administrator role on the SQL Server for the SharePoint databases.

4.  At a minimum, you must have at least .NET Framework 3.5 installed on all servers in the SharePoint Farm.

5.  Under Host Properties - Clients on the Master server, under the Windows Client - SharePoint section, make sure that the Domain Account from step 2 is specified there, with the syntax as follows:  DOMAIN\Username  (not just the Username).  Do this for every client/server member of the SharePoint farm.

6.  In the first policy (non-granular), the client selection is a Front End Web server for the SharePoint portal.  The backup selection for your first policy should be: "Microsoft SharePoint Resources:\"

7.  Create a "beds" directory under <install path>\NetBackup\logs\ on each of the SharePoint farm servers (SQL included)

8. Under Host Properties - Master Server, there is a Distributed Applications section.  For the first field, put in the name of the Front end server in the SharePoint policy, and in the 2nd field, the name of the SQL server hosting the SharePoint databases.

9.  From the Front End web server specified in the policy, Right click on the NetBackup Backup, Archive, and Restore (BAR) GUI and  select "Run as Administrator" .  You should be able to see Microsoft SharePoint Resources objects there. If you do not see objects, check the farm topology and make sure you have not missed any of the steps on that server.

For the first policy (non-GRT), make sure you do not have the "Enable Granular Recovery" box checked.

10.  You may now test you SharePoint Backup.

[watsons]

In master server host properties -> Distributed Application Restore Mapping,

Do you have this?

Frontend              Backend

uns4025-38          uns40-04
uns40-72              uns40-04

Is uns40-04 in cluster setup?

What has changed since it's working before? Any WIndows patch on the frontend or backend?

[sri_vani]

Nathan,By any chance did u verify the above list?

any luck?

[NathanNieman]

Watson yes that is what it says

As for any changes from the first to the last backup no that is what is strange.  I didn't touch or change anything.

UNS40-04 is a clustered setup just found that out.

Sri I am going back over everything and reviewing it now.

[watsons]

I would suggest opening a case with the Support to look further into it, if you have gone through all the checklist as mentioned. Collect beds & bpresolver for the support to check

It's possible bpresolver could not enumerate a certain object correctly, sometimes this happens on a cluster-ed backend where the name could have alias or using a FQDN (things like that...). 

Hate to say that, but using Netbackup for Sharepoint can really frustrate me sometimes.. 

[NathanNieman]

I will be calling this week.  sorry for the slow responce was out sick.

[sri_vani]

Nathan,

It wld be good if you cld update us...hope it gt fixed now

[NathanNieman]

I will as soon as we get this fixed.  We just added two new Sharepoint server and both are getting the same issue.

Company thinks it is ok to keep moving forward even though I can't back it up yet..  UGHHH.

[sri_vani]

May be its time for you to raise a Symantec Support they will take webex session and analyse the issue and drive you towards resolution...

[NathanNieman]

^ already working on it.

[NathanNieman · Accepted solution]

And we have a fix

http://www.symantec.com/business/support/index?page=content&id=TECH203759

Some how the key was there on one server but not the others.

Fixed with the above soltion and bam I have backups.  Will update in a day or two to make sure they are still working.