Netbackup backup with GRT issue

[NBU_13]

Hi,

I need help on this issue..

I comfigured new netbackup setup.

master/media server : Netbackup 7.5, OS windows 2008 standard

Exchange server 2007, windows 2008 standard : Netbackup 7.5 , backup configured with policy type :MS-Exchnage-Server with GRT

AD server : Windows 2008 standard, Netbackup 7.5, backup configured with policy type MS-Windows with GRT

both GRT backup jobs are getting error as below

error from AD

12/20/2012 10:26:30 - Error bpbrm (pid=5248) from client heineken.businesscontinuity.cl: ERR - Unable to initialize nbfsd.
12/20/2012 10:59:46 - Error bpbrm (pid=5248) from client heineken.businesscontinuity.cl: ERR - Error encountered while attempting to get additional files for System State:\

12/20/2012 11:01:34 - Info bpbkar32 (pid=7556) 5019 entries sent to bpdbm
12/20/2012 11:04:29 - Info bptm (pid=5752) waited for full buffer 72128 times, delayed 156551 times
12/20/2012 11:04:29 - Info bptm (pid=5752) EXITING with status 0 <----------
12/20/2012 11:04:29 - Info bpbrm (pid=5248) validating image for client heineken.businesscontinuity.cl
12/20/2012 11:04:32 - end writing; write time: 0:40:53
the requested operation was partially successful  (1)

error from Exchange server

¨from client stella.businesscontinuity.cl: ERR - Unable to initialize nbfsd.¨

Notes : I installed NFS server on maste/media server, Exchange server, AD server and I disabled server for NFS and Client for NFS on master/media server, also disabled Server for NFS on both Exchange server, AD, only Client for NFS is running on Exchange and AD servers.

on Exchange server and AD server

Netbackup client service

Netbackup legacy client service

Netbackup legacy network service

are configured with logon as administrator

Thanks. 

[RamNagalla · Accepted solution]

hey,

do you have a firewall in place.. 

can you make sure that the required ports: 29588 (nbfsd port); 2049 (nfs); 111 (portmap). are open for connection.

http://www.symantec.com/business/support/index?page=content&id=TECH190785

if there is no firewall,

did your exchage in cluster?

please porvide your Policy details and the bpbkar and nbfsd logs..

[NBU_13]

Hi,

Thanks for reply.

all ports are open in our network.

Please refer attached files for error logs.

Thanks.

[RamNagalla]

seems you missed to attach the file...

[Dyneshia]

One more thing, has this every worked before or is this a new GRT configuration.

If this is a new configuration, please sure to review : http://www.symantec.com/docs/HOWTO73076

[NBU_13]

Hi,

Please refer attached file for error logs

This is new setup for backup, I just configured netbackup for 2 new servers

[NBU_13]

Hi,

Please refer attached file for error logs

This is new setup for backup, I just configured netbackup for 2 new servers

AD backup detail logs :

12/20/2012 11:10:49 - estimated 0 kbytes needed
12/20/2012 11:10:49 - Info nbjm (pid=3304) started backup (backupid=heineken.businesscontinuity.cl_1356012649) job for client heineken.businesscontinuity.cl, policy Heineken, schedule full on storage unit Backup_dsu
12/20/2012 11:10:49 - started process bpbrm (pid=5852)
12/20/2012 11:10:55 - Info bpbrm (pid=5852) heineken.businesscontinuity.cl is the host to backup data from
12/20/2012 11:10:55 - Info bpbrm (pid=5852) reading file list from client
12/20/2012 11:10:55 - connecting
12/20/2012 11:10:57 - Info bpbrm (pid=5852) starting bpbkar32 on client
12/20/2012 11:10:57 - connected; connect time: 0:00:00
12/20/2012 11:11:07 - Info bpbkar32 (pid=7224) Backup started
12/20/2012 11:11:07 - Info bptm (pid=1160) start
12/20/2012 11:11:08 - Info bptm (pid=1160) using 262144 data buffer size
12/20/2012 11:11:08 - Info bptm (pid=1160) setting receive network buffer to 1049600 bytes
12/20/2012 11:11:08 - Info bptm (pid=1160) using 30 data buffers
12/20/2012 11:11:08 - Info bptm (pid=1160) start backup
12/20/2012 11:11:09 - Info bptm (pid=1160) backup child process is pid 5476.6060
12/20/2012 11:11:09 - Info bptm (pid=5476) start
12/20/2012 11:11:09 - begin writing
12/20/2012 11:13:57 - Error bpbrm (pid=5852) from client heineken.businesscontinuity.cl: ERR - Unable to initialize nbfsd.
12/20/2012 11:42:51 - Error bpbrm (pid=5852) from client heineken.businesscontinuity.cl: ERR - Error encountered while attempting to get additional files for System State:\
12/20/2012 11:44:33 - Info bpbkar32 (pid=7224) 5024 entries sent to bpdbm
12/20/2012 11:47:23 - Info bptm (pid=1160) waited for full buffer 72138 times, delayed 139083 times
12/20/2012 11:47:24 - Info bptm (pid=1160) EXITING with status 0 <----------
12/20/2012 11:47:24 - Info bpbrm (pid=5852) validating image for client heineken.businesscontinuity.cl
12/20/2012 11:47:27 - end writing; write time: 0:36:18
the requested operation was partially successful  (1)
 

[RamNagalla]

hi,

below it the nbfsd log

09:48:36.540 [9160.8428] <16> rpc_connect: can't create TCP connection to nbuserver (12 10060), will retry...

please check below 2 tech notes, make sure that they are fine..

http://www.symantec.com/business/support/index?page=content&id=TECH136839

http://www.symantec.com/business/support/index?page=content&id=TECH163845

below is the bpfis log, its taking source IP address as 0.0.0.0,  so please check why its taking 0.0.0.0.

[vnet_connect.c:1653] getsockopt SO_ERROR returned 10060 0x274c

09:14:22.216 [5240.5804] <16> connect_to_service: connect failed STATUS (18) CONNECT_FAILED

status: FAILED, (42) CONNECT_REFUSED; system: (10060) Connection timed out.; FROM 0.0.0.0 TO nbuserver.businesscontinuity.cl 192.168.10.122 bprd VIA pbx

status: FAILED, (42) CONNECT_REFUSED; system: (10060) Connection timed out.; FROM 0.0.0.0 TO nbuserver.businesscontinuity.cl 192.168.10.122 bprd VIA vnetd

status: FAILED, (42) CONNECT_REFUSED; system: (10060) Connection timed out.; FROM 0.0.0.0 TO nbuserver.businesscontinuity.cl 192.168.10.122 bprd

[NBU_13]

HI,

I tried as below, 2nd tech note as below

Solution

On Master Server, under Host Properties click on Clients.

Select the Exchange Client and go to the Properties of the Client.

Under Port Ranges, in the Client port window and Client reserved port window section check the "OS selected non reserved port" box.

but still, backup completed with status 1 with ERR - Unable to initialize nbfsd.

1st technote for linux media server, out media server is windows 2003

Exchange server :job details

12/20/2012 14:53:16 - Info bptm (pid=2408) start backup
12/20/2012 14:53:17 - Info bpbkar32 (pid=3828) INF - waiting for mount mutex
12/20/2012 14:53:17 - Info bptm (pid=2408) backup child process is pid 1760.4784
12/20/2012 14:53:17 - Info bptm (pid=1760) start
12/20/2012 14:53:17 - begin writing
12/20/2012 14:54:30 - Error bpbrm (pid=5892) from client stella.businesscontinuity.cl: ERR - Unable to initialize nbfsd.
12/20/2012 15:34:40 - Info bptm (pid=2408) waited for full buffer 71841 times, delayed 158709 times
12/20/2012 15:34:40 - Error bpbrm (pid=5892) from client stella.businesscontinuity.cl: ERR - Error encountered while attempting to get additional files for Microsoft Information Store:\First Storage Group\Public Folder Database\
12/20/2012 15:34:40 - Error bpbrm (pid=5892) from client stella.businesscontinuity.cl: ERR - Exchange granular restore from this image may not work.
12/20/2012 15:34:40 - Info bptm (pid=2408) EXITING with status 0 <----------
12/20/2012 15:34:40 - Info bpbrm (pid=5892) validating image for client stella.businesscontinuity.cl
12/20/2012 15:34:42 - end writing; write time: 0:41:25
the requested operation was partially successful  (1)
 

AD : Job details :

12/20/2012 15:09:44 - Info bptm (pid=3636) backup child process is pid 5040.2780
12/20/2012 15:09:44 - Info bptm (pid=5040) start
12/20/2012 15:09:44 - begin writing
12/20/2012 15:14:09 - Error bpbrm (pid=3176) from client heineken.businesscontinuity.cl: ERR - Unable to initialize nbfsd.
12/20/2012 15:52:32 - Error bpbrm (pid=3176) from client heineken.businesscontinuity.cl: ERR - Error encountered while attempting to get additional files for System State:\
12/20/2012 15:54:04 - Info bpbkar32 (pid=7852) 5023 entries sent to bpdbm
12/20/2012 15:56:54 - Info bptm (pid=3636) waited for full buffer 72130 times, delayed 180960 times
12/20/2012 15:56:54 - Info bptm (pid=3636) EXITING with status 0 <----------
12/20/2012 15:56:54 - Info bpbrm (pid=3176) validating image for client heineken.businesscontinuity.cl
12/20/2012 15:56:57 - end writing; write time: 0:47:13
the requested operation was partially successful  (1)

[Marianne]

I read the following in your opening post:

 I disabled server for NFS and Client for NFS on master/media server

Please go carefully through the NFS config section of the TN in Dyneshia's post.
Client for NFS must be enabled. See extract from HOWTO73076:

 b. Window Server 2008 or Windows 2008 R2 - To enable services for Network File System, perform the following:
      1. Open the Server Manager on Windows 2008 Server.
      2. Under the Server Manager, go to Roles.
      3. Click Add roles.
      4. Click Next.
      5. Select File Services.
      6. Select Services for Network File System.
      7. Click Finish.(Ensure that the File Services role appears in the server manager\roles)
      8. Open up Services, disable Server for NFS, enable Client for NFS set to automatic.
      9. Open a command prompt, run "sc config portmap start= auto"

[RamNagalla]

hi

check nbfsd manually, why its not initialiszeing 

check the below technote for steps to get it manually..

http://www.symantec.com/business/support/index?page=content&id=TECH124810

How to manually test NetBackup Granular Restore Technology (GRT) NFS connections between media server and client

Environment

This document applies to all operating systems supported by NetBackup for Granular Restore Technology backup and restore operations.

Solution

For a GRT backup "nbfsd" process is started on the NetBackup media server and "nbfs" process is started on the NetBackup client. Both the processes uses "NFS" (Network File Sharing) for communication.

The objective of this "nbfsd" connection check is to isolate if there is any issue with the "nbfsd", "nbfs" or "NFS". 

On the NetBackup media server.

1. Increase the logging verbosity and enable logs for nbfsd. <install_path>\Veritas\netbackup\logs\nbfsd

2. Open command prompt and execute the command:

<install_path>\Veritas\netbackup\bin\nbfsd -d -v 5 -j <job_id> -B <backup_id>

## You can specify any number for job id. It is not mandatory to have that job id present in activity monitor or the job id of the GRT backup job.
## You can specify any backup id. (Must be clientname_9 digit number e.g. client_123456789)

Example:  
  

This command will start the nbfsd process. Do not use "CNTRL + C " on the command prompt, as it will terminate the nbfsd session. Take a note of values for variable "CRED1" AND "CRED2".

If this returns back to a command prompt immediately, you probably have a port blocked. By default 7394 is used. Check bi-directional communication on that port. The client side of this test will not function unless that operation persists.

The value for "CRED1" will be used by "nbfs" to perform the NFS mount on the NetBackup client.
The value for "CRED2" will be used by "nbfs" to perform the NFS unmount on the NetBackup client.

On NetBackup client:

1. Increase the logging verbosity and enable logs for nbfsd. <install_path>\Veritas\netbackup\logs\nbfsd

2. Open command prompt and execute the command:

<install_path>\Veritas\netbackup\bin\nbfs mount -server <Media_server_name> -cred <CRED1> *

## Specify the name of the NetBackup media server where the nbfsd process is started in the above process.
## Do not forget to add * (asterisk symbol) at the end of the command line.


  

3. Use the command "net use" to get the list of the mounted drive.


  

4. Change the directory to the mounted drive letter and try to list the files.

5. If the above process is successful unmount the share using command:
<install_path>\Veritas\netbackup\bin\nbfs umount -server <servername> -cred <CRED2> <mount_pt>

hope this helps... 
 

[NBU_13]

Hi Marianne,

Let me try to enable client NFS on master/media server and try backup.

I disable Server for NFS and Client for NFS on master/media server, because I saw some symantec video, they diabled both on media server and enable client NFS on Exchange server.

Let me try enable client NFS on media server and test backup

[NBU_13]

Hi Marianne,

I tried to enable client NFS on master/media server and tried backup.

still backup job is failing with same error ERR - Unable to initialize nbfsd.:(

[NBU_13]

Hi,

I done  manually test NetBackup Granular Restore Technology (GRT) on media server, it gives below output immediately and command prompt also came.

on media server

C:\Program Files\Veritas\NetBackup\bin>nbfsd -d -v 5 -j 69 -B stella.businesscon
tinuity.cl_1356017681

PORT=7394
CRED1=5F25A94B24CD35B0D525BF64B6F074CB1F3C70553FC2194B6CFD8FC13EEF7C24D71306C913
9749664B9E8F62C62185933462C974A549237D13A0C18DC69EF572
CRED2=F4BCA78E6D0D8E72852976B552322B8AE278D0AA23672240CE2B4F9305278A1CFFD77E4836
765342E191D6959B7F01C577D70DC934D834E121B132B4CF3E43E6
NBFSD STARTED 0

C:\Program Files\Veritas\NetBackup\bin>

C:\Program Files\Veritas\NetBackup\bin>nbfs mount -server nbuserver -cred 48A185
53FF4466B3204F55373598F606E5CC42AEC90CC859DFE35AD830DFC0298CC09058E8C35E7175D42C
F40DEDA35FCFC369DA0EA4258B7A32B24192D31B79 *
connect to nbuserver failed
EXIT_STATUS=25

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -self
gethostname() returned: stella.businesscontinuity.cl
host stella.businesscontinuity.cl: stella.businesscontinuity.cl at ::1
host stella.businesscontinuity.cl: stella.businesscontinuity.cl at 192.168.10.20

aliases:     stella.businesscontinuity.cl     ::1     192.168.10.20
getfqdn(stella.businesscontinuity.cl) returned: stella.businesscontinuity.cl

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -hn stella
host stella: stella.businesscontinuity.cl at ::1
host stella: stella.businesscontinuity.cl at 192.168.10.20
aliases:     stella.businesscontinuity.cl     stella     ::1     192.168.10.20

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -ip 192.168.10.20
host 192.168.10.20: stella.businesscontinuity.cl at ::1
host 192.168.10.20: stella.businesscontinuity.cl at 192.168.10.20
aliases:     stella.businesscontinuity.cl     ::1     192.168.10.20

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -pn

C:\Program Files\Veritas\NetBackup\bin>

C:\Program Files\Veritas\NetBackup\bin>net use
New connections will be remembered.

There are no entries in the list.

C:\Program Files\Veritas\NetBackup\bin>

[RamNagalla]

hey,

did you observer this, 

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -pn

C:\Program Files\Veritas\NetBackup\bin>

bpclntcmd -pn return nothing...

this might be the reason for failure of 

C:\Program Files\Veritas\NetBackup\bin>nbfs mount -server nbuserver -cred 48A185
53FF4466B3204F55373598F606E5CC42AEC90CC859DFE35AD830DFC0298CC09058E8C35E7175D42C
F40DEDA35FCFC369DA0EA4258B7A32B24192D31B79 *
connect to nbuserver failed
EXIT_STATUS=25

there is something that you need to fix about the reverse communication between the Client and master server.

you should get the responce from the master when you run bpclntcmd -pn

please check below

1)master server Name in client registry(make sure its listed first)

2) IP resolution of master server

3) below are in master server

bpclntcmd -self

bpclntcmd -hn <master server name>

bpclntcmd -ip <master server IP>

4)In client 

bpclntcmd -hn <master server name>

bpclntcmd -ip <master server IP>

.

[NBU_13]

Hi,

Thanks you to all

Issue got resolved...

issue with firewall, its blocked ports

I created firewall policy and open port for ports: 29588 (nbfsd port); 2049 (nfs); 111 (portmap)

Now, its running fine, I done test backup, but i try to restore individual mails

its getting fail with below error.

12/21/2012 16:49:03 - begin Restore
12/21/2012 16:49:06 - restoring from image stella.businesscontinuity.cl_1356114842
12/21/2012 16:49:06 - Info bprd (pid=5660) Restoring from copy 1 of image created 12/21/12 15:34:02
12/21/2012 16:49:06 - requesting resource @aaaab
12/21/2012 16:49:06 - granted resource  MediaID=@aaaab;DiskVolume=D:\;DiskPool=Backup_pool;Path=D:\;StorageServer=nbuserver;MediaServer=nbus...
12/21/2012 16:49:09 - Info bpbrm (pid=5652) stella.businesscontinuity.cl is the host to restore to
12/21/2012 16:49:09 - Info bpbrm (pid=5652) reading file list from client
12/21/2012 16:49:10 - connecting
12/21/2012 16:49:10 - Info bpbrm (pid=5652) start nbfsd on client
12/21/2012 16:49:11 - Info bpbrm (pid=5652) start nbgre on client
12/21/2012 16:49:12 - Info tar32 (pid=0) Restore started
12/21/2012 16:49:12 - connected; connect time: 0:00:00
12/21/2012 16:49:35 - Info tar32 (pid=0) done. status: 5
12/21/2012 16:49:35 - Info tar32 (pid=0) done. status: 5: the restore failed to recover the requested files
12/21/2012 16:49:35 - Error bpbrm (pid=5652) client restore EXIT STATUS 5: the restore failed to recover the requested files
12/21/2012 16:49:36 - restored from image stella.businesscontinuity.cl_1356114842; restore time: 0:00:30
12/21/2012 16:49:41 - end Restore; elapsed time 0:00:38
MS-Exchange policy restore error  (2810)

Active directory for individual restore are getting successful

but exchange individual restore getting fail with

12/21/2012 16:49:12 - connected; connect time: 0:00:00
12/21/2012 16:49:35 - Info tar32 (pid=0) done. status: 5
12/21/2012 16:49:35 - Info tar32 (pid=0) done. status: 5: the restore failed to recover the requested files
12/21/2012 16:49:35 - Error bpbrm (pid=5652) client restore EXIT STATUS 5: the restore failed to recover the requested files
12/21/2012 16:49:36 - restored from image stella.businesscontinuity.cl_1356114842; restore time: 0:00:30
12/21/2012 16:49:41 - end Restore; elapsed time 0:00:38
MS-Exchange policy restore error (2810)

[NBU_13]

Now, its running fine, I done test backup, but i try to restore individual mails

its getting fail with below error.

12/21/2012 16:49:03 - begin Restore
12/21/2012 16:49:06 - restoring from image stella.businesscontinuity.cl_1356114842
12/21/2012 16:49:06 - Info bprd (pid=5660) Restoring from copy 1 of image created 12/21/12 15:34:02
12/21/2012 16:49:06 - requesting resource @aaaab
12/21/2012 16:49:06 - granted resource MediaID=@aaaab;DiskVolume=D:\;DiskPool=Backup_pool;Path=D:\;StorageServer=nbuserver;MediaServer=nbus...
12/21/2012 16:49:09 - Info bpbrm (pid=5652) stella.businesscontinuity.cl is the host to restore to
12/21/2012 16:49:09 - Info bpbrm (pid=5652) reading file list from client
12/21/2012 16:49:10 - connecting
12/21/2012 16:49:10 - Info bpbrm (pid=5652) start nbfsd on client
12/21/2012 16:49:11 - Info bpbrm (pid=5652) start nbgre on client
12/21/2012 16:49:12 - Info tar32 (pid=0) Restore started
12/21/2012 16:49:12 - connected; connect time: 0:00:00
12/21/2012 16:49:35 - Info tar32 (pid=0) done. status: 5
12/21/2012 16:49:35 - Info tar32 (pid=0) done. status: 5: the restore failed to recover the requested files
12/21/2012 16:49:35 - Error bpbrm (pid=5652) client restore EXIT STATUS 5: the restore failed to recover the requested files
12/21/2012 16:49:36 - restored from image stella.businesscontinuity.cl_1356114842; restore time: 0:00:30
12/21/2012 16:49:41 - end Restore; elapsed time 0:00:38
MS-Exchange policy restore error (2810)

Active directory for individual restore are getting successful

but exchange individual restore getting fail with

12/21/2012 16:49:12 - connected; connect time: 0:00:00
12/21/2012 16:49:35 - Info tar32 (pid=0) done. status: 5
12/21/2012 16:49:35 - Info tar32 (pid=0) done. status: 5: the restore failed to recover the requested files
12/21/2012 16:49:35 - Error bpbrm (pid=5652) client restore EXIT STATUS 5: the restore failed to recover the requested files
12/21/2012 16:49:36 - restored from image stella.businesscontinuity.cl_1356114842; restore time: 0:00:30
12/21/2012 16:49:41 - end Restore; elapsed time 0:00:38
MS-Exchange policy restore error (2810)