cancel
Showing results for 
Search instead for 
Did you mean: 

Backup fail after cluster logical client switch to other SAN client

albatroz19
Level 5

Hi,
i've an issue on a logical client tibcosoasb that fails with this error:

03/07/2019 17:12:41 - Error bpbrm (pid=269462) from client tibcosoadb.backup.local: ERR - Could not open file /usr/openv/netbackup/db/config/shm/tibcosoadb.backup.local_1551968976 to get shared memory information. Errno = 2: No such file or directory
03/07/2019 17:17:41 - Error bptm (pid=269463) Could not open FT Client pipe for client tibcosoadb.backup.local: pipe open failed (13369361 invalid FAT error number)
03/07/2019 17:17:42 - Info bptm (pid=269463) EXITING with status 83 <----------
03/07/2019 17:17:43 - Info bpbkar (pid=21809) done. status: 83: media open error

 

The environment it is:

Master Server - Linux RedHat - NBU 7.7.3
Media Server - Appliance 5330 - Linux RedHat - NBU 7.7.3

Client tibcosoadb and other clients present on a SAN Client physical

 


# nbemmcmd -listhosts -list_servers_in_app_cluster -clustername tibcosoadb.backup.local
NBEMMCMD, Version: 7.7.3
The following hosts were found:
client jack.backup.local (SAN CLIENT)
client petra.backup.local (SAN CLIENT)
client felder.backup.local (SAN CLIENT)
client fogerty.backup.local (SAN CLIENT)
client henley.backup.local (SAN CLIENT)
client page.backup.local (SAN CLIENT)
client blackmore.backup.local (SAN CLIENT)
client frey.backup.local (SAN CLIENT)
Command completed successfully.

# nbemmcmd -listhosts -list_servers_in_app_cluster -clustername tibcopr.backup.local
NBEMMCMD, Version: 7.7.3
The following hosts were found:
client jack.backup.local (SAN CLIENT)
client petra.backup.local (SAN CLIENT)
client felder.backup.local (SAN CLIENT)
client fogerty.backup.local (SAN CLIENT)
client henley.backup.local (SAN CLIENT)
client page.backup.local (SAN CLIENT)
client blackmore.backup.local (SAN CLIENT)
client frey.backup.local (SAN CLIENT)
Command completed successfully.

 

Client tibcosoadb.backup.local belong on san client frey.backup.local, but now it have hardware problem and we moved tibcosoadb on san client petra, but now we have the above backup error.
Other client like tibcopr present on petra, backup is ok. Then, the strange is that we switch FT transport to Lan, tibcosoadb continue to go via FT, and other client go correctly via Lan.

 

Action performed:

- Ping, telnet, port opened from client to Master and Media, viceversa
- Cleared host cache
- Restarted nbu services on Petra (Where client is it now)
- Verified with bpclntcmd ip, hn, and other option in order to verify Master, media, client name/ip configuration

Please can someone help me on this.

Thanks in advance,
Best Regards

 

11 REPLIES 11

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Have you checked if  FT service is active on the failover node (petra)?

See this TN: https://www.veritas.com/support/en_US/article.100005044

Hi Marianne,

the FT service on Petra are actives.

NB Processes
------------
root 31352 1 0 10:58 ? 00:00:00 /usr/openv/netbackup/bin/vnetd -standalone
root 31357 1 0 10:58 ? 00:00:00 /usr/openv/netbackup/bin/bpcd -standalone
root 31378 1 0 10:58 ? 00:00:02 /usr/openv/netbackup/bin/nbftclnt
root 31422 1 0 10:58 ? 00:00:00 /usr/openv/pdde/pdag/bin/mtstrmd


Shared Symantec Processes
-------------------------
root 22233 1 0 Mar07 ? 00:00:04 /opt/VRTSpbx/bin/pbx_exchange

 

From https://www.veritas.com/support/en_US/article.100005044

Solution

1. Install the NetBackup client software on each failover node OK

2. Configure the SAN client on each node according to the instructions in this chapter. Ensure that the FT service is active on all of the failover nodes (bpclntcmd or AdminConsole > San Clients) OK

3. On the NetBackup master server, use the nbemmcmd command to register the virtual node with the clients, as follows:
     a. To define NetBackup clustered clients add the virtual name to the EMM database:
         nbemmcmd –addhost –machinename virtual_name –machinetype app_cluster OK
     b. Add the virtual name to all clients in the node. For every client in the node, run the following command.
         nbemmcmd –updatehost –add_server_to_app_cluster –machinename client_name –clustername virtual_name OK

 

The other virtual client tibcopr on petra, backup goes via ft and it's ok. When i switched tibcosoadb from SAN client frey to petra, tibcosoadb'sbackup on petra do not funciton. I don't know what is the problem...

Thanks for your response.

Best Regards,

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

All I can suggest is to log a Support call with Veritas.

There is unfortunately not much experience in this community with SAN Client; even less with clustered SAN Client. 

Hi,

i knew that on virtual client tibcosoadb, Oracle use 1556 port. So i know that pbx port 1556 work on any network interface.

It's possible bind pbx service on a specified network interface in order to resolve conflict with Oracle?

Thanks in advance,
Best Regards

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Symantec/Veritas has officially registered port 1556 with iana.org.
https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xml

If anything needs to change, it's Oracle. Not NetBackup.

If you really want to change the PBX port number, you need to change it everywhere across your entire NBU domain.

I supposed that it's wrong change pbx port because is blocked by NBU.

How i may resolve this issue?
there is maybe a workaround?.

I don't understand why i have this error:

03/07/2019 17:12:41 - Error bpbrm (pid=269462) from client tibcosoadb.backup.local: ERR - Could not open file /usr/openv/netbackup/db/config/shm/tibcosoadb.backup.local_1551968976 to get shared memory information. Errno = 2: No such file or directory

Thanks and Regards

@albatroz19 @Marianne 

Incase of SAN client backups, its a shared memory (conceptually) between media server and bpbkar process on client. bpbkar thinks it is writting to a shared memory created by bptm and bptm reads from it to send data to storage hence you are getting these messages. 

PBX port is for NetBackup and SAN client backups uses PBX. So if there is any conflict with port 1556 SAN client backups will fail.

If possible reboot the cluster node then try to perform backup. If it still fails then open a case with Veritas support as logs review is needed.

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@Anshu_Pathak 

As long as Oracle is using port 1556, I doubt that a reboot will fix the problem.

Systems_Team
Moderator
Moderator
   VIP   

@Marianne:

Here's a blast from the past Smiley LOL

https://vox.veritas.com/t5/NetBackup/Netbackup-Client-network-port-1556/td-p/602621

Back in 2013 you replied on a similar issue.  Oracle DBA should change the Oracle Listener back to its default of Port 1521 and leave NetBackup on its default of 1556

Steve

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

LOL @Systems_Team 

I had to look for that post as well because I could not remember the IANA url.

Default Oracle port numbers here: https://docs.oracle.com/cd/B19306_01/install.102/b15660/app_port.htm

 

Hi All,

thank you so much for assistance.
i told to dba dmin to change the port 1556, becasue it's exclusive to Netbackup. They told me if it's possible bind port 1556 pbx_exchange to single ip address, in order to use also 1556 for oracle in other ip address like this example:

192.168.1.10 1556 netbackup
192.144.1.4 1556 oracle

In a node of cluster, they would be perform a custom configuration in order to address packet by application to 1556 port correct, to nbu or oracle. Is it possible?

Meanwhile i receive  intermittent oracle/sap failure on various client of cluster with "permission denied" on disk, like this example:

<16> non_mpx_backup_archive_verify_import: from client: ERROR: V-3-20003: Cannot open /dev/vx/dsk/TM1/TM1arc: Permission denied

 

Thanks,
Best Regards