03-08-2019 12:33 AM
Hi,
i've an issue on a logical client tibcosoasb that fails with this error:
03/07/2019 17:12:41 - Error bpbrm (pid=269462) from client tibcosoadb.backup.local: ERR - Could not open file /usr/openv/netbackup/db/config/shm/tibcosoadb.backup.local_1551968976 to get shared memory information. Errno = 2: No such file or directory
03/07/2019 17:17:41 - Error bptm (pid=269463) Could not open FT Client pipe for client tibcosoadb.backup.local: pipe open failed (13369361 invalid FAT error number)
03/07/2019 17:17:42 - Info bptm (pid=269463) EXITING with status 83 <----------
03/07/2019 17:17:43 - Info bpbkar (pid=21809) done. status: 83: media open error
The environment it is:
Master Server - Linux RedHat - NBU 7.7.3
Media Server - Appliance 5330 - Linux RedHat - NBU 7.7.3
Client tibcosoadb and other clients present on a SAN Client physical
# nbemmcmd -listhosts -list_servers_in_app_cluster -clustername tibcosoadb.backup.local
NBEMMCMD, Version: 7.7.3
The following hosts were found:
client jack.backup.local (SAN CLIENT)
client petra.backup.local (SAN CLIENT)
client felder.backup.local (SAN CLIENT)
client fogerty.backup.local (SAN CLIENT)
client henley.backup.local (SAN CLIENT)
client page.backup.local (SAN CLIENT)
client blackmore.backup.local (SAN CLIENT)
client frey.backup.local (SAN CLIENT)
Command completed successfully.
# nbemmcmd -listhosts -list_servers_in_app_cluster -clustername tibcopr.backup.local
NBEMMCMD, Version: 7.7.3
The following hosts were found:
client jack.backup.local (SAN CLIENT)
client petra.backup.local (SAN CLIENT)
client felder.backup.local (SAN CLIENT)
client fogerty.backup.local (SAN CLIENT)
client henley.backup.local (SAN CLIENT)
client page.backup.local (SAN CLIENT)
client blackmore.backup.local (SAN CLIENT)
client frey.backup.local (SAN CLIENT)
Command completed successfully.
Client tibcosoadb.backup.local belong on san client frey.backup.local, but now it have hardware problem and we moved tibcosoadb on san client petra, but now we have the above backup error.
Other client like tibcopr present on petra, backup is ok. Then, the strange is that we switch FT transport to Lan, tibcosoadb continue to go via FT, and other client go correctly via Lan.
Action performed:
- Ping, telnet, port opened from client to Master and Media, viceversa
- Cleared host cache
- Restarted nbu services on Petra (Where client is it now)
- Verified with bpclntcmd ip, hn, and other option in order to verify Master, media, client name/ip configuration
Please can someone help me on this.
Thanks in advance,
Best Regards
03-08-2019 02:42 AM
Have you checked if FT service is active on the failover node (petra)?
See this TN: https://www.veritas.com/support/en_US/article.100005044
03-08-2019 03:06 AM
Hi Marianne,
the FT service on Petra are actives.
NB Processes
------------
root 31352 1 0 10:58 ? 00:00:00 /usr/openv/netbackup/bin/vnetd -standalone
root 31357 1 0 10:58 ? 00:00:00 /usr/openv/netbackup/bin/bpcd -standalone
root 31378 1 0 10:58 ? 00:00:02 /usr/openv/netbackup/bin/nbftclnt
root 31422 1 0 10:58 ? 00:00:00 /usr/openv/pdde/pdag/bin/mtstrmd
Shared Symantec Processes
-------------------------
root 22233 1 0 Mar07 ? 00:00:04 /opt/VRTSpbx/bin/pbx_exchange
From https://www.veritas.com/support/en_US/article.100005044
Solution
1. Install the NetBackup client software on each failover node OK
2. Configure the SAN client on each node according to the instructions in this chapter. Ensure that the FT service is active on all of the failover nodes (bpclntcmd or AdminConsole > San Clients) OK
3. On the NetBackup master server, use the nbemmcmd command to register the virtual node with the clients, as follows:
a. To define NetBackup clustered clients add the virtual name to the EMM database:
nbemmcmd –addhost –machinename virtual_name –machinetype app_cluster OK
b. Add the virtual name to all clients in the node. For every client in the node, run the following command.
nbemmcmd –updatehost –add_server_to_app_cluster –machinename client_name –clustername virtual_name OK
The other virtual client tibcopr on petra, backup goes via ft and it's ok. When i switched tibcosoadb from SAN client frey to petra, tibcosoadb'sbackup on petra do not funciton. I don't know what is the problem...
Thanks for your response.
Best Regards,
03-08-2019 03:52 AM
All I can suggest is to log a Support call with Veritas.
There is unfortunately not much experience in this community with SAN Client; even less with clustered SAN Client.
03-08-2019 05:33 AM
03-08-2019 05:58 AM
Symantec/Veritas has officially registered port 1556 with iana.org.
https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xml
If anything needs to change, it's Oracle. Not NetBackup.
If you really want to change the PBX port number, you need to change it everywhere across your entire NBU domain.
03-08-2019 08:10 AM
I supposed that it's wrong change pbx port because is blocked by NBU.
How i may resolve this issue?
there is maybe a workaround?.
I don't understand why i have this error:
03/07/2019 17:12:41 - Error bpbrm (pid=269462) from client tibcosoadb.backup.local: ERR - Could not open file /usr/openv/netbackup/db/config/shm/tibcosoadb.backup.local_1551968976 to get shared memory information. Errno = 2: No such file or directory
Thanks and Regards
03-10-2019 09:37 PM
Incase of SAN client backups, its a shared memory (conceptually) between media server and bpbkar process on client. bpbkar thinks it is writting to a shared memory created by bptm and bptm reads from it to send data to storage hence you are getting these messages.
PBX port is for NetBackup and SAN client backups uses PBX. So if there is any conflict with port 1556 SAN client backups will fail.
If possible reboot the cluster node then try to perform backup. If it still fails then open a case with Veritas support as logs review is needed.
03-11-2019 02:46 AM
As long as Oracle is using port 1556, I doubt that a reboot will fix the problem.
03-11-2019 09:08 PM
Here's a blast from the past
https://vox.veritas.com/t5/NetBackup/Netbackup-Client-network-port-1556/td-p/602621
Back in 2013 you replied on a similar issue. Oracle DBA should change the Oracle Listener back to its default of Port 1521 and leave NetBackup on its default of 1556
Steve
03-11-2019 11:51 PM
LOL @Systems_Team
I had to look for that post as well because I could not remember the IANA url.
Default Oracle port numbers here: https://docs.oracle.com/cd/B19306_01/install.102/b15660/app_port.htm
03-12-2019 02:42 AM
Hi All,
thank you so much for assistance.
i told to dba dmin to change the port 1556, becasue it's exclusive to Netbackup. They told me if it's possible bind port 1556 pbx_exchange to single ip address, in order to use also 1556 for oracle in other ip address like this example:
192.168.1.10 1556 netbackup
192.144.1.4 1556 oracle
In a node of cluster, they would be perform a custom configuration in order to address packet by application to 1556 port correct, to nbu or oracle. Is it possible?
Meanwhile i receive intermittent oracle/sap failure on various client of cluster with "permission denied" on disk, like this example:
<16> non_mpx_backup_archive_verify_import: from client: ERROR: V-3-20003: Cannot open /dev/vx/dsk/TM1/TM1arc: Permission denied
Thanks,
Best Regards