06-18-2013 05:39 AM
Hi All,
we have 7.1 master server(Solaris 10), No media server. I was in middle of upgrade, everything went well until i observed a message
EMM interface initialization failed, status = 77
After install script completed i am not able to run any commands related to emm
how ever nbdb_ping works fine
Database [NBDB] is alive and well on server [NB_inblruxbkup01].
Attached install trace file
Solved! Go to Solution.
06-20-2013 06:39 AM
If this is a test then remember there are real issue when you upgrade to 7.5
You need to run the upgrade then take NBU down straight away and patch to 7.5.0.5
Once this is done the catalog migration to EMM wil start and only once that has completed will NBU start to fully function
Hope this helps
06-18-2013 06:17 AM
looks like your attachment is missing...
06-18-2013 06:37 AM
pbx service may not be running so start it up:
# /opt/VRTSpbx/bin/vxpbx_exchanged start
06-18-2013 10:45 PM
Ensure all of the interfaces on th Master have valid information for resolution.
If you have more than 110 VLANs defined and configured there is aknown issue for that.
06-18-2013 11:52 PM
Can you post the error info?
06-19-2013 12:17 AM
Hi,
Please check the vm.conf file under in /usr/openv/volmgr/
MM_SERVER_NAME = Master server Name
AUTO_PATH_CORRECTION=YES
Regards,
Pawan
06-19-2013 02:39 AM
Hi All
Attached install trace file
vm.conf
MM_SERVER_NAME = inblruxbkup01.lehman.com
MEDIA_ID_BARCODE_CHARS = 1 8 1:2:3:4:5:6
VERBOSE
There are couple of Interfaces in master server 1) is production and 2) is management IP
i checked pbx was started during installation
06-19-2013 02:41 AM
I also see lot of nbpem processes running
root 7625 1 0 16:04:43 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8444 1 0 16:08:36 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 7654 1 0 16:04:53 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8343 1 0 16:08:15 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8231 1 0 16:07:32 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 7780 1 0 16:05:25 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 7825 1 0 16:05:36 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8056 1 0 16:07:00 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8261 1 0 16:07:43 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8431 1 0 16:08:25 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8298 1 0 16:07:53 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
06-19-2013 02:42 AM
@Amit,
could you please run the command bpp -x and see if the process nbemm is running ?
./nathan
06-19-2013 02:42 AM
@Amit
bpps -x
./nathan
06-19-2013 02:44 AM
Hi Nathan,
inblruxbkup01# bpps -x
NB Processes
------------
root 5598 1 0 15:57:42 ? 3:53 /usr/openv/netbackup/bin/nbemm
root 6287 1 0 15:59:08 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 6659 1 0 16:00:59 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 6672 1 0 16:01:11 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 5627 1 0 15:57:44 ? 4:27 /usr/openv/netbackup/bin/nbjm
root 6849 1 0 16:01:53 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 6002 1 0 15:58:04 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 5601 1 0 15:57:43 ? 0:37 /usr/openv/netbackup/bin/nbrb
root 5612 1 0 15:57:43 ? 0:21 /usr/openv/netbackup/bin/bprd
root 5164 1 0 15:57:24 ? 0:00 /usr/openv/netbackup/bin/vnetd -standalone
root 6123 1 0 15:58:15 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 7685 1 0 16:05:04 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 5659 1 0 15:57:48 ? 4:02 /usr/openv/netbackup/bin/nbrmms
root 6238 1 0 15:58:47 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 5630 1 0 15:57:44 ? 0:17 /usr/openv/netbackup/bin/bpdbm
root 7498 1 0 16:03:59 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 5694 1 0 15:57:49 ? 0:01 /usr/openv/netbackup/bin/nbsl
root 6882 1 0 16:02:04 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 5741 1 0 15:57:53 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 7022 1 0 16:02:25 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 5708 1 0 15:57:50 ? 0:06 /usr/openv/netbackup/bin/nbars
root 7056 1 0 16:02:36 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 7519 1 0 16:04:11 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 5167 1 0 15:57:24 ? 0:01 /usr/openv/netbackup/bin/bpcd -standalone
root 6760 1 0 16:01:21 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 4987 1 0 15:56:32 ? 3:06 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/
root 5230 1 0 15:57:26 ? 0:09 /usr/openv/netbackup/bin/nbaudit
root 7689 1 0 16:05:15 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 5731 1 0 15:57:53 ? 0:03 /usr/openv/netbackup/bin/nbsvcmon
root 5618 1 0 15:57:44 ? 0:00 /usr/openv/netbackup/bin/bpcompatd
root 5631 5630 0 15:57:44 ? 0:19 /usr/openv/netbackup/bin/bpjobd
root 5655 1 0 15:57:47 ? 0:18 /usr/openv/netbackup/bin/nbim
root 7605 1 0 16:04:21 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 6223 1 0 15:58:36 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 5749 1 0 15:57:53 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 6209 1 0 15:58:25 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 7611 1 0 16:04:32 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8321 1 0 16:08:04 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 6338 1 0 15:59:19 ? 0:01 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 6256 1 0 15:58:57 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 6795 1 0 16:01:32 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8072 1 0 16:07:11 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8055 1 0 16:06:59 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 6408 1 0 15:59:30 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 5724 1 0 15:57:52 ? 0:02 /usr/openv/netbackup/bin/nbcssc -a NetBackup
root 6821 1 0 16:01:43 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 6884 1 0 16:02:15 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 7499 1 0 16:04:00 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 6660 1 0 16:01:00 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 5637 1 0 15:57:46 ? 4:51 /usr/openv/netbackup/bin/nbstserv
root 8159 1 0 16:07:21 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 7625 1 0 16:04:43 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8444 1 0 16:08:36 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 7654 1 0 16:04:53 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8343 1 0 16:08:15 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8231 1 0 16:07:32 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 7780 1 0 16:05:25 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 7825 1 0 16:05:36 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8056 1 0 16:07:00 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8261 1 0 16:07:43 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8431 1 0 16:08:25 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 8298 1 0 16:07:53 ? 0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 20225 1 0 15:07:51 ? 0:01 /usr/openv/netbackup/bin/admincmd/bpstsinfo -UPDATE
MM Processes
------------
root 5607 1 0 15:57:43 ? 3:09 /usr/openv/volmgr/bin/ltid
root 5640 1 0 15:57:46 ? 3:07 vmd -v
Shared Symantec Processes
-------------------------
root 26174 1 0 Jun 17 ? 0:04 /opt/VRTSpbx/bin/pbx_exchange
inblruxbkup01# bpps -x | grep nbpem | wc
44 440 4224
i wonder why so many nbpem process are running
06-19-2013 04:07 AM
Hi Amit,
stop the NBU services on master server and apply the Maintenance Release 4 and check the
Device mapping file also.
Pawan
06-19-2013 04:12 AM
I would stop NBU, make sure all processes are stopped, make sur PBX is started, restart NBU and see in what situtaion you find yourself. Seems only 1 policy was converted, how many polciies do you have on this system ?
I would also in paralell open a ticket with Symantec Support.
06-19-2013 05:26 AM
This is our fresh test setup which was initially running with 7.1 and i wanted to get some idea about 7.5 upgrade, i read some where that now imagedb is part of NBDB which was not the case with 7.1, how ever i see lot of errors like
inblruxbkup01# bperror -hoursago 4 -U
TIME SERVER/CLIENT TEXT
06/19/2013 13:43:47 unknown - nbjm waiting - nbrb is not running
06/19/2013 13:53:50 unknown - nbjm waiting - nbrb is not running
06/19/2013 14:03:53 unknown - nbjm waiting - nbrb is not running
06/19/2013 14:13:56 unknown - nbjm waiting - nbrb is not running
06/19/2013 14:23:58 unknown - nbjm waiting - nbrb is not running
06/19/2013 14:34:01 unknown - nbjm waiting - nbrb is not running
06/19/2013 14:44:03 unknown - nbjm waiting - nbrb is not running
06/19/2013 14:54:06 unknown - nbjm waiting - nbrb is not running
06/19/2013 15:04:09 unknown - nbjm waiting - nbrb is not running
06/19/2013 15:14:11 unknown - nbjm waiting - nbrb is not running
06/19/2013 15:24:14 unknown - nbjm waiting - nbrb is not running
and
06/19/2013 15:49:29 inblruxbkup01.lehman.com - connection to
nbevtmgr::VRTS_NBU_PEM_Channel on host
inblruxbkup01.lehman.com failed after 7 attempts
that is the reason i guess for emm not responding , i also noticed vxdbms.conf is still with older date (before upgrade) looks like nbu has not been upgraded properly.
Yes there is only 1 policy(catalog backup)
06-19-2013 05:46 AM
@Amit,
Could you please clarify exactly your upgrade scenario ? i am a little bit confused
What is your current major version ? 7.1 ? 7.5 ?
what is you actual patch version ? 7.1.0.3 ? 7.5.0.4? 7.5.0.5?
Please run the following command:
bpclntcmd -get_local_server_patch_version
06-19-2013 05:50 AM
inblruxbkup01# bpclntcmd -get_local_server_patch_version
7.5.0.0
it is 7.5 but i think not fully upgraded or with some issues
06-19-2013 06:13 AM
@Amit,
ok. You are using the version 7.5 which mean at this time your system is not patched at all.
As you mentionned in the beginning, you are using a Solaris 10 master server right?
Is it a cluster node or standalone ?
If it's a Cluster node , there are some particular steps that you need to follow. You can consult you netbackup installation guide for UNIX and go to the cluster section. You will find more details about it.
if it's a standalone, then extract the archive and apply the patch
I assume that your images have been migrated successfully and your catalog database is ok (you can run nbcc tools for checking the integrity of your database.
If you have upload the binaries to your master server, you can find all relevant .pdf documentation under the following directory:
<path-to-you-upload-folder>/NetBackup_7.5_LinuxR_x86_64/Doc
mine is as follow:
[root@master]# ls
favicon.ico NetBackup_AdminGuide_Sybase.pdf
NetBackup_3rdpartyattributions.pdf NetBackup_AdminGuide_Vault.pdf
NetBackup_AdminGuide_BMR.pdf NetBackup_AdminGuide_VMware.pdf
NetBackup_AdminGuide_Cloud.pdf NetBackup_AdvDisk_Guide.pdf
NetBackup_AdminGuide_ClusteredMasterServer.pdf NetBackup_BAR_GS_Guide.pdf
NetBackup_AdminGuide_DB2.pdf NetBackup_Commands.pdf
NetBackup_AdminGuide_EntVault.pdf NetBackup_Dedupe_Guide.pdf
NetBackup_AdminGuide_HighAvailability.pdf NetBackup_DeviceConfig_Guide.pdf
NetBackup_AdminGuide_Hyper-V.pdf NetBackup_GettingStarted_Guide.pdf
NetBackup_AdminGuideII_UNIXServer.pdf NetBackup_Install_UNIX.pdf
NetBackup_AdminGuideII_WinServer.pdf NetBackup_Install_Win.pdf
NetBackup_AdminGuide_Informix.pdf NetBackup_LiveUpdate_Guide.pdf
NetBackup_AdminGuideI_UNIXServer.pdf NetBackupManuals.html
NetBackup_AdminGuideI_WinServer.pdf NetBackup_OperGuide_Vault.pdf
NetBackup_AdminGuide_LotusNotes.pdf NetBackup_OST_Disk_Guide.pdf
NetBackup_AdminGuide_MSExchg_Win.pdf NetBackup_RefGuide_StatusCodes.pdf
NetBackup_AdminGuide_MSSQL_Win.pdf NetBackup_Release_Notes.pdf
NetBackup_AdminGuide_NDMP.pdf NetBackup_RepDirector_Guide.pdf
NetBackup_AdminGuide_Oracle.pdf NetBackup_SANClient_Guide.pdf
NetBackup_AdminGuide_SAP.pdf NetBackup_SecEncryp_Guide.pdf
NetBackup_AdminGuide_Search.pdf NetBackup_Troubleshoot_Guide.pdf
NetBackup_AdminGuide_SharePoint.pdf symc_logo.gif
NetBackup_AdminGuide_SnapshotClient.pdf
[root@master]# pwd
/data/netbackup/pkgs/7.5/NetBackup_7.5_LinuxR_x86_64/Doc
[root@master]#
Before patching:
1. reboot completly
2. make sure all process are up with bpps -x
3. Try connect to admin console
4. run NBCC tools to check the integrity of your catalog
5. check EMM integrity with the following command:
nbemmcmd -getemmserver
nbemmcmd -listmedia -allrecords -brief
nbemmcmd -listhosts
If all of the above sounds good, try to patch and tack the logs ;)
06-19-2013 07:46 AM
No to belabor a point, but for this specific error condition, I think this forum is not the appropriate place to do triage and look for resolution of your problem. The NetBackup support team should be handling this directly. You would be better served if you open a support case for this and will get much better assistance to resolve this matter. I am a Back Line support engineer and I am currently working a customer situation almost exaclty as yours. It is not a trivial matter.
06-19-2013 08:13 AM
I agree Jaime, since this is not a prod environment i was not keen to open up a case but looks like i will have to do that
@nathan , this is a standalone server, problem i see here is , although emm services are running none of emm commands, (nbemmcmd -listhosts, nbemmcmd -listmedia -mediaid) are respoding , I even tried to check consistency with bpdbm, but it failed.. Looks like i will have to contact support
inblruxbkup01# bpdbm -consistency 2
Checking NetBackup database consistency
NetBackup 7.5 2012020723
Failed to load the media db: cannot connect on socket (25)
check images for clients
I wil keep all of you updated
06-20-2013 02:11 AM
@Amit,
Just for information:
If i understand correctly, you are migrating from 7.1 to 7.5?
What was the previous patch level before migrating to 7.5 (eg. 7.1.0.3) ?
Did you follow all the steps and pre-req before migrating from 7.1 to 7.5 (deactivate policies, media, SLP, STU, no jobs running ...)?
Sometimes, if you miss one of those pre-req, it can lead to such situation. What Jaime recommend you is correct, this can be perfectly handle by Symantec support.
Good luck ;)