02-16-2015 05:14 PM
Hi All,
The Oracle DBA is seeing Oracle logins at 6 hour intervals, throughut the day, for each Oracle SID. Since there are over 100 instances, this is messing up his log. I am suspicious of 'nbdisco' on NetBackup 7.6 client, and wondering...
1. What it's polling frequency for 'nbdisco', and if that can be changed ?
2 Will the "REPORT_CLIENT_DISCOVERIES = FALSE" bp.conf setting disable polling for Oracle instances altogether, or just disable the reporting of instances back to the master (while continueing to poll for them) ?
3. Is there any documentation for 'nbdisco', other than its mention in "NetBackup 7.6 Oracle Admin Guide" pages 53-54 ?
-------------- Configuration --------------
NetBackup 7.6.0.3 (probably Linux based)
NetBackup 7.6.0.3 client on RHEL 6.5.
Oracle 11.2.0.3, 11.2.0.4, 12.1.0.2
____________________________
Thank you all reading.
Solved! Go to Solution.
02-16-2015 08:16 PM
Hi,
If you put it to false the service/process wont run so I would say in your case that is a good idea.
http://www.symantec.com/docs/HOWTO106494
02-17-2015 12:43 AM
From what I know to nbdisco it does not login to Oracle. It only detects running Oracle instances used by Oracle Intelligent Policies.
Update: I did a strace on the nbdisco in-house, i did not catch any calls for Oracle binary
20405 11:56:31 statfs("/opt/oracle/product/11gR2p1", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=2580302, f_bfree=1382353, f_bavail=1251281, f_files=1310720, f_ffree=1244722, f_fsid={0, 0}, f_namelen=255, f_frsize=4096}) = 0 20405 11:56:31 stat("/opt/oracle/product/11gR2p1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 20405 11:56:31 stat("/opt/oracle/product/11gR2p1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 20405 11:56:32 read(17, "#\n# This file is used by ORACLE "..., 8191) = 789 20405 11:56:32 open("/var/opt/oracle/oratab", O_RDONLY) = -1 ENOENT (No such file or directory
02-16-2015 08:16 PM
Hi,
If you put it to false the service/process wont run so I would say in your case that is a good idea.
http://www.symantec.com/docs/HOWTO106494
02-17-2015 12:43 AM
From what I know to nbdisco it does not login to Oracle. It only detects running Oracle instances used by Oracle Intelligent Policies.
Update: I did a strace on the nbdisco in-house, i did not catch any calls for Oracle binary
20405 11:56:31 statfs("/opt/oracle/product/11gR2p1", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=2580302, f_bfree=1382353, f_bavail=1251281, f_files=1310720, f_ffree=1244722, f_fsid={0, 0}, f_namelen=255, f_frsize=4096}) = 0 20405 11:56:31 stat("/opt/oracle/product/11gR2p1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 20405 11:56:31 stat("/opt/oracle/product/11gR2p1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 20405 11:56:32 read(17, "#\n# This file is used by ORACLE "..., 8191) = 789 20405 11:56:32 open("/var/opt/oracle/oratab", O_RDONLY) = -1 ENOENT (No such file or directory
02-23-2015 04:38 PM
Thank you Riaan, Nicola. I do not have access to the Oracle servers, so results are delayed while waiting for remote engineers to carry out changes.
Status Update:
1. Nbdisco was disabled today (awaiting results).
REPORT_CLIENT_DISCOVERIES = FALSE
2. Found another setting, to change the frequency of nbdisco probes, (but did not deploy it).
NBARS_DISCOVERY_TIMER = <seconds>
3. From strace performed by Nicolai, it looks like 'oratab' is being read, but would be interested in knowing exactly how nbdisco discovers databases.
4. Since unidentified database logins are reported every 6 hours (5AM, 11AM, 5PM, 11PM), I have requested a full strace of nbdisco during one of those times (awaiting results).
5. Thinking that the database logins are made over a TCP connection (local or remote), I have also requested a 'tcpdump' on a specific database listen port, at one of those times.
I plan to update this post when when more info is available from the remote engineers, and furthermore, I will get a 1st hand look myself during the 1st week of March.
Thank You!
Ken W
02-24-2015 04:01 AM
Thanks for the update Ken.
I can add that nbdisco will find running instances of oracle even when they are removed from oratab
My guess is that both oratab and list of running processes are investigated by nbdisco.
03-05-2015 09:50 AM
Nbdisco is now fully disabled, but the databases continue to get unidentified logins, once per day.
Nbdisco is not involved, therefore further discussion is outside the scope of this forum, but I will add some info in closing:
1. The VOM host agent is running on these hosts, and we are going to disable that as a test.
2. The DBA has captured the Oracle login as "SYS as SYSDBA", and snippets of the SQL code being executed follow ...
select SYS_CONTEXT('USERENV', 'SERVER_HOST'), SYS_CONTEXT('USERENV', 'DB_UNIQUE_NAME'), SYS_CONTEXT('USERENV', 'INSTANCE_NAME'), SYS_CONTEXT('USERENV', 'INSTANCE'), STARTUP_TIME, SYS_CONTEXT('USERENV', 'DB_DOMAIN'), SYS_CONTEXT('USERENV', 'SERVICE_NAME') from v$instance
select decode(upper(failover_method), NULL, 0 , 'BASIC', 1, 'PRECONNECT', 2 , 'PREPARSE', 4 , 0), decode(upper(failover_type), NULL, 1, 'NONE', 1, 'SESSION', 2, 'SELECT', 4, 'TRANSACTION', 8, 1), failover_retries, failover_delay, flags, nvl(replay_initiation_timeout, 300), decode(upper(session_state_consistency), 'STATIC', 16, 0) from service$ where name = :1
CREATE PFILE='/var/tmp/zOXQaQXsE3' FROM SPFILE='/n01/app/oracle/product/12.1.0.2/dbee/dbs/spfiledbad1.ora'
SELECT executions, end_of_fetch_count, elapsed_time/px_servers elapsed_time, cpu_time/px_servers cpu_time, buffer_gets/executions buffer_gets FROM (SELECT sum(executions) as executions, sum(case when px_servers_executions > 0 then px_servers_executions else executions end) as px_servers, sum(end_of_fetch_count) as end_of_fetch_count, sum(elapsed_time) as elapsed_time, sum(cpu_time) as cpu_time, sum(buffer_gets) as buffer_gets FROM gv$sql WHERE executions > 0 AND sql_id = :1 AND parsing_schema_name = :2)
SELECT executions, end_of_fetch_count, elapsed_time/px_servers elapsed_time, cpu_time/px_servers cpu_time, buffer_gets/executions buffer_gets FROM (SELECT sum(executions_delta) as EXECUTIONS, sum(case when px_servers_execs_delta > 0 then px_servers_execs_delta else executions_delta end) as px_servers, sum(end_of_fetch_count_delta) as end_of_fetch_count, sum(elapsed_time_delta) as ELAPSED_TIME, sum(cpu_time_delta) as CPU_TIME, sum(buffer_gets_delta) as BUFFER_GETS FROM DBA_HIST_SQLSTAT s, V$DATABASE d, DBA_HIST_SNAPSHOT sn WHERE s.dbid = d.dbid AND bitand(nvl(s.flag, 0), 1) = 0 AND sn.end_interval_time > (select systimestamp at TIME ZONE dbtimezone from dual) - 7 AND s.sql_id = :1 AND s.snap_id = sn.snap_id AND s.instance_number = sn.instance_number AND s.dbid = sn.dbid AND parsing_schema_name = :2)
select CDB from v$database
Thank You all !
Ken W
03-05-2015 09:59 AM
Maybe change the passwords and see who complains :)
03-05-2015 10:47 AM
Good tip, but that would up to the DBA as he said "I do not have access to the Oracle servers".
03-16-2015 10:29 AM
FINAL UPDATE & CONCLUSION:
1. Changing password was a no-go, because any agent that can 'su oracle', can login as SYS without a password.
2. Nbdisco and VRTSsfmh discovery was disabled using 'REPORT_CLIENT_DISCOVERIES = FALSE' (above) and VOM Server User Guide Appendix D: https://sort.symantec.com/public/documents/vom/6.0/windowsandunix/productguides/html/User_Guide/apds...
But, unidentified Oracle logins still continue.
3. Then, the Oracle DBA changed the Oracle Enterprise Manager host identification from physical to virtual hostname, and those logins disappeared. The DBA still does not understand why the hostname change made a difference, but does prove OEM is the cause, thus taking Symantec off the hook.
CONCLUSION:
Unidentifed Oracle logins originated from Oracle Enterprise Manager, not Symantec agents.
Hope this helps someone else out and saved them a week or two of trouble shooting !
Ken W
03-17-2015 01:49 AM
Thank you so much for sharing the conclusion +1