Troubleshooting on Café Com Cloud

Patching an Engineered System

Thu, 16 Aug 2018 16:54:11 -0300

Everyone in Oracle DBA world knows the key value of use an Engineered System, such as Exadata or Supercluster.

In a mission critical on premises database, as a DBA we need to consolidate many databases, plataform in different versions into an Engeneed system.

In a Engeneered System world we have different types of patch deployment that consists in a group of many other patches for differents versions of databases / applications / system /services. They are called Oracle Engineered Systems Quarterly Patch Deployment (QPD).

The key advantage of using QPD is stay with best practices patches that was recommeded by Oracle. The QDP are available for download every 4 months and if you have a contract, you can let Oracle apply for you via ACS / platinum contract. More information, you can see in https://www.oracle.com/assets/as-quarterly-patch-deployment-3042102.pdf.

Personal experiences on Engineered Systems(Exadata, Supercluster) patch apply I have some few experiences on patch apply in many enviroments even Exadata or Supercluster systems.

First, I always take an exachk. It’s extremely important to compare how health your environment are before apply. Always download the last available on MOS “Oracle Exadata Database Machine exachk or HealthCheck (Doc ID 1070954.1)”

Second, I recommend this note lecture:

Exadata Database Machine and Exadata Storage Server Supported Versions (Doc ID 888828.1)

Oracle SuperCluster Supported Software Versions - All Hardware Types (Doc ID 1567979.1)

PS. in this notes there are some references to others Engeneered Systems. If you have on your enviroment, please take a look on MOS.

So before download, as every patch I extremaly recomments read the README of each patch before start (it seems obvious, but I knew many DBAs that don’t follow this secret rule) and follow the prechecks as Oracle recommendations such as space, check if there are any conflict with another patch, etc.

Some tips based on my personal experience - In most of cases, during patching some errors related by datapatch (on Database apply) occurs. Don’t be affraid and take a look at:

Queryable Patch Inventory - Issues/Solutions for ORA-20001: Latest xml inventory is not loaded into table (Doc ID 1602089.1)

12.1 : Datapatch Fails with ERROR “KUP-04004,KUP-04017,KUP-04118,KUP-04095,ORA-29913”," fatal: libjli.so or libpicl.so.1: open failed" (Doc ID 2085653.1)

- If you have a segregated environment, always apply patches on DEV or UAT environment.

Examples of database patch apply

=====

Don’t be affraid if you get an error. It’s happen with everyone:

Log file for this invocation: /u01/app/grid/cfgtoollogs/sqlpatch/sqlpatch_2278_2018_08_16_14_39_41/sqlpatch_invocation.log

Connecting to database…OK Note: Datapatch will only apply or rollback SQL fixes for PDBs that are in an open state, no patches will be applied to closed PDBs. Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation (Doc ID 1585822.1) Bootstrapping registry and package to current versions…done

Queryable inventory could not determine the current opatch status. Execute ‘select dbms_sqlpatch.verify_queryable_inventory from dual’ and/or check the invocation log /u01/app/grid/cfgtoollogs/sqlpatch/sqlpatch_2278_2018_08_16_14_39_41/sqlpatch_invocation.log for the complete error. Prereq check failed, exiting without installing any patches.

Please refer to MOS Note 1609718.1 and/or the invocation log /u01/app/grid/cfgtoollogs/sqlpatch/sqlpatch_2278_2018_08_16_14_39_41/sqlpatch_invocation.log for information on how to resolve the above errors.

SQL Patching tool complete on Thu Aug 16 14:39:59 2018

=====

It’s so good when everything was work as design:

Log file for this invocation: /u01/app/grid/cfgtoollogs/sqlpatch/sqlpatch_23787_2018_08_16_14_47_57/sqlpatch_invocation.log

Current state of SQL patches: Bundle series DBBP: ID 180717 in the binary registry and not installed in any PDB

Adding patches to installation queue and performing prereq checks… Installation queue: For the following PDBs: CDB$ROOT PDB$SEED Nothing to roll back The following patches will be applied: 27547374 (DATABASE BUNDLE PATCH 12.1.0.2.180717)

Installing patches… Patch installation complete. Total patches installed: 2

Validating logfiles… Patch 27547374 apply (pdb CDB$ROOT): SUCCESS logfile: /u01/app/grid/cfgtoollogs/sqlpatch/27547374/22329708/27547374_apply__MGMTDB_CDBROOT_2018Aug16_14_48_44.log (no errors) Patch 27547374 apply (pdb PDB$SEED): SUCCESS logfile: /u01/app/grid/cfgtoollogs/sqlpatch/27547374/22329708/27547374_apply__MGMTDB_PDBSEED_2018Aug16_14_52_35.log (no errors) SQL Patching tool complete on Thu Aug 16 14:55:29 2018

=====

See you soon! :-)

The infamous jdbc closed connection

Mon, 11 Sep 2017 12:19:34 -0300

Sometimes you, as an DBA, is blamed for everything. The database is slow, unavailable, unpatched, and the list goes on and on.

Sometimes, in rare situations, you can prove then wrong :D

Last week we have been called to analyze a intermitent application issue. The app team blame the database, showing the “java.sql.SQLException: Closed Connection” error on app logs. Everything at database level was checked out, and rechecked again, until they call us. Long debugging hours and still no results, we tried a different approach. What about sniffing the eth card at app server..checking the communication flow between APP and Database.

And so we did it:

tcpdump -i eth0 tcp port 1521 -A -s1500 | awk ‘$1 ~ “ORA-” {i=1;split($1,t,“ORA-”);while (i <= NF) {if (i == 1) {printf("%s",“ORA-“t[2])}else {printf("%s “,$i)};i++}printf(”\n”)}’

This give us the nice output:

bla@app_blaserver:~ # tcpdump -i eth0 tcp port 1521 -A -s1500 | awk ‘$1 ~ “ORA-” {i=1;split($1,t,“ORA-”);while (i <= NF) {if (i == 1) {printf("%s”,“ORA-“t[2])}else {printf("%s “,$i)};i++}printf(”\n”)}’ tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 1500 bytes ORA-01403:no data found ORA-00913:too many values ORA-01403:no data found ORA-01403:no data found ORA-01403:no data found ORA-01403:no data found ORA-01403:no data found ORA-01403:no data found ORA-01403:no data found ORA-01403:no data found ORA-01438:value larger than specified precision allowed for this column ORA-06512:at line 2 ORA-00937:not a single-group group function ORA-01403:no data found ORA-01403:no data found ORA-00937:not a single-group group function ORA-01403:no data found ORA-00937:not a single-group group function ORA-01403:no data found ORA-00937:not a single-group group function ORA-01403:no data found

The ORA-01403 is expected after the fetch of each cursor being processed - no bad news here.

Hummmm…and when the ORA-01438/ORA-06512/ORA-00937 are raised what happens to the connection? You got it right?

After checking what was causing the errors, the intermittent issue stops, everyone was happy - incluing the DBA team :D

(You need to adapt the script to fit the listener port and eth card in your box, okay?)

As always feedbacks are very welcome.

See you around,

Hang.

db_multiblock_read_count - to be or not to be?

Thu, 25 May 2017 18:37:21 -0300

Sometimes you wanna to use things with default options…and sometimes the default is not good enough.

We all know that db_file_multiblock_read_count is bumped by default on startup, usually you get 128 blocks per read. Why?

From Oracle documentation, this value is high, but CBO will not favor full tables scans.

“Even though the default value may be a large value, the optimizer will not favor large plans if you do not set this parameter. It would do so only if you explicitly set this parameter to a large value.”

Okay. But Why? From a 10053 trace within a session, you can see that the CBO will compute db_file_multiblock_read_count always as “8”…even if you see 128 in spfile.

CBO you always use 8, if you dont set db_file_multiblock_read_count explicitly. How to know if you set db_file_multiblock_read_count as 128? Use this:

SELECT nvl(value,’NULL’) AS value FROM v$spparameter WHERE name = ‘db_file_multiblock_read_count’;

If this return *NULL*, you have not set db_file_multiblock_read_count, Oracle is showing you 128, but CBO uses 8. Sad no?

You all read the documentation and know the effects of bumping db_file_multiblock_read_count high. Full table scans will happen more often and as a result, in a OLTP system, usually you will get a call from your boss (hehe)

But there is a way to set correctly the db_file_multiblock_read_count…by testing…for example, take a look on the block below. Its silly, I know, but it can give you a magic value for this parameter.

set serveroutput on DECLARE l_count PLS_INTEGER; l_time PLS_INTEGER; l_starting_time PLS_INTEGER; l_ending_time PLS_INTEGER; BEGIN dbms_output.put_line(‘dbfmbrc seconds’); FOR l_dbfmbrc IN 1..64 LOOP EXECUTE IMMEDIATE ‘ALTER SESSION SET db_file_multiblock_read_count=’||l_dbfmbrc; EXECUTE IMMEDIATE ‘ALTER system flush buffer_cache’; EXECUTE IMMEDIATE ‘ALTER session disable parallel dml’; l_starting_time := dbms_utility.get_time(); SELECT /+ full(t) / count(*) INTO l_count FROM big_table t; l_ending_time := dbms_utility.get_time(); l_time := round((l_ending_time-l_starting_time)/100); dbms_output.put_line(l_dbfmbrc||’ ‘||l_time); END LOOP; END; /

(Thanks to Chris Antognini, you’re great)

With this block (be carefull, it will take 60-90 minutes depending of the big table’s size, as a advice, use a 5GB table), you will have values, plot a graph and see the behavior. Dont choose a high value for OLTP.

See you around!

Using oradebug - 10053 and 10046 events

Mon, 27 Feb 2017 12:00:06 -0300

Hello There! Hope this article finds you well. Today we will talk about oradebug, this little wonder in oracle, and how to set the 10046 and 10053 events to troubleshoot performance.

First of all, what is oradebug?

Oradebug is a sqlplus tool, intended to oracle support personel only - however, it can be very handy in a bunch of situations. You can use it to set specific troubleshooting events, use to diagnose on which interface your RAC is using for the interconnect messages, use to execute a hanganalyze ( the command, not me, okay? =] ), system state dumps, errorstack dumps and the list goes on.

How do I use it?

First of all, you need to attach oradebug to a specific session/process.

Using the ospid:

Get the os pid for the process, usually with and ps command, or getting the thread in windows environment. With the ospid (in this example 1177)

oradebug setospid 1177;

If you try to attach oradebug to an inactive process, you will receive:

SQL> oradebug setospid 13050; ORA-00072: process “13050” is not active

If everything went smoothly, you will receive:

SQL> oradebug setospid 16792; Oracle pid: 150, Unix process pid: 16792, image: oracle@BLASERVER12

Using the SID to attach oradebug:

First of all you need to find out the orapid for the session, running the sql below:

select p.pid orapid from v$process p, v$session s where s.sid = &SID and p.addr = s.paddr /

Example:

SQL> select p.pid orapid from v$process p, v$session s where s.sid = &SID and p.addr = s.paddr / 2 3 4 5 6 Enter value for sid: 651 old 4: where s.sid = &SID new 4: where s.sid = 651

ORAPID ———- 152

SQL> oradebug setorapid 152; Oracle pid: 152, Unix process pid: 5392, image: oracle@BLASERVER12 (TNS V1-V3)

You are ready to go!

Setting the 10046 event

oradebug event 10046 trace name context forever, level 12;

Setting the 10053 event

oradebug event 10053 trace name context forever, level 1;

Why this is important?

You can use the 10046 to get sql specific informations, such as object, parsing, elapsed time, wait events, bind variables. It is an alternativa for dbms_monitor and dbms_system, another way to generate a sessions trace, and it could be “tkprofed” as well.

If you wanna to understand the optimizer’s decicions, there is no better way that using a 10053 trace. You will understand the cost estimates, access paths, query block names, peeked binds, optimizer parameters used during the parse.

In another post I’ll give detail about the levels, and why we type “name context forever”. This is a how to post only =]

See you around!

Troubleshooting on Café Com Cloud

Patching an Engineered System

The infamous jdbc closed connection

db_multiblock_read_count - to be or not to be?

SELECT nvl(value,’*NULL*’) AS value FROM v$spparameter WHERE name = ‘db_file_multiblock_read_count’;

Using oradebug - 10053 and 10046 events

SELECT nvl(value,’NULL’) AS value FROM v$spparameter WHERE name = ‘db_file_multiblock_read_count’;