Mar 28, 2017

Apply PSU to Rac 12c on AIX7.1 (not pluggable database environment)

Recently I had opportunity to apply PSU to a rac environment in 12cR1 (not pluggable database architecture) on AIX7.1. The steps I followed are the next one:

1.- Make sure that I have the Opatch last version  (OPatch Version: 12.2.0.1.8) on all nodes (same version)
2.- Download PSU 24917825 for RAC Infra.
3.- Download PSU 24732082 for databases
4.- Take backup of ORACLE_HOME of Databases and Grid Infraestructure (also of your OraInventory) for all nodes (I suggest )

I have my install medios in the path: /u01/medios/ for my environments
(same Path on all nodes)

The PSU application on RAC nodes  can be performed  different ways. I choose this method and I will explain  you.

Apply PSU for Infraestructure GRID


The steps are the following (to do with grid user):

cd /u01/medios
unzip p24917825_121020_AIX64-5L.zip   

Note: For Opatch version 12.2.0.1.8 is not necessary to create response File with emocmrsp command.

This steps to do with root user but you will get error OPATCHAUTO-72046: Invalid wallet parameters (Doc ID 2150070.1). Here we will review patch conflicts:

export ORACLE_HOME=/u01/app/12.1.0/grid
/u01/app/12.1.0/grid/OPatch/opatchauto apply /u01/medios/24917825 –analyze


If there are not patch conflicts we can proceed to apply patch to Infra. I remember you that this patch is rolling installable. I take 30’ minutes to apply (with root user):


[nodo01]#> export ORACLE_HOME=/u01/app/12.1.0/grid
[nodo01]#>export export PATH=$PATH:$ORACLE_HOME/OPatch
[nodo01]#>/u01/app/12.1.0/grid/OPatch/opatchauto apply /u01/medios/24917825  -oh /u01/app/12.1.0/grid

OPatchauto session is initiated at Sat Feb  4 04:09:12 2017

System initialization log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2017-02-04_03-09-19AM.log.

Session log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2017-02-04_03-09-52AM.log
The id for this session is DM67

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.1.0/grid
Patch applicablity verified successfully on home /u01/app/12.1.0/grid


Verifying patch inventory on home /u01/app/12.1.0/grid
Patch inventory verified successfully on home /u01/app/12.1.0/grid


Bringing down CRS service on home /u01/app/12.1.0/grid
Prepatch operation log file location: /u01/app/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_cmh-racn1_2017-02-04_04-16-44AM.log
CRS service brought down successfully on home /u01/app/12.1.0/grid


Start applying binary patch on home /u01/app/12.1.0/grid
Successfully executed command: /usr/sbin/slibclean

Binary patch applied successfully on home /u01/app/12.1.0/grid


Starting CRS service on home /u01/app/12.1.0/grid
Postpatch operation log file location: /u01/app/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_cmh-racn1_2017-02-04_04-34-16AM.log
CRS service started successfully on home /u01/app/12.1.0/grid


Verifying patches applied on home /u01/app/12.1.0/grid
Patch verification completed with warning on home /u01/app/12.1.0/grid

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:cmh-racn1
CRS Home:/u01/app/12.1.0/grid
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/medios/24917825/21436941
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2017-02-04_03-18-06AM_1.log

Patch: /u01/medios/24917825/24732082
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2017-02-04_03-18-06AM_1.log

Patch: /u01/medios/24917825/24828633
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2017-02-04_03-18-06AM_1.log

Patch: /u01/medios/24917825/24828643
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2017-02-04_03-18-06AM_1.log



OPatchauto session completed at Sat Feb  4 04:42:40 2017
Time taken to complete the session 33 minutes, 30 seconds

Now is necessary to repeat same process  with all rest of nodes. When we have the Grid Infraestructure on all nodes with the same version, we can to apply the psu to database software.

Apply PSU to database software


Just like te previous situation, this part is rolling installation. We will start with the node one. First step is shutdown all db service in that node: (all following command with oracle user)
srvctl stop instance -d  database –I instance_name
Then we’ll check Patch conflict:

[nodo01]#>cd /u01/medios/24732082
[nodo01]#>opatch prereq CheckConflictAgainstOHWithDetail -ph ./


If all previous results fine, we’ll proceed with apply the database PSU:

[nodo01]#>cd  /u01/medios/24732082
[nodo01]#>opatch  apply
`

Oracle Interim Patch Installer version 12.2.0.1.8
Copyright (c) 2017, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
  from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version    : 12.2.0.1.8
OUI version       : 12.1.0.2.0
Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2017-02-04_04-01-02AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   23054246  24006101  24732082  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

This node is part of an Oracle Real Application Cluster.
Remote nodes: 'cmh-racn2' 'cmh-racn3' 'cmh-racn4'
Local node: 'cmh-racn1'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/12.1.0/dbhome_1')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying sub-patch '23054246' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'

Patching component oracle.rdbms.dv, 12.1.0.2.0...

Patching component oracle.rdbms.rsf, 12.1.0.2.0...

Patching component oracle.rdbms.rman, 12.1.0.2.0...

Patching component oracle.rdbms, 12.1.0.2.0...

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...

Patching component oracle.ldap.rsf, 12.1.0.2.0...
Skip copying to "/u01/app/oracle/product/12.1.0/dbhome_1/lib/libzt12.a" because it is the same as
the file in incoming patch "/u01/medios/24732082/23054246/files/lib/libzt12.a"

Patching component oracle.install.deinstalltool, 12.1.0.2.0...
Skip copying to "/u01/app/oracle/product/12.1.0/dbhome_1/lib/libldapjclnt12.so" because it is the same as
the file in incoming patch "/u01/medios/24732082/23054246/files/lib/libldapjclnt12.so"

Patching component oracle.ldap.rsf.ic, 12.1.0.2.0...

Patching component oracle.oracore.rsf, 12.1.0.2.0...

Patching component oracle.ctx, 12.1.0.2.0...

Patching component oracle.xdk, 12.1.0.2.0...
Skip copying to "/u01/app/oracle/product/12.1.0/dbhome_1/bin/xmldiff" because it is the same as
the file in incoming patch "/u01/medios/24732082/23054246/files/bin/xmldiff"
Skip copying to "/u01/app/oracle/product/12.1.0/dbhome_1/bin/xvm" because it is the same as
the file in incoming patch "/u01/medios/24732082/23054246/files/bin/xvm"
Skip copying to "/u01/app/oracle/product/12.1.0/dbhome_1/bin/xmlpatch" because it is the same as
the file in incoming patch "/u01/medios/24732082/23054246/files/bin/xmlpatch"
Skip copying to "/u01/app/oracle/product/12.1.0/dbhome_1/bin/xsl" because it is the same as
the file in incoming patch "/u01/medios/24732082/23054246/files/bin/xsl"
Skip copying to "/u01/app/oracle/product/12.1.0/dbhome_1/bin/xmlcg" because it is the same as
the file in incoming patch "/u01/medios/24732082/23054246/files/bin/xmlcg"

Patching component oracle.nlsrtl.rsf, 12.1.0.2.0...
Skip copying to "/u01/app/oracle/product/12.1.0/dbhome_1/bin/lxegen" because it is the same as
the file in incoming patch "/u01/medios/24732082/23054246/files/bin/lxegen"
Skip copying to "/u01/app/oracle/product/12.1.0/dbhome_1/bin/lcsscan" because it is the same as
the file in incoming patch "/u01/medios/24732082/23054246/files/bin/lcsscan"
Skip copying to "/u01/app/oracle/product/12.1.0/dbhome_1/bin/lxchknlb" because it is the same as
the file in incoming patch "/u01/medios/24732082/23054246/files/bin/lxchknlb"

Patching component oracle.xdk.parser.java, 12.1.0.2.0...
Skip copying to "/u01/app/oracle/product/12.1.0/dbhome_1/bin/xml" because it is the same as
the file in incoming patch "/u01/medios/24732082/23054246/files/bin/xml"

Patching component oracle.ctx.atg, 12.1.0.2.0...
Skip copying to "/u01/app/oracle/product/12.1.0/dbhome_1/lib/libanllexer12.so" because it is the same as
the file in incoming patch "/u01/medios/24732082/23054246/files/lib/libanllexer12.so"
Applying sub-patch '24006101' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'

Patching component oracle.sqlplus, 12.1.0.2.0...

Patching component oracle.rdbms, 12.1.0.2.0...

Patching component oracle.network.listener, 12.1.0.2.0...

Patching component oracle.network.rsf, 12.1.0.2.0...

Patching component oracle.rdbms.dv, 12.1.0.2.0...

Patching component oracle.rdbms.rman, 12.1.0.2.0...

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...

Patching component oracle.sqlplus.ic, 12.1.0.2.0...

Patching component oracle.rdbms.rsf, 12.1.0.2.0...
Applying sub-patch '24732082' to OH '/u01/app/oracle/product/12.1.0/dbhome_1'

Patching component oracle.rdbms.install.plugins, 12.1.0.2.0...

Patching component oracle.rdbms.rsf, 12.1.0.2.0...
Skip copying to "/u01/app/oracle/product/12.1.0/dbhome_1/rdbms/mesg/oraus.msb" because it is the same as
the file in incoming patch "/u01/medios/24732082/24732082/files/rdbms/mesg/oraus.msb"

Patching component oracle.tfa, 12.1.0.2.0...

Patching component oracle.rdbms.rman, 12.1.0.2.0...

Patching component oracle.rdbms, 12.1.0.2.0...

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
…………………..
…………………. (Here I cut this part becaouse es very very long ☹   )
………………….

Patching in rolling mode.

Remaining nodes to be patched:
'cmh-racn2' 'cmh-racn3' 'cmh-racn4'
What is the next node to be patched?



In that part the installation will ask us a question about which is the name of next hostname of Rac for the appy PSU. In this moment we will start again the database service only in this node (node 1), while we need shutdown the database service in the next one node (nodo2). When we have the instance up in the node 1 and down the service in the node , we can proceed with the installer.

When we have finished the installation on all nodes, we need to load the modified sql into the database.

Load modified sql into database (datapatch)

This is only necessary to apply once time, preferably node 1 with user oracle:


cd $ORACLE_HOME/OPatch
$ ./datapatch -verbose
SQL Patching tool version 12.1.0.2.0 Production on Wed Feb 22 05:35:11 2017
Copyright (c) 2012, 2016, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_29229074_2017_02_22_05_35_11/sqlpatch_invocation.log

Connecting to database...OK
Bootstrapping registry and package to current versions...done
Determining current state...done

Current state of SQL patches:
Patch 22674709 (Database PSU 12.1.0.2.160419, Oracle JavaVM Component (Apr2016)):
 Installed in the binary registry only
Bundle series PSU:
 ID 170117 in the binary registry and ID 160419 in the SQL registry

Adding patches to installation queue and performing prereq checks...
Installation queue:
 Nothing to roll back
 The following patches will be applied:
   22674709 (Database PSU 12.1.0.2.160419, Oracle JavaVM Component (Apr2016))
   24732082 (DATABASE PATCH SET UPDATE 12.1.0.2.170117)

Installing patches...
Patch installation complete.  Total patches installed: 2

Validating logfiles...
Patch 22674709 apply: SUCCESS
 logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/22674709/20077876/22674709_apply_CMH_2017Feb22_05_36_19.log (no errors)
Patch 24732082 apply: SUCCESS
 logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/24732082/20919987/24732082_apply_CMH_2017Feb22_05_43_58.log (no errors)
SQL Patching tool complete on Wed Feb 22 05:44:33 2017
You have mail in /usr/spool/mail/oracle



And now when we execute query to catalog we can see that our database is update with the last PSU:


We can to do the same with de oraInventory  for the grid and database ORACLE_HOME:

Remains to explain what will happen with respect to OJVM upgrade. (I’ll have soon)

Bye : )

Mar 23, 2017

Error de respaldo con Oracle 10gR2 y sobre TDP de IBM (TSM) ORA-07445: exception encountered: core dump [krbbsi()+75] [SIGFPE] [Integer divide by zero]


Les comento un tips que me fue muy útil: Hace poco luego de configurar respaldos de base de datos bajo un ambiente linux redhat 5 con Oracle 10gR2 con TDP de IBM, empezé a ejecutar respaldos de base de datos y este se caia con el error siguiente:

input archive log thread=1 sequence=228078 recid=64803 stamp=939379423
input archive log thread=1 sequence=228079 recid=64805 stamp=939379431
channel ch01: starting piece 1 at 23-MAR-17
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch01 channel at 03/23/2017 10:44:37
RMAN-10038: database session for channel ch01 terminated unexpectedly
RMAN>
Recovery Manager complete.


Al revisar el archivo de alertas se ve el siguiente error:

Thu Mar 23 10:44:37 2017
Errors in file /oracle/product/admin/g5/udump/cdcorona_ora_19026.trc:
ORA-07445: exception encountered: core dump [krbbsi()+75] [SIGFPE] [Integer divide by zero] [0x0027E1343] [] []
Thu Mar 23 11:14:38 2017


No encontré nada con el primer argumento señalado para error ORA-7445 ni en el soporte de Metalink de Oracle ni en el soporte de IBM. Asì que me puse a realizar pruebas y saqué el siguiente parámetro del canal RMAN:

  allocate channel ch01 type 'sbt_tape'
  PARMS= 'BLKSIZE=6291456,ENV=(TDPO_OPTFILE=/usr/tivoli/tsm/client/oracle/bin64/tdpo.opt)';


 Luego de sacar dicho parámetro que aun no encuentro que significa realmente (obviamente tiene que ver con el tamaño de bloque de I/O pero no sé que hace aún exactamente ) El backup anduvo sin problemas. Este problema me pasó con otros dos motores Oracle 10.2.0.4.

Espero les sirva.