Tags
Restrictions for Clusterware and Oracle ASM Upgrades
====================================================
Oracle recommends that you use CVU to check if here are any patches required for upgrading your existing Oracle Grid Infrastructure 11g release 2 or Oracle RAC database 11g Release 2 installations.
Be aware of the following restrictions and changes for upgrades to Oracle Grid Infrastructure installations, which consists of Oracle Clusterware and Oracle Automatic Storage Management (Oracle ASM):
1) To upgrade existing Oracle Clusterware installations to Oracle Grid Infrastructure 11g, your release must be greater than or equal to 10.1.0.5, 10.2.0.3, 11.1.0.6, or 11.2.
2) To upgrade existing Oracle Grid Infrastructure installations from 11.2.0.2 to a later release, you must apply patch 11.2.0.2.1 (11.2.0.2 PSU 1) or later.
3) To upgrade existing 11.1 Oracle Clusterware installations to Oracle Grid Infrastructure 11.2.0.3 or later, you must patch the release 11.1 Oracle Clusterware home with the patch for bug 7308467.
4) If you have Oracle ACFS file systems on Oracle Grid Infrastructure 11g release 2 (11.2.0.1), you upgrade Oracle Grid Infrastructure to any later version (11.2.0.2 or 11.2.0.3), and you take advantage of Redundant Interconnect Usage and add one or more additional private interfaces to the private network, then you must restart the Oracle ASM instance on each upgraded cluster member node.
5) Do not delete directories in the Grid home. For example, do not delete Grid_home/Opatch. If you delete the directory, then the Grid infrastructure installation owner cannot use Opatch to patch the grid home, and Opatch displays the error “checkdir error: cannot create Grid_home/OPatch’
6) To upgrade existing 11.2.0.1 Oracle Grid Infrastructure installations to Oracle Grid Infrastructure 11.2.0.2, you must first verify if you need to apply any mandatory patches for upgrade to succeed.
7) Oracle Clusterware and Oracle ASM upgrades are always out-of-place upgrades. With 11g release 2 (11.2), you cannot perform an in-place upgrade of Oracle Clusterware and Oracle ASM to existing homes.
8) If the existing Oracle Clusterware home is a shared home, note that you can use a non-shared home for the Oracle Grid Infrastructure for a Cluster home for Oracle Clusterware and Oracle ASM 11g release 2 (11.2).
9) With Oracle Clusterware 11g release 1 and later releases, the same user that owned the Oracle Clusterware 10g software must perform the Oracle Clusterware 11g upgrade. Before Oracle Database 11g, either all Oracle software installations were owned by the Oracle user, typically oracle, or Oracle Database software was owned by oracle, and Oracle Clusterware software was owned by a separate user, typically crs.
10) Oracle ASM and Oracle Clusterware both run in the Oracle Grid Infrastructure home.
11) During a major version upgrade to 11g release 2 (11.2), the software in the 11g release 2 (11.2) Oracle Grid Infrastructure home is not fully functional until the upgrade is completed. Running srvctl, crsctl, and other commands from the 11g release 2 (11.2) home is not supported until the final rootupgrade.sh script is run and the upgrade is complete across all nodes.
12) To manage databases in the existing earlier version (release 10.x or 11.1) database homes during the Oracle Grid Infrastructure upgrade, use the srvctl from the existing database homes.
13) During Oracle Clusterware installation, if there is a single instance Oracle ASM version on the local node, then it is converted to a clustered Oracle ASM 11g release 2 (11.2) installation, and Oracle ASM runs in the Oracle Grid Infrastructure home on all nodes.
14) With Oracle Clusterware 11g release 2 (11.2) and later, you can perform upgrades on a shared Oracle Clusterware home.
15) If a single instance (non-clustered) Oracle ASM installation is on a remote node, which is a node other than the local node (the node on which the Oracle Grid Infrastructure installation is being performed), then it will remain a single instance Oracle ASM installation. However, during installation, if you select to place the Oracle Cluster Registry (OCR) and voting disk files on Oracle ASM, then a clustered Oracle ASM installation is created on all nodes in the cluster, and the single instance Oracle ASM installation on the remote node will become nonfunctional.
Software download
=================
Below notes has the location for 11gR2 GI and RDBMS software
11.2.0.2 Patchset Location For Grid Infrastructure And RDBMS. (Doc ID 1223673.1)
11.2.0.3 Patchset Location For Grid Infrastructure And RDBMS. (Doc ID 1362393.1)
Pre-Upgrade Steps
=================
a) Configuring SSH on All Cluster Nodes :
Oracle Universal Installer uses the ssh and scp commands during installation to run remote commands on and copy files to the other cluster nodes. One must configure SSH so that these commands do not prompt for a password.
From 11.2 onwards you can configure SSH from the Oracle Universal Installer (OUI) interface during installation for the user account running the installation. The automatic configuration creates passwordless SSH connectivity between all cluster member nodes. Oracle recommends that you use the automatic procedure if possible.
b) Configuring the oracle User’s Environment:
Set the installation software owner user (grid, oracle) default file mode creation mask (umask) to 022 in the shell startup file. Setting the mask to 022 ensures that the user performing the software installation creates files with 644 permissions.
c) Installing rpm for checking Shared disk accessibility:
Install the operating system package cvuqdisk. Without cvuqdisk, Cluster Verification Utility cannot discover shared disks, and you receive the error message “Package cvuqdisk not installed” when you run Cluster Verification Utility
d) For each node, use Cluster Verification Utility to ensure that you have completed preinstallation steps. It can generate Fixup scripts to help you to prepare servers.
./runcluvfy.sh stage -pre crsinst -n girnar,himalaya -fixup -fixupdir /tmp/cvu_fixup -verbose
1. It automatially copies runfixup.sh script to all the specified nodes under /tmp/CVU_11.2.0.1.0_grid directory.
2. It checks for ntp.conf file on all the nodes. If it is available then try to see whether it is working or not. If it fails to check the liveliness then it gives error. Best is to rename ntp.conf on all the nodes so it will suggest to use CTSS and check will be passed.
NTP Configuration file check started…
Network Time Protocol Configuration(NTP) file not found on any of the nodes. Oracle Cluster Time Synchronization
Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes.
Result: Clock synchronization check using Network Time Protocol(NTP) passed
3. If any problem is detected then run fixup scripts which will resolve the basic requirement as mentioned above.
e) Software requirements:
Make sure that the minimum required OS packages are installed. Refer to below document
http://docs.oracle.com/cd/E11882_01/install.112/e22489/prelinux.htm#CIHFICFD (Refer section “2.8 Identifying Software Requirements”)
f) Apply all mandatory patches:
Apply all mandatory patches and verify the settings before starting the upgrade. Below notes has more details
Things to Consider Before Upgrading to 11.2.0.2 Grid Infrastructure (Doc ID 1312225.1)
Things to check before upgrade to 11.2.0.3 (Doc ID 1363369.1)
g) Setting Up the Network :
With this release, the single client access name (SCAN) is the hostname to provide for all clients connecting to the cluster. The SCAN is a domain name registered to at least one and up to three IP addresses, either in the domain name service (DNS) or the Grid Naming Service (GNS). The SCAN name must be at least one character long and no more than 15 characters in length, must be alphanumeric, and may contain hyphens (-). Oracle strongly recommend to use DNS or GNS for SCAN hostname resolution, To know more about SCAN please refer the below note
11gR2 Grid Infrastructure Single Client Access Name (SCAN) Explained (Doc ID 887522.1)
h) Creating and Setting permissions for the Grid infrastructure directories:
Since we are doing out of place upgrade, Grid infrastructure require to be installed on seprate directories. Grid infrastructure directories can not be under ORACLE_BASE.
$ORACLE_BASE=/u01/app
mkdir -p /u03/grid
chown grid:oinstall /u03/grid
chmod -R 775 /u03/grid
i)Backup of below mentioned files and directories:
1. Backup ORACLE_HOME, CRS_HOME and ASM_HOME on all the nodes.
2. ocr & voting disk from node1 as root user.
$CRS_HOME\bin\ocrconfig –export /tmp/ocr.bkp
Location for Voting disk file can be found with command:
$ crsctl query css votedisk
The recommended OS command for backup of Voting/OCR disk files that are placed on raw storage device is “dd”.
Eg. dd if=voting_disk_name of=backup_file_name
3. Backup oracle directory located under /etc and the init scripts and /etc/inittab
For backup of existing clusterware refer the below note
What Files To Backup In Oracle Clusterware (CRS) Installation (Doc ID 754369.1)
j) Unset enviornment variable :
For the installation owner running the installation, if you have environment variables set for the existing installation, then unset the environment variables $ORACLE_HOME and $ORACLE_SID, as these environment variables are used during upgrade.
$ unset ORACLE_BASE
$ unset ORACLE_HOME
$ unset ORACLE_SID
$ unset ORA_CRS_HOME
$ unset ORA_ASM_HOME
k) Check the readiness for clusterware upgrade using cluvfy :
Verify if your clusterware is ready for upgrade using below command
runcluvfy.sh stage -pre crsinst -upgrade [-n node_list] [-rolling] -src_crshome src_Gridhome -dest_crshome dest_Gridhome -dest_version dest_version [-fixup [-fixupdir path]] [-verbose]
Refer to below document to know more about this command
http://docs.oracle.com/cd/E11882_01/install.112/e22489/procstop.htm#BABEHGJG
Upgrade clusterware and ASM
===========================
Ensure that you have information you will need during installation, including the following:
1) An Oracle base location for Oracle Clusterware.
2) An Oracle Grid Infrastructure home location that is different from your existing Oracle Clusterware location
3) A SCAN name
4) Privileged user operating system groups to grant access to Oracle ASM data files (the OSDBA for ASM group), to grant administrative privileges to the Oracle ASM instance (OSASM group), and to grant a subset of administrative privileges to the Oracle ASM instance (OSOPER for ASM group)
5) root user access, to run scripts as root during installation
Take a vnc connection or any other xterm of session, we are ready to install Oracle clusterware. Login as the owner of existing 10g/11gR1 CRS and run the runInstaller command to start the GUI
Note : If your existing CRS and ASM are owned by different users then refer the below note
ASMCA Fails When Upgrading to 11.2 due to Different ASM/CRS User or Non-default ASM Name (Doc ID 1113073.1)
$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
$ unset ORACLE_BASE
$ unset ORACLE_HOME
$ unset ORACLE_SID
$ unset ORA_CRS_HOME
$ unset ORA_ASM_HOME
$./runInstaller
++ Select “upgrade clusterware and ASM option” from the installer screen
++ On the node selection page, select all nodes.
++ Select installation options as prompted.
++ Provide the SCAN name when prompted.
Note : If you have an existing Oracle ASM instance, you can either upgrade it at the time that you install Oracle Grid Infrastructure, or you can upgrade it after the installation, using Oracle ASM Configuration Assistant (ASMCA). However, be aware that a number of Oracle ASM features are disabled until you upgrade Oracle ASM, and Oracle Clusterware management of Oracle ASM does not function correctly until Oracle ASM is upgraded, because Oracle Clusterware only manages Oracle ASM when it is running in the Oracle Grid Infrastructure home. For this reason, Oracle recommends that if you do not upgrade Oracle ASM at the same time as you upgrade Oracle Clusterware, then you should upgrade Oracle ASM immediately afterward.
++ When prompted, run the rootupgrade.sh script on each node in the cluster that you want to upgrade.
1) Run the script on the local node first. The script shuts down the earlier release installation, replaces it with the new Oracle Clusterware release, and starts the new Oracle Clusterware installation.
If it fails on first node then stop there and resolve the rootupgrade.sh failure and only after that proceed with next steps
2) After the script completes successfully, you can run the script in parallel on all nodes except for one, which you select as the last node. When the script is run successfully on all the nodes except the last node, run the script on the last node.
++ After running the rootupgrade.sh script on the last node in the cluster, if you are upgrading from a release earlier than release 11.2.0.1, and left the check box with ASMCA marked, as is the default, then Oracle ASM Configuration Assistant runs automatically, and the Oracle Clusterware upgrade is complete. If you uncloaked the box on the interview stage of the upgrade, then ASMCA is not run automatically.
If an earlier version of Oracle Automatic Storage Management is installed, then the installer starts Oracle ASM Configuration Assistant to upgrade Oracle ASM to 11g release 2 (11.2). You can choose to upgrade Oracle ASM at this time, or upgrade it later.
Oracle recommends that you upgrade Oracle ASM at the same time that you upgrade the Oracle Clusterware binaries. Until Oracle ASM is upgraded, Oracle databases that use Oracle ASM cannot be created. Until Oracle ASM is upgraded, the 11g release 2 (11.2) Oracle ASM management tools in the Grid home (for example, srvctl) will not work.
Completing an Oracle Clusterware Upgrade when Nodes Become Unreachable
=======================================================================
If some nodes become unreachable in the middle of an upgrade, then you cannot complete the upgrade, because the upgrade script (rootupgrade.sh) did not run on the unreachable nodes. Because the upgrade is incomplete, Oracle Clusterware remains in the previous release version. You can confirm that the upgrade is incomplete by entering the command crsctl query crs activeversion.
To resolve this problem, run the rootupgrade.sh command with the -force flag using the following syntax:
Grid_home/rootupgrade -force
For example:
# /u01/app/11.2.0/grid/rootupgrade -force
This command forces the upgrade to complete. Verify that the upgrade has completed by using the command crsctl query crs activeversion. The active version should be the upgrade release.
Below is an example of force upgrade of a 6-node GI cluster.
Assuming rootupgrade.sh finishes successfully on node1, node2, but fails on node3 for non-fixable problem or node3 is down for HW issue, and the decision is to go with force upgrade, you can execute “$NEW_HOME/rootupgrade.sh” on node4 and node5, then node6. If rootupgrade.sh fails on any of the last three nodes (node4-6), you can troubleshoot and fix if possible. Assuming rootupgrade.sh succeeds on node4 and node6, but fails on node5 and can’t be fixed, now there’s two nodes in the cluster that can not be upgraded – node3 and node5, since the decision is to force upgrade, you can execute “”$NEW_HOME/rootupgrade.sh -force” as root on any one of the successful nodes (node1, node2, node4, node6) to finish the upgrade.
Do not execute rootupgrade.sh with force option before rootupgrade.sh is executed on all nodes, otherwise only the nodes where rootupgrade.sh has been executed successfully will be upgraded, the rest including those nodes where rootupgrade.sh has not been executed yet can only be removed and added back to the cluster later.
To force upgrade, “olsnodes -s” must show status “Inactive” for the failed nodes. A node will have “Inactive” status if the node is down, Oracle Clusterware is down or ocssd.bin is down.
After the force upgrade is done, if OUI is still running, go back to OUI to continue, otherwise refer to note 1360798.1 to complete Configuration Assistants. The failed nodes can be removed and added back to the cluster whenever condition allows. Continue with rest of the steps in the note only if force upgrade fails.
kpjustin 