Oracle 10g RAC installation for SLES9 system with installation scripts.

Shared disk media – NFS on NetApp network server

 

Alexei Roudnev, senior system and network engineer, Exigengroup USA.

Contact: Alexei_Roudnev@exigengroup.com

Warning. It is not OFFICIAL document. It is not supported, verified or approved by any vendor (SuSe/Novell, Oracle, Network Appliances). Use it on your own risk.


TABLE OF CONTENT:

Introduction. 1

Prerequisites and hardware. 3

Table of scripts: 3

System installation. 4

X11 access. 7

Starting installation. 8

Now it’s time to run installation scripts. 9

Data base installation. 20

Starting, stopping and managing. 29

References: 31

APPENDIX 1. Modified orarun scripts. 31

 


 

Tar file with all installation scripts available here:
 SLES9-x64-ORA10RAC.tgz or here: http://ftp.portera.com/Linux/SLES9-x64-ORA10RAC.tgz .

Introduction.

This document describes Oracle 10g release 1 (10.1.0) installation on SUSE Linux Enterprise Server 9 (SLES9) for AMD64/ET64T (x86-64) architecture, using our installation scripts. Scripts are part of our installation script library for Oracle and Linux, which was designed as automation tool and allows easy Oracle and Oracle Real Application Cluster installation and reinstallation (with guaranteed results) I believe that installation for i386 do not make any difference, and installation with other shared media (not NFS) differ in a few details only (DBCA options and file system mounting).

 

Installation is SUSE specific (uses ‘orarun’ rpm heavily), but can be modified for other Linux and can be used as ‘installation guide’ without scripts.

 

Idea of these scripts became obvious, after I read first 2 or 3 instructions how to install Oracle – it looked as some kind of magical dances around Linux, Oracles and file systems, and did not guaranteed reproduced results, especially for cluster. When it comes to SUSE Linux Enterprise Server (SLES), I found first step of such automation – ‘orarun’ package written by SUSE (now Novell) engineers. Unfortunately, this package (extremely useful for Oracle9 RAC cluster) did not eliminate numerous manual operations, especially for Oracle 10g RAC cluster installation.

 

As a result, instead of dancing around system, I began writing sets of scripts, which automate all installation for combinations of OS, Oracle and installation mode. Instead of ‘edit file’ or ‘prepare password-less ssh access’ or ‘run root’sh file as a root’, you run one of the scripts. Scripts are ordered, with numbers increasing step after step.

 

You can ask, why I did not wrote single ‘INSTALL.sh’ command, which do everything. Reason is Oracle/Linux/Shared-media complexity – single scripts works fine only if installation environment is exactly the same, and have a high chance to break if something got wrong. Other idea – system administrator must understand, what is changing, in such case as Oracle cluster installation. So, instead of single huge script, I prepared about 20 smaller once, which reflects 1:1 standard installation procedure for Oracle cluster on SuSe Enterprise Linux. It provides flexibility – if system administrator do not want to do some step by another way (for example, use another file location, or use another method for ssh access), he can always do part of installation manually, skipping few scripts.

 

Document below describe Oracle10 Real Application Cluster installation on SLES9 Enterprise Server. It was tested on:

-         Linux servers: DELL PowerEdge 2850 server;

-         NAS server – NetApp FAS270c server, with NFS license;

-         SUSE Linux Enterprise Server 9,, with Service Pack1, for x86-64 platform;

-         Oracle 10g Release 1 (10.1) for Linux x86-64;

 

I expect these scripts to work without any changes for SLES9 and Oracle10 in 32-bit mode.

 

Script sources are printed in document in blue; few scripts (files from FILE.d directory) are printed in appendix. I recommend using scripts, but if you are skilled Linux and Oracle administrator and want to do everything manually, just follow these sources as an example.

 

You do not required to read this entire document and investigate all scripts. In reality, you can do everything very fast, in 3 steps:

-         Follow prerequisites chapter and prepare installation;

-         Run scripts one by one, baying attention to scripts which runs in parallel with Oracle installer (instead of root.sh);

-         Rave everything done in 1 hour.

But, of course, the best method is – read document, understand what script is doing, run script.

 

Notice. I recommend following all tiny details of this manual. You can remove many packages and eliminate some steps, but if it will not work, it makes difficult understanding a reason. For example, I always install EVERYTHING except GNOME, and it allows me to eliminate numerous manual package selection and very possible errors. Future ‘orarun’ packages should resolve all dependencies for Oracle, but for now, it is better to avoid unnecessary work and just install all C++/Development tools and Linux tools.

 

Prerequisites and hardware.

You must have (these all are commercial products, available for evaluation and/or under special development licenses):

-         SuSe Linux Enterprise Server 9 (better with Service Pack 1);

-         Oracle 10.1.0.3 for AMD64/ET64 (2 CD or 2 cpio files) with Cluster Ready Services (1 CD or 1 cpio file).;

-         2 AMD64/ET64 servers (I used DELL 2850 servers with 2 CPU each);

-         Network Appliance (NetApp) NAS system with NFS license;

-         1Gbit Ethernet switch (or VLAN on big enterprise switch).

 

Logical connection in my example:

 

Table of scripts:

File CONFIG.sh: 7

File 010-install-orarun.sh: 8

File 015-edit-oraprofile.sh: 9

File 018-edit-rcoracle.sh: 9

File 020-start-rcoracle.sh: 10

File 030-check-oracle-user.sh: 11

File 040-create-mounts.sh: 11

File 060-setup-xntp.sh: 12

File 070-edit-services.sh: 12

File 080-ssh-genkeys.sh: 13

File 090-verify-ssh-access.sh: 14

File 200-InstCRS.1sh: 15

File 220-runRootsh.sh: 17

File 230-check-crs.sh: 18

File 300-check-env.sh: 19

File 310-InstOracle.1sh: 20

File 340-run-netca.1sh: 21

File 400-create-database.sh: 21

File /etc/profile.d/oracle.sh (FILE.d/oracle.sh): 29

File /etc/sysconfig/oracle (FILE.d/sysconfig.oracle): 31

File /etc/init.d/oracle (FILES.d/init.oracle): 33

 

 

System installation.

 

1) Allocate names and IP addresses. You will need 4 IP in access network and 2 IP in storage network.

Virtual IP are used by Oracle cluster as service IP and can float from failed server to server which takeover it’s functions. Add records for primary IP (not virtual) into hosts file (IMPORTANT! I saw problems if IP resolved into fully qualified names, not into single world names).

We used in these example:

Server

eth0-ip

eth0 virtual ip

eth1-ip

testrac11

testrac11 = 10.23.32.111

testrac11-vip = 10.23.32.113

testrac11-1 = 10.254.23.111

testrac12

testrac12 = 10.23.32.112

testrac12-vip = 10.23.32.114

testrac12-1 = 10.254.23.112

 

/etc/hosts:

10.23.32.111    testrac11     testrac11.sjclab.mydomain.com

10.23.32.112    testrac12     testrac12.sjclab.mydomain.com

10.254.32.111  testrac11-1 testrac11-1.sjclab.mydomain.com

10.254.32.112  testrac12-1 testrac12-1.sjclab.mydomain.com

10.23.32.113    testrac11-vip

10.23.32.114    testrac12-vip

 

I recommend to allow JUMBO frames in storage vlan – to do it, you must set up MTU=9000 on NetApp server and allow big MTU or jumbo frames on the switch. Do not set up jumbo frames on access vlan (except if you have reason to do it).

 

Everything will work without jumbo frames, but can create extra load on Linux servers and on the switch.

2) Create volumes on NFS server (NetApp). I created QTrees.

 

Volume name

Size

Mount as

Mount options

Goal

/voldb

10Gb

/ora/voldb

tcp,rsize=32768,wsize=3268,bg,intr,noac

Database files

/volcrs

400Mb

/ora/volcrs

tcp,rsize=32768,wsize=3268,bg,intr,noac

CRS control files

/vollog

2Gb

/ora/vollog

tcp,rsize=32768,wsize=3268,bg,intr,noac

Flash recovery area and archive logs


I used existing volumes, just mounting them few times.

3) Install SLES9 from CD. Select all except 'gnome' and few other sections:  [1]

Configure IP interfaces. If you will use jumbo frames, set up mtu=9000 on eth1 interface (using YaST2 -> Network Devices -> Change -> Interface -> Detailed setting):

DO NOT FORGET ABOUT DUPLEX PROBLEM – set up ports in AUTO on the switch and verify (using ethtool command on Linux), that server negotiated port duplex and speed with the switch.[2]

4) Install Service Pack 1 for SLES9 (recommended). DO NOT INSTALL orarun package. DO NOT INSTALL  gcc_old package.

This step is important, because Service Pack 1 addressed few performance problems (which influenced Oracle). Oracle cluster will work without it, but do not count on performance under heavy load – system memory allocation is not optimal without service pack.

Follow Service Pack 1 instruction. I install it, running ‘Yast2 -> software -> Patch CD Update’, and selecting ‘Service Pack’ location from my NFS server (so I never insert CD).

5) Prepare Oracle installation disks (I use NFS from our image server).

You will need Oracle/DB and Oracle/CRS directories (first with all CD's from Database, second with CD for Cluster Ready Services) from Oracle 10g for Linux x86-64. You can use CD and copy files from them, install from CD directly, or (if you have account and can download) you can download files from Oracle:[3]

ship.crs.lnxx86-64.cpio          ship.db_Disk2.lnxx86-64.cpio.gz
ship.db_Disk1.lnxx86-64.cpio.gz

 

And expand them, for example:

  mkdir /image/UNIX/Oracle10
 cd /image/UNIX/Oracle10
 mkdir CRS
 mkdir DB
 cd CRS
 gzip -d < $DOWNLOAD_FILES/ship.crs.lnxx86-64.cpio | cpio -idmv
 cd ..
 mkdir DB
 cd DB
 gzip -d < $DOWNLOAD_FILES/ship.db_Disk1.lnxx86-64.cpio.gz | cpio -idmv
 gzip -d < $DOWNLOAD_FILES/ship.db_Disk2.lnxx86-64.cpio.gz | cpio -idmv

(You must agree with development license if you download these files).

Anyway, you will have these directories (for example):

# ls /image/UNIX/Oracle10/CRS:
 Disk1


 
 
 
# ls /image/UNIX/Oracle10/DB:
Disk1  Disk2

 

Important. If you use Windows NFS server, do not copy disks on Windows - these CD's contain symlinks, so you must use Unix for unarchiving and copy operations.

X11 access.

Oracle installation requires X11 screen (X11 is Unix windows system). There are many X11 systems available (and you always have one when are working from Linux console in graphical mode). I usually use ‘CYGWIN’ package for Windows, when I must use Microsoft windows screen, installing:

-    Basic cygwin set;

-    Basic X11 set;

-         Open SSH;

 

After it, I can login into SuSe server using such commands:

-         Start cygwin terminal (click on icon);

-         startx & - opens xterm window;

-         slogin –X root@hostname

 

Or, better, allow XDM on SUSE Linux (open YaST2 -> system -> /etc/sysconfig editor, search for  DISPLAYMANAGER_REMOTE_ACCESS’,

Set it to ‘yes’, Apply, then run from console: ‘init 3’, then ‘init 5’), then run X in query mode:

-         X –query host-name

 

Important notices:

-         To change user when you have X11 access, use ‘sux’ instead of ‘su’. For example: ‘sux – oracle’;

-         To login into remote host with X11 forwarding, run ‘slogin –X ….’.

 

You can use any other method to get X11 access (xhosts, etc…). Just be prepared – you will need X11 access. The best method is use Linux console in graphical mode.

Starting installation.

1) Login to both nodes.

Now, login as root on both servers (open 2 xterm windows, 1 from first server, 1 from second). For example, I run from system console:

-         Click on terminal icon, and have terminal for the first server, run ‘sux –‘ to became root;

-         Click on terminal icon, then run ‘slogin –X root@second-server’ and login onto second node (with X11 forwarding).

 

 

Now, make our scripts available on the same NFS which you used for Oracle (or on any other shared media)

(I create symlink as

  ln –s /image/UNIX/SuSe9/Oracle10RAC /INST

So now I can simple enter ‘cd /INST’ to change into installation directory.)

2) Edit CONFIG.sh.

 

These file contains configuration for our installation scripts. It is in reality standard shell profile. I marked by bold lines, which are very likely to be changed.

File CONFIG.sh:

# CONFIG.sh file
#
#   Mount points - /ora/voldb, /ora/volcrs, /ora/vollog
#   Symlinks - /oradb, /oracrs, /oralog
#   Mount options - should be
VOLDB=fas-1a-1:/vol/vol0/NFS01
VOLCRS=fas-1a-1:/vol/vol0/NFS01
VOLLOG=fas-1b-1:/vol/vol0/NFS03
# Node names
NODES="testrac11 testrac12"
NODESP="testrac11-1 testrac12-1"
#
#
# XNTP server IP
XNTP_IP=10.23.32.1
#  Installation.
# We will create CRS subdirectory here
DIR_CRS=/volcrs/CRS
DIR_DB=/voldb/DB
DIR_LOG=/vollog/LOGS
#
# RPM and installation disks locations
#
RPMs=`/bin/pwd`/../RPMs
ORACLE10=`/bin/pwd`/../../Oracle10-amd64/DB
CRS10=`/bin/pwd`/../../Oracle10-amd64/CRS
ORA10CD1=$ORACLE10/Disk1
CRS10CD1=$CRS10/Disk1
RPMORARUN=$RPMs/orarun-1.8-109.5.i586.rpm
#
# Oracle HOME ID's
ORAID=10.1.0/db_1
CRSID=10.1.0/crs_1
#
# SID for the test database (we use it when edit /etc/oratab
#
if [ `hostname` = "testrac11" ]
then
     SID=test1
else
     SID=test2
fi

 

Here is a list of installation scripts:

010-install-orarun.sh
015-edit-oraprofile.sh
018-edit-rcoracle.sh
020-start-rcoracle.sh
030-check-oracle-user.sh
040-create-mounts.sh
060-setup-xntp.sh
070-edit-services.sh
080-ssh-genkeys.sh
090-verify-ssh-access.sh
200-InstCRS.1sh
210-runInventory.sh
220-runRootsh.sh
230-check-crs.sh
300-check-env.sh
300README.txt
310-InstOracle.1sh
320-RunRootSh.sh
340-run-netca.1sh
400-create-database.1sh
CONFIG.sh


Scripts which have suffix .1sh must be running on 1 node only; scripts with .sh must be running on both nodes. You can run scripts OR read their content and repeat operation manually.

Now it’s time to run installation scripts.

To run script, type in command:

            sh script-name

or

            sh script-number*sh

I recommend reading every script before running and understanding, what it is doing. Be careful to check output for possible errors., Most scripts can be easily rune few times, so you can always fix parameter (or edit script) and run script again (or can do step manually).

 

All scripts are designed to run as root – they change owner to oracle if necessary.

3) 010-install-orarun.sh - installs orarun, unlock oracle user and set up password.

On both nodes, run script 010 – ‘sh 010*.sh’. Enter new password for user 'oracle'.

File 010-install-orarun.sh:
#!/bin/bash
#       Install orarun package
#

 
 
 
 
. CONFIG.sh
rpm -U $RPMORARUN
passwd oracle
chsh -s /bin/bash oracle

Script:

-         Install orarun RPM;

-         Unlocks ‘oracle’ user by setting password and changing shell to executable one.

 

Result:

testrac12:/INST # sh 010*
Updating etc/sysconfig/oracle...
Changing password for oracle.
New password:
Bad password: it is based on a dictionary word
Re-enter new password:
Password changed
Changing login shell for oracle.
Shell changed.

4) 015-edit-oraprofile.sh - copy our version of oracle profile and edit it (set up SID).

On both nodes, run script 15.#!/bin/bash

File 015-edit-oraprofile.sh:

#
#       Edit /etc/profile.d/oracle.sh and oracle.sh
#
. CONFIG.sh
test -f /etc/profile.d/oracle.sh-ORIG && cp /etc/profile.d/oracle.sh /etc/profile.d/oracle.sh-ORIG
sed "s+ORACLE_SID=test1+ORACLE_SID=$SID+" FILES.d/oracle.sh > /tmp/oracle.sh && cp /tmp/oracle.sh /etc/profile.d/oracle.sh
chmod a+x /etc/profile.d/oracle.sh
chown oracle /etc/profile.d/oracle.sh
cd /etc/profile.d
if [ ! -f ~oracle/.profile ]
then
     cd ~oracle || exit 1
     cp -r  /etc/skel/.??* ~oracle
     touch .profile
     find . -print | xargs chown oracle
fi

Script:

-         Make reserved copy of orarun profile file;

-         Install our version of this file (see appendix);

-         Set up ORACLE_SID variable (notice, that it is different on different nodes);

-         Make this script editable for oracle user (not required but simplify many things);

-         Copy standard files into ~oracle (orarun creates user with empty directory);

-         Change owner for these files.

 

Results (SID must be different):

testrac12:/INST # sh 015*sh
testrac12:/INST # grep SID /etc/profile.d/oracle.sh
  ORACLE_SID=test2
  export ORACLE_BASE ORACLE_HOME ORA_NLS33 ORACLE_SID PATH LD_LIBRARY_PATH CLASSPATH TNS_ADMIN

5) 018-edit-rcoracle.sh - copy our version of oracle startup files for Oracle10 cluster.

On both nodes, run script 018.

File 018-edit-rcoracle.sh:

#!/bin/bash
#
#       Edit /etc/init.d/oracle
#
. CONFIG.sh
if [ ! -d /etc/init.d/ORIG ]
then
    mkdir -p /etc/init.d/ORIG
    cp /etc/init.d/oracle /etc/init.d/ORIG/.
fi
if [ ! -d /etc/sysconfig/ORIG ]
then
    mkdir -p /etc/sysconfig/ORIG
    cp /etc/sysconfig/oracle /etc/sysconfig/ORIG/.
fi
#
cp FILES.d/init.oracle /etc/init.d/oracle
cp FILES.d/sysconfig.oracle /etc/sysconfig/oracle

Script:

-         Makes reserved copy of init script and sysconfig file;

-         Install our version of these files (see appendix);


Results:

testrac11:/INST # sh 018*sh
testrac11:/INST #

 

 

6) 020-start-rcoracle.sh - start rcoracle (to set up system variables, because nothing is installed yet).
On both nodes, run script 020:

File 020-start-rcoracle.sh:

#!/bin/bash
#
#       Run rcoracle to set up kernel parameters
#
. CONFIG.sh
rcoracle start


Result:

testrac11:/INST # sh 020*sh

#############################################################################
#                 Begin of   O R A C L E   startup section                  #
#############################################################################


ORACLE_HOME directory /opt/oracle/product/10.1.0/db_1 does not exist!
Unsetting ORACLE_HOME, will try to determine it from system...
ORACLE_HOME environment variable not set.
Check /etc/profile.d/oracle.sh and /etc/oratab
Cannot find ORACLE_HOME directory .
Environment settings are wrong! Check /etc/profile.d/oracle.sh
 SETTINGS start from /etc/sysconfig/oracle
 - Set Kernel Parameters for Oracle:   yes
 - Start Oracle OCFS:                  yes
 - Start Oracle CRS:                   yes
 - Start Oracle EM:                    yes
Can't find needed file: emctl - Setting START_ORACLE_DB_EM = no
Can't find needed file: init.crs - Setting START_ORACLE_DB_CRS = no
Can't find needed file: /sbin/load_ocfs - Setting START_ORACLE_DB_OCFS = no


Setting kernel parameters for Oracle, see file
/etc/sysconfig/oracle for explanations.

Shared memory:      SHMMAX=3294967296  SHMMNI=4096  SHMALL=2097152
Semaphore values:   SEMMSL=1250  SEMMNS=32000  SEMOPM=100  SEMMNI=256
Other values:       FILE_MAX_KERNEL=131072  IP_LOCAL_PORT_RANGE=1024 65000

ULIMIT values:      MAX_CORE_FILE_SIZE_SHELL=unlimited
                    FILE_MAX_SHELL=65536  PROCESSES_MAX_SHELL=16384

Kernel parameters set for Oracle:                               done


  - Starting Oracle Cluster Filesystem...                       skipped
  - Starting Oracle CRS...                                      skipped

  - Starting Oracle EM dbconsole...                             skipped

#############################################################################
#                      End of   O R A C L E   section                       #
#############################################################################

testrac11:/INST #

7) 030-check-oracle-user.sh - this script verify Oracle user and location of Oracle installation disk.

Run it on all nodes.

File 030-check-oracle-user.sh:

#!/bin/bash
#       Check oracle user setting
#
. CONFIG.sh
if [ ! -x $ORA10CD1/runInstaller ]
then
        echo -e "\33[31m *** Wrong oracle disk location - no file $ORA10CD1/runInstaller  found \33[30m"
        exit 1
fi
echo "*** CHECK ORACLE VARIABLES *** "
sux - -c "env" oracle | grep ORACLE

Results:

testrac11:/INST # sh 030*sh
*** CHECK ORACLE VARIABLES ***
xauth:  creating new authority file /opt/oracle/.Xauthority
ORACLE_SID=test1
ORACLE_BASE=/opt/oracle
ORACLE_HOME=/opt/oracle/product/10.1.0/db_1


ORACLE_SID must be different on different nodes.

8) 040-create-mounts.sh - creates mount points and set up fstab for NFS mount, and mounts disks.

You can do it manually, using YaST2 (be careful to reproduce the same attributes at in our example) or run script (on all nodes):

File 040-create-mounts.sh:

#!/bin/sh
#
# These script creates NFS mount points. It is very likely that you will
# edit these script, or skip it and do all steps manually
#
. CONFIG.sh
#
# Recreate symlinks
rm -f /volcrs /voldb /vollog
mkdir -p /ora/voldb /ora/vollog /ora/volcrs
ln -s /ora/* /

#
grep voldb /etc/fstab || echo "$VOLDB  /ora/voldb nfs tcp,rsize=32768,wsize=3268,bg,intr,noac 0 0" >> /etc/fstab
grep volcrs /etc/fstab || echo "$VOLCRS /ora/volcrs nfs tcp,rsize=32768,wsize=3268,bg,intr,noac 0 0" >> /etc/fstab
grep vollog /etc/fstab || echo "$VOLLOG /ora/vollog nfs tcp,rsize=32768,wsize=3268,bg,intr,noac 0 0" >> /etc/fstab
#
# Some mistery - but if I mount from current directory, I see wrong names in 'df'
# So, cd to . first.
#
(cd / && mount -a)
#
mount | grep voldb  || echo "*** $VOLDB is not mounted, edit fstab manually ***"
mount | grep vollog || echo "*** $VOLDB is not mounted, edit fstab manually ***"
mount | grep volcrs || echo "*** $VOLDB is not mounted, edit fstab manually ***"

 

Results:

testrac12:/INST # sh 040*sh
fas-1a-1:/vol/vol0/NFS01  /ora/voldb nfs tcp,rsize=32768,wsize=3268,bg,intr,noac 0 0
fas-1a-1:/vol/vol0/NFS01 /ora/volcrs nfs tcp,rsize=32768,wsize=3268,bg,intr,noac 0 0
fas-1b-1:/vol/vol0/NFS03 /ora/vollog nfs tcp,rsize=32768,wsize=3268,bg,intr,noac 0 0
fas-1a-1:/vol/vol0/NFS01 on /ora/voldb type nfs (rw,tcp,rsize=32768,wsize=3268,bg,intr,noac,addr=10.254.32.105)
fas-1b-1:/vol/vol0/NFS03 on /ora/vollog type nfs (rw,tcp,rsize=32768,wsize=3268,bg,intr,noac,addr=10.254.32.106)
fas-1a-1:/vol/vol0/NFS01 on /ora/volcrs type nfs (rw,tcp,rsize=32768,wsize=3268,bg,intr,noac,addr=10.254.32.105)

IMPORTANT: you must set up noac attribute for these volumes (it is very  important), and for production system, mount in hard mode (default) so that system will never stop IO attempts. You can use intr mode, it allows to kill Oracle related processes without blind system reboot, if necessary.[4]

9) 060-setup-xntp.sh - set up XNTP service (important to synchronize clock on all nodes).

You can use YaST2, or (if xntp is installed) our script:

File 060-setup-xntp.sh:

#
#       090) Copy files.
#       Now we use this script only to start xntpd
#
. CONFIG.sh
if [ ! -f /etc/ntp.conf ]
then
  echo "server $XNTP_IP" > /etc/ntp.conf

  chkconfig xntpd on
  /etc/init.d/xntpd start
fi
echo Please be sure that you configured xntp daemon. Use YaST2.
/etc/init.d/xntpd status

Just setting XNTP server. It is important to have time synchronized on all RAC nodes.

10) 070-edit-services.sh - script removes conflicting records from /etc/services

(I am not sure if it is still necessary. Anyway, we will run it. Be careful, if you use NIS or LDAP, update them after running script and verify, that they do not keep removed records.).
Run on all nodes.

File 070-edit-services.sh:

#
. CONFIG.sh
sed '/net8-cman/d' /etc/services > /etc/services- && cp /etc/services- /etc/services && rm /etc/services-
ls -l /etc/services

 

Script removes net8-cman records from /etc/services.


Results:

        testrac11:/INST # sh 070*sh
        -rw-r--r--  1 root root 596317 Feb 28 22:17 /etc/services

 

11) 080-ssh-genkeys.sh - set up password-less acces between nodes.

Now we must set up password – less access for user oracle between all nodes. I use script below. After all, running ‘ssh node-name hostname’ should return just hostname without any additional output.

 

Attention - this script is not designed for high security (it do not use passphrase and ssh-agent); do not use it if you expose your Oracle system to public internet (it is extremely bad idea anyway).

File 080-ssh-genkeys.sh:

#
#       Generate ssh rsa and dsa keys for password-less access
#       You may want to remove this after you wil complete oracle installation
#       Or use ssh-agent and passphrase for better security
#
#
. CONFIG.sh
#
# Check DNS or HOSTS first
#
for i in $NODES $NODESP
do
    echo testing $i
    if ! ping -q -c 1 $i
    then
        echo "*** BAD NODE NAME $i, fix DNS, hosts or NODES list"
        exit 1
    fi
done
su - oracle -c "ssh-keygen -t rsa -b 2048"
su - oracle -c "ssh-keygen -t dsa -b 2048"
#
#
for i in $NODES
do
echo Copying keys to node $i
cat ~oracle/.ssh/*pub | ssh $i 'su - oracle -c "mkdir -p .ssh;touch .ssh/authorized_keys; cat >> .ssh/authorized_keys"'
su - oracle -c "ssh $i date"
echo done
done

Run it on both nodes, enter EMPTY passphrase (2 times), enter root password (2 times), and answer 'yes' on requests.
Example (I marked answers as <answer>:

testrac11:/INST # sh 080*sh
testing testrac11
PING testrac11 (10.23.32.111) 56(84) bytes of data.

--- testrac11 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.047/0.047/0.047/0.000 ms
testing testrac12
PING testrac12 (10.23.32.112) 56(84) bytes of data.

--- testrac12 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.311/0.311/0.311/0.000 ms
testing testrac11-1
PING testrac11-1 (10.254.32.111) 56(84) bytes of data.

--- testrac11-1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.040/0.040/0.040/0.000 ms
testing testrac12-1
PING testrac12-1 (10.254.32.112) 56(84) bytes of data.

--- testrac12-1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.208/1.208/1.208/0.000 ms
Generating public/private rsa key pair.
Enter file in which to save the key (/opt/oracle/.ssh/id_rsa):      <ENTER>
Created directory '/opt/oracle/.ssh'.
Enter passphrase (empty for no passphrase):                           <ENTER>
Enter same passphrase again:                                               <ENTER>
Your identification has been saved in /opt/oracle/.ssh/id_rsa.
Your public key has been saved in /opt/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
ad:db:92:f3:40:26:d6:d5:6d:95:89:b8:03:64:b8:7d oracle@testrac11
Generating public/private dsa key pair.
Enter file in which to save the key (/opt/oracle/.ssh/id_dsa):      <ENTER>
Enter passphrase (empty for no passphrase):                           <ENTER>
Enter same passphrase again:                                               <ENTER>
Your identification has been saved in /opt/oracle/.ssh/id_dsa.
Your public key has been saved in /opt/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
70:87:ea:cd:1c:12:c6:f1:fd:5e:49:ab:9d:fe:a8:5d oracle@testrac11
Copying keys to node testrac11
Password:                                                                        <ROOT PASSWORD>
The authenticity of host 'testrac11 (10.23.32.111)' can't be established.
RSA key fingerprint is fe:87:5c:2c:6f:fd:d9:0b:e5:19:dd:f3:f9:72:64:7e.
Are you sure you want to continue connecting (yes/no)? yes     <yes>
Warning: Permanently added 'testrac11,10.23.32.111' (RSA) to the list of known hosts.
Mon Feb 28 22:21:28 PST 2005
done
Copying keys to node testrac12
Password:                                                                        <ROOT PASSWORD>
The authenticity of host 'testrac12 (10.23.32.112)' can't be established.
RSA key fingerprint is e2:8b:a8:a7:13:43:4b:66:b4:b5:9f:26:49:e7:06:d4.
Are you sure you want to continue connecting (yes/no)? yes     <yes>
Warning: Permanently added 'testrac12,10.23.32.112' (RSA) to the list of known hosts.
Mon Feb 28 22:21:35 PST 2005
done
testrac11:/INST #

12) 090-verify-ssh-access.sh - verify results (password-less access between nodes).

Run these script on all nodes: sh 090-verify-ssh-access.sh. It is IMPORTANT script - it sets up identity on the very first call of ssh, so do not skip it.

File 090-verify-ssh-access.sh:

#
# Now, verify oracle password-access between nodes
#
#
#
. CONFIG.sh
#
for i in $NODES $NODESP
do
    echo -e testing $i ", \033[31m Answer yes very first time, please \033[m"
    su - oracle -c "ssh -n $i echo TEST"
    lines=`su - oracle -c "ssh -n $i echo TEST" | wc -l`
    if [ x$lines != x1 ]
    then
        echo ATTENTION. Something wrong with oracle 'ssh $i' , check it manually
    else
        echo $i OK
    fi
done

Results:

testrac11:/INST # sh 090*sh
testing testrac11 ,  Answer yes very first time, please
TEST
testrac11 OK
testing testrac12 ,  Answer yes very first time, please
TEST
testrac12 OK
testing testrac11-1 ,  Answer yes very first time, please
The authenticity of host 'testrac11-1 (10.254.32.111)' can't be established.
RSA key fingerprint is fe:87:5c:2c:6f:fd:d9:0b:e5:19:dd:f3:f9:72:64:7e.
Are you sure you want to continue connecting (yes/no)? yes                                     <yes>
Warning: Permanently added 'testrac11-1,10.254.32.111' (RSA) to the list of known hosts.
TEST
testrac11-1 OK
testing testrac12-1 ,  Answer yes very first time, please
The authenticity of host 'testrac12-1 (10.254.32.112)' can't be established.
RSA key fingerprint is e2:8b:a8:a7:13:43:4b:66:b4:b5:9f:26:49:e7:06:d4.
Are you sure you want to continue connecting (yes/no)? yes                                     <yes>
Warning: Permanently added 'testrac12-1,10.254.32.112' (RSA) to the list of known hosts.
TEST
testrac12-1 OK

Now, I recommend to verify hostnames very last time. If you used /etc/hosts, look into these files:

Example:

testrac11:/INST # cat /etc/hosts
10.23.32.111    testrac11

10.254.32.111   testrac11-1
10.23.32.112    testrac12
10.254.32.112   testrac12-1

13) 200-InstCRS.1sh - installing Cluster Ready Services. It is first real INSTALLATION step.

I use this script for CRS installation. Remember - when script asks you to run root.sh files, you better use our scripts (210* and 220*) instead - they fix some access permissions problems, which exists in Oracle installer.

Here it is. Run it on node-1 only.

File 200-InstCRS.1sh:

#
#       200) Install Oracle cluster manager
#       Enter /u02/OCRFile and /u02/VCSFile as shared files
  . CONFIG.sh
  . /etc/profile.d/oracle.sh
  # IMPORTANT. LD_ASSUME_KERNEL must be 2.4.5 here
  if [ "$LD_ASSUME_KERNEL" = "" ]
  then
     echo -e "\033[31mATTENTION. LD_ASSUME_KERNEL variable absent, set it up first\033[m"
     exit 1
  fi
  # do not work anyway # . /etc/profile.d/oracle.sh
  D=$DIR_CRS
  CRS=$CRS10CD1
  mkdir $D   > /dev/null 2>& 1
  if [ ! -d $D ]
  then
      echo "Can not create $D for CRS files"
      echo "Check $DIR_CRS variable"
      exit 1
  fi
  chown oracle:dba $D
  if [ ! -x $CRS/runInstaller ]
  then
      echo "Can not find CRS installation disk: $CRS10CD1"
      echo "verify variable CRS10CD1 in CONFIG.sh and location of CRS installation files"
      exit 1
  fi


  sux - -c "mkdir -p $ORACLE_HOME;ORACLE_HOME=$ORACLE_BASE/product/10.1.0/crs_1;mkdir -p $ORACLE_HOME;export ORACLE_HOME; echo $ORACLE+HOME;cd $CRS; ./runInstaller" oracle
sleep 5
  echo -e "\033[32mWait for runInstaller.
  ================================

   Registry file - $D/OCRFile
   Voting file   - $D/CSSFile

  =================================
  \033[m
  "

Script:

-         Change user to oracle;

-         Change ORACLE_HOME to CRS home (should be different from Database home);

-         Creates directory for shared CRS files and change it’s owner to ORACLE (by some reason, Oracle do not like when this directory is owned by root, at last I had such cases in a few installations).

-         Start Oracle installer for CRS and reminds user to expected names for CRS control files.

 

Oracle installer works in background so scripts returns short before you will see Oracle Universal Installer window.

 

Output:

testrac11:/INST # sh 200*sh
+HOME
Starting Oracle Universal Installer...

Checking installer requirements...

Checking operating system version: must be UnitedLinux-1.0, redhat-3 or SuSE-9
                                      Passed


All installer requirements met.

Checking Temp space: must be greater than 80 MB.   Actual 12318 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 4102MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2005-02-28_10-35-12PM. Please wait ...Oracle Universal Installer, Version 10.1.0.3.0 Production
Copyright (C) 1999, 2004, Oracle. All rights reserved.

Wait for runInstaller.
  ================================

   Registry file - /volcrs/CRS/OCRFile
   Voting file   - /volcrs/CRS/CSSFile

  =================================


testrac11:/INST #

Now you must install CRS. In Oracle Universal Installer:


- Verify that installation target is  /opt/oracle/product/10.1.0/crs_1;
- enter node names - testrac11 and testrac12 public, testrac11-1 and testrac12-1 private (in my example);
- specify network interface usage (eth0 public, eth1 private);
- specify Oracle Cluster Registry: /volcrs/CRS/OCRFile
- specify Voting Disk: /volcrs/CRS/CSSFile
- check summary and run install.


-
When you are prompted to run root file in inventory, run it or run script 210*.sh (you are prompted it on the very very first time only).
- When you will be prompted to run on all nodes root.sh script
   /opt/oracle/product/10.1.0/crs_1/root.sh



Run (first on node1, then when prompted - on node2) script 220 – sh 220*.sh

File 220-runRootsh.sh:

#
# Run this on every node
#
. CONFIG.sh
#
mkdir -p /opt/oracle/product/10.1.0/crs_1/log && chown oracle:dba /opt/oracle/product/10.1.0/crs_1/log
sh /opt/oracle/product/10.1.0/crs_1/root.sh
# HACK. For some mistical reason, script can not do it itself
chown oracle:dba $DIR_CRS/*File
/opt/oracle/product/10.1.0/crs_1/bin/olsnodes -n
/opt/oracle/product/10.1.0/crs_1/bin/crs_stat
echo 'export PATH=$PATH:/opt/oracle/product/10.1.0/crs_1/bin' >> ~root/.bashrc

Script:

-         Creates forgotten log directory (sometimes installer forgot to create it);

-         Run script;

-         Set up owner for shard files (sometimes root.sh set up wrong  owners);

-         Add CRS binaries into root profile.


Here is output:

testrac11:/INST # sh 220*sh
Running Oracle10 root.sh script...
\nThe following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/oracle/product/10.1.0/crs_1
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Checking to see if Oracle CRS stack is already up...
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/opt/oracle/product/10.1.0' is not owned by root
WARNING: directory '/opt/oracle/product' is not owned by root
WARNING: directory '/opt/oracle' is not owned by root
assigning default hostname testrac11 for node 1.
assigning default hostname testrac12 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: testrac11 testrac11-1 testrac11
node 2: testrac12 testrac12-1 testrac12
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Now formatting voting device: /volcrs/CRS/CSSFile
Read -1 bytes of 512 at offset 872603648 in voting device (CSSFile)
Successful in setting block0 for voting disk.
Format complete.
Adding daemons to inittab
Preparing Oracle Cluster Ready Services (CRS):
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
        testrac11
CSS is inactive on these nodes.
        testrac12
Local node checking complete.
Run root.sh on remaining nodes to start CRS daemons.
testrac11       1
testrac12       2


And on other node:

testrac12:/INST # sh 220*sh
Running Oracle10 root.sh script...
\nThe following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/oracle/product/10.1.0/crs_1
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Checking to see if Oracle CRS stack is already up...
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/opt/oracle/product/10.1.0' is not owned by root
WARNING: directory '/opt/oracle/product' is not owned by root
WARNING: directory '/opt/oracle' is not owned by root
clscfg: EXISTING configuration version 2 detected.
clscfg: version 2 is 10G Release 1.
assigning default hostname testrac11 for node 1.
assigning default hostname testrac12 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: testrac11 testrac11-1 testrac11
node 2: testrac12 testrac12-1 testrac12
clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Adding daemons to inittab
Preparing Oracle Cluster Ready Services (CRS):
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
        testrac11
        testrac12
CSS is active on all nodes.
Oracle CRS stack installed and running under init(1M)
testrac11       1
testrac12       2
CRS-0202: No resources are registered.

 

This is very important step (you can repeat in  future, if you lost your CRS shared files), it creats shared files, configure CRS and add it into /etc/inittab so that system now starts CRS automatically. You can use /etc/init.d/init.crs script to start / stop CRS, when it all is completed (rcoracle do the same).

14) 230-check-crs.sh - verify crs.
Run these script on all nodes.

File 230-check-crs.sh:

#
#       230)    Now, check if crs works
#
. ~oracle/.profile
. ~/.bashrc
rm /var/log/crs
ln -s ~oracle/product/*/crs_1/crs/log/ /var/log/crs

crsctl check install
tail /var/log/crs/*

 

Script:

-         Creates shortcut to CRS logs - /var/log/crs, so that you can always watch CRS logs by simple typing tail –f /var/log/crs;

-         Show you recent CRS logs.

 

Results:

testrac11:/INST # sh 230*sh
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
        testrac11
        testrac12
CSS is active on all nodes.
2005-02-28 22:57:01.801: Initializing EVMMgr
2005-02-28 22:57:01.984: CRSD locked during state recovery, please wait.
2005-02-28 22:57:01.987: CRSD recovered, unlocked.
2005-02-28 22:57:01.989: QS socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=ora_crsqs))
2005-02-28 22:57:01.990: UI socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=testrac11_crs_caa))
2005-02-28 22:57:01.993: E2E socket on: (ADDRESS=(PROTOCOL=tcp)(HOST=testrac11-1)(PORT=49896))
2005-02-28 22:57:01.994: Starting Threads
2005-02-28 22:57:01.994: CRS Daemon Started.
2005-02-28 22:55:16.660: CRSD-1: [CMDMAIN:2540914048] Restart waiting for Oracle CRSD to start
2005-02-28 22:57:02.082: CRSD-1: Complete Restart Application Request

 

Data base installation.

Now you have Cluster Ready Services installed and configured.

It is time to install Oracle Database.

15) 300-check-env.sh- checks environment for Oracle database.

File 300-check-env.sh:

#
#       300) check oracle env
#
echo " "
echo " Please, run me on all nodes and be sure that ORACLE_BASE is correct and ORACLE_SID is different\33[m"
echo " "
su - -c "set | grep ORACLE" oracle

Script is simple, no comments.

Results:

testrac11:/INST # sh 300*sh

 Please, run me on all nodes and be sure that ORACLE_BASE is correct and ORACLE_SID is different\33[m

ORACLE_BASE=/opt/oracle
ORACLE_HOME=/opt/oracle/product/10.1.0/db_1
ORACLE_OWNER=oracle
ORACLE_SID=test1
SET_ORACLE_KERNEL_PARAMETERS=yes
START_ORACLE_DB_CRS=yes
START_ORACLE_DB_EM=yes
START_ORACLE_DB_OCFS=yes

and

testrac12:/INST # sh 300*sh

 Please, run me on all nodes and be sure that ORACLE_BASE is correct and ORACLE_SID is different\33[m

ORACLE_BASE=/opt/oracle
ORACLE_HOME=/opt/oracle/product/10.1.0/db_1
ORACLE_OWNER=oracle
ORACLE_SID=test2
SET_ORACLE_KERNEL_PARAMETERS=yes
START_ORACLE_DB_CRS=yes
START_ORACLE_DB_EM=yes
START_ORACLE_DB_OCFS=yes

 

16) 310-InstOracle.1sh - starts Oracle Installer for the main database.

Run this script on node1 only.

File 310-InstOracle.1sh:

#
#       310) Install Oracle cluster manager
#       Enter /u02/OCRFile and /u02/VCSFile as shared files
. CONFIG.sh
D=$VOLDB
DB=$ORA10CD1
. /etc/profile.d/oracle.sh
. ~oracle/.profile


  sux - -c "cd $DB; ./runInstaller" oracle
  wait
  sleep 4
  echo -e "\033[32m  "
  echo "Install ORACLE database Enterprise Edition"
  echo "Do not create database, we wil use dbca for this purpose"
  echo "Run 320-run-root-in-X-env.sh file when prompted. Be sure that you have X allowed"
  echo "from ALL Nodes"
  echo -e "\033[m"

It starts Oracle Universal Installer for Oracle Database. (As usual, window will come AFTER script ends, because installer is always started in background):

testrac11:/INST # sh 310*sh
Starting Oracle Universal Installer...

Checking installer requirements...

Checking operating system version: must be redhat-3, SuSE-9, SuSE-8 or UnitedLinux-1.0
                                      Passed


All installer requirements met.

Checking Temp space: must be greater than 80 MB.   Actual 11786 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 4102MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2005-02-28_11-14-26PM. Please wait ...Oracle Universal Installer, Version 10.1.0.3.0 Production
Copyright (C) 1999, 2004, Oracle. All rights reserved.


Install ORACLE database Enterprise Edition
Do not create database, we wil use dbca for this purpose
Run 320-run-root-in-X-env.sh file when prompted. Be sure that you have X allowed
from ALL Nodes


Select in Oracle Installer:


- select ALL nodes;
- select Enterprise Edition;
- select DO NOT CREATE DATABASE;
- do not change group (disk);
- click INSTALL;
- when you are prompted to run root.sh script, run it (better using 320*sh script from our installation).
ATTENTION. You must have:
  - IP addresses for virtual interfaces, 1 for each node;
  - root login on remote node with X11 allowed (I use 'slogin -X -l root remote-node for such access);
and now you should configure virtual interfaces.
I enter:
  testrac11-vip.sjclab.mydomain.com
  testrac12-vip.sjclab.mydomain.com
(IP have been configured in DNS before installation).

Now you have database server installed.

17) 340-run-netca.1sh - run Network Configuration Assistant (on single node).

It starts network configuration. Configure listener and exit. Run it on one node only.

File 340-run-netca.1sh:

#!/bin/bash

#

#              340) Run netca to create listener on all nodes

#

. /etc/profile.d/oracle.sh

#

#

sux - -c "netca" oracle &

#

 

testrac11:/INST # sh 340*sh
testrac11:/INST # Oracle Net Services Configuration:
Configuring Listener:LISTENER
testrac11...
testrac12...
Listener configuration complete.
Oracle Net Services configuration successful. The exit code is 0


You must configure virtual IP (if did not do it before) and listener.

18) 400-create-database.sh - starts dbca and creates database.

File 400-create-database.sh:

#!/bin/bash

#

#              400) Create database. See guide below.

#

. CONFIG.sh

. /etc/profile.d/oracle.sh

#

echo -e "

*******************************

You are about to create a database

\033[31m I M P O R T A N T : you must change option

\033[1m filesystemio_options=directIO \033[m

during creating database.

*******************************

"

echo -n "Continue__?"

read x

sux - -c "dbca -datafileDestination $DIR_DB" oracle &

#

wait

 


Put database into /voldb/DB directory. Use name 'test'.

testrac11:/INST # sh 400*sh

 

*******************************

You are about to create a database

I M P O R T A N T : you must change option

filesystemio_options=directIO

during creating database.

*******************************

 

Continue__?

 

It opens DBCA window:




Next:

Select all nodes

 

Next:

Select database type (we use default).

 

and enter database name – test



Use defaults for notifications etc, until these window – select Cluster File System:


Specify database location - /voldb/DB:

 

Do not set up Flash Recovery data here. Accept defaults for few more screens, until you arrive to the database services. Add servicename ‘oratest  and set up service name ‘test’ here, select ‘Basic’ TAF policy.

 

Continue until initialisation parameters. Now OPEN <ALL INITIALIZATION PARAMETERS>:

 

On all parameters screen, click on SHOW ADVANCED PARAMETERS.

These is VERY IMPORTANT step – you must configure directIO filesystemio_options:

 

Now you can continue as usual, but I change location for all redo files and one control files (using /vollog/LOGS directory) as it is recommended on NetApp site:

(User ORACLE must have write access to all these directories.)

 

 

Database is created and ready. You can watch CRS activity by looking into CRS log file:

 

tail –f /var/log/crs/*

 

(We already created symlink here).  You will see output like this:

 

2005-03-02 21:51:36.331: Attempting to start `ora.testrac11.ons` on member `test

rac11`

2005-03-02 21:51:36.652: Start of `ora.testrac11.ons` on member `testrac11` succ

eeded.

2005-03-02 21:41:37.350: CRSD-1: [CMDMAIN:2540914048] Restart waiting for Oracle

 CRSD to start

2005-03-02 21:51:36.658: CRSD-1: Complete Restart Application Request

2005-03-02 21:54:12.452: Attempting to start `ora.test.test1.inst` on member `te

strac11`

2005-03-02 21:54:14.492: Start of `ora.test.test1.inst` on member `testrac11` su

cceeded.

2005-03-02 21:54:18.411: Attempting to start `ora.test.orcltest.test1.srv` on me

mber `testrac11`

2005-03-02 21:54:18.737: Start of `ora.test.orcltest.test1.srv` on member `testr

ac11` succeeded.

2005-03-02 21:54:18.794: CRS-1002: Resource ora.test.orcltest.cs is already runn

ing on member testrac12

 

Starting, stopping and managing.[5]

Clustered database is started and stopped by Cluster Ready Services (CRS) manager, so our start / stop script only starts and stops CRS processes. Moreover, Cluster Ready Services and 2 other components are installed by Oracle installation into /etc/inittab, and are automatically started by system init when it completes runlevel3 and runlevel5 initialisation:

h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null

h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null

h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null

 

-         evmd is process responsible for event exchange between cluster members;

-         cssd is cache synchronisation services daemon;

-         crsd is cluster ready services daemon, responsible for starting / stopping everything.

 

If init cannot start any of these daemons few times in a line, it suspends future attempts for 5 minutes. System script, /etc/init.d/init.crs, control flags, allowing these 3 processes to start, and sends a signal to running scripts when they are requested to stop. As a result, when you stopped CRS (or when it dies by any reason) and want to restart it, after restart is allowed, you must kill –1 1 to speed-up process respawning.

 

 I recommend to link script, provided by Oracle, with root command rccrs:

 

            ln –s /etc/init.d/init.crs /sbin/rccrs

 

Do not run other 3 (/etc/init.d/init.*) scripts provided by Oracle – they are started / stopped by init and controlled with init.crs script.

 

We already linked log directory for crs as /var/log/crs, so we can watch crs logs as

 

            tail –f /var/log/crs/*

 

You must know at least srvctl oracle command for very initial RAC management. For example, run:

testrac11:~ # srvctl status database -d test

Instance test1 is running on node testrac11

Instance test2 is running on node testrac12

Very useful tool, web based DB console, is started on port 5500 (depending of the database number – port change with every next database), so you can open

http://testrac11-vip.sjclab.mydomain.com:5500/em

login as system / NORMAL

and see EM manager web page:

 

 

This is pretty useful management tool. It is not started by crs system, so you should run ‘emctl start dbconsole’ somewhere in your scripts (after CRS started database) on every system reboot (or start it manually every time when you need it).

 

You can use our (modifying) rcoracle (/etc/init.d/oracle) script to start, stop and status-request database:

rcoracle stop – stops CRS and all cluster database instances;

rcoracle start – starts CRS (which will start instances).

rcoracle status – shows status;

rcoracle kill – dumb brute force killing all oracle processes including any oracle user processes. Used as a last resort chance to release busy resource (NFS disk for example).

 

References:

  1. http://www.novell.com/products/linuxenterpriseserver/oracle/documents.html - Oracle installation guides from SUSE (extremely useful set of documents);
  2. http://www.netapp.com/tech_library/3339.html - Oracle 10g RAC installation with RedHat Linux and Network Appliance filer (from NetApp). This document has one little bug – you should never use noac option for ORACLE_HOME volume.
  3. http://www.oracle-base.com/index.php - numerous articles about Oracle 10g;
  4. www.oracle.com - Oracle main web site (with links to Oracle Technology network, Metalink and other useful resources).
  5. http://oss.oracle.com  – Oracle and Linux project site;
  6. http://lists.suse.com/archive/suse-oracle/ - SUSE and Oracle mailing list.

APPENDIX 1. Modified orarun scripts.

These files are slightly modified components of ‘orarun’ package, which is part of SUSE Linux Enterprise Server packages. I hope to eliminate their modification with future orarun package. I print them here only to allow full understanding of this installation process.

File /etc/profile.d/oracle.sh (FILE.d/oracle.sh):

#

# These is special version of oracle.sh file, modified for Oracle 10g RAC installation on SLES9

# Do not use it in other cases.

#

# Login environment variable settings for Oracle

# The code below is done ONLY if the user is "oracle":

# Set the ULIMITs for the shell and add gcc_old to PATH if it's installed

#

 

  # Get settings, if file(s) exist(s). If not, we simply use defaults.

  if test -f /etc/sysconfig/oracle; then

    # new location as of SL 8.0 is directory /etc/sysconfig/

    . /etc/sysconfig/oracle

  else

    if test -f /etc/rc.config.d/oracle.rc.config; then

      # location is directory /etc/rc.config.d/

      . /etc/rc.config.d/oracle.rc.config

    else

      if test -f /etc/rc.config; then

      # old SuSE location was to have everything in one file

      . /etc/rc.config

      fi

    fi

  fi

 

 

  ORACLE_SID=test1

 

  ORACLE_BASE=/opt/oracle

  ORACLE_HOME=$ORACLE_BASE/product/10.1.0/db_1

 

  TNS_ADMIN=$ORACLE_HOME/network/admin

  ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data

  PATH=$PATH:$ORACLE_HOME/bin

  LD_LIBRARY_PATH=${LD_LIBRARY_PATH:+$LD_LIBRARY_PATH:}$ORACLE_HOME/lib:$ORACLE_HOME/ctx/lib

  CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib

 

  export ORACLE_BASE ORACLE_HOME ORA_NLS33 ORACLE_SID PATH LD_LIBRARY_PATH CLASSPATH TNS_ADMIN

 

  # ORACLE_TERM=xterm; export ORACLE_TERM

  # NLS_LANG=AMERICAN_AMERICA.UTF8; export NLS_LANG

 

  #

  # This requires the limits to have been increased by root

  # e.g. at boot time by the /etc/rc.d/oracle script, both

  # ulimit and kernel parameters.

  #

 

  # Add package "gcc_old" gcc 2.95.3 to path - FIRST

  #if test -x /opt/gcc295/bin/gcc; then

  #  export PATH=/opt/gcc295/bin:$PATH

  #fi

 

  # RAC (cluster) component GSD commands don't run when this is

  # set - which it is if a SuSE Java package is installed.

  unset JAVA_BINDIR JAVA_HOME

 

  # Set ulimits:

  #

  # We suppress any warning messages, so if the hard limits have not been

  # increased by root and the commands don't work we keep silent...

  # This is because the only one who needs it is the shell that starts

  # the DB server processes, and the number of warning messages created

  # here is potentially way too much and confusing

 

  # core dump file size

  ulimit -c ${MAX_CORE_FILE_SIZE_SHELL:-0} 2>/dev/null

 

  # max number of processes for user

  ulimit -u ${PROCESSES_MAX_SHELL:-16384} 2>/dev/null

 

  # max number of open files for user

  ulimit -n ${FILE_MAX_SHELL:-65536} 2>/dev/null

 

  export LD_ASSUME_KERNEL=2.4.21

 
 
 
 
File /etc/sysconfig/oracle (FILE.d/sysconfig.oracle):

#

# Note: the environment variables are set in /etc/profile.d/oracle.[c]sh

# Owner of Oracle installation; Oracle will be started as that user

#

# This is file for ORACLE 10 G CRS system

# Use original file (from orarun) for Oracle9 and Oracle8

#

ORACLE_OWNER="oracle"

 

#

# Start Oracle Cluster Filesystem (for RAC)

#

START_ORACLE_DB_OCFS="yes"

 

#

# Start Cluster Ready Services (for RAC10g; notice - they are started by inittab, so this line do not influence original start)

# Setting NO do not turn it off, but prevents this script from stopping and restarting CRS

#

START_ORACLE_DB_CRS="yes"

 

#

# Start Enterprise manager for Oracle10

#

START_ORACLE_DB_EM="yes"

 

 

#

# Set the KERNEL PARAMETERS for Oracle. Requires a 2.4 kernel, in 2.2

# kernels only SHMMAX can be set during runtime via /proc.

#

# DO NOT CHANGE ANY VALUES unless you KNOW what you are doing and why!!!

#

# Have a look at the Oracle ReleaseNotes for the Oracle product you are

# using for how to set these values. If you do not set them we will

# assume some reasonable defaults for a medium Oracle 9i database

# system (that's a pretty big and busy one!).

#

# The /proc filesystem provides access to kernel parameters and statistics

# and the /proc/sys/ system allows one to change some kernel settings

# during runtime.

# If you have the kernel sources installed (package kernel-source)

# You can find more information here:

#   /usr/src/linux/Documentation/sysctl/ (directory)

#   /usr/src/linux/Documentation/filesystems/proc.txt

#   /usr/src/linux/Documentation/networking/ip-sysctl.txt

#

SET_ORACLE_KERNEL_PARAMETERS="yes"

 

#

# Max. shared memory segment size that can be allocated.

# The ONLY parameter you should touch at all (and only if you have more than

# 4 GB of physical memory) is SHMMAX. It does not make any sense to change

# the others for Oracle!

# Kernel sources header file: /usr/src/linux/include/linux/shm.h

# Recommended: SHMMAX = 0.5*(physical memory). Higher # values are okay,

# since this parameter only sets the OS maximum:

# This setting does not affect how much shared memory is needed or used by

# Oracle8i or the operating system. It is used only to indicate the maximum

# allowable size. This setting also does not impact operating system

# kernel resources.

# The values for SHMSEG and SHMMIN cannot be changed via the proc-interface,

# but there is no need to change anything anyway!

# SHMSEG (default: 4096): max. number of shared segments per process

# SHMMIN (default: 1): min. size of a shared mem. segment in bytes

#

# SHMMAX max. size of a shared memory segment in bytes

#

SHMMAX=3294967296

#

# SHMMNI (default: 4096): max. number of shared segments system wide

# No change is needed for running Oracle!

#

SHMMNI=4096

#

# SHMALL (default: 8G [2097152]): max. shm system wide (pages)

# No change is needed for running Oracle!

#

SHMALL=2097152

 

#

# Sempahore values

# Kernel sources header file: /usr/src/linux/include/linux/sem.h

#

# SEMVMX: semaphore maximum value. Oracle recommends a value of 32767,

# which is the default in SuSE *and* the maximum value possible.

# This value cannot be changed during runtime via the /proc interface,

# but there is no need to do so anyway!

#

# SEMMSL: max. number of semaphores per id. Set to 10 plus the largest

# PROCESSES parameter of any Oracle database on the system (see init.ora).

# Max. value possible is 8000.

#

SEMMSL=1250

#

# SEMMNS: max. number of semaphores system wide. Set to the sum of the

# PROCESSES parameter for each Oracle database, adding the largest one

# twice, then add an additional 10 for each database (see init.ora).

# Max. value possible is INT_MAX (largest INTEGER value on this

# architecture, on 32-bit systems: 2147483647).

#

SEMMNS=32000

#

# SEMOPM: max. number of operations per semop call. Oracle recommends

# a value of 100. Max. value possible is 1000.

#

SEMOPM=100

#

# SEMMNI: max. number of semaphore identifiers. Oracle recommends a

# a value of (at least) 100. Max. value possible is 32768 (defined

# in include/linux/ipc.h: IPCMNI)

#

SEMMNI=256

 

#

# Defines the local port range that is used by TCP and UDP to

# choose the local port. The first number is the first, the

# second the last local port number. Default value depends on

# amount of memory available on the system:

# > 128Mb 32768-61000

# < 128Mb 1024-4999 or even less.

# This number defines number of active connections, which this

# system can issue simultaneously to systems not supporting

# TCP extensions (timestamps). With tcp_tw_recycle enabled

# (i.e. by default) range 1024-4999 is enough to issue up to

# 2000 connections per second to systems supporting timestamps.

#

IP_LOCAL_PORT_RANGE="1024 65000"

 

#

# The *_SHELL settings are for the Oracle startup script (/etc/rc.d/oracle

# and 'rcoracle') *ONLY*, it does NOT have any influence on the

# limits if you login as user 'oracle' and start Oracle from there!!!

# This sets the limits for the number of open files and processes.

# FILE_MAX_SHELL *MUST* be lower than FILE_MAX_KERNEL, obviously

#

FILE_MAX_KERNEL=131072

FILE_MAX_SHELL=65536

PROCESSES_MAX_SHELL=16384

MAX_CORE_FILE_SIZE_SHELL=unlimited

 

#

# By Andrea Arcangeli, SuSE:

# This decreases the swappiness of the kernel. It will tend to swap less. It

# will shrink the pagecache more, before falling back into swap. So

# increasing the mapped ratio will result in less cache and less swap.

# On a lowmemory machine reducing the cache, and the swap can decrease

# performance.

# On a database machine with plenty of ram, swapping some hundred mbyte

# instead may not be necessary, better to shrink the cache, in particular

# because having that much shm allocated tends to fool the VM. The VM

# can't know if the shm is fs cache too (the shm in Oracle is mostly cache

# for the filesystem).

# So going to 1000 is probably a good idea for high end servers with

# plenty of memory. Using "1000" make sense where you really know swapping

# is going to be not necessary during all the important workloads because

# you tune the machine in a way that it has enough ram to succeed w/o the

# need of swap. Using 1000 tells the VM to swap less.

#

VM_MAPPED_RATIO=1000

 

#

# Max. size of an async I/O request

#

AIO_MAX_SIZE=262144

 

 

 

File /etc/init.d/oracle (FILES.d/init.oracle):

#! /bin/sh

# Copyright (c)1995 SuSE GmbH Nuernberg, Germany.

# MODIFIED for Oracle RAC (it is not original script and is not supported) by A. Roudnev, 2005, for this guide only

#

# Author: SuSE Oracle Team <feedback@suse.de>

# Homepage: http://www.suse.com/oracle/

#

### BEGIN INIT INFO

# Provides: oracle

# Required-Start: $network $syslog $remote_fs raw

# Required-Stop:

# Default-Start: 3 5

# Default-Stop: 0 1 2 6

# Description: Start the Oracle database

### END INIT INFO

 

 

# Shell functions sourced from /etc/rc.status:

#      rc_check         check and set local and overall rc status

#      rc_status        check and set local and overall rc status

#      rc_status -v     ditto but be verbose in local rc status

#      rc_status -v -r  ditto and clear the local rc status

#      rc_failed        set local and overall rc status to failed

#      rc_reset         clear local rc status (overall remains)

#      rc_exit          exit appropriate to overall rc status

. /etc/rc.status

 

 

# catch mis-use right here at the start

if [  "$1" != "start"  -a  "$1" != "stop"  -a  "$1" != "status" -a "$1" != "restart"  -a "$1" != "kill" ]; then

    echo "Usage: $0 {start|stop|status|restart}"

    exit 1

fi

 

 

LOAD_OCFS=/sbin/load_ocfs

MOUNT=/bin/mount

UMOUNT=/bin/umount

MKDIR=/bin/mkdir

RMMOD=/sbin/rmmod

 

CHECKPROC="/sbin/checkproc"

test -x "$CHECKPROC" || CHECKPROC="test -x "

 

 

# Get settings, if file(s) exist(s). If not, we simply use defaults.

if test -f /etc/sysconfig/oracle; then

    # new location as of SL 8.0 is directory /etc/sysconfig/

    . /etc/sysconfig/oracle

else

  if test -f /etc/rc.config.d/oracle.rc.config; then

    # location is directory /etc/rc.config.d/

    . /etc/rc.config.d/oracle.rc.config

  else

    if test -f /etc/rc.config; then

    # old SuSE location was to have everything in one file

    . /etc/rc.config

    fi

  fi

fi

 

 

# Determine the base and follow a runlevel link name.

# DISABLED by default because it's very individual...

#base=${0##*/}

#link=${base#*[SK][0-9][0-9]}

# Force execution if not called by a runlevel directory.

#test $link = $base && START_ORACLE_DB="yes" && START_ORACLE_DB_LISTENER="yes" && SET_ORACLE_KERNEL_PARAMETERS="yes"

 

 

# First reset status of this service

rc_reset

 

 

#

# Get and check environment (e.g. ORACLE_HOME)

#

ora_environment()

{

  test -f /etc/profile.d/oracle.sh && . /etc/profile.d/oracle.sh

  if [ ! -z "$ORACLE_HOME" -a ! -d "$ORACLE_HOME" ]; then

    echo

    echo "${warn}ORACLE_HOME directory $ORACLE_HOME does not exist!$norm"

    echo "Unsetting ORACLE_HOME, will try to determine it from system..."

    unset ORACLE_HOME

  fi

 

  # Try /etc/oratab if it's not set in /etc/profile.d/oracle.sh

  test -z "$ORACLE_HOME" && test -f /etc/oratab &&                        \

    ORACLE_HOME=`awk -F: '/^[^#].*:.+:[YN]/ {if ($2!="") print $2; exit}' </etc/oratab` &&   \

    echo && echo "ORACLE_HOME not set, but I found this in /etc/oratab: $ORACLE_HOME" && echo

 

  if [ -z "$ORACLE_HOME" ]; then

    echo "${warn}ORACLE_HOME environment variable not set.$norm"

    echo "Check /etc/profile.d/oracle.sh and /etc/oratab"

  fi

 

  if [ ! -d "$ORACLE_HOME" ]; then

    echo "${warn}Cannot find ORACLE_HOME directory $ORACLE_HOME.$norm"

    echo "Environment settings are wrong! Check /etc/profile.d/oracle.sh"

  fi

 

  test -z "$ORACLE_OWNER" && ORACLE_OWNER="oracle"

 

 

  if [ "$1" = "start" ]; then

    echo -n " ${extd}SETTINGS $1 from /etc/sysconfig/oracle$norm"

    if [ ! -f /etc/sysconfig/oracle ]; then

      echo " - ${warn}!!! MISSING !!!$norm"

    else

      echo

    fi

    echo " - Set Kernel Parameters for Oracle:   ${SET_ORACLE_KERNEL_PARAMETERS:-no}"

    echo " - Start Oracle OCFS:                  ${START_ORACLE_DB_OCFS:-no}"

    echo " - Start Oracle CRS:                   ${START_ORACLE_DB_CRS:-no}"

    echo " - Start Oracle EM:                    ${START_ORACLE_DB_EM:-no}"

   fi

}

 

 

# Here we finally get to do the real work.

case "$1" in

  start)

    echo

    echo "#############################################################################"

    echo "#                 Begin of   O R A C L E   startup section                  #"

    echo "#############################################################################"

    echo

    ora_environment start

 

    #

    # Check if we really have all the Oracle components we are told to start

    #

    if [ ! -x $ORACLE_HOME/bin/emctl -a ${START_ORACLE_DB_EM:-no} = "yes" ]; then

      echo "${warn}Can't find needed file: emctl - Setting START_ORACLE_DB_EM = no $norm"

      START_ORACLE_DB_EM="cannot";

    fi

 

    if [ ! -x /etc/init.d/init.crs -a ${START_ORACLE_DB_CRS:-no} = "yes" ]; then

      echo "${warn}Can't find needed file: init.crs - Setting START_ORACLE_DB_CRS = no $norm"

      START_ORACLE_DB_CRS="cannot";

    fi

 

 

    if [ ! -x /sbin/load_ocfs -a ${START_ORACLE_DB_OCFS:-no} = "yes" ]; then

      echo "${warn}Can't find needed file: /sbin/load_ocfs - Setting START_ORACLE_DB_OCFS = no $norm"

      START_ORACLE_DB_OCFS="cannot";

    fi

 

    echo

 

    # Set kernel parameters for Oracle

    if [ "${SET_ORACLE_KERNEL_PARAMETERS:-no}" == "yes" ]; then

      echo

      echo "Setting kernel parameters for Oracle, see file"

      if test -f /etc/sysconfig/oracle; then

        echo "/etc/sysconfig/oracle for explanations."

      else

        echo "/etc/rc.config.d/oracle.rc.config for explanations."

      fi

      echo

 

      if  [ ! -d /proc/sys/kernel ]; then

          echo; echo "No sysctl kernel interface - cannot set kernel parameters."; echo

          rc_failed

      else

        # Set shared memory parameters

        echo -n "${extd}Shared memory:$norm    "

        test -f /proc/sys/kernel/shmmax && echo -n "  SHMMAX=${SHMMAX:-3294967296}"

        test -f /proc/sys/kernel/shmmax && echo ${SHMMAX:-3294967296} > /proc/sys/kernel/shmmax

        test -f /proc/sys/kernel/shmmni && echo -n "  SHMMNI=${SHMMNI:-4096}"

        test -f /proc/sys/kernel/shmmni && echo ${SHMMNI:-4096}       > /proc/sys/kernel/shmmni

        test -f /proc/sys/kernel/shmall && echo    "  SHMALL=${SHMALL:-2097152}"

        test -f /proc/sys/kernel/shmall && echo ${SHMALL:-2097152}    > /proc/sys/kernel/shmall

        test -f /proc/sys/kernel/shmall || echo

 

        # Set the semaphore parameters:

        # see Oracle release notes for Linux for how to set these values

        # SEMMSL, SEMMNS, SEMOPM, SEMMNI

        echo -n "${extd}Semaphore values:$norm "

        test -f /proc/sys/kernel/sem && echo -n "  SEMMSL=${SEMMSL:-1250}"

        test -f /proc/sys/kernel/sem && echo -n "  SEMMNS=${SEMMNS:-32000}"

        test -f /proc/sys/kernel/sem && echo -n "  SEMOPM=${SEMOPM:-100}"

        test -f /proc/sys/kernel/sem && echo    "  SEMMNI=${SEMMNI:-256}"

        test -f /proc/sys/kernel/sem && echo ${SEMMSL:-1250} ${SEMMNS:-32000} ${SEMOPM:-100} ${SEMMNI:-128} > /proc/sys/kernel/sem

        test -f /proc/sys/kernel/sem || echo

 

        echo -n "${extd}Other values:$norm     "

        test -f /proc/sys/fs/file-max && echo -n "  FILE_MAX_KERNEL=${FILE_MAX_KERNEL:-131072}"

        test -f /proc/sys/fs/file-max && echo ${FILE_MAX_KERNEL:-131072} > /proc/sys/fs/file-max

        test -f /proc/sys/net/ipv4/ip_local_port_range && echo "  IP_LOCAL_PORT_RANGE=${IP_LOCAL_PORT_RANGE:-"1024 65000"}"

        test -f /proc/sys/net/ipv4/ip_local_port_range && echo ${IP_LOCAL_PORT_RANGE:-"1024 65000"} > /proc/sys/net/ipv4/ip_local_port_range

        test -f /proc/sys/vm/vm_mapped_ratio && echo -n "                    VM_MAPPED_RATIO=${VM_MAPPED_RATIO:-250}"

        test -f /proc/sys/vm/vm_mapped_ratio && echo ${VM_MAPPED_RATIO:-250} > /proc/sys/vm/vm_mapped_ratio

        test -f /proc/sys/fs/aio-max-size && echo "  AIO_MAX_SIZE=${AIO_MAX_SIZE:-131072}"

        test -f /proc/sys/fs/aio-max-size && echo ${AIO_MAX_SIZE:-131072} > /proc/sys/fs/aio-max-size

        test -f /proc/sys/fs/aio-max-size || echo

 

        echo -n "${extd}ULIMIT values:$norm    "

        echo    "  MAX_CORE_FILE_SIZE_SHELL=${MAX_CORE_FILE_SIZE_SHELL:-0}"

        ulimit -c ${MAX_CORE_FILE_SIZE_SHELL:-0}

        echo -n "                    FILE_MAX_SHELL=${FILE_MAX_SHELL:-65536}"

        ulimit -n ${FILE_MAX_SHELL:-65536}

        echo    "  PROCESSES_MAX_SHELL=${PROCESSES_MAX_SHELL:-16384}"

        ulimit -u ${PROCESSES_MAX_SHELL:-16384}

 

        # Check if shmmax is really set to what we want - on some systems and

        # certain settings the result could be shmmax=0 if you set it to e.g. 4GB!

        if [ `cat /proc/sys/kernel/shmmax` != "${SHMMAX:-3294967296}" ]; then

          echo "${warn}---- WARNING - SHMMAX could not be set properly ----$norm"

          echo "   Tried to set it to: ${SHMMAX:-3294967296}"

          echo "   Value is now:       `cat /proc/sys/kernel/shmmax`"

          echo "   You might try again with a lower value."

        fi

      fi

      echo

   

      echo -n "Kernel parameters set for Oracle: "

      rc_status -v

      echo

      echo

    fi

 

    rc_reset

 

    echo -n "  - Starting Oracle Cluster Filesystem..."

    if [ "${START_ORACLE_DB_OCFS:-no}" = "yes" ]; then

       $LOAD_OCFS >& /dev/null

       rc_status -v -r

       echo -n "  - Mounting Oracle Cluster Filesystem(s)..."

       $MOUNT -a -t ocfs >& /dev/null

       rc_status -v -r

    else

       if [ ${START_ORACLE_DB_OCFS:-no} = "cannot" ]; then

         rc_status -s

       else

         rc_status -u

       fi

    fi

 

    rc_reset

 

    echo -n "  - Starting Oracle CRS..."

    if [ "${START_ORACLE_DB_CRS:-no}" = "yes" ]; then

       /etc/init.d/init.crs start >& /dev/null

       rc_status -v -r

       sleep 10

       kill -1 1

    else

       if [ ${START_ORACLE_DB_CRS:-no} = "cannot" ]; then

         rc_status -s

       else

         rc_status -u

       fi

    fi

 

    rc_reset

 

    echo

 

    echo -n "  - Starting Oracle EM dbconsole..."

    if [ "${START_ORACLE_DB_EM:-no}" = "yes" ]; then

        su - $ORACLE_OWNER -c "export ORACLE_HOME=$ORACLE_HOME TNS_ADMIN=$TNS_ADMIN; $ORACLE_HOME/bin/emctl start dbconsole >& /dev/null &"

      rc_status -v -r

    else

       if [ ${START_ORACLE_DB_EM:-no} = "cannot" ]; then

         rc_status -s

       else

         rc_status -u

       fi

    fi

 

    rc_reset

    ;;

 

 

  stop|kill)

    echo

    echo "#############################################################################"

    echo "#                 Begin of   O R A C L E   shutdown section                 #"

    echo "#############################################################################"

    echo

    ora_environment stop

 

    echo "Shutting down Oracle services (only those running)"; echo

 

    test -x $ORACLE_HOME/Apache/Apache/bin/apachectl && $CHECKPROC $ORACLE_HOME/Apache/Apache/bin/httpd  && echo -n "Shutting down Apache: " && (export ORACLE_HOME=$ORACLE_HOME TNS_ADMIN=$TNS_ADMIN; $ORACLE_HOME/Apache/Apache/bin/apachectl stop >& /dev/null; rc_status -v -r)

    test -x $AGENT_PROG                              && $CHECKPROC $ORACLE_HOME/bin/dbsnmp               && echo -n "Shutting down Agent: " && (su - $ORACLE_OWNER -c "export ORACLE_HOME=$ORACLE_HOME TNS_ADMIN=$TNS_ADMIN; $AGENT_STOP >& /dev/null"; rc_status -v -r)

    test -x $ORACLE_HOME/bin/emctl                   && echo -n                                          && echo -n "Shutting down EM console: " && (killall emagent;killall emdctl; rc_status -v -r)

    test -x $ORACLE_HOME/bin/cmctl                   && echo -n                                          && echo -n "Shutting down Connection Manager: " && (su - $ORACLE_OWNER -c "export ORACLE_HOME=$ORACLE_HOME TNS_ADMIN=$TNS_ADMIN; $ORACLE_HOME/bin/cmctl stop >& /dev/null"; rc_status -v -r)

    test -x $ORACLE_HOME/bin/lsnrctl                 && $CHECKPROC $ORACLE_HOME/bin/tnslsnr              && echo -n "Shutting down Listener: " && (su - $ORACLE_OWNER -c "export ORACLE_HOME=$ORACLE_HOME TNS_ADMIN=$TNS_ADMIN; $ORACLE_HOME/bin/lsnrctl stop >& /dev/null & "; rc_status -v -r)

    test -x $ORACLE_HOME/bin/dbshut                  && $CHECKPROC $ORACLE_HOME/bin/oracle               && echo -n "Shutting down Database: " && (su - $ORACLE_OWNER -c "export ORACLE_HOME=$ORACLE_HOME TNS_ADMIN=$TNS_ADMIN; $ORACLE_HOME/bin/dbshut >& /dev/null"; rc_status -v -r)

    test -x $ORACLE_HOME/bin/gsdctl                  && test "" != "`ps U oracle|grep 'DPROGRAM=gsd'`"   && echo -n "Shutting down GSD: " && (su - $ORACLE_OWNER -c "export ORACLE_HOME=$ORACLE_HOME TNS_ADMIN=$TNS_ADMIN; unset JAVA_BINDIR; unset JAVA_HOME; $ORACLE_HOME/bin/gsdctl stop >& /dev/null"; rc_status -v -r)

    test -x $ORACLE_HOME/oracm/bin/oracm             && $CHECKPROC $ORACLE_HOME/oracm/bin/oracm          && echo -n "Shutting down OCM: " && (killall oracm >& /dev/null; rm -f /etc/rac_on; rc_status -v -r)

    test -x /etc/init.d/init.crs                            && echo -n                                                                                        && echo -n "Turning off CRS:" && /etc/init.d/init.crs stop 

    if [ "`$MOUNT -t ocfs | grep ocfs`" != "" ]; then

      echo -n "Unmounting all OCFS filesystems: " && ($UMOUNT -t ocfs -a; rc_status -v -r)

    fi

    if [ "$1" = "kill" ]

    then

        echo Killing all ORACLE processes

                kill `ps aux | grep '^oracle' | awk '{print $2}'`

        sleep 10

                kill -9 `ps aux | grep '^oracle' | awk '{print $2}'`

    fi

 

    ;;

  status)

    echo

    echo "#############################################################################"

    echo "#                  Begin of   O R A C L E   status section                  #"

    echo "#############################################################################"

    echo

    ora_environment status

 

    echo "${extd}Kernel Parameters$norm"

    echo -n "Shared memory:"

    echo -n "  SHMMAX=" `cat /proc/sys/kernel/shmmax`

    echo -n "  SHMMNI=" `cat /proc/sys/kernel/shmmni`

    echo    "  SHMALL=" `cat /proc/sys/kernel/shmall`

    echo -n "Semaphore values:"

    echo    "  SEMMSL, SEMMNS, SEMOPM, SEMMNI: " `cat /proc/sys/kernel/sem`

    echo

 

    if [ -x $ORACLE_HOME/bin/oracle ]; then

      echo "${extd}Database-Instances$norm"

      # loop over the instances (very simple !!!)

      IFS=:

      grep -v '^\(#\|$\)' /etc/oratab | while read sid ohome autostart ; do

        state=up

        su - $ORACLE_OWNER -c "export ORACLE_SID=$sid; sqlplus /nolog" <<-! 2>/dev/null | grep ORA-01034 >/dev/null && state=down

connect / as sysdba

show sga

!

        echo "Instance $sid is $state \(autostart: $autostart\)"

      done

      echo

    fi

 

    if [ -x $ORACLE_HOME/bin/lsnrctl ]; then

      state=up

      su - $ORACLE_OWNER -c "$ORACLE_HOME/bin/lsnrctl status" | grep "[nN]o [lL]istener" >/dev/null && state=down

      echo "${extd}TNS-Listener:$norm $state"

      echo

    fi

 

    numhttpd=`ps -e | grep httpd | wc -l | sed 's/ //g'`

    state=up

    if [ "$numhttpd" -lt 1 ] ; then state=down ; fi

    echo "${extd}Web-Server (Apache httpd):$norm $state ($numhttpd processes)"

    echo

 

    if [ -x $ORACLE_HOME/bin/agentctl ]; then

      state=up

      su - $ORACLE_OWNER -c "agentctl status" | grep "Could not contact agent" >/dev/null && state=down

      echo "${extd}Intelligent Agent:$norm $state"

      echo

    fi

 

    echo "${extd}Process list for user oracle:$norm"

    ps U oracle

    ps ax | grep oracm | grep -v grep

    ps axc | grep crs | grep -v grep

    ps axc | grep css | grep -v grep

    tail -10 $ORACLE_HOME/../crs_1/crs/log/*

    ;;

  restart)

    ## Stop the service and regardless of whether it was

    ## running or not, start it again.

    $0 stop

    $0 start

    ;;

  *)

    echo "Usage: $0 {start|stop|status|restart}"

    exit 1

esac

 

echo

echo "#############################################################################"

echo "#                      End of   O R A C L E   section                       #"

echo "#############################################################################"

echo

 

# Global return value of this script is "success", always. We have too many

# individual return values...

rc_status -r

 

rc_exit

 

 

 

APPENDIX 2. List of packages, required for Oracle 10g for Linux x86-64.

Magnie Fabrizio provided this list. It is not official Oracle list.



glibc-devel-2.3.3-98.28
glibc-2.3.3-98.28
glibc-32bit-9-200407011233
glibc-devel-32bit-9-200407011229
compat-2004.7.1-1.2
compat-32bit-9-200407011229
XFree86-libs-4.3.99.902-43.22
XFree86-libs-32bit-9-200407011229
libaio-devel-0.3.98-18.4
libaio-32bit-9-200407011229
libaio-0.3.98-18.4
libaio-devel-32bit-9-200407011229
openmotif-libs-2.2.2-519.1
openmotif-2.2.2-519.1

Openmotif is needed only for 10g.
The version are the default of a SLES9 without SP1.


 



[1] I know, that it is not the best selection. But, having big enough disks, and having all database systems isolated from main network by firewalls (so that security is not an issue relating to the local system users), I do not create significant overhead (except for online updates). See Appendix 2 – list of required packages.

[2] INPORTANT. In some cases, port negotiation can take up to 20 – 30 seconds (in my combination. DELL 2850 and Cisco Catalist 2970, for example). If it happens, you should add sleep 30 delay into /etc/init.d/network startup script in start: section, before exiting, to prevent problems with following scripts.

[3] Read and obey all Oracle licenses before downloading files and installing Oracle cluster.

[4] Some documents recommend using shared $ORACLE_BASE disk. To do it, create one more volume, mount it without noac and symlink  it as /opt/oracle, before installing orarun.  Do not set noac option for this particular volume (and only for Oracle Home), if doing it – this option can make filer crazy if set on Oracle Home. I use local homes in this example.

[5] This information is provided for unskilled DBA only, and only for general orientation. You should read Oracle documentation about RAC cluster to manage any production grade database.