Focus On Oracle

Installing, Backup & Recovery, Performance Tuning,
Troubleshooting, Upgrading, Patching

当前位置: 首页 » 技术文章 » AODU

aodu(At Oracle Database Utility)之rac(一)

aodu的函数rac包含了以下内容:如何更改归档模式,常用术语,故障诊断常用脚本,设置调试模式,如何使用cvu(Cluster Vierify Utility),crsctl/srvctl常用命令,ocrconfig命令,olsnodes命令等。


[oracle@db1 ~]$ ./aodu

AT Oracle Database Utility,Release 1.1.0 on Tue Jun 14 14:28:01 2016
Copyright (c) 2014, 2015, Robin.Han.  All rights reserved.


AODU> rac ohsdba
         rac archivelog|general|abbr|diag|cvu|diaginstall|debug|eviction|diagrac|perf|srvctl|crsctl|ocrconfig|olsnodes|debug|note
AODU> rac oracle
         Currently it's for internal use only
注意:只有通过rac ohsdba会把函数中常用命令显示出来,当然也只有看过这篇文章,才会知道如何使用

AODU> rac archivelog

        ****Change archivedlog mode****
        The following steps need to be taken to enable archive logging in a RAC database environment:
        $srvctl stop database -d <db_unique_name>
        $srvctl start database -d <db_unique_name> -o mount
        $sqlplus / as sysdba
        sql> alter database archivelog;
        sql> exit;
        $ srvctl stop database -d <db_unique_name>
        $ srvctl start database -d <db_unique_name>
        sql> archive log list;
        Note:from 10g,you do not need to change parameter cluster_database

AODU> rac general     ---11gR2的特性和RAC启动顺序的说明
        ****11gR2 Clusterware Key Facts****
        11gR2 Clusterware is required to be up and running prior to installing a 11gR2 Real Application Clusters database.
        The GRID home consists of the Oracle Clusterware and ASM.  ASM should not be in a separate home.
        The 11gR2 Clusterware can be installed in "Standalone" mode for ASM and/or "Oracle Restart" single node support.
        This clusterware is a subset of the full clusterware described in this document.
        The 11gR2 Clusterware can be run by itself or on top of vendor clusterware.See the certification matrix for certified
        combinations. Ref: Note: 184875.1 "How To Check The Certification Matrix for Real Application Clusters"
        The GRID Home and the RAC/DB Home must be installed in different locations.
        The 11gR2 Clusterware requires a shared OCR files and voting files.  These can be stored on ASM or a cluster filesystem.
        The OCR is backed up automatically every 4 hours to <GRID_HOME>/cdata/<clustername>/ and can be restored via ocrconfig.
        The voting file is backed up into the OCR at every configuration change and can be restored via crsctl.
        The 11gR2 Clusterware requires at least one private network for inter-node communication and at least one
        public network for external communication.Several virtual IPs need to be registered with DNS.This includes the node VIPs (one per node),
        SCAN VIPs (three).This can be done manually via your network administrator or optionally you could configure the "GNS" (Grid Naming Service)
        in the Oracle clusterware to handle this for you (note that GNS requires its own VIP).
        A SCAN (Single Client Access Name) is provided to clients to connect to.  For more information on SCAN see Note: 887522.1
        The script at the end of the clusterware installation starts the clusterware stack.For information on troubleshooting issues see Note: 1053970.1
        Only one set of clusterware daemons can be running per node.
        On Unix, the clusterware stack is started via the init.ohasd script referenced in /etc/inittab with "respawn".
        A node can be evicted (rebooted) if a node is deemed to be unhealthy.  This is done so that the health of the entire cluster can be maintained.
        For more information on this see: Note: 1050693.1 "Troubleshooting 11.2 Clusterware Node Evictions (Reboots)"
        Either have vendor time synchronization software (like NTP) fully configured and running or have it not configured at all and
        let CTSS handle time synchronization.See Note: 1054006.1 for more information.
        If installing DB homes for a lower version, you will need to pin the nodes in the clusterware or you will see ORA-29702 errors.
        See Note 946332.1 and Note:948456.1 for more information.
        The clusterware stack can be started by either booting the machine, running "crsctl start crs" to start the clusterware stack,
        or by running "crsctl start cluster" to start the clusterware on all nodes.Note that crsctl is in the <GRID_HOME>/bin directory.
        Note that "crsctl start cluster" will only work if ohasd is running.
        The clusterware stack can be stopped by either shutting down the machine, running "crsctl stop crs" to stop the clusterware stack,
        or by running "crsctl stop cluster" to stop the clusterware on all nodes.Note that crsctl is in the <GRID_HOME>/bin directory.
        Killing clusterware daemons is not supported.
        Instance is now part of .db resources in "crsctl stat res -t" output, there is no separate .inst resource for 11gR2 instance.

        ****Cluster Start Sequence****
        This daemon spawns 4 processes.
        Level 1: OHASD Spawns:
            cssdagent - Agent responsible for spawning CSSD.
            orarootagent - Agent responsible for managing all root owned ohasd resources.
            oraagent - Agent responsible for managing all oracle owned ohasd resources.
            cssdmonitor - Monitors CSSD and node health (along wth the cssdagent).
        Level 2: OHASD rootagent spawns:
            CRSD - Primary daemon responsible for managing cluster resources.
            CTSSD - Cluster Time Synchronization Services Daemon
            ACFS (ASM Cluster File System) Drivers
        Level 2: OHASD oraagent spawns:
            MDNSD - Used for DNS lookup
            GIPCD - Used for inter-process and inter-node communication
            GPNPD - Grid Plug & Play Profile Daemon
            EVMD - Event Monitor Daemon
            ASM - Resource for monitoring ASM instances
        Level 3: CRSD spawns:
            orarootagent - Agent responsible for managing all root owned crsd resources.
            oraagent - Agent responsible for managing all oracle owned crsd resources.
        Level 4: CRSD rootagent spawns:
            Network resource - To monitor the public network
            SCAN VIP(s) - Single Client Access Name Virtual IPs
            Node VIPs - One per node
            ACFS Registery - For mounting ASM Cluster File System
            GNS VIP (optional) - VIP for GNS
        Level 4: CRSD oraagent spawns:
            ASM Resouce - ASM Instance(s) resource
            Diskgroup - Used for managing/monitoring ASM diskgroups.
            DB Resource - Used for monitoring and managing the DB and instances
            SCAN Listener - Listener for single client access name, listening on SCAN VIP
            Listener - Node listener listening on the Node VIP
            Services - Used for monitoring and managing services
            ONS - Oracle Notification Service
            eONS - Enhanced Oracle Notification Service
            GSD - For 9i backward compatibility
            GNS (optional) - Grid Naming Service - Performs name resolution

AODU> rac abbr    ---RAC术语

        ****Abbreviations, Acronyms****
        This note lists commonly used Oracle Clusterware(Cluster Ready Service or Grid Infrastructure) Related abbreviations,acronyms,terms and Procedures.
        nodename: short hostname for local node. For example, racnode1 for node
        CRS: Cluster Ready Service, name for pre-11gR2 Oracle clusterware
        GI: Grid Infrastructure, name for 11gR2 Oracle clusterware
        GI cluster: Grid Infrastructure in cluster mode
        Oracle Restart: GI Standalone, Grid Infrastructure in standalone mode
        ASM user: the OS user who installs/owns ASM. For 11gR2, ASM and grid user is the same as ASM and GI share the same ORACLE_HOME.
        For pre-11gR2 CRS cluster, ASM and CRS user can be different as ASM and CRS will be in different ORACLE_HOME.
        For pre-11gR2 single-instance ASM, ASM and local CRS user is the same as ASM and local CRS share the same home.
        CRS user: the OS user who installs/owns pre-11gR2 Oracle clusterware
        grid user: the OS user who installs/owns 11gR2 Oracle clusterware
        clusterware user: CRS or grid user which must be the same in upgrade environment
        Oracle Clusterware software owner: same as clusterware user
        clusterware home: CRS or GI home
        ORACLE_BASE:ORACLE_BASE for grid or CRS user.
        root script checkpoint file: the file that records root script ( or progress so root script
        can be re-executed, it's located in $ORACLE_BASE/Clusterware/ckptGridHA_${nodename}.xml
        OCR: Oracle Cluster Registry. To find out OCR location, execute: ocrcheck
        VD: Voting Disk. To find out voting file location, execute: crsctl query css votedisk
        Automatic OCR Backup: OCR is backed up automatically every four hours in cluster environment on OCR Master node,
        the default location is <clusterware-home>/cdata/<clustername>. To find out backup location, execute: ocrconfig -showbackup
        SCR Base: the directory where ocr.loc and olr.loc are located.
            Linux:         /etc/oracle
            Solaris:     /var/opt/oracle
            hp-ux:         /var/opt/oracle
            AIX:             /etc/oracle
        INITD Location: the directory where ohasd and init.ohasd are located.
            Linux:         /etc/init.d
            Solaris:     /etc/init.d
            hp-ux:         /sbin/init.d
            AIX:             /etc
        oratab Location: the directory where oratab is located.
            Linux:         /etc
            Solaris:     /var/opt/oracle
            hp-ux:         /etc
            AIX:             /etc
        CIL: Central Inventory Location. The location is defined by parameter inventory_loc in /etc/oraInst.loc or
        /var/opt/oracle/oraInst.loc depend on platform.
            Example on Linux:
            cat /etc/oraInst.loc | grep inventory_loc
        Disable CRS/GI: To disable clusterware from auto startup when node reboots, as root execute "crsctl disable crs".
        Replace it with "crsctl stop has" for Oracle Restart.
        DG Compatible: ASM Disk Group's compatible.asm setting. To store OCR/VD on ASM, the compatible setting must be at least,
        but on the other hand lower GI version won't work with higher compatible setting. For example, GI will have issue to
        access a DG if compatible.asm is set to When downgrading from higher GI version to lower GI version,
        if DG for OCR/VD has higher compatible, OCR/VD relocation to lower compatible setting is necessary.
        To find out compatible setting, log on to ASM and query:
        SQL> select name||' => '||compatibility from v$asm_diskgroup where name='GI';
        GI =>

        In above example, GI is the name of the interested disk group.
        To relocate OCR from higher compatible DG to lower one:
        ocrconfig -add <diskgroup>
        ocrconfig -delete <disk group>
        To relocate VD from higher compatible DG to lower one:
        crsctl replace votedisk <diskgroup>

        When upgrading Oracle Clusterware:
        OLD_HOME: pre-upgrade Oracle clusterware home - the home where existing clusterware is running off. For Oracle Restart,
        the OLD_HOME is pre-upgrade ASM home.
        OLD_VERSION: pre-upgrade Oracle clusterware version.
        NEW_HOME: new Oracle clusterware home.
        NEW_VERSION: new Oracle clusterware version.
        OCR Node: The node where backs up pre-upgrade OCR to $NEW_HOME/cdata/ocr$OLD_VERSION. In most case
        it's first node where was executed.
        Example when upgrading from to, after execution of

        ls -l $NEW_HOME/cdata/ocr*
        -rw-r--r-- 1 root root 78220 Feb 16 10:21 /ocw/b202/cdata/ocr11.

AODU> rac diag   --如何收集rac的故障诊断信息

        ****Data Gathering for All Oracle Clusterware Issues****
        TFA Collector is installed in the GI HOME and comes with GI and higher. For GI or lower,
        install the TFA Collector by referring to Document 1513912.1 for instruction on downloading and installing TFA collector.
        $GI_HOME/tfa/bin/tfactl diagcollect -from "MMM/dd/yyyy hh:mm:ss" -to "MMM/dd/yyyy hh:mm:ss"
        Format example: "Jul/1/2014 21:00:00"
        Specify the "from time" to be 4 hours before and the "to time" to be 4 hours after the time of error.
        ****Linux/Unix Platform
        a.Linux/UNIX 11gR2/12cR1
        1. Execute the following as root user:
        # script /tmp/diag.log
        # id
        # env
        # cd <temp-directory-with-plenty-free-space>
        # $GRID_HOME/bin/
        # exit
        For more information about diagcollection, check out " -help"
        The following .gz files will be generated in the current directory and need to be uploaded along with /tmp/diag.log:
        Linux/UNIX 10gR2/11gR1
        1. Execute the following as root user:
        # script /tmp/diag.log
        # id
        # env
        # cd <temp-directory-with-plenty-free-space>
        # export OCH=<CRS_HOME>
        # export ORACLE_HOME=<DB_HOME>
        # export HOSTNAME=<host>
        # $OCH/bin/ -crshome=$OCH --collect
        # exit
        The following .gz files will be generated in the current directory and need to be uploaded along with /tmp/diag.log:
        2. For 10gR2 and 11gR1, if getting an error while running, please collect /tmp/crsctl.*
        Please ensure all above information are provided from all the nodes.

        ****Windows Platform****
        b.Windows 11gR2/12cR1:
        set ORACLE_HOME=<GRID_HOME>    for example: set ORACLE_HOME=D:\app\11.2.0\grid
        set PATH=%PATH%;%ORACLE_HOME%\perl\bin
        perl %ORACLE_HOME%\bin\ --collect --crshome %ORACLE_HOME%
        The following .zip files will be generated in the current directory and need to be uploaded:
        coreData_<timestamp>.zip (only --core option specified)
        For chmosdata*:
        perl %ORACLE_HOME%\bin\ --collect --crshome %ORACLE_HOME%
        Windows 10gR2/11gR1
        set ORACLE_HOME=<DB_HOME>
        set OCH=<CRS_HOME>
        set ORACLE_BASE=<oracle-base>
        %OCH%\perl\bin\perl %OCH%\bin\ --collect

AODU> rac cvu    --如何使用CVU
        ****Cluster verify utility****
        How to Debug CVU / Collect CVU Trace Generated by RUNCLUVFY.SH (Doc ID 986822.1)
        (a) GI/CRS has been installed
        $ script /tmp/cluvfy.log
        $ $GRID_HOME/bin/cluvfy stage -pre crsinst -n <node1, node2...> -verbose
        $ $GRID_HOME/bin/cluvfy stage -post crsinst -n all -verbose
        $ exit
        (b) GI/CRS has not been installed
        run from the installation media or download from OTN
        set the environment variables CV_HOME to point to the cvu home, CV_JDKHOME to point to the JDK home and an optional
        CV_DESTLOC pointing to a writeable area on all nodes (e.g /tmp/cluvfy)
        $ cd $CV_HOME/bin
        $ script cluvfy.log
        $ cluvfy stage -pre crsinst -n <node1, node2...>
        $ exit

        ****Diagcollection options****
        To collect only a subset of logs, --adr together with --beforetime and --aftertime can be used, i.e.:
        # mkdir /tmp/collect
        # $GRID_HOME/bin/ --adr /tmp/collect -beforetime 20120218100000 -aftertime 20120218050000
        This command will copy all logs contain timestamp 2012-02-18 05:00 - 10:00 to /tmp/collect directory.
        Time is specified with YYYYMMDDHHMISS24 format. --adr points to a directory where the logs are copied to.
        From onwards, Cluster Health Monitor(CHM/OS) note 1328466.1 data can also be collected, i.e.:
        # $GRID_HOME/bin/ --chmos --incidenttime 02/18/201205:00:00 --incidentduration 05:00
        This command will collect data from 2012-02-18 05:00 to 10:00 for 5 hours. incidenttime is specified as MM/DD/YYYY24HH:MM:SS,
        incidentduration is specified as HH:MM.

        ****For 11gR2/12c:****
        Set environment variable CV_TRACELOC to a directory that's writable by grid user; trace should be generated in there once starts. For example, as grid user:
        rm -rf /tmp/cvutrace
        mkdir /tmp/cvutrace
        export CV_TRACELOC=/tmp/cvutrace
        export SRVM_TRACE=true
        export SRVM_TRACE_LEVEL=1
        <STAGE_AREA>/ stage -pre crsinst -n <node1>,<node2> -verbose

        A. STAGE_AREA refers to the location where Oracle Clusterware is unzipped.
        B. Replace above " stage -pre crsinst -n <node1>,<node2> -verbose"
        if other stage/comp needs to be traced,i.e."./ comp ocr -verbose"
        C. For, the following can be set for additional tracing from exectask command:
        export EXECTASK_TRACE=true
        The trace will be in <TMP>/CVU_<version>_<user>/exectask.trc, i.e. /tmp/CVU_12.

        ****For 10gR2, 11gR1 or 11gR2:****
        1. As crs/grid user, backup For 10gR2, it's located in <STAGE_AREA>/cluvfy/;
        and for 11gR1 and 11gR2, <STAGE_AREA>/
        cd <STAGE_AREA>
        cp runcluvfy.debug

        2. Locate the following lines in runcluvfy.debug:
        # Cleanup the home for cluster verification software
        $RM -rf $CV_HOME
        Comment out the remove command so runtime files including trace won't be removed once CVU finishes.
        # Cleanup the home for cluster verification software
        # $RM -rf $CV_HOME

        3. As crs/grid user,set environment variable CV_HOME to anywhere as long as the location is writable by crs/grid user and has 400MB of free space:
        mkdir /tmp/cvdebug
        export CV_HOME
        This step is optional, if CV_HOME is unset, CVU files will be generated in /tmp.

        4. As crs/grid user, execute runcluvfy.debug:
        export SRVM_TRACE=true
        export SRVM_TRACE_LEVEL=1
        cd <STAGE_AREA>
        ./runcluvfy.debug stage -pre crsinst -n <node1>,<node2> -verbose
        A. SRVM_TRACE_LEVEL is effective for 11gR2 only.
        B. Replace above " stage -pre crsinst -n <node1>,<node2> -verbose" if other stage/comp needs to be
        traced, i.e. "./ comp ocr -verbose"

        5. Regardless whether above command finishes or not, CVU trace should be generated in:
        10gR2: $CV_HOME/<pid>/cv/log
        11gR1: $CV_HOME/bootstrap/cv/log
        11gR2: $CV_HOME/bootstrap/cv/log
        If CV_HOME is unset, trace will be in /tmp/<pid>/cv/log or /tmp/bootstrap/cv/log depend on CVU version.

        6. Clean up temporary files generated by above runcluvfy.debug:
        rm -rf $CV_HOME/bootstrap
        On the other hand e.g. GridControl uses the installed cluvfy from CRS_HOME. So if GridControl shows errors when checking the cluster
        You may want to update the cluvfy used by GridControl.
        As cluvfy consists of many files it is best to install the newest version outside of CRS_HOME so there
        will be no conflict between the jar files and libraries used by cluvfy and CRS.
        To do so follow these steps:
        1. download the newest version of cluvfy from here and extract the files into the target directory You want to use.
        The following assumptions are made here:
        new cluvfy-home CVU_HOME = /u01/oracle/cluvfy
        2. go to CRS_HOME/bin and copy the existing file CRS_HOME/bin/cluvfy to CRS_HOME/bin/
        3. copy CVU_HOME/bin/cluvfy to CRS_HOME/bin/cluvfy
        4. edit that file and search for the line
        CRSHOME=        CRSHOME=that file and search fosearch foe
S_HOME/bin/cOME/bin/cCRS_HOME/bin/cluvfy.orgluvfy.organt to use.
ewest versiest versie of CRS_HOME so there so there cking the cluster
ime ter
ime ied as MM/DD/YYYY24HH:MYYY24HH:Mt to the JDK home and aome and al
ng. For example, 11.mple, 11.will have issue to
        ace to
        ac if compatible.asm is s.asm is s2.0.2.0. When downgradidowngradiigher GI version to lowon to lowsion,
ptionally you coly you cogure the "GNS" (Grid Na (Grid Na (Grid_OracleHome
        make the following corrections:
        CV_HOME=/u01/oracle/cluvfy           <---- check for Your environment
        ./ stage -pre dbinst -n <node1>,<node2> -verbose | tee /tmp/cluvfy_dbinst.log stage -pre  hacfg -verbose
        ./cluvfy stage -pre dbinst -n <node1>,<node2> -verbose | tee /tmp/cluvfy_dbinst.log
        ./ stage -pre crsinst -n <node1>,<node2> -verbose | tee /tmp/cluvfy_preinst.log
        ./cluvfy stage -post crsinst -n all -verbose | tee /tmp/cluvfy_postinst.log

        $ ./cluvfy comp -list
        cluvfy comp  <component-name> <component-specific options>  [-verbose]
        Valid components are:
                nodereach : checks reachability between nodes
                nodecon   : checks node connectivity
                cfs       : checks CFS integrity
                ssa       : checks shared storage accessibility
                space     : checks space availability
                sys       : checks minimum system requirements
                clu       : checks cluster integrity
                clumgr    : checks cluster manager integrity
                ocr       : checks OCR integrity
                olr       : checks OLR integrity
                ha        : checks HA integrity
                crs       : checks CRS integrity
                nodeapp   : checks node applications existence
                admprv    : checks administrative privileges
                peer      : compares properties with peers
                software  : checks software distribution
                asm       : checks ASM integrity
                acfs       : checks ACFS integrity
                gpnp      : checks GPnP integrity
                gns       : checks GNS integrity
                scan      : checks SCAN configuration
                ohasd     : checks OHASD integrity
                clocksync      : checks Clock Synchronization
                vdisk      : check Voting Disk Udev settings

        $ ./cluvfy stage -list
        cluvfy stage {-pre|-post} <stage-name> <stage-specific options>  [-verbose]
        Valid stage options and stage names are:
                -post hwos    :  post-check for hardware and operating system
                -pre  cfs     :  pre-check for CFS setup
                -post cfs     :  post-check for CFS setup
                -pre  crsinst :  pre-check for CRS installation
                -post crsinst :  post-check for CRS installation
                -pre  hacfg   :  pre-check for HA configuration
                -post hacfg   :  post-check for HA configuration
                -pre  dbinst  :  pre-check for database installation
                -pre  acfscfg  :  pre-check for ACFS Configuration.
                -post acfscfg  :  post-check for ACFS Configuration.
                -pre  dbcfg   :  pre-check for database configuration
                -pre  nodeadd :  pre-check for node addition.
                -post nodeadd :  post-check for node addition.
                -post nodedel :  post-check for node deletion.

        cluvfy comp ssa -n dbnode1,dbnode2 -s

        Datatbase logs & trace files:
        cd $(orabase)/diag/rdbms
        tar cf - $(find . -name '*.trc' -exec egrep '<date_time_search_string>' {} \; grep -v bucket) | gzip >  /tmp/database_trace_files.tar.gz
        ASM logs & trace files:
        cd $(orabase)/diag/asm/+asm/
        tar cf - $(find . -name '*.trc' -exec egrep '<date_time_search_string>' {} \; grep -v bucket) | gzip >  /tmp/asm_trace_files.tar.gz
        Clusteware logs:
        <GI home>/bin/ --collect --crs --crshome <GI home>
        OS logs:
        /var/adm/messages* or /var/log/messages* or 'errpt -a' or Windows System Event Viewer log (saved as .TXT file)

AODU> rac diaginstall   --RAC安装故障诊断信息收集
       ****Data Gathering for Oracle Clusterware Installation Issues****
         Failure before executing root script:
        For 11gR2: note 1056322.1 - Troubleshoot 11gR2 Grid Infrastructure/RAC Database runInstaller Issues
        For pre-11.2: note 406231.1 - Diagnosing RAC/RDBMS Installation Problems
         Failure while or after executing root script
        Provide files in Section "Data Gathering for All Oracle Clusterware Issues" and the following:
            root script ( or screen output
            For 11gR2: provide zip of <$ORACLE_BASE>/cfgtoollogs and <$ORACLE_BASE>/diag for grid user.
            For pre-11.2: Note 240001.1 - Troubleshooting 10g or 11.1 Oracle Clusterware Problems
        Before deconfiguring, collect the following as grid user if possible to generate a list of user resources to be
        added back to the cluster after reconfigure finishes:
        $GRID_HOME/bin/crsctl stat res -t
        $GRID_HOME/bin/crsctl stat res -p
        $GRID_HOME/bin/crsctl query css votedisk
        $GRID_HOME/bin/crsctl query crs activeversion
        $GRID_HOME/bin/crsctl check crs
        cat /etc/oracle/ocr.loc /var/opt/oracle/ocr.loc
        $GRID_HOME/bin/crsctl get css diagwait
        $GRID_HOME/bin/oifcfg iflist -p -n
        $GRID_HOME/bin/oifcfg getif
        $GRID_HOME/bin/crsctl query css votedisk
        $GRID_HOME/bin/srvctl config nodeapps -a
        $GRID_HOME/bin/srvctl config scan
        $GRID_HOME/bin/srvctl config asm -a
        $GRID_HOME/bin/srvctl config listener -l <listener-name> -a
        $DB_HOME/bin/srvctl config database -d <dbname> -a
        $DB_HOME/bin/srvctl config service -d <dbname> -s <service-name> -v
        # <$GRID_HOME>/crs/install/ -deconfig -force -verbose
        OHASD Agents do not start
        Troubleshoot Grid Infrastructure Startup Issues (Doc ID 1050908.1)
        In a nutshell, the operating system starts ohasd, ohasd starts agents to start up daemons (gipcd, mdnsd, gpnpd, ctssd, ocssd, crsd, evmd asm etc),
        and crsd starts agents that start user resources (database, SCAN, listener etc).
        OHASD.BIN will spawn four agents/monitors to start resource:
          oraagent: responsible for ora.asm, ora.evmd, ora.gipcd, ora.gpnpd, ora.mdnsd etc
          orarootagent: responsible for ora.crsd, ora.ctssd, ora.diskmon, ora.drivers.acfs etc
          cssdagent / cssdmonitor: responsible for ora.cssd(for ocssd.bin) and ora.cssdmonitor(for cssdmonitor itself)
        $GRID_HOME/bin/crsctl start res ora.crsd -init
        $GRID_HOME/bin/crsctl start res ora.evmd -init
        $GRID_HOME/bin/crsctl stop res ora.evmd -init
        ps -ef | grep <keyword> | grep -v grep | awk '{print $2}' | xargs kill -9
        If ohasd.bin can not start any of above agents properly, clusterware will not come to healthy state.

AODU> rac debug   --RAC设置调试模式
        ****Troubleshooting Steps****
        Following are the new features in 10.2/11gR1/11gR2
        Using crsctl, debugging can be turned on and off for CRS/EVM/CSS and their subcomponents. Debug levels can also be dynamically changed
        using crsctl. The debug information is persisted in OCR for use during the next startup. Debugging can be turned on for CRS managed
        resource like VIP, Instance as well.
        * Note,in the following examples, commands with "#" prompt are executed as root user,commands with "$" prompt can be executed as clusterware owner.
        1. ****Component level logging****
        # crsctl debug log css [module:level]{,module:level} ...
          - Turns on debugging for CSS
        # crsctl debug log crs [module:level]{,module:level} ...
          - Turns on debugging for CRS
        # crsctl debug log evm [module:level]{,module:level} ...
          - Turns on debugging for EVM
        For example:
        # crsctl debug log crs "CRSRTI:1,CRSCOMM:2"
        # crsctl debug log evm "EVMD:1"

        # crsctl set {log|trace} {mdns|gpnp|css|crf|crs|ctss|evm|gipc} "<name1>=<lvl1>,..."
        Set the log/trace levels for specific modules within daemons
        For example:
        # crsctl set log crs "CRSRTI=2,CRSCOMM=2"

        To list all modules:
        # crsctl lsmodules {css | crs | evm} - lists the CSS modules that can be used for debugging
        # crsctl lsmodules {mdns|gpnp|css|crf|crs|ctss|evm|gipc}
          mdns multicast Domain Name Server
          gpnp Grid Plug-n-Play Service
          css Cluster Synchronization Services
          crf Cluster Health Monitor
          crs Cluster Ready Services
          ctss Cluster Time Synchronization Service
          evm EventManager
          gipc Grid Interprocess Communications

        Logging level definition:
        level 0 = turn off
        level 2 = default
        level 3 = verbose
        level 4 = super verbose

        To check current logging level:
        For CSS:
        $ grep clssscSetDebugLevel <ocssd logs>
        For CRS / EVMD:
        $ grep "ENV Logging level for Module" <crsd / evmd logs>

        $ crsctl get log <modules> ALL
        For example:
        $ crsctl get log css ALL
        Get CSSD Module: CLSF  Log Level: 0
        Get CSSD Module: CSSD  Log Level: 2
        Get CSSD Module: GIPCCM  Log Level: 2
        Get CSSD Module: GIPCGM  Log Level: 2
        Get CSSD Module: GIPCNM  Log Level: 2
        Get CSSD Module: GPNP  Log Level: 1
        Get CSSD Module: OLR  Log Level: 0
        Get CSSD Module: SKGFD  Log Level: 0

        2. ****Component level debugging****
           Debugging can be turned on for CRS and EVM and their specific modules by setting environment variables or through crsctl.
        To turn on tracing for all modules:
        To turn on tracing for a specific sub module:

        3. ****CRS stack startup and shutdown****
           Using crsctl, the entire CRS stack and the resources can be started and stopped.
        # crsctl start crs
        # crsctl stop crs

        4. ****Diagnostics collection script -****
        This script is for collecting diagnostic information from a CRS installation. The diagnostics are necessary for development
        to be able to help with SRs, bugs and other problems that may arise in the field.
        # <$CRS_HOME>/bin/
        # <$CRS_HOME>/bin/ -crshome=$ORA_CRS_HOME --collect
        # <$GRID_HOME>/bin/
        For more details, please refer to Document 330358.1 CRS 10gR2/ 11gR1/ 11gR2 Diagnostic Collection Guide

        5. ****Unified log directory structure****
           The following log directory structure is in place for 10.2 onwards in an effort to consolidate the log files of different Clusterware
        components for easy diag information retrieval and problem analysis.
           1. Need one place where all the CRS related log files can be located.
           2. The directory structure needs to be intuitive for any user
           3. Permission, ownership issues need to be addressed
           4. Disk space issues need to be considered
            Alert file
            CRS component directories
            GI components directories:
        Log files of the respective components go to the above directories. Core files are dropped in the same directory as the logs.
        Old core files will be backed up.
        6. ****CRS and EVM Alerts****
        CRS and EVM post alert messages on the occurrence of important events.
        [NORMAL] CLSD-1201: CRSD started on host %s.
        [ERROR] CLSD-1202: CRSD aborted on host %s. Error [%s]. Details in %s.
        [ERROR] CLSD-1203: Failover failed for the CRS resource %s. Details in %s.
        [NORMAL] CLSD-1204: Recovering CRS resources for host %s
        [ERROR] CLSD-1205: Auto-start failed for the CRS resource %s. Details in %s.
        [NORMAL] CLSD-1401: EVMD started on node %s
        [ERROR] CLSD-1402: EVMD aborted on node %s. Error [%s]. Details in %s.
        [ERROR] CLSD-1402: EVMD aborted on node %s. Error [%s]. Details in %s.

        7. ****Resource debugging****
           Using crsctl, debugging can be turned on for resources.
        # crsctl debug log res <resname:level>
        - Turns on debugging for resources
        For example:
        # crsctl debug log res
        # crsctl set log res <resname>=<lvl> [-init]
        Set the log levels for resources including init resource, check crsctl stat res -t and crsctl stat res -t -init for resource name.
        For example:
        # crsctl set log res
        # crsctl set log res ora.cssdmonitor=2 -init
         This sets debugging on for the resource in the form of an OCR key.

        8. ****Health Check****
        To determine the health of the CRS stack:
        $ crsctl check crs
        To determine health of individual daemons:
        $ crsctl check css
        $ crsctl check evm
        or 11gR2:
        $ crsctl stat res -t -init
        9. ****OCR debugging****
        In 10.1, to debug OCR, the $ORA_CRS_HOME/srvm/ocrlog.ini was updated to a level higher than 0. Starting with 10.2 it is possible to
        debug the OCR at the component level using the following commands
        a)   Edit $ORA_CRS_HOME/srvm/admin/ocrlog.ini to the component level
        Eg: comploglvl="OCRAPI:5;OCRCLI:5;OCRSRV:5;OCRMAS:5;OCRCAC:5"

        b) Use the dynamic feature to update the logging into the OCR itself using the command crsctl
        Same as step one, the CRS modules name associated with OCR are:


AODU> rac eviction  --RAC重启、节点驱逐诊断信息收集
        ****Data Gathering for Node Reboot/Eviction****
        Provide files in Section "Data Gathering for All Oracle Clusterware Issues" and the followings:
        Approximate date and time of the reboot, and the hostname of the rebooted node
        OSWatcher archives which cover the reboot time at an interval of 20 seconds with private network monitoring configured.
        Note 301137.1 - OS Watcher User Guide
        Note.433472.1 - OS Watcher For Windows (OSWFW) User Guide
        For pre-11.2, zip of /var/opt/oracle/oprocd/* or /etc/oracle/oprocd/*
        For pre-11.2, OS logs - refer to Section Appendix B
        For 11gR2+, zip of /etc/oracle/lastgasp/* or /var/opt/oracle/lastgasp/*
        CHM/OS data that covers the reboot time for platforms where it is available,
        refer to Note 1328466.1 for section "How do I collect the Cluster Health Monitor data"
        The Cluster Health Monitor is integrated part of Oracle Grid Infrastructure for Linux (not on Linux Itanium) and
        Solaris (Sparc 64 and x86-64 only), so installing Oracle Grid Infrastructure on those platforms will automatically
        install the Cluster Health Monitor. AIX will have the Cluster Health Monitor starting from
        The Cluster Health Monitor is also enabled for Windows (except Windows Itanium) in
        ora.crf is the Cluster Health Monitor resource name that ohasd manages.Issue "crsctl stat res ▒Ct ▒Cinit" to check the current status
        of the Cluster Health Monitor.
        For example, issue "<GI_HOME>/bin/ --collect --crshome $ORA_CRS_HOME --chmos
        --incidenttime <start time of interesting time period> --incidentduration 05:00"

        What logs and data should I gather before logging a SR for the Cluster Health Monitor error?
        1) provide 3-4 pstack outputs over a minute for osysmond.bin
        2) output of strace -v for osysmond.bin about 2 minutes.
        3) strace -cp <osysmond.bin pid> for about 2 min
        4) oclumon dumpnodeview -v output for that node for 2 min.
        5) output of "uname -a"
        6) outpuft of "ps -eLf|grep osysmond.bin"
        7) The ologgerd and sysmond log files in the CRS_HOME/log/<host name> directory from all nodes
        How to start and stop CHM that is installed as a part of GI in 11.2 and higher?
        The ora.crf resource in 11.2 GI (and higher) is the resource for CHM, and the ora.crf resource is managed by ohasd.
        Starting and stopping ora.crf resource starts and stops CHM.
        To stop CHM (or ora.crf resource managed by ohasd)
        $GRID_HOME/bin/crsctl stop res ora.crf -init
        To start CHM (or ora.crf resource managed by ohasd)
        $GRID_HOME/bin/crsctl start res ora.crf -init
        If vendor clusterware is being used, upload the vendor clusterware logs



aodu(At Oracle Database Utility)之optim
aodu(At Oracle Database Utility)之asm(二)
aodu(At Oracle Database Utility)之asm(一)
aodu(At Oracle Database Utility)之rac(二)
aodu(At Oracle Database Utility)之rac(一)
aodu(At Oracle Database Utility)之ora600
aodu(At Oracle Database Utility)之asmdisk
aodu(At Oracle Database Utility)之unwrap
aodu(At Oracle Database Utility)之rdba
aodu(At Oracle Database Utility)之drux
aodu(At Oracle Database Utility)之time
aodu(At Oracle Database Utility)之odlog




    微  信: ohsdba





My Oracle Support

Exadata 12.1 Document

Exadata 12.2 Document

Oracle 11gR2 Document

Oracle 12cR1 Document

Oracle 12cR2 Document

GoldenGate 12c

Oracle Fusion Middleware

Mysql Document





QQ 375349564