OPATCH USERS GUIDE Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved. INTRODUCTION ------------ OPatch is a java based utility that allow the application and rolling back of interim patches to an Oracle product. The program has sub-commands that may take arguments. The program requires Java(JVM) to be present on the current system. Every ORACLE_HOME will by default have the Java in it, if not please download it from http://java.sun.com. Help is available for all commands. Recommended Java version for OPatch is Java 1.5.0 or higher. OPatch expects users to have commands such as fuser, ar and make on Unix platforms and jar on Windows platforms to be available in their PATH. ENVIRONMENTAL VARIABLES ----------------------- ORACLE_HOME: This variable must point to a valid Oracle Home and matches the value when that Oracle Home was installed. OPATCH_DEBUG: Setting this variable to "true" sets the log level to debug. OPATCH_PLATFORM_ID: OPatch has its own mechanism of finding the platform ID. If this variable to set to some numeric value, then OPatch skips its mechanism and assumes the platform ID to be the one specified here. OPATCH_NO_FUSER: Setting this variable to "true" informs the OPatch to skip the check for active executables. REAL APPLICATION CLUSTERS ------------------------- When patching a RAC installation with either a rolling patch or minimum downtime the user is prompted for which nodes to patch during the patch installation process. A rolling RAC patch is identified inside the patch and cannot be enabled when invoking the patch tool. More details are in the FAQ that should be in the same directory as this file. When patching a RAC on certain platforms, it is important that the LD_LIBRARY_PATH must be set up correctly. Please see the FAQ for details. LIMITATIONS ----------- The current version of OPatch has some limitions that will be resolved in the future. These limitations are: 1. Shared ORACLE_HOMEs (clustered file systems): Rolling RAC and minimum down options are not supported and should not be used on RAC clusers with a shared ORACLE_HOME. This is normally seen when CFS (clustered file systems) are used for the installation. Patching this type of installtion should be done on a single machine. Any instance that uses the ORACLE_HOME on a CFS that is to be patched should be stopped on all nodes before patching the installation. 2. Simultaneous installation of new instances and patching: Currently installing a new instance while patching an existing installation is not supported. Inventory corruption may result from this activity. GENERAL USAGE ------------- $ 'opatch apply ...' is to apply a patch $ 'opatch rollback ...' is to roll back a patch $ 'opatch lsinventory' is to show inventory on an Oracle Home $ 'opatch query ...' is to query a given patch from specific details $ 'opatch version' is to show the version of the OPatch being used $ 'opatch prereq ...' is to invoke some prerequiste checks $ 'opatch util ...' is to invoke some independent utilities SYNOPSIS -------- Invoking OPatch without arguments or with the 'help' sub-command returns the list of valid sub-commands to the user: Usage: opatch [ -help ] [ -r[eport] ] [ command ] command := apply lsinventory napply nrollback rollback query version prereq util := -help Displays the help message for the command. -report Print the actions without executing. example: 'opatch -help' 'opatch -help -fmw' 'opatch auto -help' 'opatch apply -help' 'opatch lsinventory -help' 'opatch napply -help' 'opatch nrollback -help' 'opatch rollback -help' 'opatch prereq -help' 'opatch util -help' APPLY ----- The apply command is used to install a interim patch. $ opatch apply -help DESCRIPTION This operation applies an interim patch to an ORACLE_HOME from the current directory. The patch location can specified using the parameter patch_location. SYNTAX opatch apply [-connectString ] [-delay ] [ -force ] [ -force_conflict ] [-init [-opatch_init_end] ] [-invPtrLoc ] [-jre ] [-local ] [-local_node ] [-minimize_downtime ] [-no_bug_superset ] [-no_inventory ] [-no_relink] [-no_sysmod] [-ocmrf ] [-oh ] [-post [-opatch_post_end] ] [-pre [-opatch_pre_end] ] [-profile_mask ] [-property_file ] [-ptlConnect ] [-ptlPassword ] [-ptlSchema ] [-remote_nodes ] [-retry ] [-verbose ] [ ] OPTIONS -all_nodes Apply the patch using all-node mode. -connectString This option can be used to specify the list of database instances on which the patch needs to be applied. The value for this option is specified as per the following syntax "SID:User:Passwd:Node". The SID is a must, others can be ignored, OPatch takes default values for it. Ex: oracle:dba:dba:mymachine,oracle1::: NOTE: If the system is not part of RAC setup, then to patch just the local node, provide the node name as empty string. -delay If -retry is specified, this options tells OPatch how many seconds it should wait before attempting to lock inventory again in case of a previous failure. -force This option is meaningful with 'silent' option and is used to override the behavior of silent under specific circumstances. -force_conflict If a conflict exist which prevents the patch from being applied, the -force_conflict flag can be used to apply the patch. OPatch will remove all the conflicting patches before applying the current patch. This will override the 'silent' behavior for conflicts and hence is meaningful only when used with 'silent' option. -init This option is used to pass parameters to the init script. This script is executed before prerequisite checks are run. The value for this option have to be enclosed in double quotes. This option should be ended by option 'opatch_init_end'. -invPtrLoc Used to locate the oraInst.loc file. Needed when the installation used the -invPtrLoc flag. This should be the path to the oraInst.loc file -jre This option tells OPatch to use JRE (java) from the specified location instead of the default location under Oracle Home. -local Patch the local node, then update inventory of the local node. Do not propagate the patch or inventory update to other nodes. -local_node This option can be used to specify to OPatch the local node name to be used for RAC mode application of the patch -minimize_downtime Only applied to RAC instances. User supplies the order of nodes to be patched. This option cannot be used in conjunction with -local option with a rolling patch. -no_bug_superset This option tells OPatch to error out if the current patch's bugs-to-fix is a super-set (or same set) of an installed patch's bugs-fixed in the Oracle Home. -no_inventory This option specifies not to update the inventory, it just updates the files in the system. If inventory is readable, then OPatch tries to read the inventory. -no_relink The nolink option does not perform any make operations. This option can be used during multiple patch applications and perform the linking step only once. OPatch does not keep track of the make operations it did not perform. It is the job of the OPatch users to make sure the execute OPatch without this option at the end, so that compilation happens once -no_sysmod This option specifies not to update the files in the system. It just updates the inventory. It also will not execute the pre and post scripts -ocmrf Give OPatch the absolute path to the OCM response file to be used for OCM configuration. -silent must be used in conjunction with -ocmrf if Oracle Home doesn't have OCM installed and configured. -oh The oracle home to work on. This takes precedence over the environment variable ORACLE_HOME -opatch_init_end This option is used to mark the end of init options. Without this option everything after init till end of the command is passed into init. This option is meaningless without -init option. -opatch_post_end This option is used to mark the end of post options. Without this option everything after post till end of the command is passed into post. This option is meaningless without -post option. -opatch_pre_end This option is used to mark the end of pre options. Without this option everything after pre till end of the command is passed into pre. This option is meaningless without -pre option. -post This option is used to pass parameters to the post script. This script is executed after application of the patch. The value for this option have to be enclosed in double quotes. This option should be ended by option 'opatch_post_end'. -pre This option is used to pass parameters to the pre script. This script is executed before application of the patch. The value for this option have to be enclosed in double quotes. This option should be ended by option 'opatch_pre_end'. -profile_mask If the patch to be applied specifies WLS patch/patchset as prerequisites, OPatch will read the WLS default patch profile. To have OPatch read non-default patch profile, specify the patch profile name with this option. -property_file The user defined property file for OPatch to use. The path to the property file should be absolute. This property file takes precedence over the one that is supplied with OPatch -ptlConnect This option can be used to specify the connect string of the portal schema -ptlPassword This option can be used to specify the password of the portal schema -ptlSchema This option can be used to specify the schema of the portal repository patch -remote_nodes This option can be used to specify to OPatch the list of remote nodes to be used for RAC mode application of the patch. The node names are comma separated and without spaces -report Just prints out the actions without executing -retry This option tells OPatch how many times it should retry in case of an inventory lock failure. -runSql This option can be used to specify OPatch to run the sql scripts and sql procedures if they are present in the given patch. -silent This suppresses any user-interaction -sqlScript This option can be used to specify the custom sql script to be run by OPatch after patching is completed -verbose This option prints more OPatch output to the screen as well as to the log file PARAMETERS Patch Location Path to the location for the patch. If the patch location is not specified, then the current directory is taken as the patch location. LSINVENTORY ----------- This command reports what's installed on the current system. $ opatch lsinventory -help CRIPTION List the inventory for a particular $ORACLE_HOME or display all installations that can be found. SYNTAX opatch lsinventory [-all ] [-all_nodes] [-bugs_fixed ] [-delay ] [-detail ] [-group_by_date ] [-invPtrLoc ] [-jre ] [-local ] [-oh ] [-patch ] [-patch_id ] [-ptlConnect ] [-ptlPassword ] [-ptlSchema ] [-property_file ] [-retry ] OPTIONS -all Report the name and installation directory for each $ORACLE_HOME found. -all_nodes Reports the patches installed on the given Oracle Home in all nodes of the RAC system. It also prints the oracle binary's size and checksum on all nodes. This option cannot be used in conjunction with -all, -detail, -patch. -bugs_fixed Reports bugs fixed by installed patches in a tabular format. Besides the bugs fixed, the report also displays the installed patches, installed times, and bug descriptions. The fixed bugs are sorted per installed patch. Default display is patches in descending order based on installed time and ascending order of bugs within each patch. You can use 'asc' (or) 'desc' with this option to enforce sort order on bugs within each patch. You can use this option with the patch (or) patch_id option to obtain sort orders with installed patches. -delay If -retry is specified, this option tells OPatch how many seconds it should wait before attempting to lock inventory again in case of a previous failure. -detail(s) Report installed products and other details. This option cannot be used in conjunction with all option. -group_by_date This option tells OPatch to group all installed patches by the date they were installed in the Oracle Home -invPtrLoc Used to locate the oraInst.loc file. When the installation used the invPtrLoc flag. This should be the path to the oraInst.loc file -jre This option tells OPatch to use JRE (java) from the specified location instead of the default location under Oracle Home -local This option tells OPatch to display inventory information of the local node only -oh The oracle home to work on. This takes precedence over the environment variable ORACLE_HOME -patch Lists the patch IDs installed in the Oracle home in ascending (asc) or descending (desc) order, which is the default, based on installed time. -patch_id Lists the patch IDs installed in the oracle home in ascending (asc) or descending (desc) order based on patch numbers. The value defaults to ascending (asc). -ptlConnect This option is used to specify the connect string credentials of the portal schema -ptlPassword This option is used to specify the password of the portal schema -ptlSchema This option is used to specify the schema of the portal repository -property_file The user defined property file for OPatch to use. The path to the property file should be absolute. This property file takes precedence over the one that is supplied with OPatch -retry This option tells OPatch how many times it should retry in case of an inventory lock failure PARAMETERS This operation does not have any required parameters. ROLLBACK -------- The rollback command removes a interim patch from the appropriate $ORACLE_HOME. $ opatch rollback -help DESCRIPTION Rollback an existing one-off patch indicated by the reference-id. SYNTAX opatch rollback -id [-connectString ] [-delay ] [-init [-opatch_init_end] ] [-invPtrLoc ] [-jre ] [-local] [-local_node ] [-no_inventory] [-no_relink] [-no_sysmod] [-oh ] [-ph ] [-post [ -opatch_post_end] ] [-pre [-opatch_pre_end] ] [-property_file ] [-ptlConnect ] [-ptlPassword ] [-ptlSchema ] [-remote_nodes ] [-retry ] [-runSql] [-silent] [-sqlScript ] [-verbose] OPTIONS -all_nodes Rollback the patch using all-nodes mode. -connectString This option can be used to specify the list of database instances on which the patch needs to be applied. The value for this option is specified as per the following syntax "SID:User:Passwd:Node". The SID is a must, others can be ignored, OPatch takes default values for it. Ex: oracle:dba:dba:mymachine,oracle1::: NOTE: If the system is not part of RAC setup, then to patch just the local node, provide the node name as empty string. -delay If -retry is specified, this option tells OPatch how many seconds it should wait before attempting to lock inventory again in case of a previous failure. -id Use 'lsinventory' option to display all patch id's. Each one-off patch is indicated by its id. To rollback a patch the id for that patch must be supplied -init This option is used to pass parameters to the init script. This script is executed before prerequisite checks are run. The value for this option have to be enclosed in double quotes. This option should be ended by option 'opatch_init_end'. -invPtrLoc Used to locate the oraInst.loc file. Needed when the installation used the invPtrLoc flag. This should be the path to the oraInst.loc file -jre This option tells OPatch to use JRE (java) from the specified location instead of the default location under Oracle Home. -local Roll back then update inventory of the local node. Do not propagate the patch or inventory update to other nodes -local_node This option can be used to specify to OPatch the local node name to be used for RAC mode rollback of the patch -no_inventory This option specifies not to update the inventory, it just updates the files in the system. If inventory is readable, then OPatch tries to read the inventory. NOTE: ph has to be used to specify the patch location. -no_relink Do not perform the make operations in the patch. This option can be used during multiple patch removals and perform the compilation step only once -no_sysmod This option specifies not to update the files in the system. It just updates the inventory. It also will not execute the pre and post scripts -oh The oracle home to work on. This takes precedence over the environment variable ORACLE_HOME -opatch_init_end This option is used to mark the end of init options. Without this option everything after init till end of the command is passed into init. This option is meaningless without -init option. -opatch_post_end This option is used to mark the end of post options. Without this option everything after post till end of the command is passed into post. This option is meaningless without -post option. -opatch_pre_end This option is used to mark the end of pre options. Without this option everything after pre till end of the command is passed into pre. This option is meaningless without -pre option. -ph Specify the valid patch directory area. Rollback will use the command types found in the patch directory to identify what commands are to be used for the current operating system -post This option is used to pass parameters to the post script. This script is executed after removal of the patch. The value for this option have to be enclosed in double quotes. This option should be ended by option 'opatch_post_end'. -pre This option is used to pass parameters to the pre script. This script is executed before removal of the patch. The value for this option have to be enclosed in double quotes. This option should be ended by option 'opatch_pre_end'. -property_file The user defined property file for OPatch to use. The path to the property file should be absolute. This property file takes precedence over the one that is supplied with OPatch -ptlConnect This option can be used to specify the connect string of the portal schema -ptlPassword This option can be used to specify the password of the portal schema -ptlSchema This option can be used to specify the schema of the portal repository patch -remote_nodes This option can be used to specify to OPatch the list of remote nodes to be used for RAC mode rollback of the patch. The node names are comma separated and without spaces -report Just prints out the actions, without executing -retry This option tells OPatch how many times should retry in case of an inventory lock failure. -runSql This option can be used to specify OPatch to run the sql scripts and sql procedures if they are present in the given patch. -silent This suppresses any user-interaction and defaults any Y|N question to Y. This option is not supported on RAC yet -sqlScript This option can be used to specify the custom sql script to be run by OPatch after patching is completed -verbose This option prints more OPatch output to the screen as well as to the log file PARAMETERS Patch ID of the patch to be rolled back unless the command is invoked within a patch area. QUERY ----- This command queries a given patch for specific details. $ opatch query -help DESCRIPTION This operation loads and prints out various information about a patch. Parameter patch_location points to where a patch is located. For example, if you specify "path", there should be sub-dir. "etc/config" under "path". SYNTAX opatch query [-all] [-is_auto_patch] [-get_base_bugs] [-get_component] [-get_os] [-get_date] [-get_patch_type] [-get_product_family] [-get_update_components] [-has_sql] [-is_online_patch] [-is_patchset_update] [-is_portal_patch] [-is_rolling_patch] [-jre ] [-oh ] [ ] OPTIONS -all Get everything available about a patch. This is equivalent to setting all available options. -is_auto_patch This option says 'true' if the patch is auto-enabled, 'false' otherwise. -get_base_bug Get bugs fixed by the patch -get_component Get components touched by the patch. -get_update_components Get all components to be upgraded in a Patchset Update. -get_date Get patch creation date and time. -get_os Get platforms for which this patch could be applied. -get_patch_type Get the patch type as specified in the patch metadata. -get_product_family Get the product family as specified in the patch metadata. -has_sql This option says 'true' if the patch has sql related actions, 'false' otherwise. -is_online_patch This option says 'true' if the patch is a online patch, 'false' otherwise. -is_patchset_update This option says 'true' if the patch is a Patchset Update, 'false' otherwise. -is_portal_patch This option says 'true' if the patch has portal actions, 'false' otherwise. -is_rolling_patch This option says 'true' if the patch is a rolling patch, 'false' otherwise. -jre This option tells OPatch to use JRE (java) from the specified location instead of the default location under Oracle Home. -oh The oracle home to work on. This takes precedence over the environment variable ORACLE_HOME. PARAMETERS The location of the patch to be queried. If the patch location is not specified, then the current directory is taken as the patch location. PREREQ ------ This command runs a given prerequiste check. $ opatch prereq -help DESCRIPTION This operation runs the prerequisite checks on an ORACLE_HOME. SYNTAX opatch prereq [-id ] [-invPtrLoc ] [-jre ] [-local_node ] [-oh ] [-ph ] [-phBaseDir ] [-phBaseFile ] [-property_file ] [-remote_nodes ] [-sid ] [-connectString ] COMMANDS CheckActiveFilesAndExecutables Check if there are any file(s) that are active, which are touched by the patch to be applied or rolled back. CheckActiveServices Check for the services that are active. Note: Applicable for Windows platforms only. CheckApplicable Check for the presence of the required components in the ORACLE_HOME and check if all the actions of the given patch(es) are applicable. CheckApplicableProduct Check if the patch is applicable for the given Oracle Home. If the patch is marked for stand-alone homes, then it can not be applied on normal oui-based home and vice versa. Also, a patch can be marked as a hybrid patch, where it is applicable for both homes. CheckCentralInventoryForOH Check if the given ORACLE_HOME is registered in the central inventory specified by the oraInst.loc file. CheckCentralInventoryForRWSession Check if a RW (read-write) session can be created for the given central inventory. CheckCentralInventoryLocation Validate the Central Inventory location. Check if it has the correct directory structure and has the inventory.xml with read permissions. CheckConflictAgainstOH Check if there are any conflicts between the patches to be applied and the patches in the OH. CheckConflictAgainstOHWithDetail Check if there are any conflicts between the patches to be applied and the patches in the OH, by giving out the detailed information about the conflicts/supersets. CheckConflictAmongPatches Check if there are any conflicts among the patches to be applied. CheckConflictAmongPatchesWithDetail Check if there are any conflicts among the patches to be applied, by giving out the detailed information about the conflicts/supersets. CheckForCRSHomeIfRAC Check if the given Central Inventory has a CRS Home registered with it, if it is a RAC setup. CheckIfOHLockedForPatching Check if the ORACLE_HOME is locked for patching by any previous unsuccessful OPatch Session. CheckInstalledOneOffs Check if all the patches provided by the user to rollback are present in the given Oracle Home. CheckOracleHome Check if the given ORACLE_HOME is valid. Check if it has the inventory files with proper permissions. CheckOraInstLocation Check if the oraInst.loc file is proper and has the read permissions. CheckOUILocation Check the ORACLE_HOME for the presence of OUI. CheckOUIVersionCompatible Check if the OUI in the ORACLE_HOME is compatible for the OPatch. CheckPatchApplicableOnCurrentPlatform Check if the given patch(es) is applicable on the current platform. CheckPatchApplyDependents Check if all the patches required by the patch currently being installed is present in the Oracle Home or not. CheckPatchRollbackDependents Check if there are any patches in the Oracle Home that are depending on the patch being currently rolled back. CheckPatchShipHome Check if the given patch to be applied has the proper structure and has the correct permissions for all the files. CheckRACNodeList Check if all the nodes in the RAC setup are valid, up and reachable. CheckRemoteCommandInvocable Check if commands can be invoked on the remote machines. CheckRemoteCopyAndRemove Check if files can be copied to and removed from the remote machines. CheckRequiredLibs Check if all the required OUI libraries are present in the given ORACLE_HOME. CheckRollbackable Check if the given patch(es) can be rollbacked from the ORACLE_HOME. CheckRollbackSid Check if the given patch(es) can be rollbacked from the ORACLE_HOME using the given SIDs. CheckStandAloneHome Check if the Oracle Home is a stand-alone home or a normal oui-based home. CheckSystemCommandAvailable Check if all the commands required for applying or rolling back the given patch are present in the system. CheckSystemSpace Check if enough system space is available for the patch(es) to be applied. CheckUserAdminPrivilege Check is the user is 'root'. Note: OPatch should not be invoked by 'root', if so then this check fails. CheckPatchsetUpdateCompatible Check if the given patches and the OUI together are Patch Set Update compatible. CheckPatchsetUpdateComponents Check if the components to be upgraded in the Patch Set Update are valid. OPTIONS -connectString This option can be used to specify the list of database instances and remote nodes. The value for this option is specified as per the following syntax "SID:User:Passwd:Node". The SID is a must, others can be ignored, OPatch takes default values for it. Ex: oracle:dba:dba:mymachine,oracle1::: NOTE: If the system is not part of RAC setup, then to specify just the local node, provide the node name as empty string. This option cannot be used along with 'sid' option. -id This option can be used to specify the patch IDs of all the patches that are to be rollbacked from the given OH. -invPtrLoc Used to locate the oraInst.loc file. Needed when the installation used the -invPtrLoc flag. This should be the path to the oraInst.loc file. -jre This option tells OPatch to use JRE (java) from the specified location instead of the default location under Oracle Home. -local_node This option can be used to specify to OPatch the local node name to be used for RAC mode application of the patch. -oh The oracle home to work on. This takes precedence over the environment variable ORACLE_HOME. -ph This option can be used to specify the path to the patch location. Ex: /tmp/101010 -phBaseDir This option can be used to specify the path to base dir where all the patches to be applied are kept. Note: The directory should contain only non-duplicate patches in zipped or unzipped format. -phBaseFile This option can be used to specify complete path to the file containing the location of the patches to be applied. -property_file The user defined property file for OPatch to use. The path to the property file should be absolute. This property file takes precedence over the one that is supplied with OPatch. -remote_nodes This option can be used to specify to OPatch the list of remote nodes to be used for RAC mode application of the patch. The node names are comma separated and without spaces. -sid This option can be used to specify the SIDs of the database instances. This option can be used only for local system operations. UTIL ---- This command runs a given utility to do a specific task. $ opatch util -help DESCRIPTION This operation invokes the chosen utilities on an ORACLE_HOME. SYNTAX opatch util [ -help ] [ COMMAND ] Use opatch util [ COMMAND ] -help to get help on a particular command. COMMANDS ApplySql Apply sql, sql procedure related actions for patches, which are already applied using Apply or NApply operation. Cleanup Remove the backup for restore area of the given patch or for all the patches. CopyListedDirs Recursively copy all directories listed in ORACLE_HOME/.patch_storage//rac/copy_dirs.txt to remote nodes. CopyListedDirsTest Recursively copy a single directory to remote nodes. CopyListedFiles Copy all files listed in ORACLE_HOME/.patch_storage//rac/copy_files.txt to remote nodes. CopyListedFilesTest Copy a single file to remote nodes. DisableOnlinePatch Disable and remove the specified online patch(es) on the given database instances. EnableOnlinePatch Install and enable the specified online patch(es) on the given database instances. InstallOCM Install and configure OCM. LoadXML Prompt for path/name of the XML file, then check if the XML is correct. RemoveListedDirs Recursively remote directories listed in ORACLE_HOME/.patch_storage//rac/remove_dirs.txt on remote nodes. RemoveListedDirsTest Recursively remove a directory on remote nodes. RemoveListedFiles Remove files listed in ORACLE_HOME/.patch_storage//rac/remove_files.txt on remote nodes. RemoveListedFilesTest Remove a single file on remote nodes. RestoreOH Restore the OracleHome to the previous state. RollbackSql Rollback sql, sql procedure related actions for patches, which are already rolled back using Rollback or NRollback operation. RunAnyCommand Run any command on remote nodes. RunLocalMake Invoke re-link on the local node. Commands are stored in ORACLE_HOME/.patch_storage//make.txt. RunRemoteMake Invoke re-link on remote nodes. Commands are stored in ORACLE_HOME/.patch_storage//rac/make_cmds.txt. UpdateRemoteNodes Propagate/Remove files/directories to/on remote nodes using files under ORACLE_HOME/.patch_storage//rac/*. UpdateOPatchVersion Update the version of OPatch in the inventory of Oracle Home. Verify Using the defined ORACLE_HOME and given patch location via -ph, the program will check to make sure the patch was applied to the ORACLE_HOME. Example: 'opatch util verify -ph /tmp/patchLoc' VERSION ------- This command prints the current version number of the patch tool. 'opatch version' In addition, OPatch has capability to apply 'n' patches, rollback 'n' patches. Please refer 'opatch napply -help', 'opatch nrollback -help' for details. ========================================================================== 10/28/09