* IBM_PROLOG_BEGIN_TAG * This is an automatically generated prolog. * * perf720 src/bos/usr/lpp/bos.perf/README.perf.tools 1.3.1.4 * * * * OBJECT CODE ONLY SOURCE MATERIALS * * COPYRIGHT International Business Machines Corp. 2002,2012 * All Rights Reserved * * The source code for this program is not published or otherwise * divested of its trade secrets, irrespective of what has been * deposited with the U.S. Copyright Office. * * IBM_PROLOG_END_TAG * * COMPONENT_NAME: (CMDPERFT) AIX Performance Analysis Tools * * FUNCTIONS: README * * ORIGINS: 30 * * (C) COPYRIGHT International Business Machines Corp. 2002 * * All Rights Reserved. * Licensed Materials - Property of IBM * * US Government Users Restricted Rights - Use, duplication or * disclosure restricted by GSA ADP Schedule Contract with IBM Corp. * Performance Management for Power Systems (PM for Power Systems) ------------------------------- PM for Power Systems (PM) helps automate the collection, archival and analysis of system performance data and returns capacity and performance reports and graphs that can help customers plan for and manage their system resources and capacity. PM collects system utilization, performance statistics and hardware configuration information. Customers are entitled to use this function to receive summary level reports for no additional charge providing the system is under warranty or covered by an IBM Hardware Maintenance Agreement. Additional 'detail' level reports are available as part of an IBM Global Technology Services fee offering. PM uses 'topasrec' persistent binary recording to collect the performance data. So, 'topasrec' persistent binary recording should always be enabled for PM Service to collect performance data. PM for Power Systems support for Shared Processor Pool (SPP) graphs and ----------------------------------------------------------------------- Cross Lpar Utilization (XLpar) on AIX --------------------------------------- PM for Power Systems support for Shared Processor Pool (SPP) graphs and total System View (XLPAR or Cross LPAR) on AIX To enable support for SPP and XLPAR on AIX v6.1 TL 08 and AIX V7.1 TL 02, it is necessary that "bos.perf.tools" fileset exist. If using HMC, follow the instruction given below in the respective Instruction Section. If using FSM, you must download and install an ifix. This ifix will be superseded by a Service Pack in 1Q '13. You will find complete instructions below in the respective Instruction Section. (NOTE: Prior AIX TL's and releases are not supported via the 'bos.perf tools' fileset. Selected release TL's are supported however via an ifix.) Enabling SPP & XLPAR will enable this partition to remotely execute commands on the HMC/FSM using SSH for gathering configuration & performance data of processor pool & other partitions. For HMC Managed Servers: ======================= If using HMC , then use 'bos.perf.tools' fileset. Specific Instruction / Requirements: ----------------------------------- To enable SPP & XLPAR, the following pre-requirements have to be configured: * POWER Hardware must be either POWER 6 or higher. * Required HMC Version 7 Release 7.3.0 or above. * Setup password-less ssh connection to HMC system using below steps - login as root - cd to home directory of the logged in user (home directory as per /etc/passwd) - cd .ssh/ - ssh-keygen -t rsa For below prompts, just hit enter key, - Enter file in which to save the key - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - scp hmc_user@hmc_name:.ssh/authorized_keys2 temp_hmc - cat id_rsa.pub >> temp_hmc - scp temp_hmc hmc_user@hmc_name:.ssh/authorized_keys2 Test it by running below command - ssh hmc_user@hmc_name date * Enable Utilization Data recording On web-based HMC tool: - log on to HMC, - click on System Management > Servers - Check the box next to the server name and select the following from the menu: - Operations > Utilization Data > Change Sampling Rate - On the Change Sampling Rate window, set the sampling rate as 5 minutes. * HMC Information: Customers must select a new TOPAS option. From the command prompt, type: 'smit topas' --> Setup Performance Management --> Change/Show HMC Information and enter HMC name and HMC user name.' * Cross LPAR Interval: From the command prompt, type: 'smit topas' --> Setup Performance Management --> Change/Show Cross LPAR Interval, press F4 ( or Esc 4 ) to get the list of options --> Select the option (15 or 30 or 60) and press Enter to set the Cross LPAR Interval. (Default is 60.) * FileSystem containing folder /var/perf/pm/daily must have more than 60 MB of free space. * Only LPARs with the "Allow performance information collection" enabled are allowed to collect data To enable/disable this, On web-based HMC tool: - log on to HMC, - click on system management > Servers > "server name" - Check box next to LPAR name and click on >> arrows - Select properties > Hardware tab > Processor tab - Check or Uncheck box "Allow performance information collection" to enable or disable respectively. * SPP and XLPAR will not be supported in VIOS. * The PM for Power Systems PM-AIX agent collects data and transmits once per night. For FSM Managed Servers: ======================= DOWNLOAD AND INSTALL IFIX: ========================= FOUR STEPS FOR INSTALLING THE IFIX Below are the instructions on which ifix file to install on the server and then how to transfer and install the ifix on the AIX server. 1) Determine the appropriate release of AIX for which the SPP_XLPAR ifix package is needed ------------------------------------------------------------------------------------ Run the below command on the AIX system lslpp -l | grep bos.perf.tools The output will be similar as below bos.perf.tools 7.1.2.0 COMMITTED Base Performance Tools bos.perf.tools 7.1.2.0 COMMITTED Base Performance Tools Look for the version number after bos.perf.tools x.x.x.xx The below table gives the SPP_XLPAR package name based on the bos.perf.tools version. You will subsequently download and install the SPP_XLPAR package as per the table below ifix bos.perf.tools versions -------------------------------------------------------- SppXlpD61V 6.1.8.0 SppXlpD71H 7.1.2.0 2) Visit the Developer Works web site to access the ifix package to download ---------------------------------------------------------------------------- Below are the steps to download the ifix 1) Go to the link https://www.ibm.com/developerworks/mydeveloperworks/files/app using a web browser 2) Search for the tag pmaix using the public tag search box from the left hand side of the page, under Public tags 3) The search result will be displayed in the center of the page associated with the pmaix tag. 4) Click on the icon left of file name to download the file. Select the file identified by 'XXX' in step 1. 5) In the Save As dialog box, change the extension from .z to .Z and save the file. (Make note of the file name). 3) Transfer the ifix package (see file name above) to the AIX server -------------------------------------------------------------------- Either use some tool like winscp OR the windows command prompt to transfer the file to the AIX server. Steps to transfer using command prompt: Open command prompt and change directory (cd) to the directory where ifix file is located. FTP to AIX server using below steps - ftp servername - User: - Password: - bin - cd /tmp - put - quit The ifix file will be transfered to the AIX server under the /tmp location. 4) Install the ifix package (see file name above) on AIX server: ------------------------------- Once the file is downloaded, make sure there is no other ifix installed for SPP or iFix which will be locking /var/perf/pm/bin/pmcfg file by using the command emgr -l It will list all the installed ifix's. If any older iFix for SPP is installed, remove it first, if not required, by using emgr -rL "LABEL Name" Change the directory to /tmp where the ifix file is located. cd /tmp Install the downloaded ifix using below command emgr -e "filename" Based on whether the ifix is installed or not, it will display the status message. Additional Specific Instructions / Requirements for the FSM environment: ------------------------------------------------------------------------ To enable SPP & XLPAR, the following pre-requirements have to be configured: * POWER Hardware must be either POWER 6 or higher. * Required FSM Version 1 Release 0.1 or above. * Setup password-less ssh connection to FSM system using below steps - login as root - cd to home directory of the logged in user (home directory as per /etc/passwd) - cd .ssh/ - ssh-keygen -t rsa For below prompts, just hit enter key, - Enter file in which to save the key - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - scp fsm_user@fsm_name:.ssh/authorized_keys2 temp_fsm - cat id_rsa.pub >> temp_fsm - scp temp_fsm fsm_user@fsm_name:.ssh/authorized_keys2 Test it by running below command - ssh fsm_user@fsm_name date * Enable Utilization Data recording - log on to FSM - On Chassis Manager Click on server - Then click on "General Actions" > "Manage Power Systems Resources" - Select server then click on "Actions" or right click on the server - Then select Operations > Utilization Data > Change Sampling Rate - Change "Sampling rate:" to 5 minutes and click "Ok" * FSM Information: Customers must select a new TOPAS option. From the command prompt, type: 'smit topas' --> Setup Performance Management --> Change/Show HMC Information and enter FSM name and FSM user name. * Cross LPAR Interval: From the command prompt, type: 'smit topas' --> Setup Performance Management --> Change/Show Cross LPAR Interval, press F4 ( or Esc 4 ) to get the list of options --> Select the option (15 or 30 or 60) and press Enter to set the Cross LPAR Interval. (Default is 60.) * FileSystem containing folder /var/perf/pm/daily must have more than 60 MB of free space. * Only LPARs with the "Allow performance information collection" enabled are allowed to collect data To enable/disable this, - log on to FSM - On Chassis Manager Click on server - Then click on "General Actions" > "Manage Power Systems Resources" - On left hand side, Click on the server to see all the LPARs - Select Lpar then "Actions" or right click on the lpar > System configuration > Manage Virtual Server - Click on Processor Tab and check "Allow Performance Collection" Box * SPP and XLPAR will not be supported in VIOS. * The PM for Power Systems PM-AIX agent collects data and transmits once per night. OPTIONAL CONFIGURATION: ---------------------- * Enable SPP: To enable the SPP, Update the HMC/FSM Information following steps as provided above. * Disable SPP & XLPAR: To disable the SPP & XLPAR, Unset/Remove HMC/FSM information following steps as above and delete the HMC/FSM name and HMC/FSM user name. * Enable XLPAR: To enable XLPAR, Update the HMC/FSM Information and set the Cross Lpar Interval following the steps as provided above. * Disable XLPar: To disable XLPAR alone and to keep SPP enabled, unset the Cross LPAR Interval. To disable both SPP and XLPAR unset HMC/FSM Information and Cross LPAR Interval. ------------------------------------------------------------ What am I agreeing to upon PM for Power Systems activation? ------------------------------------------------------------ Data collection ---------------- PM for Power Systems uses performance and capacity information from the Power Systems platform. This service collects system utilization, performance information and hardware configuration information. Once the data is collected, PM scripts process the data and prepares it for transmission to IBM for future analysis and report generation. Data availability to IBM ------------------------ You agree that IBM may use and share the data collected by PM for Power Systems within the IBM enterprise without limitation, including for purposes of problem determination, of assisting you with performance and capacity planning, of maintaining your existing and new business relationships with IBM, of notifying you of existing or projected resource constraints, and to assist us to enhance IBM products. You also agree that your data may be transferred to such entities in any country whether or not a member of the European Union. Data availability to solution providers and Business Partners ------------------------------------------------------------- You may authorize IBM to share your data with various third parties, including one or more solution providers and Business Partners to make them aware of your performance and capacity demands and to enable them to provide you with a higher level of service. You may complete the authorization process online for your selected Solution Providers and/or Business Partners: PM Data Transmission --------------------- 1. If you have previously been using PM (pmaix.rte collector code) and upgrade to a new release of AIX 6.1 TL 3 or 4 or AIX 5.3 TL 9, 10, or 11, the new AIX PM collector code will install over the top of the old collector code but you must also go into the SMIT panel to reactivate the PM Data transmission on the partition. Otherwise, it is disabled. 2. In case of fresh OS installation, it will be enabled by default and can be disabled per the instructions below or at the IBM web site: http://www.ibm.com/systems/power/support/pm/ ====== FAQs ====== Q1. What releases of AIX are supported by PM? A1. AIX 5.3 TL 11 and AIX TL 4. Subsequent support for 5.3 TL 9, and 10 and AIX 6.1 TL 2, and 3 is planned. Q2. What else must I do to get PM working beyond having installed the TL that contains the bos.perf.tools fileset ? A2. 1) From a command line you must type 'smit'. 2) Select 'Performance and Resource Scheduling'. 3) Select 'Configure Topas Options' . 4) Select 'Performance Management'. 5) You must answer the 'Enable Data Transmission' option . 6) You must update the Customer Information option to ensure it includes email address. (The email address is needed, as a 'registration key' is emailed to you once your system or partition transmits PM data the first time. Instructions will be included on how the key is used to allow you to view your entitled performance and capacity graphs on the Internet). 7) Separately, Electronic Service Agent (ESA) installation or HMC configuration for transmission of ESA information is also required as a method to transmit the PM data to IBM. 8) Make sure in either the standalone installation of ESA at the system or partition or at the HMC configuration of ESA that the 'transmit performance data' option is checked to 'yes'. Note: PM uses 'topasrec' persistent binary recording to collect the performance data. So, 'topasrec' persistent binary recording should always be enabled for PM Service to collect performance data. Q3. What happened to the old pmaix.rte fileset, which was present prior to TL upgrading? A3. pmaix.rte filset will be automatically de-installed during the upgrading of TL and its functionality will be taken over by bos.perf.tools fileset. Before the upgrade it is advised to send all collected PM data to the IBM Q4. What will happen if a customer tries to install pmaix.rte? A4. An error message will be printed saying that pmaix.rte is superseded by the bos.perf.tools fileset. It will not allow pmaix.rte fileset to install. Q5. Why am I not able to collect SAN data? A5. PM service uses "/usr/sbin/datapath" command to collect SAN data. "/usr/sbin/datapath" command is shipped with devices.sdd.61.rte fileset. Please install SDD driver to get the required data. Q6. Do I need Electronic Service Agent to transfer PM Data? A6. Yes, PM Data can be transferred to IBM either by using ESA on your HMC or on your non-HMC controlled AIX hardware. Please configure ESA to send the PM data. Please refer to the IBM Infocenter for the manual steps to transmit the PM Data. Additionally, the ESA website provides information on the features of ESA: http://www.ibm.com/support/electronic Q7. Why is Electronic Service Agent not able to transmit the data? A7. Customer needs to enable the data transmission for ESA to start collecting the PM data. Please follow the instructions on IBM Infocenter to enable the PM data transmission. or visit http://www.ibm.com/support/electronic for more ESA information Q8: I am using a HMC and attempting to transmit the PM data using ESA but it is not working. A8: Check to make sure ServiceRM is running on the HMC For additional setup instructions: http://www.ibm.com/systems/power/ support/pm/ Q9: Is PM Service supported on nmon persistent recording ? A9: No, PM Service is currently supported only on topas persistent binary recording. However, topas nmon persistent recording can run in parallel to topas binary recording. Note: PM uses 'topasrec' persistent binary recording to collect the performance data. So, 'topasrec' persistent binary recording should always be enabled for PM Service to collect performance data. The following questions refer to SMIT panel options directly or indirectly. In addition to these questions, you can learn more about the SMIT panel at: http://publib.boulder.ibm.com/infocenter/pseries/ v5r3/index.jsp?topic=/com.ibm.aix.prftungd/doc/prftungd/ smit_topas_topasout.htm Q10. Why can't I 'Enable Data Transmission'? I get an error indicating 'Persistent local binary recording is not running.' A10. Confirm that you are not in 'single user mode'. If you are, restart your server into normal mode and try the 'Enable Data Transmission' option again. If you are not in single user mode, then see 'Q11 - How do I restart my performance recording?'. After the recording has started, you can then type 'smit topas', select 'Setup Performance Management', select 'Enable Data Transmission'. Q11. How do I restart my performance recording? A11. Confirm that there is not an active recording by typing 'smit topas', selecting 'List active Recordings', and then select 'persistent'. If you see the message 'There is no active recording', then you need to start a recording by canceling (F3) back to the main topas menu. Select 'Start New Recording', select 'Start Persistent local recording', select 'binary', be sure the 'Output Path' has been specified; the default path is typically '/var/perf/daily'. If the path is not defined, the current directory may be used. You may also have to change 'Overwrite existing recording file' to yes if you get an error. Q12. I have stopped my performance recording, why is my server still sending the same performance information to IBM? A12. You will also have to Disable Data Transmission. To do this, type 'smit topas', select 'Setup Performance Management', select 'Disable Data Transmission'. The performance information will not be sent to IBM from ESA or HMC. Q13. How many days of performance data are stored on my server? A13. The default of '3' days are stored on your server. It is recommended that you keep a minimum of 3 days. To change this setting, type 'smit topas', select 'Setup Performance Management', select 'Change/Show Data Retention Period' and change the 'Retention Period (days)' to the desired number of days. Q14. How do I retransmit all of my available performance data to IBM? A14. To retransmit all of your available performance data to IBM, type 'smit topas', select 'Setup Performance Management', select 'Retransmit Recorded Data', enter zero '0' for the 'Enter the Date [YYYYMMDD]' and press ENTER. Follow the instructions on how to use ESA or HMC to send the performance file to IBM. Q15. Why can't I send performance data for the current date? A15. You can only generate a performance file for the previous day's performance recording. Q16. How do I update my customer information including e-mail address? A16. Type 'smit topas', select 'Setup Performance Management', select 'Change/Show Customer Information', make the appropriate changes. Your changes will be automatically sent the next time the performance information is transmitted to IBM. Q17. I am not able to enable PM data transmission when the persistent recording is configured with more than 1 day of recording data per file. Why ? A17. PM Data transmission can be enabled only if recorded data for each day is collected in a separate file. Re-start the recording with 'Number of Days to store per file' set to 1. ======================================================================== The rest of the information contained in this file consists of a description of a set of performance tools which may be used to evaluate and tune AIX system performance. For more detailed information on each of the tools, type "man 'toolname'". The descriptions below pertain only to AIX releases on Power Architecture. Refer to Command Reference volumes for AIX releases on other Architectures for the most current support information. ======================================================================== Some of the tools listed below were previously located in the perfagent.tools fileset prior to the AIX Version 5 release. DESCRIPTION OF TOOLS filemon: Filemon monitors a trace of file system and I/O system events, and reports performance statistics for files, virtual memory segments, logical volumes, and physical volumes. Filemon is useful to those whose applications are believed to be disk-bound, and want to know where and why. Filemon does not currently support the JFS 2 File System. For large-scale systems, it is recommended that filemon be utilized in offline mode. See the filemon manpage for additional information. fileplace: Fileplace displays the placement of a file's blocks within a file system. Logically contiguous files may be physically fragmented on disk, depending on the available free space at the time the file is created. Fileplace can be used to examine and assess the efficiency of a file's placement on disk. Fileplace does not currently support the JFS 2 File System. gennames: Gathers all the information necessary to run the tprof, filemon, netpmon or pprof commands in off-line mode. ipfilter: Extracts different operation headers from an ipreport output file and displays them in a table. Some customized nfs information regarding requests and replies is also provided. netpmon: Netpmon monitors a trace of system events, and reports on CPU usage, network device-driver I/O, internet socket calls, and NFS I/O. In its normal mode, netpmon runs in the background while one or more application programs or system commands are being executed and monitored. When tracing is stopped via a 'trcstop' command, netpmon generates all specified reports and exits. Netpmon provides only partial support most PCI-based network adapters. Netpmon does not support NFS Version 3 metrics. For large-scale systems, it is recommended that netpmon be utilized in offline mode. See the netpmon manpage for additional information. pprof: The pprof command reports CPU usage of all kernel threads running within an interval using the trace utility. rmss: Rmss provides you with a means to simulate workstations with various size memories, without having to extract and replace memory boards. Moreover, rmss provides a facility to run an application over a range of memory sizes, displaying, for each memory size, performance statistics such as the response time of the application and the amount of paging. In short, rmss is designed to help you answer the question: How much memory does a given platform need to run a given application with an acceptable level of performance? In AIX 4.3.2, rmss was updated to utilize parameters set by the vmtune command. The %pinnable, %free frame, %file frames permission rates set my vmtune (via the -M, -f/-F,-p/-P options) should be set before stealing memory (rmss -c) and reset after freeing memory (rmss -r). Previously, rmss would only allow 80% of a systems memory to be pinned (the default value set at boot time on many systems). Utilizing vmtune to adjust these values allows rmss to operate across the ranges allowed by your system administrator. The vmtune parameters should be reset the desired values once rmss tests have been completed. stripnm: Displays the symbol information of a specified object file. svmon: Svmon is a virtual memory monitor. When invoked, it captures a snapshot of the current contents of both real and virtual memory, and summarizes the contents. Svmon is useful to those who wish to determine which processes, or alternatively which segments, are con- suming the most real memory. Svmon now provides reports on Workload Management classes and tiers. See below for additional information on svmon enhancements. topas: The topas command reports selected statistics about the activity on the local system. The command uses the curses library to display its output in a format suitable for viewing on an 80x25 character based display or in a window of at least the same size on a graphical display. Topas supports additional process and WLM report screens in AIX Version 5.1. tprof: Tprof provides a detailed profile of CPU usage by an application. Tprof is like the standard program profilers prof and gprof, in that it provides an estimate of CPU usage by each routine in a program. Unlike prof and gprof, it provides a CPU profile of all processes running during an application, not just the program of interest. Additionally, it provides a profile down to the source statement level, if the associated C or Fortran source files are available. All this is accomplished with relatively little or no overhead. Tprof is useful to those whose applications are believed to be cpu-bound, and want to know where and why. For large-scale systems, it is recommended that tprof be utilized in offline mode. In conjunction with the gennames command, offline tprof allows trace files to be post-processed on other systems. ======================================================================== Large Page enablement notes for AIX Version 5.1.0.25 and above: svmon tool changes to support the Large Page model -------------------------------------------------- Objectives: ----------- The svmon command reports physical memory related information. Svmon currently reports the number of inuse, pinned and virtual mapped physical memory pages, and assumes a 4KB page size. The Large Page architecture allows mixing large and small(4KB) pages in an application address space. Svmon has to identify large page segments, large page processes and large page capability for a user and has to report large page pool related information. A new flag is added (-q flag) to make selective reports. Reports: -------- All reports are impacted by these evolutions. -Global report: (-G option) --------------------------- The second section is modified to report large page pool usage, lpage column is added: lpage pin: Number of pinned frames allocated for large page pools. lpage in use: Number of frames used in the large page pools. These values are expressed in 4KB pages For a system with a large page pool, a third section is reported describing the large page pool: pgsize: Large page size plus its unit (MB or KB) size: Number of large pages allocated at boot time for the large page pool (expressed in "pgsize" unit) free: Number of large pages chained in the large page free lists (expressed in "pgsize" unit) "-q" flag has no effect for this option. -User report: (-U option) ------------------------- A LPageCap field is added to the global user information. This field is used to report if the user has the right or not to use large pages. On a system with large page pool, the distribution between the different page sizes is detailed. A Lpage field is added to the report for each segment to display if it is a large page segment or not. "-q" flag allows to select only user with a large page capability. When a list of users is specified in conjuction with the -q option, an error is reported for each user without large page capability. -Command report: (-C option) ---------------------------- On a system with large pages, the distribution between the different page sizes is detailed. A Lpage field is added to the report for each segment to display if it is a large page segment or not. "-q" flag allows to only take into account the large page segments used by the commands. -Process report: (-P option) ---------------------------- A LPage field is added to the global process information. This field is used to report if the process uses large pages segment or not. On a system with large pages, the distribution between the different page sizes is detailed. A Lpage field is added to the report for each segment to display if it is a large page segment or not. "-q" flag allows to select only large page processes. When a list of processes is specified in conjuction with the -q option, an error is reported for each process not marked large page process. -Segment report: (-S option) ---------------------------- A Lpage field is added to the report for each segment to display if it is a large page segment or not. "-q" flag allows to select only large page segments. When a list of segments is specified in conjuction with the -q option, an error is reported for each segment that does not have the "large page" flag. -Detail report: (-D option) --------------------------- A Lpage item is added to tell if the segment is a large page one or not. When used with "-q" option, an error is reported for each segment that does not have the "large page" flag. -Frame report: (-F option) -------------------------- A Lpage item is added to tell if the frame pertains to a large page segment or not. When used with "-q" option, an error is reported for each frame that does not pertain to a large page segment. Class report: (-W option) ------------------------- On a system with a large pages pool, the distribution between the different page sizes is detailed. "-q" flag allows to only take into account the large page segments used by the classes. Tier report: (-T report) ------------------------ On a system with a large pages pool, the distribution between the different page sizes is detailed. "-q" flag allows to only take into account the large page segments used by the classes. When using the "-q" flag on a system without a large pages pool, an error is reported.