curt Command

Purpose

Generates CPU utilization report from a trace.

Syntax

curt -i inputfile [-o outputfile] [-n gensymsfile] [-m trcnmfile] [-a pidnamefile] [-f timestamp] [-l timestamp] [-r PURR] [-ehpstP] [-@ {ALL | WparList}]

Description

The curt command takes an AIX® trace file as input and produces a number of statistics related to processor (CPU) utilization and process/thread/pthread activity. The command will work with both uniprocessor and multiprocessor AIX traces if the processor clocks are properly synchronized.

The AIX trace file which is gathered using the trace command should contain at least the trace events (trace hooks) listed below. These are the events curt looks at to calculate its statistics: 
  HKWD_KERN_SVC, HKWD_KERN_SYSCRET, HKWD_KERN_FLIH, HKWD_KERN_SLIH,
  HKWD_KERN_SLIHRET, HKWD_KERN_DISPATCH, HKWD_KERN_RESUME, HKWD_KERN_IDLE,
  HKWD_SYSC_FORK, HKWD_SYSC_EXECVE, HKWD_KERN_PIDSIG, HKWD_SYSC__EXIT
  HKWD_SYSC_CRTHREAD, HKWD_KERN_INITP, HKWD_NFS_DISPATCH, HKWD_CPU_PREEMPT,
  HKWD_DR, HKWD_KERN_PHANTOM_EXTINT, HKWD_RFS4_VOPS, HHKWD_RFS4_VFSOPS, HKWD_RFS4_MISCOPS, HKWD_RFS4,
  HKWD_KERN_HCALL, HKWD_WPAR,
  HKWD_PTHREAD_VPSLEEP, HKWD_PTHREAD_GENERAL
This means that, if you specify the -j flag on your trace command, you must include these numbers for curt: 
-j 100,101,102,103,104,106,10C,119,134,135,139,200,210,215,38F,419,465,47F,488,489,48A,48D,492,4C9,605,609
Or, you can use -J curt instead.
To get the PTHREAD hooks into the trace, you must execute your pthread application using the instrumented libpthreads.a. One way to cause that to happen is to perform the following three steps before starting your application (KornShell syntax):
  1. mkdir /temp.lib; cd /temp.lib
  2. ln -s /usr/ccs/lib/perf/libpthreads.a
  3. export LIBPATH=$PWD:$LIBPATH
Putting the instrumented library directory in LIBPATH is necessary to activate the user pthread instrumentation; the temp.lib directory can be put anywhere.

Flags

Item Description
-i inputfile Specifies the input AIX trace file to be analyzed.
-o outputfile Specifies the output file (default is stdout).
-n gensymsfile Specifies a names file produced by gensyms.
-m trcnmfile Specifies a names file produced by trcnm.
-a pidnamefile Specifies a PID to process name mapping file.
-f timestamp Starts processing trace at timestamp seconds.
-l timestamp Stops processing trace at timestamp seconds.
-r PURR Uses the PURR register to calculate CPU times.
-e Outputs elapsed time information for system calls and pthread calls.
-h Displays usage text (this information).
-p Outputs detailed process information.
-s Outputs information about errors returned by system calls.
-t Outputs detailed thread information.
-P Outputs detailed pthread information.
-@ Controls the addition of workload partition information to a curt report.
You can use the -@ flag in one of the following forms:
-@
Outputs a summary of workload partitions. The summary includes the processor usage for workload partitions in various execution modes. In addition, WPAR names are shown for listed processes summarizing the processor usage by processes, threads, or pthreads.
-@ All
Outputs reports for the system and all of the workload partitions. The reports are delimited by three lines containing WPAR names or SYSTEM for the overall system.
-@ WparList
Outputs reports for the workload partitions specified by the WparList parameter, which is a comma-separated list of WPAR names. The reports are delimited by three lines containing WPAR names.

If the trace process name table is not accurate, or if more descriptive names are desired, use the -a flag to specify a PID to process name mapping file. This is a file with lines consisting of a process ID (in decimal) followed by a space followed by an ASCII string to use as the name for that process.

If the input AIX-trace file is created with the -n flag specified, curt will use that address/name table to resolve System Call and Slih addresses to names if you do not specify a -m or a -n flag on the curt command line.

If the input AIX-trace file is created in a workload partition, the curt command prints a WPAR report. The -@ flag is not allowed in this case.

Report Contents

The curt report includes the following information:

curt and Trace Information

The first lines in the curt report give the time when the curt program was executed and the command line used to invoke curt. Following that is this information about the AIX trace file being processed by curt: name, size, creation date, and the command used to gather the trace file.

The line PURR was used to calculate CPU times is printed if the -r PURR option was used and the trace file includes the PURR register.

System Summary

The first major section of the report is the System Summary. This section describes the time spent by the system as a whole (all processors) in various execution modes. These modes are as follows:
APPLICATION
The sum of times spent by all processors in User (non-privileged) mode.
SYSCALL
The sum of times spent by all processors doing System Calls. This is the portion of time that a processor spends executing in the kernel code providing services directly requested by a user process.
HCALL
The sum of times spent by all processors doing Hypervisors Calls. This is the portion of time that a processor spends executing in the hypervisor code providing services directly requested by the kernel.
KPROC
The sum of times spent by all processors executing kernel processes other than the IDLE process and NFS processes. This is the portion of time that a processor spends executing specially created dispatchable processes which only execute kernel code.
NFS
The sum of times spent by all processors executing NFS operations. NFS operations begin with RFS_DISPATCH_ENTRY and end with RFS_DISPATCH_EXIT subhooks for NFS V2/V3. NFS operations begin with start and end with done or done error for NFS V4.
FLIH
The sum of times spent by all processors in FLIHs (first level interrupt handlers).
SLIH
The sum of times spent by all processors in SLIHs (second level interrupt handlers).
DISPATCH
The sum of times spent by all processors in the AIX dispatch code. This sum includes the time spent in dispatching all threads (i.e. it includes the dispatches of the IDLE process).
IDLE DISPATCH
The sum of times spent by all processors in the AIX dispatch code where the process being dispatched was the IDLE process. Because the DISPATCH category includes the IDLE DISPATCH category's time, the IDLE DISPATCH category's time is not separately added to calculate either CPU(s) busy time or TOTAL (see below).
CPU(s) busy time
The sum of times spent by all processors executing in application, syscall, kproc, flih, slih, and dispatch modes.
IDLE
The sum of times spent by all processors executing the IDLE process.
TOTAL
The sum of CPU(s) busy time and IDLE. This number is referred to as "total processing time."
The column labeled processing total time (msec) gives the total time (in milliseconds) for the corresponding processing category. The column labeled percent total time gives the processing total time as a percentage of the TOTAL processing total time. The column labeled percent busy time gives the processing total time as a percentage of the CPU(s) busy time processing total time. The Avg. Thread Affinity is the probability that a thread was dispatched to the same processor that it last executed on.

The Total Physical CPU time (msec) is the real time the CPU(s) were running (not preempted). The Physical CPU percentage gives the Physical CPU(s) Time as a percentage of total time.

Note: In a WPAR report, the system summary information is labeled "WPAR summary".

System Application Summary

Following the System Summary is the System Application Summary, which describes the time spent in User mode in details. This section describes the time spent by all processes (on all processors) executing various parts of libpthreads.
PTHREAD
The sum of times spent by all pthreads in traced libpthreads operations.
PDISPATCH
The sum of times spent by all pthreads in the libpthreads dispatch code.
PIDLE
The sum of times spent by all pthreads in libpthreads vp_sleep code.
OTHER
The sum of time spent by all threads in user mode outside traced libpthreads operations.
APPLICATION time
The sum of times spent by all processors in user mode.
The column labeled processing total time (msec) gives the total time in milliseconds for the corresponding processing category. The column labeled percent total time gives the processing total time as a percentage of the TOTAL processing total time of System Summary. The column labeled percent application time gives the processing total time as a percentage of the APPLICATION processing total time. The Avg. Pthread Affinity is the probability that a pthread was dispatched to the same thread that it last executed on.
Note: In a WPAR report, the system application summary information is labeled "WPAR application summary".

s Summary

The WPARs Summary of the report is generated when you specify the -@ flag. The following system and system application information for workload partitions, shown as column headings in the summary, describes the time spent in all of the workload partitions in details:
appli
Percent of the total process time that was spent by the WPAR in user mode (non-privileged).
syscall
Percent of the total process time that was spent by the WPAR performing system calls.
hcall
Percent of the total process time that was spent by the WPAR performing hypervisor calls.
kproc
Percent of the total process time that was spent by the WPAR running kernel processes calls.
nfs
Percent of the total process time that was spent by the WPAR running NFS operations.
flih
Percent of the total process time that was spent by the WPAR in the first-level interrupt handlers.
slih
Percent of the total process time that was spent by the WPAR in the second-level interrupt handlers.
total
Percent of the total process time that was spent by the WPAR.
total(msec)
The sum of processor time, in milliseconds, used by the WPAR.
WPAR
The WPAR name.
Note: The WPARs Summary is generated only in an overall system report.

Per Processor Summary

Following the System Application Summary is the Per Processor Summary, which is essentially the same information but broken down on a processor by processor basis. In the description given for the System Summary, the phrase "sum of times spent by all processors" can be replaced by "time spent by this processor". The Total number of process dispatches refers to how many times AIX dispatched any non-IDLE process on this processor, while Total number of idle dispatches gives the count of IDLE process dispatches.

The Total Physical CPU time (msec) is the real time the processor was running (not preempted). The Physical CPU percentage gives the Physical CPU Time as a percentage of total time.

Physical processor affinity is the probability that a logical processor was dispatched on the same physical processor that it last executed on. Total number of preemptions is the number of times the virtual processor was redispatched on a physical CPU.

Total number of H_CEDE is the number of H_CEDE hypervisor call done by this processor; with preeemption indicates the number of H_CEDE calls resulting in preemption.

Total number of H_CONFER is the number of H_CONFER hypervisor call done by this processor; with preeemption indicates the number of H_CONFER calls resulting in preemption.

Note: A per processor summary is not generated in a WPAR report.

Per Processor Application Summary

Following each Processor Summary is the Per Processor Application Summary, which is essentially the same information as System Application Summary but broken down on a processor by processor basis.

The Total number of pthread dispatches refers to how many times libpthreads dispatched any pthread on this processor, while Total number of pthread idle dispatches gives the count of calls to vp_sleep.

Note: A per processor application summary is not generated in a WPAR report.

Application Summary

The second major section of the report is the Application Summary. The first part of this section summarizes the total system processing time on a per-thread basis (by Tid). For each thread, identified by Process ID (and name if available) and Thread ID, the summary gives the total application (same as APPLICATION above) and syscall (same as SYSCALL above) processing time in milliseconds and as the percentage of the total system processing time for all processors in the trace. In addition, the summary gives the sum of those two times, both as raw time, and as a percentage of the total processing time.

The second part of this section gives the same information on a per-process ID (by Pid) basis. The third part of this section gives the same information on a per-process name (by process type) basis.

The fourth part of this section gives similar information for kernel process threads (Kproc Summary). Since most kprocs provide a specific kernel service, the total processing time is split into two categories, operation and kernel, which loosely correspond to syscall and application for a process which always runs in kernel code. Each kproc thread is identified by name, Process ID, Thread ID and type of kproc if known. The kproc types are listed and described in a table immediately following this summary.

The fifth part of this section is the Pthread Process Summary. This section gives the total application time on multi-threaded Process (by Pid). For each process, identified by Process ID (and name if available), the summary gives the total application, pthread and other processing time in milliseconds and as the percentage of the total application time for all processors in the trace.

All five sections of the Summary are presented in sorted order from most combined processing time to least.

In all five sections of an Application Summary, the WPAR name is added to identify the thread or process if you specify the -@ flag.

Note: Pids and Tids (Process and Thread IDs) are always given in decimal.

System Calls Summary

The third major section of the report is the System Calls Summary. This section summarizes the processing time spent in system calls. For each system call (SVC), identified by kernel address (and name if available), the summary gives the number of times the SVC was called and the total processor time for all calls in milliseconds and as a percentage of total system processing time for all processors in the trace. In addition, the summary gives the average, minimum and maximum times for one call to the SVC. If the -e flag is specified, the summary gives the total elapsed time for all calls to the SVC and the average, minimum and maximum elapsed times for one call. Elapsed time is the wall-clock time from when the process starts executing the SVC in kernel mode until the process resumes executing in application mode. The Summary is presented in sorted order from most total processor time to least. If the -s flag is specified, the summary gives the number of times each error code (errno) was returned by each System Call.

The second part of this section is the Pending System Calls Summary. This part lists the System Calls which have started but not completed. The time that is given is included in the SYSCALL time for the system and the various processors and is included in the syscall time for the pthread, thread and process which issued the SVC, but is not included in the processing time for the system call in the first part of this section. The pending call is also not included in the count given in the first part of this section.

Note:
  1. System call addresses are always given in hexadecimal. Pids and Tids are always given in decimal.
  2. WPAR names are added in a System Calls Summary to identify threads or processes if you specify the -@ flag.

System Hypervisor Calls Summary

If there is hypervisor activity in the trace, an additional section is inserted at this point of the report. This major section of the report is called Hypervisor Calls Summary. This section summarizes the processing time spent in hypervisor calls. For each Hypervisor call (HCALL), identified by name (and kernel address), the summary gives the number of times the HCALL was called and the total processor time for all calls in milliseconds and as a percentage of total system processing time for all processors in the trace. In addition, the summary gives the average, minimum and maximum times for one call to the HCALL. If the -e flag is specified, the summary gives the total elapsed time for all calls to the HCALL and the average, minimum, and maximum elapsed times for one call. Elapsed time is the wall-clock time between the start and end of an hypervisor call. The summary is presented in sorted order from most total processor time to least.

The second part of this section is called Pending Hypervisor Calls Summary. This part lists the Hypervisor Calls which have started but not completed. The time that is given is included in the HCALL time for the system and the various processors and is included in the hypervisor time for the pthread, thread, and process which issued the HCALL, but is not included in the processing time for the hypervisor call in the first part of this section. The pending call is also not included in the count given in the first part of this section.

Note:
  1. Hypervisor call addresses are always given in hexadecimal. Pids and Tids are always given in decimal.
  2. WPAR names are added in a System Hypervisor Calls Summary to identify the threads or processes if you specify the -@ flag.

Pthread Calls Summary

The fourth major section of the report is the Pthread Calls Summary. This section summarizes the processing time spent in called pthread routines. For each pthread routine, identified by name, the summary gives the number of times the pthread routine was called and the total processor time for all calls, in milliseconds and as a percentage of total system processing time, for all processors in the trace. In addition, the summary gives the average, minimum and maximum times for one call to the pthread routine. If the -e flag is specified, the summary gives the total elapsed time for all calls to the pthread routine and the average, minimum and maximum elapsed times for one call. Elapsed time is the wall-clock time from when the process starts executing the pthread routine until the process exits the libpthreads code. The Summary is presented in sorted order from most total processor time to least.

The second part of this section is the Pending Pthread Calls Summary. This part lists the Pthread Calls which have started but not completed.

Note: WPAR names are added in a Pthread Calls Summary to identify threads or processes if you specify the -@ flag.

System NFS Calls Summary

This major section of the report is the System NFS Calls Summary. This section summarizes the processing time spent in NFS operations. For each NFS operation, identified by operation name and NFS version, the summary gives the number of times the operation was called and the total processor time for all calls in milliseconds and as a percentage of total NFS operation time for all operations with the same NFS version. In addition, the summary gives the average, minimum and maximum times for one call to the operation. If the -e flag is specified, the summary gives the total elapsed time for all calls to the operation and the average, minimum and maximum times for one call. The total elapsed time is also given as a percentage of total NFS operation elapsed time for all operations with the same NFS version. Elapsed time is the wall-clock time from the operation dispatch entry hook until the operation dispatch exit hook. In all cases, the summary gives the count of operation calls as a percentage of total NFS operation calls for all operations with the same NFS version. The Summary is presented in numerical order of the operation codes. The operations are presented in order of NFS Version. For NFS V4, the server operations are listed before the client operations.

The System NFS Calls Summary is followed by the Pending NFS Calls Summary. This part lists the NFS calls which have started but not completed. The time that is given is included in the NFS time for the system and the various processors and is included in the operation time for the thread and process which issued the NFS call, but is not included in the processing time for the NFS operation in the first part of this section. The pending call is also not included in the count given in the first part of this section.

Note: WPAR names are added in a System NFS Calls Summary to identify threads or processes if you specify the -@ flag.

Flih Summary

The fifth major section of the report is the Flih Summary. This section summarizes the amount of time spent in first level interrupt handlers (Flih). The first part of the summary gives the total number of entries to each Flih in the trace, as well as the total processor time for all executions of the Flih by all processors in milliseconds. In addition, the summary gives the average, minimum and maximum times for one execution. Each Flih is identified by a system-defined Flih type and a corresponding Flih name, if known.

The second part is the same information broken down on a processor by processor basis. It is possible that not all Flihs which occurred on the system will have occurred on each processor, so the Global Flih list may not be the same as the Flih list for each processor.

The second part of this section may include the Pending Flih Summary. This is a list of the Flihs which have started but not completed. The time that is given is included in the FLIH time for the system and the affected processor, but is not included in the processing time for the Flih in both parts of this section. The pending Flih is also not included in the counts given in both parts of this section.

Slih Summary

The fifth major section of the report is the Slih Summary. This section summarizes the amount of time spent in second level interrupt handlers (Slih). The first part of the summary gives the total number of entries to each Slih in the trace, as well as the total processor time for all executions of the Slih by all processors in milliseconds. In addition, the summary gives the average, minimum and maximum times for one execution. Each Slih is identified by kernel address and Slih function or module name, if known.

The second part is the same information broken down on a processor by processor basis. It is possible that not all Slihs which occurred on the system will have occurred on each processor, so the Global Slih list may not be the same as the Slih list for each processor.

The second part of this section may include the Pending Slih Summary. This is a list of the Slihs which have started but not completed. The time that is given is included in the SLIH time for the system and the affected processor, but is not included in the processing time for the Slih in both parts of this section. The pending Slih is also not included in the counts given in both parts of this section.

Detailed Process Information

This section of the report is produced when the -p flag is specified. It gives detailed information about each process found in the trace. This information is as follows:
  • The Process ID (Pid) for that process, the process name if known, and the WPAR name if you specify the -@ flag.
  • A count and a list of the Thread IDs (Tids) for that process.
  • A count and a list of Pthread IDs (Ptid) for that process, if any.
  • The time spent in application (user) mode, system call mode, and hypervisor mode is shown. For kprocs, the time spent in kernel mode and operation mode is shown instead.
  • The detail of time spent in application mode, time spent in pthread operations, time spent in libpthreads dispatch, and time spent in vp_sleep. This is printed only if there are any Ptids for the process.
  • Information on what Pthread calls were made by pthreads of this process. For NFS kprocs, information on which NFS Calls were made by threads of this process is shown instead. The -e flag also affects this output.
  • Information on what hypervisor calls were made by threads of this process. The -e flag also affects this output.
  • Information on what system calls were made by threads of this process. The -e flag also affects this output.
The processes are presented in sorted order from most combined application and syscall processing time to least.

Detailed Thread Information

This section of the report is produced when the -t flag is specified. It gives detailed information about each thread found in the trace. This information is as follows:
  • The Thread ID (Tid) and Process ID (Pid) for that thread, the process name if known, and the WPAR name if you specify the -@ flag.
  • The time spent in application (user) mode, system call mode, and hypervisor call mode is shown. For kprocs, the time spent in kernel mode and operation mode is shown instead.
  • Information on which system calls were made by this thread, including information on errors returned by the system calls if the -s flag was specified. For NFS kproc threads, information on which NFS Calls were made by this thread is shown instead. The -e flag also affects this output.
  • Information on which hypervisor calls were made by this thread. The -e flag also affects this output.
  • The processor affinity is the probability that, for any dispatch of the thread, the thread was dispatched to the same processor that it last executed on.
  • The Dispatch Histogram shows the number of times the thread was dispatched to each CPU in the system.
  • The total number of times the thread was dispatched (not including redispatches described in 7 below).
  • The number of redispatches due to interrupts being disabled indicates that the same thread which just ran was dispatched again because that thread has set the interrupt mask to INTMAX. This is shown only if nonzero.
  • The average dispatch wait time is the average elapsed time since the thread was last undispatched (i.e. average elapsed time since the thread last stopped executing).
  • How many times each type of Flih occurred while this thread was executing. Some of these types may be caused by the thread (such as DSI or ISI) while other types (such as IO) are can occur when this thread just happens to be running and are not necessarily caused by the thread itself.
The threads are presented in sorted order from most combined application and syscall processing time to least.

Detailed Pthread Information

This section of the report is produced when the -P flag is specified. It gives detailed information about each pthread found in the trace. This information is as follows:
  • The Pthread ID (Ptid) and Process ID (Pid) for that pthread, the process name if known, and the WPAR name if you specify the -@ flag.
  • The time spent in application (user) mode, kernel mode, and hypervisor mode is shown.
  • Application time detail: time spent in pthread calls, pthread dispatch, vp_sleep (pthread idle), and other application time.
  • Information on what system calls were made by this pthread, including information on errors returned by the system calls if the -s flag was specified. The -e flag also affects this output.
  • Information on what hypervisor calls were made by this pthread. The -e flag also affects this output.
  • Information on what Pthread calls were made by this pthread. The -e flag also affects this output.
  • The processor affinity is the probability that, for any dispatch of the pthread, the pthread was dispatched to the same processor that it last executed on.
  • The Dispatch Histogram for thread shows the number of times the pthread was dispatched to each CPU in the system.
  • The total number of times the pthread was dispatched (not including redispatches described in 9 below).
  • The number of redispatches due to interrupts being disabled indicates that the same pthread which just ran was dispatched again because that pthread has set the interrupt mask to INTMAX. This is shown only if non-zero.
  • The average dispatch wait time is the average elapsed time since the pthread was last undispatched by the kernel dispatcher (that is, average elapsed time since the pthread last stopped executing).
  • The thread affinity is the probability that, for any dispatch of the pthread, the pthread was dispatched to the same thread that it last executed on.
  • The Dispatch Histogram for pthread shows the number of times the pthread was dispatched to each thread in the system.
  • The total number of times the pthread was dispatched in libpthreads.
  • The average dispatch wait time is the average elapsed time since the thread was last undispatched by the libpthreads dispatcher (that is, the average elapsed time since the thread last stopped executing).
  • How many times each type of Flih occurred while this thread was executing. Some of these types may be caused by the thread (such as DSI or ISI) while other types (such as IO) are can occur when this thread just happens to be running and are not necessarily caused by the thread itself.

The pthreads are presented sorted by Pid-Ptid.

Files

Item Description
/usr/bin/curt Contains the curt command. Located in the bos.perf.tools fileset.