10% de réduction sur vos envois d'emailing --> CLIQUEZ ICI

Retour à l'accueil, cliquez ici

barrier 
Product: Cray XMT

In code, a barrier is used after a phase. The barrier delays the streams that were executing parallel operations in the phase until all the streams from the phase reach the barrier. Once all the streams reach the barrier, the streams begin work on the next phase.

blade 
Product: Cray XMT

1) A field-replaceable physical entity. A Cray XMT service blade consists of AMD Opteron sockets, memory, Cray SeaStar chips, PCI-X or PCIe cards, and a blade control processor. A Cray XMT compute blade consists of Threadstorm processors, memory, Cray SeaStar chips, and a blade control processor. 2) From a system management perspective, a logical grouping of nodes and blade control processor that monitors the nodes on that blade.

blade control processor 
Product: Cray X2, Cray XMT, Cray XT series, Cray XE series

A microprocessor on a blade that communicates with a cabinet control processor through the HSS network to monitor and control the nodes on the blade. See also blade, L0 controller, Hardware Supervisory System (HSS).

block scheduling 
Product: Cray XMT

Method of thread execution used by the compiler where contiguous blocks of loop iterations are divided equally and assigned to available streams. For example, if there are 100 loop iterations and 10 streams, the compiler assigns 10 iterations to each stream. The advantage to this method is that data in registers can be reused across adjacent iterations rather than releasing a stream after each iteration.

cabinet control processor 
Product: Cray X2, Cray XE series, Cray XMT, Cray XT series

A microprocessor in the cabinet that communicates with the HSS via the HSS network to monitor and control the devices in a system cabinet. See also Hardware Supervisory System (HSS).

cage 
Product: Cray XMT

A chassis on a Cray XMT series system. See chassis.

chassis 
Product: Cray XMT

The hardware component of a Cray XMT cabinet that houses blades. Each cabinet contains three vertically stacked chassis, and each chassis contains eight vertically mounted blades. See also cage.

Cray SeaStar chip 
Product: Cray XMT

The component of the system interconnection network that provides message routing and communication services. See also system interconnection network.

dependence analysis 
Product: Cray XMT

A technique used by the compiler to determine if any iteration of a loop depends on any other iteration (this is known as a loop-carried dependency).

dynamic scheduling 
Product: Cray XMT

In a dynamic schedule, the compiler does not bind iterations to streams at loop startup. Instead, streams compete for each iteration using a shared counter.

future 
Product: Cray XMT

Implements user-specified or explicit parallelism by starting new threads. A future is a sequence of code that can be executed by a newly created thread that is running concurrently with other threads in the program. Futures delay the execution of code if the code is using a value that is computed by a future, until the future completes. The thread that spawns the future uses parameters to pass information from the future to the waiting thread, which then executes. In a program, the term future is used as a type qualifier for a synchronization variable or as a keyword for a future statement.

induction variable 
Product: Cray XMT

A variable that is increased or decreased by a fixed amount on each iteration of a loop.

inductive loop 
Product: Cray XMT

An inductive loop is one which contains no loop-carried dependencies and has the following characteristics: a single entrance at the top of the loop; controlled by an induction variable; and has a single exit that is controlled by comparing the induction variable against an invariant.

interleaved scheduling 
Product: Cray XMT

Method of executing loop iterations used by the compiler where contiguous iterations are assigned to distinct streams. For example, for a loop with 100 iterations and 10 streams, one stream performs iterations 1, 11, 21,... while another stream performs iterations 2, 12, 22, ..., and so on. This method is typically used for triangular loops because it reduces imbalances. One disadvantage to using this method is that there is loss of data reuse between loop iterations because the stream is released at the end of the iteration.

L0 processor 
Product: Cray XMT

See blade control processor.

linear recurrence 
Product: Cray XMT

A special type of recurrence that can be parallelized.

logical machine 
Product: Cray XMT

An administrator-defined portion of a physical Cray XMT system, operating as an independent computing resource.

loop-carried dependences 
Product: Cray XMT

The value from one iteration of a loop is used during a subsequent iteration of the loop. This type of loop cannot be parallelized by the compiler.

multicore 
Product: Cascade, Cray X2, Cray XMT, Cray XT series

A processor that combines multiple independent execution engines ("cores"), each with its own cache and cache controller.

multiprocessor mode 
Product: Cray XMT

A mode that can be set at compile time that ensures that when the compiled application is run, iterations of a loop are run on multiple processors.

node 
Product: Cray XT series, Cray XMT, Cray XE series, Cray X2

For CLE systems, the logical group of processor(s), memory, and network components that acts as a network end point on the system interconnection network.

phase 
Product: Cray XMT

A set of one or more sections of code that the stream executes in parallel. Each section contains an iteration of a loop. Phases and sections are contained in control flow code generated by the compiler to control the parallel execution of a function.

recurrence 
Product: Cray XMT

A recurrence occurs when a loop uses values computed in one iteration in subsequent iterations. These subsequent uses of the value imply loop-carried dependences and thus usually prevent parallelization. To increase parallelization, use linear recurrence.

reduction 
Product: Cray XMT

A simple form of recurrence that reduces a large amount of data to a single value. It is commonly used to find the minimum and maximum elements of a vector. Although similar to a reduction, it is easier to parallelize and uses less memory.

region 
Product: Cray XMT

A region is an area in code where threads are forked in order to perform a parallel operation. The region ends at the point where the threads join back together at the end of the parallel operation.

service node 
Product: Cray XMT

Performs support functions for applications and system services such as login, network, I/O, boot, and service database (SDB). Service nodes run a version of CLE.

single-processor mode 
Product: Cray XMT

A mode that can be set at compile time that ensures that when the compiled application is run, iterations of a loop are run on a single processor.

Source : 

http://docs.cray.com/cgi-bin/craydoc.cgi?mode=Glossary;q=product%3dxmt

http://docs.cray.com/kbase/plat.html

Accéder au manuel utilisateur

http://docs.cray.com/books/S-0025-10//S-0025-10.pdf

Accéder au manuel utilisateur

New Feature http://docs.cray.com/books/S-6503-65/S-6503-65.pdf

Accéder au manuel utilisateur

Accéder au manuel utilisateur

http://docs.cray.com/books/004-2182-003/03preface.pdf Scienti?c Libraries User’s Guide 004–2151–002

http://docs.cray.com/books/004-2151-002//004-2151-002-manual.pdf PGI ® User’s Guide Parallel Fortran, C and C++ for Scientists and Engineer http://docs.cray.com/books/S-6516-61/pgi61ug.pdf PGI® User’s Guide Parallel Fortran, C and C++ for Scientists and Engineers http://docs.cray.com/books/S-6516-70/S-6516-70-mar07.pdf PAPI USER’S GUIDE http://docs.cray.com/books/S-6515-35/S-6515-35.pdf SuperLU Users' Guide James W. Demmel 1 John R. Gilbert 2 Xiaoye S. Li 3 Septemb er, 1999 Last update: October, 2003 http://docs.cray.com/books/S-6532-10/ug.pdf SuperLU Users’ Guide James W. Demmel 1 John R. Gilbert 2 Xiaoye S. Li 3 September 1999 Last update: June 2009 http://docs.cray.com/books/S-6532-20/6532-20.pdf February 2011 Programming Environments Release Announcement http://docs.cray.com/books/S-9401-1102//S-9401-1102.pdf Guide to Parallel Vector Applications 004–2182–003 http://docs.cray.com/books/004-2182-003/004-2182-003-manual.pdf CrayDoc™ Installation and Administration Guide S–2340–21 http://docs.cray.com/books/S-2340-21/S-2340-21-manual.pdf Comparing Binaries Between Cray Linux Environment (CLE) Systems, Standalone Whiteboxes, and ESLogin Nodes http://docs.cray.com/books/S-0019-10//S-0019-10.pdf Cray Application Developer's Environment User's Guid http://docs.cray.com/books/S-2396-601/S-2396-601.pdf Cray Application Developer's Environment User's Guide http://docs.cray.com/books/S-2396-60/S-2396-60.pdf Cray Application Developer's Environment User's Guid http://docs.cray.com/books/S-2396-50/S-2396-50.pdf AMD Core Math Library (ACML) Version 4.3.0 http://docs.cray.com/books/S-6511-43/S-6511-43.pdf AMD Core Math Library (ACML) Version 4.0.0 http://docs.cray.com/books/S-6511-40/acml_400_userguide.pdf Cray Fortran Reference Manual

http://docs.cray.com/books/S-3901-80/S-3901-80.pdf Cray C and C++ Reference Manual

http://docs.cray.com/books/S-2179-80/S-2179-80.pdf Lustre File System Operations Manual - Version 1.8 http://docs.cray.com/books/S-6540-1815/S-6540-1815.pdf Cray Linux Environment™ (CLE) 4.0 Software Release Overvie

http://docs.cray.com/books/S-2425-40/S-2425-40.pdf Cray XT™ System Overview : http://docs.cray.com/books/S-2423-22/S-2423-22.pdf Cray X1™ Series System Overview S–2346–25 http://docs.cray.com/books/S-2346-25/S-2346-25.pdf Migrating Applications to the Cray X1™ Series Systems S–2378–54 http://docs.cray.com/books/S-2378-54/S-2378-54.pdf intro_biolib(3) http://docs.cray.com/cgi-bin/craydoc.cgi?idx=man_search;q=id%3dintro_biolib.3;mode=Show;f=man/biolibm/30/cat3/intro_biolib.3.html Getting Started on Cray X2™ Systems S–2471–60 : http://docs.cray.com/books/S-2471-60/S-2471-60.pdf

Cray XT5h ™ System Overview S–2472–21 : http://docs.cray.com/books/S-2472-21/S-2472-21.pdf Cray® Programming Environment 6.0 Releases Overview and Installation Guide S–5212–60 http://docs.cray.com/books/S-5212-60/S-5212-60.pdf Cray® Fortran Reference Manual S–3901–60

http://docs.cray.com/books/S-3901-60/S-3901-60.pdf Cray® C and C++ Reference Manual S–2179–60 :

http://docs.cray.com/books/S-2179-60/S-2179-60.pdf Cray Performance Analysis Tools 5.3 Release Overview and Installation Guid

http://docs.cray.com/books/S-2474-53/S-2474-53.pdf Cray XMT™ System Overview http://docs.cray.com/books/S-2466-20/S-2466-20.pdf Cray XMT™ Programming Environment User's Guide http://docs.cray.com/books/S-2479-20/S-2479-20.pdf Cray XMT™ Programming Model http://docs.cray.com/books/S-2367-20/S-2367-20.pdf Cray XMT™ Debugger Reference Guid

http://docs.cray.com/books/S-2467-20/S-2467-20.pdf Cray XMT™ Performance Tools User's Guide

http://docs.cray.com/books/S-2462-20/S-2462-20.pdf Optimizing Loop-Level Parallelism in Cray XMT™ Applications :

http://docs.cray.com/books/S-2487-14/S-2487-14.pdf Limiting Loop Parallelism in Cray XMT™ Applications June 21, 2010 http://docs.cray.com/books/S-0027-14/S-0027-14.pdf Cray DVS Installation and Configuration Private S–0005–10 http://docs.cray.com/books/S-0005-10//S-0005-10.pdf Application Cleanup by ALPS and Node Health Monitoring : http://docs.cray.com/books/S-0014-22/S-0014-22.pdf Application Programmer’s I/O Guide S–3695–36 : http://docs.cray.com/books/S-3695-36/S-3695-36-manual.pdf Overview of Gemini Hardware Counters This document describes the Gemini Performance Counters and how to use them to optimize individual applications and system traf?c. Send e-mail to docs@cray.com with any comments that will help us to improve the accuracy and usability of this document. Be sure to include the title and number of the document with your comments. We value your comments and will respond to them promptly. Accessing network performance counters is desirable for application developers, system library developers (e.g. MPI), and system administrators. Application developers want to improve their application run-times or measure what affect other traf?c on the system has on their application. System library developers want to optimize their collective operations. System Administrators want to observe the system, looking for hotspots. Effective with the CrayPat (Cray performance analysis tool) version 5.1 and Cray Linux Environment (CLE) version 3.1 software releases for the Cray XE platform, users can monitor many of the performance counters that reside on the Gemini networking chip. There are two categories of Gemini performance counters available to users. NIC performance counters record information about the data moving through theNetwork Interface Controller (NIC). On the Gemini ASIC there are two NICs, each attached to a compute node. Thus, the data from the NIC performance counters re?ects network transfers beginning and ending on the node. These performance counters are read-only. Network router tile counters are available on a per-Gemini basis. There are both read-only and read/write tile counters. Each chip has 48 router tiles, arranged in a 6x8 grid. Eight processor tiles connect to each of the two Gemini NICs. Each NIC connects to a different node, running separate Linux instances. If collection at other points of the application is desired, use the CrayPat API to insert regions as described in the pat_build man page. It is recommended that you do not collect any other performance data when collecting network counters. Data collection of network counters is much more expensive than other performance data collection, and will skew other results. At the time the instrumented executable program is launched with the aprun command, a set of environment variables, PAT_RT_NWPC_*, provide access to the Gemini network performance counters. These environment variables are described in the intro_craypat man page. S–0025–10 1Using the Cray Gemini Hardware Counters 1.1 Using CrayPat to Monitor Gemini Counters The CrayPat utility pat_build instruments an executable ?le. One aspect of the instrumentation includes intercepting entries into and returns out of a function. This is known formally as tracing. Information such as time stamps and performance counter values are recorded at this time. CrayPat supports instrumentation of an application binary for collection of Gemini counters. Counter values are recorded at application runtime, and are presented to the user through a table generated by pat_report. The CrayPat user interface to request instrumentation is similar to that for processor performance counters. There is no Gemini counter display available in Cray Apprentice2 at this time. A new display will be available in a subsequent release of the Cray Apprentice2 software. Although the user interface to request network counters is similar to processor counters, there are some signi?cant differences that must be understood. Depending on the type of counters requested, some are shared across all processors within a node, some are shared between two nodes and some are shared across all applications passing through a chip. Some counters monitor all traf?c for your application, even on nodes that are not reserved for your application, and some monitor locally, that is they monitor only traf?c associated with nodes assigned to a Gemini chip and no other traf?c from the network. Users should also be aware that access to the network counters is more resource-intensive than access to the processor performance counters. Because Gemini counters are a shared resource, the system software is designed to provide dedicated access whenever possible. This is done through the Application Level Placement Scheduler (ALPS) by ensuring that an application collecting counters is not placed on the same Gemini chip as another application collecting performance counters. It does not prevent a second application from being placed on the same Gemini chip that is not collecting counters however. This compromise assures better system utilization because compute nodes are not left unavailable for use by another application. The CrayPat 5.1 release focuses on the use of the NIC and ORB counters available within the Gemini chip. The values collected from these counters are local to a node and therefore speci?c to an application. Traf?c between MPI ranks cannot be distinguished through the counters. The event names that CrayPat supports are listed at the end of this document. Network counters are only collected for the MAIN thread. Values are collected at the beginning and end of the instrumented application. Instrumentation overhead is minimal. This gives a high-level view of the program's use of the networking router in terms of the counters speci?ed. Currently the time to access counter data is too expensive to collect more frequently. A future release of CLE will address these performance limitations. 2 S–0025–10Overview of Gemini Hardware Counters Before attempting the following examples verify that your system has a Gemini network: $ module list xtpe-network-gemini Attempting to collect Gemini performance counters on a system that does not have the Gemini network will result in a fatal error: $ aprun -n 16 my_program+pat CrayPat/X: Version 5.1 Revision 3329 05/20/10 11:26:16 pat[FATAL][0]: initialization of NW performance counter API failed [No such file or directory] Example 1. Collect stalls associated with node traf?c to and from the network This example enables tracing of MAIN. $ pat_build -w my_program $ export PAT_RT_NWPC=GM_ORB_PERF_VC0_STALLED,GM_ORB_PERF_VC1_STALLED $ aprun my_program+pat Example 2. Display network counter data $ pat_report my_program+pat+11171-41tdot.xf> counter_rpt Example output from pat_report: NWPC Data by Function Group and Function Group / Function / Node Id=0='HIDE' ===================================================================== Total --------------------------------------------------------------------- Time% 100.0% Time 2.476423 secs GM_ORB_PERF_VC1_STALLED 0 GM_ORB_PERF_VC1_BLOCKED 0 GM_ORB_PERF_VC1_BLOCKED_PKT_GEN 0 GM_ORB_PERF_VC1_PKTS 48 GM_ORB_PERF_VC1_FLITS 48 GM_ORB_PERF_VC0_STALLED 111 GM_ORB_PERF_VC0_PKTS 48 GM_ORB_PERF_VC0_FLITS 201 ===================================================================== S–0025–10 3Using the Cray Gemini Hardware Counters Example 3. Collect data for a custom group of network counters In this example a user creates a group of network events in a ?le called my_nwpc_groups, one called 1 and the other called CQ_AMO: $ cat my_nwpc_groups # Group 1: Outstanding Request Buffer 1 = GM_ORB_PERF_VC1_STALLED, GM_ORB_PERF_VC1_BLOCKED, GM_ORB_PERF_VC1_BLOCKED_PKT_GEN, GM_ORB_PERF_VC1_PKTS, GM_ORB_PERF_VC1_FLITS, GM_ORB_PERF_VC0_STALLED, GM_ORB_PERF_VC0_PKTS, GM_ORB_PERF_VC0_FLITS # Group CQ_AMO: CQ_AMO = GM_AMO_PERF_COUNTER_EN, GM_AMO_PERF_CQ_FLIT_CNTR, GM_AMO_PERF_CQ_PKT_CNTR, GM_AMO_PERF_CQ_STALLED_CNTR, GM_AMO_PERF_CQ_BLOCKED_CNTR $ pat_build -w my_program $ export PAT_RT_NWPC_FILE=my_nwpc_groups $ export PAT_RT_NWPC=1,CQ_AMO $ aprun -n16 my_program+pat 4 S–0025–10Overview of Gemini Hardware Counters Example output from pat_report: NWPC Data by Function Group and Function Group / Function / Node Id=0='HIDE' ===================================================================== Total --------------------------------------------------------------------- Time% 100.0% Time 2.639046 secs GM_ORB_PERF_VC1_STALLED 72525 GM_ORB_PERF_VC1_PKTS 50457 GM_AMO_PERF_COUNTER_EN 0 GM_AMO_PERF_CQ_FLIT_CNTR 11752 GM_AMO_PERF_CQ_PKT_CNTR 5876 GM_AMO_PERF_CQ_STALLED_CNTR 5092 GM_AMO_PERF_CQ_BLOCKED_CNTR 29 ===================================================================== Example 4. Suppress instrumented entry points from recording performance data to reduce overhead This example assumes a NWPC group FMAS exists and is available for use. Because the program is traced, the PAT_RT_TRACE_FUNCTION_NAME is set to suppress any data collection by already instrumented entry points in my_program+pat. This means that NWPC values will only be recorded for the MAIN thread at the start and the end of the instrumented program. Instrumentation overhead is minimal. $ pat_build -u -g mpi my_program $ export PAT_RT_NWPC=FMAS $ export PAT_RT_TRACE_FUNCITON_NAME=*:0 $ aprun -n32 my_program+pat This gives a high-level view of the program's use of the networking router in terms of what the FMAS group describes. If more details about NWPC use during execution of the program are desired, the PAT_RT_TRACE_FUNCTION_NAME environment variable need not be set, but the signi?cant overhead injected by reading the NWPCs may make the resulting performance data inaccurate. To selectively collect NWPCs and the other performance data for traced functions, add them to the end of PAT_RT_TRACE_FUNCTION_NAME: $ export PAT_RT_TRACE_FUNCTION_NAME=0:*,mxm,MPI_Bcast S–0025–10 5Using the Cray Gemini Hardware Counters 1.2 Gemini NIC Counters To better understand how to use the NIC counters, you need to understand some of the terminology speci?c to the Gemini network architecture. The Block Transfer Engine (BTE) A Gemini network packet typically consists of one or more ?its, which are the units of ?ow control for the network. Because ?its are usually larger than the physical datapath, they are divided into phits, which are the units of data that the network can handle physically. A packet must contain at least two phits, one for the header and one for the cyclical redundancy check (CRC). The V0 counters support the request channel and the V1 counters support the response channel. A ?it/pkt ratio can tell the user if the data entering the network was not aligned, eg a ratio greater than 1 indicates misaligned data is being sent across the network. Because there is a bandwidth/pipe size difference between outgoing and incoming (outgoing is smaller), in general you will notice more stalls on the V0 (request) channel. The following counters are recommended as a way to begin using the Gemini NWPC: GM_ORB_PERF_VC0_STALLED GM_ORB_PERF_VC1_STALLED GM_ORB_PERF_VC0_PKTS GM_ORB_PERF_VC1_PKTS GM_ORB_PERF_VC0_FLITS GM_ORB_PERF_VC1_FLITS Table 1. Atomic Memory Operations Performance Counters Name Description GM_AMO_PERF_ACP_COMP_CNTR Number of Atomic Memory Operation (AMO) computations that have occurred. GM_AMO_PERF_ACP_MEM_UPDATE_CNTR Number of AMO logic cache write-throughs that have occurred. GM_AMO_PERF_ACP_STALL_CNTR Number of AMO logic pipeline stalls that have occurred. GM_AMO_PERF_AMO_HEADER_CNTR Number of request headers processed by the Decode Logic that have had an AMO computation. Error packets are not counted. GM_AMO_PERF_COUNTER_EN When set, counting is enabled. When cleared, counting is disabled. GM_AMO_PERF_CQ_BLOCKED_CNTR Number of cycles the CQ FIFO is blocked. 6 S–0025–10Overview of Gemini Hardware Counters Name Description GM_AMO_PERF_CQ_FLIT_CNTR Number of ?its (network ?ow control units) that are read from the CQ FIFO. GM_AMO_PERF_CQ_PKT_CNTR Number of packets that are read from the CQ FIFO. GM_AMO_PERF_CQ_STALLED_CNTR Number of cycles the CQ FIFO is stalled. GM_AMO_PERF_DONE_INV_CNTR Number of times a valid cache entry was invalidated because there were no more outstanding AMO requests targeting it and the last request did not have the cacheable bit set. GM_AMO_PERF_ERROR_HEADER_CNTR Number of request headers processed by the Decode Logic that have had errors. GM_AMO_PERF_FLUSH_HEADER_CNTR Number of request headers processed by the Decode Logic that have had a Flush command. Error packets are not counted. GM_AMO_PERF_FULL_INV_CNTR Number of times a valid but inactive cache entry was invalidated to make room for a new AMO address. A high value in this counter indicates that there are too many cacheable AMO addresses and that the cache is being thrashed. GM_AMO_PERF_GET_HEADER_CNTR Number of request headers processed by the Decode Logic that have had an GET command. Error packets are not counted. GM_AMO_PERF_MSGCOMP_HEADER_CNTR Number of request headers processed by the Decode Logic that have had a MsgComplete command. Error packets are not counted. GM_AMO_PERF_PUT_HEADER_CNTR Number of request headers processed by the Decode Logic that have had an PUT command. Error packets are not counted. GM_AMO_PERF_REQLIST_FULL_STALL_CNTR Number of times an AMO request causes the NRP to stall waiting for a Request List entry to become free. GM_AMO_PERF_RMT_BLOCKED_CNTR Number cycles the RMT FIFO is blocked GM_AMO_PERF_RMT_FLIT_CNTR Number of ?its that are read from the RMT FIFO GM_AMO_PERF_RMT_PKT_CNTR Number of packets that are read from the RMT FIFO GM_AMO_PERF_RMT_STALLED_CNTR Number cycles the RMT FIFO is stalled S–0025–10 7Using the Cray Gemini Hardware Counters Name Description GM_AMO_PERF_TAG_HIT_CNTR Number of AMO requests that have been processed in the Tag Store and have resulted in a cache hit. GM_AMO_PERF_TAG_MISS_CNTR Number of AMO requests that have been processed in the Tag Store and have resulted in a cache miss. GM_AMO_PERF_TAG_STALL_CNTR Number of times a GET/PUT request hits in the cache and causes the NRP to stall. Table 2. Fast Memory Access Performance Counters Name Description GM_FMA_PERF_CQ_PKT_CNT Number of packets from Fast Memory Access (FMA) to CQ. GM_FMA_PERF_CQ_STALLED_CNT Number of clock cycles FMA_CQ was stalled due to lack of credits. GM_FMA_PERF_HT_NP_REQ_FLIT_CNT Number of HT NP request ?its to FMA. GM_FMA_PERF_HT_NP_REQ_PKT_CNT Number of HT NP request packets to FMA. GM_FMA_PERF_HT_P_REQ_FLIT_CNT Number of HT P request ?its to FMA. GM_FMA_PERF_HT_P_REQ_PKT_CNT Number of HT P request packets to FMA. GM_FMA_PERF_HT_RSP_PKT_CNT Number of HT response packets from FMA to HT. GM_FMA_PERF_HT_RSP_STALLED_CNT Number of clock cycles FMA_HT_RSP was stalled due to lack of credits. GM_FMA_PERF_TARB_FLIT_CNT Number of ?its from FMA to TARB. GM_FMA_PERF_TARB_PKT_CNT Number of packets from FMA to TARB. GM_FMA_PERF_TARB_STALLED_CNT Number of clock cycles FMA_TARB was stalled due to lack of credits. 8 S–0025–10Overview of Gemini Hardware Counters Table 3. Hyper-transport Arbiter Performance Counters Name Description GM_HARB_PERF_AMO_NP_BLOCKED Number of times AMO Non-Posted Queue has an entry, but is blocked from using the Non-Posted Initiator Request output channel by the BTE Non-Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_NP_FLITS Number of ?its coming out of the AMO Non-Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_NP_PKTS Number of packets coming out of the AMO Non-Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_NP_STALLED Number of cycles the AMO Non-Posted Queue is stalled due to a lack credits on the Non-Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_P_ACP_BLOCKED Number of times AMO Posted AMO Computation Pipe Queue has an entry, but is blocked from using the Posted Initiator Request output channel by another Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_P_ACP_FLITS Number of ?its coming out of the AMO Posted AMO Computation Pipe Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). S–0025–10 9Using the Cray Gemini Hardware Counters Name Description GM_HARB_PERF_AMO_P_ACP_PKTS Number of packets coming out of the AMO Posted AMO Computation Pipe Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_P_ACP_STALLED Number of cycles the AMO Posted AMO Computation Pipe Queue is stalled due to a lack credits on the Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_P_NRP_BLOCKED Number of times AMO Posted New Request Pipe Queue has an entry, but is blocked from using the Posted Initiator Request output channel by another Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_P_NRP_FLITS Number of ?its coming out of the AMO Posted New Request Pipe Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_P_NRP_PKTS Number of packets coming out of the AMO Posted New Request Pipe Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_P_NRP_STALLED Number of cycles the AMO Posted New Request Pipe Queue is stalled due to a lack credits on the Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). 10 S–0025–10Overview of Gemini Hardware Counters Name Description GM_HARB_PERF_BTE_NP_BLOCKED Number of times AMO Non-Posted BTE Queue has an entry, but is blocked from using the Non-Posted Initiator Request output channel by another Non-Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_BTE_NP_FLITS Number of ?its coming out of the AMO Non-Posted BTE Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_BTE_NP_PKTS Number of packets coming out of the AMO Non-Posted BTE Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_BTE_NP_STALLED Number of cycles the AMO Non-Posted BTE Queue is stalled due to a lack credits on the Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_BTE_P_BLOCKED Number of times AMO Posted BTE Queue has an entry, but is blocked from using the Posted Initiator Request output channel by another Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_BTE_P_FLITS Number of ?its coming out of the AMO Posted BTE Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). S–0025–10 11Using the Cray Gemini Hardware Counters Name Description GM_HARB_PERF_BTE_P_PKTS Number of packets coming out of the AMO Posted BTE Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_BTE_P_STALLED Number of cycles the AMO Posted BTE Queue is stalled due to a lack credits on the Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_COUNTER_EN When set, counting is enabled. When clear, counting is disabled. This MMR is reset by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_IREQ_NP_FLITS Number of ?its on the non-posted initiator request output of the HARB block. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_IREQ_NP_PKTS Number of packets on the non-posted initiator request output of the HARB Block. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_IREQ_NP_STALLED Number of cycles on the non-posted initiator request output of the HARB is stalled due to a lack credits on the Non-Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). 12 S–0025–10Overview of Gemini Hardware Counters Name Description GM_HARB_PERF_IREQ_P_FLITS Number of ?its on the posted initiator request output of the HARB block. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_IREQ_P_PKTS Number of packets on the posted initiator request output of the HARB Block. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_IREQ_P_STALLED Number of cycles on the posted initiator request output of the HARB is stalled due to a lack credits on the Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_RAT_P_BLOCKED Number of times AMO Posted RAT Queue has an entry, but is blocked from using the Posted Initiator Request output channel by another Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_RAT_P_FLITS Number of ?its coming out of the AMO Posted RAT Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). S–0025–10 13Using the Cray Gemini Hardware Counters Name Description GM_HARB_PERF_RAT_P_PKTS Number of packets coming out of the AMO Posted RAT Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_RAT_P_STALLED Number of cycles the AMO Posted RAT Queue is stalled due to a lack credits on the Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). Table 4. Network Address Translation Performance Counters Name Description GM_NAT_PERF_BTE_BLOCKED Number of cycles a BTE translation is blocked due to arbitration loss. GM_NAT_PERF_BTE_STALLED Number of cycles a BTE translation is stalled due to MMR access. GM_NAT_PERF_BTE_TRANSLATIONS Number of translations performed for the BTE interface. GM_NAT_PERF_COUNTER_EN When set, counting is enabled. When cleared, counting is disabled. GM_NAT_PERF_REQ_BLOCKED Number of cycles a REQ translation is blocked due to arbitration loss. GM_NAT_PERF_REQ_STALLED Number of cycles a REQ translation is stalled due to MMR access. GM_NAT_PERF_REQ_TRANSLATIONS Number of translations performed for the REQ interface. GM_NAT_PERF_RSP_BLOCKED Number of cycles a RSP translation is blocked due to arbitration loss. GM_NAT_PERF_RSP_STALLED Number of cycles a RSP translation is stalled due to MMR access. GM_NAT_PERF_RSP_TRANSLATIONS Number of translations performed for the RSP interface. GM_NAT_PERF_TRANS_ERROR0 Number of translations that failed due to error 0 (Uncorrectable error in translation). 14 S–0025–10Overview of Gemini Hardware Counters Name Description GM_NAT_PERF_TRANS_ERROR1 Number of translations that failed due to error 1 (VMDH table invalid entry). GM_NAT_PERF_TRANS_ERROR2 Number of translations that failed due to error 2 (MDDT/MRT invalid or illegal entry). GM_NAT_PERF_TRANS_ERROR3 Number of translations that failed due to error 3 (Protection tag violation). GM_NAT_PERF_TRANS_ERROR4 Number of translations that failed due to error 4 (memory bounds error). GM_NAT_PERF_TRANS_ERROR5 Number of translations that failed due to error 5 (write permission error) Table 5. Netlink Performance Counters Name Description GM_NL_PERF_ALL_LCBS_REQS_TO_NIC_0_STALLED Number of ticks all LCBs requests have stalled to NIC 0. GM_NL_PERF_ALL_LCBS_REQS_TO_NIC_1_STALLED Number of ticks all LCBs requests have stalled to NIC 1. GM_NL_PERF_ALL_LCBS_RSP_TO_NIC_0_STALLED Number of ticks all LCBs responses have stalled to NIC 0. GM_NL_PERF_ALL_LCBS_RSP_TO_NIC_1_STALLED Number of ticks all LCBs responses have stalled to NIC 1. GM_NL_PERF_CNTRL Controls the performance counters. Writing a 1 to the Start ?eld starts the counters. Writing a 1 to the Stop ?eld stops the counters. Writing a 1 to the Clear ?eld clears the counters. GM_NL_PERF_LCB_n_REQ_CMP_22 Decompressed request data to two phit LCB_n, where n is a value from 0 to 7 that speci?es the LCB. GM_NL_PERF_LCB_n_REQ_CMP_44 Decompressed request data to one phit LCB_n, where n is a value from 0 to 7 that speci?es the LCB. GM_NL_PERF_LCB_n_REQ_TO_NIC_0 Number of requests from LCB_n to NIC 0. GM_NL_PERF_LCB_n_REQ_TO_NIC_0_STALLED Number of ticks LCB_n requests are blocked to NIC 0. GM_NL_PERF_LCB_n_REQ_TO_NIC_1 Number of requests from LCB_n to NIC 1. S–0025–10 15Using the Cray Gemini Hardware Counters Name Description GM_NL_PERF_LCB_n_REQ_TO_NIC_1_STALLED Number of ticks LCB_n requests are blocked to NIC 1. GM_NL_PERF_LCB_n_REQ_TO_PHITS Number of request phits received on LCB_n. GM_NL_PERF_LCB_n_REQ_TO_PKTS Number of request packets received on LCB_n. GM_NL_PERF_LCB_n_RSP_CMP_22 Decompressed response data to two phit LCB_n GM_NL_PERF_LCB_n_RSP_TO_NIC_1 Number of responses from LCB_n to NIC 1. GM_NL_PERF_LCB_n_RSP_TO_NIC_1_STALLED Number of ticks LCB_n responses are blocked to NIC 1. GM_NL_PERF_NIC_0_REQ_STALLED_TO_ALL_LCBS Number of ticks NIC_0 requests are blocked to all LCBs. GM_NL_PERF_NIC_0_REQ_TO_LCB_n Number of requests from NIC_0 LCB_ n. GM_NL_PERF_NIC_0_REQ_TO_LCB_n_STALLED Number of ticks NIC_0 requests are blocked to LCB_n. GM_NL_PERF_NIC_0_RSP_STALLED_TO_ALL_LCBS Number of ticks NIC_0 responses are blocked to all LCBs. GM_NL_PERF_NIC_0_RSP_TO_LCB_n Number of responses from NIC_0 LCB_ n. GM_NL_PERF_NIC_0_RSP_TO_LCB_n_STALLED Number of ticks NIC_0 responses are blocked to LCB_n. GM_NL_PERF_NIC_1_REQ_STALLED_TO_ALL_LCBS Number of ticks NIC_0 requests are blocked to all LCBs. GM_NL_PERF_NIC_1_REQ_TO_LCB_n Number of requests from NIC_1 to LCB_ n. GM_NL_PERF_NIC_1_REQ_TO_LCBn_STALLED Number of ticks NIC_1 requests are blocked to LCB_n. GM_NL_PERF_NIC_1_RSP_STALLED_TO_ALL_LCBS Number of ticks NIC_1 responses are blocked to all LCBs. GM_NL_PERF_NIC_1_RSP_TO_LCB_n Number of responses from NIC_1 LCB_ n. GM_NL_PERF_NIC_1_RSP_TO_LCB_n_STALLED Number of ticks NIC_1 responses are blocked to LCB_n. 16 S–0025–10Overview of Gemini Hardware Counters Table 6. NPT Performance Counters Name Description GM_NPT_PERF_ACP_BLOCKED_CNTR Number of cycles the ACP FIFO is blocked. GM_NPT_PERF_ACP_FLIT_CNTR Number of ?its that are read from the ACP FIFO. GM_NPT_PERF_ACP_PKT_CNTR Number of packets that are read from the ACP FIFO. GM_NPT_PERF_ACP_STALLED_CNTR Number of cycles the ACP FIFO is stalled. GM_NPT_PERF_BTE_RSP_PKT_CNTR Number of packets that are sent to the Netlink as Get or Flush responses. GM_NPT_PERF_COUNTER_EN Provides the count enable. GM_NPT_PERF_FILL_RSP_PKT_CNTR Number of packets that are sent to the AMO block as ?ll responses. GM_NPT_PERF_HTIRSP_ERR_CNTR Number of packets that are received from the HT cave and have an error status. GM_NPT_PERF_HTIRSP_FLIT_CNTR Number of ?its that are received from the HT cave. GM_NPT_PERF_HTIRSP_PKT_CNTR Number of packets that are received from the HT cave. GM_NPT_PERF_LB_BLOCKED_CNTR Number of cycles the LB FIFO is blocked. GM_NPT_PERF_LB_FLIT_CNTR Number of ?its that are read from the LB FIFO. GM_NPT_PERF_LB_PKT_CNTR Number of packets that are read from the LB FIFO. GM_NPT_PERF_LB_STALLED_CNTR Number of cycles the LB FIFO is stalled. GM_NPT_PERF_NL_RSP_PKT_CNTR Number of packets that are sent to the AMO block as ?ll responses. GM_NPT_PERF_NPT_BLOCKED_CNTR Number of cycles the NPT FIFO is blocked. GM_NPT_PERF_NPT_FLIT_CNTR Number of ?its that are read from the NPT FIFO. GM_NPT_PERF_NPT_PKT_CNTR Number of packets that are read from the NPT FIFO. GM_NPT_PERF_NPT_STALLED_CNTR Number of cycles the NPT FIFO is stalled. GM_NPT_PERF_NRP_BLOCKED_CNTR Number of cycles the NRP FIFO is blocked. GM_NPT_PERF_NRP_FLIT_CNTR Number of ?its that are read from the NRP FIFO. GM_NPT_PERF_NRP_PKT_CNTR Number of packets that are read from the NRP FIFO. GM_NPT_PERF_NRP_STALLED_CNTR Number of cycles the NRP FIFO is stalled. S–0025–10 17Using the Cray Gemini Hardware Counters Table 7. ORB Performance Counters Name Description GM_ORB_PERF_VC0_FLITS Number of ?its to come into the TX Input Queue from the SSID. GM_ORB_PERF_VC0_PKTS Number of packets to come into the TX Input Queue from the SSID. GM_ORB_PERF_VC0_STALLED Number of packets not given access to the TX Control Logic because there is not enough credits available from the NL Block, or there are no available memory locations from the ORD RAM, or a tail ?it has not been received in the ORB Input Queue when performing store-and-forward. GM_ORB_PERF_VC1_BLOCKED Number of packets not given access to the RX Control Logic because the read address and write address into the ORD RAM are attempting to access the same bank of the ORD RAM or because there is a read access to the ORD RAM from the Local Block. GM_ORB_PERF_VC1_BLOCKED_PKT_GEN Number of times the RX Response FIFO is blocked because a packet in the RX Control Logic is being translated into the format used by the rest of the NIC. GM_ORB_PERF_VC1_FLITS Number of ?its to come into the Receive Response FIFO from the network. GM_ORB_PERF_VC1_PKTS Number of packets to come into the Receive Response FIFO from the network. GM_ORB_PERF_VC1_STALLED Number of packets not given access to the RX Control Logic because there is not enough credits available from the RAT. 18 S–0025–10Overview of Gemini Hardware Counters Table 8. RAT Performance Counters Name Description GM_RAT_PERF_COUNTER_EN Enables the performance counters. GM_RAT_PERF_DATA_FLITS_VC0 Number of data ?its received on VC0 (request pipeline). GM_RAT_PERF_DATA_FLITS_VC1 Number of data ?its received on VC1 (request pipeline). GM_RAT_PERF_HEADER_FLITS_VC0 Number of header ?its received on VC0 (request pipeline). GM_RAT_PERF_HEADER_FLITS_VC1 Number of header ?its received on VC1 (request pipeline). GM_RAT_PERF_STALLED_CREDITS_VC0 Number of cycles VC0 (request pipeline) is stalled due to insuf?cient credits. GM_RAT_PERF_STALLED_CREDITS_VC1 Number of cycles VC1 (request pipeline) is stalled due to insuf?cient credits. GM_RAT_PERF_STALLED_TRANSLATION_VC0 Number of cycles VC0 (request pipeline) is stalled due to unavailable translation data. GM_RAT_PERF_STALLED_TRANSLATION_VC1 Number of cycles VC1 (request pipeline) is stalled due to unavailable translation data. GM_RAT_PERF_TRANSLATION_ERRORS_VC0 Number of translation errors seen on VC0 (request pipeline). GM_RAT_PERF_TRANSLATION_ERRORS_VC1 Number of translation errors seen on VC1 (request pipeline). GM_RAT_PERF_TRANSLATIONS_VC0 Number of translations requested on VC0 (request pipeline). GM_RAT_PERF_TRANSLATIONS_VC1 Number of translations requested on VC1 (request pipeline). S–0025–10 19Using the Cray Gemini Hardware Counters Table 9. RMT Performance Counters Name Description GM_RMT_PERF_PUT_BYTES_RX Tally of bytes received in all PUT packets that had the RMT Enable ?eld set that entered and exited the RMT with OK status. GM_RMT_PERF_PUT_CAM_EVIT PUT sequences evicted from the CAM. GM_RMT_PERF_PUT_CAM_FILL New PUT sequence packet arrived and successfully allocated in the CAM. GM_RMT_PERF_PUT_CAM_HITS Packet for PUT sequence currently stored in RMT arrived and successfully located entry in CAM. GM_RMT_PERF_PUT_CAM_MISS New PUT sequence packet arrived, but did not allocate because CAM was full. GM_RMT_PERF_PUT_PARITY Number of sequences evicted from CAM due to uncorrectable parity errors. GM_RMT_PERF_PUT_RECV_COMPLETE Number of MsgRcvComplete packets received which evicted a CAM entry. GM_RMT_PERF_PUT_TIMEOUTS Number of sequences evicted from CAM due to timeout. GM_RMT_PERF_SEND_BYTES_RX Tally of bytes received in all SEND packets that had the RMT Enable ?eld set and entered and exited the RMT with OK status. GM_RMT_PERF_SEND_CAM_EVIT SEND sequences evicted from the CAM. GM_RMT_PERF_SEND_CAM_FILL New SEND sequence packet arrived and successfully allocated in the CAM. GM_RMT_PERF_SEND_CAM_HITS Packet for SEND sequence currently stored in RMT arrived and successfully located entry in CAM. GM_RMT_PERF_SEND_CAM_MISS New SEND sequence packet arrived, but did not allocate because CAM was full. GM_RMT_PERF_SEND_PARITY Number of sequences evicted from CAM due to uncorrectable parity errors. GM_RMT_PERF_SEND_ABORTS Number of SEND sequences that were aborted. GM_RMT_PERF_SEND_TIMEOUTS Number of sequences evicted from CAM due to timeout. 20 S–0025–10Overview of Gemini Hardware Counters Table 10. SSID Performance Counters Name Description GM_SSID_PERF_COMPLETION_COUNT_1 Provides a count of completed request packet sequences. The type of sequence completions counted by this register is controlled by the SSID Performance – Completion Count Selector Register. GM_SSID_PERF_COMPLETION_COUNT_2 Provides a count of completed request packet sequences. The type of sequence completions counted by this register is controlled by the SSID Performance – Completion Count Selector Register. GM_SSID_PERF_COMPLETION_COUNT_SELECTOR Speci?es the types of completion events that are counted in the SSID Performance – Completion Count 1 Register (bits 3-0) and the SSID Performance – Completion Count 2 Register (bits 11-8). See the table of SSID_PerfCompletionCountSelect Encoding values for encoding of these ?elds. GM_SSID_PERF_OUT_STALLED_DURATION The accumulated number of cycles of cclk for which the SSID had a valid ?it available to send to the ORB but sending of the ?it had to be stalled while waiting for a credit from the ORB. This value is cleared by writing any value to this register. GM_SSID_PERF_OUTOFSSIDS_COUNT The number of Allocate SSID requests that have been received for which processing of the request had to be stalled for one or more clock cycles because a free SSID was not immediately available to service the request. This value is cleared by writing any value to this register. GM_SSID_PERF_OUTOFSSIDS_DURATION The accumulated number of cycles of cclk for which processing of Allocate SSID requests has been stalled because a free SSID is not available to service the request. This value is cleared by writing any value to this register. S–0025–10 21Using the Cray Gemini Hardware Counters Name Description GM_SSID_PERF_SSID_ALLOCATE_COUNT The total number of Allocate SSID requests that have been received, across all channels (all FMA descriptors and all BTE VCs), because this register was last cleared, and that resulted in a SSID actually being allocated. Allocate SSID requests that do not result in a SSID being allocated (i.e. redundant Allocate requests) are not counted. This value is cleared by writing any value to this register. GM_SSID_PERF_SSIDS_IN_USE Bits 7-0 specify the number of SSIDs currently in use across all Request Channels. This value is not affected by writes to this register. This ?eld is initialized to its reset value by a full reset and by an ht reset. Bits 23-16 specify the maximum number of SSIDs that have been in use simultaneously, across all channels (all FMA descriptors and all BTE Vcs), since this register was last initialized. This value is initialized to CurrentSSIDsInUse by writing any value to this register. This ?eld is initialized to its reset value by a full reset. Table 11. Transmit Arbiter Performance Counters Name Description GM_TARB_PERF_BTE_BLOCKED Transmit Arbiter (TARB) Performance BTE Blocked Count GM_TARB_PERF_BTE_FLITS TARB Performance BTE Flit Count GM_TARB_PERF_BTE_PKTS TARB Performance BTE Packet Count GM_TARB_PERF_BTE_STALLED TARB Performance BTE Stalled Count GM_TARB_PERF_FMA_BLOCKED TARB Performance FMA Blocked Count GM_TARB_PERF_FMA_FLITS TARB Performance FMA Flit Count GM_TARB_PERF_FMA_PKTS TARB Performance FMA Packet Count GM_TARB_PERF_FMA_STALLED TARB Performance FMA Stalled Count GM_TARB_PERF_LB_BLOCKED TARB Performance LB Blocked Count GM_TARB_PERF_LB_FLITS TARB Performance LB Flit Count GM_TARB_PERF_LB_PKTS TARB Performance LB Packet Count 22 S–0025–10Overview of Gemini Hardware Counters Name Description GM_TARB_PERF_LB_STALLED TARB Performance LB Stalled Count GM_TARB_PERF_OUT_FLITS TARB Performance Output Flit Count GM_TARB_PERF_OUT_PKTS TARB Performance Output Packet Count GM_TARB_PERF_OUT_STALLED TARB Performance Output Stalled Count 1.3 Gemini Tile MMRs The Gemini network consists of 48 tiles, arranged in 6 rows of 8 columns. Within each tile there are memory-mapped registers associated with the LCB and with the rest of the tile. The local block has shared connections to each row of tiles. By default, when only the name of the MMR is used, an event is counted on all 48 tiles. To address an individual tile, append the row (0-5) and column (0-7) to the name, as shown in the table. Table 12. Description of Gemini Tile MMRs Name Description GM_TILE_PERF_VC0_PHIT_CNT:n:m Number of vc0 phits read from inq buffer GM_TILE_PERF_VC1_PHIT_CNT:n:m Number of vc1 phits read from inq buffer GM_TILE_PERF_VC0_PKT_CNT:n:m Number of vc0 packets read from inq buffer GM_TILE_PERF_VC10_PKT_CNT:n:m Number of vc1 packets read from inq buffer GM_TILE_PERF_INQ_STALL:n:m Number of clock periods a valid reference is blocked from the routing pipeline. GM_TILE_PERF_CREDIT_STALL:n:m Number of clock periods a valid reference is stalled in the column buffers, waiting on transmissions credits. S–0025–10 23Using the Cray Gemini Hardware Counters © 2010 Cray Inc. All Rights Reserved. This document or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Inc. Cray, LibSci, PathScale, and UNICOS are federally registered trademarks and Active Manager, Baker, Cascade, Cray Apprentice2, Cray Apprentice2 Desktop, Cray C++ Compiling System, Cray CX, Cray CX1, Cray CX1-iWS, Cray CX1-LC, Cray CX1000, Cray CX1000-C, Cray CX1000-G, Cray CX1000-S, Cray CX1000-SC, Cray CX1000-SM, Cray CX1000-HN, Cray Fortran Compiler, Cray Linux Environment, Cray SHMEM, Cray X1, Cray X1E, Cray X2, Cray XD1, Cray XE, Cray XE6, Cray XMT, Cray XR1, Cray XT, Cray XTm, Cray XT3, Cray XT4, Cray XT5, Cray XT5 h , Cray XT5m, Cray XT6, Cray XT6m, CrayDoc, CrayPort, CRInform, ECOphlex, Gemini, Libsci, NodeKARE, RapidArray, SeaStar, SeaStar2, SeaStar2+, Threadstorm, UNICOS/lc, UNICOS/mk, and UNICOS/mp are trademarks of Cray Inc. Version 1.0 Published July 2010 Supports CrayPat release 5.1 and CLE release 3.1 running on Cray XT systems. 24 S–0025–10 Overview of Gemini Hardware Counters This document describes the Gemini Performance Counters and how to use them to optimize individual applications and system traf?c. Send e-mail to docs@cray.com with any comments that will help us to improve the accuracy and usability of this document. Be sure to include the title and number of the document with your comments. We value your comments and will respond to them promptly. Accessing network performance counters is desirable for application developers, system library developers (e.g. MPI), and system administrators. Application developers want to improve their application run-times or measure what affect other traf?c on the system has on their application. System library developers want to optimize their collective operations. System Administrators want to observe the system, looking for hotspots. Effective with the CrayPat (Cray performance analysis tool) version 5.1 and Cray Linux Environment (CLE) version 3.1 software releases for the Cray XE platform, users can monitor many of the performance counters that reside on the Gemini networking chip. There are two categories of Gemini performance counters available to users. NIC performance counters record information about the data moving through theNetwork Interface Controller (NIC). On the Gemini ASIC there are two NICs, each attached to a compute node. Thus, the data from the NIC performance counters re?ects network transfers beginning and ending on the node. These performance counters are read-only. Network router tile counters are available on a per-Gemini basis. There are both read-only and read/write tile counters. Each chip has 48 router tiles, arranged in a 6x8 grid. Eight processor tiles connect to each of the two Gemini NICs. Each NIC connects to a different node, running separate Linux instances. If collection at other points of the application is desired, use the CrayPat API to insert regions as described in the pat_build man page. It is recommended that you do not collect any other performance data when collecting network counters. Data collection of network counters is much more expensive than other performance data collection, and will skew other results. At the time the instrumented executable program is launched with the aprun command, a set of environment variables, PAT_RT_NWPC_*, provide access to the Gemini network performance counters. These environment variables are described in the intro_craypat man page. S–0025–10 1Using the Cray Gemini Hardware Counters 1.1 Using CrayPat to Monitor Gemini Counters The CrayPat utility pat_build instruments an executable ?le. One aspect of the instrumentation includes intercepting entries into and returns out of a function. This is known formally as tracing. Information such as time stamps and performance counter values are recorded at this time. CrayPat supports instrumentation of an application binary for collection of Gemini counters. Counter values are recorded at application runtime, and are presented to the user through a table generated by pat_report. The CrayPat user interface to request instrumentation is similar to that for processor performance counters. There is no Gemini counter display available in Cray Apprentice2 at this time. A new display will be available in a subsequent release of the Cray Apprentice2 software. Although the user interface to request network counters is similar to processor counters, there are some signi?cant differences that must be understood. Depending on the type of counters requested, some are shared across all processors within a node, some are shared between two nodes and some are shared across all applications passing through a chip. Some counters monitor all traf?c for your application, even on nodes that are not reserved for your application, and some monitor locally, that is they monitor only traf?c associated with nodes assigned to a Gemini chip and no other traf?c from the network. Users should also be aware that access to the network counters is more resource-intensive than access to the processor performance counters. Because Gemini counters are a shared resource, the system software is designed to provide dedicated access whenever possible. This is done through the Application Level Placement Scheduler (ALPS) by ensuring that an application collecting counters is not placed on the same Gemini chip as another application collecting performance counters. It does not prevent a second application from being placed on the same Gemini chip that is not collecting counters however. This compromise assures better system utilization because compute nodes are not left unavailable for use by another application. The CrayPat 5.1 release focuses on the use of the NIC and ORB counters available within the Gemini chip. The values collected from these counters are local to a node and therefore speci?c to an application. Traf?c between MPI ranks cannot be distinguished through the counters. The event names that CrayPat supports are listed at the end of this document. Network counters are only collected for the MAIN thread. Values are collected at the beginning and end of the instrumented application. Instrumentation overhead is minimal. This gives a high-level view of the program's use of the networking router in terms of the counters speci?ed. Currently the time to access counter data is too expensive to collect more frequently. A future release of CLE will address these performance limitations. 2 S–0025–10Overview of Gemini Hardware Counters Before attempting the following examples verify that your system has a Gemini network: $ module list xtpe-network-gemini Attempting to collect Gemini performance counters on a system that does not have the Gemini network will result in a fatal error: $ aprun -n 16 my_program+pat CrayPat/X: Version 5.1 Revision 3329 05/20/10 11:26:16 pat[FATAL][0]: initialization of NW performance counter API failed [No such file or directory] Example 1. Collect stalls associated with node traf?c to and from the network This example enables tracing of MAIN. $ pat_build -w my_program $ export PAT_RT_NWPC=GM_ORB_PERF_VC0_STALLED,GM_ORB_PERF_VC1_STALLED $ aprun my_program+pat Example 2. Display network counter data $ pat_report my_program+pat+11171-41tdot.xf> counter_rpt Example output from pat_report: NWPC Data by Function Group and Function Group / Function / Node Id=0='HIDE' ===================================================================== Total --------------------------------------------------------------------- Time% 100.0% Time 2.476423 secs GM_ORB_PERF_VC1_STALLED 0 GM_ORB_PERF_VC1_BLOCKED 0 GM_ORB_PERF_VC1_BLOCKED_PKT_GEN 0 GM_ORB_PERF_VC1_PKTS 48 GM_ORB_PERF_VC1_FLITS 48 GM_ORB_PERF_VC0_STALLED 111 GM_ORB_PERF_VC0_PKTS 48 GM_ORB_PERF_VC0_FLITS 201 ===================================================================== S–0025–10 3Using the Cray Gemini Hardware Counters Example 3. Collect data for a custom group of network counters In this example a user creates a group of network events in a ?le called my_nwpc_groups, one called 1 and the other called CQ_AMO: $ cat my_nwpc_groups # Group 1: Outstanding Request Buffer 1 = GM_ORB_PERF_VC1_STALLED, GM_ORB_PERF_VC1_BLOCKED, GM_ORB_PERF_VC1_BLOCKED_PKT_GEN, GM_ORB_PERF_VC1_PKTS, GM_ORB_PERF_VC1_FLITS, GM_ORB_PERF_VC0_STALLED, GM_ORB_PERF_VC0_PKTS, GM_ORB_PERF_VC0_FLITS # Group CQ_AMO: CQ_AMO = GM_AMO_PERF_COUNTER_EN, GM_AMO_PERF_CQ_FLIT_CNTR, GM_AMO_PERF_CQ_PKT_CNTR, GM_AMO_PERF_CQ_STALLED_CNTR, GM_AMO_PERF_CQ_BLOCKED_CNTR $ pat_build -w my_program $ export PAT_RT_NWPC_FILE=my_nwpc_groups $ export PAT_RT_NWPC=1,CQ_AMO $ aprun -n16 my_program+pat 4 S–0025–10Overview of Gemini Hardware Counters Example output from pat_report: NWPC Data by Function Group and Function Group / Function / Node Id=0='HIDE' ===================================================================== Total --------------------------------------------------------------------- Time% 100.0% Time 2.639046 secs GM_ORB_PERF_VC1_STALLED 72525 GM_ORB_PERF_VC1_PKTS 50457 GM_AMO_PERF_COUNTER_EN 0 GM_AMO_PERF_CQ_FLIT_CNTR 11752 GM_AMO_PERF_CQ_PKT_CNTR 5876 GM_AMO_PERF_CQ_STALLED_CNTR 5092 GM_AMO_PERF_CQ_BLOCKED_CNTR 29 ===================================================================== Example 4. Suppress instrumented entry points from recording performance data to reduce overhead This example assumes a NWPC group FMAS exists and is available for use. Because the program is traced, the PAT_RT_TRACE_FUNCTION_NAME is set to suppress any data collection by already instrumented entry points in my_program+pat. This means that NWPC values will only be recorded for the MAIN thread at the start and the end of the instrumented program. Instrumentation overhead is minimal. $ pat_build -u -g mpi my_program $ export PAT_RT_NWPC=FMAS $ export PAT_RT_TRACE_FUNCITON_NAME=*:0 $ aprun -n32 my_program+pat This gives a high-level view of the program's use of the networking router in terms of what the FMAS group describes. If more details about NWPC use during execution of the program are desired, the PAT_RT_TRACE_FUNCTION_NAME environment variable need not be set, but the signi?cant overhead injected by reading the NWPCs may make the resulting performance data inaccurate. To selectively collect NWPCs and the other performance data for traced functions, add them to the end of PAT_RT_TRACE_FUNCTION_NAME: $ export PAT_RT_TRACE_FUNCTION_NAME=0:*,mxm,MPI_Bcast S–0025–10 5Using the Cray Gemini Hardware Counters 1.2 Gemini NIC Counters To better understand how to use the NIC counters, you need to understand some of the terminology speci?c to the Gemini network architecture. The Block Transfer Engine (BTE) A Gemini network packet typically consists of one or more ?its, which are the units of ?ow control for the network. Because ?its are usually larger than the physical datapath, they are divided into phits, which are the units of data that the network can handle physically. A packet must contain at least two phits, one for the header and one for the cyclical redundancy check (CRC). The V0 counters support the request channel and the V1 counters support the response channel. A ?it/pkt ratio can tell the user if the data entering the network was not aligned, eg a ratio greater than 1 indicates misaligned data is being sent across the network. Because there is a bandwidth/pipe size difference between outgoing and incoming (outgoing is smaller), in general you will notice more stalls on the V0 (request) channel. The following counters are recommended as a way to begin using the Gemini NWPC: GM_ORB_PERF_VC0_STALLED GM_ORB_PERF_VC1_STALLED GM_ORB_PERF_VC0_PKTS GM_ORB_PERF_VC1_PKTS GM_ORB_PERF_VC0_FLITS GM_ORB_PERF_VC1_FLITS Table 1. Atomic Memory Operations Performance Counters Name Description GM_AMO_PERF_ACP_COMP_CNTR Number of Atomic Memory Operation (AMO) computations that have occurred. GM_AMO_PERF_ACP_MEM_UPDATE_CNTR Number of AMO logic cache write-throughs that have occurred. GM_AMO_PERF_ACP_STALL_CNTR Number of AMO logic pipeline stalls that have occurred. GM_AMO_PERF_AMO_HEADER_CNTR Number of request headers processed by the Decode Logic that have had an AMO computation. Error packets are not counted. GM_AMO_PERF_COUNTER_EN When set, counting is enabled. When cleared, counting is disabled. GM_AMO_PERF_CQ_BLOCKED_CNTR Number of cycles the CQ FIFO is blocked. 6 S–0025–10Overview of Gemini Hardware Counters Name Description GM_AMO_PERF_CQ_FLIT_CNTR Number of ?its (network ?ow control units) that are read from the CQ FIFO. GM_AMO_PERF_CQ_PKT_CNTR Number of packets that are read from the CQ FIFO. GM_AMO_PERF_CQ_STALLED_CNTR Number of cycles the CQ FIFO is stalled. GM_AMO_PERF_DONE_INV_CNTR Number of times a valid cache entry was invalidated because there were no more outstanding AMO requests targeting it and the last request did not have the cacheable bit set. GM_AMO_PERF_ERROR_HEADER_CNTR Number of request headers processed by the Decode Logic that have had errors. GM_AMO_PERF_FLUSH_HEADER_CNTR Number of request headers processed by the Decode Logic that have had a Flush command. Error packets are not counted. GM_AMO_PERF_FULL_INV_CNTR Number of times a valid but inactive cache entry was invalidated to make room for a new AMO address. A high value in this counter indicates that there are too many cacheable AMO addresses and that the cache is being thrashed. GM_AMO_PERF_GET_HEADER_CNTR Number of request headers processed by the Decode Logic that have had an GET command. Error packets are not counted. GM_AMO_PERF_MSGCOMP_HEADER_CNTR Number of request headers processed by the Decode Logic that have had a MsgComplete command. Error packets are not counted. GM_AMO_PERF_PUT_HEADER_CNTR Number of request headers processed by the Decode Logic that have had an PUT command. Error packets are not counted. GM_AMO_PERF_REQLIST_FULL_STALL_CNTR Number of times an AMO request causes the NRP to stall waiting for a Request List entry to become free. GM_AMO_PERF_RMT_BLOCKED_CNTR Number cycles the RMT FIFO is blocked GM_AMO_PERF_RMT_FLIT_CNTR Number of ?its that are read from the RMT FIFO GM_AMO_PERF_RMT_PKT_CNTR Number of packets that are read from the RMT FIFO GM_AMO_PERF_RMT_STALLED_CNTR Number cycles the RMT FIFO is stalled S–0025–10 7Using the Cray Gemini Hardware Counters Name Description GM_AMO_PERF_TAG_HIT_CNTR Number of AMO requests that have been processed in the Tag Store and have resulted in a cache hit. GM_AMO_PERF_TAG_MISS_CNTR Number of AMO requests that have been processed in the Tag Store and have resulted in a cache miss. GM_AMO_PERF_TAG_STALL_CNTR Number of times a GET/PUT request hits in the cache and causes the NRP to stall. Table 2. Fast Memory Access Performance Counters Name Description GM_FMA_PERF_CQ_PKT_CNT Number of packets from Fast Memory Access (FMA) to CQ. GM_FMA_PERF_CQ_STALLED_CNT Number of clock cycles FMA_CQ was stalled due to lack of credits. GM_FMA_PERF_HT_NP_REQ_FLIT_CNT Number of HT NP request ?its to FMA. GM_FMA_PERF_HT_NP_REQ_PKT_CNT Number of HT NP request packets to FMA. GM_FMA_PERF_HT_P_REQ_FLIT_CNT Number of HT P request ?its to FMA. GM_FMA_PERF_HT_P_REQ_PKT_CNT Number of HT P request packets to FMA. GM_FMA_PERF_HT_RSP_PKT_CNT Number of HT response packets from FMA to HT. GM_FMA_PERF_HT_RSP_STALLED_CNT Number of clock cycles FMA_HT_RSP was stalled due to lack of credits. GM_FMA_PERF_TARB_FLIT_CNT Number of ?its from FMA to TARB. GM_FMA_PERF_TARB_PKT_CNT Number of packets from FMA to TARB. GM_FMA_PERF_TARB_STALLED_CNT Number of clock cycles FMA_TARB was stalled due to lack of credits. 8 S–0025–10Overview of Gemini Hardware Counters Table 3. Hyper-transport Arbiter Performance Counters Name Description GM_HARB_PERF_AMO_NP_BLOCKED Number of times AMO Non-Posted Queue has an entry, but is blocked from using the Non-Posted Initiator Request output channel by the BTE Non-Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_NP_FLITS Number of ?its coming out of the AMO Non-Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_NP_PKTS Number of packets coming out of the AMO Non-Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_NP_STALLED Number of cycles the AMO Non-Posted Queue is stalled due to a lack credits on the Non-Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_P_ACP_BLOCKED Number of times AMO Posted AMO Computation Pipe Queue has an entry, but is blocked from using the Posted Initiator Request output channel by another Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_P_ACP_FLITS Number of ?its coming out of the AMO Posted AMO Computation Pipe Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). S–0025–10 9Using the Cray Gemini Hardware Counters Name Description GM_HARB_PERF_AMO_P_ACP_PKTS Number of packets coming out of the AMO Posted AMO Computation Pipe Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_P_ACP_STALLED Number of cycles the AMO Posted AMO Computation Pipe Queue is stalled due to a lack credits on the Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_P_NRP_BLOCKED Number of times AMO Posted New Request Pipe Queue has an entry, but is blocked from using the Posted Initiator Request output channel by another Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_P_NRP_FLITS Number of ?its coming out of the AMO Posted New Request Pipe Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_P_NRP_PKTS Number of packets coming out of the AMO Posted New Request Pipe Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_AMO_P_NRP_STALLED Number of cycles the AMO Posted New Request Pipe Queue is stalled due to a lack credits on the Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). 10 S–0025–10Overview of Gemini Hardware Counters Name Description GM_HARB_PERF_BTE_NP_BLOCKED Number of times AMO Non-Posted BTE Queue has an entry, but is blocked from using the Non-Posted Initiator Request output channel by another Non-Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_BTE_NP_FLITS Number of ?its coming out of the AMO Non-Posted BTE Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_BTE_NP_PKTS Number of packets coming out of the AMO Non-Posted BTE Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_BTE_NP_STALLED Number of cycles the AMO Non-Posted BTE Queue is stalled due to a lack credits on the Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_BTE_P_BLOCKED Number of times AMO Posted BTE Queue has an entry, but is blocked from using the Posted Initiator Request output channel by another Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_BTE_P_FLITS Number of ?its coming out of the AMO Posted BTE Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). S–0025–10 11Using the Cray Gemini Hardware Counters Name Description GM_HARB_PERF_BTE_P_PKTS Number of packets coming out of the AMO Posted BTE Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_BTE_P_STALLED Number of cycles the AMO Posted BTE Queue is stalled due to a lack credits on the Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_COUNTER_EN When set, counting is enabled. When clear, counting is disabled. This MMR is reset by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_IREQ_NP_FLITS Number of ?its on the non-posted initiator request output of the HARB block. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_IREQ_NP_PKTS Number of packets on the non-posted initiator request output of the HARB Block. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_IREQ_NP_STALLED Number of cycles on the non-posted initiator request output of the HARB is stalled due to a lack credits on the Non-Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). 12 S–0025–10Overview of Gemini Hardware Counters Name Description GM_HARB_PERF_IREQ_P_FLITS Number of ?its on the posted initiator request output of the HARB block. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_IREQ_P_PKTS Number of packets on the posted initiator request output of the HARB Block. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_IREQ_P_STALLED Number of cycles on the posted initiator request output of the HARB is stalled due to a lack credits on the Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_RAT_P_BLOCKED Number of times AMO Posted RAT Queue has an entry, but is blocked from using the Posted Initiator Request output channel by another Posted Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_RAT_P_FLITS Number of ?its coming out of the AMO Posted RAT Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). S–0025–10 13Using the Cray Gemini Hardware Counters Name Description GM_HARB_PERF_RAT_P_PKTS Number of packets coming out of the AMO Posted RAT Queue. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). GM_HARB_PERF_RAT_P_STALLED Number of cycles the AMO Posted RAT Queue is stalled due to a lack credits on the Posted Initiator Request channel. The Local Block has read/write access to the full counter. Bits 63:48 of this MMR are unimplemented and always return zero. This MMR is reset to all zeros by the chip reset (i_reset), but not by HT reset (i_ht_reset). Table 4. Network Address Translation Performance Counters Name Description GM_NAT_PERF_BTE_BLOCKED Number of cycles a BTE translation is blocked due to arbitration loss. GM_NAT_PERF_BTE_STALLED Number of cycles a BTE translation is stalled due to MMR access. GM_NAT_PERF_BTE_TRANSLATIONS Number of translations performed for the BTE interface. GM_NAT_PERF_COUNTER_EN When set, counting is enabled. When cleared, counting is disabled. GM_NAT_PERF_REQ_BLOCKED Number of cycles a REQ translation is blocked due to arbitration loss. GM_NAT_PERF_REQ_STALLED Number of cycles a REQ translation is stalled due to MMR access. GM_NAT_PERF_REQ_TRANSLATIONS Number of translations performed for the REQ interface. GM_NAT_PERF_RSP_BLOCKED Number of cycles a RSP translation is blocked due to arbitration loss. GM_NAT_PERF_RSP_STALLED Number of cycles a RSP translation is stalled due to MMR access. GM_NAT_PERF_RSP_TRANSLATIONS Number of translations performed for the RSP interface. GM_NAT_PERF_TRANS_ERROR0 Number of translations that failed due to error 0 (Uncorrectable error in translation). 14 S–0025–10Overview of Gemini Hardware Counters Name Description GM_NAT_PERF_TRANS_ERROR1 Number of translations that failed due to error 1 (VMDH table invalid entry). GM_NAT_PERF_TRANS_ERROR2 Number of translations that failed due to error 2 (MDDT/MRT invalid or illegal entry). GM_NAT_PERF_TRANS_ERROR3 Number of translations that failed due to error 3 (Protection tag violation). GM_NAT_PERF_TRANS_ERROR4 Number of translations that failed due to error 4 (memory bounds error). GM_NAT_PERF_TRANS_ERROR5 Number of translations that failed due to error 5 (write permission error) Table 5. Netlink Performance Counters Name Description GM_NL_PERF_ALL_LCBS_REQS_TO_NIC_0_STALLED Number of ticks all LCBs requests have stalled to NIC 0. GM_NL_PERF_ALL_LCBS_REQS_TO_NIC_1_STALLED Number of ticks all LCBs requests have stalled to NIC 1. GM_NL_PERF_ALL_LCBS_RSP_TO_NIC_0_STALLED Number of ticks all LCBs responses have stalled to NIC 0. GM_NL_PERF_ALL_LCBS_RSP_TO_NIC_1_STALLED Number of ticks all LCBs responses have stalled to NIC 1. GM_NL_PERF_CNTRL Controls the performance counters. Writing a 1 to the Start ?eld starts the counters. Writing a 1 to the Stop ?eld stops the counters. Writing a 1 to the Clear ?eld clears the counters. GM_NL_PERF_LCB_n_REQ_CMP_22 Decompressed request data to two phit LCB_n, where n is a value from 0 to 7 that speci?es the LCB. GM_NL_PERF_LCB_n_REQ_CMP_44 Decompressed request data to one phit LCB_n, where n is a value from 0 to 7 that speci?es the LCB. GM_NL_PERF_LCB_n_REQ_TO_NIC_0 Number of requests from LCB_n to NIC 0. GM_NL_PERF_LCB_n_REQ_TO_NIC_0_STALLED Number of ticks LCB_n requests are blocked to NIC 0. GM_NL_PERF_LCB_n_REQ_TO_NIC_1 Number of requests from LCB_n to NIC 1. S–0025–10 15Using the Cray Gemini Hardware Counters Name Description GM_NL_PERF_LCB_n_REQ_TO_NIC_1_STALLED Number of ticks LCB_n requests are blocked to NIC 1. GM_NL_PERF_LCB_n_REQ_TO_PHITS Number of request phits received on LCB_n. GM_NL_PERF_LCB_n_REQ_TO_PKTS Number of request packets received on LCB_n. GM_NL_PERF_LCB_n_RSP_CMP_22 Decompressed response data to two phit LCB_n GM_NL_PERF_LCB_n_RSP_TO_NIC_1 Number of responses from LCB_n to NIC 1. GM_NL_PERF_LCB_n_RSP_TO_NIC_1_STALLED Number of ticks LCB_n responses are blocked to NIC 1. GM_NL_PERF_NIC_0_REQ_STALLED_TO_ALL_LCBS Number of ticks NIC_0 requests are blocked to all LCBs. GM_NL_PERF_NIC_0_REQ_TO_LCB_n Number of requests from NIC_0 LCB_ n. GM_NL_PERF_NIC_0_REQ_TO_LCB_n_STALLED Number of ticks NIC_0 requests are blocked to LCB_n. GM_NL_PERF_NIC_0_RSP_STALLED_TO_ALL_LCBS Number of ticks NIC_0 responses are blocked to all LCBs. GM_NL_PERF_NIC_0_RSP_TO_LCB_n Number of responses from NIC_0 LCB_ n. GM_NL_PERF_NIC_0_RSP_TO_LCB_n_STALLED Number of ticks NIC_0 responses are blocked to LCB_n. GM_NL_PERF_NIC_1_REQ_STALLED_TO_ALL_LCBS Number of ticks NIC_0 requests are blocked to all LCBs. GM_NL_PERF_NIC_1_REQ_TO_LCB_n Number of requests from NIC_1 to LCB_ n. GM_NL_PERF_NIC_1_REQ_TO_LCBn_STALLED Number of ticks NIC_1 requests are blocked to LCB_n. GM_NL_PERF_NIC_1_RSP_STALLED_TO_ALL_LCBS Number of ticks NIC_1 responses are blocked to all LCBs. GM_NL_PERF_NIC_1_RSP_TO_LCB_n Number of responses from NIC_1 LCB_ n. GM_NL_PERF_NIC_1_RSP_TO_LCB_n_STALLED Number of ticks NIC_1 responses are blocked to LCB_n. 16 S–0025–10Overview of Gemini Hardware Counters Table 6. NPT Performance Counters Name Description GM_NPT_PERF_ACP_BLOCKED_CNTR Number of cycles the ACP FIFO is blocked. GM_NPT_PERF_ACP_FLIT_CNTR Number of ?its that are read from the ACP FIFO. GM_NPT_PERF_ACP_PKT_CNTR Number of packets that are read from the ACP FIFO. GM_NPT_PERF_ACP_STALLED_CNTR Number of cycles the ACP FIFO is stalled. GM_NPT_PERF_BTE_RSP_PKT_CNTR Number of packets that are sent to the Netlink as Get or Flush responses. GM_NPT_PERF_COUNTER_EN Provides the count enable. GM_NPT_PERF_FILL_RSP_PKT_CNTR Number of packets that are sent to the AMO block as ?ll responses. GM_NPT_PERF_HTIRSP_ERR_CNTR Number of packets that are received from the HT cave and have an error status. GM_NPT_PERF_HTIRSP_FLIT_CNTR Number of ?its that are received from the HT cave. GM_NPT_PERF_HTIRSP_PKT_CNTR Number of packets that are received from the HT cave. GM_NPT_PERF_LB_BLOCKED_CNTR Number of cycles the LB FIFO is blocked. GM_NPT_PERF_LB_FLIT_CNTR Number of ?its that are read from the LB FIFO. GM_NPT_PERF_LB_PKT_CNTR Number of packets that are read from the LB FIFO. GM_NPT_PERF_LB_STALLED_CNTR Number of cycles the LB FIFO is stalled. GM_NPT_PERF_NL_RSP_PKT_CNTR Number of packets that are sent to the AMO block as ?ll responses. GM_NPT_PERF_NPT_BLOCKED_CNTR Number of cycles the NPT FIFO is blocked. GM_NPT_PERF_NPT_FLIT_CNTR Number of ?its that are read from the NPT FIFO. GM_NPT_PERF_NPT_PKT_CNTR Number of packets that are read from the NPT FIFO. GM_NPT_PERF_NPT_STALLED_CNTR Number of cycles the NPT FIFO is stalled. GM_NPT_PERF_NRP_BLOCKED_CNTR Number of cycles the NRP FIFO is blocked. GM_NPT_PERF_NRP_FLIT_CNTR Number of ?its that are read from the NRP FIFO. GM_NPT_PERF_NRP_PKT_CNTR Number of packets that are read from the NRP FIFO. GM_NPT_PERF_NRP_STALLED_CNTR Number of cycles the NRP FIFO is stalled. S–0025–10 17Using the Cray Gemini Hardware Counters Table 7. ORB Performance Counters Name Description GM_ORB_PERF_VC0_FLITS Number of ?its to come into the TX Input Queue from the SSID. GM_ORB_PERF_VC0_PKTS Number of packets to come into the TX Input Queue from the SSID. GM_ORB_PERF_VC0_STALLED Number of packets not given access to the TX Control Logic because there is not enough credits available from the NL Block, or there are no available memory locations from the ORD RAM, or a tail ?it has not been received in the ORB Input Queue when performing store-and-forward. GM_ORB_PERF_VC1_BLOCKED Number of packets not given access to the RX Control Logic because the read address and write address into the ORD RAM are attempting to access the same bank of the ORD RAM or because there is a read access to the ORD RAM from the Local Block. GM_ORB_PERF_VC1_BLOCKED_PKT_GEN Number of times the RX Response FIFO is blocked because a packet in the RX Control Logic is being translated into the format used by the rest of the NIC. GM_ORB_PERF_VC1_FLITS Number of ?its to come into the Receive Response FIFO from the network. GM_ORB_PERF_VC1_PKTS Number of packets to come into the Receive Response FIFO from the network. GM_ORB_PERF_VC1_STALLED Number of packets not given access to the RX Control Logic because there is not enough credits available from the RAT. 18 S–0025–10Overview of Gemini Hardware Counters Table 8. RAT Performance Counters Name Description GM_RAT_PERF_COUNTER_EN Enables the performance counters. GM_RAT_PERF_DATA_FLITS_VC0 Number of data ?its received on VC0 (request pipeline). GM_RAT_PERF_DATA_FLITS_VC1 Number of data ?its received on VC1 (request pipeline). GM_RAT_PERF_HEADER_FLITS_VC0 Number of header ?its received on VC0 (request pipeline). GM_RAT_PERF_HEADER_FLITS_VC1 Number of header ?its received on VC1 (request pipeline). GM_RAT_PERF_STALLED_CREDITS_VC0 Number of cycles VC0 (request pipeline) is stalled due to insuf?cient credits. GM_RAT_PERF_STALLED_CREDITS_VC1 Number of cycles VC1 (request pipeline) is stalled due to insuf?cient credits. GM_RAT_PERF_STALLED_TRANSLATION_VC0 Number of cycles VC0 (request pipeline) is stalled due to unavailable translation data. GM_RAT_PERF_STALLED_TRANSLATION_VC1 Number of cycles VC1 (request pipeline) is stalled due to unavailable translation data. GM_RAT_PERF_TRANSLATION_ERRORS_VC0 Number of translation errors seen on VC0 (request pipeline). GM_RAT_PERF_TRANSLATION_ERRORS_VC1 Number of translation errors seen on VC1 (request pipeline). GM_RAT_PERF_TRANSLATIONS_VC0 Number of translations requested on VC0 (request pipeline). GM_RAT_PERF_TRANSLATIONS_VC1 Number of translations requested on VC1 (request pipeline). S–0025–10 19Using the Cray Gemini Hardware Counters Table 9. RMT Performance Counters Name Description GM_RMT_PERF_PUT_BYTES_RX Tally of bytes received in all PUT packets that had the RMT Enable ?eld set that entered and exited the RMT with OK status. GM_RMT_PERF_PUT_CAM_EVIT PUT sequences evicted from the CAM. GM_RMT_PERF_PUT_CAM_FILL New PUT sequence packet arrived and successfully allocated in the CAM. GM_RMT_PERF_PUT_CAM_HITS Packet for PUT sequence currently stored in RMT arrived and successfully located entry in CAM. GM_RMT_PERF_PUT_CAM_MISS New PUT sequence packet arrived, but did not allocate because CAM was full. GM_RMT_PERF_PUT_PARITY Number of sequences evicted from CAM due to uncorrectable parity errors. GM_RMT_PERF_PUT_RECV_COMPLETE Number of MsgRcvComplete packets received which evicted a CAM entry. GM_RMT_PERF_PUT_TIMEOUTS Number of sequences evicted from CAM due to timeout. GM_RMT_PERF_SEND_BYTES_RX Tally of bytes received in all SEND packets that had the RMT Enable ?eld set and entered and exited the RMT with OK status. GM_RMT_PERF_SEND_CAM_EVIT SEND sequences evicted from the CAM. GM_RMT_PERF_SEND_CAM_FILL New SEND sequence packet arrived and successfully allocated in the CAM. GM_RMT_PERF_SEND_CAM_HITS Packet for SEND sequence currently stored in RMT arrived and successfully located entry in CAM. GM_RMT_PERF_SEND_CAM_MISS New SEND sequence packet arrived, but did not allocate because CAM was full. GM_RMT_PERF_SEND_PARITY Number of sequences evicted from CAM due to uncorrectable parity errors. GM_RMT_PERF_SEND_ABORTS Number of SEND sequences that were aborted. GM_RMT_PERF_SEND_TIMEOUTS Number of sequences evicted from CAM due to timeout. 20 S–0025–10Overview of Gemini Hardware Counters Table 10. SSID Performance Counters Name Description GM_SSID_PERF_COMPLETION_COUNT_1 Provides a count of completed request packet sequences. The type of sequence completions counted by this register is controlled by the SSID Performance – Completion Count Selector Register. GM_SSID_PERF_COMPLETION_COUNT_2 Provides a count of completed request packet sequences. The type of sequence completions counted by this register is controlled by the SSID Performance – Completion Count Selector Register. GM_SSID_PERF_COMPLETION_COUNT_SELECTOR Speci?es the types of completion events that are counted in the SSID Performance – Completion Count 1 Register (bits 3-0) and the SSID Performance – Completion Count 2 Register (bits 11-8). See the table of SSID_PerfCompletionCountSelect Encoding values for encoding of these ?elds. GM_SSID_PERF_OUT_STALLED_DURATION The accumulated number of cycles of cclk for which the SSID had a valid ?it available to send to the ORB but sending of the ?it had to be stalled while waiting for a credit from the ORB. This value is cleared by writing any value to this register. GM_SSID_PERF_OUTOFSSIDS_COUNT The number of Allocate SSID requests that have been received for which processing of the request had to be stalled for one or more clock cycles because a free SSID was not immediately available to service the request. This value is cleared by writing any value to this register. GM_SSID_PERF_OUTOFSSIDS_DURATION The accumulated number of cycles of cclk for which processing of Allocate SSID requests has been stalled because a free SSID is not available to service the request. This value is cleared by writing any value to this register. S–0025–10 21Using the Cray Gemini Hardware Counters Name Description GM_SSID_PERF_SSID_ALLOCATE_COUNT The total number of Allocate SSID requests that have been received, across all channels (all FMA descriptors and all BTE VCs), because this register was last cleared, and that resulted in a SSID actually being allocated. Allocate SSID requests that do not result in a SSID being allocated (i.e. redundant Allocate requests) are not counted. This value is cleared by writing any value to this register. GM_SSID_PERF_SSIDS_IN_USE Bits 7-0 specify the number of SSIDs currently in use across all Request Channels. This value is not affected by writes to this register. This ?eld is initialized to its reset value by a full reset and by an ht reset. Bits 23-16 specify the maximum number of SSIDs that have been in use simultaneously, across all channels (all FMA descriptors and all BTE Vcs), since this register was last initialized. This value is initialized to CurrentSSIDsInUse by writing any value to this register. This ?eld is initialized to its reset value by a full reset. Table 11. Transmit Arbiter Performance Counters Name Description GM_TARB_PERF_BTE_BLOCKED Transmit Arbiter (TARB) Performance BTE Blocked Count GM_TARB_PERF_BTE_FLITS TARB Performance BTE Flit Count GM_TARB_PERF_BTE_PKTS TARB Performance BTE Packet Count GM_TARB_PERF_BTE_STALLED TARB Performance BTE Stalled Count GM_TARB_PERF_FMA_BLOCKED TARB Performance FMA Blocked Count GM_TARB_PERF_FMA_FLITS TARB Performance FMA Flit Count GM_TARB_PERF_FMA_PKTS TARB Performance FMA Packet Count GM_TARB_PERF_FMA_STALLED TARB Performance FMA Stalled Count GM_TARB_PERF_LB_BLOCKED TARB Performance LB Blocked Count GM_TARB_PERF_LB_FLITS TARB Performance LB Flit Count GM_TARB_PERF_LB_PKTS TARB Performance LB Packet Count 22 S–0025–10Overview of Gemini Hardware Counters Name Description GM_TARB_PERF_LB_STALLED TARB Performance LB Stalled Count GM_TARB_PERF_OUT_FLITS TARB Performance Output Flit Count GM_TARB_PERF_OUT_PKTS TARB Performance Output Packet Count GM_TARB_PERF_OUT_STALLED TARB Performance Output Stalled Count 1.3 Gemini Tile MMRs The Gemini network consists of 48 tiles, arranged in 6 rows of 8 columns. Within each tile there are memory-mapped registers associated with the LCB and with the rest of the tile. The local block has shared connections to each row of tiles. By default, when only the name of the MMR is used, an event is counted on all 48 tiles. To address an individual tile, append the row (0-5) and column (0-7) to the name, as shown in the table. Table 12. Description of Gemini Tile MMRs Name Description GM_TILE_PERF_VC0_PHIT_CNT:n:m Number of vc0 phits read from inq buffer GM_TILE_PERF_VC1_PHIT_CNT:n:m Number of vc1 phits read from inq buffer GM_TILE_PERF_VC0_PKT_CNT:n:m Number of vc0 packets read from inq buffer GM_TILE_PERF_VC10_PKT_CNT:n:m Number of vc1 packets read from inq buffer GM_TILE_PERF_INQ_STALL:n:m Number of clock periods a valid reference is blocked from the routing pipeline. GM_TILE_PERF_CREDIT_STALL:n:m Number of clock periods a valid reference is stalled in the column buffers, waiting on transmissions credits. S–0025–10 23Using the Cray Gemini Hardware Counters © 2010 Cray Inc. All Rights Reserved. This document or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Inc. Cray, LibSci, PathScale, and UNICOS are federally registered trademarks and Active Manager, Baker, Cascade, Cray Apprentice2, Cray Apprentice2 Desktop, Cray C++ Compiling System, Cray CX, Cray CX1, Cray CX1-iWS, Cray CX1-LC, Cray CX1000, Cray CX1000-C, Cray CX1000-G, Cray CX1000-S, Cray CX1000-SC, Cray CX1000-SM, Cray CX1000-HN, Cray Fortran Compiler, Cray Linux Environment, Cray SHMEM, Cray X1, Cray X1E, Cray X2, Cray XD1, Cray XE, Cray XE6, Cray XMT, Cray XR1, Cray XT, Cray XTm, Cray XT3, Cray XT4, Cray XT5, Cray XT5 h , Cray XT5m, Cray XT6, Cray XT6m, CrayDoc, CrayPort, CRInform, ECOphlex, Gemini, Libsci, NodeKARE, RapidArray, SeaStar, SeaStar2, SeaStar2+, Threadstorm, UNICOS/lc, UNICOS/mk, and UNICOS/mp are trademarks of Cray Inc. Version 1.0 Published July 2010 Supports CrayPat release 5.1 and CLE release 3.1 running on Cray XT systems. 24 S–0025–10 Using the Cray XMT™ for all streams Pragmas Abstract This document describes the for all streams compiler directives and how to use them to execute a block of code on multiple streams.© 2010 Cray Inc. All Rights Reserved. This document or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Inc. Cray, LibSci, and PathScale are federally registered trademarks and Active Manager, Baker, Cascade, Cray Apprentice2, Cray Apprentice2 Desktop, Cray C++ Compiling System, Cray CX, Cray CX1, Cray CX1-iWS, Cray CX1-LC, Cray CX1000, Cray CX1000-C, Cray CX1000-G, Cray CX1000-S, Cray CX1000-SC, Cray CX1000-SM, Cray CX1000-HN, Cray Fortran Compiler, Cray Linux Environment, Cray SHMEM, Cray X1, Cray X1E, Cray X2, Cray XD1, Cray XE, Cray XE6, Cray XMT, Cray XR1, Cray XT, Cray XTm, Cray XT3, Cray XT4, Cray XT5, Cray XT5 h , Cray XT5m, Cray XT6, Cray XT6m, CrayDoc, CrayPort, CRInform, ECOphlex, Gemini, Libsci, NodeKARE, RapidArray, SeaStar, SeaStar2, SeaStar2+, Threadstorm, and UNICOS/lc are trademarks of Cray Inc. UNIX, the “X device,” X Window System, and X/Open are trademarks of The Open Group in the United States and other countries. All other trademarks are the property of their respective owners. RECORD OF REVISION S–0038–14 Published October 2010 Supports 1.4 and later releases running on the Cray XMT hardware.Using the Cray XMT™ for all streams Pragmas Using the Cray XMT for all streams Pragmas Overview In some programming situations it is useful to specify that a block of code should execute exactly once on each stream of a parallel region, allowing the application to manage data on a per-thread basis. Effective with the 1.4 release two pragma compiler directives were added that support this. Description The syntax of the for all streams pragmas is as follows: #pragma mta for all streams This directive starts up a parallel region (if the code is not already in a parallel region) and cause the next statement or block of statements to be executed exactly once on every stream allocated to the region. If the pragmas appear in code that would otherwise not be parallel, they cause it to go parallel. For example, #pragma mta for all streams printf("Stream checking in\n"); would cause every stream to print the phrase "Stream checking in" once. In this example the pragma executes a block of code that increments a counter before printing the phrase: int counter = 0; #pragma mta for all streams { counter++; printf("%d streams checked in \n", counter) }; #pragma mta for all streams i of n This directive is similar to the for all streams pragma except that it also sets the variable n to the total number of streams executing the region, and the variable i to a unique per-stream identifier between 0 and n-1. For example: int i, n; int check_in_array[MAX_PROCESSORS * MAX_STREAMS_PER_PROCESSOR]; for (int i = 0; i < MAX_PROCESSORS * MAX_STREAMS_PER_PROCESSOR; i++) check_in_array[i] = 0; #pragma mta for all streams i of n { check_in_array[i] = 1; printf("Stream %d of %d checked in.\n", i, n); } Note that the integer variables i and n must be declared separately from the pragma. S–0038–14 3Using the Cray XMT™ for all streams Pragmas You can use the for all streams pragmas in conjunction with the use n streams pragma to ask the compiler to allocate a certain number of streams per processor to the parallel region executing the for all streams block. #pragma mta use 100 streams #pragma mta for all streams {// do something } Be aware, however, that there is no guarantee that the runtime will grant the requested number of streams. For example, sufficient streams may not be available due to other jobs, the OS, or other simultaneous parallel regions in the current job. Examples In the following example, taken from a breadth-first search procedure, the for all streams pragma is used to divide a data structure between threads. int processQueue(int *Q,unsigned &head, unsigned &tail, unsigned qcap, const Neighbor neighbors[], const int numNeighbors[], sync int *Marked) { #pragma mta trace "process" #pragma mta noalias *Q, *Marked, *neighbors, *numNeighbors // elements [head,tail) are readonly // we can write to other elements of Q const unsigned oldtail = tail; const unsigned oldhead = head; unsigned newhead = head; unsigned stubbed = 0; #pragma mta use 100 streams #pragma mta for all streams { unsigned outhead = 0, outtail = 0; for(;;) { // grab INBLOCK nodes (& stubs) from the input unsigned inhead = int_fetch_add(&newhead, INBLOCK); // avoid overrun unsigned intail = std::min(inhead + INBLOCK, oldtail); if (inhead>=intail) break; // stop if we ran out of work #pragma mta assert nodep *Q,*numNeighbors,*neighbors for(int i=inhead; i=0) { int begin = numNeighbors[u]; // |N| int end = numNeighbors[u+1]; // |N| #pragma mta assert nodep *Q, *neighbors, *Marked for(int j=begin;j=outtail) { outhead = int_fetch_add(&tail, OUTBLOCK); outtail = outhead+OUTBLOCK; } Q[(outhead++)%qcap] = v; // |N| }else { Marked[v] = mark; // unlock & keep mark } } } } } #ifdef PHASES stubbed += outtail-outhead; #endif // stub-out the rest of reserved space // ), where is the number of streams the compiler requests.Limiting Loop Parallelism in Cray XMT™ Application S–0027–14 Cray Inc. 7 ? Limits the number of processors used by a multiprocessor parallel loop to max(1, c / ), where is the number of streams the compiler requests for each processor used by the parallel loop. ? If c is larger than or equal to , the total number of streams used by the parallel loop will be at most c. ? If c is less than , one processor will be used and streams will be requested by the compiler. ? Limits the number of futures created for a loop that uses loop future parallelism to c. ? If multiple max concurrency c pragmas are specified on one loop, the value of c specified by the last pragma will be used. ? For collapsible loop nests, the max concurrency value specified by the outer loop (if any) will be used for the collapsed loop. ? The max concurrency c pragma is not allowed to be used on a loop that also uses the use n streams pragma. Examples The following example illustrates using the max concurrency c pragma on a single processor parallel loop. /* Use at most 95 streams. */ #pragma mta loop single processor #pragma mta max concurrency 95 for(i = 0; i < size; i++) { array[i] += array[i] + (size + i); } The following example illustrates using the max concurrency c pragma on a multiprocessor parallel loop. /* Use at most 512 streams across all processors. */ #pragma mta max concurrency 512 for(i = 0; i < size; i++) { array[i] += array[i] + (size + i); }Limiting Loop Parallelism in Cray XMT™ Application S–0027–14 Cray Inc. 8 The following example illustrates using the max concurrency c pragma on a loop that uses loop future parallelism. /* Create at most 512 futures. */ #pragma mta loop future #pragma mta max concurrency 512 for(i = 0; i < size; i++) { array[i] += array[i] + (size + i); } Multiprocessor parallel loops are allowed to use both the max n processors and max concurrency c pragmas, and can use both on a single loop. In cases where both pragmas are used, the lower bound of the number of processors estimated by the two limits will be the limit used on the loop. For example, the following code illustrates the use of both pragmas on one multiprocessor parallel loop. /* Use at most 512 streams across all processors or * at most 8 processors, whichever is smaller. */ #pragma mta max concurrency 512 #pragma mta max 8 processors for(i = 0; i < size; i++) { array[i] += array[i] + (size + i); } In the above example, if the compiler were to request 64 streams per processor, then the max concurrency 512 would estimate that 8 processors should be used for the loop (i.e., 512/64). The max 8 processors has the same limit on the number of processors so the loop would be limited to 8 processors. If the compiler instead requested 32 streams per processor, then the max concurrency 512 would estimate that 16 processors should be used, which is more than the limit of 8 specified by the max 8 processors, so the loop would be limited to 8 processors. Because the use n streams pragma cannot be used on the same loop as a max concurrency c pragma, the loop will use the default number of streams determined by the compiler. The user will need to look at the canal details for a loop to determine the default number of streams being requested by the compiler. Effect of Pragmas on Loop Fusion and Parallel Region Merging The new pragmas can prevent the compiler from fusing loops if the loops involved do not have the same limits for the max processors and max concurrency. This is because the compiler will need to put the loops into different parallel regions in order to limit the processors and/or concurrency as requested by the user. This could potentially have a negative impact on the performance of a user's application, so users may need to look at the canal output to see what loops the compiler fused.Limiting Loop Parallelism in Cray XMT™ Application S–0027–14 Cray Inc. 9 The pragmas could also prevent the compiler from merging the parallel regions for different loops into a single parallel region. The limitation for concurrency or processors specified by the new pragmas applies to the current parallel region that contains the loop with the pragmas. The compiler must ensure that all loops in a parallel region have the same limits for max processors and max concurrency. If the loops do not have matching limits, the compiler will put them in different parallel regions to ensure the user's limits on processors and/or concurrency can be correctly applied. This could potentially have a negative impact on the performance of a user's application because more time will be spent tearing down and starting new parallel regions. In the case of nested parallel regions, any limitations for concurrency or processors specified with the pragmas on either region do not affect the other region. For example, if the outer parallel region has a max 8 processors, that pragma will not affect the inner parallel region because the pragmas apply to the current parallel region only. The user can determine what loops the compiler placed in a parallel region by looking at the canal output. The “Additional Loop Details” shows which parallel region a loop is in, and the details for parallel regions state what limits for processors or concurrency (if any) are being applied to the region. The following is an example of two loops that have matching limits for max n processors that could be fused and placed into one parallel region by the compiler. #pragma mta max 64 processors for(i = 0; i < size; i++) array[i] = i; #pragma mta max 64 processors for(i = 0; i < size; i++) { array[i] += array[i] + (size + i); } The following is an example of two loops that cannot be fused or put into one parallel region because the loops specify different limits for the max processors. #pragma mta max 256 processors for(i = 0; i < size; i++) array[i] = i; #pragma mta max 512 processors for(i = 0; i < size; i++) { array[i] += array[i] + (size + i); } The following is another example of two loops that cannot be fused or put into one parallel region because the loops specify different limits for the max processors. The first loop does not use the max n processors pragma, which implies there is no user specified limit. for(i = 0; i < size; i++) array[i] = i;Limiting Loop Parallelism in Cray XMT™ Application S–0027–14 Cray Inc. 10 #pragma mta max 512 processors for(i = 0; i < size; i++) { array[i] += array[i] + (size + i); } Use Case: Applying Max Processors Pragma to GraphCT An example application that uses nested parallelism to improve system utilization and reduce contention on shared data structures is GraphCT (Graph Characterization Toolkit) [1]. GraphCT consists of multiple kernels that perform operations on a graph and the kernel focused on in this example is betweenness centrality. The betweenness centrality kernel of GraphCT is executed concurrently by a small number of threads using loop future parallelism, and each thread uses multiprocessor parallelism to compute the betweenness centrality of a node. The betweenness centrality kernel of GraphCT can see significant variance in performance due to issues with load balancing across the threads. The max n processors pragma can be used to help improve load balancing and increase utilization by evenly distributing the processors across the threads. The betweenness centrality kernel of GraphCT consists of two functions, kcentrality and kcent_core. The kcentrality function creates a small number of threads using loop future parallelism, and each of those threads calls kcent_core to compute the betweenness centrality for the nodes in the graph. Both of these functions were updated to make use of the new max n processors pragma. The changes to kcent_core are limited to applying the max n processors pragma to each parallel loop in the function. The limit for the number of processors to use per thread was determined experimentally based on the default number of threads created in kcentrality in the release version 0.4 of GraphCT, which is 20. This would give each thread approximately 6 processors on a 128P XMT system if each thread got the same number of processors. This led to trying a limit of 8 processors per thread in kcent_core. Experiments showed that using 8 processors per thread performed better than the release version of GraphCT with 20 threads and no max n processors pragmas. A power of two was chosen so the number of processors in the system could be easily divided by the number of processors used per thread. A limit of 16 processors per thread was also tested and was shown to have reasonable performance that could be very similar to the performance with a limit of 8, especially for larger graphs (scale >= 28). The following code snippets show how the max n processors pragma was used for each loop in kcent_core. In these examples, MAX_PROCS is a preprocessor macro that has been defined as 8. <...> #pragma mta max MAX_PROCS processors #pragma mta assert nodep for (j = 0; j < NV; j++) {marks[j] = sigma[NV*(K+1) + j] = 0;} <...>Limiting Loop Parallelism in Cray XMT™ Application S–0027–14 Cray Inc. 11 #pragma mta max MAX_PROCS processors #pragma mta assert nodep for (j = 0; j < (K+1)*NV; j++) { dist[j] = -1; sigma[j] = child_count[j] = 0; } <...> #pragma mta max MAX_PROCS processors #pragma mta assert no dependence #pragma mta block dynamic schedule #pragma mta use 100 streams for (j = Qstart; j < Qend; j++) { <...> #pragma mta max MAX_PROCS processors #pragma mta assert nodep #pragma mta assert no alias *sigma *Q *child *start *QHead #pragma mta use 100 streams for (n = QHead[p]; n < QHead[p+1]; n++) { <...> #pragma mta max MAX_PROCS processors for (j=0; j<(K+1)*NV; j++) delta[j] = 0.0; <...> #pragma mta max MAX_PROCS processors #pragma mta assert nodep #pragma mta block dynamic schedule #pragma mta assert no alias *sigma *Q *BC *delta *child *start *QHead #pragma mta use 100 streams for (n = Qstart; n < Qend; n++) { <...> The pragma was used on all parallel loops in the function to ensure that each thread that calls kcent_core is limited to the desired number of processors, which is 8 in this case. Also, because all of the parallel loops in kcent_core have the same limit for the max processors, the compiler will not need to put the loops into different parallel regions because of a mismatch in limits. Grouping the loops into one region can help reduce the cost of going parallel and improve performance by avoiding starting and tearing down multiple parallel regions. The kcentrality function was modified to compute the number of threads at runtime based on the number of processors used by the application and the number of processors used per thread in kcent_core. The number of threads, INC, is a preprocessor macro in version 0.4 of GraphCT. However, the modifications to kcentrality changed INC to a variable that is computed at runtime. The following code snippet shows the changes made to kcentrality. Again, MAX_PROCS used in the example below has been defined as 8.Limiting Loop Parallelism in Cray XMT™ Application S–0027–14 Cray Inc. 12 <...> /*Compute INC based on the number of processors we're using and limiting each thread to MAX_PROCS processors (in kcent_core()).*/ int INC; INC = mta_get_max_teams(); INC = INC / MAX_PROCS; INC = MTA_INT_MAX(1, INC); <...> #pragma mta loop future for(x=0; x for (int claimedk = int_fetch_add (&k, 1); claimedk < Vs; claimedk = int_fetch_add (&k, 1)) { <...> kcent_core(G, BC, K, s, Q, dist, sigma, marks, QHead, child, child_count); <...> } } <...> These changes to GraphCT helped the betweenness centrality kernel have better load balancing across the threads and achieve higher system utilization, which improved the performance and scalability of the kernel. References [1] “GraphCT – Streaming Graph Analysis”, http://trac.research.cc.gatech.edu/graphs/wiki/GraphCT, May 4, 2010. June 2004 version 6.5 TotalView New FeaturesCopyright © 1999–2004 by Etnus LLC. All rights reserved. Copyright © 1996–1998 by Dolphin Interconnect Solutions, Inc. Copyright © 1993–1996 by BBN Systems and Technologies, a division of BBN Corporation. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise without the prior written permission of Etnus LLC. (Etnus). Use, duplication, or disclosure by the Government is subject to restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS 252.227-7013. Etnus has prepared this manual for the exclusive use of its customers, personnel, and licensees. The information in this manual is subject to change without notice, and should not be construed as a commitment by Etnus. Etnus assumes no responsibility for any errors that appear in this document. TotalView and Etnus are registered trademarks of Etnus LLC. TotalView uses a modified version of the Microline widget library. Under the terms of its license, you are entitled to use these modifications. The source code is available at http://www.etnus.com/Products/TotalView/developers. All other brand names are the trademarks of their respective holders.TotalView New Features: 6.5 iii Contents New Features New Platforms and Compilers ................................................................................. 1 New and Changed GUI Features ............................................................................. 2 Tools > Memory Debugging Command Added ....................................................... 2 Node Display in the Variable Window ....................................................................... 4 STL String data types Transformed .......................................................................... 4 Type Transformations ............................................................................................... 4Contents iv 6.5TotalView New Features: version 6.5 1 New Features This booklet contains information about changes made to TotalView for version 6.5. The information in this document is to let you know what changes have occurred. You’ll find descriptions for all changes within the TotalVie w Us e r s Guide. TotalView has many features and it gives you a great number of tools for finding your program’s problems. An easy way to get acquainted with these features is to subscribe to the “Tip of the Week”. If you subscribe to this mailing list, you’ll receive an email message every week that tells you something about TotalView. ¦ All of the tips are archived on our web site at http://www.etnus.com/ Support/Tips/index.html. ¦ If you like what you see, you can subscribe at http://www.etnus.com/ mojo/mojo.cgi. New Platforms and Compilers TotalView now supports the following operating system versions: ¦ Red Hat Fedora Core 1 on x86 architectures. ¦ SuSE Linux Profession 9.0 and SuSe Linux Personal on x86 and x86-64 architectures. TotalView now supports the following compilers: ¦ gcc 3.4.0 for C and C++ on most platforms. ¦ gcc 3.4.0 for Fortran 77 on x86, x86-64, and ia64 Linux. ¦ Intel C and C++ 8.0.066 on x86 and ia64 Linux. ¦ Intel Fortran 8.0.046 on x86 and ia64 Linux ¦ Portland Group C and C++ 5.1 on x86 and x86-64 Linux. For complete information, see the TotalView Platforms Guide.New Features 2 version 6.5 New and Changed GUI Features Tools > Memory Debugging Command Added This release of TotalView adds to the memory debugging features that previously existed within TotalView. It also consolidates memory debugging interactions within one window. The TotalView Memory Debugger can help you locate many of your program’s memory problems. For example, you can: ¦ Stop execution when free(), realloc(), and other heap API problems occur. If your program tries to free memory that it can’t or shouldn’t free, the Memory Debugger can stop execution. This lets you can identify the statement that caused the problem. ¦ List leaks. The Memory Debugger can display your program’s leaks. (Leaks are memory blocks that are allocated, but which are no longer referenced.) When your program allocates a memory block, the Memory Debugger creates a backtrace. When it makes a list of your leaks, it includes this backtrace in the list. This lets you see the place where your program allocated the memory block. ¦ Paint allocated and deallocated blocks. When your program’s memory manager allocates or deallocates memory, the Memory Debugger can write a bit pattern into it. Writing this bit pattern is called painting. When you see this bit pattern in a Variable or Expression List Window, you can tell that you are using memory before your program initializes it or after your program deallocates it. Depending upon the architecture, you might even be able to force an exception when your program accesses this memory. ¦ Identify dangling pointers. A dangling pointer is a pointer that points into deallocated memory. If the pointer being displayed in a Variable is dangling, TotalView adds information to the data element so that you know about the problem. ¦ Hold onto deallocated memory. When trying to identify memory problems, holding on to memory after your program releases it can sometimes help locate problems. Holding onto freed memory is called hoarding. For example, retaining a block can sometimes force a memory error to occur. Or, when coupled with painting, you’ll be able to tell when your program is trying to access deallocated memory.New and Changed GUI Features TotalView New Features 3 After you select the Tools > Memory Debugging command, TotalView displays the following window:New Features 4 version 6.5 If memory debugging is enabled, you can tell the Memory Debugger to display information whenever execution stops. For example, here is a window showing leak information: The Backtrace Pane shows the stack frames that existed when your program allocated a memory block. The Source Pane shows the line where it made the allocation. For more information, see the Debugging Memory Problems Using TotalView document. Node Display in the Variable Window The View > Nodes command was removed. This command was only used when viewing UPC variables. You can see the nodes upon which a variable resides by right-clicking on the column headers and selecting Node. STL String data types Transformed STLView now transforms String data types. Type Transformations The way in which you create type transformations has been simplified. While older methods still work, the new methods are more direct. For information, see the “Creating Type Transformations” chapter of the TotalView Reference Guide. The Type Transformations Guide has been archived on our web site. It is will no longer be updated. However, it may be useful if you are attempting to transform a very difficult data structure or class. PGI ® User’s Guide Parallel Fortran, C and C++ for Scientists and Engineers The Portland Group™ STMicroelectronics Two Centerpointe Drive Lake Oswego, OR 97035While every precaution has been taken in the preparation of this document, The Portland Group™, a wholly-owned subsidiary of STMicroelectronics, makes no warranty for the use of its products and assumes no responsibility for any errors that may appear, or for damages resulting from the use of the information contained herein. The Portland Group retains the right to make changes to this information at any time, without notice. The software described in this document is distributed under license from STMicroelectronics and may be used or copied only in accordance with the terms of the license agreement. No part of this document may be reproduced or transmitted in any form or by any means, for any purpose other than the purchaser's personal use without the express written permission of The Portland Group. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this manual, The Portland Group was aware of a trademark claim. The designations have been printed in caps or initial caps. Thanks is given to the Parallel Tools Consortium and, in particular, to the High Performance Debugging Forum for their efforts. PGF95, PGF90, PGC++, Cluster Development Kit, CDK, PGI Unified Binary, PGI Visual Fortran, PVF and The Portland Group are trademarks and PGI, PGHPF, PGF77, PGCC, PGPROF, and PGDBG are registered trademarks of STMicroelectronics, Inc. Other brands and names are the property of their respective owners. The use of STLport, a C++ Library, is licensed separately and license, distribution and copyright notice can be found in the online documentation for a given release of the PGI compilers and tools. PGI ® User’s Guide Copyright © 1998 – 2000 The Portland Group, Inc. Copyright © 2000 – 2006 STMicroelectronics, Inc. All rights reserved. Printed in the United States of America First Printing: Release 1.7, Jun 1998 Second Printing: Release 3.0, Jan 1999 Third Printing: Release 3.1, Sep 1999 Fourth Printing: Release 3.2, Sep 2000 Fifth Printing: Release 4.0, May 2002 Sixth Printing: Release 5.0, Jun 2003 Seventh Printing: Release 5.1, Nov 2003 Eight Printing: Release 5.2, Jun 2004 Ninth Printing: Release 6.0, Mar 2005 Tenth Printing: Release 6.1, Dec 2005 Eleventh Printing: Release 6.2, Aug 2006 Twelfth printing: Release 7.0-1, December, 2006 Thirteenth printing: Release 7.1, October, 2007 Technical support: trs@pgroup.com Sales: sales@pgroup.com Web: www.pgroup.com/iii Contents Preface .................................................................................................................................... xix Audience Description ............................................................................................................ xix Compatibility and Conformance to Standards ............................................................................ xix Organization ......................................................................................................................... xx Hardware and Software Constraints ........................................................................................ xxii Conventions ........................................................................................................................ xxii Related Publications ........................................................................................................... xxvii 1. Getting Started .................................................................................................................... 1 Overview ................................................................................................................................ 1 Invoking the Command-level PGI Compilers ............................................................................... 1 Command-line Syntax ...................................................................................................... 2 Command-line Options .................................................................................................... 3 Fortran Directives and C/C++ Pragmas .............................................................................. 3 Filename Conventions .............................................................................................................. 3 Input Files ..................................................................................................................... 3 Output Files ................................................................................................................... 5 Fortran, C, and C++ Data Types ............................................................................................... 6 Parallel Programming Using the PGI Compilers ........................................................................... 7 Running SMP Parallel Programs ...................................................................................... 7 Running Data Parallel HPF Programs ................................................................................. 8 Platform-specific considerations ................................................................................................ 8 Using the PGI Compilers on Linux .................................................................................... 9 Using the PGI Compilers on Windows .............................................................................. 10 Using the PGI Compilers on SUA and SFU ........................................................................ 11 Using the PGI Compilers on Mac OS X ............................................................................. 11 Site-specific Customization of the Compilers .............................................................................. 12 Using siterc Files ........................................................................................................... 12 Using User rc Files ........................................................................................................ 12 Common Development Tasks .................................................................................................. 13 2. Using Command Line Options ....................................................................................... 15PGI® User’s Guide iv Command Line Option Overview ............................................................................................. 15 Command-line Options Syntax ......................................................................................... 15 Command-line Suboptions .............................................................................................. 16 Command-line Conflicting Options ................................................................................... 16 Help with Command-line Options ............................................................................................ 16 Getting Started with Performance ............................................................................................ 18 Using –fast and –fastsse Options ..................................................................................... 18 Other Performance-related Options ................................................................................. 19 Targeting Multiple Systems; Using the -tp Option ....................................................................... 19 Frequently-used Options ......................................................................................................... 19 3. Using Optimization & Parallelization .......................................................................... 21 Overview of Optimization ....................................................................................................... 21 Local Optimization ........................................................................................................ 22 Global Optimization ....................................................................................................... 22 Loop Optimization: Unrolling, Vectorization, and Parallelization ........................................... 22 Interprocedural Analysis (IPA) and Optimization .............................................................. 22 Function Inlining ........................................................................................................... 22 Profile-Feedback Optimization (PFO) .............................................................................. 22 Getting Started with Optimizations ........................................................................................... 23 Local and Global Optimization using -O .................................................................................. 24 Scalar SSE Code Generation ............................................................................................ 26 Loop Unrolling using –Munroll ............................................................................................... 27 Vectorization using –Mvect ..................................................................................................... 28 Vectorization Sub-options ............................................................................................... 28 Vectorization Example Using SSE/SSE2 Instructions ............................................................ 30 Auto-Parallelization using -Mconcur ......................................................................................... 32 Auto-parallelization Sub-options ...................................................................................... 33 Loops That Fail to Parallelize ......................................................................................... 34 Processor-Specific Optimization and the Unified Binary .............................................................. 36 Interprocedural Analysis and Optimization using –Mipa .............................................................. 37 Building a Program Without IPA – Single Step ................................................................... 37 Building a Program Without IPA - Several Steps ................................................................. 38 Building a Program Without IPA Using Make .................................................................... 38 Building a Program with IPA .......................................................................................... 38 Building a Program with IPA - Single Step ........................................................................ 39 Building a Program with IPA - Several Steps ..................................................................... 39 Building a Program with IPA Using Make ........................................................................ 40 Questions about IPA ...................................................................................................... 40 Profile-Feedback Optimization using –Mpfi/–Mpfo ..................................................................... 41 Default Optimization Levels ..................................................................................................... 42 Local Optimization Using Directives and Pragmas ...................................................................... 42 Execution Timing and Instruction Counting ............................................................................... 43 Portability of Multi-Threaded Programs on Linux ....................................................................... 43 libpgbind ..................................................................................................................... 44 libnuma ....................................................................................................................... 44PGI ® User’s Guide v 4. Using Function Inlining .................................................................................................. 45 Invoking Function Inlining ..................................................................................................... 45 Using an Inline Library .................................................................................................. 46 Creating an Inline Library ...................................................................................................... 47 Working with Inline Libraries ......................................................................................... 48 Updating Inline Libraries - Makefiles ............................................................................... 48 Error Detection during Inlining ............................................................................................... 49 Examples ............................................................................................................................. 49 Restrictions on Inlining .......................................................................................................... 49 5. Using OpenMP .................................................................................................................. 51 Fortran Parallelization Directives ............................................................................................. 51 C/C++ Parallelization Pragmas ............................................................................................... 52 Directive and Pragma Recognition ........................................................................................... 53 Directive and Pragma Summary Table ...................................................................................... 53 Directive and Pragma Clauses ................................................................................................. 54 Run-time Library Routines ...................................................................................................... 55 Environment Variables ........................................................................................................... 59 OMP_DYNAMIC ............................................................................................................ 59 OMP_NESTED ............................................................................................................... 59 OMP_NUM_THREADS ................................................................................................... 59 OMP_SCHEDULE ........................................................................................................... 60 OMP_STACK_SIZE ......................................................................................................... 60 OMP_WAIT_POLICY ...................................................................................................... 60 6. Using Directives and Pragmas ....................................................................................... 63 PGI Proprietary Fortran Directives ........................................................................................... 63 PGI Proprietary C and C++ Pragmas ....................................................................................... 64 PGI Proprietary Optimization Fortran Directive and C/C++ Pragma Summary ................................. 64 Scope of Fortran Directives and Command-Line options ............................................................. 66 Scope of C/C++ Pragmas and Command-Line Options ............................................................... 67 Prefetch Directives ............................................................................................................... 69 Format Requirements .................................................................................................... 70 Sample Usage ............................................................................................................... 70 !DEC$ Directive .................................................................................................................... 70 Format Requirements .................................................................................................... 71 ALIAS Directive ............................................................................................................. 71 ATTRIBUTES Directive ................................................................................................... 71 DISTRIBUTE Directive .................................................................................................... 72 ALIAS Directive ............................................................................................................. 72 C$PRAGMA C ........................................................................................................................ 72 7. Creating and Using Libraries ........................................................................................ 75 Using builtin Math Functions in C/C++ .................................................................................... 75 Creating and Using Shared Object Files on Linux ....................................................................... 76PGI® User’s Guide vi Creating and Using Shared Object Files in SFU and 32-bit SUA ..................................................... 77 Shared Object Error Message ......................................................................................... 78 Shared Object-Related Compiler Switches ......................................................................... 78 PGI Runtime Libraries on Windows ......................................................................................... 79 Creating and Using Static Libraries on Windows ........................................................................ 79 ar command ................................................................................................................ 79 ranlib command ........................................................................................................... 80 Creating and Using Dynamic-Link Libraries on Windows ............................................................. 80 Using LIB3F ........................................................................................................................ 88 LAPACK, BLAS and FFTs ......................................................................................................... 88 The C++ Standard Template Library ........................................................................................ 88 8. Using Environment Variables ........................................................................................ 89 Setting Environment Variables ................................................................................................. 89 Setting Environment Variables on Linux ............................................................................ 89 Setting Environment Variables on Windows ....................................................................... 90 Setting Environment Variables on Mac OSX ....................................................................... 90 PGI-Related Environment Variables .......................................................................................... 91 PGI Environment Variables ..................................................................................................... 92 FLEXLM_BATCH ............................................................................................................ 93 FORTRAN_OPT ............................................................................................................. 93 GMON_OUT_PREFIX ...................................................................................................... 93 LD_LIBRARY_PATH ....................................................................................................... 93 LM_LICENSE_FILE ......................................................................................................... 93 MANPATH .................................................................................................................... 94 MPSTKZ ....................................................................................................................... 94 MP_BIND ..................................................................................................................... 94 MP_BLIST .................................................................................................................... 95 MP_SPIN ..................................................................................................................... 95 MP_WARN ................................................................................................................... 95 NCPUS ......................................................................................................................... 96 NCPUS_MAX ................................................................................................................. 96 NO_STOP_MESSAGE ...................................................................................................... 96 PATH ........................................................................................................................... 96 PGI ............................................................................................................................. 96 PGI_CONTINUE ............................................................................................................. 97 PGI_OBJSUFFIX ............................................................................................................. 97 PGI_STACK_USAGE ........................................................................................................ 97 PGI_TERM ................................................................................................................... 97 PGI_TERM_DEBUG ....................................................................................................... 99 PWD ............................................................................................................................ 99 STATIC_RANDOM_SEED ................................................................................................. 99 TMP .......................................................................................................................... 100 TMPDIR ..................................................................................................................... 100 Using Environment Modules ................................................................................................. 100 Stack Traceback and JIT Debugging ....................................................................................... 101PGI ® User’s Guide vii 9. Distributing Files - Deployment .................................................................................. 103 Deploying Applications on Linux ............................................................................................ 103 Runtime Library Considerations ..................................................................................... 103 64-bit Linux Considerations .......................................................................................... 104 Linux Redistributable Files ............................................................................................ 104 Restrictions on Linux Portability .................................................................................... 104 Installing the Linux Portability Package ........................................................................... 104 Licensing for Redistributable Files ................................................................................. 105 Deploying Applications on Windows ....................................................................................... 105 PGI Redistributables .................................................................................................... 105 Microsoft Redistributables ............................................................................................ 105 Code Generation and Processor Architecture ........................................................................... 106 Generating Generic x86 Code ........................................................................................ 106 Generating Code for a Specific Processor ........................................................................ 106 Generating Code for Multiple Types of Processors in One Executable .......................................... 106 Unified Binary Command-line Switches ........................................................................... 107 Unified Binary Directives and Pragma ............................................................................. 107 10. Inter-language Calling ................................................................................................ 109 Overview of Calling Conventions ............................................................................................ 109 Inter-language Calling Considerations ..................................................................................... 110 Functions and Subroutines ................................................................................................... 110 Upper and Lower Case Conventions, Underscores .................................................................... 111 Compatible Data Types ......................................................................................................... 111 Fortran Named Common Blocks .................................................................................... 112 Argument Passing and Return Values ..................................................................................... 113 Passing by Value (%VAL) ............................................................................................. 113 Character Return Values ............................................................................................... 113 Complex Return Values ................................................................................................ 114 Array Indices ...................................................................................................................... 114 Examples ........................................................................................................................... 115 Example - Fortran Calling C .......................................................................................... 115 Example - C Calling Fortran .......................................................................................... 115 Example - C ++ Calling C ............................................................................................ 116 Example - C Calling C++ ............................................................................................. 117 Example - Fortran Calling C++ ..................................................................................... 118 Example - C++ Calling Fortran ..................................................................................... 119 Win32 Calling Conventions ................................................................................................... 120 Win32 Fortran Calling Conventions ................................................................................ 120 Symbol Name Construction and Calling Example .............................................................. 121 Using the Default Calling Convention .............................................................................. 122 Using the STDCALL Calling Convention ............................................................................ 122 Using the C Calling Convention ...................................................................................... 122 Using the UNIX Calling Convention ................................................................................. 123 11. Programming Considerations for 64-Bit Environments ....................................... 125PGI® User’s Guide viii Data Types in the 64-Bit Environment .................................................................................... 125 C/C++ Data Types ....................................................................................................... 126 Fortran Data Types ...................................................................................................... 126 Large Static Data in Linux ..................................................................................................... 126 Large Dynamically Allocated Data .......................................................................................... 126 64-Bit Array Indexing .......................................................................................................... 126 Compiler Options for 64-bit Programming .............................................................................. 127 Practical Limitations of Large Array Programming .................................................................... 128 Example: Medium Memory Model and Large Array in C ............................................................ 129 Example: Medium Memory Model and Large Array in Fortran .................................................... 130 Example: Large Array and Small Memory Model in Fortran ....................................................... 131 12. C/C++ Inline Assembly and Intrinsics ..................................................................... 133 Inline Assembly ................................................................................................................... 133 Extended Inline Assembly ..................................................................................................... 134 Output Operands ......................................................................................................... 135 Input Operands ........................................................................................................... 137 Clobber List ................................................................................................................ 138 Additional Constraints .................................................................................................. 139 Operand Aliases .......................................................................................................... 145 Assembly String Modifiers ............................................................................................. 145 Extended Asm Macros .................................................................................................. 147 Intrinsics ............................................................................................................................ 148 13. Fortran, C and C++ Data Types ................................................................................ 151 Fortran Data Types .............................................................................................................. 151 Fortran Scalars ........................................................................................................... 151 FORTRAN 77 Aggregate Data Type Extensions .................................................................. 153 Fortran 90 Aggregate Data Types (Derived Types) ............................................................ 154 C and C++ Data Types ....................................................................................................... 154 C and C++ Scalars ...................................................................................................... 154 C and C++ Aggregate Data Types .................................................................................. 156 Class and Object Data Layout ........................................................................................ 156 Aggregate Alignment .................................................................................................... 157 Bit-field Alignment ....................................................................................................... 158 Other Type Keywords in C and C++ .............................................................................. 158 14. C++ Name Mangling ................................................................................................... 159 Types of Mangling ............................................................................................................... 160 Mangling Summary .............................................................................................................. 160 Type Name Mangling ................................................................................................... 160 Nested Class Name Mangling ......................................................................................... 161 Local Class Name Mangling ........................................................................................... 161 Template Class Name Mangling ..................................................................................... 161 15. Command-Line Options Reference ........................................................................... 163PGI ® User’s Guide ix PGI Compiler Option Summary ............................................................................................. 163 Build-Related PGI Options ............................................................................................ 163 PGI Debug-Related Compiler Options ............................................................................. 166 PGI Optimization-Related Compiler Options .................................................................... 167 PGI Linking and Runtime-Related Compiler Options ......................................................... 167 C and C++ Compiler Options ............................................................................................... 168 Generic PGI Compiler Options .............................................................................................. 170 C and C++ -specific Compiler Options ................................................................................... 208 –M Options by Category ....................................................................................................... 219 –M Code Generation Controls .......................................................................... 220 –M C/C++ Language Controls .......................................................................... 223 –M Environment Controls ................................................................................ 225 –M Fortran Language Controls ......................................................................... 226 –M Inlining Controls ....................................................................................... 228 –M Optimization Controls ................................................................................ 229 –M Miscellaneous Controls .............................................................................. 238 16. OpenMP Reference Information ............................................................................... 243 Parallelization Directives and Pragmas ................................................................................... 243 ATOMIC ............................................................................................................................ 244 BARRIER ............................................................................................................................ 244 CRITICAL ... END CRITICAL and omp critical .......................................................................... 245 C$DOACROSS .................................................................................................................... 246 DO ... END DO and omp for ................................................................................................ 247 FLUSH and omp flush pragma .............................................................................................. 249 MASTER ... END MASTER and omp master pragma ................................................................. 250 ORDERED ......................................................................................................................... 251 PARALLEL ... END PARALLEL and omp parallel ....................................................................... 251 PARALLEL DO .................................................................................................................... 254 PARALLEL SECTIONS ........................................................................................................... 255 PARALLEL WORKSHARE ....................................................................................................... 256 SECTIONS … END SECTIONS .............................................................................................. 257 SINGLE ... END SINGLE ........................................................................................................ 257 THREADPRIVATE ................................................................................................................ 258 WORKSHARE ... END WORKSHARE ......................................................................................... 259 Directive and Pragma Clauses ............................................................................................... 260 Schedule Clause .......................................................................................................... 261 17. Directives and Pragmas Reference ........................................................................... 263 PGI Proprietary Fortran Directive and C/C++ Pragma Summary ................................................. 263 altcode (noaltcode) ............................................................................................................ 263 assoc (noassoc) .................................................................................................................. 264 bounds (nobounds) ........................................................................................................... 265 cncall (nocncall) ................................................................................................................ 265 concur (noconcur) ............................................................................................................ 265 depchk (nodepchk) ............................................................................................................ 265PGI® User’s Guide x eqvchk (noeqvchk) ............................................................................................................ 265 fcon (nofcon) ..................................................................................................................... 265 invarif (noinvarif) ............................................................................................................... 265 ivdep ................................................................................................................................. 266 lstval (nolstval) ................................................................................................................... 266 opt .................................................................................................................................... 266 safe (nosafe) ...................................................................................................................... 266 safe_lastval ......................................................................................................................... 266 safeptr (nosafeptr) .............................................................................................................. 267 single (nosingle) ................................................................................................................. 268 tp ...................................................................................................................................... 268 unroll (nounroll) ................................................................................................................ 268 vector (novector) ................................................................................................................ 269 vintr (novintr) .................................................................................................................... 269 18. Run-time Environment ................................................................................................ 271 Linux86 and Win32 Programming Model ................................................................................ 271 Function Calling Sequence ............................................................................................ 271 Function Return Values ................................................................................................ 273 Argument Passing ........................................................................................................ 275 Linux86-64 Programming Model ........................................................................................... 277 Function Calling Sequence ............................................................................................ 278 Function Return Values ................................................................................................ 280 Argument Passing ........................................................................................................ 281 Linux86-64 Fortran Supplement .................................................................................... 283 Win64 Programming Model .................................................................................................. 287 Function Calling Sequence ............................................................................................ 288 Function Return Values ................................................................................................ 290 Argument Passing ........................................................................................................ 291 Win64/SUA64 Fortran Supplement ................................................................................. 293 19. C++ Dialect Supported ............................................................................................... 299 Extensions Accepted in Normal C++ Mode ............................................................................. 299 cfront 2.1 Compatibility Mode ............................................................................................... 300 cfront 2.1/3.0 Compatibility Mode ......................................................................................... 301 20. C/C++ MMX/SSE Inline Intrinsics ............................................................................. 303 Using Intrinsic functions ....................................................................................................... 303 Required Header File ................................................................................................... 304 Intrinsic Data Types ..................................................................................................... 304 Intrinsic Example ........................................................................................................ 304 MMX Intrinsics ................................................................................................................... 305 SSE Intrinsics ...................................................................................................................... 306 ABM Intrinsics .................................................................................................................... 309 21. Fortran Module/Library Interfaces ........................................................................... 311PGI ® User’s Guide xi Data Types ......................................................................................................................... 311 Using DFLIB and DFPORT .................................................................................................... 312 DFLIB ........................................................................................................................ 312 DFPORT ..................................................................................................................... 312 Using the DFWIN module ..................................................................................................... 312 Supported Libraries and Modules .......................................................................................... 313 advapi32 .................................................................................................................... 313 comdlg32 ................................................................................................................... 315 dfwbase ..................................................................................................................... 315 dfwinty ....................................................................................................................... 315 gdi32 ......................................................................................................................... 316 kernel32 .................................................................................................................... 319 shell32 ....................................................................................................................... 327 user32 ....................................................................................................................... 327 winver ....................................................................................................................... 331 wsock32 .................................................................................................................... 332 22. Messages ........................................................................................................................ 333 Diagnostic Messages ............................................................................................................ 333 Phase Invocation Messages ................................................................................................... 334 Fortran Compiler Error Messages .......................................................................................... 334 Message Format .......................................................................................................... 334 Message List ............................................................................................................... 334 Fortran Runtime Error Messages ........................................................................................... 360 Message Format .......................................................................................................... 360 Message List ............................................................................................................... 360 Index ...................................................................................................................................... 363xiixiii Figures 13.1. Internal Padding in a Structure ............................................................................................. 157 13.2. Tail Padding in a Structure ................................................................................................... 158xivxv Tables 1. PGI Compilers and Commands .................................................................................................. xxvi 2. Processor Options ................................................................................................................... xxvi 1.1. Stop-after Options, Inputs and Outputs ........................................................................................ 5 1.2. Examples of Using siterc and User rc Files ................................................................................. 13 2.1. Commonly Used Command Line Options .................................................................................... 20 3.1. Optimization and –O, –g and –M Options ........................................................................ 42 5.1. Directive and Pragma Summary Table ....................................................................................... 53 5.2. Run-time Library Call Summary ................................................................................................ 55 5.3. OpenMP-related Environment Variable Summary Table ................................................................ 59 6.1. Proprietary Optimization-Related Fortran Directive and C/C++ Pragma Summary ............................. 65 8.1. PGI-related Environment Variable Summary Table ....................................................................... 91 8.2. Supported PGI_TERM Values ................................................................................................... 98 10.1. Fortran and C/C++ Data Type Compatibility ............................................................................ 111 10.2. Fortran and C/C++ Representation of the COMPLEX Type ......................................................... 112 10.3. Calling Conventions Supported by the PGI Fortran Compilers ..................................................... 120 11.1. 64-bit Compiler Options ....................................................................................................... 127 11.2. Effects of Options on Memory and Array Sizes ......................................................................... 127 11.3. 64-Bit Limitations ................................................................................................................ 128 12.1. Simple Constraints ............................................................................................................... 139 12.2. x86/x86_64 Machine Constraints .......................................................................................... 141 12.3. Multiple Alternative Constraints ............................................................................................. 143 12.4. Constraint Modifier Characters .............................................................................................. 144 12.5. Assembly String Modifier Characters ...................................................................................... 145 12.6. Intrinsic Header File Organization ......................................................................................... 148 13.1. Representation of Fortran Data Types ..................................................................................... 151 13.2. Real Data Type Ranges ........................................................................................................ 152 13.3. Scalar Type Alignment ......................................................................................................... 152 13.4. C/C++ Scalar Data Types ..................................................................................................... 154 13.5. Scalar Alignment ................................................................................................................. 155 15.1. PGI Build-Related Compiler Options ...................................................................................... 164 15.2. PGI Debug-Related Compiler Options ..................................................................................... 166 15.3. Optimization-Related PGI Compiler Options ............................................................................ 167 15.4. Linking and Runtime-Related PGI Compiler Options ................................................................. 167PGI® User’s Guide xvi 15.5. C and C++ -specific Compiler Options ................................................................................... 168 15.6. Subgroups for –help Option ................................................................................................. 179 15.7. –M Options Summary .......................................................................................................... 185 15.8. Optimization and –O, –g, –Mvect, and –Mconcur Options ........................................................ 193 16.1. Initialization of REDUCTION Variables .................................................................................... 253 16.2. Directive and Pragma Clauses .............................................................................................. 260 18.1. Register Allocation .............................................................................................................. 271 18.2. Standard Stack Frame .......................................................................................................... 272 18.3. Stack Contents for Functions Returning struct/union ................................................................. 274 18.4. Integral and Pointer Arguments ............................................................................................. 275 18.5. Floating-point Arguments ...................................................................................................... 275 18.6. Structure and Union Arguments ............................................................................................ 276 18.7. Register Allocation .............................................................................................................. 278 18.8. Standard Stack Frame .......................................................................................................... 278 18.9. Register Allocation for Example A-2 ....................................................................................... 282 18.10. Linux86-64 Fortran Fundamental Types ................................................................................ 284 18.11. Fortran and C/C++ Data Type Compatibility .......................................................................... 285 18.12. Fortran and C/C++ Representation of the COMPLEX Type ....................................................... 286 18.13. Register Allocation ............................................................................................................. 288 18.14. Standard Stack Frame ........................................................................................................ 288 18.15. Register Allocation for Example A-4 ..................................................................................... 292 18.16. Win64 Fortran Fundamental Types ....................................................................................... 293 18.17. Fortran and C/C++ Data Type Compatibility .......................................................................... 295 18.18. Fortran and C/C++ Representation of the COMPLEX Type ....................................................... 296 20.1. MMX Intrinsics (mmintrin.h) ................................................................................................ 305 20.2. SSE Intrinsics (xmmintrin.h) ................................................................................................ 306 20.3. SSE2 Intrinsics (emmintrin.h) ............................................................................................. 307 20.4. SSE3 Intrinsics (pmmintrin.h) .............................................................................................. 309 20.5. SSSE3 Intrinsics (tmmintrin.h) .............................................................................................. 309 20.6. SSE4a Intrinsics (ammintrin.h) ............................................................................................. 309 20.7. SSE4a Intrinsics (intrin.h) .................................................................................................... 310 21.1. Fortran Data Type Mappings ................................................................................................. 311xvii Examples 1.1. Hello program ......................................................................................................................... 2 2.1. Makefiles with Options ............................................................................................................ 16 3.1. Dot Product Code ................................................................................................................... 27 3.2. Unrolled Dot Product Code ...................................................................................................... 27 3.3. Vector operation using SSE instructions ..................................................................................... 31 3.4. Using SYSTEM_CLOCK code fragment ........................................................................................ 43 4.1. Sample Makefile ..................................................................................................................... 48 6.1. Prefetch Directive Use ............................................................................................................. 70 7.1. Build a DLL: Fortran ............................................................................................................... 82 7.2. Build a DLL: C ....................................................................................................................... 83 7.3. Build DLLs Containing Circular Mutual Imports: C ....................................................................... 84 7.4. Build DLLs Containing Circular Mutual Imports: Fortran ............................................................... 86 7.5. Import a Fortran module from a DLL ........................................................................................ 87 10.1. Character Return Parameters ................................................................................................ 114 10.2. COMPLEX Return Values ...................................................................................................... 114 10.3. Fortran Main Program fmain.f .............................................................................................. 115 10.4. C function cfunc_ ............................................................................................................... 115 10.5. Fortran Subroutine forts.f ..................................................................................................... 116 10.6. C Main Program cmain.c ..................................................................................................... 116 10.7. Simple C Function cfunc.c .................................................................................................... 116 10.8. C++ Main Program cpmain.C Calling a C Function .................................................................. 117 10.9. Simple C++ Function cpfunc.C with Extern C .......................................................................... 117 10.10. C Main Program cmain.c Calling a C++ Function .................................................................. 117 10.11. Fortran Main Program fmain.f calling a C++ function ............................................................ 118 10.12. C++ function cpfunc.C ...................................................................................................... 118 10.13. Fortran Subroutine forts.f ................................................................................................... 119 10.14. C++ main program cpmain.C ............................................................................................. 119 18.1. C Program Calling an Assembly-language Routine .................................................................... 277 18.2. Parameter Passing ............................................................................................................... 282 18.3. C Program Calling an Assembly-language Routine .................................................................... 283 18.4. Parameter Passing ............................................................................................................... 291 18.5. C Program Calling an Assembly-language Routine .................................................................... 293xviiixix Preface This guide is part of a set of manuals that describe how to use The Portland Group (PGI) Fortran, C, and C++ compilers and program development tools. These compilers and tools include the PGF77, PGF95, PGHPF, PGC++, and PGCC ANSI C compilers, the PGPROF profiler, and the PGDBG debugger. They work in conjunction with an x86 or x64 assembler and linker. You can use the PGI compilers and tools to compile, debug, optimize, and profile serial and parallel applications for x86 (Intel Pentium II/III/4/M, Intel Centrino, Intel Xeon, AMD Athlon XP/MP) or x64 (AMD Athlon64/Opteron/Turion, Intel EM64T, Intel Core Duo, Intel Core 2 Duo) processor-based systems. The PGI User's Guide provides operating instructions for the PGI command-level development environment. It also contains details concerning the PGI compilers' interpretation of the Fortran language, implementation of Fortran language extensions, and command-level compilation. Users are expected to have previous experience with or knowledge of the Fortran programming language. Audience Description This manual is intended for scientists and engineers using the PGI compilers. To use these compilers, you should be aware of the role of high-level languages, such as Fortran, C, and C++, as well as assembly-language in the software development process; and you should have some level of understanding of programming. The PGI compilers are available on a variety of x86 or x64 hardware platforms and operating systems. You need to be familiar with the basic commands available on your system. Compatibility and Conformance to Standards Your system needs to be running a properly installed and configured version of the compilers. For information on installing PGI compilers and tools, refer to the Release and Installation notes included with your software. For further information, refer to the following: • American National Standard Programming Language FORTRAN, ANSI X3. -1978 (1978). • ISO/IEC 1539-1 : 1991, Information technology – Programming Languages – Fortran, Geneva, 1991 (Fortran 90). • ISO/IEC 1539-1 : 1997, Information technology – Programming Languages – Fortran, Geneva, 1997 (Fortran 95).PGI® User’s Guide xx • Fortran 95 Handbook Complete ISO/ANSI Reference, Adams et al, The MIT Press, Cambridge, Mass, 1997. • High Performance Fortran Language Specification, Revision 1.0, Rice University, Houston, Texas (1993), http://www.crpc.rice.edu/HPFF. • High Performance Fortran Language Specification, Revision 2.0, Rice University, Houston, Texas (1997), http://www.crpc.rice.edu/HPFF. • OpenMP Application Program Interface, Version 2.5, May 2005, http://www.openmp.org. • Programming in VAX Fortran, Version 4.0, Digital Equipment Corporation (September, 1984). • IBM VS Fortran, IBM Corporation, Rev. GC26-4119. • Military Standard, Fortran, DOD Supplement to American National Standard Programming Language Fortran, ANSI x.3-1978, MIL-STD-1753 (November 9, 1978). • American National Standard Programming Language C, ANSI X3.159-1989. • ISO/IEC 9899:1999, Information technology – Programming Languages – C, Geneva, 1999 (C99). Organization Users typically begin by wanting to know how to use a product and often then find that they need more information and facts about specific areas of the product. Knowing how as well as why you might use certain options or perform certain tasks is key to using the PGI compilers and tools effectively and efficiently. However, once you have this knowledge and understanding, you very likely might find yourself wanting to know much more about specific areas or specific topics. Consequently, his manual is divided into the following two parts: • Part I, Compiler Usage, contains the essential information on how to use the compiler. • Part II, Reference Information, contains more detailed reference information about specific aspects of the compiler, such as the details of compiler options, directives, and more. Part I, Compiler Usage, contains these chapters: Chapter 1, “Getting Started” provides an introduction to the PGI compilers and describes their use and overall features. Chapter 2, “Using Command Line Options” provides an overview of the command-line options as well as task-related lists of options. Chapter 3, “Using Optimization & Parallelization” describes standard optimization techniques that, with little effort, allow users to significantly improve the performance of programs. Chapter 4, “Using Function Inlining” describes how to use function inlining and shows how to create an inline library. Chapter 5, “Using OpenMP” provides a description of the OpenMP Fortran parallelization directives and of the OpenMP C and C++ parallelization pragmas and shows examples of their use. Chapter 6, “Using Directives and Pragmas” provides a description of each Fortran optimization directive and C/C++ optimization pragma, and shows examples of their use.Preface xxi Chapter 7, “Creating and Using Libraries” discusses PGI support libraries, shared object files, and environment variables that affect the behavior of the PGI compilers. Chapter 8, “ Using Environment Variables” describes the environment variables that affect the behavior of the PGI compilers. Chapter 9, “Distributing Files - Deployment” describes the deployment of your files once you have built, debugged and compiled them successfully. Chapter 10, “Inter-language Calling” provides examples showing how to place C Language calls in a Fortran program and Fortran Language calls in a C program. Chapter 11, “Programming Considerations for 64-Bit Environments” discusses issues of which programmers should be aware when targeting 64-bit processors. Chapter 12, “C/C++ Inline Assembly and Intrinsics” describes how to use inline assembly code in C and C++ programs, as well as how to use intrinsic functions that map directly to x86 and x64 machine instructions. Part II, Reference Information, contains these chapters: Chapter 13, “Fortran, C and C++ Data Types” describes the data types that are supported by the PGI Fortran, C, and C++ compilers. Chapter 14, “C++ Name Mangling” describes the name mangling facility and explains the transformations of names of entities to names that include information on aspects of the entity’s type and a fully qualified name. Chapter 15, “Command-Line Options Reference” provides a detailed description of each command-line option. Chapter 16, “OpenMP Reference Information”contains detailed descriptions of each of the OpenMP directives and pragmas that PGI supports. Chapter 17, “Directives and Pragmas Reference”contains detailed descriptions of PGI’s proprietary directives and pragmas. Chapter 18, “Run-time Environment” describes the assembly language calling conventions and examples of assembly language calls. Chapter 19, “C++ Dialect Supported” lists more details of the version of the C++ language that PGC++ supports. Chapter 20, “C/C++ MMX/SSE Inline Intrinsics,” on page 303 provides tables that list the MMX Inline Intrinsics (mmintrin.h), the SSE1 inline intrinsics (xmmintrin.h), and SSE2 inline intrinsics (emmintrin.h). Chapter 21, “Fortran Module/Library Interfaces” provides a description of the Fortran module library interfaces that PVF supports, describing each property available. Chapter 22, “Messages” provides a list of compiler error messages.PGI® User’s Guide xxii Hardware and Software Constraints This guide describes versions of the PGI compilers that produce assembly code for x86 and x64 processorbased systems. Details concerning environment-specific values and defaults and system-specific features or limitations are presented in the release notes delivered with the PGI compilers. Conventions The PGI User's Guide uses the following conventions: italic Italic font is for commands, filenames, directories, arguments, options and for emphasis. Constant Width Constant width font is for examples and for language statements in the text, including assembly language statements. [ item1 ] Square brackets indicate optional items. In this case item1 is optional. { item2 | item 3} Braces indicate that a selection is required. In this case, you must select either item2 or item3. filename... Ellipsis indicate a repetition. Zero or more of the preceding item may occur. In this example, multiple filenames are allowed. FORTRAN Fortran language statements are shown in the text of this guide using upper-case characters and a reduced point size. The PGI compilers and tools are supported on both 32-bit and 64-bit variants of Linux, Windows, and Mac OS operating systems on a variety of x86-compatible processors. There are a wide variety of releases and distributions of each of these types of operating systems. The PGI User’s Guide defines the following terms with respect to these platforms: AMD64 a 64-bit processor from AMD, designed to be binary compatible with IA32 processors, and incorporating new features such as additional registers and 64-bit addressing support for improved performance and greatly increased memory range. Barcelona the Quad-Core AMD Opteron(TM) Processor, that is, Opteron Rev x10 DLL a dynamic linked library on Win32 or Win64 platforms of the form xxx.dll containing objects that are dynamically linked into a program at the time of execution. driver the compiler driver controls the compiler, linker, and assembler, and adds objects and libraries to create an executable. The -dryrun option illustrates operation of the driver. pgf77, pgf95, pghpf, pgcc, pgCCPreface xxiii (Linux), and pgcpp are drivers for the PGI compilers. A pgf90 driver is retained for compatibility with existing makefiles, even though pgf90 and pgf95 drivers are identical. Dual-core Dual-, Quad-, or Multi-core - some x64 CPUs incorporate two or four complete processor cores (functional units, registers, level 1 cache, level 2 cache, etc) on a single silicon die. These are referred to as Dual-core or Quad-core (in general, Multi-core) processors. For purposes of OpenMP or auto-parallel threads, or MPI process parallelism, these cores function as distinct processors. However, the processing cores are on a single chip occupying a single socket on the system motherboard. In PGI 7.1, there are no longer software licensing limits on OpenMP threads for Multi-core. EM64T a 64-bit IA32 processor with Extended Memory 64-bit Technology extensions that are binary compatible with AMD64 processors. This includes Intel Pentium 4, Intel Xeon, and Intel Core 2 processors. hyperthreading (HT) some IA32 CPUs incorporate extra registers that allow 2 threads to run on a single CPU with improved performance for some tasks. This is called hyperthreading and abbreviated HT. Some linux86 and linux86-64 environments treat IA32 CPUs with HT as though there were a 2nd pseudo CPU, even though there is only one physical CPU. Unless the Linux kernel is hyperthread-aware, the second thread of an OpenMP program will be assigned to the pseudo CPU, rather than a real second physical processor (if one exists in the system). OpenMP Programs can run very slowly if the second thread is not properly assigned. IA32 an Intel Architecture 32-bit processor, designed to be binary compatible with x86 processors, and incorporating new features such as streaming SIMD extensions (SSE) for improved performance. Large Arrays arrays with aggregate size larger than 2GB, which require 64-bit index arithmetic for accesses to elements of arrays. If -Mlarge_arrays is specified and -mcmodel=medium is not specified, the default small memory model is used, and all index arithmetic is performed in 64-bits. This can be a useful mode of execution for certain existing 64-bit applications that use the small memory model but allocate and manage a single contiguous data space larger than 2GB. linux86 32-bit Linux operating system running on an x86 or x64 processor-based system, with 32-bit GNU tools, utilities and libraries used by the PGI compilers to assemble and link for 32-bit execution. linux86-64 64-bit Linux operating system running on an x64 processor-based system, with 64-bit and 32-bit GNU tools, utilities and libraries used by the PGI compilers to assemble and link for execution in either linux86 or linux86-64 environments. The 32-bit development tools and execution environment under linux86-64 are considered a cross development environment for x86 processor-based applications. Mac OS X collectively, all osx86 and osx86-64 platforms supported by the PGI compilers. -mcmodel=small compiler/linker switch to produce small memory model format objects/executables in which both code (.text) and data (.bss) sections are limited to less than 2GB. This switch is the default and only possible format for linux86 32-bit executables. This switch is the default format for linux86-64 executables.PGI® User’s Guide xxiv Maximum address offset range is 32-bits, and total memory used for OS+Code+Data must be less than 2GB. -mcmodel=medium compiler/linker switch to produce medium memory model format objects/executables in which code sections are limited to less than 2GB, but data sections can be greater than 2GB. This option is supported only in linux86-64 environments. It must be used to compile any program unit that will be linked in to a 64-bit executable that will use aggregate data sets larger than 2GB and will access data requiring address offsets greater than 2GB. This option must be used to link any 64-bit executable that will use aggregate data sets greater than 2GB in size. Executables linked using -mcmodel=medium can incorporate objects compiled using -mcmodel=small as long as the small objects are from a shared library. NUMA A type of multi-processor system architecture in which the memory latency from a given processor to a given portion of memory can vary, resulting in the possibility for compiler or programming optimizations to ensure frequently accessed data is "close" to a given processor as determined by memory latency. osx86 32-bit Apple Mac OS Operating Systems running on an x86 Core 2 or Core 2 Duo processor-based system with the 32-bit Apple and GNU tools, utilities, and libraries used by the PGI compilers to assemble and link for 32-bit execution. The PGI Workstation preview supports Mac OS 10.4.9 only. osx86-64 64-bit Apple Mac OS Operating Systems running on an x64 Core 2 Duo processor-based system with the 64-bit and 32-bit Apple and GNU tools, utilities, and libraries used by the PGI compilers to assemble and link for either 64- or 32-bit execution. The PGI Workstation preview supports Mac OS 10.4.9 only. SFU Windows Services for Unix, a 32-bit-only predecessor of SUA, the Subsystem for Unix Applications. See SUA. Shared library a Linux library of the form libxxx.so containing objects that are dynamically linked into a program at the time of execution. SSE collectively, all SSE extensions supported by the PGI compilers. SSE1 32-bit IEEE 754 FPU and associated streaming SIMD extensions (SSE) instructions on Pentium III, AthlonXP* and later 32-bit x86, AMD64 and EM64T compatible CPUs, enabling scalar and packed vector arithmetic on single-precision floating-point data. SSE2 64-bit IEEE 754 FPU and associated SSE instructions on P4/Xeon and later 32-bit x86, AMD64 and EM64T compatible CPUs. SSE2 enables scalar and packed vector arithmetic on double-precision floating-point data. SSE3 additional 32-bit and 64-bit SSE instructions to enable more efficient support of arithmetic on complex floating-point data on 32-bit x86, AMD64 and EM64T compatible CPUs with so-called Prescott NewPreface xxv Instructions (PNI), such as Intel IA32 processors with EM64T extensions and newer generation (Revision E and beyond) AMD64 processors. SSE4A and ABM AMD Instruction Set enhancements for the Quad-Core AMD Opteron Processor. Support for these instructions is enabled by the -tp barcelona or -tp barcelona-64 switch. SSSE3 an extension of the SSE3 instruction set found on the Intel Core 2. Static linking a method of linking: On Linux, use - to ensure all objects are included in a generated executable at link time. Static linking causes objects from static library archives of the form libxxx.a to be linked in to your executable, rather than dynamically linking the corresponding libxxx.so shared library. Static linking of executables linked using the -mcmodel=medium option is supported. On Windows, the Windows linker links statically or dynamically depending on whether the libraries on the link-line are DLL import libraries or static libraries. By default, the static PGI libraries are included on the link line. To link with DLL versions of the PGI libraries instead of static libraries, use the -Mdll option. SUA Subsystem for UNIX-based Applications (SUA) is source-compatibility subsystem for compiling and running custom UNIX-based applications on a computer running 32-bit or 64-bit Windows server-class operating system. It provides an operating system for Portable Operating System Interface (POSIX) processes. SUA supports a package of support utilities (including shells and >300 Unix commands), case-sensitive file names, and job control. The subsystem installs separately from the Windows kernel to support UNIX functionality without any emulation. Win32 any of the 32-bit Microsoft Windows Operating Systems (XP/2000/Server 2003) running on an x86 or x64 processor-based system. On these targets, the PGI compiler products include all of the tools and libraries needed to build executables for 32-bit Windows systems. Win64 any of the 64-bit Microsoft Windows Operating Systems (XP Professional /Windows Server 2003 x64 Editions) running on an x64 processor-based system. On these targets, the PGI compiler products include all of the tools and libraries needed to build executables for 32-bit Windows systems. Windows collectively, all Win32 and Win64 platforms supported by the PGI compilers. x64 collectively, all AMD64 and EM64T processors supported by the PGI compilers. x86 a processor designed to be binary compatible with i386/i486 and previous generation processors from Intel* Corporation. Refers collectively to such processors up to and including 32-bit variants. x87 - 80-bit IEEE stack-based floating-point unit (FPU) and associated instructions on x86-compatible CPUs.PGI® User’s Guide xxvi The following table lists the PGI compilers and tools and their corresponding commands: Table 1. PGI Compilers and Commands Compiler or Tool Language or Function Command PGF77 FORTRAN 77 pgf77 PGF95 Fortran 90/95 pgf95 PGHPF High Performance Fortran pghpf PGCC C ANSI C99 and K&R C pgcc PGC++ ANSI C++ with cfront features pgcpp (pgCC) PGDBG Source code debugger pgdbg PGPROF Performance profiler pgprof In general, the designation PGF95 is used to refer to The Portland Group’s Fortran 90/95 compiler, and pgf95 is used to refer to the command that invokes the compiler. A similar convention is used for each of the PGI compilers and tools. For simplicity, examples of command-line invocation of the compilers generally reference the pgf95 command, and most source code examples are written in Fortran. Usage of the PGF77 compiler, whose features are a subset of PGF95, is similar. Usage of PGHPF, PGC++, and PGCC ANSI C99 is consistent with PGF95 and PGF77, but there are command-line options and features of these compilers that do not apply to PGF95 and PGF77 and vice versa. There are a wide variety of x86-compatible processors in use. All are supported by the PGI compilers and tools. Most of these processors are forward-compatible, but not backward-compatible, meaning that code compiled to target a given processor will not necessarily execute correctly on a previous-generation processor. The following table provides a partial list, including the most important processor types, along with the features utilized by the PGI compilers that distinguish them from a compatibility standpoint: Table 2. Processor Options Processor Prefetch SSE1 SSE2 SSE3 32-bit 64-bit Scalar FP Default AMD Athlon N N N N Y N x87 AMD Athlon XP/MP Y Y N N Y N x87 AMD Athlon64 Y Y Y N Y Y SSE AMD Opteron Y Y Y N Y Y SSE AMD Opteron Rev E Y Y Y Y Y Y SSE AMD Opteron Rev F Y Y Y Y Y Y SSE AMD Turion Y Y Y Y Y Y SSE Intel Celeron N N N N Y N x87Preface xxvii Processor Prefetch SSE1 SSE2 SSE3 32-bit 64-bit Scalar FP Default Intel Pentium II N N N N Y N x87 Intel Pentium III Y Y N N Y N x87 Intel Pentium 4 Y Y Y N Y N SSE Intel Pentium M Y Y Y N Y N SSE Intel Centrino Y Y Y N Y N SSE Intel Pentium 4 EM64T Y Y Y Y Y Y SSE Intel Xeon EM64T Y Y Y Y Y Y SSE Intel Core Duo EM64T Y Y Y Y Y Y SSE Intel Core 2 Duo EM64T Y Y Y Y Y Y SSE In this manual, the convention is to use “x86” to specify the group of processors in the previous table that are listed as “32-bit” but not “64-bit.” The convention is to use “x64” to specify the group of processors that are listed as both “32-bit” and “64-bit.” x86 processor-based systems can run only 32-bit operating systems. x64 processor-based systems can run either 32-bit or 64-bit operating systems, and can execute all 32-bit x86 binaries in either case. x64 processors have additional registers and 64-bit addressing capabilities that are utilized by the PGI compilers and tools when running on a 64-bit operating system. The prefetch, SSE1, SSE2 and SSE3 processor features further distinguish the various processors. Where such distinctions are important with respect to a given compiler option or feature, it is explicitly noted in this manual. Note that the default for performing scalar floating-point arithmetic is to use SSE instructions on targets that support SSE1 and SSE2. See section 2.3.1, Scalar SSE Code Generation, for a detailed discussion of this topic. Related Publications The following documents contain additional information related to the x86 and x64 architectures, and the compilers and tools available from The Portland Group. • PGI Fortran Reference manual describes the FORTRAN 77, Fortran 90/95, and HPF statements, data types, input/output format specifiers, and additional reference material related to use of the PGI Fortran compilers. • System V Application Binary Interface Processor Supplement by AT&T UNIX System Laboratories, Inc. (Prentice Hall, Inc.). • System V Application Binary Interface X86-64 Architecture Processor Supplement, http://www.x86- 64.org/abi.pdf. • Fortran 95 Handbook Complete ISO/ANSI Reference, Adams et al, The MIT Press, Cambridge, Mass, 1997. • Programming in VAX Fortran, Version 4.0, Digital Equipment Corporation (September, 1984). • IBM VS Fortran, IBM Corporation, Rev. GC26-4119. • The C Programming Language by Kernighan and Ritchie (Prentice Hall).PGI® User’s Guide xxviii • C: A Reference Manual by Samuel P. Harbison and Guy L. Steele Jr. (Prentice Hall, 1987). • The Annotated C++ Reference Manual by Margaret Ellis and Bjarne Stroustrup, AT&T Bell Laboratories, Inc. (Addison-Wesley Publishing Co., 1990). • OpenMP Application Program Interface, Version 2.5 May 2005 (OpenMP Architecture Review Board, 1997-2005).1 Chapter 1. Getting Started This chapter describes how to use the PGI compilers. The command used to invoke a compiler, such as the pgf95 command, is called a compiler driver. The compiler driver controls the following phases of compilation: preprocessing, compiling, assembling, and linking. Once a file is compiled and an executable file is produced, you can execute, debug, or profile the program on your system. Executables produced by the PGI compilers are unconstrained, meaning they can be executed on any compatible x86 or x64 processor-based system, regardless of whether the PGI compilers are installed on that system. Overview In general, using a PGI compiler involves three steps: 1. Produce a program source code in a file containing a .f extension or another appropriate extension, as described in “Input Files,” on page 3. This program may be one that you have written or one that you are modifying. 2. Compile the program using the appropriate compiler command. 3. Execute, debug, or profile the executable file on your system. You might also want to deploy your application, though this is not a required step. The PGI compilers allow many variations on these general program development steps. These variations include the following: • Stop the compilation after preprocessing, compiling or assembling to save and examine intermediate results. • Provide options to the driver that control compiler optimization or that specify various features or limitations. • Include as input intermediate files such as preprocessor output, compiler output, or assembler output. Invoking the Command-level PGI Compilers To translate and link a Fortran, C, or C++ program, the pgf77, pgf95, pghpf, pgcc, and pgcpp commands do the following:PGI® User’s Guide 2 1. Preprocess the source text file. 2. Check the syntax of the source text. 3. Generate an assembly language file. 4. Pass control to the subsequent assembly and linking steps. Example 1.1. Hello program Let’s look at a simple example of using the PGI compiler to create, compile, and execute a program that prints hello. Step 1: Create your program. For this example, suppose you enter the following simple Fortran program in the file hello.f: print *, "hello" end Step 2: Compile the program. When you created your program, you called it hello.f. In this example, we compile it from a shell command prompt using the default pgf95 driver option. Use the following syntax: PGI$ pgf95 hello.f PGI$ By default, the executable output is placed in the file a.out, or, on Windows platforms, in a filename based on the name of the first source or object file on the command line. However, you can use the –o option to specify an output file name. To place the executable output in the file hello, use this command: PGI$ pgf95 -o hello hello.f PGI$ Step 3: Execute the program. To execute the resulting hello program, simply type the filename at the command prompt and press the Return or Enter key on your keyboard: PGI$ hello hello PGI$ Command-line Syntax The compiler command-line syntax, using pgf95 as an example, is: pgf95 [options] [path]filename [...] Where: options is one or more command-line options, all of which are described in detail in Chapter 2, “Using Command Line Options”. path is the pathname to the directory containing the file named by filename. If you do not specify the path for a filename, the compiler uses the current directory. You must specify the path separately for each filename not in the current directory.Chapter 1. Getting Started 3 filename is the name of a source file, preprocessed source file, assembly-language file, object file, or library to be processed by the compilation system. You can specify more than one [path]filename. Command-line Options The command-line options control various aspects of the compilation process. For a complete alphabetical listing and a description of all the command-line options, refer to Chapter 2, “Using Command Line Options”. The following list provides important information about proper use of command-line options. • Case is significant for options and their arguments. • The compiler drivers recognize characters preceded by a hyphen (-) as command-line options. For example, the –Mlist option specifies that the compiler creates a listing file. Note The convention for the text of this manual is to show command-line options using a dash instead of a hyphen; for example, you see –Mlist. • The pgcpp command recognizes a group of characters preceded by a plus sign (+) as command-line options. • The order of options and the filename is not fixed. That is, you can place options before and after the filename argument on the command line. However, the placement of some options is significant, such as the –l option, in which the order of the filenames determines the search order. Note If two or more options contradict each other, the last one in the command line takes precedence. Fortran Directives and C/C++ Pragmas You can insert Fortran directives and C/C++ pragmas in program source code to alter the effects of certain command-line options and to control various aspects of the compilation process for a specific routine or a specific program loop. For more information on Fortran directives and C/C++ pragmas, refer to Chapter 5, “Using OpenMP” and Chapter 6, “Using Directives and Pragmas”. Filename Conventions The PGI compilers use the filenames that you specify on the command line to find and to create input and output files. This section describes the input and output filename conventions for the phases of the compilation process. Input Files You can specify assembly-language files, preprocessed source files, Fortran/C/C++ source files, object files, and libraries as inputs on the command line. The compiler driver determines the type of each input file by examining the filename extensions. The drivers use the following conventions:PGI® User’s Guide 4 filename.f indicates a Fortran source file. filename.F indicates a Fortran source file that can contain macros and preprocessor directives (to be preprocessed). filename.FOR indicates a Fortran source file that can contain macros and preprocessor directives (to be preprocessed). filename.F95 indicates a Fortran 90/95 source file that can contain macros and preprocessor directives (to be preprocessed). filename.f90 indicates a Fortran 90/95 source file that is in freeform format. filename.f95 indicates a Fortran 90/95 source file that is in freeform format. filename.hpf indicates an HPF source file. filename.c indicates a C source file that can contain macros and preprocessor directives (to be preprocessed). filename.i indicates a preprocessed C or C++ source file. filename.C indicates a C++ source file that can contain macros and preprocessor directives (to be preprocessed). filename.cc indicates a C++ source file that can contain macros and preprocessor directives (to be preprocessed). filename.s indicates an assembly-language file. filename.o (Linux, Apple, SFU, SUA) indicates an object file. filename.obj (Windows systems only) indicates an object file. filename.a (Linux, Apple, SFU, SUA) indicates a library of object files. filename.lib (Windows systems only) indicates a statically-linked library of object files. filename.so (Linux and SFU systems only) indicates a library of shared object files. filename.dll (Windows systems only) indicates a dynamically-linked library.Chapter 1. Getting Started 5 filename..objlib (Apple systems only) indicates a dynamically-linked library. The driver passes files with .s extensions to the assembler and files with .o, .obj, .so, .dll, .a and .lib extensions to the linker. Input files with unrecognized extensions, or no extension, are also passed to the linker. Files with a .F (Capital F) or .FOR suffix are first preprocessed by the Fortran compilers and the output is passed to the compilation phase. The Fortran preprocessor functions similar to cpp for C/C++ programs, but is built in to the Fortran compilers rather than implemented through an invocation of cpp. This design ensures consistency in the preprocessing step regardless of the type or revision of operating system under which you’re compiling. Any input files not needed for a particular phase of processing are not processed. For example, if on the command line you specify an assembly-language file (filename.s) and the –S option to stop before the assembly phase, the compiler takes no action on the assembly language file. Processing stops after compilation and the assembler does not run. In this scenario, the compilation must have been completed in a previous pass which created the .s file. For a complete description of the –S option, refer to the following section:“Output Files”. In addition to specifying primary input files on the command line, code within other files can be compiled as part of include files using the INCLUDE statement in a Fortran source file or the preprocessor #include directive in Fortran source files that use a .F extension or C and C++ source files. When linking a program with a library, the linker extracts only those library components that the program needs. The compiler drivers link in several libraries by default. For more information about libraries, refer to Chapter 7, “Creating and Using Libraries”. Output Files By default, an executable output file produced by one of the PGI compilers is placed in the file a.out, or, on Windows, in a filename based on the name of the first source or object file on the command line. As the example in the preceding section shows, you can use the –o option to specify the output file name. If you use one of the options: –F (Fortran only), –P (C/C++ only), –S or –c, the compiler produces a file containing the output of the last completed phase for each input file, as specified by the option supplied. The output file will be a preprocessed source file, an assembly-language file, or an unlinked object file respectively. Similarly, the –E option does not produce a file, but displays the preprocessed source file on the standard output. Using any of these options, the –o option is valid only if you specify a single input file. If no errors occur during processing, you can use the files created by these options as input to a future invocation of any of the PGI compiler drivers. The following table lists the stop-after options and the output files that the compilers create when you use these options. It also describes the accepted input files. Table 1.1. Stop-after Options, Inputs and Outputs Option Stop after Input Output –E preprocessing Source files. For Fortran, must have .F extension. preprocessed file to standard outPGI® User’s Guide 6 Option Stop after Input Output –F preprocessing Source files. Must have .F extension. This option is not valid for pgcc or pgcpp. preprocessed file (.f) –P preprocessing Source files. This option is not valid for pgf77, pgf95 or pghpf) preprocessed file (.i) –S compilation Source files or preprocessed files assembly-language file (.s) –c assembly Source files, preprocessed files or assemblylanguage files unlinked object file (.o or .obj) none linking Source files, preprocessed files, assemblylanguage files, object files or libraries executable file (a.out or .exe) If you specify multiple input files or do not specify an object filename, the compiler uses the input filenames to derive corresponding default output filenames of the following form, where filename is the input filename without its extension: filename.f indicates a preprocessed file, if you compiled a Fortran file using the –F option. filename.i indicates a prepossedfile, if you compiled using the –P option.. filename.lst indicates a listing file from the –Mlist option. filename.o or filename.obj indicates an object file from the –c option. filename.s indicates an assembly-language file from the –S option. Note Unless you specify otherwise, the destination directory for any output file is the current working directory. If the file exists in the destination directory, the compiler overwrites it. The following example demonstrates the use of output filename extensions. $ pgf95 -c proto.f proto1.F This produces the output files proto.o and proto1.o, or, on Windows, proto.obj and proto1.obj all of which are binary object files. Prior to compilation, the file proto1.F is preprocessed because it has a .F filename extension. Fortran, C, and C++ Data Types The PGI Fortran, C, and C++ compilers recognize scalar and aggregate data types. A scalar data type holds a single value, such as the integer value 42 or the real value 112.6. An aggregate data type consists of one or more scalar data type objects, such as an array of integer values.Chapter 1. Getting Started 7 For information about the format and alignment of each data type in memory, and the range of values each type can have on x86 or x64 processor-based systems running a 32-bit operating system, refer to Chapter 13, “Fortran, C and C++ Data Types”. For more information on x86-specific data representation, refer to the System V Application Binary Interface Processor Supplement by AT&T UNIX System Laboratories, Inc. (Prentice Hall, Inc.). This manual specifically does not address x64 processor-based systems running a 64-bit operating system, because the application binary interface (ABI) for those systems is still evolving. For the latest version of this ABI, see www.x86-64.org/abi.pdf. Parallel Programming Using the PGI Compilers The PGI compilers support three styles of parallel programming: • Automatic shared-memory parallel programs compiled using the –Mconcur option to pgf77, pgf95, pgcc, or pgcpp — parallel programs of this variety can be run on shared-memory parallel (SMP) systems such as dual-core or multi-processor workstations. • OpenMP shared-memory parallel programs compiled using the –mp option to pgf77, pgf95, pgcc, or pgcpp — parallel programs of this variety can be run on SMP systems. Carefully coded user-directed parallel programs using OpenMP directives can often achieve significant speed-ups on dual-core workstations or large numbers of processors on SMP server systems. Chapter 5, “Using OpenMP” contains complete descriptions of user-directed parallel programming. • Data parallel shared- or distributed-memory parallel programs compiled using the PGHPF High Performance Fortran compiler — parallel programs of this variety can be run on SMP workstations or servers, distributed-memory clusters of workstations, or clusters of SMP workstations or servers. Coding a data parallel version of an application can be more work than using OpenMP directives, but has the advantage that the resulting executable is usable on all types of parallel systems regardless of whether shared memory is available. See the PGHPF User’s Guide for a complete description of how to build and execute data parallel HPF programs. In this manual, the first two types of parallel programs are collectively referred to as SMP parallel programs. The third type is referred to as a data parallel program, or simply as an HPF program. Some newer CPUs incorporate two or more complete processor cores - functional units, registers, level 1 cache, level 2 cache, and so on - on a single silicon die. These CPUs are known as multi-core processors. For purposes of HPF, threads, or OpenMP parallelism, these cores function as two or more distinct processors. However, the processing cores are on a single chip occupying a single socket on a system motherboard. For purposes of PGI software licensing, a multi-core processor is treated as a single CPU. Running SMP Parallel Programs When you execute an SMP parallel program, by default it uses only one processor. To run on more than one processor, set the NCPUS environment variable to the desired number of processors, subject to a maximum of four for PGI’s workstation-class products. You can set this environment variable by issuing the following command in a Windows command prompt window:PGI® User’s Guide 8 % setenv NCPUS In a shell command window under csh, issue the following command: % setenv NCPUS In sh, ksh, or BASH command window, issue the following command: % NCPUS=; export NCPUS Note If you set NCPUS to a number larger than the number of physical processors, your program may execute very slowly. Running Data Parallel HPF Programs When you execute an HPF program, by default it will use only one processor. If you wish to run on more than one processor, use the -pghpf -np runtime option. For example, to compile and run the hello.f example defined above on one processor, you would issue the following commands: % pghpf -o hello hello.f Linking: % hello hello % To execute it on two processors, you would issue the following commands: % hello -pghpf -np 2 hello % Note If you specify a number larger than the number of physical processors, your program will execute very slowly. You still only see a single “hello” printed to your screen. This is because HPF is a single-threaded model, meaning that all statements execute with the same semantics as if they were running in serial. However, parallel statements or constructs operating on explicitly distributed data are in fact executed in parallel. The programmer must manually insert compiler directives to cause data to be distributed to the available processors. See the PGHPF User’s Guide and The High Performance Fortran Handbook for more details on constructing and executing data parallel programs on shared-memory or distributed-memory cluster systems using PGHPF. Platform-specific considerations There are nine platforms supported by the PGI Workstation and PGI Server compilers and tools: • 32-bit Linux - supported on 32-bit Linux operating systems running on either a 32-bit x86 compatible or an x64 compatible processor.Chapter 1. Getting Started 9 • 64-bit/32-bit Linux - includes all features and capabilities of the 32-bit Linux version, and is also supported on 64-bit Linux operating systems running on an x64 compatible processor. • 32-bit Windows - supported on 32-bit Windows operating systems running on either a 32-bit x86 compatible or an x64 compatible processor. • 64-bit/32-bit Windows - includes all features and capabilities of the 32-bit Windows version, and is also supported on 64-bit Windows operating systems running an x64 compatible processor. • 32-bit SFU - supported on 32-bit Windows operating systems running on either a 32-bit x86 compatible or an x64 compatible processor. • 32-bit SUA - supported on 32-bit Windows operating systems running on either a 32-bit x86 compatible or an x64 compatible processor. • 64-bit/32-bit SUA - includes all features and capabilities of the 32-bit SUA version, and is also supported on 64-bit Windows operating systems running on an x64 compatible processor. • 32-bit Apple Mac OS X - supported on 32-bit Apple Mac operating systems running on either a 32-bit or 64- bit Intel-based Mac system. • 64-bit Apple Mac OS X - supported on 64-bit Apple Mac operating systems running on a 64-bit Intel-based Mac system. The following sections describe the specific considerations required to use the PGI compilers on the various platforms: Linux, Windows, and Apple Mac OS X. Using the PGI Compilers on Linux Linux Header Files The Linux system header files contain many GNU gcc extensions. PGI supports many of these extensions, thus allowing the PGCC C and C++ compilers to compile most programs that the GNU compilers can compile. A few header files not interoperable with the PGI compilers have been rewritten and are included in $PGI/linux86/include. These files are: sigset.h, asm/byteorder.h, stddef.h, asm/ posix_types.h and others. Also, PGI’s version of stdarg.h supports changes in newer versions of Linux. If you are using the PGCC C or C++ compilers, please make sure that the supplied versions of these include files are found before the system versions. This will happen by default unless you explicitly add a –I option that references one of the system include directories. Running Parallel Programs on Linux You may encounter difficulties running auto-parallel or OpenMP programs on Linux systems when the per-thread stack size is set to the default (2MB). If you have unexplained failures, please try setting the environment variable OMP_STACK_SIZE to a larger value, such as 8MB. This can be accomplished with the command in csh: % setenv OMP_STACK_SIZE 8M in bash, sh, or ksh, use: % OMP_STACK_SIZE=8M; export OMP_STACK_SIZEPGI® User’s Guide 10 If your program is still failing, you may be encountering the hard 8 MB limit on main process stack sizes in Linux. You can work around the problem by issuing the following command in csh: % limit stacksize unlimited in bash, sh, or ksh, use: % ulimit -s unlimited Using the PGI Compilers on Windows BASH Shell Environment On Windows platforms, the tools that ship with the PGI Workstation or PGI Server command-level compilers include a full-featured shell command environment. After installation, you should have a PGI icon on your Windows desktop. Double-left-click on this icon to cause an instance of the BASH command shell to appear on your screen. Working within BASH is very much like working within the sh or ksh shells on a Linux system, but in addition BASH has a command history feature similar to csh and several other unique features. Shell programming is fully supported. A complete BASH User’s Guide is available through the PGI online manual set. Select “PGI Workstation” under Start->Programs and double-left-click on the documentation icon to see the online manual set. You must have a web browser installed on your system in order to read the online manuals. The BASH shell window is pre-initialized for usage of the PGI compilers and tools, so there is no need to set environment variables or modify your command path when the command window comes up. In addition to the PGI compiler commands referenced above, within BASH you have access to over 100 common commands and utilities, including but not limited to the following: vi emacs make tar / untar gzip / gunzip ftp sed grep / egrep / fgrep awk cat cksum cp date diff du find kill ls more / less mv printenv / env rm / rmdir touch wc If you are familiar with program development in a Linux environment, editing, compiling, and executing programs within BASH will be very comfortable. If you have not previously used such an environment, you should take time to familiarize yourself with either the vi or emacs editors and with makefiles. The emacs editor has an extensive online tutorial, which you can start by bringing up emacs and selecting the appropriate option under the pull-down help menu. You can get a thorough introduction to the construction and use of makefiles through the online Makefile User’s Guide. For library compatibility, PGI provides versions of ar and ranlib that are compatible with native Windows object-file formats. For more information on these commands, refer to “Creating and Using Static Libraries on Windows,” on page 79.Chapter 1. Getting Started 11 Windows Command Prompt The PGI Workstation entry in the Windows Start menu contains a submenu titled PGI Workstation Tools. This submenu contains a shortcut labeled PGI Command Prompt (32-bit). The shortcut is used to launch a Windows command shell using an environment pre-initialized for the use of the 32-bit PGI compilers and tools. On x64 systems, a second shortcut labeled PGI Command Prompt (64-bit) will also be present. This shortcut launches a Windows command shell using an environment pre-initialized for the use of the 64-bit PGI compilers and tools. Using the PGI Compilers on SUA and SFU Subsystem for Unix Applications (SUA and SFU) Subsystem for Unix Applications (SUA) is a source-compatibility subsystem for running Unix applications on 32-bit and 64-bit Windows server-class operating systems. PGI Workstation for Windows includes compilers and tools for SUA and its 32-bit-only predecessor, Services For Unix (SFU). SUA provides an operating system for POSIX processes. There is a package of support utilities available for download from Microsoft that provides a more complete Unix environment, including features like shells, scripting utilities, a telnet client, development tools, and so on. SUA/SFU Header Files The SUA/SFU system header files contain numerous non-standard extensions. PGI supports many of these extensions, thus allowing the PGCC C and C++ compilers to compile most programs that the GNU compilers can compile. A few header files not interoperable with the PGI compilers have been rewritten and are included in $PGI/sua32/include or $PGI/sua64/include. These files are: stdarg.h, stddef.h, and others. If you are using the PGCC C or C++ compilers, please make sure that the supplied versions of these include files are found before the system versions. This happens by default unless you explicitly add a –I option that references one of the system include directories. Running Parallel Programs on SUA and SFU You may encounter difficulties running auto-parallel or OpenMP programs on SUA/SFU systems when the per-thread stack size is set to the default (2MB). If you have unexplained failures, please try setting the environment variable OMP_STACK_SIZE to a larger value, such as 8MB. This can be accomplished with the command: in csh: % setenv OMP_STACK_SIZE 8M in bash, sh, or ksh. % OMP_STACK_SIZE=8M; export OMP_STACK_SIZE Using the PGI Compilers on Mac OS X Mac OS X Header FilesPGI® User’s Guide 12 The Mac OS X header files contain numerous non-standard extensions. PGI supports many of these extensions, thus allowing the PGCC C and C++ compilers to compile most programs that the GNU compilers can compile. A few header files not interoperable with the PGI compilers have been rewritten and are included in $PGI/ sua32/include or $PGI/sua64/include. These files are: stdarg.h, stddef.h, and others. If you are using the PGCC C or C++ compilers, please make sure that the supplied versions of these include files are found before the system versions. This will happen by default unless you explicitly add a –I option that references one of the system include directories. Running Parallel Programs on Mac OS You may encounter difficulties running auto-parallel or OpenMP programs on Mac OS X systems when the per-thread stack size is set to the default (8MB). If you have unexplained failures, please try setting the environment variable OMP_STACK_SIZE to a larger value, such as 16MB. This can be accomplished with the following command: in csh: % setenv OMP_STACK_SIZE 16M in bash, sh, or ksh. % OMP_STACK_SIZE=16M; export OMP_STACK_SIZE Site-specific Customization of the Compilers If you are using the PGI compilers and want all your users to have access to specific libraries or other files, there are special files that allow you to customize the compilers for your site. Using siterc Files The PGI compiler drivers utilize a file named siterc to enable site-specific customization of the behavior of the PGI compilers. The siterc file is located in the bin subdirectory of the PGI installation directory. Using siterc, you can control how the compiler drivers invoke the various components in the compilation tool chain. Using User rc Files In addition to the siterc file, user rc files can reside in a given user’s home directory, as specified by the user’s HOME environment variable. You can use these files to control the respective PGI compilers. All of these files are optional. On Linux and SUA these files are named .mypgf77rc, .mypgf90rc, .mypgccrc, .mypgcpprc, and .mypghpfrc. On native windows, these files are named mypgf77rc, mypgf95rc, mypgccrc, mypgcpprc, and mypghpfrc. On Windows, these files are named mypgf77rc and mypgf95rc. The following examples show how these rc files can be used to tailor a given installation for a particular purpose.Chapter 1. Getting Started 13 Table 1.2. Examples of Using siterc and User rc Files To do this... Add the line shown to the indicated file Make the libraries found in the following location available to all linux86-64 compilations. /opt/newlibs/64 set SITELIB=/opt/newlibs/64; to /opt/pgi/linux86-64/7.1/bin/siterc Make the libraries found in the following location available to all linux86 compilations. /opt/newlibs/32 set SITELIB=/opt/newlibs/32; to /opt/pgi/linux86/7.1/bin/siterc Add the following new library path to all linux86-64 compilations. /opt/local/fast append SITELIB=/opt/local/fast; to /opt/pgi/linux86-64/7.1/bin/siterc Make the following include path available to all compilations; -I/opt/acml/include set SITEINC=/opt/acml/include; to /opt/pgi/linux86/7.1/bin/siterc and / opt/pgi/linux86-64/7.1/bin/siterc Change –Mmpi to link in the following with linux86-64 compilations. /opt/mympi/64/libmpix.a set MPILIBDIR=/opt/mympi/64; set MPILIBNAME=mpix; to /opt/pgi/linux86-64/7.1/bin/siterc; Have linux86-64 compilations always add –DIS64BIT –DAMD set SITEDEF=IS64BIT AMD; to /opt/pgi/linux86-64/7.1/bin/siterc Build an F90 executable for linux86- 64 or linux86 that resolves PGI shared objects in the relative directory ./REDIST set RPATH=./REDIST ; to ~/.mypgf95rc Note This only affects the behavior of PGF95 for the given user. Common Development Tasks Now that you have a brief introduction to the compiler, let’s look at some common development tasks that you might wish to perform. • When you compile code you can specify a number of options on the command line that define specific characteristics related to how the program is compiled and linked, typically enhancing or overriding the default behavior of the compiler. For a list of the most common command line options and information on all the command line options, refer to Chapter 2, “Using Command Line Options”. • Code optimization and parallelization allow you to organize your code for efficient execution. While possibly increasing compilation time and making the code more difficult to debug, these techniques typicallyPGI® User’s Guide 14 produce code that runs significantly faster than code that does not use them. For more information on optimization and parallelization, refer to Chapter 3, “Using Optimization & Parallelization”. • Function inlining, a special type of optimization, replaces a call to a function or a subroutine with the body of the function or subroutine. This process can speed up execution by eliminating parameter passing and the function or subroutine call and return overhead. In addition, function inlining allows the compiler to optimize the function with the rest of the code. However, function inlining may also result in much larger code size with no increase in execution speed. For more information on function inlining, refer to Chapter 4, “Using Function Inlining”. • Directives and pragmas allow users to place hints in the source code to help the compiler generate better assembly code. You typically use directives and pragmas to control the actions of the compiler in a particular portion of a program without affecting the program as a whole. You place them in your source code where you want them to take effect. A directive or pragma typically stays in effect from the point where included until the end of the compilation unit or until another directive or pragma changes its status. For more information on directives and pragmas, refer to Chapter 5, “Using OpenMP”and Chapter 6, “Using Directives and Pragmas”. • A library is a collection of functions or subprograms used to develop software. Libraries contain "helper" code and data, which provide services to independent programs, allowing code and data to be shared and changed in a modular fashion. The functions and programs in a library are grouped for ease of use and linking. When creating your programs, it is often useful to incorporate standard libraries or proprietary ones. For more information on this topic, refer to Chapter 7, “Creating and Using Libraries”. • Environment variables define a set of dynamic values that can affect the way running processes behave on a computer. It is often useful to use these variables to set and pass information that alters the default behavior of the PGI compilers and the executables which they generate. For more information on these variables, refer to Chapter 8, “ Using Environment Variables”. • Deployment, though possibly an infrequent task, can present some unique issues related to concerns of porting the code to other systems. Deployment, in this context, involves distribution of a specific file or set of files that are already compiled and configured. The distribution must occur in such a way that the application executes accurately on another system which may not be configured exactly the same as the system on which the code was created. For more information on what you might need to know to successfully deploy your code, refer to Chapter 9, “Distributing Files - Deployment”. • An intrinsic is a function available in a given language whose implementation is handled specially by the compiler. Intrinsics make using processor-specific enhancements easier because they provide a C/C++ language interface to assembly instructions. In doing so, the compiler manages details that the user would normally have to be concerned with, such as register names, register allocations, and memory locations of data. For C/C++ programs, PGI provides support for MMX and SSE/SSE2/SSE3 intrinsics. For more information on these intrinsics, refer to Chapter 20, “C/C++ MMX/SSE Inline Intrinsics”.15 Chapter 2. Using Command Line Options A command line option allows you to control specific behavior when a program is compiled and linked. This chapter describes the syntax for properly using command-line options and provides a brief overview of a few of the more common options. Note For a complete list of command-line options, their descriptions and use, refer to Chapter 15, “Command-Line Options Reference,” on page 163. Command Line Option Overview Before looking at all the command-line options, first become familiar with the syntax for these options. There are a large number of options available to you, yet most users only use a few of them. So, start simple and progress into using the more advanced options. By default, the PGI 7.1 compilers generate code that is optimized for the type of processor on which compilation is performed, the compilation host. Before adding options to your command-line, review the sections“Help with Command-line Options,” on page 16 and “Frequently-used Options,” on page 19. Command-line Options Syntax On a command-line, options need to be preceded by a hyphen (-). If the compiler does not recognize an option, it passes the option to the linker. This document uses the following notation when describing options: [item] Square brackets indicate that the enclosed item is optional. {item | item} Braces indicate that you must select one and only one of the enclosed items. A vertical bar (|) separates the choices.PGI® User’s Guide 16 ... Horizontal ellipses indicate that zero or more instances of the preceding item are valid. NOTE Some options do not allow a space between the option and its argument or within an argument. When applicable, the syntax section of the option description in Chapter 15, “Command-Line Options Reference,” on page 163 contains this information. Command-line Suboptions Some options accept several suboptions. You can specify these suboptions either by using the full option statement multiple times or by using a comma-separated list for the suboptions. The following two command lines are equivalent: pgf95 -Mvect=sse -Mvect=noaltcode pgf95 -Mvect=sse,noaltcode Command-line Conflicting Options Some options have an opposite or negated counterpart. For example, both–Mvect and –Mnovect are available. –Mvect enables vectorization and –Mnovect disables it. If you used both of these commands on a command line, they would conflict. Note Rule: When you use conflicting options on a command line, the last encountered option takes precedence over any previous one. This rule is important for a number of reasons. • Some options, such as –fast, include other options. Therefore, it is possible for you to be unaware that you have conflicting options. • You can use this rule to create makefiles that apply specific flags to a set of files, as shown in Example 2.1. Example 2.1. Makefiles with Options In this makefile, CCFLAGS uses vectorization. CCNOVECTFLAGS uses the flags defined for CCFLAGS but disables vectorization. CCFLAGS=c -Mvect=sse CCNOVECTFLAGS=$(CCFLAGS) -Mnovect Help with Command-line Options If you are just getting started with the PGI compilers and tools, it is helpful to know which options are available, when to use them, and which options most users find effective.Chapter 2. Using Command Line Options 17 Using –help The –help option is useful because it provides information about all options supported by a given compiler. You can use –help in one of three ways: • Use –help with no parameters to obtain a list of all the available options with a brief one-line description of each. • Add a parameter to –help to restrict the output to information about a specific option. The syntax for this usage is this: –help For example, suppose you use the following command to restrict the output to information about the - fast option: pgf95 -help -fast The output you see is similar to this: -fast Common optimizations; includes -O2 -Munroll=c:1 -Mnoframe -Mlre In the following example, usage information for –help shows how groups of options can be listed or examined according to function $ pgf95 -help -help -help[=groups|asm|debug|language|linker|opt|other| overall|phase|prepro|suffix|switch|target|variable] Show compiler switches • Add a parameter to –help to restrict the output to a specific set of options or to a building process. The syntax for this usage is this: -help= The previous output from the command pgf95 -help -help shows the available subgroups. For example, you can use the following command to restrict the output to information about options related to debug information generation. pgf95 -help=debug The output you see is similar to this: Debugging switches: -M[no]bounds Generate code to check array bounds -Mchkfpstk Check consistency of floating point stack at subprogram calls (32-bit only) Note: This switch only works on 32-bit. On 64-bit, the switch is ignored. -Mchkstk Check for sufficient stack space upon subprogram entry -Mcoff Generate COFF format object -Mdwarf1 Generate DWARF1 debug information with -g -Mdwarf2 Generate DWARF2 debug information with -g -Mdwarf3 Generate DWARF3 debug information with -g -Melf Generate ELF format object -g Generate information for debugger -gopt Generate information for debugger without disabling optimizationsPGI® User’s Guide 18 For a complete description of subgroups, refer to “–help ,” on page 178. Getting Started with Performance One of top priorities of most users is performance and optimization. This section provides a quick overview of a few of the command-line options that are useful in improving performance. Using –fast and –fastsse Options PGI compilers implement a wide range of options that allow users a fine degree of control on each optimization phase. When it comes to optimization of code, the quickest way to start is to use –fast and –fastsse. These options create a generally optimal set of flags for targets that support SSE/SSE2 capability. They incorporate optimization options to enable use of vector streaming SIMD (SSE/SSE2) instructions for 64-bit targets. They enable vectorization with SSE instructions, cache alignment, and SSE arithmetic to flush to zero mode. Note The contents of the –fast and –fastsse options are host-dependent. Further, you should use these options on both compile and link command lines. • –fast and –fastsse typically include these options: –O2 Specifies a code optimization level of 2. –Munroll=c:1 Unrolls loops, executing multiple instances of the loop during each iteration. –Mnoframe Indicates to not generate code to set up a stack frame. –Mlre Indicates loop-carried redundancy elimination. • These additional options are also typically available when using –fast for 64-bit targets and when using –fastsse for both 32- and 64-bit targets: –Mvect=sse Generates SSE instructions. –Mscalarsse Generates scalar SSE code with xmm registers; implies –Mflushz. –Mcache_align Aligns long objects on cache-line boundaries. –Mflushz Sets SSE to flush-to-zero mode. Note For best performance on processors that support SSE instructions, use the PGF95 compiler, even for FORTRAN 77 code, and the –fast option. To see the specific behavior of –fast for your target, use the following command: pgf95 -help -fastChapter 2. Using Command Line Options 19 Other Performance-related Options While –fast and -fastsse are options designed to be the quickest route to best performance, they are limited to routine boundaries. Depending on the nature and writing style of the source code, the compiler often can perform further optimization by knowing the global context of usage of a given routine. For instance, determining the possible value range of actual parameters of a routine could enable a loop to be vectorized; similarly, determining static occurrence of calls helps to decide which routine is beneficial to inline. These types of global optimizations are under control of Inter Procedural Analysis (IPA) in PGI compilers. Option -Mipa enables Inter Procedural Analysis. -Mpi=fast is the recommended option to get best performances for global optimization. You can also add the suboption inline to enable automatic global inlining across file. You might consider using –Mipa=fast,inline. This option for inter-procedural analysis and global optimization can improve performance. You may also be able to obtain further performance improvements by experimenting with the individual –Mpgflag options detailed in the section“–M Options by Category,” on page 219. These options include –Mvect, –Munroll, –Minline, –Mconcur, and –Mpfi/–Mpfo. However, performance improvements using these options are typically application- and system-dependent. It is important to time your application carefully when using these options to ensure no performance degradations occur. For more information on optimization, refer to Chapter 3, “Using Optimization & Parallelization,” on page 21. For specific information about these options, refer to “–M Optimization Controls,” on page 229. Targeting Multiple Systems; Using the -tp Option The –tp option allows you to set the target architecture. By default, the PGI compiler uses all supported instructions wherever possible when compiling on a given system. As a result, executables created on a given system may not be usable on previous generation systems. For example, executables created on a Pentium 4 may fail to execute on a Pentium III or Pentium II. Processor-specific optimizations can be specified or limited explicitly by using the -tp option. Thus, it is possible to create executables that are usable on previous generation systems. With the exception of k8-64, k8- 64e, p7-64, and x64, any of these sub-options are valid on any x86 or x64 processor-based system. The k8-64, k8-64e, p7-64 and x64 options are valid only on x64 processor-based systems For more information about the -tp option, refer to “–tp [,target...] ,” on page 202. Frequently-used Options In addition to overall performance, there are a number of other options that many users find useful when getting started. The following table provides a brief summary of these options. For more information on these options, refer to the complete description of each option available in Chapter 15, “Command-Line Options Reference,” on page 163. Also, there are a number of suboptions available with each of the –M options listed. For more information on those options, refer to “–M Options by Category”.PGI® User’s Guide 20 Table 2.1. Commonly Used Command Line Options Option Description –fast or –fastsse These options create a generally optimal set of flags for targets that support SSE/SSE2 capability. They incorporate optimization options to enable use of vector streaming SIMD instructions (64-bit targets) and enable vectorization with SEE instructions, cache aligned and flushz. –g Instructs the compiler to include symbolic debugging information in the object module. –gopt Instructs the compiler to include symbolic debugging information in the object file, and to generate optimized code identical to that generated when –g is not specified. –help Provides information about available options. –mcmodel=medium Enables medium=model core generation for 64-bit targets; useful when the data space of the program exceeds 4GB. –Mconcur Instructs the compiler to enable auto-concurrentization of loops. If specified, the compiler uses multiple processors to execute loops that it determines to be parallelizable; thus, loop iterations are split to execute optimally in a multithreaded execution context. –Minfo Instructs the compiler to produce information on standard error. –Minline Passes options to the function inliner. –Mipa=fast,inline Enables interprocedural analysis and optimization. Also enables automatic procedure inlining. –Mneginfo Instructs the compiler to produce information on standard error. –Mpfi and –Mpfo Enable profile feedback driven optimizations. –Mkeepasm Keeps the generated assembly files. –Munroll Invokes the loop unroller to unroll loops, executing multiple instances of the loop during each iteration. This also sets the optimization level to 2 if the level is set to less than 2, or if no –O or –g options are supplied. –M[no]vect Enables/Disables the code vectorizer. --[no_]exceptions Removes exception handling from user code. –o Names the output file. –O Specifies code optimization level where is 0, 1, 2, 3, or 4. –tp [,target...] Specify the type(s) of the target processor(s) to enable generation of PGI Unified Binary executables.21 Chapter 3. Using Optimization & Parallelization Source code that is readable, maintainable, and produces correct results is not always organized for efficient execution. Normally, the first step in the program development process involves producing code that executes and produces the correct results. This first step usually involves compiling without much worry about optimization. After code is compiled and debugged, code optimization and parallelization become an issue. Invoking one of the PGI compiler commands with certain options instructs the compiler to generate optimized code. Optimization is not always performed since it increases compilation time and may make debugging difficult. However, optimization produces more efficient code that usually runs significantly faster than code that is not optimized. The compilers optimize code according to the specified optimization level. Using the –O, –Mvect, –Mipa, and –Mconcur, you can specify the optimization levels. In addition, you can use several –M switches to control specific types of optimization and parallelization. This chapter describes the optimization options displayed in the following list. –fast –Mpfi –Mvect –Mconcur –Mpfo –O –Mipa=fast –Munroll This chapter also describes how to choose optimization options to use with the PGI compilers. This overview will help if you are just getting started with one of the PGI compilers, or wish to experiment with individual optimizations. Complete specifications of each of these options is available in Chapter 15, “Command-Line Options Reference”. Overview of Optimization In general, optimization involves using transformations and replacements that generate more efficient code. This is done by the compiler and involves replacements that are independent of the particular target processor’s architecture as well as replacements that take advantage of the x86 or x64 architecture, instruction set and registers. For the discussion in this and the following chapters, optimization is divided into the following categories:PGI® User’s Guide 22 Local Optimization This optimization is performed on a block-by-block basis within a program’s basic blocks. A basic block is a sequence of statements in which the flow of control enters at the beginning and leaves at the end without the possibility of branching, except at the end. The PGI compilers perform many types of local optimization including: algebraic identity removal, constant folding, common sub-expression elimination, redundant load and store elimination, scheduling, strength reduction, and peephole optimizations. Global Optimization This optimization is performed on a program unit over all its basic blocks. The optimizer performs controlflow and data-flow analysis for an entire program unit. All loops, including those formed by IFs and GOTOs, are detected and optimized. Global optimization includes: constant propagation, copy propagation, dead store elimination, global register allocation, invariant code motion, and induction variable elimination. Loop Optimization: Unrolling, Vectorization, and Parallelization The performance of certain classes of loops may be improved through vectorization or unrolling options. Vectorization transforms loops to improve memory access performance and make use of packed SSE instructions which perform the same operation on multiple data items concurrently. Unrolling replicates the body of loops to reduce loop branching overhead and provide better opportunities for local optimization, vectorization and scheduling of instructions. Performance for loops on systems with multiple processors may also improve using the parallelization features of the PGI compilers. Interprocedural Analysis (IPA) and Optimization Interprocedural analysis (IPA) allows use of information across function call boundaries to perform optimizations that would otherwise be unavailable. For example, if the actual argument to a function is in fact a constant in the caller, it may be possible to propagate that constant into the callee and perform optimizations that are not valid if the dummy argument is treated as a variable. A wide range of optimizations are enabled or improved by using IPA, including but not limited to data alignment optimizations, argument removal, constant propagation, pointer disambiguation, pure function detection, F90/F95 array shape propagation, data placement, vestigial function removal, automatic function inlining, inlining of functions from pre-compiled libraries, and interprocedural optimization of functions from pre-compiled libraries. Function Inlining This optimization allows a call to a function to be replaced by a copy of the body of that function. This optimization will sometimes speed up execution by eliminating the function call and return overhead. Function inlining may also create opportunities for other types of optimization. Function inlining is not always beneficial. When used improperly it may increase code size and generate less efficient code. Profile-Feedback Optimization (PFO) Profile-feedback optimization (PFO) makes use of information from a trace file produced by specially instrumented executables which capture and save information on branch frequency, function and subroutine call frequency, semi-invariant values, loop index ranges, and other input data dependent information that can only be collected dynamically during execution of a program. By definition, use of profile-feedbackChapter 3. Using Optimization & Parallelization 23 optimization is a two-phase process: compilation and execution of a specially-instrumented executable, followed by a subsequent compilation which reads a trace file generated during the first phase and uses the information in that trace file to guide compiler optimizations. Getting Started with Optimizations Your first concern should be getting your program to execute and produce correct results. To get your program running, start by compiling and linking without optimization. Use the optimization level –O0 or select –g to perform minimal optimization. At this level, you will be able to debug your program easily and isolate any coding errors exposed during porting to x86 or x64 platforms. If you want to get started quickly with optimization, a good set of options to use with any of the PGI compilers is –fast –Mipa=fast. For example: $ pgf95 -fast -Mipa=fast prog.f For all of the PGI Fortran, C, and C++ compilers, the –fast, –Mipa=fast options generally produce code that is well-optimized without the possibility of significant slowdowns due to pathological cases. The –fast option is an aggregate option that includes a number of individual PGI compiler options; which PGI compiler options are included depends on the target for which compilation is performed. The –Mipa=fast option invokes interprocedural analysis including several IPA suboptions. For C++ programs, add -Minline=levels:10 --no_exceptions: $ pgcpp -fast -Mipa=fast -Minline=levels:10 --no_exceptions prog.cc Note A C++ program compiled with --no_exceptions will fail if the program uses exception handling. By experimenting with individual compiler options on a file-by-file basis, further significant performance gains can sometimes be realized. However, depending on the coding style, individual optimizations can sometimes cause slowdowns, and must be used carefully to ensure performance improvements. In addition to -fast, the optimization flags most likely to further improve performance are -O3, -Mpfi, -Mpfo, -Minline, and on targets with multiple processors -Mconcur. In addition, the –Msafeptr option can significantly improve performance of C/C++ programs in which there is known to be no pointer aliasing. However, for obvious reasons this command-line option must be used carefully. Three other options which are extremely useful are -help, -Minfo, and -dryrun. –help As described in “Help with Command-line Options,” on page 16, you can see a specification of any commandline option by invoking any of the PGI compilers with -help in combination with the option in question, without specifying any input files. For example: $ pgf95 -help -O Reading rcfile /usr/pgi/linux86-64/7.0/bin/.pgf95rcPGI® User’s Guide 24 -O[] Set optimization level, -O0 to -O4, default -O2 Or you can see the full functionality of -help itself, which can return information on either an individual option or groups of options; type: $ pgf95 -help -help Reading rcfile /usr/pgi_rel/linux86-64/7.0/bin/.pgf95rc -help[=groups|asm|debug|language|linker|opt|other|overall| phase|prepro|suffix|switch|target|variable] –Minfo You can use the -Minfo option to display compile-time optimization listings. When this option is used, the PGI compilers issue informational messages to stderr as compilation proceeds. From these messages, you can determine which loops are optimized using unrolling, SSE instructions, vectorization, parallelization, interprocedural optimizations and various miscellaneous optimizations. You can also see where and whether functions are inlined. You can use the -Mneginfo option to display informational messages listing why certain optimizations are inhibited. For more information on -Minfo, refer to “–M Optimization Controls,” on page 229 –dryrun The –dryrun option can be useful as a diagnostic tool if you need to see the steps used by the compiler driver to preprocess, compile, assemble and link in the presence of a given set of command line inputs. When you specify the –dryrun option, these steps will be printed to stderr but are not actually performed. For example, you can use this option to inspect the default and user-specified libraries that are searched during the link phase, and the order in which they are searched by the linker. The remainder of this chapter describes the –0 options, the loop unroller option –Munroll, the vectorizer option –Mvect, the auto-parallelization option –Mconcur, the interprocedural analysis optimization –Mipa, and the profile-feedback instrumentation (–Mpfi) and optimization (–Mpfo) options. You should be able to get very near optimal compiled performance using some combination of these switches. Local and Global Optimization using -O Using the PGI compiler commands with the –Olevel option (the capital O is for Optimize), you can specify any of the following optimization levels: –O0 Level zero specifies no optimization. A basic block is generated for each language statement. –O1 Level one specifies local optimization. Scheduling of basic blocks is performed. Register allocation is performed. –O2 Level two specifies global optimization. This level performs all level-one local optimization as well as leveltwo global optimization. If optimization is specified on the command line without a level, level 2 is the default.Chapter 3. Using Optimization & Parallelization 25 –O3 Level three specifies aggressive global optimization. This level performs all level-one and level-two optimizations and enables more aggressive hoisting and scalar replacement optimizations that may or may not be profitable. –O4 Level four performs all level-one, level-two, and level-three optimizations and enables hoisting of guarded invariant floating point expressions. Note If you use the -O option to specify optimization and do not specify a level, then level two optimization (-O2) is the default. Level-zero optimization specifies no optimization (–O0). At this level, the compiler generates a basic block for each statement. Performance will almost always be slowest using this optimization level. This level is useful for the initial execution of a program. It is also useful for debugging, since there is a direct correlation between the program text and the code generated. Level-one optimization specifies local optimization (–O1). The compiler performs scheduling of basic blocks as well as register allocation. Local optimization is a good choice when the code is very irregular, such as code that contains many short statements containing IF statements and does not contain loops (DO or DO WHILE statements). Although this case rarely occurs, for certain types of code, this optimization level may perform better than level-two (–O2). The PGI compilers perform many different types of local optimizations, including but not limited to: - Algebraic identity removal - Peephole optimizations - Constant folding - Redundant load and store elimination - Common subexpression elimination - Strength reductions - Local register optimization Level-two optimization (–O2 or –O) specifies global optimization. The –fast option generally will specify global optimization; however, the –fast switch varies from release to release, depending on a reasonable selection of switches for any one particular release. The –O or –O2 level performs all level-one local optimizations as well as global optimizations. Control flow analysis is applied and global registers are allocated for all functions and subroutines. Loop regions are given special consideration. This optimization level is a good choice when the program contains loops, the loops are short, and the structure of the code is regular. The PGI compilers perform many different types of global optimizations, including but not limited to: - Branch to branch elimination - Global register allocation - Constant propagation - Invariant code motion - Copy propagation - Induction variable elimination - Dead store eliminationPGI® User’s Guide 26 You can explicitly select the optimization level on the command line. For example, the following command line specifies level-two optimization which results in global optimization: $ pgf95 -O2 prog.f Specifying –O on the command-line without a level designation is equivalent to –O2. The default optimization level changes depending on which options you select on the command line. For example, when you select the –g debugging option, the default optimization level is set to level-zero (–O0). However, you can use the -gopt option to generate debug information without perturbing optimization if you need to debug optimized code. Refer to “Default Optimization Levels,” on page 42 for a description of the default levels. As noted above, the –fast option includes –O2 on all x86 and x64 targets. If you wish to override this with –O3 while maintaining all other elements of –fast, simply compile as follows: $ pgf95 -fast -O3 prog.f Scalar SSE Code Generation For all processors prior to Intel Pentium 4 and AMD Opteron/Athlon64, for example Intel Pentium III and AMD AthlonXP/MP processors, scalar floating-point arithmetic as generated by the PGI Workstation compilers is performed using x87 floating-point stack instructions. With the advent of SSE/SSE2 instructions on Intel Pentium 4/Xeon and AMD Opteron/Athlon64, it is possible to perform all scalar floating-point arithmetic using SSE/SSE2 instructions. In most cases, this is beneficial from a performance standpoint. The default on 32-bit Intel Pentium II/III (–tp p6, –tp piii, etc.) or AMD AthlonXP/MP (–tp k7) is to use x87 instructions for scalar floating-point arithmetic. The default on Intel Pentium 4/Xeon or Intel EM64T running a 32-bit operating system (–tp p7), AMD Opteron/Athlon64 running a 32-bit operating system (–tp k8-32), or AMD Opteron/Athlon64 or Intel EM64T processors running a 64-bit operating system (–tp k8-64 and –tp p7- 64 respectively) is to use SSE/SSE2 instructions for scalar floating-point arithmetic. The only way to override this default on AMD Opteron/Athlon64 or Intel EM64T processors running a 64-bit operating system is to specify an older 32-bit target (for example –tp k7 or –tp piii). Note There can be significant arithmetic differences between calculations performed using x87 instructions versus SSE/SSE2. By default, all floating-point data is promoted to IEEE 80-bit format when stored on the x87 floating-point stack, and all x87 operations are performed register-to-register in this same format. Values are converted back to IEEE 32-bit or IEEE 64-bit when stored back to memory (for REAL/float and DOUBLE PRECISION/ double data respectively). The default precision of the x87 floating-point stack can be reduced to IEEE 32-bit or IEEE 64-bit globally by compiling the main program with the –pc {32 | 64} option to the PGI Workstation compilers, which is described in detail in Chapter 2, “Using Command Line Options”. However, there is no way to ensure that operations performed in mixed precision will match those produced on a traditional loadstore RISC/UNIX system which implements IEEE 64-bit and IEEE 32-bit registers and associated floating-point arithmetic instructions. In contrast, arithmetic results produced on Intel Pentium 4/Xeon, AMD Opteron/Athlon64 or Intel EM64T processors will usually closely match or be identical to those produced on a traditional RISC/UNIX system if all scalar arithmetic is performed using SSE/SSE2 instructions. You should keep this in mind when portingChapter 3. Using Optimization & Parallelization 27 applications to and from systems which support both x87 and full SSE/SSE2 floating-point arithmetic. Many subtle issues can arise which affect your numerical results, sometimes to several digits of accuracy. Loop Unrolling using –Munroll This optimization unrolls loops, executing multiple instances of the loop during each iteration. This reduces branch overhead, and can improve execution speed by creating better opportunities for instruction scheduling. A loop with a constant count may be completely unrolled or partially unrolled. A loop with a non-constant count may also be unrolled. A candidate loop must be an innermost loop containing one to four blocks of code. The following shows the use of the –Munroll option: $ pgf95 -Munroll prog.f The –Munroll option is included as part of –fast on all x86 and x64 targets. The loop unroller expands the contents of a loop and reduces the number of times a loop is executed. Branching overhead is reduced when a loop is unrolled two or more times, since each iteration of the unrolled loop corresponds to two or more iterations of the original loop; the number of branch instructions executed is proportionately reduced. When a loop is unrolled completely, the loop’s branch overhead is eliminated altogether. Loop unrolling may be beneficial for the instruction scheduler. When a loop is completely unrolled or unrolled two or more times, opportunities for improved scheduling may be presented. The code generator can take advantage of more possibilities for instruction grouping or filling instruction delays found within the loop. Example 3.1 and Example 3.2 show the effect of code unrolling on a segment that computes a dot product. Example 3.1. Dot Product Code REAL*4 A(100), B(100), Z INTEGER I DO I=1, 100 Z = Z + A(i) * B(i) END DO END Example 3.2. Unrolled Dot Product Code REAL*4 A(100), B(100), Z INTEGER I DO I=1, 100, 2 Z = Z + A(i) * B(i) Z = Z + A(i+1) * B(i+1) END DO END Using the –Minfo option, the compiler informs you when a loop is being unrolled. For example, a message indicating the line number, and the number of times the code is unrolled, similar to the following will display when a loop is unrolled: dot: 5, Loop unrolled 5 times Using the c: and n: sub-options to –Munroll, or using –Mnounroll, you can control whether and how loops are unrolled on a file-by-file basis. Using directives or pragmas as specified in Chapter 6,PGI® User’s Guide 28 “Using Directives and Pragmas”, you can precisely control whether and how a given loop is unrolled. Refer to Chapter 2, “Using Command Line Options”, for a detailed description of the –Munroll option. Vectorization using –Mvect The –Mvect option is included as part of –fast on all x86 and x64 targets. If your program contains computationally-intensive loops, the –Mvect option may be helpful. If in addition you specify –Minfo, and your code contains loops that can be vectorized, the compiler reports relevant information on the optimizations applied. When a PGI compiler command is invoked with the –Mvect option, the vectorizer scans code searching for loops that are candidates for high-level transformations such as loop distribution, loop interchange, cache tiling, and idiom recognition (replacement of a recognizable code sequence, such as a reduction loop, with optimized code sequences or function calls). When the vectorizer finds vectorization opportunities, it internally rearranges or replaces sections of loops (the vectorizer changes the code generated; your source code’s loops are not altered). In addition to performing these loop transformations, the vectorizer produces extensive data dependence information for use by other phases of compilation and detects opportunities to use vector or packed Streaming SIMD Extensions (SSE) instructions on processors where these are supported. The –Mvect option can speed up code which contains well-behaved countable loops which operate on large REAL, REAL*4, REAL*8, INTEGER*4, COMPLEX or COMPLEX DOUBLE arrays in Fortran and their C/C++ counterparts. However, it is possible that some codes will show a decrease in performance when compiled with –Mvect due to the generation of conditionally executed code segments, inability to determine data alignment, and other code generation factors. For this reason, it is recommended that you check carefully whether particular program units or loops show improved performance when compiled with this option enabled. Vectorization Sub-options The vectorizer performs high-level loop transformations on countable loops. A loop is countable if the number of iterations is set only before loop execution and cannot be modified during loop execution. Some of the vectorizer transformations can be controlled by arguments to the –Mvect command line option. The following sections describe the arguments that affect the operation of the vectorizer. In addition, some of these vectorizer operations can be controlled from within code using directives and pragmas. For details on the use of directives and pragmas, refer to Chapter 6, “Using Directives and Pragmas,” on page 63. The vectorizer performs the following operations: • Loop interchange • Loop splitting • Loop fusion • Memory-hierarchy (cache tiling) optimizations • Generation of SSE instructions on processors where these are supported • Generation of prefetch instructions on processors where these are supported • Loop iteration peeling to maximize vector alignmentChapter 3. Using Optimization & Parallelization 29 • Alternate code generation By default, –Mvect without any sub-options is equivalent to: -Mvect=assoc,cachesize=c where c is the actual cache size of the machine. This enables the options for nested loop transformation and various other vectorizer options. These defaults may vary depending on the target system. Assoc Option The option –Mvect=assoc instructs the vectorizer to perform associativity conversions that can change the results of a computation due to a round-off error (–Mvect=noassoc disables this option). For example, a typical optimization is to change one arithmetic operation to another arithmetic operation that is mathematically correct, but can be computationally different and generate faster code. This option is provided to enable or disable this transformation, since a round-off error for such associativity conversions may produce unacceptable results. Cachesize Option The option –Mvect=cachesize:n instructs the vectorizer to tile nested loop operations assuming a data cache size of n bytes. By default, the vectorizer attempts to tile nested loop operations, such as matrix multiply, using multi-dimensional strip-mining techniques to maximize re-use of items in the data cache. SSE Option The option –Mvect=sse instructs the vectorizer to automatically generate packed SSE (Streaming SIMD Extensions), SSE2, and prefetch instructions when vectorizable loops are encountered. SSE instructions, first introduced on Pentium III and AthlonXP processors, operate on single-precision floating-point data, and hence apply only to vectorizable loops that operate on single-precision floating-point data. SSE2 instructions, first introduced on Pentium 4, Xeon and Opteron processors, operate on double-precision floating-point data. Prefetch instructions, first introduced on Pentium III and AthlonXP processors, can be used to improve the performance of vectorizable loops that operate on either 32-bit or 64-bit floating-point data. Refer to Table 2, “Processor Options,” on page xxvi for a concise list of processors that support SSE, SSE2 and prefetch instructions. Note Program units compiled with –Mvect=sse will not execute on Pentium, Pentium Pro, Pentium II or first generation AMD Athlon processors. They will only execute correctly on Pentium III, Pentium 4, Xeon, EM64T, AthlonXP, Athlon64 and Opteron systems running an SSE-enabled operating system. Prefetch Option The option –Mvect=prefetch instructs the vectorizer to automatically generate prefetch instructions when vectorizable loops are encountered, even in cases where SSE or SSE2 instructions are not generated. Usually, explicit prefetching is not necessary on Pentium 4, Xeon and Opteron because these processors supportPGI® User’s Guide 30 hardware prefetching; nonetheless, it sometimes can be worthwhile to experiment with explicit prefetching. Prefetching can be controlled on a loop-by-loop level using prefetch directives, which are described in detail in “Prefetch Directives ,” on page 69. Note Program units compiled with –Mvect=prefetch will not execute correctly on Pentium, Pentium Pro, or Pentium II processors. They will execute correctly only on Pentium III, Pentium 4, Xeon, EM64T, AthlonXP, Athlon64 or Opteron systems. In addition, the prefetchw instruction is only supported on AthlonXP, Athlon64 or Opteron systems and can cause instruction faults on non-AMD processors. For this reason, the PGI compilers do not generate prefetchw instructions by default on any target. In addition to these sub-options to –Mvect, several other sub-options are supported. Refer to the description of -M[no]vect in Chapter 15, “Command-Line Options Reference” for a detailed description of all available sub-options. Vectorization Example Using SSE/SSE2 Instructions One of the most important vectorization options is -Mvect=sse. When you use this option, the compiler automatically generates SSE and SSE2 instructions, where possible, when targeting processors on which these instructions are supported. This process can improve performance by up to a factor of two compared with the equivalent scalar code. All of the PGI Fortran, C and C++ compilers support this capability. Table 2, “Processor Options,” on page xxvi shows which x86 and x64 processors support these instructions. Prior to release 7.0 -Mvect=sse was omitted from the compiler switch -fast but included in -fastsse. Since release 7.0 -fast is synonymous with -fastsse and therefore includes -Mvect=sse. In the program in Example 3.3, “Vector operation using SSE instructions”, the vectorizer recognizes the vector operation in subroutine 'loop' when either the compiler switch -Mvect=sse or -fast is used. This example shows the compilation, informational messages, and runtime results using the SSE instructions on an AMD Opteron processor-based system, along with issues that affect SSE performance. First note that the arrays in Example 3.3 are single-precision and that the vector operation is done using a unit stride loop. Thus, this loop can potentially be vectorized using SSE instructions on any processor that supports SSE or SSE2 instructions. SSE operations can be used to operate on pairs of single-precision floatingpoint numbers, and do not apply to double-precision floating-point numbers. SSE2 instructions can be used to operate on quads of single-precision floating-point numbers or on pairs of double-precision floating-point numbers. Loops vectorized using SSE or SSE2 instructions operate much more efficiently when processing vectors that are aligned to a cache-line boundary. You can cause unconstrained data objects of size 16 bytes or greater to be cache-aligned by compiling with the –Mcache_align switch. An unconstrained data object is a data object that is not a common block member and not a member of an aggregate data structure. Note For stack-based local variables to be properly aligned, the main program or function must be compiled with –Mcache_align.Chapter 3. Using Optimization & Parallelization 31 The –Mcache_align switch has no effect on the alignment of Fortran allocatable or automatic arrays. If you have arrays that are constrained, such as vectors that are members of Fortran common blocks, you must specifically pad your data structures to ensure proper cache alignment; –Mcache_align causes only the beginning address of each common block to be cache-aligned. The following examples show the results of compiling the example code with and without –Mvect=sse. Example 3.3. Vector operation using SSE instructions program vector_op parameter (N = 9999) real*4 x(N), y(N), z(N), W(N) do i = 1, n y(i) = i z(i) = 2*i w(i) = 4*i enddo do j = 1, 200000 call loop(x,y,z,w,1.0e0,N) enddo print *, x(1),x(771),x(3618),x(6498),x(9999) end subroutine loop(a,b,c,d,s,n) integer i, n real*4 a(n), b(n), c(n), d(n),s do i = 1, n a(i) = b(i) + c(i) - s * d(i) enddo end Assume the preceding program is compiled as follows, where -Mvect=nosse disables SSE vectorization: % pgf95 -fast -Mvect=nosse -Minfo vadd.f vector_op: 4, Loop unrolled 4 times loop: 18, Loop unrolled 4 times The following output shows a sample result if the generated executable is run and timed on a standalone AMD Opteron 2.2 Ghz system: % /bin/time vadd -1.000000 -771.000 -3618.000 -6498.00 -9999.00 5.39user 0.00system 0:05.40elapsed 99%CP Now, recompile with SSE vectorization enabled, and you see results similar to these: % pgf95 -fast -Minfo vadd.f -o vadd vector_op: 4, Unrolled inner loop 8 times Loop unrolled 7 times (completely unrolled) loop: 18, Generated 4 alternate loops for the inner loop Generated vector sse code for inner loop Generated 3 prefetch instructions for this loop Notice the informational message for the loop at line 18.PGI® User’s Guide 32 • The first two lines of the message indicate that the loop has been vectorized, SSE instructions have been generated, and four alternate versions of the loop have also been generated. The loop count and alignments of the arrays determine which of these versions is executed. • The last line of the informational message indicates that prefetch instructions have been generated for three loads to minimize latency of data transfers from main memory. Executing again, you should see results similar to the following: % /bin/time vadd -1.000000 -771.000 -3618.00 -6498.00 -9999.0 3.59user 0.00system 0:03.59elapsed 100%CPU The result is a 50% speed-up over the equivalent scalar, that is, the non-SSE, version of the program. Speed-up realized by a given loop or program can vary widely based on a number of factors: • When the vectors of data are resident in the data cache, performance improvement using vector SSE or SSE2 instructions is most effective. • If data is aligned properly, performance will be better in general than when using vector SSE operations on unaligned data. • If the compiler can guarantee that data is aligned properly, even more efficient sequences of SSE instructions can be generated. • The efficiency of loops that operate on single-precision data can be higher. SSE2 vector instructions can operate on four single-precision elements concurrently, but only two double-precision elements. Note Compiling with –Mvect=sse can result in numerical differences from the executables generated with less optimization. Certain vectorizable operations, for example dot products, are sensitive to order of operations and the associative transformations necessary to enable vectorization (or parallelization). Auto-Parallelization using -Mconcur With the -Mconcur option the compiler scans code searching for loops that are candidates for autoparallelization. -Mconcur must be used at both compile-time and link-time. When the parallelizer finds opportunities for auto-parallelization, it parallelizes loops and you are informed of the line or loop being parallelized if the -Minfo option is present on the compile line. See “–M Optimization Controls,” on page 229, for a complete specification of -Mconcur. A loop is considered parallelizable if doesn't contain any cross-iteration data dependencies. Cross-iteration dependencies from reductions and expandable scalars are excluded from consideration, enabling more loops to be parallelizable. In general, loops with calls are not parallelized due to unknown side effects. Also, loops with low trip counts are not parallelized since the overhead in setting up and starting a parallel loop will likely outweigh the potential benefits. In addition, the default is to not parallelize innermost loops, since these often by definition are vectorizable using SSE instructions and it is seldom profitable to both vectorize and parallelizeChapter 3. Using Optimization & Parallelization 33 the same loop, especially on multi-core processors. Compiler switches and directives are available to let you override most of these restrictions on auto-parallelization. Auto-parallelization Sub-options The parallelizer performs various operations that can be controlled by arguments to the –Mconcur command line option. The following sections describe these arguments that affect the operation of the vectorizer. In addition, these vectorizer operations can be controlled from within code using directives and pragmas. For details on the use of directives and pragmas, refer to Chapter 6, “Using Directives and Pragmas”. By default, –Mconcur without any sub-options is equivalent to: -Mconcur=dist:block This enables parallelization of loops with blocked iteration allocation across the available threads of execution. These defaults may vary depending on the target system. Altcode Option The option –Mconcur=altcode instructs the parallelizer to generate alternate serial code for parallelized loops. If altcode is specified without arguments, the parallelizer determines an appropriate cutoff length and generates serial code to be executed whenever the loop count is less than or equal to that length. If altcode:n is specified, the serial altcode is executed whenever the loop count is less than or equal to n. If noaltcode is specified, no alternate serial code is generated. Dist Option The option –Mconcur=dist:{block|cyclic} option specifies whether to assign loop iterations to the available threads in blocks or in a cyclic (round-robin) fashion. Block distribution is the default. If cyclic is specified, iterations are allocated to processors cyclically. That is, processor 0 performs iterations 0, 3, 6, etc.; processor 1 performs iterations 1, 4, 7, etc.; and processor 2 performs iterations 2, 5, 8, etc. Cncall Option The option –Mconcur=cncall specifies that it is safe to parallelize loops that contain subroutine or function calls. By default, such loops are excluded from consideration for auto-parallelization. Also, no minimum loop count threshold must be satisfied before parallelization will occur, and last values of scalars are assumed to be safe. The environment variable NCPUS is checked at runtime for a parallel program. If NCPUS is set to 1, a parallel program runs serially, but will use the parallel routines generated during compilation. If NCPUS is set to a value greater than 1, the specified number of processors will be used to execute the program. Setting NCPUS to a value exceeding the number of physical processors can produce inefficient execution. Executing a program on multiple processors in an environment where some of the processors are being time-shared with another executing job can also result in inefficient execution. As with the vectorizer, the -Mconcur option can speed up code if it contains well-behaved countable loops and/or computationally intensive nested loops that operate on arrays. However, it is possible that some codes will show a decrease in performance on multi-processor systems when compiled with -Mconcur due to parallelization overheads, memory bandwidth limitations in the target system, false-sharing of cache lines, orPGI® User’s Guide 34 other architectural or code-generation factors. For this reason, it is recommended that you check carefully whether particular program units or loops show improved performance when compiled using this option. If the compiler is not able to successfully auto-parallelize your application, you should refer to Chapter 5, “Using OpenMP”. It is possible that insertion of explicit parallelization directives or pragmas, and use of the –mp compiler option might enable the application to run in parallel. Loops That Fail to Parallelize In spite of the sophisticated analysis and transformations performed by the compiler, programmers will often note loops that are seemingly parallel, but are not parallelized. In this subsection, we look at some examples of common situations where parallelization does not occur. Innermost Loops As noted earlier in this chapter, the PGI compilers will not parallelize innermost loops by default, because it is usually not profitable. You can override this default using the command-line option –Mconcur=innermost. Timing Loops Often, loops will occur in programs that are similar to timing loops. The outer loop in the following example is one such loop. do j = 1, 2 do i = 1, n a(i) = b(i) + c(i) 1 enddo enddo The outer loop above is not parallelized because the compiler detects a cross-iteration dependence in the assignment to a(i). Suppose the outer loop were parallelized. Then both processors would simultaneously attempt to make assignments into a(1:n). Now in general the values computed by each processor for a(1:n) will differ, so that simultaneous assignment into a(1:n) will produce values different from sequential execution of the loops. In this example, values computed for a(1:n) don’t depend on j, so that simultaneous assignment by both processors will not yield incorrect results. However, it is beyond the scope of the compilers’ dependence analysis to determine that values computed in one iteration of a loop don’t differ from values computed in another iteration. So the worst case is assumed, and different iterations of the outer loop are assumed to compute different values for a(1:n). Is this assumption too pessimistic? If j doesn’t occur anywhere within a loop, the loop exists only to cause some delay, most probably to improve timing resolution. It is not usually valid to parallelize timing loops; to do so would distort the timing information for the inner loops. Scalars Quite often, scalars will inhibit parallelization of non-innermost loops. There are two separate cases that present problems. In the first case, scalars appear to be expandable, but appear in non-innermost loops, as in the following example. do j = 1, n x = b(j) do i = 1, n a(i,j) = x + c(i,j) Chapter 3. Using Optimization & Parallelization 35 enddo enddo There are a number of technical problems to be resolved in order to recognize expandable scalars in noninnermost loops. Until this generalization occurs, scalars like x in the preceding code segment inhibit parallelization of loops in which they are assigned. In the following example, scalar k is not expandable, and it is not an accumulator for a reduction. k = 1 do i = 1, n do j = 1, n 1 a(j,i) = b(k) * x enddo k = i 2 if (i .gt. n/2) k = n - (i - n/2) enddo If the outer loop is parallelized, conflicting values are stored into k by the various processors. The variable k cannot be made local to each processor because the value of k must remain coherent among the processors. It is possible the loop could be parallelized if all assignments to k are placed in critical sections. However, it is not clear where critical sections should be introduced because in general the value for k could depend on another scalar (or on k itself), and code to obtain the value of other scalars must reside in the same critical section. In the example above, the assignment to k within a conditional at label 2 prevents k from being recognized as an induction variable. If the conditional statement at label 2 is removed, k would be an induction variable whose value varies linearly with j, and the loop could be parallelized. Scalar Last Values During parallelization, scalars within loops often need to be privatized; that is, each execution thread has its own independent copy of the scalar. Problems can arise if a privatized scalar is accessed outside the loop. For example, consider the following loop: for (i = 1; i 5.0 ) t = x[i]; } v = t; The value of t may not be computed on the last iteration of the loop. Normally, if a scalar is assigned within a loop and used following the loop, the PGI compilers save the last value of the scalar. However, if the loop is parallelized and the scalar is not assigned on every iteration, it may be difficult, without resorting to costly critical sections, to determine on what iteration t is last assigned. Analysis allows the compiler to determine that a scalar is assigned on each iteration and hence that the loop is safe to parallelize if the scalar is used later, as illustrated in the following example. for ( i = 1; i < n; i++) { if ( x[i] > 0.0 ) { t = 2.0; } else { t = 3.0; y[i] = ...t; } } v = t;PGI® User’s Guide 36 where t is assigned on every iteration of the loop. However, there are cases where a scalar may be privatizable, but if it is used after the loop, it is unsafe to parallelize. Examine the following loop in which each use of t within the loop is reached by a definition from the same iteration. for ( i = 1; i < N; i++ ){ if( x[i] > 0.0 ){ t = x[i]; ... ... y[i] = ...t; } } v = t; Here t is privatizable, but the use of t outside the loop may yield incorrect results, since the compiler may not be able to detect on which iteration of the parallelized loop t is last assigned. The compiler detects the previous cases. When a scalar is used after the loop but is not defined on every iteration of the loop, parallelization does not occur. When the programmer knows that the scalar is assigned on the last iteration of the loop, the programmer may use a directive or pragma to let the compiler know the loop is safe to parallelize. The Fortran directive safe_lastval informs the compiler that, for a given loop, all scalars are assigned in the last iteration of the loop; thus, it is safe to parallelize the loop. We could add the following line to any of our previous examples. cpgi$l safe_lastval The resulting code looks similar to this: cpgi$l safe_lastval ... for (i = 1; i 5.0 ) t = x[i]; } v = t; In addition, a command-line option –Msafe_lastval, provides this information for all loops within the routines being compiled, which essentially provides global scope. Processor-Specific Optimization and the Unified Binary Different processors have differences, some subtle, in hardware features such as instruction sets and cache size. The compilers make architecture-specific decisions about things such as instruction selection, instruction scheduling, and vectorization. By default, the PGI compilers produce code specifically targeted to the type of processor on which the compilation is performed. That is, the default is to use all supported instructions wherever possible when compiling on a given system. As a result, executables created on a given system may not be usable on previous generation systems. For example, executables created on a Pentium 4 may fail to execute on a Pentium III or Pentium II. All PGI compilers have the capability of generating unified binaries, which provide a low-overhead means for generating a single executable that is compatible with and has good performance on more than one hardware platform. You can use the –tp option to control compilation behavior by specifying the processor or processors with which the generated code is compatible. The compilers generate and combine into one executable multipleChapter 3. Using Optimization & Parallelization 37 binary code streams, each optimized for a specific platform. At runtime, the one executable senses the environment and dynamically selects the appropriate code stream. For specific information on the –tp option, see –tp [,target...] . Executable size is automatically controlled via unified binary culling. Only those functions and subroutines where the target affects the generated code have unique binary images, resulting in a code-size savings of from 10% to 90% compared to generating full copies of code for each target. Programs can use PGI Unified Binary even if all of the object files and libraries are not compiled as unified binaries. Like any other object file, you can use PGI Unified Binary object files to create programs or libraries. No special start up code is needed; support is linked in from the PGI libraries. The -Mpfi option disables generation of PGI Unified Binary. Instead, the default target auto-detect rules for the host are used to select the target processor. Interprocedural Analysis and Optimization using –Mipa The PGI Fortran, C and C++ compilers use interprocedural analysis (IPA) that results in minimal changes to makefiles and the standard edit-build-run application development cycle. Other than adding –Mipa to the command line, no other changes are required. For reference and background, the process of building a program without IPA is described below, followed by the minor modifications required to use IPA with the PGI compilers. While the PGCC compiler is used here to show how IPA works, similar capabilities apply to each of the PGI Fortran, C and C++ compilers. Note The examples use Linux file naming conventions. On Windows, ‘.o’ files would be ‘.obj’ files, and ‘a.out’ files would be ‘.exe’ files. Building a Program Without IPA – Single Step Using the pgcc command-level compiler driver, multiple source files can be compiled and linked into a single executable with one command. The following example compiles and links three source files: % pgcc -o a.out file1.c file2.c file3.c In actuality, the pgcc driver executes several steps to produce the assembly code and object files corresponding to each source file, and subsequently to link the object files together into a single executable file. Thus, the command above is roughly equivalent to the following commands performed individually: % pgcc -S -o file1.s file1.c % as -o file1.o file1.s % pgcc -S -o file2.s file2.c % as -o file2.o file2.s % pgcc -S -o file3.s file3.c % as -o file3.o file3.s % pgcc -o a.out file1.o file2.o file3.o If any of the three source files is edited, the executable can be rebuilt with the same command line: % pgcc -o a.out file1.c file2.c file3.c This always works as intended, but has the side-effect of recompiling all of the source files, even if only one has changed. For applications with a large number of source files, this can be time-consuming and inefficient.PGI® User’s Guide 38 Building a Program Without IPA - Several Steps It is also possible to use individual pgcc commands to compile each source file into a corresponding object file, and one to link the resulting object files into an executable: % pgcc -c file1.c % pgcc -c file2.c % pgcc -c file3.c % pgcc -o a.out file1.o file2.o file3.o The pgcc driver invokes the compiler and assembler as required to process each source file, and invokes the linker for the final link command. If you modify one of the source files, the executable can be rebuilt by compiling just that file and then relinking: % pgcc -c file1.c % pgcc -o a.out file1.o file2.o file3.o Building a Program Without IPA Using Make The program compilation and linking process can be simplified greatly using the make utility on systems where it is supported. Suppose you create a makefile containing the following lines: a.out: file1.o file2.o file3.o pgcc $(OPT) -o a.out file1.o file2.o file3.o file1.o: file1.c pgcc $(OPT) -c file1.c file2.o: file2.c pgcc $(OPT) -c file2.c file3.o: file3.c pgcc $(OPT) -c file3.c It is then possible to type a single make command: % make The make utility determines which object files are out of date with respect to their corresponding source files, and invokes the compiler to recompile only those source files and to relink the executable. If you subsequently edit one or more source files, the executable can be rebuilt with the minimum number of recompilations using the same single make command. Building a Program with IPA Interprocedural analysis and optimization (IPA) by the PGI compilers alters the standard and make utility command-level interfaces as little as possible. IPA occurs in three phases: • Collection: Create a summary of each function or procedure, collecting the useful information for interprocedural optimizations. This is done during the compile step if the –Mipa switch is present on the command line; summary information is collected and stored in the object file. • Propagation: Process all the object files to propagate the interprocedural summary information across function and file boundaries. This is done during the link step, when all the object files are combined, if the –Mipa switch is present on the link command line. • Recompile/Optimization: Recompile each of the object files with the propagated interprocedural information, producing a specialized object file. This process is also done during the link step when the –Mipa switch is present on the link command line.Chapter 3. Using Optimization & Parallelization 39 When linking with –Mipa, the PGI compilers automatically regenerate IPA-optimized versions of each object file, essentially recompiling each file. If there are IPA-optimized objects from a previous build, the compilers will minimize the recompile time by reusing those objects if they are still valid. They will still be valid if the IPAoptimized object is newer than the original object file, and the propagated IPA information for that file has not changed since it was optimized. After each object file has been recompiled, the regular linker is invoked to build the application with the IPAoptimized object files. The IPA-optimized object files are saved in the same directory as the original object files, for use in subsequent program builds. Building a Program with IPA - Single Step By adding the –Mipa command line switch, several source files can be compiled and linked with interprocedural optimizations with one command: % pgcc -Mipa=fast -o a.out file1.c file2.c file3.c Just like compiling without –Mipa, the driver executes several steps to produce the assembly and object files to create the executable: % pgcc -Mipa=fast -S -o file1.s file1.c % as -o file1.o file1.s % pgcc -Mipa=fast -S -o file2.s file2.c % as -o file2.o file2.s % pgcc -Mipa=fast -S -o file3.s file3.c % as -o file3.o file3.s % pgcc -Mipa=fast -o a.out file1.o file2.o file3.o In the last step, an IPA linker is invoked to read all the IPA summary information and perform the interprocedural propagation. The IPA linker reinvokes the compiler on each of the object files to recompile them with interprocedural information. This creates three new objects with mangled names: file1_ipa5_a.out.oo.o, file2_ipa5_a.out.oo.o, file2_ipa5_a.out.oo.o The system linker is then invoked to link these IPA-optimized objects into the final executable. Later, if one of the three source files is edited, the executable can be rebuilt with the same command line: % pgcc -Mipa=fast -o a.out file1.c file2.c file3.c This will work, but again has the side-effect of compiling each source file, and recompiling each object file at link time. Building a Program with IPA - Several Steps Just by adding the –Mipa command-line switch, it is possible to use individual pgcc commands to compile each source file, followed by a command to link the resulting object files into an executable: % pgcc -Mipa=fast -c file1.c % pgcc -Mipa=fast -c file2.c % pgcc -Mipa=fast -c file3.c % pgcc -Mipa=fast -o a.out file1.o file2.o file3.o The pgcc driver invokes the compiler and assembler as required to process each source file, and invokes the IPA linker for the final link command. If you modify one of the source files, the executable can be rebuilt by compiling just that file and then relinking: % pgcc -Mipa=fast -c file1.cPGI® User’s Guide 40 % pgcc -Mipa=fast -o a.out file1.o file2.o file3.o When the IPA linker is invoked, it will determine that the IPA-optimized object for file1.o (file1_ipa5_a.out.oo.o) is stale, since it is older than the object file1.o, and hence will need to be rebuilt, and will reinvoke the compiler to generate it. In addition, depending on the nature of the changes to the source file file1.c, the interprocedural optimizations previously performed for file2 and file3 may now be inaccurate. For instance, IPA may have propagated a constant argument value in a call from a function in file1.c to a function in file2.c; if the value of the argument has changed, any optimizations based on that constant value are invalid. The IPA linker will determine which, if any, of any previously created IPA-optimized objects need to be regenerated, and will reinvoke the compiler as appropriate to regenerate them. Only those objects that are stale or which have new or different IPA information will be regenerated, which saves on compile time. Building a Program with IPA Using Make As in the previous two sections, programs can be built with IPA using the make utility, just by adding the –Mipa command-line switch: OPT=-Mipa=fast a.out: file1.o file2.o file3.o pgcc $(OPT) -o a.out file1.o file2.o file3.o file1.o: file1.c pgcc $(OPT) -c file1.c file2.o: file2.c pgcc $(OPT) -c file2.c file3.o: file3.c pgcc $(OPT) -c file3.c Using the single make command invokes the compiler to generate any object files that are out-of-date, then invoke pgcc to link the objects into the executable; at link time, pgcc calls the IPA linker to regenerate any stale or invalid IPA-optimized objects. % make Questions about IPA 1. Why is the object file so large? An object file created with –Mipa contains several additional sections. One is the summary information used to drive the interprocedural analysis. In addition, the object file contains the compiler internal representation of the source file, so the file can be recompiled at link time with interprocedural optimizations. There may be additional information when inlining is enabled. The total size of the object file may be 5-10 times its original size. The extra sections are not added to the final executable. 2. What if I compile with –Mipa and link without –Mipa? The PGI compilers generate a legal object file, even when the source file is compiled with –Mipa. If you compile with –Mipa and link without –Mipa, the linker is invoked on the original object files. A legal executable will be generated; while this will not have the benefit of interprocedural optimizations, any other optimizations will apply. 3. What if I compile without –Mipa and link with –Mipa? At link time, the IPA linker must have summary information about all the functions or routines used in the program. This information is created only when a file is compiled with –Mipa. If you compileChapter 3. Using Optimization & Parallelization 41 a file without –Mipa and then try to get interprocedural optimizations by linking with –Mipa, the IPA linker will issue a message that some routines have no IPA summary information, and will proceed to run the system linker using the original object files. If some files were compiled with –Mipa and others were not, it will determine the safest approximation of the IPA summary information for those files not compiled with –Mipa, and use that to recompile the other files using interprocedural optimizations. 4. Can I build multiple applications in the same directory with –Mipa? Yes. Suppose you have three source files: main1.c, main2.c, and sub.c, where sub.c is shared between the two applications. Suppose you build the first application with –Mipa, using this command: % pgcc -Mipa=fast -o app1 main1.c sub.c The the IPA linker creates two IPA-optimized object files: main1_ipa4_app1.o sub_ipa4_app1.oo It uses them to build the first application. Now suppose you build the second application using this command: % pgcc -Mipa=fast -o app2 main2.c sub.c The IPA linker creates two more IPA-optimized object files: main2_ipa4_app2.oo sub_ipa4_app2.oo Note There are now three object files for sub.c: the original sub.o, and two IPA-optimized objects, one for each application in which it appears. Note 5. How is the mangled name for the IPA-optimized object files generated? The mangled name has '_ipa' appended, followed by the decimal number of the length of the executable file name, followed by an underscore and the executable file name itself. The suffix is changed to .oo (on Linux) or .oobj (on Windows) so linking *.o or *.obj does not pull in the IPAoptimized objects. If the IPA linker determines that the file would not benefit from any interprocedural optimizations, it does not have to recompile the file at link time and uses the original object. Profile-Feedback Optimization using –Mpfi/–Mpfo The PGI compilers support many common profile-feedback optimizations, including semi-invariant value optimizations and block placement. These are performed under control of the –Mpfi/–Mpfo command-line options. When invoked with the –Mpfi option, the PGI compilers instrument the generated executable for collection of profile and data feedback information. This information can be used in subsequent compilations that include the –Mpfo optimization option. –Mpfi must be used at both compile-time and link-time. Programs compiled with –Mpfi include extra code to collect run-time statistics and write them out to a trace file. When the resulting program is executed, a profile feedback trace file pgfi.out is generated in the current working directory.PGI® User’s Guide 42 Note Programs compiled and linked with –Mpfi execute more slowly due to the instrumentation and data collection overhead. You should use executables compiled with –Mpfi only for execution of training runs. When invoked with the –Mpfo option, the PGI compilers use data from a pgfi.out profile feedback tracefile to enable or enhance certain performance optimizations. Use of this option requires the presence of a pgfi.out trace file in the current working directory. Default Optimization Levels The following table shows the interaction between the –O ,–g, and –M options. In the table, level can be 0, 1, 2, 3 or 4, and can be vect, concur, unroll or ipa. The default optimization level is dependent upon these command-line options. Table 3.1. Optimization and –O, –g and –M Options Optimize Option Debug Option –M Option Optimization Level none none none 1 none none –M 2 none –g none 0 –O none or –g none 2 –Olevel none or –g none level –Olevel <= 2 none or –g –M 2 Code that is not optimized yet compiled using the option –O0 can be significantly slower than code generated at other optimization levels. The –M option, where is vect, concur, unroll or ipa, sets the optimization level to 2 if no –O options are supplied. The –fast and –fastsse options set the optimization level to a target-dependent optimization level if no –O options are supplied. Local Optimization Using Directives and Pragmas Command-line options let you specify optimizations for an entire source file. Directives supplied within a Fortran source file and pragmas supplied within a C or C++ source file provide information to the compiler and alter the effects of certain command-line options or the default behavior of the compiler. (Many directives have a corresponding command-line option). While a command line option affects the entire source file that is being compiled, directives and pragmas let you do the following: • Apply, or disable, the effects of a particular command-line option to selected subprograms or to selected loops in the source file (for example, an optimization). • Globally override command-line options. • Tune selected routines or loops based on your knowledge or on information obtained through profiling.Chapter 3. Using Optimization & Parallelization 43 Chapter 6, “Using Directives and Pragmas” provides details on how to add directives and pragmas to your source files. Execution Timing and Instruction Counting As this chapter shows, once you have a program that compiles, executes and gives correct results, you may optimize your code for execution efficiency. Selecting the correct optimization level requires some thought and may require that you compare several optimization levels before arriving at the best solution. To compare optimization levels, you need to measure the execution time for your program. There are several approaches you can take for timing execution. You can use shell commands that provide execution time statistics, you can include function calls in your code that provide timing information, or you can profile sections of code. Timing functions available with the PGI compilers include 3F timing routines, the SECNDS pre-declared function in PGF77 or PGF95, or the SYSTEM_CLOCK or CPU_CLOCK intrinsics in PGF95 or PGHPF. In general, when timing a program, you should try to eliminate or reduce the amount of system level activities such as program loading, I/O and task switching. The following example shows a fragment that indicates how to use SYSTEM_CLOCK effectively within an F90/ F95 or HPF program unit. Example 3.4. Using SYSTEM_CLOCK code fragment . . . integer :: nprocs, hz, clock0, clock1 real :: time integer, allocatable :: t(:) !hpf$ distribute t(cyclic) #if defined (HPF) allocate (t(number_of_processors())) #elif defined (_OPENMP) allocate (t(OMP_GET_NUM_THREADS())) #else allocate (t(1)) #endif call system_clock (count_rate=hz) ! call system_clock(count=clock0) < do work> call system_clock(count=clock1) ! t = (clock1 - clock0) time = real (sum(t)) / (real(hz) * size(t)) . . . Portability of Multi-Threaded Programs on Linux PGI has created two libraries - libpgbind and libnuma - to handle the variations between various implementations of Linux. Some older versions of Linux are lacking certain features that support multi-processor and multi-core systems, in particular, the system call 'sched_setaffinity' and the numa library libnuma. The PGI run-time library uses these features to implement some –Mconcur and –mp operations. These variations have led to the creation of two PGI libraries, libpgbind and libnuma. These libraries are used on all 32-bit and 64-bit Linux systems. These libraries are not needed on Windows.PGI® User’s Guide 44 When a program is linked with the system libnuma library, the program depends on the libnuma library in order to run. On systems without a system libnuma library, the PGI version of libnuma provides the required stubs so that the program links and executes properly. If the program is linked with libpgbind and libnuma, the differences between systems is masked by the different versions of libpgbind and libnuma. In particular, PGI provides two versions of libpgbind - one for systems with working support for sched_setaffinity and another for systems that do not. When a program is deployed to the target system, the proper set of libraries, real or stub, should be deployed with the program. This facility requires that the program be dynamically linked with libpgbind and libnuma. libpgbind On some versions of Linux, the system call sched_setaffinity does not exist or does not work. The library libpgbind is used to work around this problem. During installation, a small test program is compiled, linked, and executed. If the test program compiles, links, and executes successfully, the installed version of libpgbind calls the system sched_setaffinity, otherwise the stub version is installed. libnuma Not all systems have libnuma. Typically, only numa systems will have this library. PGI supplies a stub version of libnuma which satisfies the calls from the PGI runtime to libnuma. Note that libnuma is a shared library that is linked dynamically at runtime. The reason to have a numa library on all systems is to allow multi-threaded programs (e.g. compiled with –Mconcur or –mp ) to be compiled, linked, and executed without regard to whether the host or target systems has a numa library. When the numa library is not available, a multi-threaded program still runs because the calls to the numa library are satisfied by the PGI stub library. During installation, the installation procedure checks for the existence of a real libnuma among the system libraries. If the real library is not found, the PGI stub version is substituted.45 Chapter 4. Using Function Inlining Function inlining replaces a call to a function or a subroutine with the body of the function or subroutine. This can speed up execution by eliminating parameter passing and function/subroutine call and return overhead. It also allows the compiler to optimize the function with the rest of the code. Note that using function inlining indiscriminately can result in much larger code size and no increase in execution speed. The PGI compilers provide two categories of inlining: • Automatic inlining - During the compilation process, a hidden pass precedes the compilation pass. This hidden pass extracts functions that are candidates for inlining. The inlining of functions occurs as the source files are compiled. • Inline libraries - You create inline libraries, for example using the pgf95 compiler driver and the –Mextract and –o options. There is no hidden extract pass but you must ensure that any files that depend on the inline library use the latest version of the inline library. There are important restrictions on inlining. Inlining only applies to certain types of functions. Refer to “Restrictions on Inlining,” on page 49 for more details on function inlining limitations. This chapter describes how to use the following options related to function inlining: –Mextract –Minline –Mrecursive Invoking Function Inlining To invoke the function inliner, use the -Minline option. If you do not specify an inline library, the compiler performs a special prepass on all source files named on the compiler command line before it compiles any of them. This pass extracts functions that meet the requirements for inlining and puts them in a temporary inline library for use by the compilation pass. Several -Minline suboptions let you determine the selection criteria for functions to be inlined. These suboptions include:PGI® User’s Guide 46 except:func Inlines all eligible functions except func, a function in the source text. You can us a comma-separated list to specify multiple functions. [name:]func Inlines all functions in the source text whose name matches func. You can us a comma-separated list to specify multiple functions. [size:]n Inlines functions with a statement count less than or equal to n, the specified size. Note The size n may not exactly equal the number of statements in a selected function; the size parameter is merely a rough gauge. levels:n Inlines n level of function calling levels. The default number is one (1). Using a level greater than one indicates that function calls within inlined functions may be replaced with inlined code. This approach allows the function inliner to automatically perform a sequence of inline and extract processes. [lib:]file.ext Instructs the inliner to inline the functions within the library file file.ext. If no inline library is specified, functions are extracted from a temporary library created during an extract prepass. Tip Create the library file using the -Mextract option. If you specify both a function name and a size n, the compiler inlines functions that match the function name or have n or fewer statements. If a name is used without a keyword, then a name with a period is assumed to be an inline library and a name without a period is assumed to be a function name. If a number is used without a keyword, the number is assumed to be a size. In the following example, the compiler inlines functions with fewer than approximately 100 statements in the source file myprog.f and writes the executable code in the default output file a.out. $ pgf95 -Minline=size:100 myprog.f Refer to “–M Options by Category,” on page 219 for more information on the -Minline options. Using an Inline Library If you specify one or more inline libraries on the command line with the -Minline option, the compiler does not perform an initial extract pass. The compiler selects functions to inline from the specified inline library. If you also specify a size or function name, all functions in the inline library meeting the selection criteria are selected for inline expansion at points in the source text where they are called. If you do not specify a function name or a size limitation for the -Minline option, the compiler inlines every function in the inline library that matches a function in the source text.Chapter 4. Using Function Inlining 47 In the following example, the compiler inlines the function proc from the inline library lib.il and writes the executable code in the default output file a.out. $ pgf95 -Minline=name:proc,lib:lib.il myprog.f The following command line is equivalent to the preceding line, with the exception that in the following example does not use the keywords name: and lib:. You typically use keywords to avoid name conflicts when you use an inline library name that does not contain a period. Otherwise, without the keywords, a period informs the compiler that the file on the command line is an inline library. $ pgf95 -Minline=proc,lib.il myprog.f Creating an Inline Library You can create or update an inline library using the -Mextract command-line option. If you do not specify selection criteria with the -Mextract option, the compiler attempts to extract all subprograms. Several -Mextract options let you determine the selection criteria for creating or updating an inline library. These selection criteria include: func Extracts the function func. You can us a comma-separated list to specify multiple functions. [name:]func Extracts the functions whose name matches func, a function in the source text. [size:]n Limits the size of the extracted functions to functions with a statement count less than or equal to n, the specified size. Note The size n may not exactly equal the number of statements in a selected function; the size parameter is merely a rough gauge. [lib:]ext.lib Stores the extracted information in the library directory ext.lib. If no inline library is specified, functions are extracted to a temporary library created during an extract prepass for use during the compilation stage. When you use the -Mextract option, only the extract phase is performed; the compile and link phases are not performed. The output of an extract pass is a library of functions available for inlining. This output is placed in the inline library file specified on the command line with the –o filename specification. If the library file exists, new information is appended to it. If the file does not exist, it is created. You can use a command similar to the following: $ pgf95 -Mextract=lib:lib.il myfunc.f You can use the -Minline option with the -Mextract option. In this case, the extracted library of functions can have other functions inlined into the library. Using both options enables you to obtain more than one level of inlining. In this situation, if you do not specify a library with the –Minline option, the inline processPGI® User’s Guide 48 consists of two extract passes. The first pass is a hidden pass implied by the –Minline option, during which the compiler extracts functions and places them into a temporary library. The second pass uses the results of the first pass but puts its results into the library that you specify with the –o option. Working with Inline Libraries An inline library is implemented as a directory with each inline function in the library stored as a file using an encoded form of the inlinable function. A special file named TOC in the inline library directory serves as a table of contents for the inline library. This is a printable, ASCII file which can be examined to find out information about the library contents, such as names and sizes of functions, the source file from which they were extracted, the version number of the extractor which created the entry, etc. Libraries and their elements can be manipulated using ordinary system commands. • Inline libraries can be copied or renamed. • Elements of libraries can be deleted or copied from one library to another. • The ls or dir command can be used to determine the last-change date of a library entry. Dependencies When a library is created or updated using one of the PGI compilers, the last-change date of the library directory is updated. This allows a library to be listed as a dependence in a makefile or a PVF property and ensures that the necessary compilations are performed when a library is changed. Updating Inline Libraries - Makefiles If you use inline libraries you need to be certain that they remain up to date with the source files into which they are inlined. One way to assure inline libraries are updated is to include them in a makefile. The makefile fragment in the following example assumes the file utils.f contains a number of small functions used in the files parser.f and alloc.f. The makefile also maintains the inline library utils.il. The makefile updates the library whenever you change utils.f or one of the include files it uses. In turn, the makefile compiles parser.f and alloc.f whenever you update the library. Example 4.1. Sample Makefile SRC = mydir FC = pgf95 FFLAGS = -O2 main.o: $(SRC)/main.f $(SRC)/global.h $(FC) $(FFLAGS) -c $(SRC)/main.f utils.o: $(SRC)/utils.f $(SRC)/global.h $(SRC)/utils.h $(FC) $(FFLAGS) -c $(SRC)/utils.f utils.il: $(SRC)/utils.f $(SRC)/global.h $(SRC)/utils.h $(FC) $(FFLAGS) -Mextract=15 -o utils.il utils.f parser.o: $(SRC)/parser.f $(SRC)/global.h utils.il $(FC) $(FFLAGS) -Minline=utils.il -c $(SRC)/parser.f alloc.o: $(SRC)/alloc.f $(SRC)/global.h utils.il $(FC) $(FFLAGS) -Minline=utils.il -c $(SRC)/alloc.f myprog: main.o utils.o parser.o alloc.o $(FC) -o myprog main.o utils.o parser.o alloc.oChapter 4. Using Function Inlining 49 Error Detection during Inlining To request inlining information from the compiler when you invoke the inliner, specify the –Minfo=inline option. For example: $ pgf95 -Minline=mylib.il -Minfo=inline myext.f Examples Assume the program dhry consists of a single source file dhry.f. The following command line builds an executable file for dhry in which proc7 is inlined wherever it is called: $ pgf95 dhry.f -Minline=proc7 The following command lines build an executable file for dhry in which proc7 plus any functions of approximately 10 or fewer statements are inlined (one level only). Note The specified functions are inlined only if they are previously placed in the inline library, temp.il, during the extract phase. $ pgf95 dhry.f -Mextract=lib:temp.il $ pgf95 dhry.f -Minline=10,proc7,temp.il Using the same source file dhry.f, the following example builds an executable for dhry in which all functions of roughly ten or fewer statements are inlined. Two levels of inlining are performed. This means that if function A calls function B, and B calls C, and both B and C are inlinable, then the version of B which is inlined into A will have had C inlined into it. $ pgf95 dhry.f -Minline=size:10,levels:2 Restrictions on Inlining The following Fortran subprograms cannot be extracted: • Main or BLOCK DATA programs. • Subprograms containing alternate return, assigned GO TO, DATA, SAVE, or EQUIVALENCE statements. • Subprograms containing FORMAT statements. • Subprograms containing multiple entries. A Fortran subprogram is not inlined if any of the following applies: • It is referenced in a statement function. • A common block mismatch exists; in other words, the caller must contain all common blocks specified in the callee, and elements of the common blocks must agree in name, order, and type (except that the caller's common block can have additional members appended to the end of the common block). • An argument mismatch exists; in other words, the number and type (size) of actual and formal parameters must be equal.PGI® User’s Guide 50 • A name clash exists, such as a call to subroutine xyz in the extracted subprogram and a variable named xyz in the caller. The following types of C and C++ functions cannot be inlined: • Functions containing switch statements • Functions which reference a static variable whose definition is nested within the function • Function which accept a variable number of arguments Certain C/C++ functions can only be inlined into the file that contains their definition: • Static functions • Functions which call a static function • Functions which reference a static variable51 Chapter 5. Using OpenMP The PGF77 and PGF95 Fortran compilers support the OpenMP Fortran Application Program Interface. The PGCC ANSI C and C++ compilers support the OpenMP C/C++ Application Program Interface. The OpenMP shared-memory parallel programming model is defined by a collection of compiler directives or pragmas, library routines, and environment variables that can be used to specify shared-memory parallelism in Fortran, C and C++ programs. The Fortran directives and C/C++ pragmas include a parallel region construct for writing coarse grain SPMD programs, work-sharing constructs which specify that DO loop iterations or C/C++ for loop iterations should be split among the available threads of execution, and synchronization constructs. The data environment is controlled either by using clauses on the directives or pragmas, or with additional directives or pragmas. Run-time library routines are provided to query the parallel runtime environment, for example to determine how many threads are participating in execution of a parallel region. Finally, environment variables are provided to control the execution behavior of parallel programs. For more information on OpenMP, see www.openmp.org. Fortran directives and C/C++ pragmas allow users to place hints in the source code to help the compiler generate better assembly code. You typically use directives and pragmas to control the actions of the compiler in a particular portion of a program without affecting the program as a whole. You place them in your source code where you want them to take effect. Typically they stay in effect from the point where included until the end of the compilation unit or until another directive or pragma changes its status. Fortran Parallelization Directives Parallelization directives are comments in a program that are interpreted by the PGI Fortran compilers when the option –mp is specified on the command line. The form of a parallelization directive is: sentinel directive_name [clauses] With the exception of the SGI-compatible DOACROSS directive, the sentinel must comply with these rules: • Be one of these: !$OMP, C$OMP, or *$OMP. • Must start in column 1 (one). • Must appear as a single word without embedded white space. • The sentinel marking a DOACROSS directive is C$.PGI® User’s Guide 52 The directive_name can be any of the directives listed in Table 5.1, “Directive and Pragma Summary Table,” on page 53. The valid clauses depend on the directive. Chapter 16, “OpenMP Reference Information” provides a list of directives and their clauses, their usage, and examples. In addition to the sentinel rules, the directive must also comply with these rules: • Standard Fortran syntax restrictions, such as line length, case insensitivity, and so on, apply to the directive line. • Initial directive lines must have a space or zero in column six. • Continuation directive lines must have a character other than a space or a zero in column six. Continuation lines for C$DOACROSS directives are specified using the C$& sentinel. • Directives which are presented in pairs must be used in pairs. Clauses associated with directives have these characteristics: • The order in which clauses appear in the parallelization directives is not significant. • Commas separate clauses within the directives, but commas are not allowed between the directive name and the first clause. • Clauses on directives may be repeated as needed, subject to the restrictions listed in the description of each clause. C/C++ Parallelization Pragmas Parallelization pragmas are #pragma statements in a C or C++ program that are interpreted by the PGCC C and C++ compilers when the option -mp is specified on the command line. The form of a parallelization pragma is: #pragma omp pragma_name [clauses] The format for pragmas include these standards: • The pragmas follow the conventions of the C and C++ standards. • Whitespace can appear before and after the #. • Preprocessing tokens following the #pragma omp are subject to macro replacement. • The order in which clauses appear in the parallelization pragmas is not significant. • Spaces separate clauses within the pragmas. • Clauses on pragmas may be repeated as needed subject to the restrictions listed in the description of each clause. For the purposes of the OpenMP pragmas, a C/C++ structured block is defined to be a statement or compound statement (a sequence of statements beginning with { and ending with }) that has a single entry and a single exit. No statement or compound statement is a C/C++ structured block if there is a jump into or out of that statement.Chapter 5. Using OpenMP 53 Directive and Pragma Recognition The compiler option –mp enables recognition of the parallelization directives and pragmas. The use of this option also implies: –Mreentrant Local variables are placed on the stack and optimizations, such as -Mnoframe, that may result in nonreentrant code are disabled. –Miomutex For directives, critical sections are generated around Fortran I/O statements. For pragmas, calls to I/O library functions are system-dependent and are not necessarily guaranteed to be thread-safe. I/O library calls within parallel regions should be protected by critical regions, as shown in the examples in Chapter 16, “OpenMP Reference Information”, to ensure they function correctly on all systems. Directive and Pragma Summary Table The following table provides a brief summary of the directives and pragmas that PGI supports. For complete information on these statement and examples, refer to Chapter 16, “OpenMP Reference Information”. Table 5.1. Directive and Pragma Summary Table Fortran Directive and C/C++ Pragma Description “ATOMIC ,” on page 244 omp atomic Semantically equivalent to enclosing a single statement in the CRITCIAL...END CRITICAL directive or omp critical pragma. Note: Only certain statements are allowed. “BARRIER,” on page 244 omp barrier Synchronizes all threads at a specific point in a program so that all threads complete work to that point before any thread continues. “CRITICAL ... END CRITICAL and omp critical ,” on page 245 Defines a subsection of code within a parallel region, a critical section, which is executed one thread at a time. “DO ... END DO and omp for ,” on page 247 Provides a mechanism for distribution of loop iterations across the available threads in a parallel region. “C$DOACROSS ,” on page 246 Specifies that the compiler should parallelize the loop to which it applies, even though that loop is not contained within a parallel region. “FLUSH and omp flush pragma ,” on page 249 When this appears, all processor-visible data items, or, when a list is present (FLUSH [list]), only those specified in the list, are written to memory, thus ensuring that all the threads in a team have a consistent view of certain objects in memory. “MASTER ... END MASTER and omp master pragma ” Designates code that executes on the master thread and that is skipped by the other threads.PGI® User’s Guide 54 Fortran Directive and C/C++ Pragma Description “ORDERED ,” on page 251 omp ordered Defines a code block that is executed by only one thread at a time, and in the order of the loop iterations; this makes the ordered code block sequential, while allowing parallel execution of statements outside the code block. “PARALLEL DO ,” on page 254 omp parallel for Enables you to specify which loops the compiler should parallelize. “PARALLEL ... END PARALLEL and omp parallel ,” on page 251 Supports a fork/join execution model in which a single thread executes all statements until a parallel region is encountered. “PARALLEL SECTIONS ,” on page 255 omp parallel sections Defines a non-iterative work-sharing construct without the need to define an enclosing parallel region. “PARALLEL WORKSHARE ,” on page 256 Provides a short form method for including a WORKSHARE directive inside a PARALLEL construct. “SECTIONS … END SECTIONS ,” on page 257 omp sections Defines a non-iterative work-sharing construct within a parallel region. “SINGLE ... END SINGLE,” on page 257S omp master Designates code that executes on a single thread and that is skipped by the other threads. “THREADPRIVATE ,” on page 258 omp threadprivate When a common block or variable that is initialized appears in this directive or pragma, each thread’s copy is initialized once prior to its first use. “WORKSHARE ... END WORKSHARE,” on page 259 omp for Provides a mechanism to effect parallel execution of noniterative but implicitly data parallel constructs. Directive and Pragma Clauses Some directives and pragmas accept clauses that further allow a user to control the scope attributes of variables for the duration of the directive or pragma. Not all clauses are allowed on all directives, so the clauses that are valid are included with the description of the directive and pragma. Typically, if no data scope clause is specified for variables, the default scope is share. Table 16.2, “Directive and Pragma Clauses ,” on page 260 provides a brief summary of the clauses associated with OPENMP directives and pragmas that PGI supports.Chapter 5. Using OpenMP 55 For complete information on these clauses, refer to the OpenMP documentation available on the WorldWide Web. Run-time Library Routines User-callable functions are available to the Fortran and to the OpenMP C/C++ programmer to query and alter the parallel execution environment. Any C/C++ program unit that invokes these functions should include the statement #include . The omp.h include file contains definitions for each of the C/C++ library routines and two required type definitions. For example, to use the omp_get_num_threads function, use this syntax: #include int omp_get_num_threads(void); The following table summarizes the run-time library calls. Note The Fortran call is shown first followed by the equivalent C++ call. Table 5.2. Run-time Library Call Summary Run-time Library Call with Examples omp_get_num_threads Returns the number of threads in the team executing the parallel region from which it is called. When called from a serial region, this function returns 1. A nested parallel region is the same as a single parallel region. By default, the value returned by this function is equal to the value of the environment variable OMP_NUM_THREADS or to the value set by the last previous call to omp_set_num_threads(). Fortran integer omp_get_num_threads() C/C++ #include int omp_get_num_threads(void); omp_set_num_threads Sets the number of threads to use for the next parallel region. This subroutine or function can only be called from a serial region of code. If it is called from within a parallel region, or from within a subroutine or function that is called from within a parallel region, the results are undefined. Further, this subroutine or function has precedence over the OMP_NUM_THREADS environment variable. Fortran subroutine omp_set_num_threads(scalar_integer_exp) C/C++ #include void omp_set_num_threads(int num_threads); omp_get_thread_num Returns the thread number within the team. The thread number lies between 0 and omp_get_num_threads()-1. When called from a serial region, this function returns 0. A nested parallel region is the same as a single parallel region. Fortran integer omp_get_thread_num()PGI® User’s Guide 56 Run-time Library Call with Examples C/C++ #include int omp_get_thread_num(void); omp_get_max_threads Returns the maximum value that can be returned by calls to omp_get_num_threads(). If omp_set_num_threads() is used to change the number of processors, subsequent calls to omp_get_max_threads() return the new value. Further, this function returns the maximum value whether executing from a parallel or serial region of code. Fortran integer function omp_get_max_threads() C/C++ #include void omp_get_max_threads(void) omp_get_num_procs Returns the number of processors that are available to the program Fortran integer function omp_get_num_procs() C/C++ #include int omp_get_num_procs(void); omp_get_stack_size Returns the value of the OpenMP internal control variable that specifies the size that is used to create a stack for a newly created thread. This value may not be the size of the stack of the current thread. Fortran !omp_get_stack_size interface function omp_get_stack_size () use omp_lib_kinds integer ( kind=OMP_STACK_SIZE_KIND ) :: omp_get_stack_size end function omp_get_stack_size end interface C/C++ #include size_t omp_get_stack_size(void); omp_set_stack_size Changes the value of the OpenMP internal control variable that specifies the size to be used to create a stack for a newly created thread. The integer argument specifies the stack size in kilobytes. The size of the stack of the current thread cannot be changed. In the PGI implementation, all OpenMP or auto-parallelization threads are created just prior to the first parallel region; therefore, only calls to omp_set_stack_size() that occur prior to the first region have an effect. Fortran: subroutine omp_set_stack_size(integer(KIND=OMP_STACK_SIZE_KIND)) C/C++ #include void omp_set_stack_size(size_t); omp_in_parallel Returns whether or not the call is within a parallel region. Returns .TRUE.for directives and non-zero for pragmas if called from within a parallel region and .FALSE. for directives and zero for pragmas if called outside of a parallel region. When calledChapter 5. Using OpenMP 57 Run-time Library Call with Examples from within a parallel region that is serialized, for example in the presence of an IF clause evaluating .FALSE.for directives and zero for pragmas, the function returns .FALSE. for directives and zero for pragmas. Fortran logical function omp_in_parallel() C/C++ #include int omp_in_parallel(void); omp_set_dynamic Allows automatic dynamic adjustment of the number of threads used for execution of parallel regions. This function is recognized, but currently has no effect. Fortran subroutine omp_set_dynamic(scalar_logical_exp) C/C++ #include void omp_set_dynamic(int dynamic_threads); omp_get_dynamic Allows the user to query whether automatic dynamic adjustment of the number of threads used for execution of parallel regions is enabled. This function is recognized, but currently always returns .FALSE.for directives and zero for pragmas. Fortran logical function omp_get_dynamic() C/C++ #include void omp_get_dynamic(void); omp_set_nested Allows enabling/disabling of nested parallel regions. This function is recognized, but currently has no effect. Fortran subroutine omp_set_nested(scalar_logical_exp) C/C++ #include void omp_set_nested(int nested); omp_get_nested Allows the user to query whether dynamic adjustment of the number of threads available for execution of parallel regions is enabled. This function is recognized, but currently always returns .FALSE. for directives and zero for pragmas. Fortran logical function omp_get_nested() C/C++ #include int omp_get_nested(void); omp_get_wtime Returns the elapsed wall clock time, in seconds, as a DOUBLE PRECISION value for directives and as a floating-point double value for pragmas. Times returned are per-thread times, and are not necessarily globally consistent across all threads. Fortran double precision function omp_get_wtime() C/C++ #include double omp_get_wtime() omp_get_wtickPGI® User’s Guide 58 Run-time Library Call with Examples Returns the resolution of omp_get_wtime(), in seconds, as a DOUBLE PRECISION value for Fortran directives and as a floating-point double value for C/C++ pragmas. Fortran double precision function omp_get_wtick() C/C++ #include double omp_get_wtick() omp_init_lock Initializes a lock associated with the variable lock for use in subsequent calls to lock routines. The initial state of the lock is unlocked. If the variable is already associated with a lock, it is illegal to make a call to this routine. Fortran subroutine omp_init_lock(integer_var) C/C++ #include void omp_init_lock(omp_lock_t *lock); void omp_init_nest_lock(omp_nest_lock_t *lock); omp_destroy_lock Disassociates a lock associated with the variable. Fortran subroutine omp_destroy_lock(integer_var) C/C++ #include void omp_destroy_lock(omp_lock_t *lock); void omp_destroy_nest_lock(omp_nest_lock_t *lock); omp_set_lock Causes the calling thread to wait until the specified lock is available. The thread gains ownership of the lock when it is available. If the variable is not already associated with a lock, it is illegal to make a call to this routine. Fortran subroutine omp_set_lock(integer_var) C/C++ #include void omp_set_lock(omp_lock_t *lock); void omp_set_nest_lock(omp_nest_lock_t *lock); omp_unset_lock Causes the calling thread to release ownership of the lock associated with integer_var. If the variable is not already associated with a lock, it is illegal to make a call to this routine. Fortran subroutine omp_unset_lock(integer_var) C/C++ #include void omp_unset_lock(omp_lock_t *lock); void omp_unset_nest_lock(omp_nest_lock_t *lock); omp_test_lock Causes the calling thread to try to gain ownership of the lock associated with the variable. The function returns .TRUE.for directives and non-zero for pragmas if the thread gains ownership of the lock; otherwise it returns .FALSE. for directives and zero for pragmas. If the variable is not already associated with a lock, it is illegal to make a call to this routine. Fortran logical function omp_test_lock(integer_var) C/C++ #include int omp_test_lock(omp_lock_t *lock);Chapter 5. Using OpenMP 59 Run-time Library Call with Examples int omp_test_nest_lock(omp_nest_lock_t *lock); Environment Variables You can use OpenMP environment variables to control the behavior of OpenMP programs. These environment variables allow you to set and pass information that can alter the behavior of directives and pragmas. The following summary table is a quick reference for the OPENMP environment variables that PGI uses. Detailed descriptions of each of these variables immediately follows the table. Table 5.3. OpenMP-related Environment Variable Summary Table Environment Variable Default Description OMP_DYNAMIC FALSE Currently has no effect. Typically enables (TRUE) or disables (FALSE) the dynamic adjustment of the number of threads. OMP_NESTED FALSE Currently has no effect. Typically enables (TRUE) or disables (FALSE) nested parallelism. OMP_NUM_THREADS 1 Specifies the number of threads to use during execution of parallel regions. OMP_SCHEDULE STATIC with chunk size of 1 Specifies the type of iteration scheduling and optionally the chunk size to use for omp for and omp parallel for loops that include the run-time schedule clause. OMP_STACK_SIZE Overrides the default stack size for a newly created thread. OMP_WAIT_POLICY ACTIVE Sets the behavior of idle threads, defining whether they spin or sleep when idle. The values are ACTIVE and PASSIVE. OMP_DYNAMIC OMP_DYNAMIC currently has no effect. Typically this variable enables (TRUE) or disables (FALSE) the dynamic adjustment of the number of threads. OMP_NESTED OMP_NESTED currently has no effect. Typically this variable enables (TRUE) or disables (FALSE) nested parallelism. OMP_NUM_THREADS OMP_NUM_THREADS specifies the number of threads to use during execution of parallel regions. The default value for this variable is 1. For historical reasons, the environment variable NCPUS is supported with the same functionality. In the event that both OMP_NUM_THREADS and NCPUS are defined, the value of OMP_NUM_THREADS takes precedence.PGI® User’s Guide 60 NOTE OMP_NUM_THREADS threads is used to execute the program regardless of the number of physical processors available in the system. As a result, you can run programs using more threads than physical processors and they execute correctly. However, performance of programs executed in this manner can be unpredictable, and oftentimes will be inefficient. OMP_SCHEDULE OMP_SCHEDULE specifies the type of iteration scheduling to use for DO and PARALLEL DO loop directives and for omp for and omp parallel for loop pragmas that include the SCHEDULE(RUNTIME) clause, described in “Schedule Clause,” on page 261. The default value for this variable is STATIC If the optional chunk size is not set, a chunk size of 1 is assumed except in the case of a static schedule. For a static schedule, the default is as defined in “DO ... END DO and omp for ,” on page 247. Examples of the use of OMP_SCHEDULE are as follows: For Fortran: $ setenv OMP_SCHEDULE "STATIC, 5" $ setenv OMP_SCHEDULE "GUIDED, 8" $ setenv OMP_SCHEDULE "DYNAMIC" For C/C++: $ setenv OMP_SCHEDULE "static, 5" $ setenv OMP_SCHEDULE "guided, 8" $ setenv OMP_SCHEDULE "dynamic" OMP_STACK_SIZE OMP_STACK_SIZE is an OpenMP 3.0 feature that controls the size of the stack for newly-created threads. This variable overrides the default stack size for a newly created thread. The value is a decimal integer followed by an optional letter B, K, M, or G, to specify bytes, kilobytes, megabytes, and gigabytes, respectively. If no letter is used, the default is kilobytes. There is no space between the value and the letter; for example, one megabyte is specified 1M. The following example specifies a stack size of 8 megabytes. $ setenv OMP_STACK_SIZE 8M The API functions related to OMP_STACK_SIZE are omp_set_stack_size and omp_get_stack_size. The environment variable OMP_STACK_SIZE is read on program start-up. If the program changes its own environment, the variable is not re-checked. This environment variable takes precedence over MPSTKZ, described in “MPSTKZ,” on page 94. Once a thread is created, its stack size cannot be changed. In the PGI implementation, threads are created prior to the first parallel region and persist for the life of the program. The stack size of the main program is set at program start-up and is not affected by OMP_STACK_SIZE. For more information on controlling the program stack size in Linux, refer to “Running Parallel Programs on Linux,” on page 9. OMP_WAIT_POLICY OMP_WAIT_POLICY sets the behavior of idle threads - specifically, whether they spin or sleep when idle. The values are ACTIVE and PASSIVE, with ACTIVE the default. The behavior defined by OMP_WAIT_POLICY is also shared by threads created by auto-parallelization.Chapter 5. Using OpenMP 61 • Threads are considered idle when waiting at a barrier, when waiting to enter a critical region, or when unemployed between parallel regions. • Threads waiting for critical sections always busy wait (ACTIVE). • Barriers always busy wait (ACTIVE), with calls to sched_yield determined by the environment variable MP_SPIN, described in “MP_SPIN,” on page 95. • Unemployed threads during a serial region can either busy wait using the barrier (ACTIVE) or politely wait using a mutex (PASSIVE). This choice is set by OMP_WAIT_POLICY, so the default is ACTIVE. When ACTIVE is set, idle threads consume 100% of their CPU allotment spinning in a busy loop waiting to restart in a parallel region. This mechanism allows for very quick entry into parallel regions, a condition which is good for programs that enter and leave parallel regions frequently. When PASSIVE is set, idle threads wait on a mutex in the operating system and consume no CPU time until being restarted. Passive idle is best when a program has long periods of serial activity or when the program runs on a multi-user machine or otherwise shares CPU resources.6263 Chapter 6. Using Directives and Pragmas It is often useful to be able to alter the effects of certain command line options or default behavior of the compiler. Fortran directives and C/C++ pragmas provide pragmatic information that control the actions of the compiler in a particular portion of a program without affecting the program as a whole. That is, while a command line option affects the entire source file that is being compiled, directives and pragmas apply, or disable, the effects of a command line option to selected subprograms or to selected loops in the source file, for example, to optimize a specific area of code. Use directives and pragmas to tune selected routines or loops. PGI Proprietary Fortran Directives PGI Fortran compilers support proprietary directives that may have any of the following forms: !pgi$g directive !pgi$r directive !pgi$l directive !pgi$ directive Note If the input is in fixed format, the comment character must begin in column 1 and either * or C is allowed in place of !. The scope indicator occurs after the $; this indicator controls the scope of the directive. Some directives ignore the scope indicator. The valid scopes, shown above, are: g (global) indicates the directive applies to the end of the source file. r (routine) indicates the directive applies to the next subprogram. l (loop) indicates the directive applies to the next loop (but not to any loop contained within the loop body). Loop-scoped directives are only applied to DO loops.PGI® User’s Guide 64 blank indicates that the default scope for the directive is applied. The body of the directive may immediately follow the scope indicator. Alternatively, any number of blanks may precede the name of the directive. Any names in the body of the directive, including the directive name, may not contain embedded blanks. Blanks may surround any special characters, such as a comma or an equal sign. The directive name, including the directive prefix, may contain upper or lower case letters, and the case is not significant. Case is significant for any variable names that appear in the body of the directive if the command line option –Mupcase is selected. For compatibility with other vendors’ directives, the prefix cpgi$ may be substituted with cdir$ or cvd$. Note If the input is in fixed format, the comment character must begin in column 1. PGI Proprietary C and C++ Pragmas Pragmas may be supplied in a C/C++ source file to provide information to the compiler. Many pragmas have a corresponding command-line option. Pragmas may also toggle an option, selectively enabling and disabling the option. The general syntax of a pragma is: #pragma [ scope ] pragma-body The optional scope field is an indicator for the scope of the pragma; some pragmas ignore the scope indicator. The valid scopes are: global indicates the pragma applies to the entire source file. routine indicates the pragma applies to the next function. loop indicates the pragma applies to the next loop (but not to any loop contained within the loop body). Loopscoped pragmas are only applied to for and while loops. If a scope indicator is not present, the default scope, if any, is applied. Whitespace must appear after the pragma keyword and between the scope indicator and the body of the pragma. Whitespace may also surround any special characters, such as a comma or an equal sign. Case is significant for the names of the pragmas and any variable names that appear in the body of the pragma. PGI Proprietary Optimization Fortran Directive and C/C++ Pragma Summary The following table summarizes the supported Fortran directives and C/C++ pragmas. The following terms are useful in understanding the table.Chapter 6. Using Directives and Pragmas 65 • Functionality is a brief summary of the way to use the directive or pragma. For a complete description, refer to Chapter 17, “Directives and Pragmas Reference,” on page 263. • Many of the directives and pragmas can be preceded by NO. The default entry indicates the default for the directive or pragma. N/A appears if a default does not apply. • The scope entry indicates the allowed scope indicators for each directive or pragma, with L for loop, R for routine, and G for global. The default scope is surrounded by parentheses and N/A appears if the directive or pragma is not available in the given language. Note The “*” in the scope indicates this: For routine-scoped directive The scope includes the code following the directive or pragma until the end of the routine. For globally-scoped directive The scope includes the code following the directive or pragma until the end of the file rather than for the entire file. The name of a directive or pragma may also be prefixed with –M. For example, the directive –Mbounds is equivalent to bounds and –Mopt is equivalent to opt; and the pragma –Mnoassoc is equivalent to noassoc, and –Mvintr is equivalent to vintr. Table 6.1. Proprietary Optimization-Related Fortran Directive and C/C++ Pragma Summary Directive or pragma Functionality Default Fortran Scope C/C++ Scope altcode (noaltcode) Do/don’t generate alternate code for vectorized and parallelized loops. altcode (L)RG (L)RG assoc (noassoc) Do/don’t perform associative transformations. assoc (L)RG (L)RG bounds (nobounds) Do/don’t perform array bounds checking. nobounds (R)G* (R)G cncall (nocncall) Loops are considered for parallelization, even if they contain calls to user-defined subroutines or functions, or if their loop counts do not exceed usual thresholds. nocncall (L)RG (L)RG concur (noconcur) Do/don’t enable auto-concurrentization of loops. concur (L)RG (L)RG depchk (nodepchk) Do/don’t ignore potential data dependencies. depchk (L)RG (L)RG eqvchk (noeqvchk) Do/don’t check EQUIVALENCE for data dependencies. eqvchk (L)RG N/A fcon (nofcon) Do/don’t assume unsuffixed real constants are single precision. nofcon N/A (R)GPGI® User’s Guide 66 Directive or pragma Functionality Default Fortran Scope C/C++ Scope invarif (noinvarif) Do/don’t remove invariant if constructs from loops. invarif (L)RG (L)RG ivdep Ignore potential data dependencies. ivdep (L)RG N/A lstval (nolstval) Do/don’t compute last values. lstval (L)RG (L)RG opt Select optimization level. N/A (R)G (R)G safe (nosafe) Do/don’t treat pointer arguments as safe. safe N/A (R)G safe_lastval Parallelize when loop contains a scalar used outside of loop. not enabled (L) (L) safeptr (nosafeptr) Do/don’t ignore potential data dependencies to pointers. nosafeptr N/A L(R)G single (nosingle) Do/don’t convert float parameters to double. nosingle N/A (R)G* tp Generate PGI Unified Binary code optimized for specified targets. N/A (R)G (R)G unroll (nounroll) Do/don’t unroll loops. nounroll (L)RG (L)RG vector (novector) Do/don't perform vectorizations. vector (L)RG* (L)RG vintr (novintr) Do/don’t recognize vector intrinsics. vintr (L)RG (L)RG Scope of Fortran Directives and Command-Line options During compilation the effect of a directive may be to either turn an option on, or turn an option off. Directives apply to the section of code following the directive, corresponding to the specified scope, which may include the following loop, the following routine, or the rest of the program. This section presents several examples that show the effect of directives as well as their scope. Consider the following Fortran code: integer maxtime, time parameter (n = 1000, maxtime = 10) double precision a(n,n), b(n,n), c(n,n) do time = 1, maxtime do i = 1, n do j = 1, n c(i,j) = a(i,j) + b(i,j) enddo enddo enddo end When compiled with –Mvect, both interior loops are interchanged with the outer loop. $ pgf95 -Mvect dirvect1.f Directives alter this behavior either globally or on a routine or loop by loop basis. To assure that vectorization is not applied, use the novector directive with global scope.Chapter 6. Using Directives and Pragmas 67 cpgi$g novector integer maxtime, time parameter (n = 1000, maxtime = 10) double precision a(n,n), b(n,n), c(n,n) do time = 1, maxtime do i = 1, n do j = 1, n c(i,j) = a(i,j) + b(i,j) enddo enddo enddo end In this version, the compiler disables vectorization for the entire source file. Another use of the directive scoping mechanism turns an option on or off locally, either for a specific procedure or for a specific loop: integer maxtime, time parameter (n = 1000, maxtime = 10) double precision a(n,n), b(n,n), c(n,n) cpgi$l novector do time = 1, maxtime do i = 1, n do j = 1, n c(i,j) = a(i,j) + b(i,j) enddo enddo enddo end Loop level scoping does not apply to nested loops. That is, the directive only applies to the following loop. In this example, the directive turns off vector transformations for the top-level loop. If the outer loop were a timing loop, this would be a practical use for a loop-scoped directive. Scope of C/C++ Pragmas and Command-Line Options During compilation a pragma either turns an option on or turns an option off. Pragmas apply to the section of code corresponding to the specified scope - either the entire file, the following loop, or the following or current routine. This section presents several examples showing the effect of pragmas and the use of the pragma scope indicators. Note In all cases, pragmas override a corresponding command-line option. For pragmas that have only routine and global scope, there are two rules for determining the scope of a pragma. We cover these special scope rules at the end of this section. Consider the program: main() { float a[100][100], b[100][100], c[100][100]; int time, maxtime, n, i, j; maxtime=10; n=100; for (time=0; time[,[,...]] where is any valid variable, member, or array element reference.PGI® User’s Guide 70 Format Requirements NOTE The sentinel for prefetch directives is c$mem, which is distinct from the cpgi$ sentinel used for optimization directives. Any prefetch directives that use the cpgi$ sentinel will be ignored by the PGI compilers. • The "c" must be in column 1. • Either * or ! is allowed in place of c. • The scope indicators g, r and l used with the cpgi$ sentinel are not supported. • The directive name, including the directive prefix, may contain upper or lower case letters and is case insensitive (case is not significant). • Any variable names that appear in the body of the directive are case sensitive if the command line option –Mupcase is selected. Sample Usage Example 6.1. Prefetch Directive Use This example uses prefetch directives to prefetch data in a matrix multiplication inner loop where a row of one source matrix has been gathered into a contiguous vector. real*8 a(m,n), b(n,p), c(m,p), arow(n) ... do j = 1, p c$mem prefetch arow(1),b(1,j) c$mem prefetch arow(5),b(5,j) c$mem prefetch arow(9),b(9,j) do k = 1, n, 4 c$mem prefetch arow(k+12),b(k+12,j) c(i,j) = c(i,j) + arow(k) * b(k,j) c(i,j) = c(i,j) + arow(k+1) * b(k+1,j) c(i,j) = c(i,j) + arow(k+2) * b(k+2,j) c(i,j) = c(i,j) + arow(k+3) * b(k+3,j) enddo enddo This pattern of prefetch directives causes the compiler to emit prefetch instructions whereby elements of arow and b are fetched into the data cache starting four iterations prior to first use. By varying the prefetch distance in this way, it is sometimes possible to reduce the effects of main memory latency and improve performance. !DEC$ Directive PGI Fortran compilers for Microsoft Windows support several de-facto standard Fortran directives that help with interlanguage calling and importing and exporting routines to and from DLLs. These directives all take the form: !DEC$ directiveChapter 6. Using Directives and Pragmas 71 Format Requirements You must follow the following format requirements for the directive to be recognized in your program: • The directive must begin in line 1 when the file is fixed format or compiled with –Mfixed. • The directive prefix !DEC$ requires a space between the prefix and the directive keyword ATTRIBUTES. • The ! must begin the prefix when compiling Fortran 90 freeform format. • The characters C or * can be used in place of ! in either form of the prefix when compiling fixed-form (F77- style) format. • The directives are completely case insensitive. ALIAS Directive This directive specifies an alternative name with which to resolve a routine. The syntax for the ALIAS directive is either of the following: !DEC$ ALIAS routine_name , external_name !DEC$ ALIAS routine_name : external_name In this syntax, external_name is used as the external name for the specified routine_name. If external_name is an identifier name, the name (in uppercase) is used as the external name for the specified routine_name. If external_name is a character constant, it is used as-is; the string is not changed to uppercase, nor are blanks removed. You can also supply an alias for a routine using the ATTRIBUTES directive, described in the next section: !DEC$ ATTIRIBUTES ALIAS : 'alias_name' :: routine_name This directive specifies an alternative name with which to resolve a routine, as illustrated in the following code fragment that provides external names for three routines. In this fragment, the external name for sub1 is name1, for sub2 is name2, and for sub3 is name3. subroutine sub !DEC$ alias sub1 , 'name1' !DEC$ alias sub2 : 'name2' !DEC$ attributes alias : 'name3' :: sub3 ATTRIBUTES Directive !DEC$ ATTRIBUTES where is one of: ALIAS : 'alias_name' :: routine_name Specifies an alternative name with which to resolve routine_name. C :: routine_name Specifies that the routine routine_name will have its arguments passed by value. When a routine marked C is called, arguments, except arrays, are sent by value. For characters, only the first character is passed. The standard Fortran calling convention is pass by reference.PGI® User’s Guide 72 DLLEXPORT :: name Specifies that 'name' is being exported from a DLL. DLLIMPORT :: name Specifies that 'name' is being imported from a DLL. REFERENCE :: name Specifies that the argument 'name' is being passed by reference. Often this attribute is used in conjunction with STDCALL, where STDCALL refers to an entire routine; then individual arguments are modified with REFERENCE. STDCALL :: routine_name Specifies that routine 'routine_name' will have its arguments passed by value. When a routine marked STDCALL is called, arguments (except arrays and characters) will be sent by value. The standard Fortran calling convention is pass by reference. VALUE :: name Specifies that the argument 'name' is being passed by value. DISTRIBUTE Directive The syntax for the DISTRIBUTE directive is either of the following: !DEC$ DISTRIBUTE POINT !DEC$ DISTRIBUTEPOINT This directive is front-end based, and tells the compiler at what point within a loop to split into two loops. subroutine dist(a,b,n) integer i integer n integer a(*) integer b(*) do i = 1,n a(i) = a(i)+2 !DEC$ DISTRIBUTE POINT b(i) = b(i)*4 enddo end subroutine ALIAS Directive !DEC$ ALIAS is the same as !DEC$ ATTRIBUTES ALIAS C$PRAGMA C When programs are compiled using one of the PGI Fortran compilers on Linux, Win64, OSX, and SUA systems, an underscore is appended to Fortran global names, including names of functions, subroutines, and common blocks. This mechanism distinguishes Fortran name space from C/C++ name space. You can use C$PRAGMA C in the Fortran program to call a C/C++ function from Fortran. The statement would look similar to this:Chapter 6. Using Directives and Pragmas 73 C$PRAGMA C(name[,name]...) NOTE This statement directs the compiler to recognize the routine 'name' as a C function, thus preventing the Fortran compiler from appending an underscore to the routine name. On Win32 systems the C$PRAGMA C as well as the attributes C and STDCALL may effect other changes on argument passing as well as on the names of the routine. For more information on this topic, refer to “Win32 Calling Conventions,” on page 120.7475 Chapter 7. Creating and Using Libraries A library is a collection of functions or subprograms that are grouped for reference and ease of linking. This chapter discusses issues related to PGI-supplied compiler libraries. Specifically, it addresses the use of C/C++ builtin functions in place of the corresponding libc routines, creation of dynamically linked libraries, known as shared objects or shared libraries, and math libraries. Note This chapter does not duplicate material related to using libraries for inlining, described in “Creating an Inline Library,” on page 47 or information related to run-time library routines available to OpenMP programmers, described in “Run-time Library Routines,” on page 55. This chapter has examples that include the following options related to creating and using libraries. –Bdynamic –fpic –Mmakeimplib –Bstatic –implib –o –c –l –shared –def –Mmakedll Using builtin Math Functions in C/C++ The name of the math header file is math.h. Include the math header file in all of your source files that use a math library routine as in the following example, which calculates the inverse cosine of pi/3. #include #define PI 3.1415926535 void main() { double x, y; x = PI/3.0; y = acos(x); }PGI® User’s Guide 76 Including math.h will cause PGCC C and C++ to use builtin functions, which are much more efficient than library calls. In particular, the following intrinsics calls will be processed using builtins if you include math.h: abs atan atan2 cos exp fabs fmax fmaxf fmin fminf log log10 pow sin sqrt tan Creating and Using Shared Object Files on Linux All of the PGI Fortran, C, and C++ compilers support creation of shared object files. Unlike statically linked object and library files, shared object files link and resolve references with an executable at runtime via a dynamic linker supplied with your operating system. The PGI compilers must generate position independent code to support creation of shared objects by the linker. However, this is not the default. You must create object files with position independent code and shared object files that will include them. The following steps describe how to create and use a shared object file. 1. Create an object file with position independent code. To do this, compile your code with the appropriate PGI compiler using the –fpic option, or one of the equivalent options, such as –fPIC, –Kpic, and –KPIC, which are supported for compatibility with other systems. For example, use the following command to create an object file with position independent code using pgf95: % pgf95 -c -fpic tobeshared.f 2. Produce a shared object file. To do this, use the appropriate PGI compiler to invoke the linker supplied with your system. It is customary to name such files using a .so filename extension. On Linux, you do this by passing the –shared option to the linker: % pgf95 -shared -o tobeshared.so tobeshared.o Note Compilation and generation of the shared object can be performed in one step using both the –fpic option and the appropriate option for generation of a shared object file. 3. Use a shared object file. To do this, us the appropriate PGI compiler to compile and link the program which will reference functions or subroutines in the shared object file, and list the shared object on the link line, as shown here: % pgf95 -o myprog myprog.f tobeshared.so 4. Make the executable available. You now have an executable myprog which does not include any code from functions or subroutines in tobeshared.so, but which can be executed and dynamically linked to that code.Chapter 7. Creating and Using Libraries 77 By default, when the program is linked to produce myprog, no assumptions are made on the location of tobeshared.so. Therefore, for myprog to execute correctly, you must initialize the environment variable LD_LIBRARY_PATH to include the directory containing tobeshared.so. If LD_LIBRARY_PATH is already initialized, it is important not to overwrite its contents. Assuming you have placed tobeshared.so in a directory /home/myusername/bin, you can initialize LD_LIBRARY_PATH to include that directory and preserve its existing contents, as shown in the following: % setenv LD_LIBRARY_PATH "$LD_LIBRARY_PATH":/home/myusername/bin If you know that tobeshared.so will always reside in a specific directory, you can create the executable myprog in a form that assumes this using the –R link-time option. For example, you can link as follows: % pgf95 -o myprog myprof.f tobeshared.so -R/home/myusername/bin Note As with the –L option, there is no space between –R and the directory name. If the –R option is used, it is not necessary to initialize LD_LIBRARY_PATH. In the previous example, the dynamic linker will always look in /home/myusername/bin to resolve references to tobeshared.so. By default, if the LD_LIBRARY_PATH environment variable is not set, the linker will only search /usr/lib and /lib for shared objects. The command ldd is a useful tool when working with shared object files and executables that reference them. When applied to an executable, as shown in the following example, ldd lists all shared object files referenced in the executable along with the pathname of the directory from which they will be extracted. % ldd myprog If the pathname is not hard-coded using the–R option, and if LD_LIBRARY_PATH is not initialized, the pathname is listed as “not found”. For more information on ldd, its options and usage, see the online man page for ldd. Creating and Using Shared Object Files in SFU and 32-bit SUA Note The information included in this section is valid for 32-bit only. The 32-bit version of PGI Workstation for SFU and SUA uses the GNU ld for its linker, unlike previous versions that used the Windows LINK.EXE. With this change, the PGI compilers and tools for SFU and 32-bit SUA are now able to generate shared object (.so) files. You use the –shared switch to generate a shared object file. The following example creates a shared object file, hello.so, and then creates a program called hello that uses it. 1. Create a shared object file. To produce a shared object file, use the appropriate PGI compiler to invoke the linker supplied with your system. It is customary to name such files using a .so filename extension. In the following example, we use hello.so:PGI® User’s Guide 78 % pgcc -shared hello.c -o hello.so 2. Create a program that uses the shared object, in this example, hello.so: % pgcc hi.c hello.so -o hello Shared Object Error Message When running a program that uses a shared object, you may encounter an error message similar to the following: hello: error in loading shared libraries hello.so: cannot open shared object file: No such file or directory This error message either means that the shared object file does not exist or that the location of this file is not specified in your LD_LIBRARY_PATH variable. To specify the location of the .so, add the shared object’s directory to your LD_LIBRARY_PATH variable. For example, the following command adds the current directory to your LD_LIBRARY_PATH variable using C shell syntax: % setenv LD_LIBRARY_PATH "$LD_LIBRARY_PATH":"./" Shared Object-Related Compiler Switches The following switches support shared object files in SFU and SUA. For more detailed information on these switches, refer to Chapter 15, “Command-Line Options Reference,” on page 163. –shared Used to produce shared libraries –Bdynamic Passed to linker; specify dynamic binding Note On Windows, -Bstatic and -Bdynamic must be used for both compiling and linking. –Bstatic Passed to linker; specify static binding –Bstatic_pgi Use to link static PGI libraries with dynamic system libraries; implies –Mnorpath. –L Passed to linker; add directory to library search path. –Mnorpath Don't add –rpath paths to link line. –Mnostartup Do not use standard linker startup file. –Mnostdlib Do not use standard linker libraries. –R Passed to linker; just link symbols from object, or add directory to run time search path.Chapter 7. Creating and Using Libraries 79 PGI Runtime Libraries on Windows The PGI runtime libraries on Windows are available in both static and dynamicallyy-linked (DLL) versions. The static libraries are used by default. • You can use the dynamically-linked version of the routine by specifying –Bdynamic at both compile and link time. • You can explicitly specify static linking, the default, by using -Bstatic at compile and link time. For details on why you might choose one type of linking over another type, refer to “Creating and Using Dynamic-Link Libraries on Windows,” on page 80. Creating and Using Static Libraries on Windows The Microsoft Library Manager (LIB.EXE) is the tool that is typically used to create and manage a static library of object files on Windows. LIB is provided with the PGI compilers as part of the Microsoft Open Tools. Refer to www.msdn2.com for a complete LIB reference - searching for LIB.EXE. For a list of available options, invoke LIB with the /? switch. For compatibility with legacy makefiles, PGI provides wrappers for LIB and LINK called ar. This version of ar is compatible with Womdpws amd pbject-file formats. PGi also provides ranlib as a placeholder for legacy makefile support. ar command The ar command is a legacy archive wrapper that interprets legacy ar command line options and translates these to LINK/LIB options. You can use it to create libraries of object files. Syntax: The syntax for the ar command is this: ar [options] [archive] [object file]. Where: • The first argument must be a command line switch, and the leading dash on the first option is optional. • The single character options, such as –d and –v, may be combined into a single option, as –dv. Thus, ar dv, ar -dv, and ar -d -v all mean the same thing. • The first non-switch argument must be the library name. • One (and only one) of –d, –r, –t, or –x must appear on the command line. Options The options available for the ar command are these:PGI® User’s Guide 80 –c This switch is for compatibility; it is ignored. –d The named object files are deleted from the library. –r The named object files are replaced in or added to the library. ranlib command The ranlib command is a wrapper that allows use of legacy scripts and makefiles that use the ranlib command. The command actually does nothing; it merely exists for compatibility. Syntax: The syntax for the ranlib command is this: DOS> ranlib [options] [archive] Options The options available for the ranlib command are these: –help Short help information is printed out. –V Version information is printed out. Creating and Using Dynamic-Link Libraries on Windows There are several differences between static and dynamic-link libraries on Windows. Libraries of either type are used when resolving external references for linking an executable, but the process differs for each type of library. When linking with a static library, the code needed from the library is incorporated into the executable. When linking with a DLL, external references are resolved using the DLL's import library, not the DLL itself. The code in the DLL associated with the external references does not become a part of the executable. The DLL is loaded when the executable that needs it is run. For the DLL to be loaded in this manner, the DLL must be in your path. Static libraries and DLLs also handle global data differently. Global data in static libraries is automatically accessible to other objects linked into an executable. Global data in a DLL can only be accessed from outside the DLL if the DLL exports the data and the image that uses the data imports it. To this end the C compilers support the Microsoft storage class extensions __declspec(dllimport) and __declspec(dllexport). These extensions may appear as storage class modifiers and enable functions and data to be imported and exported: extern int __declspec(dllimport) intfunc(); float __declspec(dllexport) fdata;Chapter 7. Creating and Using Libraries 81 The PGI Fortran compilers support the DEC$ATTRIBUTES extensions DLLIMPORT and DLLEXPORT: cDEC$ ATTRIBUTES DLLEXPORT :: object [,object] ... cDEC$ ATTRIBUTES DLLIMPORT :: object [,object] ... Here c is one of C, c, !, or *. object is the name of the subprogram or common block that is exported or imported. Note that common block names are enclosed within slashes (/). In example: cDEC$ ATTRIBUTES DLLIMPORT :: intfunc !DEC$ ATTRIBUTES DLLEXPORT :: /fdata/ For more information on these extensions, refer to “!DEC$ Directive,” on page 70. The Examples in this section further illustrate the use of these extensions. To create a DLL from the command line, use the –Mmakedll option. The following switches apply to making and using DLLs with the PGI compilers: –Bdynamic Compile for and link to the DLL version of the PGI runtime libraries. This flag is required when linking with any DLL built by the PGI compilers. This flag corresponds to the /MD flag used by Microsoft’s cl compilers. –Bstatic Compile for and link to the static version of the PGI runtime libraries. This flag corresponds to the /MT flag used by Microsoft’s cl compilers. –Mmakedll Generate a dynamic-link library or DLL. Implies –Bdynamic. –Mmakeimplib Generate an import library without generating a DLL. Use this flag when you want to generate an import library for a DLL but are not yet ready to build the DLL itself. This situation might arise, for example, when building DLLs with mutual imports, as shown in Example 7.4, “Build DLLs Containing Circular Mutual Imports: Fortran,” on page 86. –o Passed to the linker. Name the DLL or import library . –def When used with –Mmakedll, this flag is passed to the linker and a .def file named is generated for the DLL. The .def file contains the symbols exported by the DLL. Generating a .def file is not required when building a DLL but can be a useful debugging tool if the DLL does not contain the symbols that you expect it to contain. When used with –Mmakeimplib, this flag is passed to lib which requires a .def file to create an import library. The .def file can be empty if the list of symbols to export are passed to lib on the command line or explicitly marked as dllexport in the source code. –implib Passed to the linker. Generate an import library named for the DLL. A DLL’s import library is the interface used when linking an executable that depends on routines in a DLL.PGI® User’s Guide 82 To use the PGI compilers to create an executable that links to the DLL form of the runtime, use the compiler flag –Bdynamic. The executable built will be smaller than one built without –Bdynamic; the PGI runtime DLLs, however, must be available on the system where the executable is run. The –Bdynamic flag must be used when an executable is linked against a DLL built by the PGI compilers. The following examples outline how to use –Bdynamic, –Mmakedll and –Mmakeimplib to build and use DLLs with the PGI compilers. Example 7.1. Build a DLL: Fortran In this example we build a DLL out of a single source file, object1.f, which exports data and a subroutine using DLLEXPORT. The main source file, prog1.f, uses DLLIMPORT to import the data and subroutine from the DLL. object1.f subroutine sub1(i) !DEC$ ATTRIBUTES DLLEXPORT :: sub1 integer i common /acommon/ adata integer adata !DEC$ ATTRIBUTES DLLEXPORT :: /acommon/ print *, "sub1 adata", adata print *, "sub1 i ", i adata = i end prog1.f program prog1 common /acommon/ adata integer adata external sub1 !DEC$ ATTRIBUTES DLLIMPORT:: sub1, /acommon/ adata = 11 call sub1(12) print *, "main adata", adata end Step 1: Create the DLL obj1.dll and its import library obj1.lib using the following series of commands: % pgf95 -Bdynamic -c object1.f % pgf95 -Mmakedll object1.obj -o obj1.dll Step 2: Compile the main program: % pgf95 -Bdynamic -o prog1 prog1.f -defaultlib:obj1 The –Mdll switch causes the compiler to link against the PGI runtime DLLs instead of the PGI runtime static libraries. The –Mdll switch is required when linking against any PGI-compiled DLL, such as obj1.dll. The #defaultlib: switch specifies that obj1.lib, the DLL’s import library, should be used to resolve imports. Step 3: Ensure that obj1.dll is in your path, then run the executable prog1 to determine if the DLL was successfully created and linked:Chapter 7. Creating and Using Libraries 83 % prog1 sub1 adata 11 sub1 i 12 main adata 12 Should you wish to change obj1.dll without changing the subroutine or function interfaces, no rebuilding of prog1 is necessary. Just recreate obj1.dll and the new obj1.dll is loaded at runtime. Example 7.2. Build a DLL: C In this example, we build a DLL out of a single source file, object2.c, which exports data and a subroutine using __declspec(dllexport). The main source file, prog2.c, uses __declspec(dllimport) to import the data and subroutine from the DLL. object2.c int __declspec(dllexport) data; void __declspec(dllexport) func2(int i) { printf("func2: data == %d\n", data); printf("func2: i == %d\n", i); data = i; } prog2.c int __declspec(dllimport) data; void __declspec(dllimport) func2(int); int main() { data = 11; func2(12); printf("main: data == %d\n",data); return 0; } Step 1: Create the DLL obj2.dll and its import library obj2.lib using the following series of commands: % pgcc -Bdynamic -c object2.c % pgcc -Mmakedll object2.obj -o obj2.dll Step 2: Compile the main program: % pgcc -Bdynamic -o prog2 prog2.c -defaultlib:obj2 The –Bdynamic switch causes the compiler to link against the PGI runtime DLLs instead of the PGI runtime static libraries. The –Bdynamic switch is required when linking against any PGI-compiled DLL such as obj2.dll. The #defaultlib: switch specifies that obj2.lib, the DLL’s import library, should be used to resolve the imported data and subroutine in prog2.c. Step 3: Ensure that obj2.dll is in your path, then run the executable prog2 to determine if the DLL was successfully created and linked:PGI® User’s Guide 84 % prog2 func2: data == 11 func2: i == 12 main: data == 12 Should you wish to change obj2.dll without changing the subroutine or function interfaces, no rebuilding of prog2 is necessary. Just recreate obj2.dll and the new obj2.dll is loaded at runtime. Example 7.3. Build DLLs Containing Circular Mutual Imports: C In this example we build two DLLs, obj3.dll and obj4.dll, each of which imports a routine that is exported by the other. To link the first DLL, the import library for the second DLL must be available. Usually an import library is created when a DLL is linked. In this case, however, the second DLL cannot be linked without the import library for the first DLL. When such circular imports exist, an import library for one of the DLLs must be created in a separate step without creating the DLL. The PGI drivers call the Microsoft lib tool to create import libraries in this situation. Once the DLLs are built, we can use them to build the main program. /* object3.c */ void __declspec(dllimport) func_4b(void); void __declspec(dllexport) func_3a(void) { printf("func_3a, calling a routine in obj4.dll\n"); func_4b(); } void __declspec(dllexport) func_3b(void) { printf("func_3b\n"); } /* object4.c */ void __declspec(dllimport) func_3b(void); void __declspec(dllexport) func_4a(void) { printf("func_4a, calling a routine in obj3.dll\n"); func_3b(); } void __declspec(dllexport) func_4b(void) { printf("func_4b\n"); } /* prog3.c */ void __declspec(dllimport) func_3a(void); void __declspec(dllimport) func_4a(void); int main() { func_3a(); func_4a(); return 0; }Chapter 7. Creating and Using Libraries 85 Step 1: Use –Mmakeimplib with the PGI compilers to build an import library for the first DLL without building the DLL itself. % pgcc -Bdynamic -c object3.c % pgcc -Mmakeimplib -o obj3.lib object3.obj The –def= option can also be used with –Mmakeimplib. Use a .def file when you need to export additional symbols from the DLL. A .def file is not needed in this example because all symbols are exported using __declspec(dllexport). Step 2: Use the import library, obj3.lib, created in Step 1, to link the second DLL. % pgcc -Bdynamic -c object4.c % pgcc -Mmakedll -o obj4.dll object4.obj -defaultlib:obj3 Step 3: Use the import library, obj4.lib, created in Step 2, to link the first DLL. % pgcc -Mmakedll -o obj3.dll object3.obj -defaultlib:obj4 Step 4: Compile the main program and link against the import libraries for the two DLLs. % pgcc -Bdynamic prog3.c -o prog3 -defaultlib:obj3 -defaultlib:obj4 Step 5: Execute prog3.exe to ensure that the DLLs were create properly. % prog3 func_3a, calling a routine in obj4.dll func_4b func_4a, calling a routine in obj3.dll func_3bPGI® User’s Guide 86 Example 7.4. Build DLLs Containing Circular Mutual Imports: Fortran In this example we build two DLLs when each DLL is dependent on the other, and use them to build the main program. In the following source files, object2.f95 makes calls to routines defined in object3.f95, and vice versa. This situation of mutual imports requires two steps to build each DLL. In this example we build two DLLs, obj2.dll and obj3.dll, each of which imports a routine that is exported by the other. To link the first DLL, the import library for the second DLL must be available. Usually an import library is created when a DLL is linked. In this case, however, the second DLL cannot be linked without the import library for the first DLL. When such circular imports exist, an import library for one of the DLLs must be created in a separate step without creating the DLL. The PGI drivers call the Microsoft lib tool to create import libraries in this situation. Once the DLLs are built, we can use them to build the main program. object2.f95 subroutine func_2a external func_3b !DEC$ ATTRIBUTES DLLEXPORT :: func_2a !DEC$ ATTRIBUTES DLLIMPORT :: func_3b print*,"func_2a, calling a routine in obj3.dll" call func_3b() end subroutine subroutine func_2b !DEC$ ATTRIBUTES DLLEXPORT :: func_2b print*,"func_2b" end subroutine object3.f95 subroutine func_3a external func_2b !DEC$ ATTRIBUTES DLLEXPORT :: func_3a !DEC$ ATTRIBUTES DLLIMPORT :: func_2b print*,"func_3a, calling a routine in obj2.dll" call func_2b() end subroutine subroutine func_3b !DEC$ ATTRIBUTES DLLEXPORT :: func_3b print*,"func_3b" end subroutine prog2.f95 program prog2 external func_2a external func_3a !DEC$ ATTRIBUTES DLLIMPORT :: func_2a !DEC$ ATTRIBUTES DLLIMPORT :: func_3a call func_2a() call func_3a() end program Step 1: Use –Mmakeimplib with the PGI compilers to build an import library for the first DLL without building the DLL itself. % pgf95 -Bdynamic -c object2.f95 % pgf95 -Mmakeimplib -o obj2.lib object2.objChapter 7. Creating and Using Libraries 87 Tip The -def= option can also be used with -Mmakeimplib. Use a .def file when you need to export additional symbols from the DLL. A .def file is not needed in this example because all symbols are exported using DLLEXPORT. Step 2: Use the import library, obj2.lib, created in Step 1, to link the second DLL. % pgf95 -Bdynamic -c object3.f95 % pgf95 -Mmakedll -o obj3.dll object3.obj -defaultlib:obj2 Step 3: Use the import library, obj3.lib, created in Step 2, to link the first DLL. % pgf95 -Mmakedll -o obj2.dll object2.obj -defaultlib:obj3 Step 4: Compile the main program and link against the import libraries for the two DLLs. % pgf95 -Bdynamic prog2.f95 -o prog2 -defaultlib:obj2 -defaultlib:obj3 Step 5: Execute prog2 to ensure that the DLLs were created properly: % prog2 func_2a, calling a routine in obj3.dll func_3b func_3a, calling a routine in obj2.dll func_2b Example 7.5. Import a Fortran module from a DLL In this example we import a Fortran module from a DLL. We use the source file my_module_def.f90 to create a DLL containing a Fortran module. We then use the source file my_module_use.f90 to build a program that imports and uses the Fortran module from my_module_def.f90. defmod.f90 module testm type a_type integer :: an_int end type a_type type(a_type) :: a, b !DEC$ ATTRIBUTES DLLEXPORT :: a,b contains subroutine print_a !DEC$ ATTRIBUTES DLLEXPORT :: print_a write(*,*) a%an_int end subroutine subroutine print_b !DEC$ ATTRIBUTES DLLEXPORT :: print_b write(*,*) b%an_int end subroutine end module usemod.f90 use testm a%an_int = 1 b%an_int = 2 call print_a call print_b endPGI® User’s Guide 88 Step 1: Create the DLL. % pgf90 -Mmakedll -o defmod.dll defmod.f90 Creating library defmod.lib and object defmod.exp Step 2: Create the exe and link against the import library for the imported DLL. % pgf90 -Bdynamic -o usemod usemod.f90 -defaultlib:defmod.lib Step 3: Run the exe to ensure that the module was imported from the DLL properly. % usemod 1 2 Using LIB3F The PGI Fortran compilers include complete support for the de facto standard LIB3F library routines on both Linux and Windows operating systems. See the PGI Fortran Reference manual for a complete list of available routines in the PGI implementation of LIB3F. LAPACK, BLAS and FFTs Pre-compiled versions of the public domain LAPACK and BLAS libraries are included with the PGI compilers. The LAPACK library is called liblapack.a or on Windows, liblapack.lib. The BLAS library is called libblas.a or on Windows, libblas.lib. These libraries are installed to $PGI//lib, where is replaced with the appropriate target name (linux86, linux86-64, osx86, osx86-64, win32, win64, sfu32, sua32, or sua64). To use these libraries, simply link them in using the –l option when linking your main program: % pgf95 myprog.f -llapack -lblas Highly optimized assembly-coded versions of BLAS and certain FFT routines may be available for your platform. In some cases, these are shipped with the PGI compilers. See the current release notes for the PGI compilers you are using to determine if these optimized libraries exist, where they can be downloaded (if necessary), and how to incorporate them into your installation as the default. The C++ Standard Template Library The PGC++ compiler includes a bundled copy of the STLPort Standard C++ Library. See the online Standard C++ Library tutorial and reference manual at www.stlport.com for further details and licensing.89 Chapter 8. Using Environment Variables Environment variables allow you to set and pass information that can alter the default behavior of the PGI compilers and the executables which they generate. This chapter includes explanations of the environment variables specific to PGI compilers. Other environment variables are referenced and documented in other sections of this User’s Guide or the PGI Tools Guide. • You use OpenMP environment variables to control the behavior of OpenMP programs. For consistency related to the OpenMP environment, the details of the OpenMP-related environment variables are included in Chapter 5, “Using OpenMP”. • You can use environment variables to control the behavior of the PGDBG debugger or PGPROF profiler. For a description of environment variables that affect these tools, refer to the PGI Tools Guide. Setting Environment Variables Before we look at the environment variables that you might use with the PGI compilers and tools, let’s take a look at how to set environment variables. To illustrate how to set these variables in various environments, lets look at how a user might initialize the shell environment prior to using the PGI compilers and tools. Setting Environment Variables on Linux Let’s assume that you want access to the PGI products when you log on. Let’s further assume that you installed the PGI compilers in /opt/pgi and that the license file is in /opt/pgi/license.dat. For access at startup, you can add the following lines to your startup file. In csh, use these commands: % setenv PGI /opt/pgi % setenv MANPATH "$MANPATH":$PGI/linux86/7.1/man % setenv LM_LICENSE_FILE $PGI/license.dat % set path = ($PGI/linux86/7.1/bin $path) In bash, sh or ksh, use these commands: % PGI=/opt/pgi; export PGI PGI® User’s Guide 90 % MANPATH=$MANPATH:$PGI/linux86/7.1/man; export MANPATH % LM_LICENSE_FILE=$PGI/license.dat; export LM_LICENSE_FILE % PATH=$PGI/linux86/7.1/bin:$PATH; export PATH Setting Environment Variables on Windows In Windows, when you access PGI Workstation 7.1 (Start | PGI Workstation 7.1), you have two options that PGI provides for setting your environment variables - either the DOS command environment or the Cygwin Bash environment. When you open either of these shells available to you, the default environment variables are already set and available to you. You may want to use other environment variables, such as the OpenMP ones. This section explains how to do that. Suppose that your home directory is C:tmp. The following examples show how you might set the temporary directory to your home directory, and then verify that it is set. Command prompt: From PGI Workstation 7.1, select PGI Workstation Tools | PGI Command Prompt (32-bit or 64-bit), and enter the following: DOS> set TMPDIR=C:tmp DOS> echo %TMPDIR% C:\tmp DOS> Cygwin Bash prompt: From PGI Workstation 7.1, select PGI Workstation (32-bit or 64-bit) and at the Cygwin Bash prompt, enter the following PGI$ export TMPDIR=C:\\tmp PGI$ echo $TMPDIR C:\tmp PGI$ Setting Environment Variables on Mac OSX Let’s assume that you want access to the PGi products when you log on. Let’s further assume that you installed the PGI compilers in /opt/pgi and that the license file is in /opt/pgi/license.dat. For access at startup, you can add the following lines to your startup file. For x64 osx86-64 in a csh: % set path = (/opt/pgi/osx86-64/7.0/bin $path) % setenv MANPATH "$MANPATH":/opt/pgi/osx86-64/7.0/man For x64 osx86-64 in a bash, zsh, or ksh: % PATH=/opt/pgi/osx86-64/7.0/bin:$PATH; export PATH % MANPATH=$MANPATH:/opt/pgi/osx86-64/7.0/man; export MANPATH For x64 osx86 in a csh: % set path = (/opt/pgi/osx86/7.0/bin $path)Chapter 8. Using Environment Variables 91 % setenv MANPATH "$MANPATH":/opt/pgi/osx86/7.0/man For x64 osx86 in a bash, zsh, or ksh: % PATH=/opt/pgi/osx86/7.0/bin:$PATH % export PATH % MANPATH=$MANPATH:/opt/pgi/osx86/7.0/man % export MANPATH PGI-Related Environment Variables For easy reference, the following summary table provides a quick listing of the OpenMP and PGI compilerrelated environment variables. Later in this chapter are more detailed descriptions of the environment variables specific to PGI compilers and the executables they generate. Table 8.1. PGI-related Environment Variable Summary Table Environment Variable Description FLEXLM_BATCH (Windows only) When set to 1, prevents interactive pop-ups from appearing by sending all licensing errors and warnings to standard out rather than to a pop-up window. FORTRAN_OPT Allows the user to specify that the PGI Fortran compilers user VAX I/ O conventions. GMON_OUT_PREFIX Specifies the name of the output file for programs tha are compiler and linked with the –pg option. LD_LIBRARY_PATH Specifies a colon-separated set of directories where libraries should first be searched, prior to searching the standard set of directories. LM_LICENSE_FILE Specifies the full path of the license file that is required for running the PGI software. On Windows, LM_LICENSE _FILE does not need to be set. MANPATH Sets the directories that are seacrhed for manual pages associated with the command that the user types. MPSTKZ Increases the size of the stacks used by threads executing in parallel regions. The value should be an integer concatenated with M or m to specify stack sizes of n megabytes. MP_BIND Specifies whether to bind processes or threads executing in a parallel region to a physical processor. MP_BLIST When MP_BIND is yes, this variable specifically defines the threadCPU relationship, overriding the default values. MP_SPIN Specifies the number of times to check a semaphore before calling sched_yield() (on Linux) or _sleep() (on Windows). MP_WARN Allows you to eliminate certain default warning messages. NCPUS Sets the number of processes or threads used in parallel regions. NCPUS_MAX Limits the maximum number of processors or threads that can be used in a parallel region.PGI® User’s Guide 92 Environment Variable Description NO_STOP_MESSAGE If used, the execution of a plain STOP statement does not produce the message FORTRAN STOP. OMP_DYNAMIC Currently has no effect. Enables (TRUE) or disables (FALSE) the dynamic adjustment of the number of threads. The default is FALSE. OMP_NESTED Currently has no effect. Enables (TRUE) or disables (FALSE) nested parallelism. The default is FALSE. OMP_NUM_THREADS Specifies the number of threads to use during execution of parallel regions. Default is 1. OMP_SCHEDULE Specifies the type of iteration scheduling and, optionally, the chunk size to use for omp for and omp parallel for loops that include the run-time schedule clause. The default is STATIC with chunk size = 1. OMP_STACK_SIZE Overrides the default stack size for a newly created thread. OMP_WAIT_POLICY Sets the behavior of idle threads, defining whether they spin or sleep when idle. The values are ACTIVE and PASSIVE. The default is ACTIVE. PATH Determines which locations are searched for commands the user may type. PGI Specifies, at compile-time, the root directory where the PGI compilers and tools are installed. PGI_CONTINUE If set, when a program compiled with–Mchkfpstk is executed, the stack is automatically cleaned up and execution then continues. PGI_OBJSUFFIX Allows you to control the suffix on generated object files. PGI_STACK_USAGE (Windows only) Allows you to explicitly set stack properties for your program. PGI_TERM Controls the stack traceback and just-in-time debugging functionality. PGI_TERM_DEBUG Overrides the default behavior when PGI_TERM is set to debug. PWD Allows you to display the current directory. STATIC_RANDOM_SEED Forces the seed returned by RANDOM_SEED to be constant. TMP Sets the directory to use for temporary files created during execution of the PGI compilers and tools; interchangeable with TMPDIR. TMPDIR Sets the directory to use for temporary files created during execution of the PGI compilers and tools. PGI Environment Variables You use the environment variables listed in Table 8.1, “PGI-related Environment Variable Summary Table” to alter the default behavior of the PGI compilers and the executables which they generate. This section provides more detailed descriptions about the variables in this table that are not OpenMP environment variables.Chapter 8. Using Environment Variables 93 FLEXLM_BATCH By default, on Windows the license server creates interactive pop-up messages to issue warning and errors. You can use the environment variable FLEXLM_BATCH to prevent interactive pop-up windows. To do this, set the environment variable FLEXLM_BATCH to 1. The following csh example prevents interactive pop-up messages for licensing warnings and errors: % set FLEXLM_BATCH = 1; FORTRAN_OPT FORTRAN_OPT allows the user to specify that the PGI Fortran compilers user VAX I/O conventions. • If FORTRAN_OPT exists and contains the value vaxio, the record length in the open statement is in units of 4-byte words, and the $ edit descriptor only has an effect for lines beginning with a space or a plus sign (+). • If this variable exists and contains the value format_relaxed, an I/O item corresponding to a numerical edit descriptor (such as F, E, I, and so on) is not required to be a type implied by the descriptor. The following example causes the PGI Fortran compilers to use VAX I/O conventions: $ setenv FORTRAN_OPT vaxio GMON_OUT_PREFIX GMON_OUT_PREFIX specifies the name of the output file for programs that are compiled and linked with the -pg option. The default name is gmon.out.a. If GMON_OUT_PREFIX is set, the name of the output file has GMON_OUT_PREFIX as a prefix. Further, the suffix is the pid of the running process. The prefix and suffix are separated by a dot. For example, if the output file is mygmon, then the full filename may look something similar to this: GMON_OUT_PREFIX.mygmon.0012348567. The following example causes the PGI Fortran compilers to use pgout as the output file for programs compiled and linked with the -pg option. $ setenv GMON_OUT_PREFIX pgout LD_LIBRARY_PATH The LD_LIBRARY_PATH variable is a colon-separated set of directories specifying where libraries should first be searched, prior to searching the standard set of directories. This variable is useful when debugging a new library or using a nonstandard library for special purposes. The following csh example adds the current directory to your LD_LIBRARY_PATH variable. % setenv LD_LIBRARY_PATH "$LD_LIBRARY_PATH":"./" LM_LICENSE_FILE The LM_LICENSE_FILE variable specifies the full path of the license file that is required for running the PGI software.PGI® User’s Guide 94 For example, once the license file is in place, you can execute the following csh commands to make the products you have purchased accessible and to initialize your environment for use of FLEXlm. These commands assume that you use the default installation directory: /opt/pgi % setenv PGI /opt/pgi % setenv LM_LICENSE_FILE "$LM_LICENSE_FILE":/opt/pgi/license.dat To set the environment variable LM_LICENSE_FILE to the full path of the license key file, do this: 1. Open the System Properties dialog: Start | Control Panel | System. 2. Select the Advanced tab. 3. Click the Environment Variables button. • If LM_LICENSE_FILE is not already an environment variable, create a new system variable for it. Set its value to the full path, including the name of the file, for the license key file, license.dat. • If LM_LICENSE_FILE already exists as an environment variable, append the path to the license file to the variable’s current value using a semi-colon to separate entrie • If LM_LICENSE_FILE is not already an environment variable, create a new system variable for it. Set its value to the full path, including the name of the file, for the license key file, license.dat. • If LM_LICENSE_FILE already exists as an environment variable, append the path to the license file to the variable’s current value using a semi-colon to separate entrie MANPATH The MANPATH variable sets the directories that are searched for manual pages associated with the commands that the user types. When using PGI products, it is important that you set your PATH to include the location of the PGI products and then set the MANPATH variable to include the man pages associated with the products. The following csh example targets x64 linux86-64 version of the compilers and tool s and allows the user access to the manual pages associated with them. % set path = (/opt/pgi/linux86-64/7.1/bin $path % setenv MANPATH "$MANPATH":/opt/pgi/linux86-64/7.1/man MPSTKZ MPSTKZ increases the size of the stacks used by threads executing in parallel regions. You typically use this variable with programs that utilize large amounts of thread-local storage in the form of private variables or local variables in functions or subroutines called within parallel regions. The value should be an integer concatenated with M or m to specify stack sizes of n megabytes. For example, the following setting specifies a stack size of 8 megabytes. $ setenv MPSTKZ 8M MP_BIND You can set MP_BIND to yes or y to bind processes or threads executing in a parallel region to physical processor. Set it to no or n to disable such binding. The default is to not bind processes to processors. ThisChapter 8. Using Environment Variables 95 variable is an execution-time environment variable interpreted by the PGI runtime-support libraries. It does not affect the behavior of the PGI compilers in any way. Note The MP_BIND environment variable is not supported on all platforms. $ setenv MP_BIND y MP_BLIST MP_BLIST allows you to specifically define the thread-CPU relationship. Note This variable is only in effect when MP_BIND is yes . While the MP_BIND variable binds processors or threads to a physical processor, MP_BLIST allows you to specifically define which thread is associated with which processor. The list defines the processor-thread relationship order, beginning with thread 0. This list overrides the default binding. For example, the following setting for MP_BLIST maps CPUs 3, 2, 1 and 0 to threads 0, 1, 2 and 3 respectively. $ setenv MP_BLIST=3,2,1,0 MP_SPIN When a thread executing in a parallel region enters a barrier, it spins on a semaphore. You can use MP_SPIN to specify the number of times it checks the semaphore before calling sched_yield() (on Linux) or _sleep() (on Windows). These calls cause the thread to be re-scheduled, allowing other processes to run. The default values are 100 (on Linux) and 10000 (on Windows). $ setenv MP_SPIN 200 MP_WARN MP_WARN allows you to eliminate certain default warning messages. By default, a warning is printed to stderr if you execute an OpenMP or auto-parallelized program with NCPUS or OMP_NUM_THREADS set to a value larger than the number of physical processors in the system. For example, if you produce a parallelized executable a.out and execute as follows on a system with only one processor: % setenv OMP_NUM_THREADS 2 % a.out Warning: OMP_NUM_THREADS or NCPUS (2) greater than available cpus (1) FORTRAN STOP Setting MP_WARN to NO eliminates these warning messages.PGI® User’s Guide 96 NCPUS You can use the NCPUS environment variable to set the number of processes or threads used in parallel regions. The default is to use only one process or thread, which is known as serial mode. Note OMP_NUM_THREADS has the same functionality as NCPUS. For historical reasons, PGi supports the environment variable NCPUS. If both OMP_NUM_THREADS and NCPUS are set, the value of OMP_NUM_THREADS takes precedence. Warning Setting NCPUS to a value larger than the number of physical processors or cores in your system can cause parallel programs to run very slowly. NCPUS_MAX You can use the NCPUS_MAX environment variable to limit the maximum number of processes or threads used in a parallel program. Attempts to dynamically set the number of processes or threads to a higher value, for example using set_omp_num_threads(), will cause the number of processes or threads to be set at the value of NCPUS_MAX rather than the value specified in the function call. NO_STOP_MESSAGE If the NO_STOP_MESSAGE variable exists, the execution of a plain STOP statement does not produce the message FORTRAN STOP. The default behavior of the PGI Fortran compilers is to issue this message. PATH The PATH variable sets the directories that are searched for commands that the user types. When using PGI products, it is important that you set your PATH to include the location of the PGI products. You can also use this variable to specify that you want to use only the linux86 version of the compilers and tools, or to target linux86 as the default. The following csh example targets x64 linux86-64 version of the compilers and tools. % set path = (/opt/pgi/linux86-64/7.1/bin $path) PGI The PGI environment variable specifies the root directory where the PGI compilers and tools are installed. This variable is recognized at compile-time. If it is not set, the default value depends on your system as well as which compilers are installed: • On Linux, the default value of this variable is /opt/pgi. • On Windows, the default value is C:\Program Files\PGI, where C represents the system drive. If both 32- and 64-bit compilers are installed, the 32-bit compilers are inC:\Program Files (x86)\ PGIChapter 8. Using Environment Variables 97 • For SFU/SUA compilers, the default value of this variable is /opt/pgi in the SFU/SUA file system. The corresponding Windows-style path is C:/SFU/opt/pgi for SFU and C:/WINDOWS/SUA/opt/pgi for SUA, where C represents the system drive. In most cases, if the PGI environment variable is not set, the PGI compilers and tools dynamically determine the location of this root directory based on the instance of the compiler or tool that was invoked. However, there are still some dependencies on the PGI environment variable, and it can be used as a convenience when initializing your environment for use of the PGI compilers and tools. For example, assuming you use csh and want the 64-bit linux86-64 versions of the PGI compilers and tools to be the default, you would use this syntax: % setenv PGI /usr/pgi % setenv MANPATH "$MANPATH":$PGI/linux86/6.0/man % setenv LM_LICENSE_FILE $PGI/license.dat % set path = ($PGI/linux86-64/6.0/bin $path) PGI_CONTINUE You set the PGI_CONTINUE variable to specify the actions to take before continuing with execution. For example, if the PGI_CONTINUE environment variable is set and a program compiled with –Mchkfpstk is executed, the stack is automatically cleaned up and execution then continues. If PGI_CONTINUE is set to verbose, the stack is automatically cleaned up, a warning message is printed, and then execution continues. Note There is a performance penalty associated with the stack cleanup. PGI_OBJSUFFIX You can set the PGI_OBJSUFFIX environment variable to generate object files that have a specific suffix. For example, if you set PGI_OBJSUFFIX to .o, the object files have a suffix of .o rather than .obj. PGI_STACK_USAGE (Windows only) The PGI_STACK_USAGE variable (for Windows only) allows you to explicitly set stack properties for your program. When the user compiles a program with the –Mchkstk option and sets the PGI_STACK_USAGE environment variable to any value, the program displays the stack space allocated and used after the program exits. You might see something similar to the following message: thread 0 stack: max 8180KB, used 48KB This message indicates that the program used 48KB of a 8180KB allocated stack. For more information on the –Mchkstk option, refer to –Mchkstk. PGI_TERM The PGI_TERM environment variable controls the stack traceback and just-in-time debugging functionality. The runtime libraries use the value of ‘ to determine what action to take when a program abnormally terminates.PGI® User’s Guide 98 The value of PGI_TERM is a comma-separated list of options. The commands for setting the environment variable follow. • In csh: % setenv PGI_TERM option[,option...] • In bash or sh: $ PGI_TERM=option[,option...] $ export PGI_TERM • In the Windows Command Prompt: C:\> set PGI_TERM=option[,option...] Table 8.2 lists the supported values for option. Following the table is a complete description of each option that indicates specifically how you might apply the option. By default, all of these options are disabled. Table 8.2. Supported PGI_TERM Values [no]debug Enables/disables just-in-time debugging (debugging invoked on error) [no]trace Enables/disables stack traceback on error [no]signal Enables/disables establishment of signal handlers for common signals that cause program termination [no]abort Enables/disables calling the system termination routine abort() [no]debug This enables/disables just-in-time debugging. The default is nodebug. When PGI_TERM is set to debug, the following command is invoked on error, unless you use PGI_TERM_DEBUG to override this default. pgdbg -text -attach is the process ID of the process being debugged. The PGI_TERM_DEBUG environment variable may be set to override the default setting. For more information, refer to “PGI_TERM_DEBUG,” on page 99. [no]trace This enables/disables the stack traceback. The default is notrace. [no]signal This enables/disables the establishing signal handlers for the most common signals that cause program termination. The default is nosignal. You can set trace and debug automatically enables signal. Specifically setting nosignal allows you to override this behavior.Chapter 8. Using Environment Variables 99 [no]abort This enables/disables calling the system termination routine abort(). The default is noabort. When noabort is in effect the process terminates by calling _exit(127). On Linux and SUA, when abort is in effect, the abort routine creates a core file and exits with code 127. On Windows, when abort is in effect, the abort routine exits with the status of the exception received. For example, if the program receives an access violation, abort() exits with status 0xC0000005. A few runtime errors just print an error message and call exit(127), regardless of the status of PGI_TERM. These are mainly errors such as specifying an invalid environment variable value where a traceback would not be useful. If it appears that abort() does not generate core files on a Linux system, be sure to unlimit the coredumpsize. You can do this in these ways: • Using csh: % limit coredumpsize unlimited % setenv PGI_TERM abort • Using bash or sh: $ ulimit -c unlimited $ export PGI_TERM=abort To debug a core file with pgdbg, start pgdbg with the -core option. For example, to view a core file named “core” for a program named “a.out”: $ pgdbg -core core a.out For more information on why to use this variable, refer to “Stack Traceback and JIT Debugging,” on page 101. PGI_TERM_DEBUG The PGI_TERM_DEBUG variable may be set to override the default behavior when PGI_TERM is set to debug. The value of PGI_TERM_DEBUG should be set to the command line used to invoke the program. For example: gdb --quiet --pid %d The first occurrence of %d in the PGI_TERM_DEBUG string will be replaced by the process id. The program named in the PGI_TERM_DEBUG string must be found on the currentPATH or specified with a full path name. PWD The PWD variable allows you to display the current directory. STATIC_RANDOM_SEED You can use STATIC_RANDOM_SEED to force the seed returned by the Fortran 90/95 RANDOM_SEED intrinsic to be constant. The first call to RANDOM_SEED without arguments resets the random seed to aPGI® User’s Guide 100 default value, then advances the seed by a variable amount based on time. Subsequent calls to RANDOM_SEED without arguments reset the random seed to the same initial value as the first call. Unless the time is exactly the same, each time a program is run a different random number sequence is generated. Setting the environment variable STATIC_RANDOM_SEED to YES forces the seed returned by RANDOM_SEED to be constant, thereby generating the same sequence of random numbers at each execution of the program. TMP You can use TMP to specify the directory to use for placement of any temporary files created during execution of the PGI compilers and tools. This variable is interchangeable with TMPDIR. TMPDIR You can use TMPDIR to specify the directory to use for placement of any temporary files created during execution of the PGI compilers and tools. Using Environment Modules On Linux, if you use the Environment Modules package, that is, the module load command, PGI 7.1 includes a script to set up the appropriate module files. Assuming your installation base directory is /opt/pgi, and your MODULEPATH environment variable is / usr/local/Modules/modulefiles, execute this command: % /opt/pgi/linux86/7.1-1/etc/modulefiles/pgi.module.install \ -all -install /usr/local/Modules/modulefiles This command creates module files for all installed versions of the PGI compilers. You must have write permission to the modulefiles directory to enable the module commands: % module load pgi32/7.1 % module load pgi64/7.1 % module load pgi/7.1 where "pgi/7.1" uses the 32-bit compilers on a 32-bit system and uses 64-bit compilers on a 64-bit system. To see what versions are available, use this command: % module avail pgi The module load command sets or modifies the environment variables as indicated in the following table. This Environment Variable... Is set or modified to ... CC Full path to pgcc V Path to pgCC V Full path to pgCC CXX Path to pgCC FC Full path to pgf95 F77 Full path to pgf77Chapter 8. Using Environment Variables 101 This Environment Variable... Is set or modified to ... F90 Full path to pgf95 LD_LIBRARY_PATH Prepends the PGI library directory MANPATH Prepends the PGI man page directory PATH Prepends the PGI compiler and tools bin directory PGI The base installation directory PGI does not provide support for the Environment Modules package. For more information about the package, go to: modules.sourceforge.net. Stack Traceback and JIT Debugging When a programming error results in a run-time error message or an application exception, a program will usually exit, perhaps with an error message. The PGI run-time library includes a mechanism to override this default action and instead print a stack traceback, start a debugger, or (on Linux) create a core file for postmortem debugging. The stack traceback and just-in-time debugging functionality is controlled by an environment variable, PGI_TERM. The run-time libraries use the value of PGI_TERM to determine what action to take when a program abnormally terminates. When the PGI runtime library detects an error or catches a signal, it calls the routine pgi_stop_here() prior to generating a stack traceback or starting the debugger. The pgi_stop_here routine is a convenient spot to set a breakpoint when debugging a program. For more information on PGI_Term and the supported values, refer to “PGI_TERM,” on page 97.102103 Chapter 9. Distributing Files - Deployment Once you have successfully built, debugged and tuned your application, you may want to distribute it to users who need to run it on a variety of systems. This chapter addresses how to effectively distribute applications built using PGI compilers and tools. The application must be installed in such a way that the it executes accurately on a system other than the one on which it was built, and which may be configured differently. Deploying Applications on Linux To successfully deploy your application on Linux, there are a number of issues to consider, including these: • Runtime Libraries • 64-bit Linux Systems • Redistribution of Files • Linux Portability of files and packages • Licensing Runtime Library Considerations On Linux systems, the system runtime libraries can be linked to an application either statically, or dynamically, For example, for the C runtime library, libc, you can use either the static version libc.a or the shared object libc.so. If the application is intended to run on Linux systems other than the one on which it was built, it is generally safer to use the shared object version of the library. This approach ensures that the application uses a version of the library that is compatible with the system on which the application is running. Further, it works best when the application is linked on a system that has an equivalent or earlier version of the system software than the system on which the application will be run. Note Building on a newer system and running the application on an older system may not produce the desired output.PGI® User’s Guide 104 To use the shared object version of a library, the application must also link to shared object versions of the PGI runtime libraries. To execute an application built in such a way on a system on which PGI compilers are not installed, those shared objects must be available.To build using the shared object versions of the runtime libraries, use the -Bdynamic option, as shown here: $ pgf90 -Bdynamic myprog.f90 64-bit Linux Considerations On 64-bit Linux systems, 64-bit applications that use the -mcmodel=medium option sometimes cannot be successfully linked statically. Therefore, users with executables built with the -mcmodel=medium option may need to use shared libraries, linking dynamically. Also, runtime libraries built using the -fpic option use 32-bit offsets, so they sometimes need to reside near other runtime libs in a shared area of Linux program memory. Note If your application is linked dynamically using shared objects, then the shared object versions of the PGI runtime are required. Linux Redistributable Files There are two methods for installing the shared object versions of the runtime libraries required for applications built with PGI compilers and tools: Linux Portability Package and manual distribution. PGI provides the Linux Portability Package, an installation package that can be downloaded from the PGI web site. In addition, when the PGI compilers are installed, there is a directory named REDIST for each platform (linux86 and linux86-64) that contains the redistributed shared object libraries. These may be redistributed by licensed PGI customers under the terms of the PGI End-User License Agreement. Restrictions on Linux Portability You cannot expect to be able to run an executable on any given Linux machine. Portability depends on the system you build on as well as how much your program uses system routines that may have changed from Linux release to Linux release. For example, one area of significant change between some versions of Linux is in libpthread.so. PGI compilers use this shared object for the options -Mconcur (auto-parallel) and - mp (OpenMP) programs. Typically, portability is supported for forward execution, meaning running a program on the same or a later version of Linux; but not for backward compatibility, that is, running on a prior release. For example, a user who compiles and links a program under Suse 9.1 should not expect the program to run without incident on a Red Hat 8.0 system, which is an earlier version of Linux. It may run, but it is less likely. Developers might consider building applications on earlier Linux versions for wider usage. Installing the Linux Portability Package You can download the Linux Portability Packages from the Downloads page at http://www.pgroup.com. First download the package you need, then untar it, and run the install script. Then you can add the installation directory to your library path.Chapter 9. Distributing Files - Deployment 105 To use the installed libraries, you can either modify /etc/ld.so.conf and run ldconfig(1) or modify the environment variable LD_LIBRARY_PATH, as shown here: setenv LD_LIBRARY_PATH /usr/local/pgi or export LD_LIBRARY_PATH=/usr/local/pgi Licensing for Redistributable Files The installation of the Linux Portability Package presents the standard PGI usage license. The libs can be distributed for use with PGI compiled applications, within the provisions of that license. The files in the REDIST directories may be redistributed under the terms of the End-User License Agreement for the product in which they were included. Deploying Applications on Windows Windows programs may be linked statically or dynamically. • A statically linked program is completely self-contained, created by linking to static versions of the PGI and Microsoft runtime libraries. • A dynamically linked program depends on separate dynamically-linked libraries (DLLs) that must be installed on a system for the application to run on that system. Although it may be simpler to install a statically linked executable, there are advantages to using the DLL versions of the runtime, including these: • Executable binary file size is smaller. • Multiple processes can use DLLs at once, saving system resources. • New versions of the runtime can be installed and used by the application without rebuilding the application. Dynamically-linked Windows programs built with PGI compilers depend on dynamic run-time library files (DLLs). These DLLs must be distributed with such programs to enable them to execute on systems where the PGI compilers are not installed. These redistributable libraries include both PGI runtime libraries and Microsoft runtime libraries. PGI Redistributables PGI Redistributable directories contain all of the PGI Linux runtime library shared object files or Windows dynamically- linked libraries that can be re-distributed by PGI 7.1 licensees under the terms of the PGI Enduser License Agreement (EULA). Microsoft Redistributables The PGI products on Windows include Microsoft Open Tools. The Microsoft Open Tools directory contains a subdirectory named redist. PGI licensees may redistribute the files contained in this directory in accordance with the terms of the PGI End-User License Agreement.PGI® User’s Guide 106 Microsoft supplies installation packages, vcredist_x86.exe and vcredist_x64.exe, containing these runtime files. You can download these packages from www.microsoft.com. Code Generation and Processor Architecture The PGI compilers can generate much more efficient code if they know the specific x86 processor architecture on which the program will run. When preparing to deploy your application, you should determine whether you want the application to run on the widest possible set of x86 processors, or if you want to restrict the application to run on a specific processor or set of processors. The restricted approach allows you to optimize performance for that set of processors. Different processors have differences, some subtle, in hardware features, such as instruction sets and cache size. The compilers make architecture-specific decisions about such things as instruction selection, instruction scheduling, and vectorization, all of which can have a profound effect on the performance of your application. Processor- specific code generation is controlled by the -tp option, described in “–tp [,target...] ,” on page 202. When an application is compiled without any -tp options, the compiler generates code for the type of processor on which the compiler is run. Generating Generic x86 Code To generate generic x86 code, use one of the following forms of the-tp option on your command line: -tp px ! generate code for any x86 cpu type -tp p6 ! generate code for Pentium 2 or greater While both of these examples are good choices for portable execution, most users have Pentium 2 or greater CPUs. Generating Code for a Specific Processor You can use the -tp option to request that the compiler generate code optimized for a specific processor. The PGI Release Notes contains a list of supported processors or you can look at the -tp entry in the compiler output generated by using the -help option, described in “–help ,” on page 178. Generating Code for Multiple Types of Processors in One Executable PGI unified binaries provide a low-overhead method for a single program to run well on a number of hardware platforms. All 64-bit PGI compilers can produce PGI Unified Binary programs that contain code streams fully optimized and supported for both AMD64 and Intel EM64T processors using the -tp target option. The compilers generate and combine multiple binary code streams into one executable, where each stream is optimized for a specific platform. At runtime, this one executable senses the environment and dynamically selects the appropriate code stream. Different processors have differences, some subtle, in hardware features, such as instruction sets and cache size. The compilers make architecture-specific decisions about such things as instruction selection, instructionChapter 9. Distributing Files - Deployment 107 scheduling, and vectorization. PGI unified binaries provide a low-overhead means for a single program to run well on a number of hardware platforms. Executable size is automatically controlled via unified binary culling. Only those functions and subroutines where the target affects the generated code will have unique binary images, resulting in a code-size savings of 10-90% compared to generating full copies of code for each target. Programs can use PGI Unified Binary even if all of the object files and libraries are not compiled as unified binaries. Like any other object file, you can use PGI Unified Binary object files to create programs or libraries. No special start up code is needed; support is linked in from the PGI libraries. The -Mpfi option disables generation of PGI Unified Binary. Instead, the default target auto-detect rules for the host are used to select the target processor. Unified Binary Command-line Switches The PGI Unified Binary command-line switch is an extension of the target processor switch, -tp, which may be applied to individual files during compilation . The target processor switch, -tp, accepts a comma-separated list of 64-bit targets and generates code optimized for each listed target. The following example generates optimized code for three targets: -tp k8-64,p7-64,core2-64 A special target switch, -tp x64, is the same as -tp k8-64, p7-64s. Unified Binary Directives and Pragma Unified binary directives and pragmas may be applied to functions, subroutines, or whole files. The directives and pragmas cause the compiler to generate PGI Unified Binary code optimized for one or more targets. No special command line options are needed for these pragmas and directives to take effect. The syntax of the Fortran directive is this: pgi$[g|r| ] pgi tp [target]... where the scope is g (global), r (routine) or blank. The default is r, routine. For example, the following syntax indicates that the whole file, represented by g, should be optimized for both k8_64 and p7_64. pgi$g pgi tp k8_64 p7_64 The syntax of the C/C++ pragma is this: #pragma [global|routine|] tp [target]... where the scope is global, routine, or blank. The default is routine. For example, the following syntax indicates that the next function should be optimized for k8_64, p7_64, and core2_64. #pragma routine tp k8_64 p7_64 core2_64108109 Chapter 10. Inter-language Calling This chapter describes inter-language calling conventions for C, C++, and Fortran programs using the PGI compilers. The following sections describe how to call a Fortran function or subroutine from a C or C++ program and how to call a C or C++ function from a Fortran program. For information on calling assembly language programs, refer to Chapter 18, “Run-time Environment”. This chapter provides examples that use the following options related to inter-language calling. For more information on these options, refer to Chapter 15, “Command-Line Options Reference,” on page 163. -c -Mnomain Overview of Calling Conventions This chapter includes information on the following topics: • Functions and subroutines in Fortran, C, and C++ • Naming and case conversion conventions • Compatible data types • Argument passing and special return values • Arrays and indexes • Win32 calling conventions The sections “Inter-language Calling Considerations,” on page 110 through“Example - C++ Calling Fortran,” on page 119 describe how to perform inter-language calling using the Linux/Win64/SUA convention. Default Fortran calling conventions for Win32 differ, although Win32 programs compiled using the -Munix Fortran command-line option use the Linux/Win64 convention rather than the default Win32 conventions. All information in those sections pertaining to compatibility of arguments applies to Win32 as well. For details on the symbol name and argument passing conventions used on Win32 platforms, refer to “Win32 Calling Conventions,” on page 120.PGI® User’s Guide 110 Inter-language Calling Considerations In general, when argument data types and function return values agree, you can call a C or C++ function from Fortran as well as call a Fortran function from C or C++. When data types for arguments do not agree, you may need to develop custom mechanisms to handle them. For example, the Fortran COMPLEX type has a matching type in C99 but does not have a matching type in C90; however, it is still possible to provide inter-language calls but there are no general calling conventions for such cases. Note • If a C++ function contains objects with constructors and destructors, calling such a function from either C or Fortran is not possible unless the initialization in the main program is performed from a C++ program in which constructors and destructors are properly initialized. • In general, you can call a C or Fortran function from C++ without problems as long as you use the extern "C" keyword to declare the function in the C++ program. This declaration prevents name mangling for the C function name. If you want to call a C++ function from C or Fortran, you also have to use the extern "C" keyword to declare the C++ function. This keeps the C++ compiler from mangling the name of the function. • You can use the __cplusplus macro to allow a program or header file to work for both C and C++. For example, the following defines in the header file stdio.h allow this file to work for both C and C++. #ifndef _STDIO_H #define _STDIO_H #ifdef __cplusplus extern "C" { #endif /* __cplusplus */ . . /* Functions and data types defined... */ . #ifdef __cplusplus } #endif /* __cplusplus */ #endif • C++ member functions cannot be declared extern, since their names will always be mangled. Therefore, C++ member functions cannot be called from C or Fortran. Functions and Subroutines Fortran, C, and C++ define functions and subroutines differently. For a Fortran program calling a C or C++ function, observe the following return value convention: • When a C or C++ function returns a value, call it from Fortran as a function. • When a C or C++ function does not return a value, call it as a subroutine. For a C/C++ program calling a Fortran function, the call should return a similar type. Table 10.1, “Fortran and C/C++ Data Type Compatibility,” on page 111 lists compatible types. If the call is to a Fortran subroutine,Chapter 10. Inter-language Calling 111 a Fortran CHARACTER function, or a Fortran COMPLEX function, call it from C/C++ as a function that returns void. The exception to this convention is when a Fortran subroutine has alternate returns; call such a subroutine from C/C++ as a function returning int whose value is the value of the integer expression specified in the alternate RETURN statement. Upper and Lower Case Conventions, Underscores By default on Linux, Win64, OSX, and SUA systems, all Fortran symbol names are converted to lower case. C and C++ are case sensitive, so upper-case function names stay upper-case. When you use inter-language calling, you can either name your C/C++ functions with lower-case names, or invoke the Fortran compiler command with the option –Mupcase, in which case it will not convert symbol names to lower-case. When programs are compiled using one of the PGI Fortran compilers on Linux, Win64, OSX, and SUA systems, an underscore is appended to Fortran global names (names of functions, subroutines and common blocks). This mechanism distinguishes Fortran name space from C/C++ name space. Use these naming conventions: • If you call a C/C++ function from Fortran, you should rename the C/C++ function by appending an underscore or use C$PRAGMA C in the Fortran program. For more information on C$PRAGMA C, refer to “C$PRAGMA C,” on page 72. • If you call a Fortran function from C/C++, you should append an underscore to the Fortran function name in the calling program. Compatible Data Types Table 10.1 shows compatible data types between Fortran and C/C++. Table 10.2, “Fortran and C/C++ Representation of the COMPLEX Type,” on page 112 shows how the Fortran COMPLEX type may be represented in C/C++. If you can make your function/subroutine parameters as well as your return values match types, you should be able to use inter-language calling. Table 10.1. Fortran and C/C++ Data Type Compatibility Fortran Type (lower case) C/C++ Type Size (bytes) character x char x 1 character*n x char x[n] n real x float x 4 real*4 x float x 4 real*8 x double x 8 double precision double x 8 integer x int x 4 integer*1 x signed char x 1 integer*2 x short x 2 integer*4 x int x 4 integer*8 x long long x 8PGI® User’s Guide 112 Fortran Type (lower case) C/C++ Type Size (bytes) logical x int x 4 logical*1 x char x 1 logical*2 x short x 2 logical*4 int x 4 logical*8 long long x 8 Table 10.2. Fortran and C/C++ Representation of the COMPLEX Type Fortran Type (lower case) C/C++ Type Size (bytes) complex x struct {float r,i;} x; 8 float complex x; complex*8 x struct {float r,i;} x; 8 float complex x; 8 double complex x struct {double dr,di;} x; 16 double complex x; 16 complex *16 x struct {double dr,di;} x; 16 double complex x; 16 Note For C/C++, the complex type implies C99 or later. Fortran Named Common Blocks A named Fortran common block can be represented in C/C++ by a structure whose members correspond to the members of the common block. The name of the structure in C/C++ must have the added underscore. For example, the Fortran common block: INTEGER I COMPLEX C DOUBLE COMPLEX CD DOUBLE PRECISION D COMMON /COM/ i, c, cd, d is represented in C with the following equivalent: extern struct { int i; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; and in C++ with the following equivalent:Chapter 10. Inter-language Calling 113 extern "C" struct { int i; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; Tip For global or external data sharing, extern “C” is not required. Argument Passing and Return Values In Fortran, arguments are passed by reference, that is, the address of the argument is passed, rather than the argument itself. In C/C++, arguments are passed by value, except for strings and arrays, which are passed by reference. Due to the flexibility provided in C/C++, you can work around these differences. Solving the parameter passing differences generally involves intelligent use of the & and * operators in argument passing when C/C++ calls Fortran and in argument declarations when Fortran calls C/C++. For strings declared in Fortran as type CHARACTER, an argument representing the length of the string is also passed to a calling function. On Linux systems, or when using the UNIX calling convention on Windows (-Munix), the compiler places the length argument(s) at the end of the parameter list, following the other formal arguments. The length argument is passed by value, not by reference. Passing by Value (%VAL) When passing parameters from a Fortran subprogram to a C/C++ function, it is possible to pass by value using the %VAL function. If you enclose a Fortran parameter with %VAL(), the parameter is passed by value. For example, the following call passes the integer i and the logical bvar by value. integer*1 i logical*1 bvar call cvalue (%VAL(i), %VAL(bvar)) Character Return Values “Functions and Subroutines,” on page 110 describes the general rules for return values for C/C++ and Fortran inter-language calling. There is a special return value to consider. When a Fortran function returns a character, two arguments need to be added at the beginning of the C/C++ calling function’s argument list: • The address of the return character or characters • The length of the return character Example 10.1, “Character Return Parameters” illustrates the extra parameters, tmp and 10, supplied by the caller:PGI® User’s Guide 114 Example 10.1. Character Return Parameters ! Fortran function returns a character CHARACTER*(*) FUNCTION CHF( C1,I) CHARACTER*(*) C1 INTEGER I END /* C declaration of Fortran function */ extern void chf_(); char tmp[10]; char c1[9]; int i; chf_(tmp, 10, c1, &i, 9); If the Fortran function is declared to return a character value of constant length, for example CHARACTER*4 FUNCTION CHF(), the second extra parameter representing the length must still be supplied, but is not used. NOTE The value of the character function is not automatically NULL-terminated. Complex Return Values When a Fortran function returns a complex value, an argument needs to be added at the beginning of the C/ C++ calling function’s argument list; this argument is the address of the complex return value. Example 10.2, “COMPLEX Return Values” illustrates the extra parameter, cplx, supplied by the caller. Example 10.2. COMPLEX Return Values COMPLEX FUNCTION CF(C, I) INTEGER I . . . END extern void cf_(); typedef struct {float real, imag;} cplx; cplx c1; int i; cf_(&c1, &i); Array Indices C/C++ arrays and Fortran arrays use different default initial array index values. By default, C/C++ arrays start at 0 and Fortran arrays start at 1. If you adjust your array comparisons so that a Fortran second element is compared to a C/C++ first element, and adjust similarly for other elements, you should not have problems working with this difference. If this is not satisfactory, you can declare your Fortran arrays to start at zero. Another difference between Fortran and C/C++ arrays is the storage method used. Fortran uses columnmajor order and C/C++ use row-major order. For one-dimensional arrays, this poses no problems. For twodimensional arrays, where there are an equal number of rows and columns, row and column indexes can simply be reversed. For arrays other than single dimensional arrays, and square two-dimensional arrays, interlanguage function mixing is not recommended.Chapter 10. Inter-language Calling 115 Examples This section contains examples that illustrate inter-language calling. Example - Fortran Calling C Example 10.4, “C function cfunc_” shows a C function that is called by the Fortran main program shown in Example 10.3, “Fortran Main Program fmain.f”. Notice that each argument is defined as a pointer, since Fortran passes by reference. Also notice that the C function name uses all lower-case and a trailing "_". Example 10.3. Fortran Main Program fmain.f logical*1 bool1 character letter1 integer*4 numint1, numint2 real numfloat1 double precision numdoub1 integer*2 numshor1 external cfunc call cfunc (bool1, letter1, numint1, numint2, + numfloat1, numdoub1, numshor1) write( *, "(L2, A2, I5, I5, F6.1, F6.1, I5)") + bool1, letter1, numint1, numint2, numfloat1, + numdoub1, numshor1 end Example 10.4. C function cfunc_ #define TRUE 0xff #define FALSE 0 void cfunc_( bool1, letter1, numint1, numint2, numfloat1,\ numdoub1, numshor1, len_letter1) char *bool1, *letter1; int *numint1, *numint2; float *numfloat1; double *numdoub1; short *numshor1; int len_letter1; { *bool1 = TRUE; *letter1 = 'v'; *numint1 = 11; *numint2 = -44; *numfloat1 = 39.6 ; *numdoub1 = 39.2; *numshor1 = 981; } Compile and execute the program fmain.f with the call to cfunc_ using the following command lines: $ pgcc -c cfunc.c $ pgf95 cfunc.o fmain.f Executing the a.out file should produce the following output: T v 11 -44 39.6 39.2 981 Example - C Calling Fortran Example 10.6, “C Main Program cmain.c” shows a C main program that calls the Fortran subroutine shown in Example 10.5, “Fortran Subroutine forts.f”. Notice that each call uses the & operator to pass by reference. Also notice that the call to the Fortran subroutine uses all lower-case and a trailing "_".PGI® User’s Guide 116 Example 10.5. Fortran Subroutine forts.f subroutine forts ( bool1, letter1, numint1 + numint2, numfloat1, numdoub1, numshor1) logical*1 bool1 character letter1 integer numint1, numint2 double precision numdoub1 real numfloat1 integer*2 numshor1 bool1 = .true. letter1 = "v" numint1 = 11 numint2 = -44 numdoub1 = 902 numfloat1 = 39.6 numshor1 = 299 return end Example 10.6. C Main Program cmain.c main () { char bool1, letter1; int numint1, numint2; float numfloat1; double numdoub1; short numshor1; extern void forts_ (); forts_(&bool1,&letter1,&numint1,&numint2,&numfloat1,&numdoub1,&numshor1, 1); printf(" %s %c %d %d %3.1f %.0f %d\n", bool1?"TRUE":"FALSE",letter1,numint1, numint2, numfloat1, numdoub1, numshor1); } To compile this Fortran subroutine and C program, use the following commands: $ pgcc -c cmain.f $ pgf95 -Mnomain cmain.o forts.f Executing the resulting a.out file should produce the following output: TRUE v 11 -44 39.6 902 299 Example - C ++ Calling C Example 10.8, “C++ Main Program cpmain.C Calling a C Function” shows a C++ main program that calls the C function shown in Example 10.7, “Simple C Function cfunc.c”. Example 10.7. Simple C Function cfunc.c void cfunc(num1, num2, res) int num1, num2, *res; { printf("func: a = %d b = %d ptr c = %x\n",num1,num2,res); *res=num1/num2; printf("func: res = %d\n",*res); }Chapter 10. Inter-language Calling 117 Example 10.8. C++ Main Program cpmain.C Calling a C Function xtern "C" void cfunc(int n, int m, int *p); #include main() { int a,b,c; a=8; b=2; cout << "main: a = "< extern "C" { extern void forts_(char *,char *,int *,int *, float *,double *,short *); } main () { char bool1, letter1; int numint1, numint2; float numfloat1; double numdoub1; short numshor1; forts_(&bool1,&letter1,&numint1,&numint2,&numfloat1, &numdoub1,&numshor1); cout << " bool1 = "; bool1?cout << "TRUE ":cout << "FALSE "; cout < 2GB in size. Note that if you execute with the above settings in your environment, you may see the following: % bigadd Segmentation fault Execution fails because the stack size is not large enough. Try resetting the stack size in your environment: % limit stacksize 3000M PGI® User’s Guide 130 Note that ‘limit stacksize unlimited’ will probably not provide as large a stack as we are using above. % bigadd a[0]=1 b[0]=2 c[0]=3 n=599990000 a[599990000]=5.9999e+08 b[599990000]=1.19998e+09 c[599990000]=1.79997e+09 The size of the bss section of the bigadd executable is now larger than 2GB: % size –-format=sysv bigadd | grep bss .bss 4800000008 5245696 % size -–format=sysv bigadd | grep Total Total 4800005080 Example: Medium Memory Model and Large Array in Fortran The following example works with both the PGF95 and PGF77 compilers included in Release 7.0. Both compilers use 64-bit addresses and index arithmetic when the –mcmodel=medium option is used. Consider the following example: % cat mat.f program mat integer i, j, k, size, l, m, n parameter (size=16000) ! >2GB parameter (m=size,n=size) real*8 a(m,n),b(m,n),c(m,n),d do i = 1, m do j = 1, n a(i,j)=10000.0D0*dble(i)+dble(j) b(i,j)=20000.0D0*dble(i)+dble(j) enddo enddo !$omp parallel !$omp do do i = 1, m do j = 1, n c(i,j) = a(i,j) + b(i,j) enddo enddo !$omp do do i=1,m do j = 1, n d = 30000.0D0*dble(i)+dble(j)+dble(j) if(d .ne. c(i,j)) then print *,”err i=”,i,”j=”,j print *,”c(i,j)=”,c(i,j) print *,”d=”,d stop endif enddo enddo !$omp end parallel print *, “M =”,M,”, N =”,N print *, “c(M,N) = “, c(m,n) end When compiled with the PGF95 compiler using –mcmodel=medium: % pgf95 –mp –o mat mat.f –i8 –mcmodel=mediumChapter 11. Programming Considerations for 64-Bit Environments 131 % setenv OMP_NUM_THREADS 2 % mat M = 16000 , N = 16000 c(M,N) = 480032000.0000000 Example: Large Array and Small Memory Model in Fortran The following example uses large, dynamically-allocated arrays. The code is divided into a main and subroutine so you could put the subroutine into a shared library. Dynamic allocation of large arrays saves space in the size of executable and saves time initializing data. Further, the routines can be compiled with 32- bit compilers, by just decreasing the parameter size below. % cat mat_allo.f90 program mat_allo integer i, j integer size, m, n parameter (size=16000) parameter (m=size,n=size) double precision, allocatable::a(:,:),b(:,:),c(:,:) allocate(a(m,n), b(m,n), c(m,n)) do i = 100, m, 1 do j = 100, n, 1 a(i,j) = 10000.0D0 * dble(i) + dble(j) b(i,j) = 20000.0D0 * dble(i) + dble(j) enddo enddo call mat_add(a,b,c,m,n) print *, “M =”,m,”,N =”,n print *, “c(M,N) = “, c(m,n) end subroutine mat_add(a,b,c,m,n) integer m, n, i, j double precision a(m,n),b(m,n),c(m,n) !$omp do do i = 1, m do j = 1, n c(i,j) = a(i,j) + b(i,j) enddo enddo return end % pgf95 –o mat_allo mat_allo.f90 –i8 –Mlarge_arrays -mp -fast132133 Chapter 12. C/C++ Inline Assembly and Intrinsics Inline Assembly Inline Assembly lets you specify machine instructions inside a "C" function. The format for an inline assembly instruction is this: { asm | __asm__ } ("string"); The asm statement begins with the asm or __asm__ keyword. The __asm__ keyword is typically used in header files that may be included in ISO "C" programs. "string" is one or more machine specific instructions separated with a semi-colon (;) or newline (\n) character. These instructions are inserted directly into the compiler's assembly-language output for the enclosing function. Some simple asm statements are: asm ("cli"); asm ("sti"); The asm statements above disable and enable system interrupts respectively. In the following example, the eax register is set to zero. asm( "pushl %eax\n\t" "movl $0, %eax\n\t" "popl %eax"); Notice that eax is pushed on the stack so that it is it not clobbered. When the statement is done with eax, it is restored with the popl instruction. Typically a program uses macros that enclose asm statements. The interrupt constructs shown above are used in the following two examples: #define disableInt __asm__ ("cli"); #define enableInt __asm__ ("sti");PGI® User’s Guide 134 Extended Inline Assembly “Inline Assembly,” on page 133 explains how to use inline assembly to specify machine specific instructions inside a "C" function. This approach works well for simple machine operations such as disabling and enabling system interrupts. However, inline assembly has three distinct limitations: 1. The programmer must choose the registers required by the inline assembly. 2. To prevent register clobbering, the inline assembly must include push and pop code for registers that get modified by the inline assembly. 3. There is no easy way to access stack variables in an inline assembly statement. Extended Inline Assembly was created to address these limitations. The format for extended inline assembly, also known as extended asm, is as follows: { asm | __asm__ } [ volatile | __volatile__ ] ("string" [: [output operands]] [: [input operands]] [: [clobberlist]]); • Extended asm statements begin with the asm or __asm__ keyword. Typically the __asm__ keyword is used in header files that may be included by ISO "C" programs. • An optional volatile or __volatile__ keyword may appear after the asm keyword. This keyword instructs the compiler not to delete, move significantly, or combine with any other asm statement. Like __asm__, the __volatile__ keyword is typically used with header files that may be included by ISO "C" programs. • "string" is one or more machine specific instructions separated with a semi-colon (;) or newline (\n) character. The string can also contain operands specified in the [output operands], [input operands], and [clobber list]. The instructions are inserted directly into the compiler's assembly-language output for the enclosing function. • The [output operands], [input operands], and [clobber list] items each describe the effect of the instruction for the compiler. For example: asm( "movl %1, %%eax\n" "movl %%eax, %0":"=r" (x) : "r" (y) : "%eax" ); where "=r" (x) is an output operand "r" (y) is an input operand. "%eax" is the clobber list consisting of one register, "%eax". The notation for the output and input operands is a constraint string surrounded by quotes, followed by an expression, and surrounded by parentheses. The constraint string describes how the input and output operands are used in the asm "string". For example, "r" tells the compiler that the operand is a register. The "=" tells the compiler that the operand is write only, which means that a value is stored in an output operand's expression at the end of the asm statement. Each operand is referenced in the asm "string" by a percent "%" and its number. The first operand is number 0, the second is number 1, the third is number 2, and so on. In the preceding example, "%0" references the output operand, and "%1" references the input operand. The asm "string" also contains "%%eax", which references machine register "%eax". Hard coded registers like "%eax" should be specified in the clobber list to prevent conflicts with other instructions in the compiler's assembly-language output.Chapter 12. C/C++ Inline Assembly and Intrinsics 135 [output operands], [input operands], and [clobber list] items are described in more detail in the following sections. Output Operands The [output operands] are an optional list of output constraint and expression pairs that specify the result(s) of the asm statement. An output constraint is a string that specifies how a result is delivered to the expression. For example, "=r" (x) says the output operand is a write-only register that stores its value in the "C" variable x at the end of the asm statement. An example follows: int x; void example() { asm( "movl $0, %0" : "=r" (x) ); } The previous example assigns 0 to the "C" variable x. For the function in this example, the compiler produces the following assembly. If you want to produce an assembly listing, compile the example with the pgcc -S compiler option: example: ..Dcfb0: pushq %rbp ..Dcfi0: movq %rsp, %rbp ..Dcfi1: ..EN1: ## lineno: 8 movl $0, %eax movl %eax, x(%rip) ## lineno: 0 popq %rbp ret In the generated assembly shown, notice that the compiler generated two statements for the asm statement at line number 5. The compiler generated "movl $0, %eax" from the asm "string". Also notice that %eax appears in place of "%0" because the compiler assigned the %eax register to variable x. Since item 0 is an output operand, the result must be stored in its expression (x). The instruction movl %eax, x(%rip) assigns the output operand to variable x. In addition to write-only output operands, there are read/write output operands designated with a "+" instead of a "=". For example, "+r" (x) tells the compiler to initialize the output operand with variable x at the beginning of the asm statement. To illustrate this point, the following example increments variable x by 1: int x=1; void example2() { asm( "addl $1, %0" : "+r" (x) ); } To perform the increment, the output operand must be initialized with variable x. The read/write constraint modifier ("+") instructs the compiler to initialize the output operand with its expression. The compiler generates the following assembly code for the example2() function:PGI® User’s Guide 136 example2: ..Dcfb0: pushq %rbp ..Dcfi0: movq %rsp, %rbp ..Dcfi1: ..EN1: ## lineno: 5 movl x(%rip), %eax addl $1, %eax movl %eax, x(%rip) ## lineno: 0 popq %rbp ret From the example(2) code, two extraneous moves are generated in the assembly: one movl for initializing the output register and a second movl to write it to variable x. To eliminate these moves, use a memory constraint type instead of a register constraint type, as shown in the following example: int x=1; void example2() { asm( "addl $1, %0" : "+m" (x) ); } The compiler generates a memory reference in place of a memory constraint. This eliminates the two extraneous moves: example2: ..Dcfb0: pushq %rbp ..Dcfi0: movq %rsp, %rbp ..Dcfi1: ..EN1: ## lineno: 5 addl $1, x(%rip) ## lineno: 0 popq %rbp ret Because the assembly uses a memory reference to variable x, it does not have to move x into a register prior to the asm statement; nor does it need to store the result after the asm statement. Additional constraint types are found in “Additional Constraints,” on page 139. The examples thus far have used only one output operand. Because extended asm accepts a list of output operands, asm statements can have more than one result. For example: void example4() { int x=1; int y=2; asm( "addl $1, %1\n" "addl %1, %0": "+r" (x), "+m" (y) ); } The example above increments variable y by 1 then adds it to variable x. Multiple output operands are separated with a comma. The first output operand is item 0 ("%0") and the second is item 1 ("%1") in the asm "string". The resulting values for x and y are 4 and 3 respectively.Chapter 12. C/C++ Inline Assembly and Intrinsics 137 Input Operands The [input operands] are an optional list of input constraint and expression pairs that specify what "C" values are needed by the asm statement. The input constraints specify how the data is delivered to the asm statement. For example, "r" (x) says that the input operand is a register that has a copy of the value stored in "C" variable x. Another example is "m" (x) which says that the input item is the memory location associated with variable x. Other constraint types are discussed in “Additional Constraints,” on page 139. An example follows: void example5() { int x=1; int y=2; int z=3; asm( "addl %2, %1\n" "addl %2, %0" : "+r" (x), "+m" (y) : "r" (z) ); } The previous example adds variable z, item 2, to variable x and variable y. The resulting values for x and y are 4 and 5 respectively. Another type of input constraint worth mentioning here is the matching constraint. A matching constraint is used to specify an operand that fills both an input as well as an output role. An example follows: int x=1; void example6() { asm( "addl $1, %1" : "=r" (x) : "0" (x) ); } The previous example is equivalent to the example2() function shown in “Output Operands,” on page 135. The constraint/expression pair, "0" (x), tells the compiler to initialize output item 0 with variable x at the beginning of the asm statement. The resulting value for x is 2. Also note that "%1" in the asm "string" means the same thing as "%0" in this case. That is because there is only one operand with both an input and an output role. Matching constraints are very similar to the read/write output operands mentioned in “Output Operands,” on page 135. However, there is one key difference between read/write output operands and matching constraints. The matching constraint can have an input expression that differs from its output expression. The example below uses different values for the input and output roles: int x; int y=2; void example7() { asm( "addl $1, %1" : "=r" (x) : "0" (y) ); } The compiler generates the following assembly for example7(): example7: ..Dcfb0: pushq %rbp ..Dcfi0:PGI® User’s Guide 138 movq %rsp, %rbp ..Dcfi1: ..EN1: ## lineno: 8 movl y(%rip), %eax addl $1, %eax movl %eax, x(%rip) ## lineno: 0 popq %rbp ret Variable x gets initialized with the value stored in y, which is 2. After adding 1, the resulting value for variable x is 3. Because matching constraints perform an input role for an output operand, it does not make sense for the output operand to have the read/write ("+") modifier. In fact, the compiler disallows matching constraints with read/write output operands. The output operand must have a write only ("=") modifier. Clobber List The [clobber list] is an optional list of strings that hold machine registers used in the asm "string". Essentially, these strings tell the compiler which registers may be clobbered by the asm statement. By placing registers in this list, the programmer does not have to explicitly save and restore them as required in traditional inline assembly (described in “Inline Assembly,” on page 133). The compiler takes care of any required saving and restoring of the registers in this list. Each machine register in the [clobber list] is a string separated by a comma. The leading '%' is optional in the register name. For example, "%eax" is equivalent to "eax". When specifying the register inside the asm "string", you must include two leading '%' characters in front of the name (for example., "%%eax"). Otherwise, the compiler will behave as if a bad input/output operand was specified and generate an error message. An example follows: void example8() { int x; int y=2; asm( "movl %1, %%eax\n" "movl %1, %%edx\n" "addl %%edx, %%eax\n" "addl %%eax, %0" : "=r" (x) : "0" (y) : "eax", "edx" ); } The code shown above uses two hard-coded registers, eax and edx. It performs the equivalent of 3*y and assigns it to x, producing a result of 6. In addition to machine registers, the clobber list may contain the following special flags: "cc" The asm statement may alter the condition code register. "memory" The asm statement may modify memory in an unpredictable fashion.Chapter 12. C/C++ Inline Assembly and Intrinsics 139 The "memory" flag causes the compiler not to keep memory values cached in registers across the asm statement and not to optimize stores or loads to that memory. For example: asm("call MyFunc":::"memory"); This asm statement contains a "memory" flag because it contains a call. The callee may otherwise clobber registers in use by the caller without the "memory" flag. The following function uses extended asm and the "cc" flag to compute a power of 2 that is less than or equal to the input parameter n. #pragma noinline int asmDivideConquer(int n) { int ax = 0; int bx = 1; asm ( "LogLoop:\n" "cmp %2, %1\n" "jnle Done\n" "inc %0\n" "add %1,%1\n" "jmp LogLoop\n" "Done:\n" "dec %0\n" :"+r" (ax), "+r" (bx) : "r" (n) : "cc"); return ax; } The "cc" flag is used because the asm statement contains some control flow that may alter the condition code register. The #pragma noinline statement prevents the compiler from inlining the asmDivideConquer()function. If the compiler inlines asmDivideConquer(), then it may illegally duplicate the labels LogLoop and Done in the generated assembly. Additional Constraints Operand constraints can be divided into four main categories: • Simple Constraints • Machine Constraints • Multiple Alternative Constraints • Constraint Modifiers Simple Constraints The simplest kind of constraint is a string of letters or characters, known as Simple Constraints, such as the "r" and "m" constraints introduced in “Output Operands,” on page 135. Table 12.1, “Simple Constraints” describes these constraints. Table 12.1. Simple Constraints Constraint Description whitespace Whitespace characters are ignored.PGI® User’s Guide 140 Constraint Description E An immediate floating point operand. F Same as "E". g Any general purpose register, memory, or immediate integer operand is allowed. i An immediate integer operand. m A memory operand. Any address supported by the machine is allowed. n Same as "i". o Same as "m". p An operand that is a valid memory address. The expression associated with the constraint is expected to evaluate to an address (for example, "p" (&x) ). r A general purpose register operand. X Same as "g". 0,1,2,..9 Matching Constraint. See “Input Operands,” on page 137 for a description. The following example uses the general or "g" constraint, which allows the compiler to pick an appropriate constraint type for the operand; the compiler chooses from a general purpose register, memory, or immediate operand. This code lets the compiler choose the constraint type for "y". void example9() { int x, y=2; asm( "movl %1, %0\n" : "=r" (x) : "g" (y) ); } This technique can result in more efficient code. For example, when compiling example9() the compiler replaces the load and store of y with a constant 2. The compiler can then generate an immediate 2 for the y operand in the example. The assembly generated by pgcc for our example is as follows: example9: ..Dcfb0: pushq %rbp ..Dcfi0: movq %rsp, %rbp ..Dcfi1: ..EN1: ## lineno: 3 movl $2, %eax ## lineno: 6 popq %rbp ret In this example, notice the use of $2 for the "y" operand. Of course, if y is always 2, then the immediate value may be used instead of the variable with the "i" constraint, as shown here: void example10()Chapter 12. C/C++ Inline Assembly and Intrinsics 141 { int x; asm( "movl %1, %0\n" : "=r" (x) : "i" (2) ); } Compiling example10() with pgcc produces assembly similar to that produced for example9(). Machine Constraints Another category of constraints is Machine Constraints. The x86 and x86_64 architectures have several classes of registers. To choose a particular class of register, you can use the x86/x86_64 machine constraints described in Table 12.2, “x86/x86_64 Machine Constraints”. Table 12.2. x86/x86_64 Machine Constraints Constraint Description a a register (e.g., %al, %ax, %eax, %rax) A Specifies a or d registers. This is used primarily for holding 64-bit integer values on 32 bit targets. The d register holds the most significant bits and the a register holds the least significant bits. b b register (e.g, %bl, %bx, %ebx, %rbx) c c register (e.g., %cl, %cx, %ecx, %rcx) C Not supported. d d register (e.g., %dl, %dx, %edx, %rdx) D di register (e.g., %dil, %di, %edi, %rdi) e Constant in range of 0xffffffff to 0x7fffffff f Not supported. G Floating point constant in range of 0.0 to 1.0. I Constant in range of 0 to 31 (e.g., for 32-bit shifts). J Constant in range of 0 to 63 (e.g., for 64-bit shifts) K Constant in range of 0 to 127. L Constant in range of 0 to 65535. M Constant in range of 0 to 3 constant (e.g., shifts for lea instruction). N Constant in range of 0 to 255 (e.g., for out instruction). q Same as "r" simple constraint. Q Same as "r" simple constraint. R Same as "r" simple constraint. S si register (e.g., %sil, %si, %edi, %rsi) t Not supported. u Not supported.PGI® User’s Guide 142 Constraint Description x XMM SSE register y Not supported. Z Constant in range of 0 to 0x7fffffff. The following example uses the "x" or XMM register constraint to subtract c from b and store the result in a. double example11() { double a; double b = 400.99; double c = 300.98; asm ( "subpd %2, %0;" :"=x" (a) : "0" (b), "x" (c) ); return a; } The generated assembly for this example is this: example11: ..Dcfb0: pushq %rbp ..Dcfi0: movq %rsp, %rbp ..Dcfi1: ..EN1: ## lineno: 4 movsd .C00128(%rip), %xmm1 movsd .C00130(%rip), %xmm2 movapd %xmm1, %xmm0 subpd %xmm2, %xmm0; ## lineno: 10 ## lineno: 11 popq %rbp ret If a specified register is not available, the pgcc and pgcpp compilers issue an error message. For example, pgcc and pgcpp reserves the "%ebx" register for Position Independent Code (PIC) on 32-bit system targets. If a program has an asm statement with a "b" register for one of the operands, the compiler will not be able to obtain that register when compiling for 32-bit with the -fPIC switch (which generates PIC). To illustrate this point, the following example is compiled for a 32-bit target using PIC: void example12() { int x=1; int y=1; asm( "addl %1, %0\n" : "+a" (x) : "b" (y) ); } Compiling with the "-tp p7" switch chooses a 32-bit target.Chapter 12. C/C++ Inline Assembly and Intrinsics 143 % pgcc example12.c -fPIC -c -tp p7 PGC-S-0354-Can't find a register in class 'BREG' for extended ASM operand 1 (example12.c: 3) PGC/x86 Linux/x86 Rel Dev: compilation completed with severe errors Multiple Alternative Constraints Sometimes a single instruction can take a variety of operand types. For example, the x86 permits registerto-memory and memory-to-register operations. To allow this flexibility in inline assembly, use multiple alternative constraints. An alternative is a series of constraints for each operand. To specify multiple alternatives, separate each alternative with a comma. Table 12.3. Multiple Alternative Constraints Constraint Description , Separates each alternative for a particular operand. ? Ignored ! Ignored The following example uses multiple alternatives for an add operation. void example13() { int x=1; int y=1; asm( "addl %1, %0\n" : "+ab,cd" (x) : "db,cam" (y) ); } example13() has two alternatives for each operand: "ab,cd" for the output operand and "db,cam" for the input operand. Each operand must have the same number of alternatives; however, each alternative can have any number of constraints (for example, the output operand in example13() has two constraints for its second alternative and the input operand has three for its second alternative). The compiler first tries to satisfy the left-most alternative of the first operand (for example, the output operand in example13()). When satisfying the operand, the compiler starts with the left-most constraint. If the compiler cannot satisfy an alternative with this constraint (for example, if the desired register is not available), it tries to use any subsequent constraints. If the compiler runs out of constraints, it moves on to the next alternative. If the compiler runs out of alternatives, it issues an error similar to the one mentioned in example12(). If an alternative is found, the compiler uses the same alternative for subsequent operands. For example, if the compiler chooses the "c" register for the output operand in example13(), then it will use either the "a" or "m" constraint for the input operand. Constraint Modifiers Characters that affect the compiler's interpretation of a constraint are known as Constraint Modifiers. Two constraint modifiers, the "=" and the "+", were introduced in “Output Operands,” on page 135. Table 12.4 summarizes each constraint modifier.PGI® User’s Guide 144 Table 12.4. Constraint Modifier Characters Constraint Modifier Description = This operand is write-only. It is valid for output operands only. If specified, the "=" must appear as the first character of the constraint string. + This operand is both read and written by the instruction. It is valid for output operands only. The output operand is initialized with its expression before the first instruction in the asm statement. If specified, the "+" must appear as the first character of the constraint string. & A constraint or an alternative constraint, as defined in “Multiple Alternative Constraints,” on page 143, containing an "&" indicates that the output operand is an early clobber operand. This type operand is an output operand that may be modified before the asm statement finishes using all of the input operands. The compiler will not place this operand in a register that may be used as an input operand or part of any memory address. % Ignored. # Characters following a "#" up to the first comma (if present) are to be ignored in the constraint. * The character that follows the "*" is to be ignored in the constraint. The "=" and "+" modifiers apply to the operand, regardless of the number of alternatives in the constraint string. For example, the "+" in the output operand of example13() appears once and applies to both alternatives in the constraint string. The "&", "#", and "*" modifiers apply only to the alternative in which they appear. Normally, the compiler assumes that input operands are used before assigning results to the output operands. This assumption lets the compiler reuse registers as needed inside the asm statement. However, if the asm statement does not follow this convention, the compiler may indiscriminately clobber a result register with an input operand. To prevent this behavior, apply the early clobber "&" modifier. An example follows: void example15() { int w=1; int z; asm( "movl $1, %0\n" "addl %2, %0\n" "movl %2, %1" : "=a" (w), "=r" (z) : "r" (w) ); } The previous code example presents an interesting ambiguity because "w" appears both as an output and as an input operand. So, the value of "z" can be either 1 or 2, depending on whether the compiler uses the same register for operand 0 and operand 2. The use of constraint "r" for operand 2 allows the compiler to pick any general purpose register, so it may (or may not) pick register "a" for operand 2. This ambiguity can be eliminated by changing the constraint for operand 2 from "r" to "a" so the value of "z" will be 2, or by adding an early clobber "&" modifier so that "z" will be 1. The following example shows the same function with an early clobber "&" modifier:Chapter 12. C/C++ Inline Assembly and Intrinsics 145 void example16() { int w=1; int z; asm( "movl $1, %0\n" "addl %2, %0\n" "movl %2, %1" : "=&a" (w), "=r" (z) : "r" (w) ); } Adding the early clobber "&" forces the compiler not to use the "a" register for anything other than operand 0. Operand 2 will therefore get its own register with its own copy of "w". The result for "z" in example16() is 1. Operand Aliases Extended asm specifies operands in assembly strings with a percent '%' followed by the operand number. For example, "%0" references operand 0 or the output item "=&a" (w) in function example16() shown above. Extended asm also supports operand aliasing, which allows use of a symbolic name instead of a number for specifying operands. An example follows: void example17() { int w=1, z=0; asm( "movl $1, %[output1]\n" "addl %[input], %[output1]\n" "movl %[input], %[output2]" : [output1] "=&a" (w), [output2] "=r" (z) : [input] "r" (w)); } In example17(), "%[output1]" is an alias for "%0", "%[output2]" is an alias for "%1", and "%[input]" is an alias for "%2". Aliases and numeric references can be mixed, as shown in the following example: void example18() { int w=1, z=0; asm( "movl $1, %[output1]\n" "addl %[input], %0\n" "movl %[input], %[output2]" : [output1] "=&a" (w), [output2] "=r" (z) : [input] "r" (w)); } In example18(), "%0" and "%[output1]" both represent the output operand. Assembly String Modifiers Special character sequences in the assembly string affect the way the assembly is generated by the compiler. For example, the "%" is an escape sequence for specifying an operand, "%%" produces a percent for hard coded registers, and "\n" specifies a new line. Table 12.5, “Assembly String Modifier Characters”summarizes these modifiers, known as Assembly String Modifiers. Table 12.5. Assembly String Modifier Characters Modifier Description \ Same as \ in printf format strings.PGI® User’s Guide 146 Modifier Description %* Adds a '*' in the assembly string. %% Adds a '%' in the assembly string. %A Adds a '*' in front of an operand in the assembly string. (For example, %A0 adds a '*' in front of operand 0 in the assembly output.) %B Produces the byte op code suffix for this operand. (For example, %b0 produces 'b' on x86 and x86_64.) %L Produces the word op code suffix for this operand. (For example, %L0 produces 'l' on x86 and x86_64.) %P If producing Position Independent Code (PIC), the compiler adds the PIC suffix for this operand. (For example, %P0 produces @PLT on x86 and x86_64.) %Q Produces a quad word op code suffix for this operand if is supported by the target. Otherwise, it produces a word op code suffix. (For example, %Q0 produces 'q' on x86_64 and 'l' on x86.) %S Produces 's' suffix for this operand. (For example, %S0 produces 's' on x86 and x86_64.) %T Produces 't' suffix for this operand. (For example, %S0 produces 't' on x86 and x86_64.) %W Produces the half word op code suffix for this operand. (For example, %W0 produces 'w' on x86 and x86_64.) %a Adds open and close parentheses ( ) around the operand. %b Produces the byte register name for an operand. (For example, if operand 0 is in register 'a', then %b0 will produce '%al'.) %c Cuts the '$' character from an immediate operand. %k Produces the word register name for an operand. (For example, if operand 0 is in register 'a', then %k0 will produce '%eax'.) %q Produces the quad word register name for an operand if the target supports quad word. Otherwise, it produces a word register name. (For example, if operand 0 is in register 'a', then %q0 produces %rax on x86_64 or %eax on x86.) %w Produces the half word register name for an operand. (For example, if operand 0 is in register 'a', then %w0 will produce '%ax'.) %z Produces an op code suffix based on the size of an operand. (For example, 'b' for byte, 'w' for half word, 'l' for word, and 'q' for quad word.) %+ %C %D %F %O %X %f %h %l %n %s %y Not Supported. These modifiers begin with either a backslash "\" or a percent "%".Chapter 12. C/C++ Inline Assembly and Intrinsics 147 The modifiers that begin with a backslash "\" (e.g., "\n") have the same effect as they do in a printf format string. The modifiers that are preceded with a "%" are used to modify a particular operand. These modifiers begin with either a backslash "\" or a percent "%" For example, "%b0" means, "produce the byte or 8 bit version of operand 0". If operand 0 is a register, it will produce a byte register such as %al, %bl, %cl, and so on. Consider this example: void example19() { int a = 1; int *p = &a; asm ("add%z0 %q1, %a0" : "=&p" (p) : "r" (a), "0" (p) ); } On an x86 target, the compiler produces the following instruction for the asm string shown in the preceding example: addl %ecx, (%eax) The "%z0" modifier produced an 'l' (lower-case 'L') suffix because the size of pointer p is 32 bits on x86. The "%q1" modifier produced the word register name for variable a. The "%a0" instructs the compiler to add parentheses around operand 0, hence "(%eax)". On an x86_64 target, the compiler produces the following instruction for the above asm string shown in the preceding example: addq %rcx, (%rax) The "%z0" modifier produced a 'q' suffix because the size of pointer p is 64-bit on x86_64. Because x86_64 supports quad word registers, the "%q1" modifier produced the quad word register name (%rax) for variable a. Extended Asm Macros As with traditional inline assembly, described in“Inline Assembly,” on page 133, extended asm can be used in a macro. For example, you can use the following macro to access the runtime stack pointer. #define GET_SP(x) \ asm("mov %%sp, %0": "=m" (##x):: "%sp" ); void example20() { void * stack_pointer; GET_SP(stack_pointer); } The GET_SP macro assigns the value of the stack pointer to whatever is inserted in its argument (for example, stack_pointer). Another "C" extension known as statement expressions is used to write the GET_SP macro another way: #define GET_SP2 ({ \ void *my_stack_ptr; \ asm("mov %%sp, %0": "=m" (my_stack_ptr) :: "%sp" ); \PGI® User’s Guide 148 my_stack_ptr; \ }) void example21() { void * stack_pointer = GET_SP2; } The statement expression allows a body of code to evaluate to a single value. This value is specified as the last instruction in the statement expression. In this case, the value is the result of the asm statement, my_stack_ptr. By writing an asm macro with a statement expression, the asm result may be assigned directly to another variable (for example, void * stack_pointer = GET_SP2) or included in a larger expression, such as: void * stack_pointer = GET_SP2 - sizeof(long). Which style of macro to use depends on the application. If the asm statement needs to be a part of an expression, then a macro with a statement expression is a good approach. Otherwise, a traditional macro, like GET_SP(x), will probably suffice. Intrinsics Inline intrinsic functions map to actual x86 or x64 machine instructions. Intrinsics are inserted inline to avoid the overhead of a function call. The compiler has special knowledge of intrinsics, so with use of intrinsics, better code may be generated as compared to extended inline assembly code. The PGI Workstation version 7.0 or higher compiler intrinsics library implements MMX, SSE, SS2, SSE3, SSSE3, SSE4a, and ABM instructions. The intrinsic functions are available to C and C++ programs on Linux and Windows. Unlike most functions which are in libraries, intrinsics are implemented internally by the compiler. A program can call the intrinsic functions from C/C++ source code after including the corresponding header file. The intrinsics are divided into header files as follows: Table 12.6. Intrinsic Header File Organization Instructions Header File MMX mmintrin.h SSE xmmintrin.h SSE2 emmintrin.h SSE3 pmmintrin.h SSSE3 tmmintrin.h SSE4a ammintrin.h ABM intrin.h The following is a simple example program that calls XMM intrinsics. #include int main(){ __m128 __A, __B, result;Chapter 12. C/C++ Inline Assembly and Intrinsics 149 __A = _mm_set_ps(23.3, 43.7, 234.234, 98.746); __B = _mm_set_ps(15.4, 34.3, 4.1, 8.6); result = _mm_add_ps(__A,__B); return 0; }150151 Chapter 13. Fortran, C and C++ Data Types This chapter describes the scalar and aggregate data types recognized by the PGI Fortran, C, and C++ compilers, the format and alignment of each type in memory, and the range of values each type can have on x86 or x64 processor-based systems running a 32-bit operating system. For more information on x86- specific data representation, refer to the System V Application Binary Interface, Processor Supplement, listed in “Related Publications,” on page xxvii. This chapter specifically does not address x64 processor-based systems running a 64-bit operating system, because the application binary interface (ABI) for those systems is still evolving. For the latest version of the ABI, refer to http://www.x86-64.org/abi.pdf. Fortran Data Types Fortran Scalars A scalar data type holds a single value, such as the integer value 42 or the real value 112.6. The next table lists scalar data types, their size, format and range. Table 13.2, “Real Data Type Ranges,” on page 152 shows the range and approximate precision for Fortran real data types. Table 13.3, “Scalar Type Alignment,” on page 152 shows the alignment for different scalar data types. The alignments apply to all scalars, whether they are independent or contained in an array, a structure or a union. Table 13.1. Representation of Fortran Data Types Fortran Data Type Format Range INTEGER 2's complement integer -2 31 to 2 31 -1 INTEGER*2 2's complement integer -32768 to 32767 INTEGER*4 2's complement integer INTEGER*8 2's complement integer -2 63 to 2 63 -1 LOGICAL 32-bit value true or false LOGICAL*1 8-bit value true or falsePGI® User’s Guide 152 Fortran Data Type Format Range LOGICAL*2 16-bit value true or false LOGICAL*4 32-bit value true or false LOGICAL*8 64-bit value true or false BYTE 2's complement -128 to 127 REAL Single-precision floating point 10 -37 to 1038 (1) REAL*4 Single-precision floating point 10 -37 to 1038 (1) REAL*8 Double-precision floating point 10 -307 to 1038 (1) DOUBLE PRECISION Double-precision floating point 10 -307 to 1038 (1) COMPLEX Single-precision floating point 10 -37 to 1038 (1) DOUBLE COMPLEX Double-precision floating point 10 -307 to 1038 (1) COMPLEX*16 Double-precision floating point 10 -307 to 1038 (1) CHARACTER*n Sequence of n bytes (1) Approximate value The logical constants .TRUE. and .FALSE. are all ones and all zeroes, respectively. Internally, the value of a logical variable is true if the least significant bit is one and false otherwise. When the option –Munixlogical is set, a logical variable with a non-zero value is true and with a zero value is false. Table 13.2. Real Data Type Ranges Data Type Binary Range Decimal Range Digits of Precision REAL -2 -126 to 2 128 10 -37 to 1038 (1) 7-8 REAL*8 -2 -1022 to 2 1024 10 -307 to 1038 (1) 15-16 Table 13.3. Scalar Type Alignment This Type... ...Is aligned on this size boundary LOGICAL*1 1-byte LOGICAL*2 2-byte LOGICAL*4 4-byte LOGICAL*8 8-byte BYTE 1-byteChapter 13. Fortran, C and C++ Data Types 153 This Type... ...Is aligned on this size boundary INTEGER*2 2-byte INTEGER*4 4-byte INTEGER*8 8-byte REAL*4 4-byte REAL*8 8-byte COMPLEX*8 4-byte COMPLEX*16 8-byte FORTRAN 77 Aggregate Data Type Extensions The PGF77 compiler supports de facto standard extensions to FORTRAN 77 that allow for aggregate data types. An aggregate data type consists of one or more scalar data type objects. You can declare the following aggregate data types: array consists of one or more elements of a single data type placed in contiguous locations from first to last. structure is a structure that can contain different data types. The members are allocated in the order they appear in the definition but may not occupy contiguous locations. union is a single location that can contain any of a specified set of scalar or aggregate data types. A union can have only one value at a time. The data type of the union member to which data is assigned determines the data type of the union after that assignment. The alignment of an array, a structure or union (an aggregate) affects how much space the object occupies and how efficiently the processor can address members. Arrays use the alignment of their members. Array types align according to the alignment of the array elements. For example, an array of INTEGER*2 data aligns on a 2-byte boundary. Structures and Unions align according to the alignment of the most restricted data type of the structure or union. In the next example, the union aligns on a 4-byte boundary since the alignment of c, the most restrictive element, is four. STRUCTURE /astr/ UNION MAP INTEGER*2 a ! 2 bytes END MAP PGI® User’s Guide 154 MAP BYTE b ! 1 byte END MAP MAP INTEGER*4 c ! 4 bytes END MAP END UNION END STRUCTURE Structure alignment can result in unused space called padding. Padding between members of the structure is called internal padding. Padding between the last member and the end of the space is called tail padding. The offset of a structure member from the beginning of the structure is a multiple of the member’s alignment. For example, since an INTEGER*2 aligns on a 2-byte boundary, the offset of an INTEGER*2 member from the beginning of a structure is a multiple of two bytes. Fortran 90 Aggregate Data Types (Derived Types) The Fortran 90 standard added formal support for aggregate data types. The TYPE statement begins a derived type data specification or declares variables of a specified user-defined type. For example, the following would define a derived type ATTENDEE: TYPE ATTENDEE CHARACTER(LEN=30) NAME CHARACTER(LEN=30) ORGANIZATION CHARACTER (LEN=30) EMAIL END TYPE ATTENDEE In order to declare a variable of type ATTENDEE and access the contents of such a variable, code such as the following would be used: TYPE (ATTENDEE) ATTLIST(100) . . . ATTLIST(1)%NAME = ‘JOHN DOE’ C and C++ Data Types C and C++ Scalars Table 13.4, “C/C++ Scalar Data Types”lists C and C++ scalar data types, providing their size and format. The alignment of a scalar data type is equal to its size. Table 13.5, “Scalar Alignment,” on page 155 shows scalar alignments that apply to individual scalars and to scalars that are elements of an array or members of a structure or union. Wide characters are supported (character constants prefixed with an L). The size of each wide character is 4 bytes. Table 13.4. C/C++ Scalar Data Types Data Type Size (bytes) Format Range unsigned char 1 ordinal 0 to 255 [signed] char 1 2's complement integer -128 to 127 unsigned short 2 ordinal 0 to 65535 [signed] short 2 2's complement integer -32768 to 32767Chapter 13. Fortran, C and C++ Data Types 155 Data Type Size (bytes) Format Range unsigned int 4 ordinal 0 to 2 32 -1 [signed] int 4 2's complement integer -2 31 to 2 31 -1 [signed] long [int] (32-bit operating systems and win64) 4 2's complement integer -2 31 to 2 31 -1 [signed] long [int] (linux86- 64 and sua64) 8 2's complement integer -2 63 to 2 63 -1 unsigned long [int] (32-bit operating systems and win64) 4 ordinal 0 to 2 32 -1 unsigned long [int] (linux86- 64 and sua64) 8 ordinal 0 to 2 64 -1 [signed] long long [int] 8 2's complement integer -2 63 to 2 63 -1 unsigned long long [int] 8 ordinal 0 to 2 64 -1 float 4 IEEE single-precision floating-point 10 -37 to 10 38 (1) double 8 IEEE double-precision floating-point 10 -307 to 10 308 (1) long double 8 IEEE double-precision floating-point 10 -307 to 10 308 (1) bit field (2) (unsigned value) 1 to 32 bits ordinal 0 to 2 size -1, where size is the number of bits in the bit field bit field (2) (signed value) 1 to 32 bits 2's complement integer -2 size-1 to 2 size-1 -1, where size is the number of bits in the bit field pointer 4 address 0 to 2 32 -1 enum 4 2's complement integer -2 31 to 2 31 -1 (1) Approximate value (2) Bit fields occupy as many bits as you assign them, up to 4 bytes, and their length need not be a multiple of 8 bits (1 byte) Table 13.5. Scalar Alignment Data Type Alignment on this size boundary char 1-byte boundary, signed or unsigned. short 2-byte boundary, signed or unsigned. int 4-byte boundary, signed or unsigned.PGI® User’s Guide 156 Data Type Alignment on this size boundary enum 4-byte boundary. pointer 4-byte boundary. float 4-byte boundary. double 8-byte boundary. long double 8-byte boundary. long [int] 32-bit on Win64 4-byte boundary, signed or unsigned. long [int] linux86-64, sua64 8-byte boundary, signed or unsigned. long long [int] 8-byte boundary, signed or unsigned. C and C++ Aggregate Data Types An aggregate data type consists of one or more scalar data type objects. You can declare the following aggregate data types: array consists of one or more elements of a single data type placed in contiguous locations from first to last. class (C++ only) is a class that defines an object and its member functions. The object can contain fundamental data types or other aggregates including other classes. The class members are allocated in the order they appear in the definition but may not occupy contiguous locations. struct is a structure that can contain different data types. The members are allocated in the order they appear in the definition but may not occupy contiguous locations. When a struct is defined with member functions, its alignment rules are the same as those for a class. union is a single location that can contain any of a specified set of scalar or aggregate data types. A union can have only one value at a time. The data type of the union member to which data is assigned determines the data type of the union after that assignment. Class and Object Data Layout Class and structure objects with no virtual entities and with no base classes, that is just direct data field members, are laid out in the same manner as C structures. The following section describes the alignment and size of these C-like structures. C++ classes (and structures as a special case of a class) are more difficult to describe. Their alignment and size is determined by compiler generated fields in addition to user-specified fields. The following paragraphs describe how storage is laid out for more general classes. The user is warned that the alignment and size of a class (or structure) is dependent on the existence and placement of direct and virtual base classes and of virtual function information. The information that follows is for informational purposes only, reflects the current implementation, and is subject to change. Do not make assumptions about the layout of complex classes or structures. All classes are laid out in the same general way, using the following pattern (in the sequence indicated):Chapter 13. Fortran, C and C++ Data Types 157 • First, storage for all of the direct base classes (which implicitly includes storage for non-virtual indirect base classes as well): • When the direct base class is also virtual, only enough space is set aside for a pointer to the actual storage, which appears later. • In the case of a non-virtual direct base class, enough storage is set aside for its own non-virtual base classes, its virtual base class pointers, its own fields, and its virtual function information, but no space is allocated for its virtual base classes. • Next, storage for the class’s own fields. • Next, storage for virtual function information (typically, a pointer to a virtual function table). • Finally, storage for its virtual base classes, with space enough in each case for its own non-virtual base classes, virtual base class pointers, fields, and virtual function information. Aggregate Alignment The alignment of an array, a structure or union (an aggregate) affects how much space the object occupies and how efficiently the processor can address members. Arrays align according to the alignment of the array elements. For example, an array of short data type aligns on a 2-byte boundary. Structures and Unions align according to the most restrictive alignment of the enclosing members. For example the union un1 below aligns on a 4-byte boundary since the alignment of c, the most restrictive element, is four: union un1 { short a; /* 2 bytes */ char b; /* 1 byte */ int c; /* 4 bytes */ }; Structure alignment can result in unused space, called padding. Padding between members of a structure is called internal padding. Padding between the last member and the end of the space occupied by the structure is called tail padding. Figure 13.1, “Internal Padding in a Structure,” on page 157, illustrates structure alignment. Consider the following structure: struct strc1 { char a; /* occupies byte 0 */ short b; /* occupies bytes 2 and 3 */ char c; /* occupies byte 4 */ int d; /* occupies bytes 8 through 11 */ }; Figure 13.1. Internal Padding in a StructurePGI® User’s Guide 158 Figure 13.2, “Tail Padding in a Structure,” on page 158, shows how tail padding is applied to a structure aligned on a doubleword (8 byte) boundary. struct strc2{ int m1[4]; /* occupies bytes 0 through 15 */ double m2; /* occupies bytes 16 through 23 */ short m3; /* occupies bytes 24 and 25 */ } st; Bit-field Alignment Bit-fields have the same size and alignment rules as other aggregates, with several additions to these rules: • Bit-fields are allocated from right to left. • A bit-field must entirely reside in a storage unit appropriate for its type. Bit-fields never cross unit boundaries. • Bit-fields may share a storage unit with other structure/union members, including members that are not bitfields. • Unnamed bit-field's types do not affect the alignment of a structure or union. • Items of [signed/unsigned] long long type may not appear in field declarations on 32-bit systems. Figure 13.2. Tail Padding in a Structure Other Type Keywords in C and C++ The void data type is neither a scalar nor an aggregate. You can use void or void* as the return type of a function to indicate the function does not return a value, or as a pointer to an unspecified data type, respectively. The const and volatile type qualifiers do not in themselves define data types, but associate attributes with other types. Use const to specify that an identifier is a constant and is not to be changed. Use volatile to prevent optimization problems with data that can be changed from outside the program, such as memory#mapped I/O buffers.159 Chapter 14. C++ Name Mangling Name mangling transforms the names of entities so that the names include information on aspects of the entity’s type and fully qualified name. This ability is necessary since the intermediate language into which a program is translated contains fewer and simpler name spaces than there are in the C++ language; specifically: • Overloaded function names are not allowed in the intermediate language. • Classes have their own scopes in C++, but not in the generated intermediate language. For example, an entity x from inside a class must not conflict with an entity x from the file scope. • External names in the object code form a completely flat name space. The names of entities with external linkage must be projected onto that name space so that they do not conflict with one another. A function f from a class A, for example, must not have the same external name as a function f from class B. • Some names are not names in the conventional sense of the word, they're not strings of alphanumeric characters, for example: operator=. There are two main problems here: 1. Generating external names that will not clash. 2. Generating alphanumeric names for entities with strange names in C++. Name mangling solves these problems by generating external names that will not clash, and alphanumeric names for entities with strange names in C++. It also solves the problem of generating hidden names for some behind-the-scenes language support in such a way that they match up across separate compilations. You see mangled names if you view files that are translated by PGC++, and you do not use tools that demangle the C++ names. Intermediate files that use mangled names include the assembly and object files created by the pgcpp command. To view demangled names, use the tool pgdecode, which takes input from stdin. prompt> pgdecode g__1ASFf A::g(float) The name mangling algorithm for the PGC++ compiler is the same as that for cfront, and, except for a few minor details, also matches the description in Section 7.2, Function Name Encoding, of The Annotated C++ Reference Manual (ARM). Refer to the ARM for a complete description of name mangling.PGI® User’s Guide 160 Types of Mangling The following entity names are mangled: • Function names including non-member function names are mangled, to deal with overloading. Names of functions with extern "C" linkage are not mangled. • Mangled function names have the function name followed by __ followed by F followed by the mangled description of the types of the parameters of the function. If the function is a member function, the mangled form of the class name precedes the F. If the member function is static, an S also precedes the F. int f(float); // f__Ff class A int f(float); // f__1AFf static int g(float); // g__1ASFf ; • Special and operator function names, like constructors and operator=(). The encoding is similar to that for normal functions, but a coded name is used instead of the routine name: class A int operator+(float); // __pl__1Aff A(float); // __ct__1Aff ; int operator+(A, float); // __pl__F1Af • Static data member names. The mangled form is the member name followed by __ followed by the mangled form of the class name: class A static int i; // i__1A ; • Names of variables generated for virtual function tables. These have names like vtblmangled-classname or vtblmangled-base-class-namemangled-class-name. • Names of variables generated to contain runtime type information. These have names like Ttypeencoding and TIDtype-encoding. Mangling Summary This section lists some of the C++ entities that are mangled and provides some details on the mangling algorithm. For more details, refer to The Annotated C++ Reference Manual. Type Name Mangling Using PGC++, each type has a corresponding mangled encoding. For example, a class type is represented as the class name preceded by the number of characters in the class name, as in 5abcde for abcde. Simple types are encoded as lower-case letters, as in i for int or f for float. Type modifiers and declarators are encoded as upper-case letters preceding the types they modify, as in U for unsigned or P for pointer.Chapter 14. C++ Name Mangling 161 Nested Class Name Mangling Nested class types are encoded as a Q followed by a digit indicating the depth of nesting, followed by a _, followed by the mangled-form names of the class types in the fully-qualified name of the class, from outermost to innermost: class A class B // Q2_1A1B ; ; Local Class Name Mangling The name of the nested class itself is mangled to the form described above with a prefix __, which serves to make the class name distinct from all user names. Local class names are encoded as L followed by a number (which has no special meaning; it’s just an identifying number assigned to the class) followed by __ followed by the mangled name of the class (this is not in the ARM, and cfront encodes local class names slightly differently): void f() class A // L1__1A} ; ; This form is used when encoding the local class name as a type. It’s not necessary to mangle the name of the local class itself unless it's also a nested class. Template Class Name Mangling Template classes have mangled names that encode the arguments of the template: template class abc ; abc x; abc__pt__3_ii This describes two template arguments of type int with the total length of template argument list string, including the underscore, and a fixed string, indicates parameterized type as well, the name of the class template.162163 Chapter 15. Command-Line Options Reference A command-line option allows you to specify specific behavior when a program is compiled and linked. Compiler options perform a variety of functions, such as setting compiler characteristics, describing the object code to be produced, controlling the diagnostic messages emitted, and performing some preprocessor functions. Most options that are not explicitly set take the default settings. This reference chapter describes the syntax and operation of each compiler option. For easy reference, the options are arranged in alphabetical order. For an overview and tips on which options are best for which tasks, refer to Chapter 2, “Using Command Line Options,” on page 15, which also provides summary tables of the different options. This chapter uses the following notation: [item] Square brackets indicate that the enclosed item is optional. {item | item} Braces indicate that you must select one and only one of the enclosed items. A vertical bar (|) separates the choices. ... Horizontal ellipses indicate that zero or more instances of the preceding item are valid. PGI Compiler Option Summary The following tables include all the PGI compiler options that are not language-specific. The options are separated by category for easier reference. For a complete description of each option, see the detailed information later in this chapter. Build-Related PGI Options The options included in the following table are the ones you use when you are initially building your program or application.PGI® User’s Guide 164 Table 15.1. PGI Build-Related Compiler Options Option Description –# Display invocation information. –### Show but do not execute the driver commands (same as –dryrun). –c Stops after the assembly phase and saves the object code in filename.o. –D Defines a preprocessor macro. –d Prints additional information from the preprocessor. –dryrun Show but do not execute driver commands. –E Stops after the preprocessing phase and displays the preprocessed file on the standard output. –F Stops after the preprocessing phase and saves the preprocessed file in filename.f (this option is only valid for the PGI Fortran compilers). --flagcheck Simply return zero status if flags are correct. –flags Display valid driver options. –fpic (Linux only) Generate position-independent code. –fPIC (Linux only) Equivalent to –fpic. –G (Linux only) Passed to the linker. Instructs the linker to produce a shared object file. –g77libs (Linux only) Allow object files generated by g77 to be linked into PGI main programs. –help Display driver help message. –I Adds a directory to the search path for #include files. –i2, –i4 and –i8 –i2: Treat INTEGER variables as 2 bytes. –i4: Treat INTEGER variables as 4 bytes. –i8: Treat INTEGER and LOGICAL variables as 8 bytes and use 64- bits for INTEGER*8 operations. –K Requests special compilation semantics with regard to conformance to IEEE 754. --keeplnk If the compiler generates a temporary indirect file for a long linker command, preserves the temporary file instead of deleting it. –L Specifies a library directory. –l Loads a library. –m Displays a link map on the standard output. –M Selects variations for code generation and optimization. –mcmodel=mediumChapter 15. Command-Line Options Reference 165 Option Description (–tp k8-64 and –tp p7-64 targets only) Generate code which supports the medium memory model in the linux86-64 environment. –module (F90/F95/HPF only) Save/search for module files in directory . –mp[=align,[no]numa] Interpret and process user-inserted shared-memory parallel programming directives (see Chapters 5 and 6). –noswitcherror Ignore unknown command line switches after printing an warning message. –o Names the object file. –pc (–tp px/p5/p6/piii targets only) Set precision globally for x87 floating-point calculations; must be used when compiling the main program. may be one of 32, 64 or 80. –pg Instrument the generated executable to produce a gprof-style gmon.out sample-based profiling trace file (–qp is also supported, and is equivalent). –pgf77libs Append PGF77 runtime libraries to the link line. –pgf90libs Append PGF90/PGF95 runtime libraries to the link line. –Q Selects variations for compiler steps. –R (Linux only) Passed to the Linker. Hard code into the search path for shared object files. –r Creates a relocatable object file. –r4 and –r8 –r4: Interpret DOUBLE PRECISION variables as REAL. –r8: Interpret REAL variables as DOUBLE PRECISION. –rc file Specifies the name of the driver's startup file. –s Strips the symbol-table information from the object file. –S Stops after the compiling phase and saves the assembly–language code in filename.s. –shared (Linux only) Passed to the linker. Instructs the linker to generate a shared object file. Implies –fpic. –show Display driver's configuration parameters after startup. –silent Do not print warning messages. –soname Pass the soname option and its argument to the linker. –time Print execution times for the various compilation steps. –tp [,target...] Specify the type(s) of the target processor(s).PGI® User’s Guide 166 Option Description –u Initializes the symbol table with , which is undefined for the linker. An undefined symbol triggers loading of the first member of an archive library. –U Undefine a preprocessor macro. –V[release_number] Displays the version messages and other information, or allows invocation of a version of the compiler other than the default. –v Displays the compiler, assembler, and linker phase invocations. –W Passes arguments to a specific phase. –w Do not print warning messages. PGI Debug-Related Compiler Options The options included in the following table are the ones you typically use when you are debugging your program or application. Table 15.2. PGI Debug-Related Compiler Options Option Description –C Exposes Ansi warnings only. –c Instrument the generated executable to perform array bounds checking at runtime. –E Stops after the preprocessing phase and displays the preprocessed file on the standard output. --flagcheck Simply return zero status if flags are correct. –flags Display valid driver options. –g Includes debugging information in the object module. –gopt Includes debugging information in the object module, but forces assembly code generation identical to that obtained when is not present on the command line. –K Requests special compilation semantics with regard to conformance to IEEE 754. --keeplnk If the compiler generates a temporary indirect file for a long linker command, preserves the temporary file instead of deleting it. –M Selects variations for code generation and optimization. –pc (–tp px/p5/p6/piii targets only) Set precision globally for x87 floating-point calculations; must be used when compiling the main program. may be one of 32, 64 or 80. –Mprof=timeChapter 15. Command-Line Options Reference 167 Option Description Instrument the generated executable to produce a gprof-style gmon.out sample-based profiling trace file (–qp is also supported, and is equivalent). PGI Optimization-Related Compiler Options The options included in the following table are the ones you typically use when you are optimizing your program or application code. Table 15.3. Optimization-Related PGI Compiler Options Option Description –fast Generally optimal set of flags for targets that support SSE capability. –fastsse Generally optimal set of flags for targets that include SSE/SSE2 capability. –M Selects variations for code generation and optimization. –mp[=align,[no]numa] Interpret and process user-inserted shared-memory parallel programming directives (see Chapters 5 and 6). –nfast Generally optimal set of flags for the target. Doesn’t use SSE. –O Specifies code optimization level where is 0, 1, 2, 3, or 4. –pc (–tp px/p5/p6/piii targets only) Set precision globally for x87 floating-point calculations; must be used when compiling the main program. may be one of 32, 64 or 80. –Mprof=time Instrument the generated executable to produce a gprof-style gmon.out sample-based profiling trace file (-qp is also supported, and is equivalent). PGI Linking and Runtime-Related Compiler Options The options included in the following table are the ones you typically use to define parameters related to linking and running your program or application code. Table 15.4. Linking and Runtime-Related PGI Compiler Options Option Description –byteswapio (Fortran only) Swap bytes from big-endian to little-endian or vice versa on input/output of unformatted data –fpic (Linux only) Generate position-independent code. –fPIC (Linux only) Equivalent to –fpic. –G (Linux only) Passed to the linker. Instructs the linker to produce a shared object file.PGI® User’s Guide 168 Option Description –g77libs (Linux only) Allow object files generated by g77 to be linked into PGI main programs. –i2, –i4 and –i8 –i2: Treat INTEGER variables as 2 bytes. –i4: Treat INTEGER variables as 4 bytes. –i8: Treat INTEGER and LOGICAL variables as 8 bytes and use 64- bits for INTEGER*8 operations. –K Requests special compilation semantics with regard to conformance to IEEE 754. –M Selects variations for code generation and optimization. –mcmodel=medium (–tp k8-64 and –tp p7-64 targets only) Generate code which supports the medium memory model in the linux86-64 environment. –shared (Linux only) Passed to the linker. Instructs the linker to generate a shared object file. Implies –fpic. –soname Pass the soname option and its argument to the linker. –tp [,target...] Specify the type(s) of the target processor(s). C and C++ Compiler Options There are a large number of compiler options specific to the PGCC and PGC++ compilers, especially PGC++. The next table lists several of these options, but is not exhaustive. For a complete list of available options, including an exhaustive list of PGC++ options, use the –help command-line option. For further detail on a given option, use –help and specify the option explicitly. The majority of these options are related to building your program or application. Table 15.5. C and C++ -specific Compiler Options Option Description –A (pgcpp only) Accept proposed ANSI C++, issuing errors for non-conforming code. –a (pgcpp only) Accept proposed ANSI C++, issuing warnings for non-conforming code. --[no_]alternative_tokens (pgcpp only) Enable/disable recognition of alternative tokens. These are tokens that make it possible to write C++ without the use of the , , [, ], #, &, and ^ and characters. The alternative tokens include the operator keywords (e.g., and, bitand, etc.) and digraphs. The default is -–no_alternative_tokens. –B Allow C++ comments (using //) in C source. –b (pgcpp only) Compile with cfront 2.1 compatibility. This accepts constructs and a version of C++ that is not partChapter 15. Command-Line Options Reference 169 Option Description of the language definition but is accepted by cfront. EDG option. –b3 (pgcpp only) Compile with cfront 3.0 compatibility. See –b above. --[no_]bool (pgcpp only) Enable or disable recognition of bool. The default value is ––bool. – –[no_]builtin Do/don’t compile with math subroutine builtin support, which causes selected math library routines to be inlined. The default is ––builtin. --cfront_2.1 (pgcpp only) Enable compilation of C++ with compatibility with cfront version 2.1. --cfront_3.0 (pgcpp only) Enable compilation of C++ with compatibility with cfront version 3.0. --compress_names (pgcpp only) Create a precompiled header file with the name filename. --dependencies (see –M) (pgcpp only) Print makefile dependencies to stdout. --dependencies_to_file filename (pgcpp only) Print makefile dependencies to file filename. --display_error_number (pgcpp only) Display the error message number in any diagnostic messages that are generated. --diag_error tag (pgcpp only) Override the normal error severity of the specified diagnostic messages. --diag_remark tag (pgcpp only) Override the normal error severity of the specified diagnostic messages. --diag_suppress tag (pgcpp only) Override the normal error severity of the specified diagnostic messages. --diag_warning tag (pgcpp only) Override the normal error severity of the specified diagnostic messages. -e (pgcpp only) Set the C++ front-end error limit to the specified . --[no_]exceptions (pgcpp only) Disable/enable exception handling support. The default is ––exceptions ––gnu_extensions (pgcpp only) Allow GNU extensions like “include next” which are required to compile Linux system header files. --[no]llalign (pgcpp only) Do/don’t align longlong integers on integer boundaries. The default is ––llalign. –M Generate make dependence lists. –MD Generate make dependence lists.PGI® User’s Guide 170 Option Description –MD,filename (pgcpp only) Generate make dependence lists and print them to file filename. --optk_allow_dollar_in_id_chars (pgcpp only) Accept dollar signs in identifiers. –P Stops after the preprocessing phase and saves the preprocessed file in filename.i. -+p (pgcpp only) Disallow all anachronistic constructs. cfront option --pch (pgcpp only) Automatically use and/or create a precompiled header file. --pch_dir directoryname (pgcpp only) The directory dirname in which to search for and/or create a precompiled header file. --[no_]pch_messages (pgcpp only) Enable/ disable the display of a message indicating that a precompiled header file was created or used. --preinclude= (pgcpp only) Specify file to be included at the beginning of compilation so you can set system-dependent macros, types, and so on. -suffix (see–P ) (pgcpp only) Use with –E, –F, or –P to save intermediate file in a file with the specified suffix. –t Control instantiation of template functions. EDG option --use_pch filename (pgcpp only) Use a precompiled header file of the specified name as part of the current compilation. --[no_]using_std (pgcpp only) Enable/disable implicit use of the std namespace when standard header files are included. –X (pgcpp only) Allow $ in names. Generic PGI Compiler Options The following descriptions are for the PGI options. For easy reference, the options are arranged in alphabetical order. For a list of options by tasks, refer to Chapter 2, “Using Command Line Options,” on page 15. –# Displays the invocations of the compiler, assembler and linker. Default: The compiler does not display individual phase invocations. Usage:The following command-line requests verbose invocation information. $ pgf95 -# prog.f Description: The –# option displays the invocations of the compiler, assembler and linker. These invocations are command-lines created by the driver from your command-line input and the default value.Chapter 15. Command-Line Options Reference 171 Related options:–Minfo, –V, –v. –### Displays the invocations of the compiler, assembler and linker, but does not execute them. Default: The compiler does not display individual phase invocations. Usage:The following command-line requests verbose invocation information. $ pgf95 -### myprog.f Description: Use the –### option to display the invocations of the compiler, assembler and linker but not to execute them. These invocations are command lines created by the compiler driver from the PGIRC files and the command-line options. Related options: –#, –dryrun, –Minfo, –V –Bdynamic Compiles for and links to the DLL version of the PGI runtime libraries. Default: The compiler uses static libraries. Usage:You can create the DLL obj1.dll and its import library obj1.lib using the following series of commands: % pgf95 -Bdynamic -c object1.f % pgf95 -Mmakedll object1.obj -o obj1.dll Then compile the main program using this command: $ pgf95 -# prog.f For a complete example, refer to Example 7.1, “Build a DLL: Fortran,” on page 82. Description: Use this option to compile for and link to the DLL version of the PGI runtime libraries. This flag is required when linking with any DLL built by the PGI compilers. This flag corresponds to the /MD flag used by Microsoft’s cl compilers. Note On Windows, -Bdynamic must be used for both compiling and linking. When you use the PGI compiler flag –Bdynamic to create an executable that links to the DLL form of the runtime, the executable built is smaller than one built without –Bdynamic. The PGI runtime DLLs, however, must be available on the system where the executable is run. The –Bdynamic flag must be used when an executable is linked against a DLL built by the PGI compilers. Related options:–Bstatic, –Mdll –Bstatic Compiles for and links to the static version of the PGI runtime libraries.PGI® User’s Guide 172 Default: The compiler uses static libraries. Usage:The following command line explicitly compiles for and links to the static version of the PGI runtime libraries: % pgf95 -Bstatic -c object1.f Description: You can use this option to explicitly compile for and link to the static version of the PGI runtime libraries. Note On Windows, -Bstatic must be used for both compiling and linking. For more information on using static libraries on Windows, refer to “Creating and Using Static Libraries on Windows,” on page 79. Related options:–Bdynamic, –Mdll –byteswapio Swaps the byte-order of data in unformatted Fortran data files on input/output. Default: The compiler does not byte-swap data on input/output. Usage: The following command-line requests that byte-swapping be performed on input/output. $ pgf95 -byteswapio myprog.f Description: Use the –byteswapio option to swap the byte-order of data in unformatted Fortran data files on input/output. When this option is used, the order of bytes is swapped in both the data and record control words; the latter occurs in unformatted sequential files. You can use option to convert big-endian format data files produced by most RISC workstations and high-end servers to the little-endian format used on x86 or x64 systems on the fly during file reads/writes. This option assumes that the record layouts of unformatted sequential access and direct access files are the same on the systems. It further assumes that the IEEE representation is used for floating-point numbers. In particular, the format of unformatted data files produced by PGI Fortran compilers is identical to the format used on Sun and SGI workstations; this format allows you to read and write unformatted Fortran data files produced on those platforms from a program compiled for an x86 or x64 platform using the –byteswapio option. Related options: –C Enables array bounds checking. Default: The compiler does not enable array bounds checking. Usage: In this example, the compiler instruments the executable produced from myprog.f to perform array bounds checking at runtime:Chapter 15. Command-Line Options Reference 173 $ pgf95 -C myprog.f Description: Use this option to enable array bounds checking. If an array is an assumed size array, the bounds checking only applies to the lower bound. If an array bounds violation occurs during execution, an error message describing the error is printed and the program terminates. The text of the error message includes the name of the array, the location where the error occurred (the source file and the line number in the source), and information about the out of bounds subscript (its value, its lower and upper bounds, and its dimension). Related options: –Mbounds. –c Halts the compilation process after the assembling phase and writes the object code to a file. Default: The compiler produces an executable file (does not use the –c option). Usage: In this example, the compiler produces the object file myprog.o in the current directory. $ pgf95 -c myprog.f Description: Use the –c option to halt the compilation process after the assembling phase and write the object code to a file. If the input file is filename.f, the output file is filename.o. Related options: –E, –Mkeepasm, –o, and –S. –d Prints additional information from the preprocessor. Default: Syntax: -d[D|I|M|N] -dD Print macros and values from source files. -dI Print include file names. -dM Print macros and values, including predefined and command-line macros. -dN Print macro names from source files. Usage: In the following example, the compiler prints macro names from the source file. $ pgf95 -dN myprog.f Description: Use the -d option to print additional information from the preprocessor.PGI® User’s Guide 174 Related options: –E, –D, –U. –D Creates a preprocessor macro with a given value. Note You can use the –D option more than once on a compiler command line. The number of active macro definitions is limited only by available memory. Syntax: -Dname[=value] Where name is the symbolic name and value is either an integer value or a character string. Default: If you define a macro name without specifying a value, the preprocessor assigns the string 1 to the macro name. Usage: In the following example, the macro PATHLENGTH has the value 256 until a subsequent compilation. If the –D option is not used, PATHLENGTH is set to 128. $ pgf95 -DPATHLENGTH=256 myprog.F The source text in myprog.F is this: #ifndef PATHLENGTH #define PATHLENGTH 128 #endif SUBROUTINE SUB CHARACTER*PATHLENGTH path ... END Use the –D option to create a preprocessor macro with a given value. The value must be either an integer or a character string. You can use macros with conditional compilation to select source text during preprocessing. A macro defined in the compiler invocation remains in effect for each module on the command line, unless you remove the macro with an #undef preprocessor directive or with the –U option. The compiler processes all of the –U options in a command line after processing the –D options. Related options: –U –dryrun Displays the invocations of the compiler, assembler, and linker but does not execute them. Default: The compiler does not display individual phase invocations. Usage: The following command-line requests verbose invocation information. $ pgf95 -dryrun myprog.fChapter 15. Command-Line Options Reference 175 Description: Use the –dryrun option to display the invocations of the compiler, assembler, and linker but not have them executed. These invocations are command lines created by the compiler driver from the PGIRC file and the command-line supplied with –dryrun. Related options: –Minfo, –V, –### –E Halts the compilation process after the preprocessing phase and displays the preprocessed output on the standard output. Default: The compiler produces an executable file. Usage: In the following example the compiler displays the preprocessed myprog.f on the standard output. $ pgf95 -E myprog.f Description: Use the –E option to halt the compilation process after the preprocessing phase and display the preprocessed output on the standard output. Related options: –C, –c, –Mkeepasm, –o, –F, –S. –F Stops compilation after the preprocessing phase. Default: The compiler produces an executable file. Usage: In the following example the compiler produces the preprocessed file myprog.f in the current directory. $ pgf95 -F myprog.F Description: Use the –F option to halt the compilation process after preprocessing and write the preprocessed output to a file. If the input file is filename.F, then the output file is filename.f. Related options: –c,–E, –Mkeepasm, –o, –S –fast Enables vectorization with SEE instructions, cache alignment, and flushz for 64-bit targets. Default: The compiler enables vectorization with SEE instructions, cache alignment, and flushz. Usage: In the following example the compiler produces vector SEE code when targeting a 64-bit machine. $ pgf95 -fast vadd.f95 Description: When you use this option, a generally optimal set of options is chosen for targets that support SSE capability. In addition, the appropriate –tp option is automatically included to enable generation of code optimized for the type of system on which compilation is performed. This option enables vectorization with SEE instructions, cache alignment, and flushz.PGI® User’s Guide 176 Note Auto-selection of the appropriate –tp option means that programs built using the –fastsse option on a given system are not necessarily backward-compatible with older systems. Note C/C++ compilers enable –Mautoinline with –fast. Related options: –nfast, –O, –Munroll, –Mnoframe, –Mscalarsse, –Mvect, –Mcache_align, –tp –fastsse Synonymous with –fast. --flagcheck Causes the compiler to check that flags are correct then exit. Default: The compiler begins a compile without the additional step to first validate that flags are correct. Usage: In the following example the compiler checks that flags are correct, and then exits. $ pgf95 --flagcheck myprog.f Description: Use this option to make the compiler check that flags are correct and then exit. If flags are all correct then the compiler returns a zero status. Related options: –flags Displays driver options on the standard output. Default: The compiler does not display the driver options. Usage: In the following example the user requests information about the known switches. $ pgf95 -flags Description: Use this option to display driver options on the standard output. When you use this option with –v, in addition to the valid options, the compiler lists options that are recognized and ignored. Related options: –#, –###, –v –fpic (Linux only) Generates position-independent code suitable for inclusion in shared object (dynamically linked library) files. Default: The compiler does not generate position-independent code.Chapter 15. Command-Line Options Reference 177 Usage: In the following example the resulting object file, myprog.o, can be used to generate a shared object. $ pgf95 -fpic myprog.f (Linux only) Use the -fpic option to generate position-independent code suitable for inclusion in shared object (dynamically linked library) files. Related options: –shared, –fPIC, –G, –R –fPIC (Linux only) Equivalent to –fpic. Provided for compatibility with other compilers. –G (Linux only) Instructs the linker to produce a shared object file. Default: The compiler does not instruct the linker to produce a shared object file. Usage: In the following example the linker produces a shared object file. $ pgf95 -G myprog.f Description: (Linux only) Use this option to pass information to the linker that instructs the linker to produce a shared object file. Related options: –fpic, –shared, –R –g Instructs the compiler to include symbolic debugging information in the object module. Default: The compiler does not put debugging information into the object module. Usage: In the following example, the object file a.out contains symbolic debugging information. $ pgf95 -g myprog.f Description: Use the –g option to instruct the compiler to include symbolic debugging information in the object module. Debuggers, such as PGDBG, require symbolic debugging information in the object module to display and manipulate program variables and source code. If you specify the –g option on the command-line, the compiler sets the optimization level to –O0 (zero), unless you specify the –O option. For more information on the interaction between the –g and –O options, see the –O entry. Symbolic debugging may give confusing results if an optimization level other than zero is selected. Note Including symbolic debugging information increases the size of the object module. Related options:–OPGI® User’s Guide 178 –gopt Instructs the compiler to include symbolic debugging information in the object file, and to generate optimized code identical to that generated when –g is not specified. Default: The compiler does not put debugging information into the object module. Usage: In the following example, the object file a.out contains symbolic debugging information. $ pgf95 -gopt myprog.f Description: Using –g alters how optimized code is generated in ways that are intended to enable or improve debugging of optimized code. The –gopt option instructs the compiler to include symbolic debugging information in the object file, and to generate optimized code identical to that generated when –g is not specified. Related options: –g77libs (Linux only) Used on the link line, this option instructs the pgf95 driver to search the necessary g77 support libraries to resolve references specific to g77 compiled program units. Note The g77 compiler must be installed on the system on which linking occurs in order for this option to function correctly. Default: The compiler does not search g77 support libraries to resolve references at link time. Usage: The following command-line requests that g77 support libraries be searched at link time: $ pgf95 -g77libs myprog.f g77_object.o Description: (Linux only) Use the –g77libs option on the link line if you are linking g77-compiled program units into a pgf95-compiled main program using the pgf95 driver. When this option is present, the pgf95 driver searches the necessary g77 support libraries to resolve references specific to g77 compiled program units. Related options: –help Used with no other options, –help displays options recognized by the driver on the standard output. When used in combination with one or more additional options, usage information for those options is displayed to standard output. Default: The compiler does not display usage information. Usage: In the following example, usage information for –Minline is printed to standard output. $ pgcc -help -Minline -Minline[=lib:||except:|Chapter 15. Command-Line Options Reference 179 name:|size:|levels:] Enable function inlining lib: Use extracted functions from extlib Inline function func except: Do not inline function func name: Inline function func size: Inline only functions smaller than n levels: Inline n levels of functions -Minline Inline all functions that were extracted In the following example, usage information for –help shows how groups of options can be listed or examined according to function $ pgcc -help -help -help[=groups|asm|debug|language|linker|opt|other| overall|phase|prepro|suffix|switch|target|variable] Show compiler switches Description: Use the –help option to obtain information about available options and their syntax. You can use –help in one of three ways: • Use –help with no parameters to obtain a list of all the available options with a brief one-line description of each. • Add a parameter to –help to restrict the output to information about a specific option. The syntax for this usage is this: -help • Add a parameter to –help to restrict the output to a specific set of options or to a building process. The syntax for this usage is this: -help= The following table lists and describes the subgroups available with –help. –help=groups Gives available groups for help. Table 15.6. Subgroups for –help Option Use this –help option To get this information... –help=asm A list of options specific to the assembly phase. –help=debug A list of options related to debug information generation. –help=groups A list of available groups to use with the help option. –help=language A list of language-specific options. –help=linker A list of options specific to link phase. –help=opt A list of options specific to optimization phase. –help=other A list of other options, such as ansi conformance pointer aliasing for C. –help=overall A list of option generic to any compiler.PGI® User’s Guide 180 Use this –help option To get this information... –help=phase A list of build process phases and to which compiler they apply. –help=prepro A list of options specific to preprocessing phase. –help=suffix A list of known file suffixes and to which phases they apply. –help=switch A list of all known options, this is equivalent to usage of –help without any parameter. –help=target A list of options specific to target processor. –help=variable A list of all variables and their current value. They can be redefined on the command line using syntax VAR=VALUE. For more examples of –help, refer to “Help with Command-line Options,” on page 16. Related options: –#, –###, –show, –V, –flags –I Adds a directory to the search path for files that are included using either the INCLUDE statement or the preprocessor directive #include. Default: The compiler searches only certain directories for included files. • For gcc-lib includes: /usr/lib64/gcc-lib • For system includes: /usr/linclude Syntax: -Idirectory Where directory is the name of the directory added to the standard search path for include files. Usage: In the following example, the compiler first searches the directory mydir and then searches the default directories for include files. $ pgf95 -Imydir Description: Adds a directory to the search path for files that are included using the INCLUDE statement or the preprocessor directive #include. Use the –I option to add a directory to the list of where to search for the included files. The compiler searches the directory specified by the –I option before the default directories. The Fortran INCLUDE statement directs the compiler to begin reading from another file. The compiler uses two rules to locate the file: 1. If the file name specified in the INCLUDE statement includes a path name, the compiler begins reading from the file it specifies. 2. If no path name is provided in the INCLUDE statement, the compiler searches (in order):Chapter 15. Command-Line Options Reference 181 • Any directories specified using the –I option (in the order specified.) • The directory containing the source file • The current directory For example, the compiler applies rule (1) to the following statements: INCLUDE '/bob/include/file1' (absolute path name) INCLUDE '../../file1' (relative path name) and rule (2) to this statement: INCLUDE 'file1' Related options: –Mnostdinc –i2, –i4 and –i8 Treat INTEGER and LOGICAL variables as either two, four, or eight bytes. Default: The compiler treats INTERGER and LOGICAL variables as four bytes. Usage: In the following example using the i8 switch causes the integer variables to be treated as 64 bits. $ pgf95 -I8 int.f int.f is a function similar to this: int.f print *, “Integer size:”, bit_size(i) end Description: Use this option to treat INTEGER and LOGICAL variables as either two, four, or eight bytes. INTEGER*8 values not only occupy 8 bytes of storage, but operations use 64 bits, instead of 32 bits. Related options: –K Requests that the compiler provide special compilation semantics. Default: The compiler does not provide special compilation semantics. Syntax: –K Where flag is one of the following: ieee Perform floating-point operations in strict conformance with the IEEE 754 standard. Some optimizations are disabled, and on some systems a more accurate math library is linked if –Kieee is used during the link step.PGI® User’s Guide 182 noieee Default flag. Use the fastest available means to perform floating-point operations, link in faster non-IEEE libraries if available, and disable underflow traps. PIC (Linux only) Generate position-independent code. Equivalent to –fpic. Provided for compatibility with other compilers. pic (Linux only) Generate position-independent code. Equivalent to –fpic. Provided for compatibility with other compilers. trap=option [,option]... Controls the behavior of the processor when floating-point exceptions occur. Possible options include: • fp • align (ignored) • inv • denorm • divz • ovf • unf • inexact Usage: In the following example, the compiler performs floating-point operations in strict conformance with the IEEE 754 standard $ pgf95 -Kieee myprog.f Description: Use -K to instruct the compile to provide special compilation semantics. The default is –Knoieee. –Ktrap is only processed by the compilers when compiling main functions or programs. The options inv, denorm, divz, ovf, unf, and inexact correspond to the processor’s exception mask bits: invalid operation, denormalized operand, divide-by-zero, overflow, underflow, and precision, respectively. Normally, the processor’s exception mask bits are on, meaning that floating-point exceptions are masked—the processor recovers from the exceptions and continues. If a floating-point exception occurs and its corresponding mask bit is off, or “unmasked”, execution terminates with an arithmetic exception (C's SIGFPE signal). –Ktrap=fp is equivalent to –Ktrap=inv,divz,ovf. Note The PGI compilers do not support exception-free execution for–Ktrap=inexact. The purpose of this hardware support is for those who have specific uses for its execution, along with the appropriate signal handlers for handling exceptions it produces. It is not designed for normal floating point operation code support. Related options:Chapter 15. Command-Line Options Reference 183 --keeplnk (Windows only.) Preserves the temporary file when the compiler generates a temporary indirect file for a long linker command. Usage: In the following example the compiler preserves each temporary file rather than deleting it. $ pgf95 --keeplnk myprog.f Description: If the compiler generates a temporary indirect file for a long linker command, use this option to instruct the compiler to preserve the temporary file instead of deleting it. Related options: –L Specifies a directory to search for libraries. Note Multiple –L options are valid. However, the position of multiple –L options is important relative to –l options supplied. Syntax: -Ldirectory Where directory is the name of the library directory. Default: The compiler searches the standard library directory. Usage: In the following example, the library directory is /lib and the linker links in the standard libraries required by PGF95 from this directory. $ pgf95 -L/lib myprog.f In the following example, the library directory /lib is searched for the library file libx.a and both the directories /lib and /libz are searched for liby.a. $ pgf95 -L/lib -lx -L/libz -ly myprog.f Use the –L option to specify a directory to search for libraries. Using –L allows you to add directories to the search path for library files. Related options:-l –l Instructs the linker to load the specified library. The linker searches in addition to the standard libraries. Note The linker searches the libraries specified with –l in order of appearance before searching the standard libraries.PGI® User’s Guide 184 Syntax: -llibrary Where library is the name of the library to search. Usage: In the following example, if the standard library directory is /lib the linker loads the library /lib/ libmylib.a, in addition to the standard libraries. $ pgf95 myprog.f -lmylib Description: Use this option to instruct the linker to load the specified library. The compiler prepends the characters lib to the library name and adds the .a extension following the library name. The linker searches each library specifies before searching the standard libraries. Related options:–L –m Displays a link map on the standard output. Default: The compiler does display the link map and does not use the –m option. Usage:When the following example is executed on Windows, pgf95 creates a link map in the file myprog.map. $ pgf95 -m myprog.f Description: Use this option to display a link map. • On Linux, the map is written to stdout. • On Windows, the map is written to a .map file whose name depends on the executable. If the executable is myprog.f, the map file is in myprog.map. Related options: –c, –o, -s, –u –M Selects options for code generation. The options are divided into the following categories: Code generation Fortran Language Controls Optimization Environment C/C++ Language Controls Miscellaneous Inlining The following table lists and briefly describes the options alphabetically and includes a field showing the category. For more details about the options as they relate to these categories, refer to “–M Options by Category,” on page 219.Chapter 15. Command-Line Options Reference 185 Table 15.7. –M Options Summary pgflag Description Category allocatable=95|03 Controls whether to use Fortran 95 or Fortran 2003 semantics in allocatable array assignments. Fortran Language anno Annotate the assembly code with source code. Miscellaneous [no]autoinline C/C++ when a function is declared with the inline keyword, inline it at –O2 and above. Inlining [no]asmkeyword Specifies whether the compiler allows the asm keyword in C/C++ source files (pgcc and pgcpp only). C/C++ Language [no]backslash Determines how the backslash character is treated in quoted strings (pgf77, pgf95, and pghpf only). Fortran Language [no]bounds Specifies whether array bounds checking is enabled or disabled. Miscellaneous – –[no_]builtin Do/don’t compile with math subroutine builtin support, which causes selected math library routines to be inlined (pgcc and pgcpp only). Optimization byteswapio Swap byte-order (big-endian to little-endian or vice versa) during I/O of Fortran unformatted data. Miscellaneous cache_align Where possible, align data objects of size greater than or equal to 16 bytes on cache-line boundaries. Optimization chkfpstk Check for internal consistency of the x87 FP stack in the prologue of a function and after returning from a function or subroutine call (–tp px/p5/p6/ piii targets only). Miscellaneous chkptr Check for NULL pointers (pgf95 and pghpf only). Miscellaneous chkstk Check the stack for available space upon entry to and before the start of a parallel region. Useful when many private variables are declared. Miscellaneous concur Enable auto-concurrentization of loops. Multiple processors or cores will be used to execute parallelizable loops. Optimization cpp Run the PGI cpp-like preprocessor without performing subsequent compilation steps. Miscellaneous cray Force Cray Fortran (CF77) compatibility (pgf77, pgf95, and pghpf only). Optimization [no]daz Do/don’t treat denormalized numbers as zero. Code Generation [no]dclchk Determines whether all program variables must be declared (pgf77, pgf95, and pghpf only). Fortran LanguagePGI® User’s Guide 186 pgflag Description Category [no]defaultunit Determines how the asterisk character (“*”) is treated in relation to standard input and standard output (regardless of the status of I/O units 5 and 6, pgf77, pgf95, and pghpf only). Fortran Language [no]depchk Checks for potential data dependencies. Optimization [no]dse Enables [disables] dead store elimination phase for programs making extensive use of function inlining. Optimization [no]dlines Determines whether the compiler treats lines containing the letter "D" in column one as executable statements (pgf77, pgf95, and pghpf only). Fortran Language dll Link with the DLL version of the runtime libraries (Windows only). Miscellaneous dollar,char Specifies the character to which the compiler maps the dollar sign code (pgf77, pgf95, and pghpf only). Fortran Language dwarf1 When used with –g, generate DWARF1 format debug information. Code Generation dwarf2 When used with –g, generate DWARF2 format debug information. Code Generation dwarf3 When used with –g, generate DWARF3 format debug information. Code Generation extend Instructs the compiler to accept 132-column source code; otherwise it accepts 72-column code (pgf77, pgf95, and pghpf only). Fortran Language extract invokes the function extractor. Inlining fcon Instructs the compiler to treat floating-point constants as float data types (pgcc and pgcpp only). C/C++ Language fixed Instructs the compiler to assume F77-style fixed format source code (pgf95 and pghpf only). Fortran Language [no]flushz Do/don’t set SSE flush-to-zero mode Code Generation [no]fprelaxed[=option] Perform certain floating point intrinsic functions using relaxed precision. Optimization free Instructs the compiler to assume F90-style free format source code (pgf95 and pghpf only). Fortran Language func32 The compiler aligns all functions to 32-byte boundaries. Code Generation gccbug[s] Matches behavior of certain gcc bugs MiscellaneousChapter 15. Command-Line Options Reference 187 pgflag Description Category noi4 Determines how the compiler treats INTEGER variables (pgf77, pgf95, and pghpf only). Optimization info Prints informational messages regarding optimization and code generation to standard output as compilation proceeds. Miscellaneous inform Specifies the minimum level of error severity that the compiler displays. Miscellaneous inline Invokes the function inliner. Inlining [no]ipa Invokes inter-procedural analysis and optimization. Optimization [no]iomutex Determines whether critical sections are generated around Fortran I/O calls (pgf77, pgf95, and pghpf only). Fortran Language keepasm Instructs the compiler to keep the assembly file. Miscellaneous [no]large_arrays Enables support for 64-bit indexing and single static data objects of size larger than 2GB. Code Generation lfs Links in libraries that allow file I/O to files of size larger than 2GB on 32-bit systems (32-bit Linux only). Environment [no]lre Disable/enable loop-carried redundancy elimination. Optimization list Specifies whether the compiler creates a listing file. Miscellaneous makedll Generate a dynamic link library (DLL) (Windows only). Miscellaneous makeimplib Passes the -def switch to the librarian without a deffile, when used without –def:deffile. Miscellaneous mpi=option Link to MPI libraries: MPICH1, MPICH2, or Microsoft MPI libraries Code Generation [no]loop32 Aligns/does not align innermost loops on 32 byte boundaries with –tp barcelona Code Generation [no]movnt Force/disable generation of non-temporal moves and prefetching. Code Generation neginfo Instructs the compiler to produce information on why certain optimizations are not performed. Miscellaneous noframe Eliminates operations that set up a true stack frame pointer for functions. Optimization nomain When the link step is called, don’t include the object file that calls the Fortran main program (pgf77, pgf95, and pghpf only). Code GenerationPGI® User’s Guide 188 pgflag Description Category noopenmp When used in combination with the –mp option, causes the compiler to ignore OpenMP parallelization directives or pragmas, but still process SGI-style parallelization directives or pragmas. Miscellaneous nopgdllmain Do not link the module containing the default DllMain() into the DLL (Windows only). Miscellaneous norpath On Linux, do not add –rpath paths to the link line. Miscellaneous nosgimp When used in combination with the –mp option, causes the compiler to ignore SGI-style parallelization directives or pragmas, but still process OpenMP directives or pragmas. Miscellaneous [no]stddef Instructs the compiler to not recognize the standard preprocessor macros. Environment nostdinc Instructs the compiler to not search the standard location for include files. Environment nostdlib Instructs the linker to not link in the standard libraries. Environment [no]onetrip Determines whether each DO loop executes at least once (pgf77, pgf95, and pghpf only). Language novintr Disable idiom recognition and generation of calls to optimized vector functions. Optimization pfi Instrument the generated code and link in libraries for dynamic collection of profile and data information at runtime. Optimization pfo Read a pgfi.out trace file and use the information to enable or guide optimizations. Optimization [no]prefetch Enable/disable generation of prefetch instructions. Optimization preprocess Perform cpp-like preprocessing on assembly language and Fortran input source files. Miscellaneous prof Set profile options; function-level and line-level profiling are supported. Code Generation [no]r8 Determines whether the compiler promotes REAL variables and constants to DOUBLE PRECISION (pgf77, pgf95, and pghpf only). Optimization [no]r8intrinsics Determines how the compiler treats the intrinsics CMPLX and REAL (pgf77, pgf95, and pghpf only). Optimization [no]recursive Allocate (do not allocate) local variables on the stack, this allows recursion. SAVEd, data-initialized, Code GenerationChapter 15. Command-Line Options Reference 189 pgflag Description Category or namelist members are always allocated statically, regardless of the setting of this switch (pgf77, pgf95, and pghpf only). [no]reentrant Specifies whether the compiler avoids optimizations that can prevent code from being reentrant. Code Generation [no]ref_externals Do/don’t force references to names appearing in EXTERNAL statements (pgf77, pgf95, and pghpf only). Code Generation safeptr Instructs the compiler to override data dependencies between pointers and arrays (pgcc and pgcpp only). Optimization safe_lastval In the case where a scalar is used after a loop, but is not defined on every iteration of the loop, the compiler does not by default parallelize the loop. However, this option tells the compiler it safe to parallelize the loop. For a given loop, the last value computed for all scalars make it safe to parallelize the loop. Code Generation [no]save Determines whether the compiler assumes that all local variables are subject to the SAVE statement (pgf77, pgf95, and pghpf only). Fortran Language [no]scalarsse Do/don’t use SSE/SSE2 instructions to perform scalar floating-point arithmetic. Optimization schar Specifies signed char for characters (pgcc and pgcpp only - also see uchar). C/C++ Language [no]second_underscore Do/don’t add the second underscore to the name of a Fortran global if its name already contains an underscore (pgf77, pgf95, and pghpf only). Code Generation [no]signextend Do/don’t extend the sign bit, if it is set. Code Generation [no]single Do/don’t convert float parameters to double parameter characters (pgcc and pgcpp only). C/C++ Language [no]smart Do/don’t enable optional post-pass assembly optimizer. Optimization [no]smartalloc[=huge| huge:|hugebss] Add a call to the routine mallopt in the main routine. Supports large TLBs on Linux and Windows. Tip. To be effective, this switch must be specified when compiling the file containing the Fortran, C, or C++ main program. Environment standard Causes the compiler to flag source code that does not conform to the ANSI standard (pgf77, pgf95, and pghpf only). Fortran LanguagePGI® User’s Guide 190 pgflag Description Category [no]stride0 Do/do not generate alternate code for a loop that contains an induction variable whose increment may be zero (pgf77, pgf95, and pghpf only). Code Generation uchar Specifies unsigned char for characters (pgcc and pgcpp only - also see schar). C/C++ Language unix Uses UNIX calling and naming conventions for Fortran subprograms (pgf77, pgf95, and pghpf for Win32 only). Code Generation [no]nounixlogical Determines whether logical .TRUE. and .FALSE. are determined by non-zero (TRUE) and zero (FALSE) values for unixlogical. With nounixlogical, the default, -1 values are TRUE and 0 values are FALSE (pgf77, pgf95, and pghpf only). Fortran Language [no]unroll Controls loop unrolling. Optimization [no]upcase Determines whether the compiler allows uppercase letters in identifiers (pgf77, pgf95, and pghpf only). Fortran Language varargs Forces Fortran program units to assume calls are to C functions with a varargs type interface (pgf77 and pgf95 only). Code Generation [no]vect Do/don’t invoke the code vectorizer. Optimization –mcmodel=medium (For use only on 64-bit Linux targets) Generates code for the medium memory model in the linux86-64 execution environment. Implies –Mlarge_arrays. Default: The compiler generates code for the small memory model. Usage: The following command line requests position independent code be generated, and the –mcmodel=medium option be passed to the assembler and linker: $ pgf95 -mcmodel=medium myprog.f Description: The default small memory model of the linux86-64 environment limits the combined area for a user’s object or executable to 1GB, with the Linux kernel managing usage of the second 1GB of address for system routines, shared libraries, stacks, and so on. Programs are started at a fixed address, and the program can use a single instruction to make most memory references. The medium memory model allows for larger than 2GB data areas, or .bss sections. Program units compiled using either –mcmodel=medium or –fpic require additional instructions to reference memory. The effect on performance is a function of the data-use of the application. The –mcmodel=medium switch must be used at both compile time and link time to create 64-bit executables. Program units compiled for the default small memory model can be linked into medium memory model executables as long as they are compiled with –fpic, or position-independent.Chapter 15. Command-Line Options Reference 191 The linux86-64 environment provides static libxxx.a archive libraries that are built with and without –fpic, and dynamic libxxx.so shared object libraries that are compiled –fpic. The –mcmodel=medium link switch implies the –fpic switch and will utilize the shared libraries by default. Similarly, the $PGI/linux86-64// lib directory contains the libraries for building small memory model codes, and the $PGI/linux86-64// libso directory contains shared libraries for building –mcmodel=medium and –fpic executables. Note –mcmodel=medium -fpic is not allowed to create shared libraries. However, you can create static archive libraries (.a) that are –fpic. Related options:–Mlarge_arrays –module Allows you to specify a particular directory in which generated intermediate .mod files should be placed. Default: The compiler places .mod files in the current working directory, and searches only in the current working directory for pre-compiled intermediate .mod files. Usage: The following command line requests that any intermediate module file produced during compilation of myprog.f be placed in the directory mymods; specifically, the file ./mymods/myprog.mod is used. $ pgf95 -module mymods myprog.f Description: Use the –module option to specify a particular directory in which generated intermediate .mod files should be placed. If the –module option is present, and USE statements are present in a compiled program unit, then is searched for .mod intermediate files prior to a search in the default local directory. Related options: –mp[=align,[no]numa] Instructs the compiler to interpret user-inserted OpenMP shared-memory parallel programming directives and pragmas, and to generate an executable file which will utilize multiple processors in a shared-memory parallel system. Default: The compiler ignores user-inserted shared-memory parallel programming directives and pragmas. Usage: The following command line requests processing of any shared-memory directives present in myprog.f: $ pgf95 -mp myprog.f Description: Use the –mp option to instruct the compiler to interpret user-inserted OpenMP shared-memory parallel programming directives and to generate an executable file which utilizes multiple processors in a shared-memory parallel system. The align sub-option forces loop iterations to be allocated to OpenMP processes using an algorithm that maximizes alignment of vector sub-sections in loops that are both parallelized and vectorized for SSE. ThisPGI® User’s Guide 192 allocation can improve performance in program units that include many such loops. It can also result in loadbalancing problems that significantly decrease performance in program units with relatively short loops that contain a large amount of work in each iteration. The numa suboption uses libnuma on systems where it is available. For a detailed description of this programming model and the associated directives and pragmas, refer to Chapter 5, “Using OpenMP”. Related options: –Mconcur and –Mvect –nfast A generally optimal set of options is chosen depending on the target system. In addition, the appropriate –tp option is automatically included to enable generation of code optimized for the type of system on which compilation is performed. Note Auto-selection of the appropriate –tp option means that programs built using the –fast option on a given system are not necessarily backward-compatible with older systems. Usage: In the following example, the compiler selects a generally optimal set of options for the target system. $ pgf95 -nfast myprog.f Description: Use this option to instruct the compiler to select a generally optimal set of options for the target system. In addition, the appropriate –tp option is automatically included to enable generation of code optimized for the type of system on which compilation is performed. Related options: –O, –Munroll, –Mnoframe, –Mvect, –tp, –Mscalarsse –noswitcherror Issues warnings instead of errors for unknown switches. Ignores unknown command line switches after printing an warning message. Default: The compiler prints an error message and then halts. Usage: In the following example, the compiler ignores unknown command line switches after printing an warning message. $ pgf95 -noswitcherror myprog.f Description: Use this option to instruct the compiler to ignore unknown command line switches after printing an warning message. Tip You can configure this behavior in the siterc file by adding: set NOSWITCHERROR=1. Related options:None.Chapter 15. Command-Line Options Reference 193 –O Invokes code optimization at the specified level. Default: The compiler optimizes at level 2 (correct?) Syntax: –O [level] Where level is an integer from 0 to 4. Usage: In the following example, since no –O option is specified, the compiler sets the optimization to level 1. $ pgf95 myprog.f In the following example, since no optimization level is specified and a –O option is specified, the compiler sets the optimization to level 2. $ pgf95 -O myprog.f Description: Use this option to invoke code optimization at the specified level - one of the following: 0 creates a basic block for each statement. Neither scheduling nor global optimization is done. To specify this level, supply a 0 (zero) argument to the –O option. 1 schedules within basic blocks and performs some register allocations, but does no global optimization. 2 performs all level-1 optimizations, and also performs global scalar optimizations such as induction variable elimination and loop invariant movement. 3 level-three specifies aggressive global optimization. This level performs all level-one and level-two op-timizations and enables more aggressive hoisting and scalar replacement optimizations that may or may not be profitable. 4 level-four performs all level-one, level-two, and level-three op-timizations and enables hoisting of guarded invariant floating point expressions. Table 15.8 shows the interaction between the –O option, –g option, –Mvect, and –Mconcur options. Table 15.8. Optimization and –O, –g, –Mvect, and –Mconcur Options Optimize Option Debug Option –M Option Optimization Level none none none 1 none none –Mvect 2PGI® User’s Guide 194 Optimize Option Debug Option –M Option Optimization Level none none –Mconcur 2 none –g none 0 –O none or –g none 2 –Olevel none or –g none level –Olevel < 2 none or –g –Mvect 2 –Olevel < 2 none or –g –Mconcur 2 Unoptimized code compiled using the option –O0 can be significantly slower than code generated at other optimization levels. Like the –Mvect option, the –Munroll option sets the optimization level to level-2 if no –O or –g options are supplied. The –gopt option is recommended for generation of debug information with optimized code. For more information on optimization, see Chapter 3, “Using Optimization & Parallelization”. Related options: –g, –M, –gopt –o Names the executable file. Use the –o option to specify the filename of the compiler object file. The final output is the result of linking. Syntax: –o filename Where filename is the name of the file for the compilation output. The filename must not have a .f extension. Default: The compiler creates executable filenames as needed. If you do not specify the –o option, the default filename is the linker output file a.out. Usage: In the following example, the executable file is myprog instead of the default a.out. $ pgf95 myprog.f -o myprog Related options: –c, –E, –F, –S –pc Note This option is available only for –tp px/p5/p6/piii targets. Allows you to control the precision of operations performed using the x87 floating point unit, and their representation on the x87 floating point stack. Syntax: –pc { 32 | 64 | 80 }Chapter 15. Command-Line Options Reference 195 Usage: $ pgf95 -pc 64 myprog.c Description: The x87 architecture implements a floating-point stack using 8 80-bit registers. Each register uses bits 0-63 as the significant, bits 64-78 for the exponent, and bit 79 is the sign bit. This 80-bit real format is the default format, called the extended format. When values are loaded into the floating point stack they are automatically converted into extended real format. The precision of the floating point stack can be controlled, however, by setting the precision control bits (bits 8 and 9) of the floating control word appropriately. In this way, you can explicitly set the precision to standard IEEE double-precision using 64 bits, or to single precision using 32 bits. 1 The default precision is system dependent. To alter the precision in a given program unit, the main program must be compiled with the same -pc option. The command line option –pc val lets the programmer set the compiler’s precision preference. Valid values for val are: • 32 single precision • 64 double precision • 80 extended precision Extended Precision Option – Operations performed exclusively on the floating-point stack using extended precision, without storing into or loading from memory, can cause problems with accumulated values within the extra 16 bits of extended precision values. This can lead to answers, when rounded, that do not match expected results. For example, if the argument to sin is the result of previous calculations performed on the floating-point stack, then an 80-bit value used instead of a 64-bit value can result in slight discrepancies. Results can even change sign due to the sin curve being too close to an x-intercept value when evaluated. To maintain consistency in this case, you can assure that the compiler generates code that calls a function. According to the x86 ABI, a function call must push its arguments on the stack (in this way memory is guaranteed to be accessed, even if the argument is an actual constant.) Thus, even if the called function simply performs the inline expansion, using the function call as a wrapper to sin has the effect of trimming the argument precision down to the expected size. Using the –Mnobuiltin option on the command line for C accomplishes this task by resolving all math routines in the library libm, performing a function call of necessity. The other method of generating a function call for math routines, but one that may still produce the inline instructions, is by using the –Kieee switch. A second example illustrates the precision control problem using a section of code to determine machine precision: program find_precision w = 1.0 100 w=w+w y=w+1 z=y-w if (z .gt. 0) goto 100 C now w is just big enough that |((w+1)-w)-1| >= 1 ... print*,w 1 According to Intel documentation, this only affects the x87 operations of add, subtract, multiply, divide, and square root. In particular, it does not appear to affect the x87 transcendental instructions.PGI® User’s Guide 196 end In this case, where the variables are implicitly real*4, operations are performed on the floating-point stack where optimization removed unnecessary loads and stores from memory. The general case of copy propagation being performed follows this pattern: a = x y = 2.0 + a Instead of storing x into a, then loading a to perform the addition, the value of x can be left on the floatingpoint stack and added to 2.0. Thus, memory accesses in some cases can be avoided, leaving answers in the extended real format. If copy propagation is disabled, stores of all left-hand sides will be performed automatically and reloaded when needed. This will have the effect of rounding any results to their declared sizes. For the above program, w has a value of 1.8446744E+19 when executed using default (extended) precision. If, however, –Kieee is set, the value becomes 1.6777216E+07 (single precision.) This difference is due to the fact that –Kieee disables copy propagation, so all intermediate results are stored into memory, then reloaded when needed. Copy propagation is only disabled for floating-point operations, not integer. With this particular example, setting the –pc switch will also adjust the result. The switch –Kieee also has the effect of making function calls to perform all transcendental operations. Although the function still produces the x86 machine instruction for computation (unless in C the –Mnobuiltin switch is set), arguments are passed on the stack, which results in a memory store and load. Finally, –Kieee also disables reciprocal division for constant divisors. That is, for a/b with unknown a and constant b, the expression is usually converted at compile time to a*(1/b), thus turning an expensive divide into a relatively fast scalar multiplication. However, numerical discrepancies can occur when this optimization is used. Understanding and correctly using the –pc, –Mnobuiltin, and Kieee switches should enable you to produce the desired and expected precision for calculations which utilize floating-point operations. Related options: –pg (Linux only) Instructs the compiler to instrument the generated executable for gprof-style sample-based profiling. Usage: In the following example the program is compiled for profiling using pgdbg or gprof. $ pgf95 -pg myprog.c Default: The compiler does not instrument the generated executable for gprof-style profiling. Description: Use this option to instruct the compiler to instrument the generated executable for gprof-style sample-based profiling. You must use this option at both the compile and link steps. A gmon.out style trace is generated when the resulting program is executed, and can be analyzed using gprof or pgprof. –pgf77libs Instructs the compiler to append PGF77 runtime libraries to the link line.Chapter 15. Command-Line Options Reference 197 Default: The compiler does not append the PGF77 runtime libraries to the link line. Usage: In the following example a .c main program is linked with an object file compiled with pgf77. $ pgcc main.c myf77.o -pgf77libs Description: Use this option to instruct the compiler to append PGF77 runtime libraries to the link line. Related options:–pgf90libs –pgf90libs Instructs the compiler to append PGF90/PGF95 runtime libraries to the link line. Default: The compiler does not append the PGF90/PGF95 runtime libraries to the link line. Usage: In the following example a .c main program is linked with an object file compiled with pgf95. $ pgf95 main.c myf95.o -pgf90libs Description: Use this option to instruct the compiler to append PGF90/PGF95 runtime libraries to the link line. Related options:-pgf77libs –Q Selects variations for compilation. There are four uses for the –Q option. Usage: The following examples show the different –Q options. $ pgf95 -Qdir /home/comp/new hello.f $ pgf95 -Qoption ld,-s hello.f $ pgf95 -Qpath /home/test hello.f $ pgf95 -Qproduce .s hello.f Description: Use this option to select variations for compilation. As illustrated in the Usage section, there are four varieties for the –Q option. The first variety, using the dir keyword, lets you supply a directory parameter that indicates the directory where the compiler driver is located. -Qdirdirectory The second variety, using the option keyword, lets you supply the option opt to the program prog. The prog parameter can be one of pgftn, as, or ld. -Qoptionprog,opt The third –Q variety, using the path keyword, lets you supply an additional pathname to the search path for the compiler’s required .o files. -QpathpathnamePGI® User’s Guide 198 The fourth –Q variety, using the produce keyword, lets you choose a stop-after location for the compilation based on the supplied sourcetype parameter. Valid sourcetypes are: .i, .c, .s and .o, which respectively indicate the stop-after locations: preprocessing, compiling, assembling, or linking. -Qproducesourcetype Related options: –p –R (Linux only) Instructs the linker to hard-code the pathname into the search path for generated shared object (dynamically linked library) files. Note There cannot be a space between R and . Usage: In the following example, at runtime the a.out executable searches the specified directory, in this case /home./Joe/myso, for shared objects. $ pgf95 -Rm/home/Joe/myso myprog.f Description: Use this option to instruct the compiler to pass information to the linker to hard-code the pathname into the search path for shared object (dynamically linked library) files. Related options: –fpic, –shared, –G –r Linux only. Creates a relocatable object file. Default: The compiler does not create a relocatable object file and does not use the –r option. Usage: In this example, pgf95 creates a relocatable object file. $ pgf95 -r myprog.f Use this option to create a relocatable object file. Related options: –c, –o, –s, –u –r4 and –r8 Interprets DOUBLE PRECISION variables as REAL (–r4) or REAL variables as DOUBLE PRECISION (–r8). Usage: In this example, the double precision variables are interpreted as REAL. $ pgf95 -r4 myprog.f Description: Interpret DOUBLE PRECISION variables as REAL (–r4) or REAL variables as DOUBLE PRECISION (–r8). Related options: –i2, –i4, –i8, –nor8Chapter 15. Command-Line Options Reference 199 –rc Specifies the name of the driver startup configuration file. If the file or pathname supplied is not a full pathname, the path for the configuration file loaded is relative to the $DRIVER path (the path of the currently executing driver). If a full pathname is supplied, that file is used for the driver configuration file. Syntax: -rc [path] filename Where path is either a relative pathname, relative to the value of $DRIVER, or a full pathname beginning with "/ ". Filename is the driver configuration file. Default: The driver uses the configuration file .pgirc. Usage: In the following example, the file .pgf95rctest, relative to /usr/pgi/linux86/bin, the value of $DRIVER, is the driver configuration file. $ pgf95 -rc .pgf95rctest myprog.f Description: Use this option to specify the name of the driver startup configuration file. If the file or pathname supplied is not a full pathname, the path for the configuration file loaded is relative to the $DRIVER path - the path of the currently executing driver. If a full pathname is supplied, that file is used for the driver configuration file. Related options: –show –rpath Linux only. Syntax: -rpath path Speicifes the name of the dirver startip configuration file, where path is either a relative pathname, or a full pathname beginning with "/". Default: The driver uses the configuration file .pgirc. Usage: In the following example, the file .pgf95rctest, relative to /usr/pgi/linux86/bin, the value of $DRIVER, is the driver configuration file. $ pgf95 -rc .pgf95rctest myprog.f Description: Use this option to specify the name of the driver startup configuration file. If the file or pathname supplied is not a full pathname, the path for the configuration file loaded is relative to the $DRIVER path - the path of the currently executing driver. If a full pathname is supplied, that file is used for the driver configuration file. Related options: –show –s (Linux only) Strips the symbol-table information from the executable file.PGI® User’s Guide 200 Default: The compiler includes all symbol-table information and does not use the –s option. Usage: In this example, pgf95 strips symbol-table information from the a.out. executable file. $ pgf95 -s myprog.f Description: Use this option to strip the symbol-table information from the executable. Related options: –c, –o, –u –S Stops compilation after the compiling phase and writes the assembly-language output to a file. Default: The compiler does not produce a .s file. Usage: In this example, pgf95 produces the file myprog.s in the current directory. $ pgf95 -S myprog.f Description: Use this option to stop compilation after the compiling phase and then write the assemblylanguage output to a file. If the input file is filename.f, then the output file is filename.s. Related options: –c, –E, –F, –Mkeepasm, –o –shared (Linux only) Instructs the compiler to pass information to the linker to produce a shared object (dynamically linked library) file. Default: The compiler does not pass information to the linker to produce a shared object file. Usage: In the following example the compiler passes information to the linker to produce the shared object file: myso.so. $ pgf95 -shared myprog.f -o myso.so Description: Use this option to instruct the compiler to pass information to the linker to produce a shared object (dynamically linked library) file. Related options: –fpic, –G, –R –show Produces driver help information describing the current driver configuration. Default: The compiler does not show driver help information. Usage: In the following example, the driver displays configuration information to the standard output after processing the driver configuration file. $ pgf95 -show myprog.f Description: Use this option to produce driver help information describing the current driver configuration.Chapter 15. Command-Line Options Reference 201 Related options: –V, –v, –###, –help, –rc –silent Do not print warning messages. Default: The compiler prints warning messages. Usage: In the following example, the driver does not display warning messages. $ pgf95 -silent myprog.f Description: Use this option to suppress warning messages. Related options: –v, –V, –w –soname (Linux only.) The compiler recognizes the –soname option and passes it through to the linker. Default: The compiler does not recognize the –soname option. Usage: In the following example, the driver passes the soname option and its argument through to the linker. $ pgf95 -soname library.so myprog.f Description: Use this option to instruct the compiler to recognize the –soname option and pass it through to the linker. Related options: –stack (Windows only.) Allows you to explicitly set stack properties for your program. Default: If –stack is not specified, then the defaults are as followed: Win32 Setting is -stack:2097152,2097152, which is approximately 2MB for reserved and committed bytes. Win64 No default setting Syntax: -stack={ (reserved bytes)[,(committed bytes)] }{, [no]check } Usage: The following example demonstrates how to reserve 524,288 stack bytes (512KB), commit 262,144 stack bytes for each routine (256KB), and disable the stack initialization code with the nocheck argument. $ pgf95 -stack=524288,262144,nocheck myprog.f Description: Use this option to explicitly set stack properties for your program. The –stack option takes one or more arguments: (reserved bytes), (committed bytes), [no]check.PGI® User’s Guide 202 reserved bytes Specifies the total stack bytes required in your program. committed bytes Specifies the number of stack bytes that the Operating System will allocate for each routine in your program. This value must be less than or equal to the stack reserved bytes value. Default for this argument is 4096 bytes [no]check Instructs the compiler to generate or not to generate stack initialization code upon entry of each routine. Check is the default, so stack initialization code is generated. Stack initialization code is required when a routine's stack exceeds the committed bytes size. When your committed bytes is equal to the reserved bytes or equal to the stack bytes required for each routine, then you can turn off the stack initialization code using the -stack=nocheck compiler option. If you do this, the compiler assumes that you are specifying enough committed stack space; and therefore, your program does not have to manage its own stack size. For more information on determining the amount of stack required by your program, refer to –Mchkstk compiler option, described in “–M Miscellaneous Controls”. Note -stack=(reserved bytes),(committed bytes) are linker options. -stack=[no]check is a compiler option. If you specify -stack=(reserved bytes),(committed bytes) on your compile line, it is only used during the link step of your build. Similarly, –stack=[no]check can be specified on your link line, but its only used during the compile step of your build. Related options:–Mchkstk –time Print execution times for various compilation steps. Default: The compiler does not print execution times for compilation steps. Usage: In the following example, pgf95 prints the execution times for the various compilation steps. $ pgf95 -time myprog.f Description: Use this option to print execution times for various compilation steps. Related options: –# –tp [,target...] Sets the target architecture.Chapter 15. Command-Line Options Reference 203 Default: The PGI compilers produce code specifically targeted to the type of processor on which the compilation is performed. In particular, the default is to use all supported instructions wherever possible when compiling on a given system. The default style of code generation is auto-selected depending on the type of processor on which compilation is performed. Further, the –tp x64 style of unified binary code generation is only enabled by an explicit –tp x64 option. Note Executables created on a given system may not be usable on previous generation systems. (For example, executables created on a Pentium 4 may fail to execute on a Pentium III or Pentium II.) Usage: In the following example, pgf95 sets the target architecture to EM64T: $ pgf95 -tp p7-64 myprog.f Description: Use this option to set the target architecture. By default, the PGI compiler uses all supported instructions wherever possible when compiling on a given system. As a result, executables created on a given system may not be usable on previous generation systems. For example, executables created on a Pentium 4 may fail to execute on a Pentium III or Pentium II. Processor-specific optimizations can be specified or limited explicitly by using the –tp option. Thus, it is possible to create executables that are usable on previous generation systems. With the exception of k8-64, k8- 64e, p7-64, and x64, any of these sub-options are valid on any x86 or x64 processor-based system. The k8-64, k8-64e, p7-64 and x64 options are valid only on x64 processor-based systems. The –tp x64 option generates unified binary object and executable files, as described in the section called “Using –tp to Generate a Unified Binary”. The following list is the possible sub-options for –tp and the processors that each sub-option is intended to target: k8-32 generate 32-bit code for AMD Athlon64, AMD Opteron and compatible processors. k8-64 generate 64-bit code for AMD Athlon64, AMD Opteron and compatible processors. k8-64e generate 64-bit code for AMD Opteron Revision E, AMD Turion, and compatible processors. p6 generate 32-bit code for Pentium Pro/II/III and AthlonXP compatible processors. p7 generate 32-bit code for Pentium 4 and compatible processors. p7-64 generate 64-bit code for Intel P4/Xeon EM64T and compatible processors. core2 generate 32-bit code for Intel Core 2 Duo and compatible processors.PGI® User’s Guide 204 core2-64 generate 64-bit code for Intel Core 2 Duo EM64T and compatible processors. piii generate 32-bit code for Pentium III and compatible processors, including support for single-precision vector code using SSE instructions. px generate 32-bit code that is usable on any x86 processor-based system. x64 generate 64-bit unified binary code including full optimizations and support for both AMD and Intel x64 processors. Refer to Table 2, “Processor Options,” on page xxvi for a concise list of the features of these processors that distinguish them as separate targets when using the PGI compilers and tools. Syntax for 64-bit targets: -tp {k8-64 | k8-64e | p7-64 | core2-64 | x64} Syntax for 32-bit targets: -tp {k8-32 | p6 | p7 | core2 | piii | px} Using –tp to Generate a Unified Binary Different processors have differences, some subtle, in hardware features such as instruction sets and cache size. The compilers make architecture-specific decisions about such things as instruction selection, instruction scheduling, and vectorization. Any of these decisions can have significant effects on performance and compatibility. PGI unified binaries provide a low-overhead means for a single program to run well on a number of hardware platforms. You can use the –tp option to produce PGI Unified Binary programs. The compilers generate, and combine into one executable, multiple binary code streams, each optimized for a specific platform. At runtime, this one executable senses the environment and dynamically selects the appropriate code stream. The target processor switch, –tp, accepts a comma-separated list of 64-bit targets and will generate code optimized for each listed target. For example, the following switch generates optimized code for three targets: k8-64, p7-64, and core2-64. Syntax for optimizing for multiple targets: -tp k8-64,p7-64,core2-64 The –tp k8-64 and –tp k8-64e options result in generation of code supported on and optimized for AMD x64 processors, while the –tp p7-64 option results in generation of code that is supported on and optimized for Intel x64 processors. Performance of k8-64 or k8-64e code executed on Intel x64 processors, or of p7-64 code executed on AMD x64 processors, can often be significantly less than that obtained with a native binary. The special –tp x64 option is equivalent to –tp k8-64,p7-64. This switch produces PGI Unified Binary programs containing code streams fully optimized and supported for both AMD64 and Intel EM64T processors.Chapter 15. Command-Line Options Reference 205 For more information on unified binaries, refer to “Processor-Specific Optimization and the Unified Binary,” on page 36. Related options: –u Initializes the symbol-table with , which is undefined for the linker. Default: The compiler does not use the –u option. Syntax: -usymbol Where symbol is a symbolic name. Usage: In this example, pgf95 initializes symbol-table with , $ pgf95 -utest myprog.f Description: Use this option to initialize the symbol-table with , which is undefined for the linker. An undefined symbol triggers loading of the first member of an archive library. Related options: –c, –o, –s –U Undefines a preprocessor macro. Syntax: -Usymbol Where symbol is a symbolic name. Usage: The following examples undefine the macro test. $ pgf95 -Utest myprog.F $ pgf95 -Dtest -Utest myprog.F Description: Use this option to undefine a preprocessor macro. You can also use the #undef pre-processor directive to undefine macros. Related options: –D,–Mnostddef. –V[release_number] Displays additional information, including version messages. Further, if a release_number is appended, the compiler driver attempts to compile using the specified release instead of the default release. Note There can be no space between –V and release_number.PGI® User’s Guide 206 Default: The compiler does not display version information and uses the release specified by your path to compile. Usage: The following command-line shows the output using the –V option. % pgf95 -V myprog.f The following command-line causes PGF95 to compile using the 5.2 release instead of the default release. % pgcc -V5.2 myprog.c Description: Use this option to display additional information, including version messages or, if a release_number is appended, to instruct the compiler driver to attempt to compile using the specified release instead of the default release. The specified release must be co-installed with the default release, and must have a release number greater than or equal to 4.1, which was the first release that supported this functionality. Related options: –Minfo, –v –v Displays the invocations of the compiler, assembler, and linker. Default: The compiler does not display individual phase invocations. Usage: In the following example you use –v to see the commands sent to compiler tools, assembler, and linker. $ pgf95 -v myprog.f90 Description: Use the –v option to display the invocations of the compiler, assembler, and linker. These invocations are command lines created by the compiler driver from the files and the –W options you specify on the compiler command-line. Related options: –Minfo, –, V, –W –W Passes arguments to a specific phase. Syntax: -W{0 | a | l },option[,option...] Note You cannot have a space between the –W and the single-letter pass identifier, between the identifier and the comma, or between the comma and the option. 0 (the number zero) specifies the compiler.Chapter 15. Command-Line Options Reference 207 a specifies the assembler. l (lowercase letter l) specifies the linker. option is a string that is passed to and interpreted by the compiler, assembler or linker. Options separated by commas are passed as separate command line arguments. Usage: In the following example the linker loads the text segment at address 0xffc00000 and the data segment at address 0xffe00000. $ pgf95 -Wl,-k,-t,0xffc00000,-d,0xffe00000 myprog.f Description: Use this option to pass arguments to a specific phase. You can use the –W option to specify options for the assembler, compiler, or linker. Note A given PGI compiler command invokes the compiler driver, which parses the command-line, and generates the appropriate commands for the compiler, assembler, and linker. Related options: –w Do not print warning messages. Default: The compiler prints warning messages. Usage: In the following example no warning messages are printed. $ pgf95 -w myprog.f Description: Use the –w option to not print warning messages. Sometimes the compiler issues many warning in which you may have no interest. You can use this option to not issue those warnings. Related options:–silent –Xs Use legacy standard mode for C and C++. Default:None. Usage: In the following example the compiler uses legacy standard mode. $ pgcc -XS myprog.c Description: Use this option to use legacy standard mode for C and C++. This option implies - alias=traditional. Related options:-alias, –XtPGI® User’s Guide 208 –Xt Use legacy traditional mode for C and C++. Default:None. Usage: In the following example the compiler uses legacy traditional mode. $ pgcc -XStmyprog.c Description: Use this option to use legacy standard mode for C and C++. This option implies - alias=traditional. Related options:-alias, –Xs C and C++ -specific Compiler Options There are a large number of compiler options specific to the PGCC and PGC++ compilers, especially PGC++. This section provides the details of several of these options, but is not exhaustive. For a complete list of available options, including an exhaustive list of PGC++ options, use the –help command-line option. For further detail on a given option, use –help and specify the option explicitly, as described in –help . –A (pgcpp only) Instructs the PGC++ compiler to accept code conforming to the proposed ANSI C++ standard, issuing errors for non-conforming code. Default: By default, the compiler accepts code conforming to the standard C++ Annotated Reference Manual. Usage: The following command-line requests ANSI conforming C++. $ pgcpp -A hello.cc Description: Use this option to instruct the PGC++ compiler to accept code conforming to the proposed ANSI C++ standard and to issues errors for non-conforming code. Related options:–a, –b and +p. –a (pgcpp only) Instructs the PGC++ compiler to accept code conforming to the proposed ANSI C++ standard, issuing warnings for non-conforming code. Default: By default, the compiler accepts code conforming to the standard C++ Annotated Reference Manual. Usage: The following command-line requests ANSI conforming C++, issuing warnings for non-conforming code. $ pgcpp -a hello.cc Description: Use this option to instruct the PGC++ compiler to accept code conforming to the proposed ANSI C++ standard and to issues warnings for non-conforming code.Chapter 15. Command-Line Options Reference 209 Related options:–A, –b and +p. –alias select optimizations based on type-based pointer alias rules in C and C++. Syntax: -alias=[ansi|traditional] Default:None Usage: The following command-line enables optimizations. $ pgcpp -alias=ansi hello.cc Description: Use this option to select optimizations based on type-based pointer alias rules in C and C++. ansi Enable optimizations using ANSI C type-based pointer disambiguation traditional Disable type-based pointer disambiguation Related options: --[no_]alternative_tokens (pgcpp only) Enables or disables recognition of alternative tokens. These are tokens that make it possible to write C++ without the use of the comma (,) , [, ], #, &, ^, and characters. The alternative tokens include the operator keywords (e.g., and, bitand, etc.) and digraphs. The default behavior is --no_alternative_tokens. Default:. The default behavior is that the recognition of alternative tokens is disabled: -- no_alternative_tokens. Usage: The following command-line enables alternative token recognition. $ pgcpp --alternative_tokens hello.cc (pgcpp only) Use this option to enable or disable recognition of alternative tokens. These tokens make it possible to write C++ without the use of the comma (,), [, ], #, &, ^, and characters. The alternative tokens include digraphs and the operator keywords, such as and, bitand, and so on. The default behavior is -- no_alternative_tokens. Related options: –B (pgcc and pgcpp only) Enables use of C++ style comments starting with // in C program units. Default: The PGCC ANSI and K&R C compiler does not allow C++ style comments. Usage: In the following example the compiler accepts C++ style comments.PGI® User’s Guide 210 $ pgcc -B myprog.cc Description: Use this option to enable use of C++ style comments starting with // in C program units. Related options: –b (pgcpp only) Enables compilation of C++ with cfront 2.1 compatibility and acceptance of anachronisms. Default: The compiler does not accept cfront language constructs that are not part of the C++ language definition. Usage: In the following example the compiler accepts cfront constructs. $ pgcpp -b myprog.cc Description: Use this option to enable compilation of C++ with cfront 2.1 compatibility. The compiler then accepts language constructs that, while not part of the C++ language definition, are accepted by the AT&T C++ Language System (cfront release 2.1). This option also enables acceptance of anachronisms. Related options: ––cfront2.1, –b3 , ––cfront3.0, +p, –A –b3 (pgcpp only) Enables compilation of C++ with cfront 3.0 compatibility and acceptance of anachronisms. Default: The compiler does not accept cfront language constructs that are not part of the C++ language definition. Usage: In the following example, the compiler accepts cfront constructs. $ pgcpp -b3 myprog.cc Description: Use this option to enable compilation of C++ with cfront 3.0 compatibility. The compiler then accepts language constructs that, while not part of the C++ language definition, are accepted by the AT&T C++ Language System (cfront release 3.0). This option also enables acceptance of anachronisms. Related options: ––cfront2.1, –b, ––cfront3.0, +p, –A --[no_]bool (pgcpp only) Enables or disables recognition of bool. Default: The compile recognizes bool: --bool. Usage: In the following example, the compiler does not recognize bool. $ pgcpp --no_bool myprog.ccChapter 15. Command-Line Options Reference 211 Description: Use this option to enable or disable recognition of bool. Related options: – –[no_]builtin Compile with or without math subroutine builtin support. Default: The default is to compile with math subroutine support: ––built. Usage: In the following example, the compiler does not build with math subroutine support. $ pgcpp --no_builtin myprog.cc Description: Use this option to enable or disable compiling with math subroutine builtin support. When you compile with math subroutine builtin support, the selected math library routines are inlined. Related options: --cfront_2.1 (pgcpp only) Enables compilation of C++ with cfront 2.1 compatibility and acceptance of anachronisms. Default: The compiler does not accept cfront language constructs that are not part of the C++ language definition. Usage: In the following example, the compiler accepts cfront constructs. $ pgcpp --cfront_2.1 myprog.cc Description: Use this option to enable compilation of C++ with cfront 2.1 compatibility. The compiler then accepts language constructs that, while not part of the C++ language definition, are accepted by the AT&T C++ Language System (cfront release 2.1). This option also enables acceptance of anachronisms. Related options: –b, –b3, ––cfront3.0, +p, –A --cfront_3.0 (pgcpp only) Enables compilation of C++ with cfront 3.0 compatibility and acceptance of anachronisms. Default: The compiler does not accept cfront language constructs that are not part of the C++ language definition. Usage: In the following example, the compiler accepts cfront constructs. $ pgcpp --cfront_3.0 myprog.cc Description: Use this option to enable compilation of C++ with cfront 3.0 compatibility. The compiler then accepts language constructs that, while not part of the C++ language definition, are accepted by the AT&T C++ Language System (cfront release 3.0). This option also enables acceptance of anachronisms.PGI® User’s Guide 212 Related options: ––cfront2.1, –b, –b3, +p, –A --compress_names Compresses long function names in the file. Default: The compiler does not compress names: --no_compress_names. Usage: In the following example, the compiler compresses long function names. $ pgcpp --ccompress_names yprog.cc Description: Use this option to specify to compress long function names. Highly nested template parameters can cause very long function names. These long names can cause problems for older assemblers. Users encountering these problems should compileall C++ code, including library code with the switch -- compress_name. Libraries supplied by PGI work with --compress_names. Related options: --create_pch filename (pgcpp only) If other conditions are satisfied, create a precompiled header file with the specified name. Note If --pch (automatic PCH mode) appears on the command line following this option, its effect is erased. Default: The compiler does not create a precompiled header file. Usage: In the following example, the compiler creates a precompiled header file, hdr1. $ pgcpp --create_pch hdr1 myprog.cc Description: If other conditions are satisfied, use this option to create a precompiled header file with the specified name. Related options: --diag_error tag (pgcpp only) Overrides the normal error severity of the specified diagnostic messages. Default: The compiler does not override normal error severity. Description: Use this option to override the normal error severity of the specified diagnostic messages. The message(s) may be specified using a mnemonic error tag or using an error number. ? Related options:--diag_remark tag, --diag_suppress tag, --diag_warning tag, --display_error_number --diag_remark tag (pgcpp only) Overrides the normal error severity of the specified diagnostic messages.Chapter 15. Command-Line Options Reference 213 Default: The compiler does not override normal error severity. Description: Use this option to override the normal error severity of the specified diagnostic messages. The message(s) may be specified using a mnemonic error tag or using an error number. Related options: --diag_error tag, --diag_suppress tag, --diag_warning tag, --display_error_number --diag_suppress tag (pgcpp only) Overrides the normal error severity of the specified diagnostic messages. Default: The compiler does not override normal error severity. Usage: In the following example, the compiler overrides the normal error severity ofthe specified diagnostic messages.. $ pgcpp --diag_suppress error_tag prog.cc Description: Use this option to override the normal error severity of the specified diagnostic messages. The message(s) may be specified using a mnemonic error tag or using an error number. Related options:--diag_error tag, --diag_remark tag, --diag_warning tag, --diag_error_number --diag_warning tag (pgcpp only) Overrides the normal error severity of the specified diagnostic messages. Default: The compiler does not override normal error severity. Usage: In the following example, the compiler overrides the normal error severity of the specified diagnostic messages. $ pgcpp --diag_suppress an_error_tag myprog.cc Description: Use this option to override the normal error severity of the specified diagnostic messages. The message(s) may be specified using a mnemonic error tag or using an error number. Related options: --diag_error tag, --diag_remark tag, --diag_suppress tag, --diag_error_number --display_error_number (pgcpp only) Displays the error message number in any diagnostic messages that are generated. The option may be used to determine the error number to be used when overriding the severity of a diagnostic message. Default: The compiler does not display error message numbers for generated diagnostic messages. Usage: In the following example, the compiler displays the error message number for any generated diagnostic messages.PLEASE PROVIDE ONE $ pgcpp --display_error_number myprog.cc Description: Use this option to display the error message number in any diagnostic messages that are generated. You can use this option to determine the error number to be used when overriding the severity of a diagnostic message.PGI® User’s Guide 214 Related options: --diag_error tag, --diag_remark tag, --diag_suppress tag, --diag_warning tag -e (pgcpp only) Set the C++ front-end error limit to the specified . --[no_]exceptions (pgcpp only) Enables or disables exception handling support. Default: The compiler provides exception handling support: --exceptions. Usage: In the following example, the compiler does not provide exception handling support. PLEASE PROVIDE ONE $ pgcpp --no_exceptions myprog.cc Description: Use this option to enable or disable exception handling support. Related options: ––gnu_extensions (pgcpp only) Allows GNU extensions. Default: The compiler does not allow GNU extensions. Usage: In the following example, the compiler allows GNU extensions. $ pgcpp --gnu_extensions myprog.cc Description: Use this option to allow GNU extensions, such as “include next”, which are required to compile Linux system header files. Related options: --[no]llalign (pgcpp only) Enables or disables alignment of long long integers on long long boundaries. Default: The compiler aligns long long integers on long long boundaries: --llalign. Usage: In the following example, the compiler does not align long long integers on long long boundaries. $ pgcpp --nollalign myprog.cc Description: Use this option to allow enable or disable alignment of long long integers on long long boundaries. Related options: –M Generates a list of make dependencies and prints them to stdout.Chapter 15. Command-Line Options Reference 215 Note The compilation stops after the preprocessing phase. Default: The compiler does not generate a list of make dependencies. Usage: In the following example, the compiler generates a list of make dependencies. $ pgcpp -M myprog.cc Description: Use this option to generate a list of make dependencies and prints them to stdout. Related options:–MD, –P, –suffix –MD Generates a list of make dependencies and prints them to a file. Default: The compiler does not generate a list of make dependencies. Usage: In the following example, the compiler generates a list of make dependencies and prints them to the file myprog.d. $ pgcpp -MD myprog.cc Description: Use this option to generate a list of make dependencies and prints them to a file. The name of the file is determined by the name of the file under compilation.dependencies_file. Related options:–M, –P, –suffix --optk_allow_dollar_in_id_chars (pgcpp only) Accepts dollar signs ($) in identifiers. Default: The compiler does not accept dollar signs ($) in identifiers. Usage: In the following example, the compiler allows dollar signs ($) in identifiers. $ pgcpp -optk_allow_dollar_in_id_chars myprog.cc Description: Use this option to instruct the compiler to accept dollar signs ($) in identifiers. –P Halts the compilation process after preprocessing and writes the preprocessed output to a file. Default: The compiler produces an executable file. Usage: In the following example, the compiler produces the preprocessed file myprog.i in the current directory. $ pgcpp -P myprog.ccPGI® User’s Guide 216 Description: Use this option to halt the compilation process after preprocessing and write the preprocessed output to a file. If the input file is filename.c or filename.cc., then the output file is filename.i. Note Use the –suffix option with this option to save the intermediate file in a file with the specified suffix. Related options: –C,–c,–E, –Mkeepasm, –o, –S -+p (pgcpp only) Disallow all anachronistic constructs. Default: The compiler disallows all anachronistic constructs. Usage: In the following example, the compiler disallows all anachronistic constructs. $ pgcpp -+p myprog.cc Description: Use this option to disallow all anachronistic constructs. Related options: --pch (pgcpp only) Automatically use and/or create a precompiled header file. Note If --use_pch or --create_pch (manual PCH mode) appears on the command line following this option, this option has no effect. Default: The compiler does not automatically use or create a precompiled header file. Usage: In the following example, the compiler automatically uses a precompiled header file. $ pgcpp --pch myprog.cc Description: Use this option to automatically use and/or create a precompiled header file. Related options: --pch_dir directoryname (pgcpp only) Specifies the directory in which to search for and/or create a precompiled header file. The compiler searches your PATH for precompiled header files / use or create a precompiled header file. Usage: In the following example, the compiler searches in the directory myhdrdir for a precompiled header file. $ pgcpp --pch_dir myhdrdir myprog.ccChapter 15. Command-Line Options Reference 217 Description: Use this option to specify the directory in which to search for and/or create a precompiled header file. You may use this option with automatic PCH mode (--pch) or manual PCH mode (--create_pch or --use_pch). Related options:--create_pch, --pch, --use_pch --[no_]pch_messages (pgcpp only) Enables or disables the display of a message indicating that the current compilation used or created a precompiled header file. The compiler displays a message when it uses or creates a precompiled header file. In the following example, no message is displayed when the precompiled header file located in myhdrdir is used in the compilation. $ pgcpp --pch_dir myhdrdir --no_pch_messages myprog.cc Description: Use this option to enable or disable the display of a message indicating that the current compilation used or created a precompiled header file. Related options:--pch_dir, --preinclude= (pgcpp only) Specifies the name of a file to be included at the beginning of the compilation. In the following example, the compiler includes the file incl_file.c at the beginning of the compilation. me $ pgcpp --preinclude=incl_file.c myprog.cc Description: Use this option to specify the name of a file to be included at the beginning of the compilation. For example, you can use this option to set system-dependent macros and types. Related options: --use_pch filename (pgcpp only) Uses a precompiled header file of the specified name as part of the current compilation. Note If --pch (automatic PCH mode) appears on the command line following this option, its effect is erased. Default: The compiler does not use a precompiled header file. In the following example, the compiler uses the precompiled header file, hdr1 as part of the current compilation. $ pgcpp --use_pch hdr1 myprog.ccPGI® User’s Guide 218 Use a precompiled header file of the specified name as part of the current compilation. If --pch (automatic PCH mode) appears on the command line following this option, its effect is erased. Related options:--create_pch, --pch_dir, --pch_messages --[no_]using_std (pgcpp only) Enables or disables implicit use of the std namespace when standard header files are included. Default:The compiler uses std namespace when standard header files are included: --using_std. Usage: The following command-line disables implicit use of the std namespace: $ pgcpp --no_using_std hello.cc Description: Use this option to enable or disable implicit use of the std namespace when standard header files are included in the compilation. Related options: –t (pgcpp only) Control instantiation of template functions. –t [arg] Default:No templates are instantiated. Usage: In the following example, all templates are instantiated. $ pgcpp -tall myprog.cc Description: Use this option to control instantiation of template functions. The argument is one of the following: all Instantiates all functions whether or not they are used. local Instantiates only the functions that are used in this compilation, and forces those functions to be local to this compilation. Note: This may cause multiple copies of local static variables. If this occurs, the program may not execute correctly. none Instantiates no functions. (this is the default) used Instantiates only the functions that are used in this compilation. Usage: In the following example, all templates are instantiated. $ pgcppChapter 15. Command-Line Options Reference 219 -tall myprog.cc –X (pgcpp only) Generates cross-reference information and places output in the specified file. Syntax: –Xfoo where foo is the specifies file for the cross reference information. Default: The compiler does not generate cross-reference information. Usage: In the following example, the compiler generates cross-reference information, placing it in the file: xreffile. $ pgcpp -Xxreffile myprog.cc Description: Use this option to generate cross-reference information and place output in the specified file. This is an EDG option. Related options: --zc_eh (Linux only) Generates zero-overhead exceptionregions. Default:The compiler does not to use --zc_eh but instead uses --sjlj_eh, which implements exception handling with setjmp and longjmp. Usage: The following command-line enables zero-overhead exception regions: $ pgcpp --zc_eh ello.cc Description: Use this option to generate zero-overhead exception regions. The --zc_eh option defers the cost of exception handling until an exception is thrown. For a program with many exception regions and few throws, this option may lead to improved run-time performance. This option is compatible with C++ code that was compiled with previous version if PGI C++. Note The --zc_eh option is available only on newer Linux systems that supply the system unwind libraries in libgcc_eh and on Windows. Related options: –M Options by Category This section describes each of the options available with –M by the categories: Code generation Fortran Language Controls OptimizationPGI® User’s Guide 220 C/C++ Language Controls Inlining Miscellaneous Environment For a complete alphabetical list of all the options, refer to “ –M Options Summary,” on page 185. The following sections provide detailed descriptions of several, but not all, of the –M options. For a complete alphabetical list of all the options, refer to “ –M Options Summary,” on page 185. These options are grouped according to categories and are listed with exact syntax, defaults, and notes concerning similar or related options. For the latest information and description of a given option, or to see all available options, use the –help command-line option, described in“–help ,” on page 178. –M Code Generation Controls This section describes the –M options that control code generation. Default: For arguments that you do not specify, the default code generation controls are these: nodaz noreentrant nostride0 noflushz noref_externals signextend norecursive nosecond_underscore Related options: –D, –I, –L, –l, –U Syntax: Description and Related Options –Mdaz Set IEEE denormalized input values to zero; there is a performance benefit but misleading results can occur, such as when dividing a small normalized number by a denormalized number. To take effect, this option must be set for the main program. –Mnodaz Do not treat denormalized numbers as zero.To take effect, this option must be set for the main program. –Mdwarf1 Generate DWARF1 format debug information; must be used in combination with –g. –Mdwarf2 Generate DWARF2 format debug information; must be used in combination with –g. –Mdwarf3 Generate DWARF3 format debug information; must be used in combination with –g. –Mflushz Set SSE flush-to-zero mode; if a floating-point underflow occurs, the value is set to zero.To take effect, this option must be set for the main program. –Mnoflushz Do not set SSE flush-to-zero mode; generate underflows.To take effect, this option must be set for the main program.Chapter 15. Command-Line Options Reference 221 –Mfunc32 Align functions on 32-byte boundaries. –Mlarge_arrays Enable support for 64-bit indexing and single static data objects larger than 2GB in size. This option is default in the presence of –mcmodel=medium. Can be used separately together with the default small memory model for certain 64-bit applications that manage their own memory space. For more information, refer to Chapter 11, “Programming Considerations for 64-Bit Environments”. –Mmpi=option -Mmpi adds the include and library options to the compile and link commands necessary to build an MPI application using MPI librariews installed with the PGI Cluister Development Kit (CDK). On Linux, this option inserts -I$MPIDIR/include into the compile line and -L$MPIDIR/lib into the link line. The specifies option determines whether to select MPICH-1 or MPICH-2 headers and libraries. The base directories for MPICH-1 and MPICH-2 are set in localrc. On Windows, this option inserts -I$MCCP_HOME/IncludeIncludeinto the compile line and - L$CCP_HOME/lib into the link line. The -Mmpi options are as specified: • –Mmpi=mpich1 - Selects preconfigured MPICH-1 communication libraries. • –Mmpi=mpich2 - Selects preconfigured MPICH-2 communication libraries. • –Mmpi=msmpi - Select Microsoft MSMPI libraries. Note The user can set the environment variables MPIDIR and MPILIBNAME to override the default values for the MPI directory and library name. MPICH1 and MPICH2 apply only for PGI CDK Cluster Development Kit; MSMPI applies only on Microsoft Compute Cluster systems. For –Mmpi=msmpi to work, the CCP_HOME environment variable must be set. When the Microsoft Compute Cluster SDK is installed, this variable is typically set to point to the MSMPI library directory. –Mnolarge_arrays Disable support for 64-bit indexing and single static data objects larger than 2GB in size. When placed after –mcmodel=medium on the command line, disables use of 64-bit indexing for applications that have no single data object larger than 2GB. –Mnomain Instructs the compiler not to include the object file that calls the Fortran main program as part of the link step. This option is useful for linking programs in which the main program is written in C/C++ and one or more subroutines are written in Fortran (pgf77, pgf95, and pghpf only). –M[no]movnt Instructs the compiler to generate nontemporal move and prefetch instructions even in cases where the compiler cannot determine statically at compile-time that these instructions will be beneficial.PGI® User’s Guide 222 –Mprof[=option[,option,...]] Set performance profiling options. Use of these options causes the resulting executable to create a performance profile that can vbe viewed and analyzed with the PGPROF performance profiler. In the descriptions that follow, PGI-style profiling implies compiler-generated source instrumentation. MPICHstyle profiling implies the use of instrumented wrappers for MPI library routines. The option argument can be any of the following: dwarf Generate limited DWARF symbol information sufficient for most performance profilers. func Perform PGI-style function-level profiling. hwcts Generate a profile using event-based sampling of hardware counters via the PAPI interface. (linux86- 64 platforms only; PAPI must be installed). lines Perform PGI-style line-level profiling. mpich1 Perform MPICH-style profiling for MPICH-1. Implied –Mmpi=mpich1. (Linux only). mpich2 Perform MPICH-style profiling for MPICH-2. Implies –Mmpi=mpich2. (Linux with MPI support licence privileges only.) msmpi Perform MPICH-style profiling for Microsoft MSMPI. Implies –Mmpi=msmpi. (Microsoft Compute Cluster Server only ). For -Mprof=msmpi to work, the CCP_HOME environment variable must be set. This variable is typically set when the Microsoft Compute Cluster SDK is installed. time Generate a profile using time-based instruction-level statistical sampling. This is equivalent to -pg, except that the profile is saved to a file names pgprof.out rather than gmon.out. –Mrecursive instructs the compiler to allow Fortran subprograms to be called recursively. –Mnorecursive Fortran subprograms may not be called recursively. –Mref_externals force references to names appearing in EXTERNAL statements (pgf77, pgf95, and pghpf only). –Mnoref_externals do not force references to names appearing in EXTERNAL statements (pgf77, pgf95, and pghpf only). –Mreentrant instructs the compiler to avoid optimizations that can prevent code from being reentrant.Chapter 15. Command-Line Options Reference 223 –Mnoreentrant instructs the compiler not to avoid optimizations that can prevent code from being reentrant. –Msecond_underscore instructs the compiler to add a second underscore to the name of a Fortran global symbol if its name already contains an underscore. This option is useful for maintaining compatibility with object code compiled using g77, which uses this convention by default (pgf77, pgf95, and pghpf only). –Mnosecond_underscore instructs the compiler not to add a second underscore to the name of a Fortran global symbol if its name already contains an underscore (pgf77, pgf95, and pghpf only). –Msignextend instructs the compiler to extend the sign bit that is set as a result of converting an object of one data type to an object of a larger signed data type. –Mnosignextend instructs the compiler not to extend the sign bit that is set as the result of converting an object of one data type to an object of a larger data type. –Msafe_lastval In the case where a scalar is used after a loop, but is not defined on every iteration of the loop, the compiler does not by default parallelize the loop. However, this option tells the compiler it’s safe to parallelize the loop. For a given loop the last value computed for all scalars make it safe to parallelize the loop. –Mstride0 instructs the compiler to inhibit certain optimizations and to allow for stride 0 array references. This option may degrade performance and should only be used if zero-stride induction variables are possible. –Mnostride0 instructs the compiler to perform certain optimizations and to disallow for stride 0 array references. –Munix use UNIX symbol and parameter passing conventions for Fortran subprograms (pgf77, pgf95, and pghpf for Win32 only). –Mvarargs force Fortran program units to assume procedure calls are to C functions with a varargs-type interface (pgf77 and pgf95 only). –M C/C++ Language Controls This section describes the –M options that affect C/C++ language interpretations by the PGI C and C++ compilers. These options are only valid to the pgcc and pgcpp compiler drivers. Default: For arguments that you do not specify, the defaults are as follows: noasmkeyword nosingle dollar,_ schar Usage:PGI® User’s Guide 224 In this example, the compiler allows the asm keyword in the source file. $ pgcc -Masmkeyword myprog.c In the following example, the compiler maps the dollar sign to the dot character. $ pgcc -Mdollar,. myprog.c In the following example, the compiler treats floating-point constants as float values. $ pgcc -Mfcon myprog.c In the following example, the compiler does not convert float parameters to double parameters. $ pgcc -Msingle myprog.c Without –Muchar or with –Mschar, the variable ch is a signed character: char ch; signed char sch; If –Muchar is specified on the command line: $ pgcc -Muchar myprog.c char ch above is equivalent to: unsigned char ch; Syntax: Description and Related Options –Masmkeyword instructs the compiler to allow the asm keyword in C source files. The syntax of the asm statement is as follows: asm("statement"); Where statement is a legal assembly-language statement. The quote marks are required. Note. The current default is to support gcc's extended asm, where the syntax of extended asm includes asm strings. The –M[no]asmkeyword switch is useful only if the target device is a Pentium 3 or older cpu type (–tp piii|p6|k7|athlon|athlonxp|px). –Mnoasmkeyword instructs the compiler not to allow the asm keyword in C source files. If you use this option and your program includes the asm keyword, unresolved references will be generated –Mdollar,char char specifies the character to which the compiler maps the dollar sign ($). The PGCC compiler allows the dollar sign in names; ANSI C does not allow the dollar sign in names. –Mfcon instructs the compiler to treat floating-point constants as float data types, instead of double data types. This option can improve the performance of single-precision code. –Mschar specifies signed char characters. The compiler treats "plain" char declarations as signed char.Chapter 15. Command-Line Options Reference 225 –Msingle do not to convert float parameters to double parameters in non-prototyped functions. This option can result in faster code if your program uses only float parameters. However, since ANSI C specifies that routines must convert float parameters to double parameters in non-prototyped functions, this option results in non#ANSI conformant code. –Mnosingle instructs the compiler to convert float parameters to double parameters in non-prototyped functions. –Muchar instructs the compiler to treat "plain" char declarations as unsigned char. –M Environment Controls This section describes the –M options that control environments. Default: For arguments that you do not specify, the default environment option depends on your configuration. Syntax: Description and Related Options –Mlfs (32-bit Linux only) link in libraries that enable file I/O to files larger than 2GB (Large File Support). –Mnostartup instructs the linker not to link in the standard startup routine that contains the entry point (_start) for the program. Note If you use the –Mnostartup option and do not supply an entry point, the linker issues the following error message: Warning: cannot find entry symbol _start –M[no]smartalloc[=huge|h[uge:|hugebss] adds a call to the routine mallopt in the main routine. This option supports large TLBs on Linux and Windows. This option must be used to compile the main routine to enable optimized malloc routines. The option arguments can be any of the following: huge Link in the huge page runtime library Enables large 2-megabyte pages to be allocated. The effect is to reduce the number of TLB entries required to execute a program. This option is most effective on Barcelona and Core 2 systems; older architectures do not have enough TLB entries for this option to be benefitical. By itself, the huge suboption tries to allocate as many huge pages as required. huge: Link the huge page runtime library and allocate n huge pages. Use this suboption to limit the number of huge pages allocated to n.PGI® User’s Guide 226 You can also limit the pages allocated by using the environment variable PGI_HUGE_PAGES. hugebss Puts the BSS section in huge pages; attempts to put a program's unititlaized data section into huge pages. Tip. To be effective, this switch must be specified when compiling the file containing the Fortran, C, or C++ main program. –M[no]stddef instructs the compiler not to predefine any macros to the preprocessor when compiling a C program. –Mnostdinc instructs the compiler to not search the standard location for include files. –Mnostdlib instructs the linker not to link in the standard libraries libpgftnrtl.a, libm.a, libc.a and libpgc.a in the library directory lib within the standard directory. You can link in your own library with the –l option or specify a library directory with the –L option. –M Fortran Language Controls This section describes the –M options that affect Fortran language interpretations by the PGI Fortran compilers. These options are valid only for the pghpf, pgf77 and pgf95 compiler drivers. Default: For arguments that you do not specify, the defaults are as follows: nobackslash noiomutex nodclchk noonetrip nodefaultunit nosave nodlines nounixlogical dollar,_ noupcase Syntax: Description and Related Options –Mallocatable=95|03 controls whether Fortran 95 or Fortran 2003 semantics are used in allocatable array assignments. The default behavior is to use Fortran 95 semantics; the 03 option instructs the compiler to use Fortran 2003 semantics. –Mbackslash the compiler treats the backslash as a normal character, and not as an escape character in quoted strings. –Mnobackslash the compiler recognizes a backslash as an escape character in quoted strings (in accordance with standard C usage). –Mdclchk the compiler requires that all program variables be declared.Chapter 15. Command-Line Options Reference 227 –Mnodclchk the compiler does not require that all program variables be declared. –Mdefaultunit the compiler treats "*" as a synonym for standard input for reading and standard output for writing. –Mnodefaultunit the compiler treats "*" as a synonym for unit 5 on input and unit 6 on output. –Mdlines the compiler treats lines containing "D" in column 1 as executable statements (ignoring the "D"). –Mnodlines the compiler does not treat lines containing "D" in column 1 as executable statements (does not ignore the "D"). –Mdollar,char char specifies the character to which the compiler maps the dollar sign. The compiler allows the dollar sign in names. –Mextend with –Mextend, the compiler accepts 132-column source code; otherwise it accepts 72-column code. –Mfixed with –Mfixed, the compiler assumes input source files are in FORTRAN 77-style fixed form format. –Mfree with –Mfree, the compiler assumes the input source files are in Fortran 90/95 freeform format. –Miomutex the compiler generates critical section calls around Fortran I/O statements. –Mnoiomutex the compiler does not generate critical section calls around Fortran I/O statements. –Monetrip the compiler forces each DO loop to execute at least once. –Mnoonetrip the compiler does not force each DO loop to execute at least once. This option is useful for programs written for earlier versions of Fortran. –Msave the compiler assumes that all local variables are subject to the SAVE statement. Note that this may allow older Fortran programs to run, but it can greatly reduce performance. –Mnosave the compiler does not assume that all local variables are subject to the SAVE statement. –Mstandard the compiler flags non-ANSI–conforming source code. –Munixlogical directs the compiler to treat logical values as true if the value is non-zero and false if the value is zero (UNIX F77 convention.) When –Munixlogical is enabled, a logical value or test that is non-zero isPGI® User’s Guide 228 .TRUE., and a value or test that is zero is .FALSE.. In addition, the value of a logical expression is guaranteed to be one (1) when the result is .TRUE.. –Mnounixlogical Directs the compiler to use the VMS convention for logical values for true and false. Even values are true and odd values are false. –Mupcase the compiler allows uppercase letters in identifiers. With –Mupcase, the identifiers "X" and "x" are different, and keywords must be in lower case. This selection affects the linking process: if you compile and link the same source code using –Mupcase on one occasion and –Mnoupcase on another, you may get two different executables (depending on whether the source contains uppercase letters). The standard libraries are compiled using the default –Mnoupcase. –Mnoupcase the compiler converts all identifiers to lower case. This selection affects the linking process: If you compile and link the same source code using –Mupcase on one occasion and –Mnoupcase on another, you may get two different executables (depending on whether the source contains uppercase letters). The standard libraries are compiled using –Mnoupcase. –M Inlining Controls This section describes the –M options that control function inlining. Before looking at all the options, let’s look at a couple examples. Usage: In the following example, the compiler extracts functions that have 500 or fewer statements from the source file myprog.f and saves them in the file extract.il. $ pgf95 -Mextract=500 -oextract.il myprog.f In the following example, the compiler inlines functions with fewer than approximately 100 statements in the source file myprog.f and writes the executable code in the default output file a.out. $ pgf95 -Minline=size:100 myprog.f Related options: –o, –Mextract Syntax: Description and Related Options –M[no]autoinline instructs the compiler to inline a C/C++ function at –O2 and above when it is declared with the inline keyword. –Mextract[=option[,option,...]] Extracts functions from the file indicated on the command line and creates or appends to the specified extract directory where option can be any of: name:func instructs the extractor to extract function func from the file. size:number instructs the extractor to extract functions with number or fewer, statements from the file.Chapter 15. Command-Line Options Reference 229 lib:filename.ext Use directory filename.ext as the extract directory (required in order to save and re-use inline libraries). If you specify both name and size, the compiler extracts functions that match func, or that have number or fewer statements. For examples of extracting functions, see Chapter 4, “Using Function Inlining”. –Minline[=option[,option,...]] This passes options to the function inliner, where the option can be any of these: except:func instructs the inliner to inline all eligible functions except func, a function in the source text. Multiple functions can be listed, comma-separated. [name:]func instructs the inliner to inline the function func. The func name should be a non-numeric string that does not contain a period. You can also use a name: prefix followed by the function name. If name: is specified, what follows is always the name of a function. [lib:]filename.ext instructs the inliner to inline the functions within the library file filename.ext. The compiler assumes that a filename.ext option containing a period is a library file. Create the library file using the –Mextract option. You can also use a lib: prefix followed by the library name. If lib: is specified, no period is necessary in the library name. Functions from the specified library are inlined. If no library is specified, functions are extracted from a temporary library created during an extract prepass. levels:number instructs the inliner to perform number levels of inlining. The default number is 1. [no]reshape instructs the inliner to allow (disallow)inlining in Fortran even when array shapes do not match. The default is -Minline=noreshape, except with -Mconcur or -mp, where the default is -Minline=reshape. [size:]number instructs the inliner to inline functions with number or fewer statements. You can also use a size: prefix followed by a number. If size: is specified, what follows is always taken as a number. If you specify both func and number, the compiler inlines functions that match the function name or have number or fewer statements. For examples of inlining functions, refer to Chapter 4, “Using Function Inlining”. –M Optimization Controls This section describes the –M options that control optimization. Before looking at all the options, let’s look at the defaults. Default: For arguments that you do not specify, the default optimization control options are as follows: depchk noipa nounroll nor8 i4 nolre novect nor8intrinsics nofprelaxed noprefetchPGI® User’s Guide 230 Note If you do not supply an option to –Mvect, the compiler uses defaults that are dependent upon the target system. Usage: In this example, the compiler invokes the vectorizer with use of packed SSE instructions enabled. $ pgf95 -Mvect=sse -Mcache_align myprog.f Related options: –g, –O Syntax: Description and Related Options –Mcache_align Align unconstrained objects of length greater than or equal to 16 bytes on cache-line boundaries. An unconstrained object is a data object that is not a member of an aggregate structure or common block. This option does not affect the alignment of allocatable or automatic arrays. Note: To effect cache-line alignment of stack-based local variables, the main program or function must be compiled with –Mcache_align. –Mconcur[=option [,option,...]] Instructs the compiler to enable auto-concurrentization of loops. If –Mconcur is specified, multiple processors will be used to execute loops that the compiler determines to be parallelizable. Where option is one of the following: [no]altcode:n Instructs the parallelizer to generate alternate serial code for parallelized loops. If altcode is specified without arguments, the parallelizer determines an appropriate cutoff length and generates serial code to be executed whenever the loop count is less than or equal to that length. If altcode:n is specified, the serial altcode is executed whenever the loop count is less than or equal to n. If noaltcode is specified, the parallelized version of the loop is always executed regardless of the loop count. cncall Calls in parallel loops are safe to parallelize. Loops containing calls are candidates for parallelization. Also, no minimum loop count threshold must be satisfied before parallelization will occur, and last values of scalars are assumed to be safe. dist:block Parallelize with block distribution (this is the default). Contiguous blocks of iterations of a parallelizable loop are assigned to the available processors. dist:cyclic Parallelize with cyclic distribution. The outermost parallelizable loop in any loop nest is parallelized. If a parallelized loop is innermost, its iterations are allocated to processors cyclically. For example, if there are 3 processors executing a loop, processor 0 performs iterations 0, 3, 6, etc.; processor 1 performs iterations 1, 4, 7, etc.; and processor 2 performs iterations 2, 5, 8, etc. [no]innermost Enable parallelization of innermost loops. The default is to not parallelize innermost loops, since it is usually not profitable on dual-core processors.Chapter 15. Command-Line Options Reference 231 noassoc Disables parallelization of loops with reductions. When linking, the –Mconcur switch must be specified or unresolved references will result. The NCPUS environment variable controls how many processors or cores are used to execute parallelized loops. Note This option applies only on shared-memory multi-processor (SMP) or multi-core processorbased systems. –Mcray[=option[,option,...]] (pgf77 and pgf95 only) Force Cray Fortran (CF77) compatibility with respect to the listed options. Possible values of option include: pointer for purposes of optimization, it is assumed that pointer-based variables do not overlay the storage of any other variable. –Mdepchk instructs the compiler to assume unresolved data dependencies actually conflict. –Mnodepchk instructs the compiler to assume potential data dependencies do not conflict. However, if data dependencies exist, this option can produce incorrect code. –Mdse Enables a dead store elimination phase that is useful for programs that rely on extensive use of inline function calls for performance. This is disabled by default. –Mnodse (default) Disables the dead store elimination phase. –M[no]fpapprox[=option] Perform certain fp operations using low-precision approximation. By default -Mfpapprox is not used. If -Mfpapprox is used without suboptions, it defaults to use approximate div, sqrt, and rsqrt. The available suboptions are these: div Approximate floating point division sqrt Approximate floating point square root rsqrt Approximate floating point reciprocal square root –M[no]fpmisalign Instructs the compiler to allow (not allow) vector arithmetic instructions with memory operands that are not aligned on 16-byte boundaries. The default is -Mnofpmisalign on all processors.PGI® User’s Guide 232 Note Applicable only with one of these options: –tp barcelona or –tp barcelona-64 –Mfprelaxed[=option] instructs the compiler to use relaxed precision in the calculation of some intrinsic functions. Can result in improved performance at the expense of numerical accuracy. The possible values for option are: div Perform divide using relaxed precision. noorder Perform reciprocal square root (1/sqrt) using relaxed precision. order Perform reciprocal square root (1/sqrt) using relaxed precision. rsqrt Perform reciprocal square root (1/sqrt) using relaxed precision. sqrt Perform square root with relaxed precision. With no options, –Mfprelaxed generates relaxed precision code for those operations that generate a significant performance improvement, depending on the target processor. –Mnofprelaxed (default) instructs the compiler not to use relaxed precision in the calculation of intrinsic functions. –Mi4 (pgf77 and pgf95 only) the compiler treats INTEGER variables as INTEGER*4. –Mipa=[,[,…]] Pass options to the interprocedural analyzer. Note: –Mipa implies –O2, and the minimum optimization level that can be specified in combination with –Mipa is –O2. For example, if you specify –Mipa –O1 on the command line, the optimization level will automatically be elevated to –O2 by the compiler driver. It is typical and recommended to use –Mipa=fast. Many of the following sub-options can be prefaced with no, which reverses or disables the effect of the sub-option if it’s included in an aggregate sub-option like –Mipa=fast. The choices of option are: [no]align recognize when targets of a pointer dummy are aligned; default is noalign. [no]arg remove arguments replaced by const, ptr; default is noarg. [no]cg generate call graph information for viewing using the pgicg command-line utility; default is nocg. [no]const perform interprocedural constant propagation; default is const.Chapter 15. Command-Line Options Reference 233 except: used with inline to specify functions which should not be inlined; default is to inline all eligible functions according to internally defined heuristics. [[no]f90ptr F90/F95 pointer disambiguation across calls; default is nof90ptr fast choose IPA options generally optimal for the target. Use –help to see the settings for –Mipa=fast on a given target. force force all objects to re-compile regardless of whether IPA information has changed. [no]globals optimize references to global variables; default is noglobals. inline[:n] perform automatic function inlining. If the optional :n is provided, limit inlining to at most n levels. IPA-based function inlining is performed from leaf routines upward. ipofile save IPA information in a .ipo file rather than incorporating it into the object file. [no]keepobj keep the optimized object files, using file name mangling, to reduce re-compile time in subsequent builds default is keepobj. [no]libc optimize calls to certain standard C library routines.; default is nolibc. [no]libinline allow inlining of routines from libraries; implies –Mipa=inline; default is nolibinline. [no]libopt allow recompiling and optimization of routines from libraries using IPA information; default is nolibopt. [no]localarg equivalent to arg plus externalization of local pointer targets; default is nolocalarg. main: specify a function to appear as a global entry point; may appear multiple times; disables linking. [no]ptr enable pointer disambiguation across procedure calls; default is noptr. [no]pure pure function detection; default is nopure. required return an error condition if IPA is inhibited for any reason, rather than the default behavior of linking without IPA optimization.PGI® User’s Guide 234 [no]reshape enables or disables Fortran inline with mismatched array shapes. safe:[|] declares that the named function, or all functions in the named library, are safe; a safe procedure does not call back into the known procedures and does not change any known global variables. Without –Mipa=safe, any unknown procedures will cause IPA to fail. [no]safeall declares that all unknown procedures are safe; see –Mipa=safe; default is nosafeall. [no]shape perform Fortran 90 array shape propagation; default is noshape. summary only collect IPA summary information when compiling; this prevents IPA optimization of this file, but allows optimization for other files linked with this file. [no]vestigial remove uncalled (vestigial) functions; default is novestigial. –M[no]loop32 Aligns or does not align innermost loops on 32 byte boundaries with –tp barcelona. Small loops on barcelona may run fast if aligned on 32-byte boundaries; however, in practice, most assemblers do not yet implement efficient padding causing some programs to run more slowly with this default. Use -Mloop32 on systems with an assembler tuned for barcleona. The default is -Mnoloop32. –Mlre[=array | assoc | noassoc] Enables loop-carried redundancy elimination, an optimization that can reduce the number of arithmetic operations and memory references in loops. array treat individual array element references as candidates for possible loop-carried redundancy elimination. The default is to eliminate only redundant expressions involving two or more operands. assoc allow expression re-association; specifying this sub-option can increase opportunities for loop-carried redundancy elimination but may alter numerical results. noassoc disallow expression re-association. –Mnolre Disables loop-carried redundancy elimination. –Mnoframe Eliminates operations that set up a true stack frame pointer for every function. With this option enabled, you cannot perform a traceback on the generated code and you cannot access local variables. –Mnoi4 (pgf77 and pgf95 only) the compiler treats INTEGER variables as INTEGER*2.Chapter 15. Command-Line Options Reference 235 –Mpfi generate profile-feedback instrumentation; this includes extra code to collect run-time statistics and dump them to a trace file for use in a subsequent compilation. –Mpfi must also appear when the program is linked. When the resulting program is executed, a profile feedback trace file pgfi.out is generated in the current working directory; see –Mpfo. Note compiling and linking with –Mpfi adds significant runtime overhead to almost any executable; you should use executables compiled with –Mpfi only for execution of training runs. –Mpfo enable profile-feedback optimizations; requires the presence of a pgfi.out profile-feedback trace file in the current working directory. See –Mpfi. –Mprefetch[=option [,option...]] enables generation of prefetch instructions on processors where they are supported. Possible values for option include: d:m set the fetch-ahead distance for prefetch instructions to m cache lines. n:p set the maximum number of prefetch instructions to generate for a given loop to p. nta use the prefetchnta instruction. plain use the prefetch instruction (default). t0 use the prefetcht0 instruction. w use the AMD-specific prefetchw instruction. –Mnoprefetch Disables generation of prefetch instructions. –M[no]propcond Enables or disables constant propagation from assertions derived from equality conditionals. The default is enabled. –Mr8 (pgf77, pgf95 and pghpf only) the compiler promotes REAL variables and constants to DOUBLE PRECISION variables and constants, respectively. DOUBLE PRECISION elements are 8 bytes in length. –Mnor8 (pgf77, pgf95 and pghpf only) the compiler does not promote REAL variables and constants to DOUBLE PRECISION. REAL variables will be single precision (4 bytes in length).PGI® User’s Guide 236 –Mr8intrinsics (pgf77, and pgf95 only) the compiler treats the intrinsics CMPLX and REAL as DCMPLX and DBLE, respectively. –Mnor8intrinsics (pgf77, and pgf95 only) the compiler does not promote the intrinsics CMPLX and REAL to DCMPLX and DBLE, respectively. –Msafeptr[=option[,option,...]] (pgcc and pgcpp only) instructs the C/C++ compiler to override data dependencies between pointers of a given storage class. Possible values of option include: all assume all pointers and arrays are independent and safe for aggressive optimizations, and in particular that no pointers or arrays overlap or conflict with each other. arg instructs the compiler that arrays and pointers are treated with the same copyin and copyout semantics as Fortran dummy arguments. global instructs the compiler that global or external pointers and arrays do not overlap or conflict with each other and are independent. local/auto instructs the compiler that local pointers and arrays do not overlap or conflict with each other and are independent. static instructs the compiler that static pointers and arrays do not overlap or conflict with each other and are independent. –Mscalarsse Use SSE/SSE2 instructions to perform scalar floating-point arithmetic (this option is valid only on –tp {p7 | k8-32 | k8-64} targets). –Mnoscalarsse Do not use SSE/SSE2 instructions to perform scalar floating-point arithmetic; use x87 instructions instead (this option is not valid in combination with the –tp k8-64 option). –Msmart instructs the compiler driver to invoke a post-pass assembly optimization utility. –Mnosmart instructs the compiler not to invoke an AMD64-specific post-pass assembly optimization utility. –M[no]traceback Adds debug information for runtime traceback for use with the environment variable $PGI_TERM. By default, traceback is enabled for f77 and f90 and disabled for C and C++. Setting setTRACEBACK=OFF; in siterc or .mypg*rc also disables default traceback. Using ON instead of OFF enables default traceback.Chapter 15. Command-Line Options Reference 237 –Munroll[=option [,option...]] invokes the loop unroller to executing multiple instances of the loop during each iteration. This also sets the optimization level to 2 if the level is set to less than 2, or if no –O or –g options are supplied. The option is one of the following: c:m instructs the compiler to completely unroll loops with a constant loop count less than or equal to m, a supplied constant. If this value is not supplied, the m count is set to 4. m: instructs the compiler to unroll multi- block loops n times. This option is useful for loops that have conditional statements. If n is not supplied, the default value is 4. The default setting is not to enable - Munroll=m. n: instructs the compiler to unroll single-block loops n times, a loop that is not completely unrolled, or has a non-constant loop count. If n is not supplied, the unroller computes the number of times a candidate loop is unrolled. –Mnounroll instructs the compiler not to unroll loops. -M[no]vect[=option [,option,...]] (disable) enable the code vectorizer, where option is one of the following: altcode Instructs the vectorizer to generate alternate code (altcode) for vectorized loops when appropriate. For each vectorized loop the compiler decides whether to generate altcode and what type or types to generate, which may be any or all of: altcode without iteration peeling, altcode with non-temporal stores and other data cache optimizations, and altcode based on array alignments calculated dynamically at runtime. The compiler also determines suitable loop count and array alignment conditions for executing the altcode. This option is enabled by default. noaltcode This disables alternate code generation for vectorized loops. assoc Instructs the vectorizer to enable certain associativity conversions that can change the results of a computation due to roundoff error. A typical optimization is to change an arithmetic operation to an arithmetic operation that is mathematically correct, but can be computationally different, due to round-off error noassoc Instructs the vectorizer to disable associativity conversions. cachesize:n Instructs the vectorizer, when performing cache tiling optimizations, to assume a cache size of n. The default is set using per-processor type, either using the –tp switch or auto-detected from the host computer. [no]gather Vectorize loops containing indirect array references, such as this one:PGI® User’s Guide 238 sum = 0.d0 do k=d(j),d(j+1)-1 sum = sum + a(k)*b(c(k)) enddo The default is -Mvect=gather. [no]sizelimit Generate vector code for all loops where possible regardless of the number of statements in the loop. This overrides a heuristic in the vectorizer that ordinarily prevents vectorization of loops with a number of statements that exceeds a certain threshold. The default is nosizelimit. smallvect[:n] Instructs the vectorizer to assume that the maximum vector length is less than or equal to n. The vectorizer uses this information to eliminate generation of the stripmine loop for vectorized loops wherever possible. If the size n is omitted, the default is 100. Note: No space is allowed on either side of the colon (:). sse Instructs the vectorizer to search for vectorizable loops and, wherever possible, make use of SSE, SSE2, and prefetch instructions. –Mnovect instructs the compiler not to perform vectorization; can be used to override a previous instance of –Mvect on the command-line, in particular for cases where –Mvect is included in an aggregate option such as –fastsse. –Mnovintr instructs the compiler not to perform idiom recognition or introduce calls to hand-optimized vector functions. –M Miscellaneous Controls Default: For arguments that you do not specify, the default miscellaneous options are as follows: inform nobounds nolist warn Usage: In the following example, the compiler includes Fortran source code with the assembly code. $ pgf95 -Manno -S myprog.f In the following example, the compiler displays information about inlined functions with fewer than approximately 20 source lines in the source file myprog.f. $ pgf95 -Minfo=inline -Minline=20 myprog.f In the following example, the assembler does not delete the assembly file myprog.s after the assembly pass. $ pgf95 -Mkeepasm myprog.f In the following example, the compiler creates the listing file myprog.lst.Chapter 15. Command-Line Options Reference 239 $ pgf95 -Mlist myprog.f In the following example, array bounds checking is enabled. $ pgf95 -Mbounds myprog.f Related options: –m, –S, –V, –v Syntax: Description and Related Options –Manno annotate the generated assembly code with source code when either the –S or –Mkeepasm options are used. –Mbounds enables array bounds checking. If an array is an assumed size array, the bounds checking only applies to the lower bound. If an array bounds violation occurs during execution, an error message describing the error is printed and the program terminates. The text of the error message includes the name of the array, the location where the error occurred (the source file and the line number in the source), and information about the out of bounds subscript (its value, its lower and upper bounds, and its dimension). For example: PGFTN-F-Subscript out of range for array a (a.f: 2) subscript=3, lower bound=1, upper bound=2, dimension=2 –Mnobounds disables array bounds checking. –Mbyteswapio swap byte-order from big-endian to little-endian or vice versa upon input/output of Fortran unformatted data files. –Mchkfpstk (32-bit only) instructs the compiler to check for internal consistency of the x87 floating-point stack in the prologue of a function and after returning from a function or subroutine call. Floating-point stack corruption may occur in many ways, one of which is Fortran code calling floating-point functions as subroutines (i.e., with the CALL statement). If the PGI_CONTINUE environment variable is set upon execution of a program compiled with –Mchkfpstk, the stack will be automatically cleaned up and execution will continue. There is a performance penalty associated with the stack cleanup. If PGI_CONTINUE is set to verbose, the stack will be automatically cleaned up and execution will continue after printing of a warning message. Note This switch is only valid for 32-bit. On 64-bit it is ignored. –Mchkptr instructs the compiler to check for pointers that are de-referenced while initialized to NULL (pgf95 and pghpf only). –Mchkstk instructs the compiler to check the stack for available space in the prologue of a function and before the start of a parallel region. Prints a warning message and aborts the program gracefully if stack space is insufficient. Useful when many local and private variables are declared in an OpenMP program.PGI® User’s Guide 240 If the user also sets the PGI_STACK_USAGE environment variable to any value, then the program displays the stack space allocated and used after the program exits. For example, you might see something similar the following message: thread 0 stack: max 8180KB, used 48KB This message indicates that the program used 48KB of a 8180KB allocated stack. For more information on the PGI_STACK_USAGE, refer to“PGI_STACK_USAGE,” on page 97. This information is useful when you want to explicitly set a reserved and committed stack size for your programs, such as using the –stack option on Windows. –Mcpp[=option [,option,...]] run the PGI cpp-like preprocessor without execution of any subsequent compilation steps. This option is useful for generating dependence information to be included in makefiles. Note Only one of the m, md, mm or mmd options can be present; if multiple of these options are listed, the last one listed is accepted and the others are ignored. The option is one or more of the following: m print makefile dependencies to stdout. md print makefile dependencies to filename.d, where filename is the root name of the input file being processed. mm print makefile dependencies to stdout, ignoring system include files. mmd print makefile dependencies to filename.d, where filename is the root name of the input file being processed, ignoring system include files. [no]comment (don’t) retain comments in output. [suffix:] use as the suffix of the output file containing makefile dependencies. –Mdll (Windows only) link with the DLL versions of the runtime libraries. This flag is required when linking with any DLL built by any of The Portland Group compilers. This option implies –D_DLL, which defines the preprocessor symbol _DLL. –Mgccbug[s] match the behavior of certain gcc bugs. –Minfo[=option [,option,...]] instructs the compiler to produce information on standard error, where option is one of the following:Chapter 15. Command-Line Options Reference 241 all instructs the compiler to produce all available –Minfo information. [no]file instructs the compiler to print or not print source file names as they are compiled. The default is to print the names, -Minfo=file. inline instructs the compiler to display information about extracted or inlined functions. This option is not useful without either the –Mextract or –Minline option. ipa instructs the compiler to display information about interprocedural optimizations. loop instructs the compiler to display information about loops, such as information on vectorization. opt instructs the compiler to display information about optimization. mp instructs the compiler to display information about parallelization. time instructs the compiler to display compilation statistics. unroll instructs the compiler to display information about loop unrolling. –Mneginfo[=option [,option,...]] instructs the compiler to produce information on standard error, where option is one of the following: all instructs the compiler to produce all available information on why various optimizations are not performed. concur instructs the compiler to produce all available information on why loops are not automatically parallelized. In particular, if a loop is not parallelized due to potential data dependence, the variable(s) that cause the potential dependence will be listed in the –Mneginfo messages. loop instructs the compiler to produce information on why memory hierarchy optimizations on loops are not performed. –Minform,level instructs the compiler to display error messages at the specified and higher levels, where level is one of the following: fatal instructs the compiler to display fatal error messages. inform instructs the compiler to display all error messages (inform, warn, severe and fatal).PGI® User’s Guide 242 severe instructs the compiler to display severe and fatal error messages. warn instructs the compiler to display warning, severe and fatal error messages. –Mkeepasm instructs the compiler to keep the assembly file as compilation continues. Normally, the assembler deletes this file when it is finished. The assembly file has the same filename as the source file, but with a .s extension. –Mlist instructs the compiler to create a listing file. The listing file is filename.lst, where the name of the source file is filename.f. –Mmakedll (Windows only) generate a dynamic link library (DLL). –Mmakeimplib (Windows only) when used without -def:deffile, passes the -def switch to the librarian without a deffile. –Mnolist the compiler does not create a listing file. This is the default. –Mnoopenmp when used in combination with the –mp option, causes the compiler to ignore OpenMP parallelization directives or pragmas, but still process SGI-style parallelization directives or pragmas. –Mnosgimp when used in combination with the –mp option, causes the compiler to ignore SGI-style parallelization directives or pragmas, but still process OpenMP parallelization directives or pragmas. –Mnopgdllmain (Windows only) do not link the module containing the default DllMain() into the DLL. This flag applies to building DLLs with the PGF95 and PGHPF compilers. If you want to replace the default DllMain() routine with a custom DllMain(), use this flag and add the object containing the custom DllMain() to the link line. The latest version of the default DllMain() used by PGF95 and PGHPF is included in the Release Notes for each release; the PGF95- and PGHPF-specific code in this routine must be incorporated into the custom version of DllMain() to ensure the appropriate function of your DLL. –Mnorpath ( Linux only) Do not add –rpath to the link line. –Mpreprocess perform cpp-like preprocessing on assembly and Fortran input source files.243 Chapter 16. OpenMP Reference Information The PGF77 and PGF95 Fortran compilers support the OpenMP Fortran Application Program Interface. The PGCC ANSI C and C++ compilers support the OpenMP C/C++ Application Program Interface. This chapter contains detailed descriptions of each of the OpenMP Fortran directives and C/C++ pragmas that PGI supports. In addition, the section“Directive and Pragma Clauses,” on page 260 contains information about the clauses associated with the directives and pragmas. Parallelization Directives and Pragmas Parallelization directives, as described in Chapter 5, “Using OpenMP”, are comments in a program that are interpreted by the PGI Fortran compilers when the option -mp is specified on the command line. The form of a parallelization directive is: sentinel directive_name [clauses] Parallelization pragmas are #pragma statements in a C or C++ program that are interpreted by the PGCC C and C++ compilers when the option -mp is specified on the command line. The form of a parallelization pragma is: #pragma omp pragma_name [clauses] The examples given with each section use the routines omp_get_num_threads() and omp_get_thread_num(). For more information, refer to “Run-time Library Routines,” on page 55. They return the number of threads currently in the team executing the parallel region and the thread number within the team, respectively. Note Directives which are presented in pairs must be used in pairs. This section describes the details of these directives and pragmas that were summarized in Chapter 5, “Using OpenMP”. For each directive and pragma, this section describes the overall purpose, the syntax, the clauses associated with it, the usage, and examples of how to use it.PGI® User’s Guide 244 ATOMIC The OpenMP ATOMIC directive is semantically equivalent to a single statement in a CRITICAL...END CRITICAL directive or the omp critical pragma. !$OMP ATOMIC Syntax: !$OMP ATOMIC #pragma omp atomic < C/C++ expression statement > Usage: The ATOMIC directive is semantically equivalent to enclosing the following single statement in a CRITICAL / END CRITICAL directive pair. The omp atomic pragma is semantically equivalent to subjecting the following single C/C++ expression statement to an omp critical pragma. The statements must be one of the following forms: For Directives: x = x operator expr x = expr operator x x = intrinsic (x, expr) x = intrinsic (expr, x) For Pragmas: x = expr x++ ++x x-- --x where x is a scalar variable of intrinsic type, expr is a scalar expression that does not reference x, intrinsic is one of MAX, MIN, IAND, IOR, or IEOR, operator is one of +, *, -, /, .AND., .OR., .EQV., or .NEQV., and is not overloaded and is one of +, *, -, /, &, ^, |, << or >>. BARRIER The OpenMP BARRIER directive defines a point in a program where each thread waits for all other threads to arrive before continuing with program execution. Syntax: !$OMP BARRIER #pragma omp barrier Usage: There may be occasions in a parallel region, when it is necessary that all threads complete work to that point before any thread is allowed to continue. The BARRIER directive or omp barrier pragma synchronizes all threads at such a point in a program. Multiple barrier points are allowed within a parallel region. The BARRIER directive and omp barrier pragma must either be executed by all threads executing the parallel region or by none of them.Chapter 16. OpenMP Reference Information 245 CRITICAL ... END CRITICAL and omp critical The CRITICAL ...END CRITICAL directive and omp critical pragma require a thread to wait until no other thread is executing within a critical section. Syntax: !$OMP CRITICAL [(name)] < Fortran code executed in body of critical section > !$OMP END CRITICAL [(name)] #pragma omp critical [(name)] < C/C++ structured block > !$OMP CRITICAL [(name)] < Fortran code executed in body of critical section > !$OMP END CRITICAL [(name)] Usage: Within a parallel region, there may exist subregions of code that will not execute properly when executed by multiple threads simultaneously. This issue is often due to a shared variable that is written and then read again. The CRITICAL ... END CRITICAL directive pair and the omp critical pragma define a subsection of code within a parallel region, referred to as a critical section, which is executed one thread at a time. The first thread to arrive at a critical section is the first to execute the code within the section. The second thread to arrive does not begin execution of statements in the critical section until the first thread exits the critical section. Likewise, each of the remaining threads wait its turn to execute the statements in the critical section. You can use the optional name argument to identify the critical region. Names that identify critical regions have external linkage and are in a name space separate from the name spaces used by labels, tags, members, and ordinary identifiers. If a name argument appears on a CRITICAL directive, the same name must appear on the END CRITICAL directive. Note Critical sections cannot be nested, and any such specifications are ignored. Branching into or out of a critical section is illegal. Example of Critical...End Critical directive: PROGRAM CRITICAL_USE REAL A(100,100),MX, LMX INTEGER I, J MX = -1.0 LMX = -1.0 CALL RANDOM_SEED() CALL RANDOM_NUMBER(A) !$OMP PARALLEL PRIVATE(I), FIRSTPRIVATE(LMX) !$OMP DO DO J=1,100 DO I=1,100 LMX = MAX(A(I,J),LMX) ENDDO ENDDOPGI® User’s Guide 246 !$OMP CRITICAL MX = MAX(MX,LMX) !$OMP END CRITICAL !$OMP END PARALLEL PRINT *,"MAX VALUE OF A IS ", MX END Example of omp critical pragma #include main(){ int a[100][100], mx=-1,lmx=-1, i, j; for (j=0; j<100; j++) for (i=0; i<100; i++) a[i][j]=1+(int)(10.0*rand()/(RAND_MAX+1.0)); #pragma omp parallel private(i) firstprivate(lmx) { #pragma omp for for (j=0; j<100; j++) for (i=0; i<100; i++) lmx = (lmx > a[i][j]) ? lmx : a[i][j]; #pragma omp critical mx = (mx > lmx) ? mx : lmx; } printf ("max value of a is %d\n",mx); } This program could also be implemented without the critical region by declaring MX as a reduction variable and performing the MAX calculation in the loop using MX directly rather than using LMX. Refer to “PARALLEL ... END PARALLEL and omp parallel ” and “DO ... END DO and omp for ” for more information on how to use the REDUCTION clause on a parallel DO loop. Example of omp critical pragma #include main(){ int a[100][100], mx=-1, lmx=-1, i, j; for (j=0; j<100; j++) for (i=0; i<100; i++) a[i][j]=1+(int)(10.0*rand()/(RAND_MAX+1.0)); #pragma omp parallel private(i) firstprivate(lmx) { #pragma omp for for (j=0; j<100; j++) for (i=0; i<100; i++) lmx = (lmx > a[i][j]) ? lmx : a[i][j]; #pragma omp critical mx = (mx > lmx) ? mx : lmx; } printf ("max value of a is %d\n",mx); } C$DOACROSS The C$DOACROSS directive, while not part of the OpenMP standard, is supported for compatibility with programs parallelized using legacy SGI-style directives. Syntax:Chapter 16. OpenMP Reference Information 247 C$DOACROSS [ Clauses ] < Fortran DO loop to be executed in parallel > #pragma omp parallel [clauses] < C/C++ structured block > Clauses: [ {PRIVATE | LOCAL} (list) ] [ {SHARED | SHARE} (list) ] [ MP_SCHEDTYPE={SIMPLE | INTERLEAVE} ] [ CHUNK= ] [ IF (logical_expression) ] Usage: The C$DOACROSS directive has the effect of a combined parallel region and parallel DO loop applied to the loop immediately following the directive. It is very similar to the OpenMP PARALLEL DO directive, but provides for backward compatibility with codes parallelized for SGI systems prior to the OpenMP standardization effort. The C$DOACROSS directive must not appear within a parallel region. It is a shorthand notation that tells the compiler to parallelize the loop to which it applies, even though that loop is not contained within a parallel region. While this syntax is more convenient, it should be noted that if multiple successive DO loops are to be parallelized it is more efficient to define a single enclosing parallel region and parallelize each loop using the OpenMP DO directive. A variable declared PRIVATE or LOCAL to a C$DOACROSS loop is treated the same as a private variable in a parallel region or DO (see above). A variable declared SHARED or SHARE to a C$DOACROSS loop is shared among the threads, meaning that only 1 copy of the variable exists to be used and/or modified by all of the threads. This is equivalent to the default status of a variable that is not listed as PRIVATE in a parallel region or DO (this same default status is used in C$DOACROSS loops as well). DO ... END DO and omp for The OpenMP DO ...END DO directive and omp for pragma support parallel execution and the distribution of loop iterations across available threads in a parallel region. Syntax: !$OMP DO [Clauses] < Fortran DO loop to be executed in parallel !$OMP END DO [NOWAIT] #pragma omp for [Clauses] < C/C++ for loop to be executed in parallel > Clauses: For Directives: PRIVATE(list) FIRSTPRIVATE(list) LASTPRIVATE(list) REDUCTION({operator | intrinsic } : list) SCHEDULE (type [, chunk]) ORDERED For Pragmas: private(list) firstprivate(list) lastprivate(list) reduction(operator: list) schedule (kind[, chunk]) ordered nowaitPGI® User’s Guide 248 Usage: The real purpose of supporting parallel execution is the distribution of work across the available threads. The DO ... END DO directive pair and the omp for pragma provide a convenient mechanism for the distribution of loop iterations across the available threads in a parallel region. While you can explicitly manage work distribution with constructs such as the following one, these constructs are not in the form of directives or pragmas. Examples: For Directives: IF (omp_get_thread_num() .EQ. 0) THEN ... ELSE IF (omp_get_thread_num() .EQ. 1) THEN ... ENDIF For Pragmas: if (omp_get_thread_num() == 0) { ... } else if (omp_get_thread_num() == 1) { ... }t Tips Remember these items about clauses in the DO...END DO directives and omp for pragmas: • Variables declared in a PRIVATE list are treated as private to each processor participating in parallel execution of the loop, meaning that a separate copy of the variable exists on each processor. • Variables declared in a FIRSTPRIVATE list are PRIVATE, and in addition are initialized from the original object existing before the construct. • Variables declared in a LASTPRIVATE list are PRIVATE, and in addition the thread that executes the sequentially last iteration updates the version of the object that existed before the construct. • The REDUCTION clause for the directive is described in “PARALLEL ... END PARALLEL and omp parallel ,” on page 251 and the reduction clause for the pragma is described in“Directive and Pragma Clauses,” on page 260. • The SCHEDULE clause is explained in the following section. • If ORDERED code blocks are contained in the dynamic extent of the DO directive, the ORDERED clause must be present. For more information on ORDERED code blocks, refer to “ORDERED ”. • If ordered code blocks are contained in the dynamic extent of the for directive, the ordered clause must be present. See “ORDERED ,” on page 251 for more information on ordered code blocks. • The DO ... END DO directive pair directs the compiler to distribute the iterative DO loop immediately following the !$OMP DO directive across the threads available to the program. The DO loop is executed in parallel by the team that was started by an enclosing parallel region. If the !$OMP END DO directive is not specified, the !$OMP DO is assumed to end with the enclosed DO loop. DO ... END DO directive pairs may not be nested. Branching into or out of a !$OMP DO loop is not supported. • The omp for pragma directs the compiler to distribute the iterative for loop immediately following across the threads available to the program. The for loop is executed in parallel by the team that was started by anChapter 16. OpenMP Reference Information 249 enclosing parallel region. Branching into or out of an omp for loop is not supported, and omp for pragmas may not be nested. • By default, there is an implicit barrier after the end of the parallel loop; the first thread to complete its portion of the work waits until the other threads have finished their portion of work. If NOWAIT is specified, the threads will not synchronize at the end of the parallel loop. In addition to the preceding items, remember these items about !$OMP DO loops and omp for loops: • The DO loop index variable is always private. • The for loop index variable is always private and must be a signed integer. • !$OMP DO loops and omp for loops must be executed by all threads participating in the parallel region or none at all. • The END DO directive is optional, but if it is present it must appear immediately after the end of the enclosed DO loop. • The for loop must be a structured block and its execution must not be terminated by break. • Values of the loop control expressions and the chunk expressions must be the same for all threads executing the loop. Examples: Example of Do...END DO directive PROGRAM DO_USE REAL A(1000), B(1000) DO I=1,1000 B(I) = FLOAT(I) ENDDO !$OMP PARALLEL !$OMP DO DO I=1,1000 A(I) = SQRT(B(I)); ENDDO ... !$OMP END PARALLEL ... END Example of omp for pragma #include #include main(){ float a[1000], b[1000]; int i; for (i=0; i<1000; i++) b[i] = i; #pragma omp parallel { #pragma omp for for (i=0; i<1000; i++) a[i] = sqrt(b[i]); ... } ... } The SCHEDULE clause specifies how iterations of the DO or for loop are divided up between processors. For more information on this clause, refer to “Schedule Clause,” on page 261. FLUSH and omp flush pragma The OpenMP FLUSH directive and omp flush pragma ensure that all processor-visible data items, or only those specified in list when it’s present, are written back to memory at the point at which the directive appears. Syntax:PGI® User’s Guide 250 !$OMP FLUSH [(list)] #pragma omp flush [(list)] Usage: The FLUSH directive ensures that all processor-visible data items, or only those specified in the list, when it is present, are written back to memory at the point at which the directive or pragma appears. MASTER ... END MASTER and omp master pragma The MASTER....END MASTER directive and omp master pragma allow the user to designate code that must execute on a master thread and that is skipped by other threads in the team of threads. Syntax: !$OMP MASTER < Fortran code executed in body of MASTER section > !$OMP END MASTER #pragma omp master < C/C++ structured block > Usage: A master thread is a single thread of control that begins an OpenMP program and which is present for the duration of the program. In a parallel region of code, there may be a sub-region of code that should execute only on the master thread. Instead of ending the parallel region before this subregion and then starting it up again after this subregion, the MASTER ... END MASTER directive pair or omp master pragma allows the user to conveniently designate code that executes on the master thread and is skipped by the other threads. There is no implied barrier on entry to or exit from a master section of code. Nested master sections are ignored. Further, branching into or out of a master section is not supported. Examples: Example of MASTER...END MASTER directive PROGRAM MASTER_USE INTEGER A(0:1) INTEGER omp_get_thread_num A=-1 !$OMP PARALLEL A(omp_get_thread_num()) = omp_get_thread_num() !$OMP MASTER PRINT *, "YOU SHOULD ONLY SEE THIS ONCE" !$OMP END MASTER !$OMP END PARALLEL PRINT *, "A(0)=", A(0), " A(1)=", A(1) END Example of omp master pragma #include #include main(){ int a[2]={-1,-1};Chapter 16. OpenMP Reference Information 251 #pragma omp parallel { a[omp_get_thread_num()] = omp_get_thread_num(); #pragma omp master printf("YOU SHOULD ONLY SEE THIS ONCE\n"); } printf("a[0]=%d, a[1]=%d\n",a[0],a[1]); } ORDERED The OpenMP ORDERED directive and omp ordered pragma allow the user to identify a portion of code within a ordered code block that must be executed in the original, sequential order, while allowing parallel execution of statements outside the code block. Syntax: !$OMP ORDERED < Fortran code block executed by processor > !$OMP END ORDERED #pragma omp ordered < C/C++ structured block > Usage: The ORDERED directive can appear only in the dynamic extent of a DO or PARALLEL DO directive that includes the ORDERED clause. The ordered pragma can appear only in the dynamic extent of a for or parallel for pragma that includes the ordered clause. The structured code block between the ORDERED / END ORDERED directives or after the ordered pragma is executed by only one thread at a time, and in the order of the loop iterations. This sequentializes the ordered code block while allowing parallel execution of statements outside the code block. The following additional restrictions apply to the ORDERED directive and ordered pragma: • The ordered code block must be a structured block. It is illegal to branch into or out of the block. • It is illegal to branch into or out of the block. • A given iteration of a loop with a DO directive or omp for pragma cannot execute the same ORDERED directive or omp ordered pragma more than once, and cannot execute more than one ORDERED directive or omp ordered pragma. PARALLEL ... END PARALLEL and omp parallel The OpenMP PARALLEL ...END PARALLEL directive and OpenMP omp parallel pragma support a fork/join execution model in which a single thread executes all statements until a parallel region is encountered. Syntax: !$OMP PARALLEL [Clauses] < Fortran code executed in body of parallel region > !$OMP END PARALLEL #pragma omp parallel [clauses] < C/C++ structured block > Clauses:PGI® User’s Guide 252 For Directives: PRIVATE(list) SHARED(list) DEFAULT(PRIVATE | SHARED | NONE) FIRSTPRIVATE(list) REDUCTION([{operator | intrinsic}:] list) COPYIN(list) IF(scalar_logical_expression) NUM_THREADS(scalar_integer_expression) For Pragmas: private(list) shared(list) default(shared | none) firstprivate(list) reduction(operator: list) copyin (list) if (scalar_expression) num_threads(scalar_integer_expression) Usage: This directive pair or pragma declares a region of parallel execution. It directs the compiler to create an executable in which the statements within the structured block, such as between PARALLEL and PARALLEL END for directives, are executed by multiple lightweight threads. The code that lies within this structured block is called a parallel region. The OpenMP parallelization directives or pragmas support a fork/join execution model in which a single thread executes all statements until a parallel region is encountered. At the entrance to the parallel region, a system-dependent number of symmetric parallel threads begin executing all statements in the parallel region redundantly. These threads share work by means of work-sharing constructs such as parallel DO loops or For loops. • The number of threads in the team is controlled by the OMP_NUM_THREADS environment variable. If OMP_NUM_THREADS is not defined, the program executes parallel regions using only one processor. • Branching into or out of a parallel region is not supported. • All other shared-memory parallelization directives or pragmas must occur within the scope of a parallel region. Nested PARALLEL ... END PARALLEL directive pairs or omp parallel pragmas are not supported and are ignored. • There is an implicit barrier at the end of the parallel region, which, in the directive, is denoted by the END PARALLEL directive. When all threads have completed execution of the parallel region, a single thread resumes execution of the statements that follow. NOTE By default, there is no work distribution in a parallel region. Each active thread executes the entire region redundantly until it encounters a directive or pragma that specifies work distribution. For work distribution, see the DO, PARALLEL DO, or DOACROSS directives or the omp for pragma. Examples: PARALELL ... END PARALLEL directive example: PROGRAM WHICH_PROCESSOR_AM_I INTEGER A(0:1) INTEGER omp_get_thread_num A(0) = -1 A(1) = -1 !$OMP PARALLEL A(omp_get_thread_num()) = omp parallel pragma Example #include #include main(){ int a[2]={-1,-1}; #pragma omp parallel { a[omp_get_thread_num()] =Chapter 16. OpenMP Reference Information 253 omp_get_thread_num() !$OMP END PARALLEL PRINT *, "A(0)=",A(0), " A(1)=",A(1) END omp_get_thread_num(); } printf("a[0] = %d, a[1] = %d",a[0],a[1]); } Clause Usage: PRIVATE: The variables specified in a PRIVATE list are private to each thread in a team. In effect, the compiler creates a separate copy of each of these variables for each thread in the team. When an assignment to a private variable occurs, each thread assigns to its local copy of the variable. When operations involving a private variable occur, each thread performs the operations using its local copy of the variable. Tips about private variables: • Variables declared private in a parallel region are undefined upon entry to the parallel region. If the first use of a private variable within the parallel region is in a right-hand-side expression, the results of the expression will be undefined (i.e., this is probably a coding error). • Variables declared private in a parallel region are undefined when serial execution resumes at the end of the parallel region. SHARED: Variables specified in a SHARED list are shared between all threads in a team, meaning that all threads access the same storage area for SHARED data. DEFAULT: The DEFAULT clause lets you specify the default attribute for variables in the lexical extent of the parallel region. Individual clauses specifying PRIVATE, SHARED, and so on, override the declared DEFAULT. Specifying DEFAULT(NONE) declares that there is no implicit default; thus, each variable in the parallel region must be explicitly listed with an attribute of PRIVATE, SHARED, FIRSTPRIVATE, LASTPRIVATE, or REDUCTION. FIRSTPRIVATE: Variables that appear in the list of a FIRSTPRIVATE clause are subject to the same semantics as PRIVATE variables; however, these variables are initialized from the original object existing prior to entering the parallel region. REDUCTION: Variables that appear in the list of a REDUCTION clause must be SHARED. A private copy of each variable in list is created for each thread as if the PRIVATE clause had been specified. Each private copy is initialized according to the operator as specified in the following table: Table 16.1. Initialization of REDUCTION Variables For Directives For Pragmas Operator / Intrinsic Initialization Operator Initialization + 0 + 0 * 1 * 1 - 0 - 0 .AND. .TRUE. & ~0 .OR. .FALSE. | 0PGI® User’s Guide 254 For Directives For Pragmas Operator / Intrinsic Initialization Operator Initialization .EQV. .TRUE. ^ 0 .NEQV. .FALSE. && 1 MAX Smallest Representable Number || 0 MIN Largest Representable Number IAND All bits on IOR 0 IEOR 0 At the end of the parallel region, a reduction is performed on the instances of variables appearing in list using operator or intrinsic as specified in the REDUCTION clause. The initial value of each REDUCTION variable is included in the reduction operation. If the {operator | intrinsic}: portion of the REDUCTION clause is omitted, the default reduction operator is “+” (addition). The COPYIN clause applies only to THREADPRIVATE common blocks. In the presence of the COPYIN clause, data from the master thread’s copy of the common block is copied to the threadprivate copies upon entry to the parallel region. In the presence of an IF clause, the parallel region is executed in parallel only if the corresponding scalar_logical_expression evaluates to .TRUE.. Otherwise, the code within the region is executed by a single processor, regardless of the value of the environment variable OMP_NUM_THREADS. If the NUM_THREADS clause is present, the corresponding scalar_integer_expression must evaluate to a positive integer value. This value sets the maximum number of threads used during execution of the parallel region. A NUM_THREADS clause overrides either a previous call to the library routine omp_set_num_threads() or the setting of the OMP_NUM_THREADS environment variable. PARALLEL DO The OpenMP PARALLEL DO directive is a shortcut for a PARALLEL region that contains a single DO directive. Note The OpenMP PARALLEL DO or DO directive must be immediately followed by a DO statement (DOstmt as defined by R818 of the ANSI Fortran standard). If you place another statement or an OpenMP directive between the PARALLEL DO or DO directive and the DO statement, the compiler issues a syntax error. Syntax: !$OMP PARALLEL DO [CLAUSES] < Fortran DO loop to be executed in parallel > #pragma omp parallel [clauses] < C/C++ structured block >Chapter 16. OpenMP Reference Information 255 [!$OMP END PARALLEL DO] Clauses: PRIVATE(list) SHARED(list) DEFAULT(PRIVATE | SHARED | NONE) FIRSTPRIVATE(list) LASTPRIVATE(list) REDUCTION({operator | intrinsic} : list) COPYIN (list) IF(scalar_logical_expression) NUM_THREADS(scalar_integer_expression) SCHEDULE (type [, chunk]) ORDERED Usage: The semantics of the PARALLEL DO directive are identical to those of a parallel region containing only a single parallel DO loop and directive. Note that the END PARALLEL DO directive is optional. The available clauses are as defined in “PARALLEL ... END PARALLEL and omp parallel ,” on page 251 and “DO ... END DO and omp for ”. PARALLEL SECTIONS The OpenMP PARALLEL SECTIONS / END SECTIONS directive pair and the omp parallel sections pragma define tasks to be executed in parallel; that is, they define a non-iterative work-sharing construct without the need to define an enclosing parallel region. Syntax: !$OMP PARALLEL SECTIONS [CLAUSES] [!$OMP SECTION] < Fortran code block executed by processor i > [!$OMP SECTION] < Fortran code block executed by processor j > ... !$OMP END SECTIONS [NOWAIT] #pragma omp parallel sections [clauses] { [#pragma omp section] < C/C++ structured block executed by processor i > [#pragma omp section] < C/C++ structured block executed by processor j > ... } Clauses: Directive clauses: PRIVATE(list) SHARED(list) DEFAULT(PRIVATE | SHARED | NONE) FIRSTPRIVATE(list) LASTPRIVATE(list) REDUCTION({operator | intrinsic} : list) COPYIN (list) IF(scalar_logical_expression) NUM_THREADS(scalar_integer_expression) Pragma clauses: private(list) shared(list) default(shared | none) firstprivate(list) lastprivate (list) reduction({operator: list) copyin (list) if (scalar_expression) num_threads(scalar_integer_expression)PGI® User’s Guide 256 nowait Usage: The PARALLEL SECTIONS / END SECTIONS directive pair and the omp parallel sections pragma define a noniterative work-sharing construct without the need to define an enclosing parallel region. Each section is executed by a single processor. If there are more processors than sections, some processors will have no work and will jump to the implied barrier at the end of the construct. If there are more sections than processors, one or more processors will execute more than one section. A SECTION directive may only appear within the lexical extent of the enclosing PARALLEL SECTIONS / END SECTIONS directives. In addition, the code within the PARALLEL SECTIONS / END SECTIONS directives must be a structured block, and the code in each SECTION must be a structured block. Semantics are identical to a parallel region containing only an omp sections pragma and the associated structured block. The available clauses are as defined in “PARALLEL ... END PARALLEL and omp parallel ,” on page 251 and “DO ... END DO and omp for ”. PARALLEL WORKSHARE The OpenMP PARALLEL WORKSHARE directive u... Syntax: !$OMP PARALLEL WORKSHARE [CLAUSES] < Fortran structured block to be executed in parallel > [!$OMP END PARALLEL WORKSHARE] !$OMP PARALLEL DO [CLAUSES] < Fortran DO loop to be executed in parallel > [!$OMP END PARALLEL DO] #pragma omp parallel [clauses] < C/C++ structured block > Clauses: Directive clauses: PRIVATE(list) SHARED(list) DEFAULT(PRIVATE | SHARED | NONE) FIRSTPRIVATE(list) LASTPRIVATE(list) REDUCTION({operator | intrinsic} : list) COPYIN (list) IF(scalar_logical_expression) NUM_THREADS(scalar_integer_expression) SCHEDULE (type [, chunk]) ORDERED Pragma clauses: private(list) shared(list) default(shared | none) firstprivate(list) lastprivate (list) reduction({operator: list) copyin (list) if (scalar_expression) num_threads(scalar_integer_expression) nowait Usage: The semantics of the PARALLEL WORKSHARE directive are identical to those of a parallel region containing a single WORKSHARE construct.Chapter 16. OpenMP Reference Information 257 The END PARALLEL WORKSHARE directive is optional, and that NOWAIT may not be specified on an END PARALLEL WORKSHARE directive. The available clauses are as defined in “PARALLEL ... END PARALLEL and omp parallel ,” on page 251. SECTIONS … END SECTIONS The OpenMP SECTIONS / END SECTIONS directive pair and the omp sections pragma define a non-iterative work-sharing construct within a parallel region which each section is executed by a single processor. Syntax: !$OMP SECTIONS [ Clauses ] [!$OMP SECTION] < Fortran code block executed by processor i > [!$OMP SECTION] < Fortran code block executed by processor j > ... !$OMP END SECTIONS [NOWAIT] #pragma omp sections [ Clauses ] { [#pragma omp section] < C/C++ structured block executed by processor i > [#pragma omp section] < C/C++ structured block executed by processor j > ... } Clauses: For Directives: PRIVATE (list) FIRSTPRIVATE (list) LASTPRIVATE (list) REDUCTION({operator|intrinsic} : list) For Pragmas: private (list) firstprivate (list) lastprivate (list) reduction(operator: list) nowait Usage: The SECTIONS / END SECTIONS directive pair and the omp sections pragma define a non-iterative worksharing construct within a parallel region. Each section is executed by a single processor. If there are more processors than sections, some processors have no work and thus jump to the implied barrier at the end of the construct. If there are more sections than processors, one or more processors must execute more than one section. A SECTION directive or omp sections pragma may only appear within the lexical extent of the enclosing SECTIONS / END SECTIONS directives or omp sections pragma. In addition, the code within the SECTIONS / END SECTIONS directives or omp sections pragma must be a structured block, and the code in each section must be a structured block. The available clauses are as defined in “PARALLEL ... END PARALLEL and omp parallel ,” on page 251 and “DO ... END DO and omp for ”. SINGLE ... END SINGLE The SINGLE ...END SINGLE directive or omp parallel pragma designates code that executes on a single thread and that is skipped by the other threads.PGI® User’s Guide 258 Syntax: !$OMP SINGLE [Clauses] < Fortran code executed in body of SINGLE processor section > !$OMP END SINGLE [NOWAIT] #pragma omp parallel [clauses] < C/C++ structured block > Clauses: PRIVATE(list) FIRSTPRIVATE(list) COPYPRIVATE(list) Usage: In a parallel region of code, there may be a sub-region of code that will only execute correctly on a single thread. Instead of ending the parallel region before this subregion and then starting it up again after this subregion, the SINGLE ... END SINGLE directive pair lets you conveniently designate code that executes on a single thread and is skipped by the other threads. There is an implied barrier on exit from a SINGLE ... END SINGLE section of code unless the optional NOWAIT clause is specified. Nested single process sections are ignored. Branching into or out of a single process section is not supported. Examples: PROGRAM SINGLE_USE INTEGER A(0:1) INTEGER omp_get_thread_num() !$OMP PARALLEL A(omp_get_thread_num()) = omp_get_thread_num() !$OMP SINGLE PRINT *, "YOU SHOULD ONLY SEE THIS ONCE" !$OMP END SINGLE !$OMP END PARALLEL PRINT *, "A(0)=", A(0), " A(1)=", A(1) END The section“PARALLEL ... END PARALLEL and omp parallel ,” on page 251 describes PRIVATE and FIRSTPRIVATE clause. The COPYPRIVATE clause causes the variables in list to be copied from the private copies in the single thread that executes the SINGLE region to the other copies in all other threads of the team at the end of the SINGLE region. The COPYPRIVATE clause must not be used with NOWAIT. THREADPRIVATE The OpenMP THREADPRIVATE directive identifies a Fortran common block as being private to each thread. The omp threadprivate pragma identifies a global variable as being private to each thread. Syntax: !$OMP THREADPRIVATE (list) #pragma omp threadprivate (list) Usage:Chapter 16. OpenMP Reference Information 259 Where list is a comma-separated list of named variables to be made private to each thread or named common blocks to be made private to each thread but global within the thread. Common block names must appear between slashes, such as /common_block_name/. This directive must appear in the declarations section of a program unit after the declaration of any common blocks or variables listed. On entry to a parallel region, data in a THREADPRIVATE common block or variable is undefined unless COPYIN is specified on the PARALLEL directive. When a common block or variable that is initialized using DATA statements appears in a THREADPRIVATE directive, each thread’s copy is initialized once prior to its first use. Where list is a list of variables to be made private to each thread but global within the thread. This pragma must appear in the declarations section of a program unit after the declaration of any variables listed. On entry to a parallel region, data in a threadprivate variable is undefined unless copyin is specified on the omp parallel pragma. When a variable appears in an omp threadprivate pragma, each thread’s copy is initialized once at an unspecified point prior to its first use as the master copy would be initialized in a serial execution of the program. Restrictions: The following restrictions apply to the THREADPRIVATE directive or omp threadprivate pragma: • The THREADPRIVATE directive must appear after every declaration of a thread private common block. • The omp threadprivate pragma must appear after the declaration of every threadprivate variable included in list. • Only named common blocks can be made thread private • It is illegal for a THREADPRIVATE common block or its constituent variables to appear in any clause other than a COPYIN clause. • A variable can appear in a THREADRIVATE directive only in the scope in which it is declared. It must not be an element of a common block or be declared in an EQUIVALENCE statement. • A variable that appears in a THREADPRIVATE directive and is not declared in the scope of a module must have the SAVE attribute. • If a variable is specified in an omp threadprivate pragma in one translation unit, it must be specified in an omp threadprivate pragma in every translation unit in which it appears. • The address of an omp threadprivate variable is not an address constant. • The address of an omp threadprivate variable is not an address constant. • An omp threadprivate variable must not have an incomplete type or a reference type. WORKSHARE ... END WORKSHARE The OpenMP WORKSHARE … END WORKSHARE directive pair or omp parallel pragma provides a mechanism to effect parallel execution of non-iterative but implicitly data parallel constructs. Syntax: !$OMP WORKSHARE < Fortran structured block to be #pragma omp parallel [clauses] < C/C++ structured block >PGI® User’s Guide 260 executed in parallel > !$OMP END WORKSHARE [NOWAIT] Usage: The Fortran structured block enclosed by the WORKSHARE … END WORKSHARE directive pair can consist only of the following types of statements and constructs: • Array assignments • Scalar assignments • FORALL statements or constructs • WHERE statements or constructs • OpenMP ATOMIC , CRITICAL or PARALLEL constructs The work implied by the above statements and constructs is split up between the threads executing the WORKSHARE construct in a way that is guaranteed to maintain standard Fortran semantics. The goal of the WORKSHARE construct is to effect parallel execution of non-iterative but implicitly data parallel array assignments, FORALL, and WHERE statements and constructs intrinsic to the Fortran language beginning with Fortran 90. The Fortran structured block contained within a WORKSHARE construct must not contain any userdefined function calls unless the function is ELEMENTAL. Directive and Pragma Clauses Some directives and pragmas accept clauses that further allow a user to control the scope attributes of variables for the duration of the directive or pragma. Not all clauses are allowed on all directives, so the clauses that are valid are included with the description of the directive and pragma. Typically, if no data scope clause is specified for variables, the default scope is share. The following table provides a brief summary of the clauses associated with OPENMP directives and pragmas that PGI supports. For complete information on these clauses, refer to the OpenMP documentation available on the WorldWide Web. Table 16.2. Directive and Pragma Clauses Clause Description copyin Allows threads to access the master thread's value, for a threadprivate variable. You assign the same value to threadprivate variables for each thread in the team executing the parallel region. Then, for each variable specified, the value of the variable in the master thread of the team is copied to the threadprivate copies at the beginning of the parallel region. copyprivate(list) Specifies that one or more variables should be shared among all threads. This clause provides a mechanism to use a private variable to broadcast a value from one member of a team to the other members. default (OpenMP) Specifies the behavior of unscoped variables in a parallel region, such as the data-sharing attributes of variables.Chapter 16. OpenMP Reference Information 261 Clause Description firstprivate(list) Specifies that each thread should have its own instance of a variable, and that each variable in the list should be initialized with the value of the original variable, because it exists before the parallel construct. if (OpenMP) Specifies whether a loop should be executed in parallel or in serial. lastprivate(list) Specifies that the enclosing context's version of the variable is set equal to the private version of whichever thread executes the final iteration (for-loop construct) or last section (#pragma sections). nowait Overrides the barrier implicit in a directive. num_threads Sets the number of threads in a thread team. ordered (OpenMP Clauses) Required on a parallel for (OpenMP) statement if an ordered (OpenMP Directives) directive is to be used in the loop. private (OpenMP) Specifies that each thread should have its own instance of a variable. reduction({operator | intrinsic } : list) Specifies that one or more variables that are private to each thread are the subject of a reduction operation at the end of the parallel region. schedule(type [,chunk]) Applies to the for (OpenMP) directive, allowing the user to specify the chunking method for parallelization. Work is assigned to threads in different manners depending on the scheduling type or chunk size used. shared (OpenMP) Specifies that one or more variables should be shared among all threads. All threads within a team access the same storage area for shared variables Schedule Clause The SCHEDULE clause specifies how iterations of the DO or for loop are divided up between processors. Given a SCHEDULE (type [, chunk]) clause, the type can be STATIC, DYNAMIC, GUIDED, or RUNTIME, defined in the following list. Note For pragmas, the values for the clause are lower case static, dynamic, guided, or runtime. For simplicity, we use the directive uppercase value in the following information. • When SCHEDULE (STATIC, chunk) is specified, iterations are allocated in contiguous blocks of size chunk. The blocks of iterations are statically assigned to threads in a round-robin fashion in order of the thread ID numbers. The chunk must be a scalar integer expression. If chunk is not specified, a default chunk size is chosen equal to: (number_of_iterations + omp_num_threads() - 1) / omp_num_threads() • When SCHEDULE (DYNAMIC, chunk) is specified, iterations are allocated in contiguous blocks of size chunk. As each thread finishes a piece of the iteration space, it dynamically obtains the next set of iterations. The chunk must be a scalar integer expression. If no chunk is specified, a default chunk size is chosen equal to 1.PGI® User’s Guide 262 • When SCHEDULE (GUIDED, chunk) is specified, the chunk size is reduced in an exponentially decreasing manner with each dispatched piece of the iteration space. Chunk specifies the minimum number of iterations to dispatch each time, except when there are less than chunk iterations remaining to be processed, at which point all remaining iterations are assigned. If no chunk is specified, a default chunk size is chosen equal to 1. • When SCHEDULE (RUNTIME) is specified, the decision regarding iteration scheduling is deferred until runtime. The schedule type and chunk size can be chosen at runtime by setting the OMP_SCHEDULE environment variable. If this environment variable is not set, the resulting schedule is equivalent to SCHEDULE(STATIC).263 Chapter 17. Directives and Pragmas Reference As we mentioned in Chapter 6, “Using Directives and Pragmas,” on page 63, PGI Fortran compilers support proprietary directives and pragmas. This chapter contains detailed descriptions of PGI’s proprietary directives and pragmas. PGI Proprietary Fortran Directive and C/C++ Pragma Summary Directives are Fortran comments and pragmas are C/C++ comments that the user may supply in a source file to provide information to the compiler. These comments alter the effects of certain command line options or default behavior of the compiler. They provide pragmatic information that control the actions of the compiler in a particular portion of a program without affecting the program as a whole. That is, while a command line option affects the entire source file that is being compiled, directives and pragmas apply, or disable, the effects of a command line option to selected subprograms or to selected loops in the source file, for example, to optimize a specific area of code. Use directives and pragmas to tune selected routines or loops. The Fortran directives may have any of the following forms: !pgi$g directive !pgi$r directive !pgi$l directive !pgi$ directive Directives and pragmas override corresponding command-line options. For usage information such as the scope and related command-line options, refer to Chapter 6, “Using Directives and Pragmas,” on page 63. altcode (noaltcode) This directive or pragma instructs the compiler to generate alternate code for vectorized or parallelized loops. The noaltcode directive or pragma disables generation of alternate code. Scope: This directive or pragma affects the compiler only when –Mvect=sse or –Mconcur is enabled on the command line.PGI® User’s Guide 264 cpgi$ altcode Enables alternate code (altcode) generation for vectorized loops. For each loop the compiler decides whether to generate altcode and what type(s) to generate, which may be any or all of: altcode without iteration peeling, altcode with non-temporal stores and other data cache optimizations, and altcode based on array alignments calculated dynamically at runtime. The compiler also determines suitable loop count and array alignment conditions for executing the alternate code. cpgi$ altcode alignment For a vectorized loop, if possible generate an alternate vectorized loop containing additional aligned moves which is executed if a runtime array alignment test is passed. cpgi$ altcode [(n)] concur For each auto-parallelized loop, generate an alternate serial loop to be executed if the loop count is less than or equal to n. If n is omitted or n is 0, the compiler determines a suitable value of n for each loop. cpgi$ altcode [(n)] concurreduction This directive sets the loop count threshold for parallelization of reduction loops to n. For each autoparallelized reduction loop, generate an alternate serial loop to be executed if the loop count is less than or equal to n. If n is omitted or n is 0, the compiler determines a suitable value of n for each loop. cpgi$ altcode [(n)] nontemporal For a vectorized loop, if possible generate an alternate vectorized loop containing non-temporal stores and other cache optimizations to be executed if the loop count is greater than n. If n is omitted or n is 1, the compiler determines a suitable value of n for each loop. The alternate code is optimized for the case when the data referenced in the loop does not all fit in level 2 cache. cpgi$ altcode [(n)] nopeel For a vectorized loop where iteration peeling is performed by default, if possible generate an alternate vectorized loop without iteration peeling to be executed if the loop count is less than or equal to n. If n is omitted or n is 1, the compiler determines a suitable value of n for each loop, and in some cases it may decide not to generate an alternate unpeeled loop. cpgi$ altcode [(n)] vector For each vectorized loop, generate an alternate scalar loop to be executed if the loop count is less than or equal to n. If n is omitted or n is 1, the compiler determines a suitable value of n for each loop. cpgi$ noaltcode This directive sets the loop count thresholds for parallelization of all innermost loops to 0, and disables alternate code generation for vectorized loops. assoc (noassoc) This directive or pragma toggles the effects of the –Mvect=noassoc command-line option, an optimization –M control. By default, when scalar reductions are present the vectorizer may change the order of operations so that it can generate better code (e.g., dot product). Such transformations may change the result of the computation due to roundoff error. The noassoc directive disables these transformations. Scope: This directive or pragma affects the compiler only when –Mvect=sse is enabled on the command line.Chapter 17. Directives and Pragmas Reference 265 bounds (nobounds) This directive or pragma alters the effects of the –Mbounds command line option. This directive enables the checking of array bounds when subscripted array references are performed. By default, array bounds checking is not performed. cncall (nocncall) Loops within the specified scope are considered for parallelization, even if they contain calls to user-defined subroutines or functions. A nocncall directive cancels the effect of a previous cncall. concur (noconcur) This directive or pragma alters the effects of the –Mconcur command-line option. The directive instructs the auto-parallelizer to enable auto-concurrentization of loops. If concur is specified, multiple processors will be used to execute loops which the auto-parallelizer determines to be parallelizable. The noconcur directive disables these transformations, but use of concur overrides previous noconcur statements. Scope: This directive or pragma affects the compiler only when –Mconcur is enabled on the command line. depchk (nodepchk) This directive or pragma alters the effects of the –Mdepchk command line option. When potential data dependencies exist, the compiler, by default, assumes that there is a data dependence that in turn may inhibit certain optimizations or vectorizations. nodepchk directs the compiler to ignore unknown data dependencies. eqvchk (noeqvchk) When examining data dependencies, noeqvchk directs the compiler to ignore any dependencies between variables appearing in EQUIVALENCE statements. fcon (nofcon) This C/C++ pragma alters the effects of the –Mfcon (a –M Language control) command-line option. The pragma instructs the compiler to treat non-suffixed floating-point constants as float rather than double. By default, all non-suffixed floating-point constants are treated as double. Note Only routine or global scopes are allowed for this C/C++ pragma. invarif (noinvarif) This directive or pragma has no corresponding command-line option. Normally, the compiler removes certain invariant if constructs from within a loop and places them outside of the loop. The directive noinvarif directs the compiler to not move such constructs. The directive invarif toggles a previous noinvarif.PGI® User’s Guide 266 ivdep The ivdep directive is equivalent to the directive nodepchk. lstval (nolstval) This directive or pragma has no corresponding command-line option. The compiler determines whether the last values for loop iteration control variables and promoted scalars need to be computed. In certain cases, the compiler must assume that the last values of these variables are needed and therefore computes their last values. The directive nolstval directs the compiler not to compute the last values for those cases. opt The syntax of this directive or pragma is: cpgi$ opt= where, the optional is r or g and is an integer constant representing the optimization level to be used when compiling a subprogram (routine scope) or all subprograms in a file (global scope). The opt directive overrides the value specified by the command line option –On. safe (nosafe) This C/C++ pragma has no corresponding command-line option. By default, the compiler assumes that all pointer arguments are unsafe. That is, the storage located by the pointer can be accessed by other pointers. The forms of the safe pragma are: #pragma [scope] [no]safe #pragma safe (variable [, variable]...) where scope is either global or routine. When the pragma safe is not followed by a variable name (or name list), all pointer arguments appearing in a routine (if scope is routine) or all routines (if scope is global) will be treated as safe. If variable names occur after safe, each name is the name of a pointer argument in the current function. The named argument is considered to be safe. Note If just one variable name is specified, the surrounding parentheses may be omitted. safe_lastval During parallelization scalars within loops need to be privatized. Problems are possible if a scalar is accessed outside the loop. in this exampl a problem results since the value of t may not be computed on the last iteration of the loop.: do i = 1, N if( f(x(i)) > 5.0 then) t = x(i) endif enddoChapter 17. Directives and Pragmas Reference 267 v = t If a scalar assigned within a loop is used outside the loop, we normally save the last value of the scalar. Essentially the value of the scalar on the "last iteration" is saved, in this case when i = N. If the loop is parallelized and the scalar is not assigned on every iteration, it may be difficult to determine on what iteration t is last assigned, without resorting to costly critical sections. Analysis allows the compiler to determine if a scalar is assigned on every iteration, thus the loop is safe to parallelize if the scalar is used later. An example loop is: do i = 1, N if( x(i) > 0.0 ) then t = 2.0 else t = 3.0 endif y(i) = ...t... enddo v = t where t is assigned on every iteration of the loop. However, there are cases where a scalar may be privatizable. If it is used after the loop, it is unsafe to parallelize. Examine this loop: do i = 1,N if( x(i) > 0.0 ) then t = x(i) ... ... y(i) = ...t.. endif enddo v = t where each use of t within the loop is reached by a definition from the same iteration. Here t is privatizable, but the use of t outside the loop may yield incorrect results since the compiler may not be able to detect on which iteration of the parallelized loop t is assigned last. The compiler detects the above cases. Where a scalar is used after the loop, but is not defined on every iteration of the loop, parallelization will not occur. If you know that the scalar is assigned on the last iteration of the loop, making it safe to parallelize the loop, a directive or pragma is available to let the compiler know the loop is safe to parallelize. Use the following C pragma to tell the compiler that for a given loop the last value computed for all scalars make it safe to parallelize the loop: cpgi$l safe_lastva #pragma loop safe_lastval In addition, a command-line option,-Msafe_lastval, provides this information for all loops within the routines being compiled, essentially providing global scope. safeptr (nosafeptr) The directive or pragma safeptr directs the compiler to treat pointer variables of the indicated storage class as safe. The pragma nosafeptr directs the compiler to treat pointer variables of the indicated storage class as unsafe. This pragma alters the effects of the –Msafeptr command-line option.PGI® User’s Guide 268 The syntax of this pragma is: cpgi$[] value #pragma [scope] value where value is: [no]safeptr={arg|local|auto|global|static|all},... Note that the values local and auto are equivalent. For example, in a file containing multiple functions, the command-line option –Msafeptr might be helpful for one function, but can’t be used because another function in the file would produce incorrect results. In such a file, the safeptr pragma, used with routine scope could improve performance and produce correct results. single (nosingle) The pragma single directs the compiler not to implicityly convert float values to double non-prototyped functions. This can result in faster code if the program uses only float parameters. Note Since ANSI C specifies that floats must be converted to double, this pragma results in non-ANSI conforming code. Valid only for routine or global scope. tp Note The tp directive or pragma can only be applied at the routine or global level. For more information about these levels, refer to“Scope of C/C++ Pragmas and Command-Line Options,” on page 67. You use the directive or pragma tp to specify one or more processor targets for which to generate code. cpgi$ tp [target]... See Table 2, “Processor Options,” on page xxvi for a list of targets that can be used as parameters to the tp directive. For more information on unified binaries, refer to “Processor-Specific Optimization and the Unified Binary,” on page 36. unroll (nounroll) Note The unroll directive or pragma has no effect on vectorized loops. The directive or pragma nounroll disables loop unrolling while unroll enables unrolling. The directive or pragma takes arguments c and n. A c specifies that c (complete unrolling should be turned on or off) An n specifies that n (count) unrolling should be turned on or off. In addition, the following arguments may be added to the unroll directive: cpgi$ unroll = c:vChapter 17. Directives and Pragmas Reference 269 This sets the threshold to which c unrolling applies; v is a constant; a loop whose constant loop count is <= v is completely unrolled. cpgi$ unroll = n:v This adjusts threshold to which n unrolling applies; v is a constant; a loop to which n unrolling applies is unrolled v times. The directives unroll and nounroll only apply if –Munroll is selected on the command line. vector (novector) The directive or pragma novector is used to disable vectorization. The directive vector is used to re-enable vectorization after a previous novector directive. The directives vector and novector only apply if –Mvect has been selected on the command line. vintr (novintr) The directive or pragma novintr directs the vectorizer to disable recognition of vector intrinsics. The directive vintr is used to re-enable recognition of vector intrinsics after a previous novintr directive. The directives vintr and novintr only apply if –Mvect has been selected on the command line.270271 Chapter 18. Run-time Environment This chapter describes the programming model supported for compiler code generation, including register conventions and calling conventions for x86 and x64 processor-based systems. It addresses these conventions for processors running linux86 or Win32 operating systems, for processors running linux86-64 operating systems, and for processors running Win64 operating systems. Note In this chapter we sometimes refer to word, halfword, and double word. The equivalent byte information is word (4 byte), halfword (2 byte), and double word (8 byte). Linux86 and Win32 Programming Model This section defines compiler and assembly language conventions for the use of certain aspects of an x86 processor running a linux86 or Win32 operating system. These standards must be followed to guarantee that compilers, application programs, and operating systems written by different people and organizations will work together. The conventions supported by the PGCC ANSI C compiler implement the application binary interface (ABI) as defined in the System V Application Binary Interface: Intel Processor Supplement and the System V Application Binary Interface, listed in the “Related Publications” section in the Preface. Function Calling Sequence This section describes the standard function calling sequence, including the stack frame, register usage, and parameter passing. Register Usage Conventions The following table defines the standard for register allocation. The 32-bit x86 Architecture provides a number of registers. All the integer registers and all the floating-point registers are global to all procedures in a running program. Table 18.1. Register Allocation Type Name Purpose General %eax integer return valuePGI® User’s Guide 272 Type Name Purpose %edx dividend register (for divide operations) %ecx count register (shift and string operations) %ebx local register variable %ebp optional stack frame pointer %esi local register variable %edi local register variable %esp stack pointer Floating-point %st(0) floating-point stack top, return value %st(1) floating-point next to stack top %st(...) %st(7) floating-point stack bottom In addition to the registers, each function has a frame on the run-time stack. This stack grows downward from high addresses. The next table shows the stack frame organization. Table 18.2. Standard Stack Frame Position Contents Frame 4n+8 (%ebp) argument word n previous 8 (%ebp) argument word 0 4 (%ebp) return address 0 (%ebp) caller's %ebp current -4 (%ebp) n bytes of local -n (%ebp) variables and temps Several key points concerning the stack frame: • The stack is kept double word aligned • Argument words are pushed onto the stack in reverse order so the rightmost argument in C call syntax has the highest address. A dummy word may be pushed ahead of the rightmost argument in order to preserve doubleword alignment. All incoming arguments appear on the stack, residing in the stack frame of the caller. • An argument’s size is increased, if necessary, to make it a multiple of words. This may require tail padding, depending on the size of the argument. All registers on an x86 system are global and thus visible to both a calling and a called function. Registers %ebp, %ebx, %edi, %esi, and %esp are non-volatile across function calls. Therefore, a function must preserve these registers’ values for its caller. Remaining registers are volatile (scratch). If a calling function wants to preserve such a register value across a function call, it must save its value explicitly.Chapter 18. Run-time Environment 273 Some registers have assigned roles in the standard calling sequence: %esp The stack pointer holds the limit of the current stack frame, which is the address of the stack’s bottommost, valid word. At all times, the stack pointer should point to a word-aligned area. %ebp The frame pointer holds a base address for the current stack frame. Consequently, a function has registers pointing to both ends of its frame. Incoming arguments reside in the previous frame, referenced as positive offsets from %ebp, while local variables reside in the current frame, referenced as negative offsets from %ebp. A function must preserve this register value for its caller. %eax Integral and pointer return values appear in %eax. A function that returns a structure or union value places the address of the result in %eax. Otherwise, this is a scratch register. %esi, %edi These local registers have no specified role in the standard calling sequence. Functions must preserve their values for the caller. %ecx, %edx Scratch registers have no specified role in the standard calling sequence. Functions do not have to preserve their values for the caller. %st(0) Floating-point return values appear on the top of the floating point register stack; there is no difference in the representation of single or double-precision values in floating point registers. If the function does not return a floating point value, then the stack must be empty. %st(1) - %st(7) Floating point scratch registers have no specified role in the standard calling sequence. These registers must be empty before entry and upon exit from a function. EFLAGS The flags register contains the system flags, such as the direction flag and the carry flag. The direction flag must be set to the “forward” (i.e., zero) direction before entry and upon exit from a function. Other user flags have no specified role in the standard calling sequence and are not reserved. Floating Point Control Word The control word contains the floating-point flags, such as the rounding mode and exception masking. This register is initialized at process initialization time and its value must be preserved. Signals can interrupt processes. Functions called during signal handling have no unusual restriction on their use of registers. Moreover, if a signal handling function returns, the process resumes its original execution path with registers restored to their original values. Thus, programs and compilers may freely use all registers without danger of signal handlers changing their values. Function Return Values Functions Returning Scalars or No Value • A function that returns an integral or pointer value places its result in register %eax.PGI® User’s Guide 274 • A function that returns a long long integer value places its result in the registers %edx and %eax. The most significant word is placed in %edx and the least significant word is placed in %eax. • A floating-point return value appears on the top of the floating point stack. The caller must then remove the value from the floating point stack, even if it does not use the value. Failure of either side to meet its obligations leads to undefined program behavior. The standard calling sequence does not include any method to detect such failures nor to detect return value type mismatches. Therefore, the user must declare all functions properly. There is no difference in the representation of single-, double- or extended-precision values in floating-point registers. • Functions that return no value (also called procedures or void functions) put no particular value in any register. • A call instruction pushes the address of the next instruction (the return address) onto the stack. The return instruction pops the address off the stack and effectively continues execution at the next instruction after the call instruction. A function that returns a scalar or no value must preserve the caller's registers as described above. Additionally, the called function must remove the return address from the stack, leaving the stack pointer (%esp) with the value it had before the call instruction was executed. Functions Returning Structures or Unions If a function returns a structure or union, then the caller provides space for the return value and places its address on the stack as argument word zero. In effect, this address becomes a hidden first argument. A function that returns a structure or union also sets %eax to the value of the original address of the caller's area before it returns. Thus, when the caller receives control again, the address of the returned object resides in register %eax and can be used to access the object. Both the calling and the called functions must cooperate to pass the return value successfully: • The calling function must supply space for the return value and pass its address in the stack frame; • The called function must use the address from the frame and copy the return value to the object so supplied; • The called function must remove this address from the stack before returning. Failure of either side to meet its obligation leads to undefined program behavior. The standard function calling sequence does not include any method to detect such failures nor to detect structure and union type mismatches. Therefore, you must declare the function properly. The following table illustrates the stack contents when the function receives control, after the call instruction, and when the calling function again receives control, after the ret instruction. Table 18.3. Stack Contents for Functions Returning struct/union Position After Call After Return Position 4n+8 (%esp) argument word n argument word n 4n-4 (%esp) 8 (%esp) argument word 1 argument word 1 0 (%esp)Chapter 18. Run-time Environment 275 Position After Call After Return Position 4 (%esp) value address undefined 0 (%esp) return address The following sections of this appendix describe where arguments appear on the stack. The examples are written as if the function prologue described above had been used. Argument Passing Integral and Pointer Arguments As mentioned, a function receives all its arguments through the stack; the last argument is pushed first. In the standard calling sequence, the first argument is at offset 8(%ebp), the second argument is at offset 12(%ebp), as previously shown in Table 18.3, “Stack Contents for Functions Returning struct/union”. Functions pass all integer-valued arguments as words, expanding or padding signed or unsigned bytes and halfwords as needed. Table 18.4. Integral and Pointer Arguments Call Argument Stack Address g(1, 2, 3, (void *)0); 1 8 (%ebp) 2 12 (%ebp) 3 16 (%ebp) (void *) 0 20 (%ebp) Floating-Point Arguments The stack also holds floating-point arguments: single-precision values use one word and double-precision use two. The example below uses only double-precision arguments. Table 18.5. Floating-point Arguments Call Argument Stack Address h(1.414, 1, 2.998e10); word 0, 1.414 8 (%ebp) word 1, 1.414 12 (%ebp) 1 16 (%ebp) word 0 2.998e10 20 (%ebp) word 1, 2.998e10 24 (%ebp) Structure and Union Arguments Structures and unions can have byte, halfword, or word alignment, depending on the constituents. An argument’s size is increased, if necessary, to make it a multiple of words. This may require tail padding,PGI® User’s Guide 276 depending on the size of the argument. Structure and union arguments are pushed onto the stack in the same manner as integral arguments, described above. This provides call-by-value semantics, letting the called function modify its arguments without affecting the calling function’s object. In the example below, the argument, s, is a structure consisting of more than 2 words. Table 18.6. Structure and Union Arguments Call Argument Stack Address i(1,s); 1 8 (%ebp) word 0, s 12 (%ebp) word 1, s 16 (%ebp) ... ... Implementing a Stack In general, compilers and programmers must maintain a software stack. Register %esp is the stack pointer. Register %esp is set by the operating system for the application when the program is started. The stack must be a grow-down stack. A separate frame pointer enables calls to routines that change the stack pointer to allocate space on the stack at run-time (e.g. alloca). Some languages can also return values from a routine allocated on stack space below the original top-of-stack pointer. Such a routine prevents the calling function from using %esp-relative addressing to get at values on the stack. If the compiler does not call routines that leave %esp in an altered state when they return, a frame pointer is not needed and is not used if the compiler option –Mnoframe is specified. Although not required, the stack should be kept aligned on 8-byte boundaries so that 8-byte locals are favorably aligned with respect to performance. PGI's compilers allocate stack space for each routine in multiples of 8 bytes. Variable Length Parameter Lists. Parameter passing in registers can handle a variable number of parameters. The C language uses a special method to access variable-count parameters. The stdarg.h and varargs.h files define several functions to access these parameters. A C routine with variable parameters must use the va_start macro to set up a data structure before the parameters can be used. The va_arg macro must be used to access the successive parameters. C Parameter Conversion. In C, for a called prototyped function, the parameter type in the called function must match the argument type in the calling function. If the called function is not prototyped, the calling convention uses the types of the arguments but promotes char or short to int, and unsigned char or unsigned short to unsigned int and promotes float to double, unless you use the ##Msingle option. For more information on the –Msingle option, refer to Chapter 3. If the called function is prototyped, the unused bits of a register containing a char or short parameter are undefined and the called function must extend the sign of the unused bits when needed.Chapter 18. Run-time Environment 277 Calling Assembly Language Programs Example 18.1. C Program Calling an Assembly-language Routine /* File: testmain.c */ main(){ long l_para1 = 0x3f800000; float f_para2 = 1.0; double d_para3 = 0.5; float f_return; extern float sum_3 (long para1, float para2, double para3); f_return = sum_3(l_para1,f_para2, d_para3); printf("Parameter one, type long = %08x\n",l_para1); printf("Parameter two, type float = %f\n",f_para2); printf("Parameter three, type double = %g\n",d_para3); printf("The sum after conversion = %f\n",f_return); } # File: sum_3.s # Computes ( para1 + para2 ) + para3 .text .align 4 .long .EN1-sum_3+0xc8000000 .align 16 .globl sum_3 sum_3: pushl %ebp movl %esp,%ebp subl $8,%esp ..EN1: fildl 8(%ebp) fadds 12(%ebp) faddl 16(%ebp) fstps -4(%ebp) flds -4(%ebp) addl $8,%esp leave ret .type sum_3,@function .size sum_3,.-sum_3 Linux86-64 Programming Model This section defines compiler and assembly language conventions for the use of certain aspects of an x64 processor running a linux86-64 operating system. These standards must be followed to guarantee that compilers, application programs, and operating systems written by different people and organizations will work together. The conventions supported by the PGCC ANSI C compiler implement the application binary interface (ABI) as defined in the System V Application Binary Interface: AMD64 Architecture Processor Supplement and the System V Application Binary Interface, listed in the “Related Publications” section in the Preface. Note The programming model used for Win64 and SUA64 differs from the Linux86-64 model. For more information, refer to “Win64 Programming Model,” on page 287.PGI® User’s Guide 278 Function Calling Sequence This section describes the standard function calling sequence, including the stack frame, register usage, and parameter passing. Register Usage Conventions. The following table defines the standard for register allocation. The x64 Architecture provides a variety of registers. All the general purpose registers, XMM registers, and x87 registers are global to all procedures in a running program. Table 18.7. Register Allocation Type Name Purpose General %rax 1st return register %rbx callee-saved; optional base pointer %rcx pass 4th argument to functions %rdx pass 3rd argument to functions; 2nd return register %rsp stack pointer %rbp callee-saved; optional stack frame pointer %rsi pass 2nd argument to functions %rdi pass 1st argument to functions %r8 pass 5th argument to functions %r9 pass 6th argument to functions %r10 temporary register; pass a function’s static chain pointer %r11 temporary register %r12-r15 callee-saved registers XMM %xmm0-%xmm1 pass and return floating point arguments %xmm2-%xmm7 pass floating point arguments %xmm8-%xmm15 temporary registers x87 %st(0) temporary register; return long double arguments %st(1) temporary register; return long double arguments %st(2) - %st(7) temporary registers In addition to the registers, each function has a frame on the run-time stack. This stack grows downward from high addresses. The next table shows the stack frame organization. Table 18.8. Standard Stack Frame Position Contents Frame 8n+16 (%rbp) argument eightbyte n previousChapter 18. Run-time Environment 279 Position Contents Frame . . . 16 (%rbp) argument eightbyte 0 8 (%rbp) return address current 0 (%rbp) caller's %rbp current -8 (%rbp) unspecified . . . 0 (%rsp) variable size -128 (%rsp) red zone Key points concerning the stack frame: • The end of the input argument area is aligned on a 16-byte boundary. • The 128-byte area beyond the location of %rsp is called the red zone and can be used for temporary local data storage. This area is not modified by signal or interrupt handlers. • A call instruction pushes the address of the next instruction (the return address) onto the stack. The return instruction pops the address off the stack and effectively continues execution at the next instruction after the call instruction. A function must preserve non-volatile registers (described below). Additionally, the called function must remove the return address from the stack, leaving the stack pointer (%rsp) with the value it had before the call instruction was executed. All registers on an x64 system are global and thus visible to both a calling and a called function. Registers %rbx, %rsp, %rbp, %r12, %r13, %r14, and %r15 are non-volatile across function calls. Therefore, a function must preserve these registers’ values for its caller. Remaining registers are volatile (scratch). If a calling function wants to preserve such a register value across a function call, it must save its value explicitly. Registers are used extensively in the standard calling sequence. The first six integer and pointer arguments are passed in these registers (listed in order): %rdi, %rsi, %rdx, %rcx, %r8, %r9. The first eight floating point arguments are passed in the first eight XMM registers: %xmm0, %xmm1, …, %xmm7. The registers %rax and %rdx are used to return integer and pointer values. The registers %xmm0 and %xmm1 are used to return floating point values. Additional registers with assigned roles in the standard calling sequence: %rsp The stack pointer holds the limit of the current stack frame, which is the address of the stack’s bottommost, valid word. The stack must be 16-byte aligned. %rbp The frame pointer holds a base address for the current stack frame. Consequently, a function has registers pointing to both ends of its frame. Incoming arguments reside in the previous frame, referenced as positive offsets from %rbp, while local variables reside in the current frame, referenced as negative offsets from %rbp. A function must preserve this register value for its caller.PGI® User’s Guide 280 RFLAGS The flags register contains the system flags, such as the direction flag and the carry flag. The direction flag must be set to the “forward” (i.e., zero) direction before entry and upon exit from a function. Other user flags have no specified role in the standard calling sequence and are not preserved. Floating Point Control Word The control word contains the floating-point flags, such as the rounding mode and exception masking. This register is initialized at process initialization time and its value must be preserved. Signals can interrupt processes. Functions called during signal handling have no unusual restriction on their use of registers. Moreover, if a signal handling function returns, the process resumes its original execution path with registers restored to their original values. Thus, programs and compilers may freely use all registers without danger of signal handlers changing their values. Function Return Values Functions Returning Scalars or No Value • A function that returns an integral or pointer value places its result in the next available register of the sequence %rax, %rdx. • A function that returns a floating point value that fits in the XMM registers returns this value in the next available XMM register of the sequence %xmm0, %xmm1. • An X87 floating-point return value appears on the top of the floating point stack in %st(0) as an 80-bit X87 number. If this X87 return value is a complex number, the real part of the value is returned in %st(0) and the imaginary part in %st(1). • A function that returns a value in memory also returns the address of this memory in %rax. • Functions that return no value (also called procedures or void functions) put no particular value in any register. Functions Returning Structures or Unions A function can use either registers or memory to return a structure or union. The size and type of the structure or union determine how it is returned. If a structure or union is larger than 16 bytes, it is returned in memory allocated by the caller. To determine whether a 16-byte or smaller structure or union can be returned in one or more return registers, examine the first eight bytes of the structure or union. The type or types of the structure or union’s fields making up these eight bytes determine how these eight bytes will be returned. If the eight bytes contain at least one integral type, the eight bytes will be returned in %rax even if non-integral types are also present in the eight bytes. If the eight bytes only contain floating point types, these eight bytes will be returned in %xmm0. If the structure or union is larger than eight bytes but smaller than 17 bytes, examine the type or types of the fields making up the second eight bytes of the structure or union. If these eight bytes contain at least one integral type, these eight bytes will be returned in %rdx even if non-integral types are also present in the eight bytes. If the eight bytes only contain floating point types, these eight bytes will be returned in %xmm1.Chapter 18. Run-time Environment 281 If a structure or union is returned in memory, the caller provides the space for the return value and passes its address to the function as a “hidden” first argument in %rdi. This address will also be returned in %rax. Argument Passing Integral and Pointer Arguments Integral and pointer arguments are passed to a function using the next available register of the sequence %rdi, %rsi, %rdx, %rcx, %r8, %r9. After this list of registers has been exhausted, all remaining integral and pointer arguments are passed to the function via the stack. Floating-Point Arguments Float and double arguments are passed to a function using the next available XMM register taken in the order from %xmm0 to %xmm7. After this list of registers has been exhausted, all remaining float and double arguments are passed to the function via the stack. Structure and Union Arguments Structure and union arguments can be passed to a function in either registers or on the stack. The size and type of the structure or union determine how it is passed. If a structure or union is larger than 16 bytes, it is passed to the function in memory. To determine whether a 16-byte or smaller structure or union can be passed to a function in one or two registers, examine the first eight bytes of the structure or union. The type or types of the structure or union’s fields making up these eight bytes determine how these eight bytes will be passed. If the eight bytes contain at least one integral type, the eight bytes will be passed in the first available general purpose register of the sequence %rdi, %rsi, %rdx, %rcx, %r8, %r9 even if non-integral types are also present in the eight bytes. If the eight bytes only contain floating point types, these eight bytes will be passed in the first available XMM register of the sequence from %xmm0 to %xmm7. If the structure or union is larger than eight bytes but smaller than 17 bytes, examine the type or types of the fields making up the second eight bytes of the structure or union. If the eight bytes contain at least one integral type, the eight bytes will be passed in the next available general purpose register of the sequence %rdi, %rsi, %rdx, %rcx, %r8, %r9 even if non-integral types are also present in the eight bytes. If these eight bytes only contain floating point types, these eight bytes will be passed in the next available XMM register of the sequence from %xmm0 to %xmm7. If the first or second eight bytes of the structure or union cannot be passed in a register for some reason, the entire structure or union must be passed in memory. Passing Arguments on the Stack If there are arguments left after every argument register has been allocated, the remaining arguments are passed to the function on the stack. The unassigned arguments are pushed on the stack in reverse order, with the last argument pushed first. Table 18.9, “Register Allocation for Example A-2” shows the register allocation and stack frame offsets for the function declaration and call shown in the following example. Both table and example are adapted from System V Application Binary Interface: AMD64 Architecture Processor Supplement.PGI® User’s Guide 282 Example 18.2. Parameter Passing typedef struct { int a, b; double d; } structparam; structparam s; int e, f, g, h, i, j, k; float flt; double m, n; extern void func(int e, int f, structparam s, int g, int h, float flt, double m, double n, int i, int j, int k); void func2() { func(e, f, s, g, h, flt, m, n, i, j, k); } Table 18.9. Register Allocation for Example A-2 General Purpose Registers Floating Point Registers Stack Frame Offset %rdi: e %xmm0: s.d 0: j %rsi: f %xmm1: flt 8: k %rdx: s.a,s.b %xmm2: m %rcx: g %xmm3: n %r8: h %r9: i Implementing a Stack In general, compilers and programmers must maintain a software stack. The stack pointer, register %rsp, is set by the operating system for the application when the program is started. The stack must grow downwards from high addresses. A separate frame pointer enables calls to routines that change the stack pointer to allocate space on the stack at run-time (e.g. alloca). Some languages can also return values from a routine allocated on stack space below the original top-of-stack pointer. Such a routine prevents the calling function from using %rsp-relative addressing for values on the stack. If the compiler does not call routines that leave %rsp in an altered state when they return, a frame pointer is not needed and may not be used if the compiler option –Mnoframe is specified. The stack must be kept aligned on 16-byte boundaries. Variable Length Parameter Lists. Parameter passing in registers can handle a variable number of parameters. The C language uses a special method to access variable-count parameters. The stdarg.h and varargs.h files define several functions to access these parameters. A C routine with variable parameters must use the va_start macro to set up a data structure before the parameters can be used. The va_arg macro must be used to access the successive parameters.Chapter 18. Run-time Environment 283 For calls that use varargs or stdargs, the register %rax acts as a hidden argument whose value is the number of XMM registers used in the call. C Parameter Conversion. In C, for a called prototyped function, the parameter type in the called function must match the argument type in the calling function. If the called function is not prototyped, the calling convention uses the types of the arguments but promotes char or short to int, and unsigned char or unsigned short to unsigned int and promotes float to double, unless you use the ##Msingle option. For more information on the –Msingle option, refer to Chapter 3. Calling Assembly Language Programs Example 18.3. C Program Calling an Assembly-language Routine /* File: testmain.c */ #include int main() { long l_para1 = 2; float f_para2 = 1.0; double d_para3 = 0.5; float f_return; extern float sum_3(long para1, float para2, double para3); f_return = sum_3(l_para1, f_para2, d_para3); printf("Parameter one, type long = %ld\n", l_para1); printf("Parameter two, type float = %f\n", f_para2); printf("Parameter three, type double = %f\n", d_para3); printf("The sum after conversion = %f\n", f_return); return 0; } # File: sum_3.s # Computes ( para1 + para2 ) + para3 .text .align 16 .globl sum_3 sum_3: pushq %rbp movq %rsp, %rbp cvtsi2ssq %rdi, %xmm2 addss %xmm0, %xmm2 cvtss2sd %xmm2,%xmm2 addsd %xmm1, %xmm2 cvtsd2ss %xmm2, %xmm2 movaps %xmm2, %xmm0 popq %rbp ret .type sum_3, @function .size sum_3,.-sum_3 Linux86-64 Fortran Supplement Sections A2.4.1 through A2.4.4 define the Fortran supplement to the ABI for x64 Linux. The register usage conventions set forth in that document remain the same for Fortran.PGI® User’s Guide 284 Fortran Fundamental Types Table 18.10. Linux86-64 Fortran Fundamental Types Fortran Type Size (bytes) Alignment (bytes) INTEGER 4 4 INTEGER*1 1 1 INTEGER*2 2 2 INTEGER*4 4 4 INTEGER*8 8 8 LOGICAL 4 4 LOGICAL*1 1 1 LOGICAL*2 2 2 LOGICAL*4 4 4 LOGICAL*8 8 8 BYTE 1 1 CHARACTER*n n 1 REAL 4 4 REAL*4 4 4 REAL*8 8 8 DOUBLE PRECISION 8 8 COMPLEX 8 4 COMPLEX*8 8 4 COMPLEX*16 16 8 DOUBLE COMPLEX 16 8 A logical constant is one of: • .TRUE. • .FALSE. The logical constants .TRUE. and .FALSE. are defined to be the four-byte values -1 and 0 respectively. A logical expression is defined to be .TRUE. if its least significant bit is 1 and .FALSE. otherwise. Note that the value of a character is not automatically NULL-terminated. Naming Conventions By default, all globally visible Fortran symbol names (subroutines, functions, common blocks) are converted to lower-case. In addition, an underscore is appended to Fortran global names to distinguish the Fortran name space from the C/C++ name space.Chapter 18. Run-time Environment 285 Argument Passing and Return Conventions Arguments are passed by reference (i.e. the address of the argument is passed, rather than the argument itself). In contrast, C/C++ arguments are passed by value. When passing an argument declared as Fortran type CHARACTER, an argument representing the length of the CHARACTER argument is also passed to the function. This length argument is a four-byte integer passed by value, and is passed at the end of the parameter list following the other formal arguments. A length argument is passed for each CHARACTER argument; the length arguments are passed in the same order as their respective CHARACTER arguments. A Fortran function, returning a value of type CHARACTER, adds two arguments to the beginning of its argument list. The first additional argument is the address of the area created by the caller for the return value; the second additional argument is the length of the return value. If a Fortran function is declared to return a character value of constant length, for example CHARACTER*4 FUNCTION CHF(), the second extra parameter representing the length of the return value must still be supplied. A Fortran complex function returns its value in memory. The caller provides space for the return value and passes the address of this storage as if it were the first argument to the function. Alternate return specifiers of a Fortran function are not passed as arguments by the caller. The alternate return function passes the appropriate return value back to the caller in %rax. The handling of the following Fortran 90 features is implementation-defined: internal procedures, pointer arguments, assumed-shape arguments, functions returning arrays, and functions returning derived types. Inter-language Calling Inter-language calling between Fortran and C/C++ is possible if function/subroutine parameters and return values match types. If a C/C++ function returns a value, call it from Fortran as a function, otherwise, call it as a subroutine. If a Fortran function has type CHARACTER or COMPLEX, call it from C/C++ as a void function. If a Fortran subroutine has alternate returns, call it from C/C++ as a function returning int; the value of such a subroutine is the value of the integer expression specified in the alternate RETURN statement. If a Fortran subroutine does not contain alternate returns, call it from C/C++ as a void function. The following table provides the C/C++ data type corresponding to each Fortran data type. Table 18.11. Fortran and C/C++ Data Type Compatibility Fortran Type C/C++ Type Size (bytes) CHARACTER*n x char x[n] n REAL x float x 4 REAL*4 x float x 4 REAL*8 x double x 8 DOUBLE PRECISION x double x 8 INTEGER x int x 4 INTEGER*1 x signed char x 1PGI® User’s Guide 286 Fortran Type C/C++ Type Size (bytes) INTEGER*2 x short x 2 INTEGER*4 x int x 4 INTEGER*8 x long x, or long long x 8 LOGICAL x int x 4 LOGICAL*1 x char x 1 LOGICAL*2 x short x 2 LOGICAL*4 x int x 4 LOGICAL*8 x long x, or long long x 8 Table 18.12. Fortran and C/C++ Representation of the COMPLEX Type Fortran Type (lower case) C/C++ Type Size (bytes) complex x struct {float r,i;} x; 8 float complex x; complex*8 x struct {float r,i;} x; 8 float complex x; 8 double complex x struct {double dr,di;} x; 16 double complex x; 16 complex *16 x struct {double dr,di;} x; 16 double complex x; 16 Note For C/C++, the complex type implies C99 or later. Arrays C/C++ arrays and Fortran arrays use different default initial array index values. By default, C/C++ arrays start at 0 and Fortran arrays start at 1. A Fortran array can be declared to start at zero. Another difference between Fortran and C/C++ arrays is the storage method used. Fortran uses columnmajor order and C/C++ use row-major order. For one-dimensional arrays, this poses no problems. For two-dimensional arrays, where there are an equal number of rows and columns, row and column indexes can simply be reversed. Inter-language function mixing is not recommended for arrays other than single dimensional arrays and square two-dimensional arrays. Structures, Unions, Maps, and Derived Types. Fields within Fortran structures and derived types, and multiple map declarations within a Fortran union, conform to the same alignment requirements used by C structures.Chapter 18. Run-time Environment 287 Common Blocks. A named Fortran common block can be represented in C/C++ by a structure whose members correspond to the members of the common block. The name of the structure in C/C++ must have the added underscore. For example, the Fortran common block: INTEGER I, J COMPLEX C DOUBLE COMPLEX CD DOUBLE PRECISION D COMMON /COM/ i, j, c, cd, d is represented in C with the following equivalent: extern struct { int i; int j; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; and in C++ with the following equivalent: extern "C" struct { int i; int j; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; Note that the compiler-provided name of the BLANK COMMON block is implementation specific. Calling Fortran COMPLEX and CHARACTER functions from C/C++ is not as straightforward as calling other types of Fortran functions. Additional arguments must be passed to the Fortran function by the C/C++ caller. A Fortran COMPLEX function returns its value in memory; the first argument passed to the function must contain the address of the storage for this value. A Fortran CHARACTER function adds two arguments to the beginning of its argument list. The following example of calling a Fortran CHARACTER function from C/C++ illustrates these caller-provided extra parameters: CHARACTER*(*) FUNCTION CHF(C1, I) CHARACTER*(*) C1 INTEGER I END extern void chf_(); char tmp[10]; char c1[9]; int i; chf_(tmp, 10, c1, &i, 9); The extra parameters tmp and 10 are supplied for the return value, while 9 is supplied as the length of c1. Refer to Section 2.8, Argument Passing and Return Conventions, for additional information. Win64 Programming Model This section defines compiler and assembly language conventions for the use of certain aspects of an x64 processor running a Win64 operating system, including SUA64. These standards must be followed to guarantee that compilers, application programs, and operating systems written by different people and organizationsPGI® User’s Guide 288 will work together. The conventions supported by the PGCC ANSI C compiler implement the application binary interface (ABI) as defined in the AMD64 Software Conventions document. Function Calling Sequence This section describes the standard function calling sequence, including the stack frame, register usage, and parameter passing. Register Usage Conventions. The following table defines the standard for register allocation. The 64-bit AMD64 and EM64T Architectures provide a number of registers. All the general purpose registers, XMM registers, and x87 registers are global to all procedures in a running program. Table 18.13. Register Allocation Type Name Purpose General %rax return value register %rbx callee-saved %rcx pass 1st argument to functions %rdx pass 2nd argument to functions %rsp stack pointer %rbp callee-saved; optional stack frame pointer %rsi callee-saved %rdi callee-saved %r8 pass 3rd argument to functions %r9 pass 4th argument to functions %r10-%r11 temporary registers; used in syscall/sysret instructions %r12-r15 callee-saved registers XMM %xmm0 pass 1st floating point argument; return value register %xmm1 pass 2nd floating point argument %xmm2 pass 3rd floating point argument %xmm3 pass 4th floating point argument %xmm4-%xmm5 temporary registers %xmm6-%xmm15 callee-saved registers In addition to the registers, each function has a frame on the run-time stack. This stack grows downward from high addresses. The next table shows the stack frame organization. Table 18.14. Standard Stack Frame Position Contents Frame 8n-120 (%rbp) argument eightbyte n previousChapter 18. Run-time Environment 289 Position Contents Frame . . . -80 (%rbp) argument eightbyte 5 -88 (%rbp) %r9 home -96 (%rbp) %r8 home -104 (%rbp) %rdx home -112 (%rbp) %rcx home -120 (%rbp) return address current -128 (%rbp) caller's %rbp . . . 0 (%rsp) variable size Key points concerning the stack frame: • The parameter area at the bottom of the stack must contain enough space to hold all the parameters needed by any function call. Space must be set aside for the four register parameters to be “homed” to the stack even if there are less than four register parameters used in a given call. • Sixteen-byte alignment of the stack is required except within a function’s prolog and within leaf functions. All registers on an x64 system are global and thus visible to both a calling and a called function. Registers %rbx, %rsp, %rbp, %rsi, %rdi, %r12, %r13, %r14, and %r15 are non-volatile. Therefore, a called function must preserve these registers’ values for its caller. Remaining registers are scratch. If a calling function wants to preserve such a register value across a function call, it must save a value in its local stack frame. Registers are used in the standard calling sequence. The first four arguments are passed in registers. Integral and pointer arguments are passed in these general purpose registers (listed in order): %rcx, %rdx, %r8, %r9. Floating point arguments are passed in the first four XMM registers: %xmm0, %xmm1, %xmm2, %xmm3. Registers are assigned using the argument’s ordinal position in the argument list. For example, if a function’s first argument is an integral type and its second argument is a floating-point type, the first argument will be passed in the first general purpose register (%rcx) and the second argument will be passed in the second XMM register (%xmm1); the first XMM register and second general purpose register are ignored. Arguments after the first four are passed on the stack. Integral and pointer type return values are returned in %rax. Floating point return values are returned in %xmm0. Additional registers with assigned roles in the standard calling sequence: %rsp The stack pointer holds the limit of the current stack frame, which is the address of the stack’s bottommost, valid word. The stack pointer should point to a 16-byte aligned area unless in the prolog or a leaf function.PGI® User’s Guide 290 %rbp The frame pointer, if used, can provide a way to reference the previous frame on the stack. Details are implementation dependent. A function must preserve this register value for its caller. MXCSR The flags register MXCSR contains the system flags, such as the direction flag and the carry flag. The six status flags (MXCSR[0:5]) are volatile; the remainder of the register is nonvolatile. x87 Floating Point Control Word (FPCSR) The control word contains the floating-point flags, such as the rounding mode and exception masking. This register is initialized at process initialization time and its value must be preserved. Signals can interrupt processes. Functions called during signal handling have no unusual restriction on their use of registers. Moreover, if a signal handling function returns, the process resumes its original execution path with registers restored to their original values. Thus, programs and compilers may freely use all registers without danger of signal handlers changing their values. Function Return Values Functions Returning Scalars or No Value • A function that returns an integral or pointer value that fits in 64 bits places its result in %rax. • A function that returns a floating point value that fits in the XMM registers returns this value in %xmm0. • A function that returns a value in memory via the stack places the address of this memory (passed to the function as a “hidden” first argument in %rcx) in %rax. • Functions that return no value (also called procedures or void functions) put no particular value in any register. • A call instruction pushes the address of the next instruction (the return address) onto the stack. The return instruction pops the address off the stack and effectively continues execution at the next instruction after the call instruction. A function that returns a scalar or no value must preserve the caller's registers as described above. Additionally, the called function must remove the return address from the stack, leaving the stack pointer (%rsp) with the value it had before the call instruction was executed. Functions Returning Structures or Unions A function can use either registers or the stack to return a structure or union. The size and type of the structure or union determine how it is returned. A structure or union is returned in memory if it is larger than 8 bytes or if its size is 3, 5, 6, or 7 bytes. A structure or union is returned in %rax if its size is 1, 2, 4, or 8 bytes. If a structure or union is to be returned in memory, the caller provides space for the return value and passes its address to the function as a “hidden” first argument in %rcx. This address will also be returned in %rax.Chapter 18. Run-time Environment 291 Argument Passing Integral and Pointer Arguments Integral and pointer arguments are passed to a function using the next available register of the sequence %rcx, %rdx, %r8, %r9. After this list of registers has been exhausted, all remaining integral and pointer arguments are passed to the function via the stack. Floating-Point Arguments Float and double arguments are passed to a function using the next available XMM register of the sequence %xmm0, %xmm1, %xmm2, %xmm3. After this list of registers has been exhausted, all remaining XMM floating-point arguments are passed to the function via the stack. Array, Structure, and Union Arguments Arrays and strings are passed to functions using a pointer to caller-allocated memory. Structure and union arguments of size 1, 2, 4, or 8 bytes will be passed as if they were integers of the same size. Structures and unions of other sizes will be passed as a pointer to a temporary, allocated by the caller, and whose value contains the value of the argument. The caller-allocated temporary memory used for arguments of aggregate type must be 16-byte aligned. Passing Arguments on the Stack Registers are assigned using the argument’s ordinal position in the argument list. For example, if a function’s first argument is an integral type and its second argument is a floating-point type, the first argument will be passed in the first general purpose register (%rcx) and the second argument will be passed in the second XMM register (%xmm1); the first XMM register and second general purpose register are ignored. Arguments after the first four are passed on the stack; they are pushed on the stack in reverse order, with the last argument pushed first. Table 18.15, “Register Allocation for Example A-4” shows the register allocation and stack frame offsets for the function declaration and call shown in the following example. Example 18.4. Parameter Passing typedef struct { int i; float f; } struct1; int i; float f; double d; long l; long long ll; struct1 s1; extern void func (int i, float f, struct1 s1, double d, long long ll, long l); func (i, f, s1, d, ll, l);PGI® User’s Guide 292 Table 18.15. Register Allocation for Example A-4 General Purpose Registers Floating Point Registers Stack Frame Offset %rcx: i %xmm0: 32: ll %rdx: %xmm1: f 40: l %r8: s1.i, s1.f %xmm2: %r9: %xmm3: d Implementing a Stack In general, compilers and programmers must maintain a software stack. The stack pointer, register %rsp, is set by the operating system for the application when the program is started. The stack must grow downwards from high addresses. A separate frame pointer enables calls to routines that change the stack pointer to allocate space on the stack at run-time (e.g. alloca). Some languages can also return values from a routine allocated on stack space below the original top-of-stack pointer. Such a routine prevents the calling function from using %rsp-relative addressing to get at values on the stack. If the compiler does not call routines that leave %rsp in an altered state when they return, a frame pointer is not needed and is not used if the compiler option –Mnoframe is specified. The stack must always be 16-byte aligned except within the prolog and within leaf functions. Variable Length Parameter Lists. Parameter passing in registers can handle a variable number of parameters. The C language uses a special method to access variable-count parameters. The stdarg.h and varargs.h files define several functions to access these parameters. A C routine with variable parameters must use the va_start macro to set up a data structure before the parameters can be used. The va_arg macro must be used to access the successive parameters. For unprototyped functions or functions that use varargs, floating-point arguments passed in registers must be passed in both an XMM register and its corresponding general purpose register. C Parameter Conversion. In C, for a called prototyped function, the parameter type in the called function must match the argument type in the calling function. If the called function is not prototyped, the calling convention uses the types of the arguments but promotes char or short to int, and unsigned char or unsigned short to unsigned int and promotes float to double, unless you use the –Msingle option. For more information on the –Msingle option, refer to Chapter 3. If the called function is prototyped, the unused bits of a register containing a char or short parameter are undefined and the called function must extend the sign of the unused bits when needed.Chapter 18. Run-time Environment 293 Calling Assembly Language Programs Example 18.5. C Program Calling an Assembly-language Routine /* File: testmain.c */ main() { long l_para1 = 0x3f800000; float f_para2 = 1.0; double d_para3 = 0.5; float f_return; extern float sum_3 (long para1, float para2, double para3); f_return = sum_3(l_para1,f_para2, d_para3); printf("Parameter one, type long = %08x\n",l_para1); printf("Parameter two, type float = %f\n",f_para2); printf("Parameter three, type double = %g\n",d_para3); printf("The sum after conversion = %f\n",f_return); } # File: sum_3.s # Computes ( para1 + para2 ) + para3 .text .align 16 .globl sum_3 sum_3: pushq %rbp leaq 128(%rsp), %rbp cvtsi2ss %ecx, %xmm0 addss %xmm1, %xmm0 cvtss2sd %xmm0, %xmm0 addsd %xmm2, %xmm0 cvtsd2ss %xmm0, %xmm0 popq %rbp ret .type sum_3,@function .size sum_3,.-sum_3 Win64/SUA64 Fortran Supplement Sections A3.4.1 through A3.4.4 define the Fortran supplement to the AMD64 Software Conventions for Win64. The register usage conventions set forth in that document remain the same for Fortran. Fortran Fundamental Types Table 18.16. Win64 Fortran Fundamental Types Fortran Type Size (bytes) Alignment (bytes) INTEGER 4 4 INTEGER*1 1 1 INTEGER*2 2 2 INTEGER*4 4 4 INTEGER*8 8 8 LOGICAL 4 4PGI® User’s Guide 294 Fortran Type Size (bytes) Alignment (bytes) LOGICAL*1 1 1 LOGICAL*2 2 2 LOGICAL*4 4 4 LOGICAL*8 8 8 BYTE 1 1 CHARACTER*n n 1 REAL 4 4 REAL*4 4 4 REAL*8 8 8 DOUBLE PRECISION 8 8 COMPLEX 8 4 COMPLEX*8 8 4 COMPLEX*16 16 8 DOUBLE COMPLEX 16 8 A logical constant is one of: • .TRUE. • .FALSE. The logical constants .TRUE. and .FALSE. are defined to be the four-byte values -1 and 0 respectively. A logical expression is defined to be .TRUE. if its least significant bit is 1 and .FALSE. otherwise. Note that the value of a character is not automatically NULL-terminated. Fortran Naming Conventions By default, all globally visible Fortran symbol names (subroutines, functions, common blocks) are converted to lower-case. In addition, an underscore is appended to Fortran global names to distinguish the Fortran name space from the C/C++ name space. Fortran Argument Passing and Return Conventions Arguments are passed by reference (i.e. the address of the argument is passed, rather than the argument itself). In contrast, C/C++ arguments are passed by value. When passing an argument declared as Fortran type CHARACTER, an argument representing the length of the CHARACTER argument is also passed to the function. This length argument is a four-byte integer passed by value, and is passed at the end of the parameter list following the other formal arguments. A length argument is passed for each CHARACTER argument; the length arguments are passed in the same order as their respective CHARACTER arguments.Chapter 18. Run-time Environment 295 A Fortran function, returning a value of type CHARACTER, adds two arguments to the beginning of its argument list. The first additional argument is the address of the area created by the caller for the return value; the second additional argument is the length of the return value. If a Fortran function is declared to return a character value of constant length, for example CHARACTER*4 FUNCTION CHF(), the second extra parameter representing the length of the return value must still be supplied. A Fortran complex function returns its value in memory. The caller provides space for the return value and passes the address of this storage as if it were the first argument to the function. Alternate return specifiers of a Fortran function are not passed as arguments by the caller. The alternate return function passes the appropriate return value back to the caller in %rax. The handling of the following Fortran 90 features is implementation-defined: internal procedures, pointer arguments, assumed-shape arguments, functions returning arrays, and functions returning derived types. Inter-language Calling Inter-language calling between Fortran and C/C++ is possible if function/subroutine parameters and return values match types. If a C/C++ function returns a value, call it from Fortran as a function, otherwise, call it as a subroutine. If a Fortran function has type CHARACTER or COMPLEX, call it from C/C++ as a void function. If a Fortran subroutine has alternate returns, call it from C/C++ as a function returning int; the value of such a subroutine is the value of the integer expression specified in the alternate RETURN statement. If a Fortran subroutine does not contain alternate returns, call it from C/C++ as a void function. The following table provides the C/C++ data type corresponding to each Fortran data type. Table 18.17. Fortran and C/C++ Data Type Compatibility Fortran Type C/C++ Type Size (bytes) CHARACTER*n x char x[n] n REAL x float x 4 REAL*4 x float x 4 REAL*8 x double x 8 DOUBLE PRECISION x double x 8 INTEGER x int x 4 INTEGER*1 x signed char x 1 INTEGER*2 x short x 2 INTEGER*4 x int x 4 INTEGER*8 x long long x 8 LOGICAL x int x 4 LOGICAL*1 x char x 1 LOGICAL*2 x short x 2 LOGICAL*4 x int x 4PGI® User’s Guide 296 Fortran Type C/C++ Type Size (bytes) LOGICAL*8 x long long x 8 Table 18.18. Fortran and C/C++ Representation of the COMPLEX Type Fortran Type (lower case) C/C++ Type Size (bytes) complex x struct {float r,i;} x; 8 float complex x; complex*8 x struct {float r,i;} x; 8 float complex x; 8 double complex x struct {double dr,di;} x; 16 double complex x; 16 complex *16 x struct {double dr,di;} x; 16 double complex x; 16 Note For C/C++, the complex type implies C99 or later. Arrays C/C++ arrays and Fortran arrays use different default initial array index values. By default, C/C++ arrays start at 0 and Fortran arrays start at 1. A Fortran array can be declared to start at zero. Another difference between Fortran and C/C++ arrays is the storage method used. Fortran uses columnmajor order and C/C++ use row-major order. For one-dimensional arrays, this poses no problems. For two-dimensional arrays, where there are an equal number of rows and columns, row and column indexes can simply be reversed. Inter-language function mixing is not recommended for arrays other than single dimensional arrays and square two-dimensional arrays. Structures, Unions, Maps, and Derived Types. Fields within Fortran structures and derived types, and multiple map declarations within a Fortran union, conform to the same alignment requirements used by C structures. Common Blocks. A named Fortran common block can be represented in C/C++ by a structure whose members correspond to the members of the common block. The name of the structure in C/C++ must have the added underscore. For example, the Fortran common block: INTEGER I, J COMPLEX C DOUBLE COMPLEX CD DOUBLE PRECISION D COMMON /COM/ i, j, c, cd, dChapter 18. Run-time Environment 297 is represented in C with the following equivalent: extern struct { int i; int j; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; and in C++ with the following equivalent: extern "C" struct { int i; int j; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; Note that the compiler-provided name of the BLANK COMMON block is implementation specific. Calling Fortran COMPLEX and CHARACTER functions from C/C++ is not as straightforward as calling other types of Fortran functions. Additional arguments must be passed to the Fortran function by the C/C++ caller. A Fortran COMPLEX function returns its value in memory; the first argument passed to the function must contain the address of the storage for this value. A Fortran CHARACTER function adds two arguments to the beginning of its argument list. The following example of calling a Fortran CHARACTER function from C/C++ illustrates these caller-provided extra parameters: CHARACTER*(*) FUNCTION CHF(C1, I) CHARACTER*(*) C1 INTEGER I END extern void chf_(); char tmp[10]; char c1[9]; int i; chf_(tmp, 10, c1, &i, 9); The extra parameters tmp and 10 are supplied for the return value, while 9 is supplied as the length of c1. Refer to “Argument Passing and Return Values,” on page 113, for additional information.298299 Chapter 19. C++ Dialect Supported The PGC++ compiler accepts the C++ language of the ISO/IEC 14882:1998 C++ standard, except for Exported Templates.PGC++ optionally accepts a number of features erroneously accepted by cfront version 2.1 or 3.0. Using the -b option, PGC++ accepts these features, which may never have been legal C++, but have found their way into some user’s code. Command-line options provide full support of many C++ variants, including strict standard conformance. PGC++ provides command line options that enable the user to specify whether anachronisms and/or cfront 2.1/3.0 compatibility features should be accepted. Extensions Accepted in Normal C++ Mode The following extensions are accepted in all modes (except when strict ANSI violations are diagnosed as errors, see the –A option): • A friend declaration for a class may omit the class keyword: class A { friend B; // Should be "friend class B" }; • Constants of scalar type may be defined within classes: class A { const int size = 10; int a[size]; }; • In the declaration of a class member, a qualified name may be used: struct A{ int A::f(); // Should be int f(); } • The preprocessing symbol c_plusplus is defined in addition to the standard __cplusplus. • An assignment operator declared in a derived class with a parameter type matching one of its base classes is treated as a "default'' assignment operator --- that is, such a declaration blocks the implicit generation of a copy assignment operator. (This is cfront behavior that is known to be relied upon in at least one widely used library.) Here's an example: struct A { } ;PGI® User’s Guide 300 struct B : public A { B& operator=(A&); }; • By default, as well as in cfront-compatibility mode, there will be no implicit declaration of B::operator=(const B&), whereas in strict-ANSI mode B::operator=(A&) is not a copy assignment operator and B::operator=(const B&) is implicitly declared. • Implicit type conversion between a pointer to an extern “C” function and a pointer to an extern “C++” function is permitted. Here’s an example: extern "C" void f(); // f’s type has extern "C" linkage void (*pf) () // pf points to an extern "C++" function = &f; // error unless implicit conv is allowed cfront 2.1 Compatibility Mode The following extensions are accepted in cfront 2.1 compatibility mode in addition to the extensions listed in the following 2.1/3.0 section (i.e., these are things that were corrected in the 3.0 release of cfront): • The dependent statement of an if, while, do-while, or for is not considered to define a scope. The dependent statement may not be a declaration. Any objects constructed within the dependent statement are destroyed at exit from the dependent statement. • Implicit conversion from integral types to enumeration types is allowed. • A non-const member function may be called for a const object. A warning is issued. • A const void * value may be implicitly converted to a void * value, e.g., when passed as an argument. • When, in determining the level of argument match for overloading, a reference parameter is initialized from an argument that requires a non-class standard conversion, the conversion counts as a user-defined conversion. (This is an outright bug, which unfortunately happens to be exploited in some class libraries.) • When a builtin operator is considered alongside overloaded operators in overload resolution, the match of an operand of a builtin type against the builtin type required by the builtin operator is considered a standard conversion in all cases (e.g., even when the type is exactly right without conversion). • A reference to a non-const type may be initialized from a value that is a const-qualified version of the same type, but only if the value is the result of selecting a member from a const class object or a pointer to such an object. • A cast to an array type is allowed; it is treated like a cast to a pointer to the array element type. A warning is issued. • When an array is selected from a class, the type qualifiers on the class object (if any) are not preserved in the selected array. (In the normal mode, any type qualifiers on the object are preserved in the element type of the resultant array.) • An identifier in a function is allowed to have the same name as a parameter of the function. A warning is issued.Chapter 19. C++ Dialect Supported 301 • An expression of type void may be supplied on the return statement in a function with a void return type. A warning is issued. • cfront has a bug that causes a global identifier to be found when a member of a class or one of its base classes should actually be found. This bug is not emulated in cfront compatibility mode. • A parameter of type "const void *'' is allowed on operator delete; it is treated as equivalent to "void *". • A period (".") may be used for qualification where "::" should be used. Only "::'' may be used as a global qualifier. Except for the global qualifier, the two kinds of qualifier operators may not be mixed in a given name (i.e., you may say A::B::C or A.B.C but not A::B.C or A.B::C). A period may not be used in a vacuous destructor reference nor in a qualifier that follows a template reference such as A::B. • cfront 2.1 does not correctly look up names in friend functions that are inside class definitions. In this example function f should refer to the functions and variables (e.g., f1 and a1) from the class declaration. Instead, the global definitions are used. int a1; int e1; void f1(); class A { int a1; void f1(); friend void f() { int i1 = a1; // cfront uses global a1 f1(); // cfront uses global f1 } }; • Only the innermost class scope is (incorrectly) skipped by cfront as illustrated in the following example. int a1; int b1; struct A { static int a1; class B { static int b1; friend void f() { int i1 = a1; // cfront uses A::a1 int j1 = b1; // cfront uses global b1 } }; }; • operator= may be declared as a nonmember function. (This is flagged as an anachronism by cfront 2.1) • A type qualifier is allowed (but ignored) on the declaration of a constructor or destructor. For example: class A { A() const; // No error in cfront 2.1 mode }; cfront 2.1/3.0 Compatibility Mode The following extensions are accepted in both cfront 2.1 and cfront 3.0 compatibility mode (i.e., these are features or problems that exist in both cfront 2.1 and 3.0):PGI® User’s Guide 302 • Type qualifiers on the this parameter may to be dropped in contexts such as this example: struct A { void f() const; }; void (A::*fp)() = &A::f; This is actually a safe operation. A pointer to a const function may be put into a pointer to non-const, because a call using the pointer is permitted to modify the object and the function pointed to will actually not modify the object. The opposite assignment would not be safe. • Conversion operators specifying conversion to void are allowed. • A nonstandard friend declaration may introduce a new type. A friend declaration that omits the elaborated type specifier is allowed in default mode, but in cfront mode the declaration is also allowed to introduce a new type name. struct A { friend B; }; • The third operator of the ? operator is a conditional expression instead of an assignment expression. • A reference to a pointer type may be initialized from a pointer value without use of a temporary even when the reference pointer type has additional type qualifiers above those present in the pointer value. For example, int *p; const int *&r = p; // No temporary used • A reference may be initialized with a null.303 Chapter 20. C/C++ MMX/SSE Inline Intrinsics An intrinsic is a function available in a given language whose implementation is handled specifically by the compiler. Typically, an intrinsic substitutes a sequence of automatically-generated instructions for the original function call. Since the compiler has an intimate knowledge of the intrinsic function, it can better integrate it and optimize it for the situation. PGI provides support for MMX and SSE/SSE2/SSE3/SSSE3/SSE4a/ABM intrinsics in C/C++ programs. The definitions of the intrinsics are in the inline library libintrinsics.il, which is automatically included in your compilation. Intrinsics make the use of processor-specific enhancements easier because they provide a C/C++ language interface to assembly instructions. In doing so, the compiler manages things that the user would normally have to be concerned with, such as register names, register allocations, and memory locations of data. This chapter contains these seven tables associated with inline intrinsics: • A table of MMX inline intrinsics (mmintrin.h) • A table of SSE inline intrinsics (xmmintrin.h) • A table of SSE2 inline intrinsics (emmintrin.h) • A table of SSE3 inline intrinsics (pmmintrin.h) • A table of SSSE3 inline intrinsics (tmmintrin.h) • A table of SSE4a inline intrinsics (ammintrin.h) • A table of ABM inline intrinsics (intrin.h) Using Intrinsic functions The definitions of the intrinsics are provided in the inline library libintrinsics.il., which is automatically included when you compile.PGI® User’s Guide 304 Required Header File To call these intrinsic functions from a C/C++ source, you must include the corresponding header file - one of the following: • For MMX, use mmintrin.h • For SSE, use xmmintrin.h • For SSE2, use emmintrin.h • For SSE3, use pmmintrin.h • For SSSE3 use tmmintrin.h • For SSE4a use ammintrin.h • For ABM use intrin.h Intrinsic Data Types The following table describes the data types that are defined for intrinsics: Data Types Defined in Description __m64 mmintrin.h For use with MMX intrinsics, this 64-bit data type stores one 64-bit or two 32-bit integer values. __m128 xmmintrin.h For use with SSE intrinsics, this 128-bit data type, aligned on 16-byte boundaries, stores four single-precision floating point values. __m128d emmintrin.h For use with SSE2/SSE3 intrinsics, this 128-bit data type, aligned on 16-byte boundaries, stores two double-precision floating point values. __m128i emmintrin.h For use with SSE2/SSE3 intrinsics, this 128-bit data type, aligned on 16-byte boundaries, stores two 64-bit integer values. Intrinsic Example The MMX/SSE intrinsics include functions for initializing variables of the types defined in the preceding table. The following sample program, example.c, illustrates the use of the SSE intrinsics _mm_add_ps and _mm_set_ps. #include int main(){ __m128 A, B, result; A = _mm_set_ps(23.3, 43.7, 234.234, 98.746); /* initialize A */ B = _mm_set_ps(15.4, 34.3, 4.1, 8.6); /* initialize B */ result = _mm_add_ps(A, B); return 0; } To compile this program, use the following command: $ pgcc example.c -o myprog The inline library libintrinsics.il is automatically inlined.Chapter 20. C/C++ MMX/SSE Inline Intrinsics 305 MMX Intrinsics PGI supports a set of MMX Intrinsics which allow the use of the MMX instructions directly from C/C++ code, without writing the assembly instructions. The following table lists the MMX intrinsics that PGI supports. Note Intrinsics with a * are only available on 64-bit systems. Table 20.1. MMX Intrinsics (mmintrin.h) _mm_empty _m_paddd _m_psllw _m_pand _m_empty _mm_add_si64 _mm_slli_pi16 _mm_andnot_si64 _mm_cvtsi32_si64 _mm_adds_pi8 _m_psllwi _m_pandn _m_from_int _m_paddsb _mm_sll_pi32 _mm_or_si64 _mm_cvtsi64x_si64* _mm_adds_pi16 _m_pslld _m_por _mm_set_pi64x* _m_paddsw _mm_slli_pi32 _mm_xor_si64 _mm_cvtsi64_si32 _mm_adds_pu8 _m_pslldi _m_pxor _m_to_int _m_paddusb _mm_sll_si64 _mm_cmpeq_pi8 _mm_cvtsi64_si64x* _mm_adds_pu16 _m_psllq _m_pcmpeqb _mm_packs_pi16* _m_paddusw _mm_slli_si64 _mm_cmpgt_pi8 _m_packsswb _mm_sub_pi8 _m_psllqi _m_pcmpgtb _mm_packs_pi32 _m_psubb _mm_sra_pi16 _mm_cmpeq_pi16 _m_packssdw _mm_sub_pi16 _m_psraw _m_pcmpeqw _mm_packs_pu16 _m_psubw _mm_srai_pi16 _mm_cmpgt_pi16 _m_packuswb _mm_sub_pi32 _m_psrawi _m_pcmpgtw _mm_unpackhi_pi8 _m_psubd _mm_sra_pi32 _mm_cmpeq_pi32 _m_punpckhbw _mm_sub_si64 _m_psrad _m_pcmpeqd _mm_unpackhi_pi16 _mm_subs_pi8 _mm_srai_pi32 _mm_cmpgt_pi32 _m_punpckhwd _m_psubsb _m_psradi _m_pcmpgtd _mm_unpackhi_pi32 _mm_subs_pi16 _mm_srl_pi16 _mm_setzero_si64 _m_punpckhdq _m_psubsw _m_psrlw _mm_set_pi32 _mm_unpacklo_pi8 _mm_subs_pu8 _mm_srli_pi16 _mm_set_pi16 _m_punpcklbw _m_psubusb _m_psrlwi _mm_set_pi8 _mm_unpacklo_pi16 _mm_subs_pu16 _mm_srl_pi32 _mm_setr_pi32 _m_punpcklwd _m_psubusw _m_psrld _mm_setr_pi16 _mm_unpacklo_pi32 _mm_madd_pi16 _mm_srli_pi32 _mm_setr_pi8 _m_punpckldq _m_pmaddwd _m_psrldi _mm_set1_pi32 _mm_add_pi8 _mm_mulhi_pi16 _mm_srl_si64 _mm_set1_pi16PGI® User’s Guide 306 _m_paddb _m_pmulhw _m_psrlq _mm_set1_pi8 _mm_add_pi16 _mm_mullo_pi16 _mm_srli_si64 _m_paddw _m_pmullw _m_psrlqi _mm_add_pi32 _mm_sll_pi16 _mm_and_si64 SSE Intrinsics PGI supports a set of SSE Intrinsics which allow the use of the SSE instructions directly from C/C++ code, without writing the assembly instructions. The following tables list the SSE intrinsics that PGI supports. Note Intrinsics with a * are only available on 64-bit systems. Table 20.2. SSE Intrinsics (xmmintrin.h) _mm_add_ss _mm_comige_ss _mm_load_ss _mm_sub_ss _mm_comineq_ss _mm_load1_ps _mm_mul_ss _mm_ucomieq_ss _mm_load_ps1 _mm_div_ss _mm_ucomilt_ss _mm_load_ps _mm_sqrt_ss _mm_ucomile_ss _mm_loadu_ps _mm_rcp_ss _mm_ucomigt_ss _mm_loadr_ps _mm_rsqrt_ss _mm_ucomige_ss _mm_set_ss _mm_min_ss _mm_ucomineq_ss _mm_set1_ps _mm_max_ss _mm_cvtss_si32 _mm_set_ps1 _mm_add_ps _mm_cvt_ss2si _mm_set_ps _mm_sub_ps _mm_cvtss_si64x* _mm_setr_ps _mm_mul_ps _mm_cvtps_pi32 _mm_store_ss _mm_div_ps _mm_cvt_ps2pi _mm_store_ps _mm_sqrt_ps _mm_cvttss_si32 _mm_store1_ps _mm_rcp_ps _mm_cvtt_ss2si _mm_store_ps1 _mm_rsqrt_ps _mm_cvttss_si64x* _mm_storeu_ps _mm_min_ps _mm_cvttps_pi32 _mm_storer_ps _mm_max_ps _mm_cvtt_ps2pi _mm_move_ss _mm_and_ps _mm_cvtsi32_ss _mm_extract_pi16 _mm_andnot_ps _mm_cvt_si2ss _m_pextrw _mm_or_ps _mm_cvtsi64x_ss* _mm_insert_pi16 _mm_xor_ps _mm_cvtpi32_ps _m_pinsrw _mm_cmpeq_ss _mm_cvt_pi2ps _mm_max_pi16 _mm_cmplt_ss _mm_movelh_ps _m_pmaxswChapter 20. C/C++ MMX/SSE Inline Intrinsics 307 _mm_cmple_ss _mm_setzero_ps _mm_max_pu8 _mm_cmpgt_ss _mm_cvtpi16_ps _m_pmaxub _mm_cmpge_ss _mm_cvtpu16_ps _mm_min_pi16 _mm_cmpneq_ss _mm_cvtpi8_ps _m_pminsw _mm_cmpnlt_ss _mm_cvtpu8_ps _mm_min_pu8 _mm_cmpnle_ss _mm_cvtpi32x2_ps _m_pminub _mm_cmpngt_ss _mm_movehl_ps _mm_movemask_pi8 _mm_cmpnge_ss _mm_cvtps_pi16 _m_pmovmskb _mm_cmpord_ss _mm_cvtps_pi8 _mm_mulhi_pu16 _mm_cmpunord_ss _mm_shuffle_ps _m_pmulhuw _mm_cmpeq_ps _mm_unpackhi_ps _mm_shuffle_pi16 _mm_cmplt_ps _mm_unpacklo_ps _m_pshufw _mm_cmple_ps _mm_loadh_pi _mm_maskmove_si64 _mm_cmpgt_ps _mm_storeh_pi _m_maskmovq _mm_cmpge_ps _mm_loadl_pi _mm_avg_pu8 _mm_cmpneq_ps _mm_storel_pi _m_pavgb _mm_cmpnlt_ps _mm_movemask_ps _mm_avg_pu16 _mm_cmpnle_ps _mm_getcsr _m_pavgw _mm_cmpngt_ps _MM_GET_EXCEPTION_STATE _mm_sad_pu8 _mm_cmpnge_ps _MM_GET_EXCEPTION_MASK _m_psadbw _mm_cmpord_ps _MM_GET_ROUNDING_MODE _mm_prefetch _mm_cmpunord_ps _MM_GET_FLUSH_ZERO_MODE _mm_stream_pi _mm_comieq_ss _mm_setcsr _mm_stream_ps _mm_comilt_ss _MM_SET_EXCEPTION_STATE _mm_sfence _mm_comile_ss _MM_SET_EXCEPTION_MASK _mm_pause _mm_comigt_ss _MM_SET_ROUNDING_MODE _MM_TRANSPOSE4_PS _MM_SET_FLUSH_ZERO_MODE Table 20.3. SSE2 Intrinsics (emmintrin.h) _mm_load_sd _mm_cmpge_sd _mm_cvtps_pd _mm_srl_epi32 _mm_load1_pd _mm_cmpneq_sd _mm_cvtsd_si32 _mm_srl_epi64 _mm_load_pd1 _mm_cmpnlt_sd _mm_cvtsd_si64x* _mm_slli_epi16 _mm_load_pd _mm_cmpnle_sd _mm_cvttsd_si32 _mm_slli_epi32 _mm_loadu_pd _mm_cmpngt_sd _mm_cvttsd_si64x* _mm_slli_epi64 _mm_loadr_pd _mm_cmpnge_sd _mm_cvtsd_ss _mm_srai_epi16 _mm_set_sd _mm_cmpord_sd _mm_cvtsi32_sd _mm_srai_epi32PGI® User’s Guide 308 _mm_set1_pd _mm_cmpunord_sd _mm_cvtsi64x_sd* _mm_srli_epi16 _mm_set_pd1 _mm_comieq_sd _mm_cvtss_sd _mm_srli_epi32 _mm_set_pd _mm_comilt_sd _mm_unpackhi_pd _mm_srli_epi64 _mm_setr_pd _mm_comile_sd _mm_unpacklo_pd _mm_and_si128 _mm_setzero_pd _mm_comigt_sd _mm_loadh_pd _mm_andnot_si128 _mm_store_sd _mm_comige_sd _mm_storeh_pd _mm_or_si128 _mm_store_pd _mm_comineq_sd _mm_loadl_pd _mm_xor_si128 _mm_store1_pd _mm_ucomieq_sd _mm_storel_pd _mm_cmpeq_epi8 _mm_store_pd1 _mm_ucomilt_sd _mm_movemask_pd _mm_cmpeq_epi16 _mm_storeu_pd _mm_ucomile_sd _mm_packs_epi16 _mm_cmpeq_epi32 _mm_storer_pd _mm_ucomigt_sd _mm_packs_epi32 _mm_cmplt_epi8 _mm_move_sd _mm_ucomige_sd _mm_packus_epi16 _mm_cmplt_epi16 _mm_add_pd _mm_ucomineq_sd _mm_unpackhi_epi8 _mm_cmplt_epi32 _mm_add_sd _mm_load_si128 _mm_unpackhi_epi16 _mm_cmpgt_epi8 _mm_sub_pd _mm_loadu_si128 _mm_unpackhi_epi32 _mm_cmpgt_epi16 _mm_sub_sd _mm_loadl_epi64 _mm_unpackhi_epi64 _mm_srl_epi16 _mm_mul_pd _mm_store_si128 _mm_unpacklo_epi8 _mm_cmpgt_epi32 _mm_mul_sd _mm_storeu_si128 _mm_unpacklo_epi16 _mm_max_epi16 _mm_div_pd _mm_storel_epi64 _mm_unpacklo_epi32 _mm_max_epu8 _mm_div_sd _mm_movepi64_pi64 _mm_unpacklo_epi64 _mm_min_epi16 _mm_sqrt_pd _mm_move_epi64 _mm_add_epi8 _mm_min_epu8 _mm_sqrt_sd _mm_setzero_si128 _mm_add_epi16 _mm_movemask_epi8 _mm_min_pd _mm_set_epi64 _mm_add_epi32 _mm_mulhi_epu16 _mm_min_sd _mm_set_epi32 _mm_add_epi64 _mm_maskmoveu_si128 _mm_max_pd _mm_set_epi64x* _mm_adds_epi8 _mm_avg_epu8 _mm_max_sd _mm_set_epi16 _mm_adds_epi16 _mm_avg_epu16 _mm_and_pd _mm_set_epi8 _mm_adds_epu8 _mm_sad_epu8 _mm_andnot_pd _mm_set1_epi64 _mm_adds_epu16 _mm_stream_si32 _mm_or_pd _mm_set1_epi32 _mm_sub_epi8 _mm_stream_si128 _mm_xor_pd _mm_set1_epi64x* _mm_sub_epi16 _mm_stream_pd _mm_cmpeq_pd _mm_set1_epi16 _mm_sub_epi32 _mm_movpi64_epi64 _mm_cmplt_pd _mm_set1_epi8 _mm_sub_epi64 _mm_lfence _mm_cmple_pd _mm_setr_epi64 _mm_subs_epi8 _mm_mfence _mm_cmpgt_pd _mm_setr_epi32 _mm_subs_epi16 _mm_cvtsi32_si128 _mm_cmpge_pd _mm_setr_epi16 _mm_subs_epu8 _mm_cvtsi64x_si128* _mm_cmpneq_pd _mm_setr_epi8 _mm_subs_epu16 _mm_cvtsi128_si32Chapter 20. C/C++ MMX/SSE Inline Intrinsics 309 _mm_cmpnlt_pd _mm_cvtepi32_pd _mm_madd_epi16 _mm_cvtsi128_si64x* _mm_cmpnle_pd _mm_cvtepi32_ps _mm_mulhi_epi16 _mm_srli_si128 _mm_cmpngt_pd _mm_cvtpd_epi32 _mm_mullo_epi16 _mm_slli_si128 _mm_cmpnge_pd _mm_cvtpd_pi32 _mm_mul_su32 _mm_shuffle_pd _mm_cmpord_pd _mm_cvtpd_ps _mm_mul_epu32 _mm_shufflehi_epi16 _mm_cmpunord_pd _mm_cvttpd_epi32 _mm_sll_epi16 _mm_shufflelo_epi16 _mm_cmpeq_sd _mm_cvttpd_pi32 _mm_sll_epi32 _mm_shuffle_epi32 _mm_cmplt_sd _mm_cvtpi32_pd _mm_sll_epi64 _mm_extract_epi16 _mm_cmple_sd _mm_cvtps_epi32 _mm_sra_epi16 _mm_insert_epi16 _mm_cmpgt_sd _mm_cvttps_epi32 _mm_sra_epi32 Table 20.4. SSE3 Intrinsics (pmmintrin.h) _mm_addsub_ps _mm_moveldup_ps _mm_loaddup_pd _mm_mwait _mm_hadd_ps _mm_addsub_pd _mm_movedup_pd _mm_hsub_ps _mm_hadd_pd _mm_lddqu_si128 _mm_movehdup_ps _mm_hsub_pd _mm_monitor Table 20.5. SSSE3 Intrinsics (tmmintrin.h) _mm_hadd_epi16 _mm_hsubs_pi16 _mm_sign_pi16 _mm_hadd_epi32 _mm_maddubs_epi16 _mm_sign_pi32 _mm_hadds_epi16 _mm_maddubs_pi16 _mm_alignr_epi8 _mm_hadd_pi16 _mm_mulhrs_epi16 _mm_alignr_pi8 _mm_hadd_pi32 _mm_mulhrs_pi16 _mm_abs_epi8 _mm_hadds_pi16 _mm_shuffle_epi8 _mm_abs_epi16 _mm_hsub_epi16 _mm_shuffle_pi8 _mm_abs_epi32 _mm_hsub_epi32 _mm_sign_epi8 _mm_abs_pi8 _mm_hsubs_epi16 _mm_sign_epi16 _mm_abs_pi16 _mm_hsub_pi16 _mm_sign_epi32 _mm_abs_pi32 _mm_hsub_pi32 _mm_sign_pi8 Table 20.6. SSE4a Intrinsics (ammintrin.h) _mm_stream_sd _mm_extract_si64 _mm_insert_si64 _mm_stream_ss _mm_extracti_si64 _mm_inserti_si64 ABM Intrinsics PGI supports a set of ABM Intrinsics which allow the use of the ABM instructions directly from C/C++ code, without writing the assembly instructions. The following table lists the ABM intrinsics that PGI supports.PGI® User’s Guide 310 Table 20.7. SSE4a Intrinsics (intrin.h) __lzcnt16 __lzcnt64 __popcnt __rdtscp __lzcnt __popcnt16 __popcnt64311 Chapter 21. Fortran Module/Library Interfaces PGI Visual Fortran provides access to a number of libraries that export C interfaces by using Fortran modules. PGI uses this mechanism to support the Win32 API and Unix/Linux portability libraries. This chapter describes the Fortran module library interfaces that PGIsupports, describing each property available. Data Types Because the Win32 API and Portability interfaces resolve to C language libraries, it is important to understand how the data types compare within the two languages. Here is a table summarizing how C types correspond with Fortran types for some of the more common data types: Table 21.1. Fortran Data Type Mappings C Win32 Data Type Fortran Data Type BOOL LOGICAL(4) BYTE BYTE CHAR CHARACTER SHORT, WORD INTEGER(2) DWORD, INT, LONG INTEGER(4) LONG LONG INTEGER(8) FLOAT REAL(4) DOUBLE REAL(8) x86 Pointers INTEGER(4) x64 Pointers INTEGER(8) For more information on data types, refer to “Fortran Data Types,” on page 151.PGI® User’s Guide 312 Using DFLIB and DFPORT PGI also includes Fortran module interfaces to libraries supporting some standard C library and Unix/Linux system call functionality. These functions are provided by the DFLIB and DFPORT modules. To utilize these modules, add the appropriate USE statement: USE DFLIB USE DFPORT DFLIB The following table lists the functions that DFLIB includes: commitqq gettim setenvqq getdrivedirqq renamefileqq signalqq getenvqq runqq systemqq DFPORT The following table lists the functions that DFPORT includes: abort access alarm besj0 besj1 besjn besy0 besy1 besyn chdir chmod ctime date dbesj0 dbesj1 dbesjn dbesy0 dbesy1 dbesyn dffrac dflmax dflmin drandm dsecnds dtime etime exit fdate ffrac fgetc flmax flmin flush fork fputc free fseek fseek64 fstat ftell ftell64 gerror getarg getc getcwd getenv getfd getgid getlog getpid getuid gmtime hostnm iargc idate ierrno inmax ioinit irand1 irand2 irandm isatty itime kill link lnblnk loc long lstat ltime malloc mclock outstr perror putc putenv qsort rand1 rand2 random rename rindex rtc secnds short signal sleep srand1 srand2 stat stime symlnk system time timef times ttynam unlink wait Using the DFWIN module The DFWIN module includes all the modules needed to access the Win32 API. You can use modules supporting specific portions of the Win32 API separately. DFWIN is the only module you need to access the Fortran interfaces to the Win32 API. Simply add the USE DFWIN line to your Fortran code to use this module.Chapter 21. Fortran Module/Library Interfaces 313 To utilize any of the Win32 API interfaces, add a Fortran USE statement for the library or module that includes it. For example, to use user32.lib, add the following Fortran USE statement: USE DFWIN For a mapping of module names to the corresponding Win32 API library and header files, refer to The Microsoft Windows API documentation.. The function calls made through the module interfaces ultimately resolve to C Language interfaces, so some accommodation for inter-language calling conventions must be made in the Fortran application. These accommodations include: • On x64 platforms, pointers and pointer types such as HANDLE, HINSTANCE, WPARAM, and HWND must be treated as 8-byte quantities (INTEGER(8)). On x86 (32-bit) platforms, these are 4-byte quantities ((INTEGER(4)). • In general, C makes calls by value while Fortran makes calls by reference. • When doing Windows development one must sometimes provide callback functions for message processing, dialog processing, etc. These routines are called by the Windows system when events are processed. In order to provide the expected function signature for a callback function, the user may need to use the STDCALL attribute directive (!DEC$ATTRIBUTE::STDCALL) in the declaration. See the PVF examples for more detail on how to implement callbacks. Supported Libraries and Modules The following tables provide lists of the functions in each library or module that PGI supports in DFWIN. advapi32 The following table lists the functions that advapi32 includes: AccessCheckAndAuditAlarm AccessCheckByType AccessCheckByTypeAndAuditAlarm AccessCheckByTypeResultList AccessCheckByTypeResultListAndAuditAlarm AccessCheckByTypeResultListAndAuditAlarmByHandle AddAccessAllowedAce AddAccessAllowedAceEx AddAccessAllowedObjectAce AddAccessDeniedAce AddAccessDeniedAceEx AddAccessDeniedObjectAce AddAce AddAuditAccessAce AddAuditAccessAceEx AddAuditAccessObjectAce AdjustTokenGroups AdjustTokenPrivileges AllocateAndInitializeSid AllocateLocallyUniqueId AreAllAccessesGranted AreAnyAccessesGranted BackupEventLog CheckTokenMembership ClearEventLog CloseEncryptedFileRaw CloseEventLog ConvertToAutoInheritPrivateObjectSecurity CopySid CreatePrivateObjectSecurity CreatePrivateObjectSecurityEx CreatePrivateObjectSecurityWithMultipleInheritancePGI® User’s Guide 314 CreateProcessAsUser CreateProcessWithLogonW CreateProcessWithTokenW CreateRestrictedToken CreateWellKnownSid DecryptFile DeleteAce DeregisterEventSource DestroyPrivateObjectSecurity DuplicateToken DuplicateTokenEx EncryptFile EqualDomainSid EqualPrefixSid EqualSid FileEncryptionStatus FindFirstFreeAce FreeSid GetAce GetAclInformation GetCurrentHwProfile GetEventLogInformation GetFileSecurity GetKernelObjectSecurity GetLengthSid GetNumberOfEventLogRecords GetOldestEventLogRecord GetPrivateObjectSecurity GetSecurityDescriptorControl GetSecurityDescriptorDacl GetSecurityDescriptorGroup GetSecurityDescriptorLength GetSecurityDescriptorOwner GetSecurityDescriptorRMControl GetSecurityDescriptorSacl GetSidIdentifierAuthority GetSidLengthRequired GetSidSubAuthority GetSidSubAuthorityCount GetTokenInformation GetUserName GetWindowsAccountDomainSid ImpersonateAnonymousToken ImpersonateLoggedOnUser ImpersonateNamedPipeClient ImpersonateSelf InitializeAcl InitializeSecurityDescriptor InitializeSid IsTextUnicode IsTokenRestricted IsTokenUntrusted IsValidAcl IsValidSecurityDescriptor IsValidSid IsWellKnownSid LogonUser LogonUserEx LookupAccountName LookupAccountSid LookupPrivilegeDisplayName LookupPrivilegeName LookupPrivilegeValue MakeAbsoluteSD MakeAbsoluteSD2 MakeSelfRelativeSD MapGenericMask NotifyChangeEventLog ObjectCloseAuditAlarm ObjectDeleteAuditAlarm ObjectOpenAuditAlarm ObjectPrivilegeAuditAlarmChapter 21. Fortran Module/Library Interfaces 315 OpenBackupEventLog OpenEncryptedFileRaw OpenEventLog OpenProcessToken OpenThreadToken PrivilegeCheck PrivilegedServiceAuditAlarm ReadEncryptedFileRaw ReadEventLog RegisterEventSource ReportEvent RevertToSelf SetAclInformation SetFileSecurity SetKernelObjectSecurity SetPrivateObjectSecurity SetPrivateObjectSecurityEx SetSecurityDescriptorControl SetSecurityDescriptorDacl SetSecurityDescriptorGroup SetSecurityDescriptorOwner SetSecurityDescriptorRMControl SetSecurityDescriptorSacl SetThreadToken SetTokenInformation WriteEncryptedFileRaw comdlg32 The following table lists the functions that comdlg32 includes: AfxReplaceText ChooseColor ChooseFont CommDlgExtendedError FindText GetFileTitle GetOpenFileName GetSaveFileName PageSetupDlg PrintDlg PrintDlgEx ReplaceText dfwbase These are the functions that dfwbase includes: chartoint LoByte MakeWord chartoreal LoWord MakeWparam CopyMemory LoWord64 PaletteIndex GetBlueValue MakeIntAtom PaletteRGB GetGreenValue MakeIntResource PrimaryLangID GetRedValue MakeLangID RGB HiByte MakeLCID RtlCopyMemory HiWord MakeLong SortIDFromLCID HiWord64 MakeLParam SubLangID inttochar MakeLResult dfwinty These are the functions that dfwinty includes:PGI® User’s Guide 316 dwNumberOfFunctionKeys rdFunction gdi32 These are the functions that gdi32 includes: AbortDoc AbortPath AddFontMemResourceEx AddFontResource AddFontResourceEx AlphaBlend AngleArc AnimatePalette Arc ArcTo BeginPath BitBlt CancelDC CheckColorsInGamut ChoosePixelFormat Chord CloseEnhMetaFile CloseFigure CloseMetaFile ColorCorrectPalette ColorMatchToTarget CombineRgn CombineTransform CopyEnhMetaFile CopyMetaFile CreateBitmap CreateBitmapIndirect CreateBrushIndirect CreateColorSpace CreateCompatibleBitmap CreateCompatibleDC CreateDC CreateDIBitmap CreateDIBPatternBrush CreateDIBPatternBrushPt CreateDIBSection CreateDiscardableBitmap CreateEllipticRgn CreateEllipticRgnIndirect CreateEnhMetaFile CreateFont CreateFontIndirect CreateFontIndirectEx CreateHalftonePalette CreateHatchBrush CreateIC CreateMetaFile CreatePalette CreatePatternBrush CreatePen CreatePenIndirect CreatePolygonRgn CreatePolyPolygonRgn CreateRectRgn CreateRectRgnIndirect CreateRoundRectRgn CreateScalableFontResource CreateSolidBrush DeleteColorSpace DeleteDC DeleteEnhMetaFile DeleteMetaFile DeleteObject DescribePixelFormat DeviceCapabilities DPtoLP DrawEscape Ellipse EndDoc EndPage EndPath EnumEnhMetaFile EnumFontFamilies EnumFontFamiliesEx EnumFonts EnumICMProfiles EnumMetaFile EnumObjects EqualRgn Escape ExcludeClipRect ExtCreatePen ExtCreateRegion ExtEscape ExtFloodFill ExtSelectClipRgn ExtTextOut FillPath FillRgn FixBrushOrgEx FlattenPath FloodFill FrameRgn GdiComment GdiFlush GdiGetBatchLimitChapter 21. Fortran Module/Library Interfaces 317 GdiSetBatchLimit GetArcDirection GetAspectRatioFilterEx GetBitmapBits GetBitmapDimensionEx GetBkColor GetBkMode GetBoundsRect GetBrushOrgEx GetCharABCWidthsA GetCharABCWidthsFloat GetCharABCWidthsI GetCharABCWidthsW GetCharacterPlacement GetCharWidth GetCharWidth32 GetCharWidthFloat GetCharWidthI GetClipBox GetClipRgn GetColorAdjustment GetColorSpace GetCurrentObject GetCurrentPositionEx GetDCBrushColor GetDCOrgEx GetDCPenColor GetDeviceCaps GetDeviceGammaRamp GetDIBColorTable GetDIBits GetEnhMetaFile GetEnhMetaFileBits GetEnhMetaFileDescriptionA GetEnhMetaFileDescriptionW GetEnhMetaFileHeader GetEnhMetaFilePaletteEntries GetEnhMetaFilePixelFormat GetFontData GetFontLanguageInfo GetFontUnicodeRanges GetGlyphIndices GetGlyphOutline GetGraphicsMode GetICMProfileA GetICMProfileW GetKerningPairs GetLayout GetLogColorSpace GetMapMode GetMetaFile GetMetaFileBitsEx GetMetaRgn GetMiterLimit GetNearestColor GetNearestPaletteIndex GetObject GetObjectType GetOutlineTextMetrics GetPaletteEntries GetPath GetPixel GetPixelFormat GetPolyFillMode GetRandomRgn GetRasterizerCaps GetRegionData GetRgnBox GetROP2 GetStockObject GetStretchBltMode GetSystemPaletteEntries GetSystemPaletteUse GetTextAlign GetTextCharacterExtra GetTextCharset GetTextCharsetInfo GetTextColor GetTextExtentExPoint GetTextExtentExPointI GetTextExtentPoint GetTextExtentPoint32 GetTextExtentPointI GetTextFace GetTextMetrics GetViewportExtEx GetViewportOrgEx GetWindowExtEx GetWindowOrgEx GetWinMetaFileBits GetWorldTransform GradientFill IntersectClipRect InvertRgn LineDD LineTo LPtoDP MaskBlt ModifyWorldTransform MoveToEx OffsetClipRgn OffsetRgn OffsetViewportOrgEx OffsetWindowOrgEx PaintRgn PatBlt PathToRegion PiePGI® User’s Guide 318 PlayEnhMetaFile PlayEnhMetaFileRecord PlayMetaFile PlayMetaFileRecord PlgBlt PolyBezier PolyBezierTo PolyDraw Polygon Polyline PolylineTo PolyPolygon PolyPolyline PolyTextOut PtInRegion PtVisible RealizePalette Rectangle RectInRegion RectVisible RemoveFontMemResourceEx RemoveFontResource RemoveFontResourceEx ResetDC ResizePalette RestoreDC RoundRect SaveDC ScaleViewportExtEx ScaleWindowExtEx SelectClipPath SelectClipRgn SelectObject SelectPalette SetAbortProc SetArcDirection SetBitmapBits SetBitmapDimensionEx SetBkColor SetBkMode SetBoundsRect SetBrushOrgEx SetColorAdjustment SetColorSpace SetDCBrushColor SetDCPenColor SetDeviceGammaRamp SetDIBColorTable SetDIBits SetDIBitsToDevice SetEnhMetaFileBits SetGraphicsMode SetICMMode SetICMProfile SetLayout SetMapMode SetMapperFlags SetMetaFileBitsEx SetMetaRgn SetMiterLimit SetPaletteEntries SetPixel SetPixelFormat SetPixelV SetPolyFillMode SetRectRgn SetROP2 SetStretchBltMode SetSystemPaletteUse SetTextAlign SetTextCharacterExtra SetTextColor SetTextJustification SetViewportExtEx SetViewportOrgEx SetWindowExtEx SetWindowOrgEx SetWinMetaFileBits SetWorldTransform StartDoc StartPage StretchBlt StretchDIBits StrokeAndFillPath StrokePath SwapBuffers TextOut TranslateCharsetInfo TransparentBlt UnrealizeObject UpdateColors UpdateICMRegKey wglCopyContext wglCreateContext wglCreateLayerContext wglDeleteContext wglDescribeLayerPlane wglGetCurrentContext wglGetCurrentDC wglGetLayerPaletteEntries wglGetProcAddress wglMakeCurrent wglRealizeLayerPalette wglSetLayerPaletteEntries wglShareLists wglSwapLayerBuffers wglSwapMultipleBuffers wglUseFontBitmapsChapter 21. Fortran Module/Library Interfaces 319 wglUseFontOutlines WidenPath kernel32 These are the functions that kernel32 includes: ActivateActCtx AddAtom AddConsoleAlias AddRefActCtx AddVectoredContinueHandler AddVectoredExceptionHandler AllocateUserPhysicalPages AllocConsole AreFileApisANSI AssignProcessToJobObject AttachConsole BackupRead BackupSeek BackupWrite Beep BeginUpdateResource BindIoCompletionCallback BuildCommDCB BuildCommDCBAndTimeouts CallNamedPipe CancelDeviceWakeupRequest CancelIo CancelTimerQueueTimer CancelWaitableTimer CheckNameLegalDOS8Dot3 CheckRemoteDebuggerPresent ClearCommBreak ClearCommError CloseHandle CommConfigDialog CompareFileTime ConnectNamedPipe ContinueDebugEvent ConvertFiberToThread ConvertThreadToFiber ConvertThreadToFiberEx CopyFile CopyFileEx CreateActCtx CreateConsoleScreenBuffer CreateDirectory CreateDirectoryEx CreateEvent CreateFiber CreateFiberEx CreateFile CreateFileMapping CreateHardLink CreateIoCompletionPort CreateJobObject CreateJobSet CreateMailslot CreateMemoryResourceNotification CreateMutex CreateNamedPipe CreatePipe CreateProcess CreateRemoteThread CreateSemaphore CreateTapePartition CreateThread CreateTimerQueue CreateTimerQueueTimer CreateWaitableTimerPGI® User’s Guide 320 DeactivateActCtx DebugActiveProcess DebugActiveProcessStop DebugBreak DebugBreakProcess DebugSetProcessKillOnExit DecodePointer DecodeSystemPointer DefineDosDevice DeleteAtom DeleteCriticalSection DeleteFiber DeleteFile DeleteTimerQueue DeleteTimerQueueEx DeleteTimerQueueTimer DeleteVolumeMountPoint DeviceIoControl DisableThreadLibraryCalls DisconnectNamedPipe DnsHostnameToComputerName DosDateTimeToFileTime DuplicateHandle EncodePointer EncodeSystemPointer EndUpdateResource EnterCriticalSection EnumResourceLanguages EnumResourceNames EnumResourceTypes EnumSystemFirmwareTables EraseTape EscapeCommFunction ExitProcess ExitThread ExpandEnvironmentStrings FatalAppExit FatalExit FileTimeToDosDateTime FileTimeToLocalFileTime FileTimeToSystemTime FillConsoleOutputAttribute FillConsoleOutputCharacter FindActCtxSectionGuid FindActCtxSectionString FindAtom FindClose FindCloseChangeNotification FindFirstChangeNotification FindFirstFile FindFirstFileEx FindFirstVolume FindFirstVolumeMountPoint FindNextChangeNotification FindNextFile FindNextVolume FindNextVolumeMountPoint FindResource FindResourceEx FindVolumeClose FindVolumeMountPointClose FlsAlloc FlsFree FlsGetValue FlsSetValue FlushConsoleInputBuffer FlushFileBuffers FlushInstructionCache FlushViewOfFile FormatMessage FreeConsole FreeEnvironmentStringsChapter 21. Fortran Module/Library Interfaces 321 FreeLibrary FreeLibraryAndExitThread FreeResource FreeUserPhysicalPages GenerateConsoleCtrlEvent GetAtomName GetBinaryType GetCommandLine GetCommConfig GetCommMask GetCommModemStatus GetCommProperties GetCommState GetCommTimeouts GetCompressedFileSize GetComputerName GetConsoleAlias GetConsoleAliases GetConsoleAliasesLength GetConsoleAliasExes GetConsoleAliasExesLength GetConsoleCP GetConsoleCursorInfo GetConsoleDisplayMode GetConsoleFontSize GetConsoleMode GetConsoleOutputCP GetConsoleProcessList GetConsoleScreenBufferInfo GetConsoleSelectionInfo GetConsoleTitle GetConsoleWindow GetCurrentActCtx GetCurrentConsoleFont GetCurrentDirectory GetCurrentProcess GetCurrentProcessId GetCurrentProcessorNumber GetCurrentThread GetCurrentThreadId GetDefaultCommConfig GetDevicePowerState GetDiskFreeSpace GetDiskFreeSpaceEx GetDllDirectory GetDriveType GetEnvironmentStrings GetEnvironmentVariable GetExitCodeProcess GetExitCodeThread GetFileAttributes GetFileAttributesEx GetFileInformationByHandle GetFileSize GetFileSizeEx GetFileTime GetFileType GetFirmwareEnvironmentVariable GetFullPathName GetHandleInformation GetLargePageMinimum GetLargestConsoleWindowSize GetLastError GetLocalTime GetLogicalDrives GetLogicalDriveStrings GetLogicalProcessorInformation GetLongPathName GetMailslotInfo GetModuleFileName GetModuleHandle GetModuleHandleExPGI® User’s Guide 322 GetNamedPipeHandleState GetNamedPipeInfo GetNativeSystemInfo GetNumaAvailableMemoryNode GetNumaHighestNodeNumber GetNumaNodeProcessorMask GetNumaProcessorNode GetNumberOfConsoleInputEvents GetNumberOfConsoleMouseButtons GetOverlappedResult GetPriorityClass GetPrivateProfileInt GetPrivateProfileSection GetPrivateProfileSectionNames GetPrivateProfileString GetPrivateProfileStruct GetProcAddress GetProcessAffinityMask GetProcessHandleCount GetProcessHeap GetProcessHeaps GetProcessId GetProcessIdOfThread GetProcessIoCounters GetProcessPriorityBoost GetProcessShutdownParameters GetProcessTimes GetProcessVersion GetProcessWorkingSetSize GetProcessWorkingSetSizeEx GetProfileInt GetProfileSection GetProfileString GetQueuedCompletionStatus GetShortPathName GetStartupInfo GetStdHandle GetSystemDirectory GetSystemFirmwareTable GetSystemInfo GetSystemRegistryQuota GetSystemTime GetSystemTimeAdjustment GetSystemTimeAsFileTime GetSystemWindowsDirectory GetSystemWow64Directory GetTapeParameters GetTapePosition GetTapeStatus GetTempFileName GetTempPath GetThreadContext GetThreadId GetThreadIOPendingFlag GetThreadPriority GetThreadPriorityBoost GetThreadSelectorEntry GetThreadTimes GetTickCount GetTimeZoneInformation GetVersion GetVersionEx GetVolumeInformation GetVolumeNameForVolumeMountPoint GetVolumePathName GetVolumePathNamesForVolumeName GetWindowsDirectory GetWriteWatch GlobalAddAtom GlobalAlloc GlobalCompact GlobalDeleteAtomChapter 21. Fortran Module/Library Interfaces 323 GlobalFindAtom GlobalFix GlobalFlags GlobalFree GlobalGetAtomName GlobalHandle GlobalLock GlobalMemoryStatus GlobalMemoryStatusEx GlobalReAlloc GlobalSize GlobalUnfix GlobalUnlock GlobalUnWire GlobalWire HeapAlloc HeapCompact HeapCreate HeapDestroy HeapFree HeapLock HeapQueryInformation HeapReAlloc HeapSetInformation HeapSize HeapUnlock HeapValidate HeapWalk InitAtomTable InitializeCriticalSection InitializeCriticalSectionAndSpinCount InitializeSListHead InterlockedCompareExchange InterlockedCompareExchange64 InterlockedDecrement InterlockedExchange InterlockedExchangeAdd InterlockedFlushSList InterlockedIncrement InterlockedPopEntrySList InterlockedPushEntrySList IsBadCodePtr IsBadHugeReadPtr IsBadHugeWritePtr IsBadReadPtr IsBadStringPtr IsBadWritePtr IsDebuggerPresent IsProcessInJob IsProcessorFeaturePresent IsSystemResumeAutomatic LeaveCriticalSection LoadLibrary LoadLibraryEx LoadModule LoadResource LocalAlloc LocalCompact LocalFileTimeToFileTime LocalFlags LocalFree LocalHandle LocalLock LocalReAlloc LocalShrink LocalSize LocalUnlock LockFile LockFileEx LockResource lstrcat lstrcmpPGI® User’s Guide 324 lstrcmpi lstrcpy lstrcpyn lstrlen MapUserPhysicalPages MapUserPhysicalPagesScatter MapViewOfFile MapViewOfFileEx MoveFile MoveFileEx MoveFileWithProgress MulDiv NeedCurrentDirectoryForExePath OpenEvent OpenFile OpenFileMapping OpenJobObject OpenMutex OpenProcess OpenSemaphore OpenThread OpenWaitableTimer OutputDebugString PeekConsoleInput PeekNamedPipe PostQueuedCompletionStatus PrepareTape ProcessIdToSessionId PulseEvent PurgeComm QueryActCtxW QueryDepthSList QueryDosDevice QueryInformationJobObject QueryMemoryResourceNotification QueryPerformanceCounter QueryPerformanceFrequency QueueUserAPC QueueUserWorkItem RaiseException ReadConsole ReadConsoleInput ReadConsoleOutput ReadConsoleOutputAttribute ReadConsoleOutputCharacter ReadDirectoryChangesW ReadFile ReadFileEx ReadFileScatter ReadProcessMemory RegisterWaitForSingleObject RegisterWaitForSingleObjectEx ReleaseActCtx ReleaseMutex ReleaseSemaphore RemoveDirectory RemoveVectoredContinueHandler RemoveVectoredExceptionHandler ReOpenFile ReplaceFile RequestDeviceWakeup RequestWakeupLatency ResetEvent ResetWriteWatch RestoreLastError ResumeThread ScrollConsoleScreenBuffer SearchPath SetCommBreak SetCommConfig SetCommMask SetCommStateChapter 21. Fortran Module/Library Interfaces 325 SetCommTimeouts SetComputerName SetComputerNameEx SetConsoleActiveScreenBuffer SetConsoleCP SetConsoleCtrlHandler SetConsoleCursorInfo SetConsoleCursorPosition SetConsoleMode SetConsoleOutputCP SetConsoleScreenBufferSize SetConsoleTextAttribute SetConsoleTitle SetConsoleWindowInfo SetCriticalSectionSpinCount SetCurrentDirectory SetDefaultCommConfig SetDllDirectory SetEndOfFile SetEnvironmentStrings SetEnvironmentVariable SetErrorMode SetEvent SetFileApisToANSI SetFileApisToOEM SetFileAttributes SetFilePointer SetFilePointerEx SetFileShortName SetFileTime SetFileValidData SetFirmwareEnvironmentVariable SetHandleCount SetHandleInformation SetInformationJobObject SetLastError SetLocalTime SetMailslotInfo SetMessageWaitingIndicator SetNamedPipeHandleState SetPriorityClass SetProcessAffinityMask SetProcessPriorityBoost SetProcessShutdownParameters SetProcessWorkingSetSize SetProcessWorkingSetSizeEx SetStdHandle SetSystemTime SetSystemTimeAdjustment SetTapeParameters SetTapePosition SetThreadAffinityMask SetThreadContext SetThreadExecutionState SetThreadIdealProcessor SetThreadPriority SetThreadPriorityBoost SetThreadStackGuarantee SetTimerQueueTimer SetTimeZoneInformation SetUnhandledExceptionFilter SetupComm SetVolumeLabel SetVolumeMountPoint SetWaitableTimer SignalObjectAndWait SizeofResource Sleep SleepEx SuspendThread SwitchToFiber SwitchToThreadPGI® User’s Guide 326 SystemTimeToFileTime SystemTimeToTzSpecificLocalTime TerminateJobObject TerminateProcess TerminateThread TlsAlloc TlsFree TlsGetValue TlsSetValue TransactNamedPipe TransmitCommChar TryEnterCriticalSection TzSpecificLocalTimeToSystemTime UnhandledExceptionFilter UnlockFile UnlockFileEx UnmapViewOfFile UnregisterWait UnregisterWaitEx UpdateResource VerifyVersionInfo VirtualAlloc VirtualAllocEx VirtualFree VirtualFreeEx VirtualLock VirtualProtect VirtualProtectEx VirtualQuery VirtualQueryEx VirtualUnlock WaitCommEvent WaitForDebugEvent WaitForMultipleObjects WaitForMultipleObjectsEx WaitForSingleObject WaitForSingleObjectEx WaitNamedPipe WinExec Wow64DisableWow64FsRedirection Wow64EnableWow64FsRedirection Wow64RevertWow64FsRedirection WriteConsole WriteConsoleInput WriteConsoleOutput WriteConsoleOutputAttribute WriteConsoleOutputCharacter WriteFile WriteFileEx WriteFileGather WritePrivateProfileSection WritePrivateProfileString WritePrivateProfileStruct WriteProcessMemory WriteProfileSection WriteProfileString WriteTapemark WTSGetActiveConsoleSessionId ZombifyActCtx _hread _hwrite _lclose _lcreat _llseek _lopen _lread _lwriteChapter 21. Fortran Module/Library Interfaces 327 shell32 These are the functions that shell32 includes: DoEnvironmentSubst ShellExecuteEx DragAcceptFiles Shell_NotifyIcon DragFinish SHEmptyRecycleBin DragQueryFile SHFileOperation DragQueryPoint SHFreeNameMappings DuplicateIcon SHGetDiskFreeSpaceEx ExtractAssociatedIcon SHGetFileInfo ExtractIcon SHGetNewLinkInfo ExtractIconEx SHInvokePrinterCommand FindExecutable SHIsFileAvailableOffline IsLFNDrive SHLoadNonloadedIconOverlayIdentifiers SHAppBarMessage SHQueryRecycleBin SHCreateProcessAsUserW SHSetLocalizedName ShellAbout WinExecError ShellExecute user32 These are the functions that user32 includes: ActivateKeyboardLayout AdjustWindowRect AdjustWindowRectEx AllowSetForegroundWindow AnimateWindow AnyPopup AppendMenu ArrangeIconicWindows AttachThreadInput BeginDeferWindowPos BeginPaint BringWindowToTop BroadcastSystemMessage BroadcastSystemMessageEx CallMsgFilter CallNextHookEx CallWindowProc CascadeWindows ChangeClipboardChain ChangeDisplaySettings ChangeDisplaySettingsEx ChangeMenu CharLower CharLowerBuff CharNext CharNextEx CharPrev CharPrevEx CharToOem CharToOemBuff CharUpper CharUpperBuff CheckDlgButton CheckMenuItem CheckMenuRadioItem CheckRadioButton ChildWindowFromPoint ChildWindowFromPointEx ClientToScreen ClipCursor CloseClipboard CloseDesktop CloseWindow CloseWindowStation CopyAcceleratorTablePGI® User’s Guide 328 CopyCursor CopyIcon CopyImage CopyRect CountClipboardFormats CreateAcceleratorTable CreateCaret CreateCursor CreateDesktop CreateDialogIndirectParam CreateDialogParam CreateIcon CreateIconFromResource CreateIconFromResourceEx CreateIconIndirect CreateMDIWindow CreateMenu CreatePopupMenu CreateWindow CreateWindowEx CreateWindowStation DeferWindowPos DefFrameProc DefMDIChildProc DefRawInputProc DefWindowProc DeleteMenu DeregisterShellHookWindow DestroyAcceleratorTable DestroyCaret DestroyCursor DestroyIcon DestroyMenu DestroyWindow DialogBoxIndirectParam DialogBoxParam1 DialogBoxParam2 DisableProcessWindowsGhosting DispatchMessage DlgDirList DlgDirListComboBox DlgDirSelectComboBoxEx DlgDirSelectEx DragDetect DragObject DrawAnimatedRects DrawCaption DrawEdge DrawFocusRect DrawFrameControl DrawIcon DrawIconIndirect DrawMenuBar DrawState DrawText DrawTextEx EmptyClipboard EnableMenuItem EnableScrollBar EnableWindow EndDeferWindowPos EndDialog EndMenu EndPaint EndTask EnumChildWindows EnumClipboardFormats EnumDesktops EnumDesktopWindows EnumDisplayDevices EnumDisplayMonitors EnumDisplaySettings EnumDisplaySettingsEx EnumProps EnumPropsEx EnumThreadWindows EnumWindows EnumWindowStations EqualRect ExcludeUpdateRgn ExitWindowsEx FillRect FindWindow FindWindowEx FlashWindow FlashWindowEx FrameRect GetActiveWindow GetAltTabInfo GetAncestor GetAsyncKeyState GetCapture GetCaretBlinkTime GetCaretPos GetClassInfo GetClassInfoEx GetClassLong GetClassLongPtr GetClassName GetClassWord GetClientRect GetClipboardData GetClipboardFormatName GetClipboardOwner GetClipboardSequenceNumber GetClipboardViewer GetClipCursor GetComboBoxInfoChapter 21. Fortran Module/Library Interfaces 329 GetCursor GetCursorInfo GetCursorPos GetDC GetDCEx GetDesktopWindow GetDialogBaseUnits GetDlgCtrlID GetDlgItem GetDlgItemInt GetDlgItemText GetDoubleClickTime GetFocus GetForegroundWindow GetGuiResources GetGUIThreadInfo GetIconInfo GetInputState GetKBCodePage GetKeyboardLayout GetKeyboardLayoutList GetKeyboardLayoutName GetKeyboardState GetKeyboardType GetKeyNameText GetKeyState GetLastActivePopup GetLastInputInfo GetLayeredWindowAttributes GetListBoxInfo GetMenu GetMenuBarInfo GetMenuCheckMarkDimensions GetMenuContextHelpId GetMenuDefaultItem GetMenuInfo GetMenuItemCount GetMenuItemID GetMenuItemInfo GetMenuItemRect GetMenuState GetMenuString GetMessage GetMessageExtraInfo GetMessagePos GetMessageTime GetMonitorInfo GetMouseMovePointsEx GetNextDlgGroupItem GetNextDlgTabItem GetOpenClipboardWindow GetParent GetPriorityClipboardFormat GetProcessDefaultLayout GetProcessWindowStation GetProp GetQueueStatus GetRawInputBuffer GetRawInputData GetRawInputDeviceInfo GetRawInputDeviceList GetRegisteredRawInputDevices GetScrollBarInfo GetScrollInfo GetScrollPos GetScrollRange GetShellWindow GetSubMenu GetSysColor GetSysColorBrush GetSystemMenu GetSystemMetrics GetTabbedTextExtent GetThreadDesktop GetTitleBarInfo GetTopWindow GetUpdateRect GetUpdateRgn GetUserObjectInformation GetUserObjectSecurity GetWindow GetWindowContextHelpId GetWindowDC GetWindowInfo GetWindowLong GetWindowLongPtr GetWindowModuleFileName GetWindowPlacement GetWindowRect GetWindowRgn GetWindowRgnBox GetWindowText GetWindowTextLength GetWindowThreadProcessId GetWindowWord GrayString HideCaret HiliteMenuItem InflateRect InSendMessage InSendMessageEx InsertMenu InsertMenuItem InternalGetWindowText IntersectRect InvalidateRect InvalidateRgn InvertRectPGI® User’s Guide 330 IsCharAlpha IsCharAlphaNumeric IsCharLower IsCharUpper IsChild IsClipboardFormatAvailable IsDialogMessage IsDlgButtonChecked IsGUIThread IsHungAppWindow IsIconic IsMenu IsRectEmpty IsWindow IsWindowEnabled IsWindowUnicode IsWindowVisible IsWinEventHookInstalled IsWow64Message IsZoomed keybd_event KillTimer LoadAccelerators LoadBitmap LoadCursor1 LoadCursor2 LoadCursorFromFile LoadIcon1 LoadIcon2 LoadImage LoadKeyboardLayout LoadMenu1 LoadMenu2 LoadMenuIndirect LoadString LockSetForegroundWindow LockWindowUpdate LockWorkStation LookupIconIdFromDirectory LookupIconIdFromDirectoryEx LRESULT MapDialogRect MapVirtualKey MapVirtualKeyEx MapWindowPoints MenuItemFromPoint MessageBeep MessageBox MessageBoxEx MessageBoxIndirect ModifyMenu1 ModifyMenu2 MonitorFromPoint MonitorFromRect MonitorFromWindow mouse_event MoveWindow MsgWaitForMultipleObjects MsgWaitForMultipleObjectsEx NotifyWinEvent OemKeyScan OemToChar OemToCharBuff OffsetRect OpenClipboard OpenDesktop OpenIcon OpenInputDesktop OpenWindowStation PaintDesktop PeekMessage PostMessage PostQuitMessage PostThreadMessage PrintWindow PrivateExtractIcons PtInRect RealChildWindowFromPoint RealGetWindowClass RedrawWindow RegisterClass RegisterClassEx RegisterClipboardFormat RegisterDeviceNotification RegisterHotKey RegisterRawInputDevices RegisterShellHookWindow RegisterWindowMessage ReleaseCapture ReleaseDC RemoveMenu RemoveProp ReplyMessage ScreenToClient ScrollDC ScrollWindow ScrollWindowEx SendDlgItemMessage SendInput SendMessage SendMessageCallback SendMessageTimeout SendNotifyMessage SetActiveWindow SetCapture SetCaretBlinkTime SetCaretPos SetClassLongChapter 21. Fortran Module/Library Interfaces 331 SetClassLongPtr SetClassWord SetClipboardData SetClipboardViewer SetCursor SetCursorPos SetDebugErrorLevel SetDlgItemInt SetDlgItemText SetDoubleClickTime SetFocus SetForegroundWindow SetKeyboardState SetLastErrorEx SetLayeredWindowAttributes SetMenu SetMenuContextHelpId SetMenuDefaultItem SetMenuInfo SetMenuItemBitmaps SetMenuItemInfo SetMessageExtraInfo SetMessageQueue SetParent SetProcessDefaultLayout SetProcessWindowStation SetProp SetRect SetRectEmpty SetScrollInfo SetScrollPos SetScrollRange SetSysColors SetSystemCursor SetThreadDesktop SetTimer SetUserObjectInformation SetUserObjectSecurity SetWindowContextHelpId SetWindowLong SetWindowLongPtr SetWindowPlacement SetWindowPos SetWindowRgn SetWindowsHook SetWindowsHookEx SetWindowText SetWindowWord SetWinEventHook ShowCaret ShowCursor ShowOwnedPopups ShowScrollBar ShowWindow ShowWindowAsync SubtractRect SwapMouseButton SwitchDesktop SwitchToThisWindow SystemParametersInfo TabbedTextOut TileWindows ToAscii ToAsciiEx ToUnicode ToUnicodeEx TrackMouseEvent TrackPopupMenu TrackPopupMenuEx TranslateAccelerator TranslateMDISysAccel TranslateMessage UnhookWindowsHook UnhookWindowsHookEx UnhookWinEvent UnionRect UnloadKeyboardLayout UnregisterClass UnregisterDeviceNotification UnregisterHotKey UpdateLayeredWindow UpdateLayeredWindowIndirect UpdateWindow UserHandleGrantAccess ValidateRect ValidateRgn VkKeyScan VkKeyScanEx WaitForInputIdle WaitMessage WindowFromDC WindowFromPoint WinHelp wsprintf wvsprintf winver These are the functions that winver includes: GetFileVersionInfo VerInstallFilePGI® User’s Guide 332 GetFileVersionInfoSize VerLanguageName VerFindFile VerQueryValue wsock32 These are the functions that wsock32 includes: accept AcceptEx bind closesocket connect GetAcceptExSockaddrs getpeername gethostname getprotobyname getprotobynumber getservbyname getservbyport getsockname getsockopt htonl htons inet_addr inet_ntoa ioctlsocket listen ntohl ntohs recv select send sendto setsockopt shutdown socket TransmitFile WSAAsyncGetHostByName WSAAsyncGetProtoByName WSAAsyncGetProtoByNumber WSAAsyncGetServByName WSAAsyncGetServByPort WSAAsyncSelect WSACancelAsyncRequest WSACancelBlockingCall WSACleanup WSAGetLastError WSAIsBlocking WSARecvEx WSASetBlockingHook WSASetLastError WSAStartup333 Chapter 22. Messages This chapter describes the various messages that the compiler produces. These messages include the sign-on message and diagnostic messages for remarks, warnings, and errors. The compiler always displays any error messages, along with the erroneous source line, on the screen. If you specify the –Mlist option, the compiler places any error messages in the listing file. You can also use the –v option to display more information about the compiler, assembler, and linker invocations and about the host system. For more information on the –Mlist and –v options, refer to Chapter 2, “Using Command Line Options”. Diagnostic Messages Diagnostic messages provide syntactic and semantic information about your source text. Syntactic information includes information such as syntax errors. Semantic includes in-formation includes such as unreachable code. You can specify that the compiler displays error messages at a certain level with the -Minform option. The compiler messages refer to a severity level, a message number, and the line number where the error occurs. The compiler can also display internal error messages on standard errors. If your compilation produces any internal errors, contact The Portland Group’s technical reporting service by sending e-mail to trs@pgroup.com. If you use the listing file option –Mlist, the compiler places diagnostic messages after the source lines in the listing file, in the following format: PGFTN-etype-enum-message (filename: line) Where: etype is a character signifying the severity level enum is the error number message is the error messagePGI® User’s Guide 334 filename is the source filename line is the line number where the compiler detected an error. Phase Invocation Messages You can display compiler, assembler, and linker phase invocations by using the –v command line option. For further information about this option, see Chapter 2, “Using Command Line Options”. Fortran Compiler Error Messages This section presents the error messages generated by the PGF77 and PGF95 compilers. The compilers display error messages in the program listing and on standard output; and can also display internal error messages on standard error. Message Format Each message is numbered. Each message also lists the line and column number where the error occurs. A dollar sign ($) in a message represents information that is specific to each occurrence of the message. Message List Error message severities: I informative W warning S severe error F fatal error V variable V000 Internal compiler error. $ $ This message indicates an error in the compiler, rather than a user error – although it may be possible for a user error to cause an internal error. The severity may vary; if it is informative or warning, correct object code was probably generated, but it is not safe to rely on this. Regardless of the severity or cause, internal errors should be reported to trs@pgroup.com. F001 Source input file name not specified On the command line, source file name should be specified either before all the switches, or after them. F002 Unable to open source input file: $ Source file name misspelled, file not in current working directory, or file is read protected.Chapter 22. Messages 335 F003 Unable to open listing file Probably, user does not have write permission for the current working directory. F004 $ $ Generic message for file errors. F005 Unable to open temporary file Compiler uses directory "/usr/tmp" or "/tmp" in which to create temporary files. If neither of these directories is available on the node on which the compiler is being used, this error will occur. S006 Input file empty Source input file does not contain any Fortran statements other than comments or compiler directives. F007 Subprogram too large to compile at this optimization level $ Internal compiler data structure overflow, working storage exhausted, or some other non-recoverable problem related to the size of the subprogram. If this error occurs at opt 2, reducing the opt level to 1 may work around the problem. Moving the subprogram being compiled to its own source file may eliminate the problem. If this error occurs while compiling a subprogram of fewer than 2000 statements it should be reported to the compiler maintenance group as a possible compiler problem. F008 Error limit exceeded The compiler gives up because too many severe errors were issued; the error limit can be reset on the command line. F009 Unable to open assembly file Probably, user does not have write permission for the current working directory. F010 File write error occurred $ Probably, file system is full. S011 Unrecognized command line switch: $ Refer to PDS reference document for list of allowed compiler switches. S012 Value required for command line switch: $ Certain switches require an immediately following value, such as "-opt 2". V000 Internal compiler error. $ $ This message indicates an error in the compiler, rather than a user error – although it may be possible for a user error to cause an internal error. The severity may vary; if it is informative or warning, correct object code was probably generated, but it is not safe to rely on this. Regardless of the severity or cause, internal errors should be reported to trs@pgroup.com. F001 Source input file name not specified On the command line, source file name should be specified either before all the switches, or after them. F002 Unable to open source input file: $ Source file name misspelled, file not in current working directory, or file is read protected. F003 Unable to open listing file Probably, user does not have write permission for the current working directory.PGI® User’s Guide 336 F004 $ $ Generic message for file errors. F005 Unable to open temporary file Compiler uses directory "/usr/tmp" or "/tmp" in which to create temporary files. If neither of these directories is available on the node on which the compiler is being used, this error will occur. S006 Input file empty Source input file does not contain any Fortran statements other than comments or compiler directives. F007 Subprogram too large to compile at this optimization level $ Internal compiler data structure overflow, working storage exhausted, or some other non-recoverable problem related to the size of the subprogram. If this error occurs at opt 2, reducing the opt level to 1 may work around the problem. Moving the subprogram being compiled to its own source file may eliminate the problem. If this error occurs while compiling a subprogram of fewer than 2000 statements it should be reported to the compiler maintenance group as a possible compiler problem. F008 Error limit exceeded The compiler gives up because too many severe errors were issued; the error limit can be reset on the command line. F009 Unable to open assembly file Probably, user does not have write permission for the current working directory. F010 File write error occurred $ Probably, file system is full. S011 Unrecognized command line switch: $ Refer to PDS reference document for list of allowed compiler switches. S012 Value required for command line switch: $ Certain switches require an immediately following value, such as "-opt 2". S013 Unrecognized value specified for command line switch: $ S014 Ambiguous command line switch: $ Too short an abbreviation was used for one of the switches. W015 Hexadecimal or octal constant truncated to fit data type I016 Identifier, $, truncated to 31 chars An identifier may be at most 31 characters in length; characters after the 31st are ignored. S017 Unable to open include file: $ File is missing, read protected, or maximum include depth (10) exceeded. Remember that the file name should be enclosed in quotes. S018 Illegal label $ $ Used for label ’field’ errors or illegal values. E.g., in fixed source form, the label field (first five characters) of the indicated line contains a non-numeric character.Chapter 22. Messages 337 S019 Illegally placed continuation line A continuation line does not follow an initial line, or more than 99 continuation lines were specified. S020 Unrecognized compiler directive Refer to user’s manual for list of allowed compiler directives. S021 Label field of continuation line is not blank The first five characters of a continuation line must be blank. S022 Unexpected end of file - missing END statement S023 Syntax error - unbalanced $ Unbalanced parentheses or brackets. W024 CHARACTER or Hollerith constant truncated to fit data type A character or hollerith constant was converted to a data type that was not large enough to contain all of the characters in the constant. This type conversion occurs when the constant is used in an arithmetic expression or is assigned to a non-character variable. The character or hollerith constant is truncated on the right, that is, if 4 characters are needed then the first 4 are used and the remaining characters are discarded. W025 Illegal character ($) - ignored The current line contains a character, possibly non-printing, which is not a legal Fortran character (characters inside of character or Hollerith constants cannot cause this error). As a general rule, all non-printing characters are treated as white space characters (blanks and tabs); no error message is generated when this occurs. If for some reason, a non-printing character is not treated as a white space character, its hex representation is printed in the form dd where each d is a hex digit. S026 Unmatched quote S027 Illegal integer constant: $ Integer constant is too large for 32 bit word. S028 Illegal real or double precision constant: $ S029 Illegal $ constant: $ Illegal hexadecimal, octal, or binary constant. A hexadecimal constant consists of digits 0..9 and letters A..F or a..f; any other character in a hexadecimal constant is illegal. An octal constant consists of digits 0..7; any other digit or character in an octal constant is illegal. A binary constant consists of digits 0 or 7; any other digit or character in a binary constant is illegal. S030 Explicit shape must be specified for $ S031 Illegal data type length specifier for $ The data type length specifier (e.g. 4 in INTEGER*4) is not a constant expression that is a member of the set of allowed values for this particular data type. W032 Data type length specifier not allowed for $ The data type length specifier (e.g. 4 in INTEGER*4) is not allowed in the given syntax (e.g. DIMENSION A(10)*4). S033 Illegal use of constant $ A constant was used in an illegal context, such as on the left side of an assignment statement or as the target of a data initialization statement.PGI® User’s Guide 338 S034 Syntax error at or near $ I035 Predefined intrinsic $ loses intrinsic property An intrinsic name was used in a manner inconsistent with the language definition for that intrinsic. The compiler, based on the context, will treat the name as a variable or an external function. S036 Illegal implicit character range First character must alphabetically precede second. S037 Contradictory data type specified for $ The indicated identifier appears in more than one type specification statement and different data types are specified for it. S038 Symbol, $, has not been explicitly declared The indicated identifier must be declared in a type statement; this is required when the IMPLICIT NONE statement occurs in the subprogram. W039 Symbol, $, appears illegally in a SAVE statement $ An identifier appearing in a SAVE statement must be a local variable or array. S040 Illegal common variable $ Indicated identifier is a dummy variable, is already in a common block, or has previously been defined to be something other than a variable or array. W041 Illegal use of dummy argument $ This error can occur in several situations. It can occur if dummy arguments were specified on a PROGRAM statement. It can also occur if a dummy argument name occurs in a DATA, COMMON, SAVE, or EQUIVALENCE statement. A program statement must have an empty argument list. S042 $ is a duplicate dummy argument S043 Illegal attempt to redefine $ $ An attempt was made to define a symbol in a manner inconsistent with an earlier definition of the same symbol. This can happen for a number of reasons. The message attempts to indicate the situation that occurred. intrinsic - An attempt was made to redefine an intrinsic function. A symbol that represents an intrinsic function may be redefined if that symbol has not been previously verified to be an intrinsic function. For example, the intrinsic sin can be defined to be an integer array. If a symbol is verified to be an intrinsic function via the INTRINSIC statement or via an intrinsic function reference then it must be referred to as an intrinsic function for the remainder of the program unit. symbol - An attempt was made to redefine a symbol that was previously defined. An example of this is to declare a symbol to be a PARAMETER which was previously declared to be a subprogram argument. S044 Multiple declaration for symbol $ A redundant declaration of a symbol has occurred. For example, an attempt was made to declare a symbol as an ENTRY when that symbol was previously declared as an ENTRY. S045 Data type of entry point $ disagrees with function $ The current function has entry points with data types inconsistent with the data type of the current function. For example, the function returns type character and an entry point returns type complex.Chapter 22. Messages 339 S046 Data type length specifier in wrong position The CHARACTER data type specifier has a different position for the length specifier from the other data types. Suppose, we want to declare arrays ARRAYA and ARRAYB to have 8 elements each having an element length of 4 bytes. The difference is that ARRAYA is character and ARRAYB is integer. The declarations would be CHARACTER ARRAYA(8)*4 and INTEGER ARRAYB*4(8). S047 More than seven dimensions specified for array S048 Illegal use of ’*’ in declaration of array $ An asterisk may be used only as the upper bound of the last dimension. S049 Illegal use of ’*’ in non-subroutine subprogram The alternate return specifier ’*’ is legal only in the subroutine statement. Programs, functions, and block data are not allowed to have alternate return specifiers. S050 Assumed size array, $, is not a dummy argument S051 Unrecognized built-in % function The allowable built-in functions are %VAL, %REF, %LOC, and %FILL. One was encountered that did not match one of these allowed forms. S052 Illegal argument to %VAL or %LOC S053 %REF or %VAL not legal in this context The built-in functions %REF and %VAL can only be used as actual parameters in procedure calls. W054 Implicit character $ used in a previous implicit statement An implicit character has been given an implied data type more than once. The implied data type for the implicit character is changed anyway. W055 Multiple implicit none statements The IMPLICIT NONE statement can occur only once in a subprogram. W056 Implicit type declaration The -Mdclchk switch and an implicit declaration following an IMPLICIT NONE statement will produce a warning message for IMPLICIT statements. S057 Illegal equivalence of dummy variable, $ Dummy arguments may not appear in EQUIVALENCE statements. S058 Equivalenced variables $ and $ not in same common block A common block variable must not be equivalenced with a variable in another common block. S059 Conflicting equivalence between $ and $ The indicated equivalence implies a storage layout inconsistent with other equivalences. S060 Illegal equivalence of structure variable, $ STRUCTURE and UNION variables may not appear in EQUIVALENCE statements. S061 Equivalence of $ and $ extends common block backwards W062 Equivalence forces $ to be unaligned EQUIVALENCE statements have defined an address for the variable which has an alignment not optimal for variables of its data type. This can occur when INTEGER and CHARACTER data are equivalenced, for instance.PGI® User’s Guide 340 I063 Gap in common block $ before $ S064 Illegal use of $ in DATA statement implied DO loop The indicated variable is referenced where it is not an active implied DO index variable. S065 Repeat factor less than zero S066 Too few data constants in initialization statement S067 Too many data constants in initialization statement S068 Numeric initializer for CHARACTER $ out of range 0 through 255 A CHARACTER*1 variable or character array element can be initialized to an integer, octal, or hexadecimal constant if that constant is in the range 0 through 255. S069 Illegal implied DO expression The only operations allowed within an implied DO expression are integer +, -, *, and /. S070 Incorrect sequence of statements $ The statement order is incorrect. For instance, an IMPLICIT NONE statement must precede a specification statement which in turn must precede an executable statement. S071 Executable statements not allowed in block data S072 Assignment operation illegal to $ $ The destination of an assignment operation must be a variable, array reference, or vector reference. The assignment operation may be by way of an assignment statement, a data statement, or the index variable of an implied DO-loop. The compiler has determined that the identifier used as the destination, is not a storage location. The error message attempts to indicate the type of entity used. entry point - An assignment to an entry point that was not a function procedure was attempted. external procedure - An assignment to an external procedure or a Fortran intrinsic name was attempted. if the identifier is the name of an entry point that is not a function, an external procedure... S073 Intrinsic or predeclared, $, cannot be passed as an argument S074 Illegal number or type of arguments to $ $ The indicated symbol is an intrinsic or generic function, or a predeclared subroutine or function, requiring a certain number of arguments of a fixed data type. S075 Subscript, substring, or argument illegal in this context for $ This can happen if you try to doubly index an array such as ra(2)(3). This also applies to substring and function references. S076 Subscripts specified for non-array variable $ S077 Subscripts omitted from array $ S078 Wrong number of subscripts specified for $ S079 Keyword form of argument illegal in this context for $$Chapter 22. Messages 341 S080 Subscript for array $ is out of bounds S081 Illegal selector $ $ S082 Illegal substring expression for variable $ Substring expressions must be of type integer and if constant must be greater than zero. S083 Vector expression used where scalar expression required A vector expression was used in an illegal context. For example, iscalar = iarray, where a scalar is assigned the value of an array. Also, character and record references are not vectorizable. S084 Illegal use of symbol $ $ This message is used for many different errors. S085 Incorrect number of arguments to statement function $ S086 Dummy argument to statement function must be a variable S087 Non-constant expression where constant expression required S088 Recursive subroutine or function call of $ A function may not call itself. S089 Illegal use of symbol, $, with character length = * Symbols of type CHARACTER*(*) must be dummy variables and must not be used as statement function dummy parameters and statement function names. Also, a dummy variable of type CHARACTER*(*) cannot be used as a function. S090 Hollerith constant more than 4 characters In certain contexts, Hollerith constants may not be more than 4 characters long. S091 Constant expression of wrong data type S092 Illegal use of variable length character expression A character expression used as an actual argument, or in certain contexts within I/O statements, must not consist of a concatenation involving a passed length character variable. W093 Type conversion of expression performed An expression of some data type appears in a context which requires an expression of some other data type. The compiler generates code to convert the expression into the required type. S094 Variable $ is of wrong data type $ The indicated variable is used in a context which requires a variable of some other data type. S095 Expression has wrong data type An expression of some data type appears in a context which requires an expression of some other data type. S096 Illegal complex comparison The relations .LT., .GT., .GE., and .LE. are not allowed for complex values. S097 Statement label $ has been defined more than once More than one statement with the indicated statement number occurs in the subprogram. S098 Divide by zeroPGI® User’s Guide 342 S099 Illegal use of $ Aggregate record references may only appear in aggregate assignment statements, unformatted I/O statements, and as parameters to subprograms. They may not appear, for example, in expressions. Also, records with differing structure types may not be assigned to one another. S100 Expression cannot be promoted to a vector An expression was used that required a scalar quantity to be promoted to a vector illegally. For example, the assignment of a character constant string to a character array. Records, too, cannot be promoted to vectors. S101 Vector operation not allowed on $ Record and character typed entities may only be referenced as scalar quantities. S102 Arithmetic IF expression has wrong data type The parenthetical expression of an arithmetic if statement must be an integer, real, or double precision scalar expression. S103 Type conversion of subscript expression for $ The data type of a subscript expression must be integer. If it is not, it is converted. S104 Illegal control structure $ This message is issued for a number of errors involving IF-THEN statements and DO loops. If the line number specified is the last line (END statement) of the subprogram, the error is probably an unterminated DO loop or IF-THEN statement. S105 Unmatched ELSEIF, ELSE or ENDIF statement An ELSEIF, ELSE, or ENDIF statement cannot be matched with a preceding IF-THEN statement. S106 DO index variable must be a scalar variable The DO index variable cannot be an array name, a subscripted variable, a PARAMETER name, a function name, a structure name, etc. S107 Illegal assigned goto variable $ S108 Illegal variable, $, in NAMELIST group $ A NAMELIST group can only consist of arrays and scalars which are not dummy arguments and pointer-based variables. I109 Overflow in $ constant $, constant truncated at left A non-decimal (hexadecimal, octal, or binary) constant requiring more than 64-bits produces an overflow. The constant is truncated at left (e.g. ’1234567890abcdef1’x will be ’234567890abcdef1’x). I110 I111 Underflow of real or double precision constant I112 Overflow of real or double precision constant S113 Label $ is referenced but never defined S114 Cannot initialize $ W115 Assignment to DO variable $ in loop S116 Illegal use of pointer-based variable $ $Chapter 22. Messages 343 S117 Statement not allowed within a $ definition The statement may not appear in a STRUCTURE or derived type definition. S118 Statement not allowed in DO, IF, or WHERE block I119 Redundant specification for $ Data type of indicated symbol specified more than once. I120 Label $ is defined but never referenced I121 Operation requires logical or integer data types An operation in an expression was attempted on data having a data type incompatible with the operation. For example, a logical expression can consist of only logical elements of type integer or logical. Real data would be invalid. I122 Character string truncated Character string or Hollerith constant appearing in a DATA statement or PARAMETER statement has been truncated to fit the declared size of the corresponding identifier. W123 Hollerith length specification too big, reduced The length specifier field of a hollerith constant specified more characters than were present in the character field of the hollerith constant. The length specifier was reduced to agree with the number of characters present. S124 Relational expression mixes character with numeric data A relational expression is used to compare two arithmetic expressions or two character expressions. A character expression cannot be compared to an arithmetic expression. I125 Dummy procedure $ not declared EXTERNAL A dummy argument which is not declared in an EXTERNAL statement is used as the subprogram name in a CALL statement, or is called as a function, and is therefore assumed to be a dummy procedure. This message can result from a failure to declare a dummy array. I126 Name $ is not an intrinsic function I127 Optimization level for $ changed to opt 1 $ W128 Integer constant truncated to fit data type: $ An integer constant will be truncated when assigned to data types smaller than 32-bits, such as a BYTE. I129 Floating point overflow. Check constants and constant expressions I130 Floating point underflow. Check constants and constant expressions I131 Integer overflow. Check floating point expressions cast to integer I132 Floating pt. invalid oprnd. Check constants and constant expressions I133 Divide by 0.0. Check constants and constant expressionsPGI® User’s Guide 344 S134 Illegal attribute $ $ W135 Missing STRUCTURE name field A STRUCTURE name field is required on the outermost structure. W136 Field-namelist not allowed The field-namelist field of the STRUCTURE statement is disallowed on the outermost structure. W137 Field-namelist is required in nested structures W138 Multiply defined STRUCTURE member name $ A member name was used more than once within a structure. W139 Structure $ in RECORD statement not defined A RECORD statement contains a reference to a STRUCTURE that has not yet been defined. S140 Variable $ is not a RECORD S141 RECORD required on left of $ S142 $ is not a member of this RECORD S143 $ requires initializer W144 NEED ERROR MESSAGE $ $ This is used as a temporary message for compiler development. W145 %FILL only valid within STRUCTURE block The %FILL special name was used outside of a STRUCTURE multiline statement. It is only valid when used within a STRUCTURE multiline statement even though it is ignored. S146 Expression must be character type S147 Character expression not allowed in this context S148 Reference to $ required An aggregate reference to a record was expected during statement compilation but another data type was found instead. S149 Record where arithmetic value required An aggregate record reference was encountered when an arithmetic expression was expected. S150 Structure, Record, derived type, or member $ not allowed in this context A structure, record, or member reference was found in a context which is not supported. For example, the use of structures, records, or members within a data statement is disallowed. S151 Empty TYPE, STRUCTURE, UNION, or MAP TYPE - ENDTYPE, STRUCTURE - ENDSTRUCTURE, UNION - ENDUNION MAP - ENDMAP declaration contains no members. S152 All dimension specifiers must be ’:’ S153 Array objects are not conformable $ S154 DISTRIBUTE target, $, must be a processorChapter 22. Messages 345 S155 $ $ S156 Number of colons and triplets must be equal in ALIGN $ with $ S157 Illegal subscript use of ALIGN dummy $ - $ S158 Alternate return not specified in SUBROUTINE or ENTRY An alternate return can only be used if alternate return specifiers appeared in the SUBROUTINE or ENTRY statements. S159 Alternate return illegal in FUNCTION subprogram An alternate return cannot be used in a FUNCTION. S160 ENDSTRUCTURE, ENDUNION, or ENDMAP does not match top S161 Vector subscript must be rank-one array W162 Not equal test of loop control variable $ replaced with < or > test. S163 S164 Overlapping data initializations of $ An attempt was made to data initialize a variable or array element already initialized. S165 $ appeared more than once as a subprogram A subprogram name appeared more than once in the source file. The message is applicable only when an assembly file is the output of the compiler. S166 $ cannot be a common block and a subprogram A name appeared as a common block name and a subprogram name. The message is applicable only when an assembly file is the output of the compiler. I167 Inconsistent size of common block $ A common block occurs in more than one subprogram of a source file and its size is not identical. The maximum size is chosen. The message is applicable only when an assembly file is the output of the compiler. S168 Incompatible size of common block $ A common block occurs in more than one subprogram of a source file and is initialized in one subprogram. Its initialized size was found to be less than its size in the other subprogram(s). The message is applicable only when an assembly file is the output of the compiler. W169 Multiple data initializations of common block $ A common block is initialized in more than one subprogram of a source file. Only the first set of initializations apply. The message is applicable only when an assembly file is the output of the compiler. W170 F90 extension: $ $ Use of a nonstandard feature. A description of the feature is provided. W171 F90 extension: nonstandard statement type $ W172 F90 extension: numeric initialization of CHARACTER $ A CHARACTER*1 variable or array element was initialized with a numeric value.PGI® User’s Guide 346 W173 F90 extension: nonstandard use of data type length specifier W174 F90 extension: type declaration contains data initialization W175 F90 extension: IMPLICIT range contains nonalpha characters W176 F90 extension: nonstandard operator $ W177 F90 extension: nonstandard use of keyword argument $ W178 W179 F90 extension: use of structure field reference $ W180 F90 extension: nonstandard form of constant W181 F90 extension: & alternate return W182 F90 extension: mixed non-character and character elements in COMMON $ W183 F90 extension: mixed non-character and character EQUIVALENCE ($,$) W184 Mixed type elements (numeric and/or character types) in COMMON $ W185 Mixed numeric and/or character type EQUIVALENCE ($,$) S186 Argument missing for formal argument $ S187 Too many arguments specified for $ S188 Argument number $ to $: type mismatch S189 Argument number $ to $: association of scalar actual argument to array dummy argument S190 Argument number $ to $: non-conformable arrays S191 Argument number $ to $ cannot be an assumed-size array S192 Argument number $ to $ must be a label W193 Argument number $ to $ does not match INTENT (OUT) W194 INTENT(IN) argument cannot be defined - $ S195 Statement may not appear in an INTERFACE block $ S196 Deferred-shape specifiers are required for $ S197 Invalid qualifier or qualifier value (/$) in OPTIONS statement An illegal qualifier was found or a value was specified for a qualifier which does not expect a value. In either case, the qualifier for which the error occurred is indicated in the error message.Chapter 22. Messages 347 S198 $ $ in ALLOCATE/DEALLOCATE W199 Unaligned memory reference A memory reference occurred whose address does not meet its data alignment requirement. S200 Missing UNIT/FILE specifier S201 Illegal I/O specifier - $ S202 Repeated I/O specifier - $ S203 FORMAT statement has no label S204 $ $ Miscellaneous I/O error. S205 Illegal specification of scale factor The integer following + or - has been omitted, or P does not follow the integer value. S206 Repeat count is zero S207 Integer constant expected in edit descriptor S208 Period expected in edit descriptor S209 Illegal edit descriptor S210 Exponent width not used in the Ew.dEe or Gw.dEe edit descriptors S211 Internal I/O not allowed in this I/O statement S212 Illegal NAMELIST I/O Namelist I/O cannot be performed with internal, unformatted, formatted, and list-directed I/O. Also, I/O lists must not be present. S213 $ is not a NAMELIST group name S214 Input item is not a variable reference S215 Assumed sized array name cannot be used as an I/O item or specifier An assumed sized array was used as an item to be read or written or as an I/O specifier (i.e., FMT = arrayname). In these contexts the size of the array must be known. S216 STRUCTURE/UNION cannot be used as an I/O item S217 ENCODE/DECODE buffer must be a variable, array, or array element S218 Statement labeled $ $ S219 S220 Redefining predefined macro $PGI® User’s Guide 348 S221 #elif after #else A preprocessor #elif directive was found after a #else directive; only #endif is allowed in this context. S222 #else after #else A preprocessor #else directive was found after a #else directive; only #endif is allowed in this context. S223 #if-directives too deeply nested Preprocessor #if directive nesting exceeded the maximum allowed (currently 10). S224 Actual parameters too long for $ The total length of the parameters in a macro call to the indicated macro exceeded the maximum allowed (currently 2048). W225 Argument mismatch for $ The number of arguments supplied in the call to the indicated macro did not agree with the number of parameters in the macro’s definition. F226 Can’t find include file $ The indicated include file could not be opened. S227 Definition too long for $ The length of the macro definition of the indicated macro exceeded the maximum allowed (currently 2048). S228 EOF in comment The end of a file was encountered while processing a comment. S229 EOF in macro call to $ The end of a file was encountered while processing a call to the indicated macro. S230 EOF in string The end of a file was encountered while processing a quoted string. S231 Formal parameters too long for $ The total length of the parameters in the definition of the indicated macro exceeded the maximum allowed (currently 2048). S232 Identifier too long The length of an identifier exceeded the maximum allowed (currently 2048). S233 W234 Illegal directive name The sequence of characters following a # sign was not an identifier. W235 Illegal macro name A macro name was not an identifier. S236 Illegal number $ The indicated number contained a syntax error. F237 Line too long The input source line length exceeded the maximum allowed (currently 2048).Chapter 22. Messages 349 W238 Missing #endif End of file was encountered before a required #endif directive was found. W239 Missing argument list for $ A call of the indicated macro had no argument list. S240 Number too long The length of a number exceeded the maximum allowed (currently 2048). W241 Redefinition of symbol $ The indicated macro name was redefined. I242 Redundant definition for symbol $ A definition for the indicated macro name was found that was the same as a previous definition. F243 String too long The length of a quoted string exceeded the maximum allowed (currently 2048). S244 Syntax error in #define, formal $ not identifier A formal parameter that was not an identifier was used in a macro definition. W245 Syntax error in #define, missing blank after name or arglist There was no space or tab between a macro name or argument list and the macro’s definition. S246 Syntax error in #if A syntax error was found while parsing the expression following a #if or #elif directive. S247 Syntax error in #include The #include directive was not correctly formed. W248 Syntax error in #line A #line directive was not correctly formed. W249 Syntax error in #module A #module directive was not correctly formed. W250 Syntax error in #undef A #undef directive was not correctly formed. W251 Token after #ifdef must be identifier The #ifdef directive was not followed by an identifier. W252 Token after #ifndef must be identifier The #ifndef directive was not followed by an identifier. S253 Too many actual parameters to $ The number of actual arguments to the indicated macro exceeded the maximum allowed (currently 31). S254 Too many formal parameters to $ The number of formal arguments to the indicated macro exceeded the maximum allowed (currently 31). F255 Too much pushback The preprocessor ran out of space while processing a macro expansion. The macro may be recursive.PGI® User’s Guide 350 W256 Undefined directive $ The identifier following a # was not a directive name. S257 EOF in #include directive End of file was encountered while processing a #include directive. S258 Unmatched #elif A #elif directive was encountered with no preceding #if or #elif directive. S259 Unmatched #else A #else directive was encountered with no preceding #if or #elif directive. S260 Unmatched #endif A #endif directive was encountered with no preceding #if, #ifdef, or #ifndef directive. S261 Include files nested too deeply The nesting depth of #include directives exceeded the maximum (currently 20). S262 Unterminated macro definition for $ A newline was encountered in the formal parameter list for the indicated macro. S263 Unterminated string or character constant A newline with no preceding backslash was found in a quoted string. I264 Possible nested comment The characters /* were found within a comment. S265 S266 S267 W268 Cannot inline subprogram; common block mismatch W269 Cannot inline subprogram; argument type mismatch This message may be Severe if have gone too far to undo inlining process. F270 Missing -exlib option W271 Can’t inline $ - wrong number of arguments I272 Argument of inlined function not used S273 Inline library not specified on command line (-inlib switch) F274 Unable to access file $/TOC S275 Unable to open file $ while extracting or inlining F276 Assignment to constant actual parameter in inlined subprogram I277 Inlining of function $ may result in recursion S278 Chapter 22. Messages 351 W279 Possible use of $ before definition in $ The optimizer has detected the possibility that a variable is used before it has been assigned a value. The names of the variable and the function in which the use occurred are listed. The line number, if specified, is the line number of the basic block containing the use of the variable. W280 Syntax error in directive $ messages 280-300 rsvd for directive handling W281 Directive ignored - $ $ S300 Too few data constants in initialization of derived type $ S301 $ must be TEMPLATE or PROCESSOR S302 Unmatched END$ statement S303 END statement for $ required in an interface block S304 EXIT/CYCLE statement must appear in a DO/DOWHILE loop$$ S305 $ cannot be named, $ S306 $ names more than one construct S307 $ must have the construct name $ S308 DO may not terminate at an EXIT, CYCLE, RETURN, STOP, GOTO, or arithmetic IF S309 Incorrect name, $, specified in END statement S310 $ $ Generic message for MODULE errors. W311 Non-replicated mapping for $ array, $, ignored W312 Array $ should be declared SEQUENCE W313 Subprogram $ called within INDEPENDENT loop not PURE E314 IPA: actual argument $ is a label, but dummy argument $ is not an asterisk The call passes a label to the subprogram; the corresponding dummy argument in the subprogram should be an asterisk to declare this as the alternate return. I315 IPA: routine $, $ constant dummy arguments This many dummy arguments are being replaced by constants due to interprocedural analysis. I316 IPA: routine $, $ INTENT(IN) dummy arguments This many dummy arguments are being marked as INTENT(IN) due to interprocedural analysis. I317 IPA: routine $, $ array alignments propagated This many array alignments were propagated by interprocedural analysis. I318 IPA: routine $, $ distribution formats propagated This many array distribution formats were propagated by interprocedural analysis.PGI® User’s Guide 352 I319 IPA: routine $, $ distribution targets propagated This many array distribution targets were propagated by interprocedural analysis. I320 IPA: routine $, $ common blocks optimized This many mapped common blocks were optimized by interprocedural analysis. I321 IPA: routine $, $ common blocks not optimized This many mapped common blocks were not optimized by interprocedural analysis, either because they were declared differently in different routines, or they did not appear in the main program. I322 IPA: analyzing main program $ Interprocedural analysis is building the call graph and propagating information with the named main program. I323 IPA: collecting information for $ Interprocedural analysis is saving information for the current subprogram for subsequent analysis and propagation. W324 IPA file $ appears to be out of date W325 IPA file $ is for wrong subprogram: $ W326 Unable to open file $ to propagate IPA information to $ I327 IPA: $ subprograms analyzed I328 IPA: $ dummy arguments replaced by constants I329 IPA: $ INTENT(IN) dummy arguments should be INTENT(INOUT) I330 IPA: $ dummy arguments changed to INTENT(IN) I331 IPA: $ inherited array alignments replaced I332 IPA: $ transcriptive distribution formats replaced I333 IPA: $ transcriptive distribution targets replaced I334 IPA: $ descriptive/prescriptive array alignments verified I335 IPA: $ descriptive/prescriptive distribution formats verified I336 IPA: $ descriptive/prescriptive distribution targets verified I337 IPA: $ common blocks optimized I338 IPA: $ common blocks not optimized S339 Bad IPA contents file: $ S340 Bad IPA file format: $ S341 Unable to create file $ while analyzing IPA information S342 Unable to open file $ while analyzing IPA information S343 Unable to open IPA contents file $Chapter 22. Messages 353 S344 Unable to create file $ while collecting IPA information F345 Internal error in $: table overflow Analysis failed due to a table overflowing its maximum size. W346 Subprogram $ appears twice The subprogram appears twice in the same source file; IPA will ignore the first appearance. F347 Missing -ipalib option Interprocedural analysis, enabled with the -ipacollect, -ipaanalyze, or -ipapropagate options, requires the - ipalib option to specify the library directory. W348 Common /$/ $ has different distribution target The array was declared in a common block with a different distribution target in another subprogram. W349 Common /$/ $ has different distribution format The array was declared in a common block with a different distribution format in another subprogram. W350 Common /$/ $ has different alignment The array was declared in a common block with a different alignment in another subprogram. W351 Wrong number of arguments passed to $ The subroutine or function statement for the given subprogram has a different number of dummy arguments than appear in the call. W352 Wrong number of arguments passed to $ when bound to $ The subroutine or function statement for the given subprogram has a different number of dummy arguments than appear in the call to the EXTERNAL name given. W353 Subprogram $ is missing A call to a subroutine or function with this name appears, but it could not be found or analyzed. I354 Subprogram $ is not called No calls to the given subroutine or function appear anywhere in the program. W355 Missing argument in call to $ A nonoptional argument is missing in a call to the given subprogram. I356 Array section analysis incomplete Interprocedural analysis for array section arguments is incomplete; some information may not be available for optimization. I357 Expression analysis incomplete Interprocedural analysis for expression arguments is incomplete; some information may not be available for optimization. W358 Dummy argument $ is EXTERNAL, but actual is not subprogram The call statement passes a scalar or array to a dummy argument that is declared EXTERNAL. W359 SUBROUTINE $ passed to FUNCTION dummy argument $ The call statement passes a subroutine name to a dummy argument that is used as a function.PGI® User’s Guide 354 W360 FUNCTION $ passed to FUNCTION dummy argument $ with different result type The call statement passes a function argument to a function dummy argument, but the dummy has a different result type. W361 FUNCTION $ passed to SUBROUTINE dummy argument $ The call statement passes a function name to a dummy argument that is used as a subroutine. W362 Argument $ has a different type than dummy argument $ The type of the actual argument is different than the type of the corresponding dummy argument. W363 Dummy argument $ is a POINTER but actual argument $ is not The dummy argument is a pointer, so the actual argument must be also. W364 Array or array expression passed to scalar dummy argument $ The actual argument is an array, but the dummy argument is a scalar variable. W365 Scalar or scalar expression passed to array dummy argument $ The actual argument is a scalar variable, but the dummy argument is an array. F366 Internal error: interprocedural analysis fails An internal error occurred during interprocedural analysis; please report this to the compiler maintenance group. If user errors were reported when collecting IPA information or during IPA analysis, correcting them may avoid this error. I367 Array $ bounds cannot be matched to formal argument Passing a nonsequential array to a sequential dummy argument may require copying the array to sequential storage. The most common cause is passing an ALLOCATABLE array or array expression to a dummy argument that is declared with explicit bounds. Declaring the dummy argument as assumed shape, with bounds (:,:,:), will remove this warning. W368 Array-valued expression passed to scalar dummy argument $ The actual argument is an array-valued expression, but the dummy argument is a scalar variable. W369 Dummy argument $ has different rank than actual argument The actual argument is an array or array-valued expression with a different rank than the dummy argument. W370 Dummy argument $ has different shape than actual argument The actual argument is an array or array-valued expression with a different shape than the dummy argument; this may require copying the actual argument into sequential storage. W371 Dummy argument $ is INTENT(IN) but may be modified The dummy argument was declared as INTENT(IN), but analysis has found that the argument may be modified; the INTENT(IN) declaration should be changed. W372 Cannot propagate alignment from $ to $ The most common cause is when passing an array with an inherited alignment to a dummy argument with noninherited alignment.Chapter 22. Messages 355 I373 Cannot propagate distribution format from $ to $ The most common cause is when passing an array with a transcriptive distribution format to a dummy argument with prescriptive or descriptive distribution format. I374 Cannot propagate distribution target from $ to $ The most common cause is when passing an array with a transcriptive distribution target to a dummy argument with prescriptive or descriptive distribution target. I375 Distribution format mismatch between $ and $ Usually this arises when the actual and dummy arguments are distributed in different dimensions. I376 Alignment stride mismatch between $ and $ This may arise when the actual argument has a different stride in its alignment to its template than does the dummy argument. I377 Alignment offset mismatch between $ and $ This may arise when the actual argument has a different offset in its alignment to its template than does the dummy argument. I378 Distribution target mismatch between $ and $ This may arise when the actual and dummy arguments have different distribution target sizes. I379 Alignment of $ is too complex The alignment specification of the array is too complex for interprocedural analysis to verify or propagate; the program will work correctly, but without the benefit of IPA. I380 Distribution format of $ is too complex The distribution format specification of the array is too complex for interprocedural analysis to verify or propagate; the program will work correctly, but without the benefit of IPA. I381 Distribution target of $ is too complex The distribution target specification of the array is too complex for interprocedural analysis to verify or propagate; the program will work correctly, but without the benefit of IPA. I382 IPA: $ subprograms analyzed Interprocedural analysis succeeded in finding and analyzing this many subprograms in the whole program. I383 IPA: $ dummy arguments replaced by constants Interprocedural analysis has found this many dummy arguments in the whole program that can be replaced by constants. I384 IPA: $ dummy arguments changed to INTENT(IN) Interprocedural analysis has found this many dummy arguments in the whole program that are not modified and can be declared as INTENT(IN). W385 IPA: $ INTENT(IN) dummy arguments should be INTENT(INOUT) Interprocedural analysis has found this many dummy arguments in the whole program that were declared as INTENT(IN) but should be INTENT(INOUT). I386 IPA: $ array alignments propagated Interprocedural analysis has found this many array dummy arguments that could have the inherited array alignment replaced by a descriptive alignment.PGI® User’s Guide 356 I387 IPA: $ array alignments verified Interprocedural analysis has verified that the prescriptive or descriptive alignments of this many array dummy arguments match the alignments of the actual argument. I388 IPA: $ array distribution formats propagated Interprocedural analysis has found this many array dummy arguments that could have the transcriptive distribution format replaced by a descriptive format. I389 IPA: $ array distribution formats verified Interprocedural analysis has verified that the prescriptive or descriptive distribution formats of this many array dummy arguments match the formats of the actual argument. I390 IPA: $ array distribution targets propagated Interprocedural analysis has found this many array dummy arguments that could have the transcriptive distribution target replaced by a descriptive target. I391 IPA: $ array distribution targets verified Interprocedural analysis has verified that the prescriptive or descriptive distribution targets of this many array dummy arguments match the targets of the actual argument. I392 IPA: $ common blocks optimized Interprocedural analysis has found this many common blocks that could be optimized. I393 IPA: $ common blocks not optimized Interprocedural analysis has found this many common blocks that could not be optimized, either because the common block was not declared in the main program, or because it was declared differently in different subprograms. I394 IPA: $ replaced by constant value The dummy argument was replaced by a constant as per interprocedural analysis. I395 IPA: $ changed to INTENT(IN) The dummy argument was changed to INTENT(IN) as per interprocedural analysis. I396 IPA: array alignment propagated to $ The template alignment for the dummy argument was changed as per interprocedural analysis. I397 IPA: distribution format propagated to $ The distribution format for the dummy argument was changed as per interprocedural analysis. I398 IPA: distribution target propagated to $ The distribution target for the dummy argument was changed as per interprocedural analysis. I399 IPA: common block $ not optimized The given common block was not optimized by interprocedural analysis either because it was not declared in the main program, or because it was declared differently in different subprograms. E400 IPA: dummy argument $ is an asterisk, but actual argument is not a label The subprogram expects an alternate return label for this argument.Chapter 22. Messages 357 E401 Actual argument $ is a subprogram, but Dummy argument $ is not declared EXTERNAL The call statement passes a function or subroutine name to a dummy argument that is a scalar variable or array. E402 Actual argument $ is illegal E403 Actual argument $ and formal argument $ have different ranks The actual and formal array arguments differ in rank, which is allowed only if both arrays are declared with the HPF SEQUENCE attribute. E404 Sequential array section of $ in argument $ is not contiguous When passing an array section to a formal argument that has the HPF SEQUENCE attribute, the actual argument must be a whole array with the HPF SEQUENCE attribute, or an array section of such an array where the section is a contiguous sequence of elements. E405 Array expression argument $ may not be passed to sequential dummy argument $ When the dummy argument has the HPF SEQUENCE attribute, the actual argument must be a whole array with the HPF SEQUENCE attribute or a contiguous array section of such an array, unless an INTERFACE block is used. E406 Actual argument $ and formal argument $ have different character lengths The actual and formal array character arguments have different character lengths, which is allowed only if both character arrays are declared with the HPF SEQUENCE attribute, unless an INTERFACE block is used. W407 Argument $ has a different character length than dummy argument $ The character length of the actual argument is different than the length specified for the corresponding dummy argument. W408 Specified main program $ is not a PROGRAM The main program specified on the command line is a subroutine, function, or block data subprogram. W409 More than one main program in IPA directory: $ and $ There is more than one main program analyzed in the IPA directory shown. The first one found is used. W410 No main program found; IPA analysis fails. The main program must appear in the IPA directory for analysis to proceed. W411 Formal argument $ is DYNAMIC but actual argument is an expression W412 Formal argument $ is DYNAMIC but actual argument $ is not I413 Formal argument $ has two reaching distributions and may be a candidate for cloningPGI® User’s Guide 358 I414 $ and $ may be aliased and one of them is assigned Interprocedural analysis has determined that two formal arguments because the same variable is passed in both argument positions, or one formal argument and a global or COMMON variable may be aliased, because the global or COMMON variable is passed as an actual argument. If either alias is assigned in the subroutine, unexpected results may occur; this message alerts the user that this situation is disallowed by the Fortran standard. F415 IPA fails: incorrect IPA file Interprocedural analysis saves its information in special IPA files in the specified IPA directory. One of these files has been renamed or corrupted. This can arise when there are two files with the same prefix, such as ’a.hpf’ and ’a.f90’. E416 Argument $ has the SEQUENCE attribute, but the dummy parameter $ does not When an actual argument is an array with the SEQUENCE attribute, the dummy parameter must have the SEQUENCE attribute or an INTERFACE block must be used. E417 Interface block for $ is a SUBROUTINE but should be a FUNCTION E418 Interface block for $ is a FUNCTION but should be a SUBROUTINE E419 Interface block for $ is a FUNCTION has wrong result type W420 Earlier $ directive overrides $ directive W421 $ directive can only appear in a function or subroutine E422 Nonconstant DIM= argument is not supported E423 Constant DIM= argument is out of range E424 Equivalence using substring or vector triplets is not allowed E425 A record is not allowed in this context E426 WORD type cannot be converted E427 Interface block for $ has wrong number of arguments E428 Interface block for $ should have $ E429 Interface block for $ should not have $ E430 Interface block for $ has wrong $ W431 Program is too large for Interprocedural Analysis to complete W432 Illegal type conversion $ E433 Subprogram $ called within INDEPENDENT loop not LOCAL W434 Incorrect home array specification ignoredChapter 22. Messages 359 S435 Array declared with zero size An array was declared with a zero or negative dimension bound, as ’real a(-1)’, or an upper bound less than the lower bound, as ’real a(4:2)’. W436 Independent loop not parallelized$ W437 Type $ will be mapped to $ Where DOUBLE PRECISION is not supported, it is mapped to REAL, and similarly for COMPLEX(16) or COMPLEX*32. E438 $ $ not supported on this platform This construct is not supported by the compiler for this target. S439 An internal subprogram cannot be passed as argument - $ S440 Defined assignment statements may not appear in WHERE statement or WHERE block S441 $ may not appear in a FORALL block E442 Adjustable-length character type not supported on this host - $ $ S443 EQUIVALENCE of derived types not supported on this host - $ S444 Derived type in EQUIVALENCE statement must have SEQUENCE attribute - $ A variable or array with derived type appears in an EQUIVALENCE statement. The derived type must have the SEQUENCE attribute, but does not. E445 Array bounds must be integer $ $ The expressions in the array bounds must be integer. S446 Argument number $ to $: rank mismatch The number of dimensions in the array or array expression does not match the number of dimensions in the dummy argument. S447 Argument number $ to $ must be a subroutine or function name S448 Argument number $ to $ must be a subroutine name S449 Argument number $ to $ must be a function name S450 Argument number $ to $: kind mismatch S451 Arrays of derived type with a distributed member are not supported S452 Assumed length character, $, is not a dummy argument S453 Derived type variable with pointer member not allowed in IO - $ $PGI® User’s Guide 360 S454 Subprogram $ is not a module procedure Only names of module procedures declared in this module or accessed through USE association can appear in a MODULE PROCEDURE statement. S455 A derived type array section cannot appear with a member array section - $ A reference like A(:)%B(:), where ’A’ is a derived type array and ’B’ is a member array, is not allowed; a section subscript may appear after ’A’ or after ’B’, but not both. S456 Unimplemented for data type for MATMUL S457 Illegal expression in initialization S458 Argument to NULL() must be a pointer S459 Target of NULL() assignment must be a pointer S460 ELEMENTAL procedures cannot be RECURSIVE S461 Dummy arguements of ELEMENATAL procedures must be scalar S462 Arguments and return values of ELEMENATAL procedures cannot have the POINTER attribute S463 Arguments of ELEMENATAL procedures cannot be procedures S464 An ELEMENTAL procedure cannot be passed as argument - $ Fortran Runtime Error Messages This section presents the error messages generated by the runtime system. The runtime system displays error messages on standard output. Message Format The messages are numbered but have no severity indicators because they all terminate program execution. Message List Here are the runtime error messages: 201 illegal value for specifier An improper specifier value has been passed to an I/O runtime routine. Example: within an OPEN statement, form='unknown'. 202 conflicting specifiers Conflicting specifiers have been passed to an I/O runtime routine. Example: within an OPEN statement, form='unformatted', blank='null'. 203 record length must be specified A recl specifier required for an I/O runtime routine has not been passed. Example: within an OPEN statement, access='direct' has been passed, but the record length has not been specified (recl=specifier).Chapter 22. Messages 361 204 illegal use of a readonly file Self explanatory. Check file and directory modes for readonly status. 205 'SCRATCH' and 'SAVE'/'KEEP' both specified In an OPEN statement, a file disposition conflict has occurred. Example: within an OPEN statement, status='scratch' and dispose='keep' have been passed. 206 attempt to open a named file as 'SCRATCH' 207 file is already connected to another unit 208 'NEW' specified for file that already exists 209 'OLD' specified for file that does not exist 210 dynamic memory allocation failed Memory allocation operations occur only in conjunction with namelist I/O. The most probable cause of fixed buffer overflow is exceeding the maximum number of simultaneously open file units. 211 invalid file name 212 invalid unit number A file unit number less than or equal to zero has been specified. 215 formatted/unformatted file conflict Formatted/unformatted file operation conflict. 217 attempt to read past end of file 219 attempt to read/write past end of record For direct access, the record to be read/written exceeds the specified record length. 220 write after last internal record 221 syntax error in format string A runtime encoded format contains a lexical or syntax error. 222 unbalanced parentheses in format string 223 illegal P or T edit descriptor - value missing 224 illegal Hollerith or character string in format An unknown token type has been found in a format encoded at run-time. 225 lexical error -- unknown token type 226 unrecognized edit descriptor letter in format An unexpected Fortran edit descriptor (FED) was found in a runtime format item. 228 end of file reached without finding group 229 end of file reached while processing group 230 scale factor out of range -128 to 127 Fortran P edit descriptor scale factor not within range of -128 to 127. 231 error on data conversionPGI® User’s Guide 362 233 too many constants to initialize group item 234 invalid edit descriptor An invalid edit descriptor has been found in a format statement. 235 edit descriptor does not match item type Data types specified by I/O list item and corresponding edit descriptor conflict. 236 formatted record longer than 2000 characters 237 quad precision type unsupported 238 tab value out of range A tab value of less than one has been specified. 239 entity name is not member of group 242 illegal operation on direct access file 243 format parentheses nesting depth too great 244 syntax error - entity name expected 245 syntax error within group definition 246 infinite format scan for edit descriptor 248 illegal subscript or substring specification 249 error in format - illegal E, F, G or D descriptor 250 error in format - number missing after '.', '-', or '+' 251 illegal character in format string 252 operation attempted after end of file 253 attempt to read non-existent record (direct access) 254 illegal repeat count in format363 Index Symbols 64-Bit Programming, 125 A Arguments passing, 113 passing by value, 113 Arrays indices, 114 Auto-parallelization, 32 B Basic block, 22 Blocks Fortran named common, 112 Bounds checking, 239 C C/C++ Builtin Functions, 75 C/C++ Math Header File, 75 C$PRAGMA C, 72 C++ Name Mangling, 159 C++ Standard Template Library, 88 Cache tiling failed cache tiling, 241 with -Mvect, 237 Command line case sensitivity, 2 include files, 5 option order, 3 Command-line Options, 3, 15, 163, 184 -#, 170, 170 -###, 171, 171 -A, 208 -a, 208 -alias, 209 -B, 209 -b, 210 -b3, 210 --Bdynamic, 171, 171 Build-related, 163 -byteswapio, 172 -C, 172 -c, 173 --cfront_2.1, 211 --cfront_3.0, 211 --compress_names, 212 --create_pch, 212 -d, 173 -D, 174 Debug-related, 166, 167, 167 --diag_error, 212 --diag_remark, 212 --diag_suppress, 213 --diag_warning, 213 --display_error_number, 213, 214, 214 -dryrun, 174 -E, 175 -F, 175 -fast, 175 -fastsse, 176 -flagcheck, 176 -flags, 176 -fpic, 176, 177 -fPIC, 177 -G, 177 -g, 177 -g77libs, 178 Generic PGI options, 170 -gopt, 178 -help, 178 -I, 180 -i2, -i4 and -i8, 181 --keeplnk, 183 -Kflag, 181 -L, 183 -l, 183 -M, 214 -Mallocatable, 226 -Manno, 239, 242 -Masmkeyword, 224 -Mbackslash, 226 -Mbounds, 239 -Mbyteswapio, 239 -Mcache_align, 230 -Mchkfpstk, 239 -Mchkptr, 239 -Mchkstk, 239 -mcmodel=medium, 126 -mcmodel=small, 126 -Mconcur, 230 -Mcpp, 240 -Mcray, 231 -MD, 215 -Mdaz, 220 -Mdclchk, 226 -Mdefaultunit, 227 -Mdepchk, 231 -Mdlines, 227 -Mdll, 240 -Mdollar, 224, 227 -Mdse, 231 -Mdwarf1, 220 -Mdwarf2, 220 -Mdwarf3, 220 -Mextend, 227 -Mextract, 228 -Mfcon, 224 -Mfixed, 227 -Mflushz, 220 -Mfprelaxed, 232 -Mfree, 227 -Mfunc32, 221 -Mgccbugs, 240 -Mi4, 232 -Minform, 241 -Minline, 229 -Miomutex, 227 -Mipa, 232 -Mkeepasm, 242 -Mlarge_arrays, 126, 126, 221 -Mlfs, 225 -Mlist, 242PGI® User’s Guide 364 -Mloop32, 234 -Mlre, 234 -Mmakedll, 242, 242 -Mneginfo, 241 -Mnoasmkeyword, 224 -Mnobackslash, 226 -Mnobounds, 239 -Mnodaz, 220 -Mnodclchk, 227 -Mnodefaultunit, 227 -Mnodepchk, 231 -Mnodlines, 227 -Mnodse, 231 -Mnoflushz, 220 -Mnofprelaxed, 232 -Mnoframe, 234 -Mnoi4, 234 -Mnoiomutex, 227 -Mnolarge_arrays, 221, 221 -Mnolist, 242 -Mnoloop32, 234 -Mnolre, 234 -Mnomain, 221 -Mnoonetrip, 227 -Mnoopenmp, 242 -Mnopgdllmain, 242 -Mnoprefetch, 235 -Mnor8, 235 -Mnor8intrinsics, 236 -Mnorecursive, 222 -Mnoreentrant, 223 -Mnoref_externals, 222 -Mnosave, 227 -Mnoscalarsse, 236 -Mnosecond_underscore, 223 -Mnosgimp, 242 -Mnosignextend, 223 -Mnosingle, 225 -Mnosmart, 236 -Mnostartup, 225, 225 -Mnostddef, 226 -Mnostdlib, 226, 226 -Mnostride0, 223 -Mnounixlogical, 228 -Mnounroll, 237 -Mnoupcase, 228, 228 -Mnovect, 238 -Mnovintr, 238 -module, 191 -Monetrip, 227 -mp, 191 -Mpfi, 235 -Mpfo, 235 -Mprefetch, 235 -Mpreprocess, 242 -Mprof, 222 -Mr8, 235 -Mr8intrinsics, 236 -Mrecursive, 222 -Mreentrant, 222 -Mref_externals, 222 -Msafe_lastval, 223 -Msafeptr, 236 -Msave, 227 -Mscalarsse, 236 -Mschar, 224 -Msecond_underscore, 223 -Msignextend, 223 -Msingle, 225 -Msmart, 236 -Mstandard, 227 -Mstride0, 223 -Muchar, 225 -Munix, 223 -Munixlogical, 227 -Munroll, 237 -Mupcase, 228 -Mvarargs, 223 -Mvect, 237 -nfast, 192 --llalign, 214, 221, 231, 231, 235, 236 --alternative_tokens, 209, 209, 210, 214, 217, 218, 219 -nontemporal moves, 221 -noswitcherror, 192 -O, 193 -o, 194 -- optk_allow_dollar_in_id_chars, 215 -P, 215 -pc, 194 --pch, 216, 216, 219 --pch_dir, 216 -pg, 196, 196, 197 --preinclude, 217 -Q, 197 -R, 198 -r, 198, 198 -r4 and -r8, 198 -rc, 199 -rpath, 199 -s, 184, 199 -S, 200 -shared, 200 -show, 200 -silent, 201 -soname, 201 -stack, 201 syntax, 2 -t, 218 -time, 202 -tp, 202 -u, 205 -U, 205 --use_pch, 217 -V, 205 -v, 206 -W, 206 -w, 207 -Xs, 207 -Xt, 208 Compilation driver, 1 Compilers Invoke at command level, 1 PGC++, xxvi PGF77, xxvi PGF95, xxvi PGHPF, xxvi Constraints *, 144, 144 &, 144 %, 144 +, 144 =, 144 character, 139 machine, 141Index 365 machine, example, 142 modifiers, 143 multiple alternative, 143 operand, 139 operant aliases, 145 simple, 139 cpp, 5 D Data Types, 6, 151 Aggregate, 6 attributes, 158 bit-fields, 158 C/C++ aggregate alignment, 157 C/C++ scalar data types, 154 C/C++ struct, 156 C/C++ void, 158 C++ class and object layout, 156 C++ classes, 156 DEC structures, 153 DEC Unions, 153 F90 derived types, 154 Fortran Scalars, 151 internal padding, 157 scalars, 6 tail padding, 157 Deployment, 103 Directives C/C++, 3, 3 Fortran, 3 optimization, 63, 263 Parallelization, 51, 243, 263 prefetch, 69 scope, 66 E Environment variables, 89, 89, 91, 92 FLEXLM_BATCH, 91, 93 FORTRAN_OPT, 93 GMON_OUT_PREFIX, 93 LD_LIBRARY_PATH, 91 MANPATH, 93, 93, 94 MP_BIND, 94 MP_BLIST, 95 MP_SPIN, 95 MP_WARN, 95 NCPUS, 96 NCPUS_MAX, 96 NO_STOP_MESSAGE, 96 OMP_STACK_SIZE, 9, 11, 12, 59, 60, 92 OMP_WAIT_POLICY, 59, 60, 92 PATH, 92, 96 PGI, 96 PGI_CONTINUE, 97 PGI_OBJSUFFIX, 97 PGI_TERM, 97 PGI_TERM_DEBUG, 97, 99, 240 PWD, 99 STATIC_RANDOM_SEED, 99 TMPDIR, 100, 100 F Filename Conventions, 3 extensions, 3 Input Files, 3 Output Files, 5 Floating-point stack, 194 Fortran directive summary, 64 named common blocks, 112 Fortran Parallelization Directives ATOMIC, 251 DOACROSS, 246, 246 Function Inlining inlining and makefiles, 48 inlining examples, 49 inlining restrictions, 49 H Hello example, 2 I Inline Assembly clobber list, 138 Inter-language Calling, 109 %VAL, 113 arguments and return values, 113 array indices, 114 C$PRAGMA C, 72 C++ calling C, 116 C++ calling Fortran, 119 C calling C++, 117 character case conventions, 111 character return values, 113 compatible data types, 111 Fortran calling C, 115 Fortran calling C++, 118 underscores, 72, 111 L Language options, 223 Libraries -Bdynamic option, 171, 171 BLAS, 88 FFTs, 88 LAPACK, 88 LIB3F, 88 shared object files, 76 Linux, 9 Header Files, 9 large static data, 126 Parallelization, 9 Listing Files, 239, 242, 242 Loops failed auto-parallelization, 34 innermost, 34 scalars, 34 timing, 34 Loop unrolling, 27 M MAC OS Parallelization, 12 MAC OS X Header Files, 11 Mangling C++ names, 159 types, 160 Modifiers assembly string, 145 characters, 145 N Name mangling local class, 161 nested class, 161PGI® User’s Guide 366 template class, 161 type, 160 O OpenMP C/C++ Pragmas, 51, 243 flush, 249, 250 ordered, 251 parallel, 251 parallel sections, 255, 256 OpenMP C/C++ Support Routines omp_destroy_lock(), 58 omp_destroy_nest_lock(), 58 omp_get_dynamic(), 57 omp_get_max_threads(), 56 omp_get_nested(), 57 omp_get_num_procs(), 56 omp_get_num_threads(), 55, 55 omp_get_stack_size(), 56 omp_get _thread_num(), 55 omp_get_wtick(), 58 omp_get_wtime(), 57 omp_in_parallel(), 56 omp_init_lock(), 58 omp_init_nest_lock(), 58 omp_set_dynamic(), 57 omp_set_lock(), 58 omp_set_nest_lock(), 58 omp_set_nested(), 57 omp_set_num_threads(), 55 omp_set_stack_size(), 56 omp_test_lock(), 58 omp_test_nest_lock(), 58 omp_unset_lock(), 58 omp_unset_nest_lock(), 58 OpenMP C++ Pragmas omp critical, 245 OpenMP environment variables MPSTKZ, 91, 94 OMP_DYNAMIC, 59, 59, 59, 60, 92, 92 OMP_NESTED, 59, 59, 92 OMP_NUM_THREADS, 59, 59, 92 OpenMP Fortran Directives, 51, 243, 263 ATOMIC, 244 BARRIER, 244 CRITICAL, 245 DO, 247 FLUSH, 249 MASTER, 250 ORDERED, 251 PARALLEL, 251 PARALLEL DO, 254, 254 PARALLEL SECTIONS, 255 PARALLEL WORKSHARE, 256 SECTIONS, 257, 257 SINGLE, 257 THREADPRIVATE, 258 WORKSHARE, 259 OpenMP Fortran Support Routines omp_destroy_lock(), 58 omp_get_dynamic(), 57 omp_get_max_threads(), 56 omp_get_nested(), 57 omp_get_num_procs(), 56 omp_get_num_threads(), 55, 55 omp_get_stack_size(), 56 omp_get_thread_num(), 55 omp_get_wtick(), 58 omp_get_wtime(), 57 omp_in_parallel(), 56 omp_init_lock(), 58 omp_set_dynamic(), 57 omp_set_lock(), 58 omp_set_nested(), 57 omp_set_num_threads(), 55 omp_set_stack_size(), 56 omp_test_lock(), 58 omp_unset_lock(), 58 Operand aliases, 145 modifier *, 144, 144 modifier &, 144 modifier %, 144 modifier +, 144 modifier =, 144 Operand constaints see constraints, 139 Operand constraints machine, 141 Optimization, 63, 263 C/C++ pragmas, 42, 64 C/C++ pragmas scope, 67 cache tiling, 237 Fortran directives, 42, 63, 263 Fortran directives scope, 66 function inlining, 14, 22, 45 global, 25 global optimization, 22, 25 inline libraries, 46 Inter-Procedural Analysis, 22 IPA, 22 local, 25 local optimization, 22 loop optimization, 22 loops, 234, 234, 234 loop unrolling, 22, 27 no level specified, 25 none, 25 -O, 193 -O0, 24 -O1, 24 -O2, 24 -O3, 25, 25 -Olevel, 24 parallelization, 22, 32 PFO, 22 pointers, 236 prefetching, 235, 235, 235 profile-feedback (PFO), 41 Profile-Feedback Optimization, 22 vectorization, 22, 28 Options Mchkfpstk, 97 P Parallelization, 32 auto-parallelization, 32 failed auto-parallelization, 34, 241 Mac OS, 12 -Mconcur auto-parallelization, 230 NCPUS environment variable, 33 safe_lastval, 35 user-directed, 191 Parallelization Directives, 51, 243, 263Index 367 Parallelization Pragmas, 51, 243 Pragmas C/C++, 3 optimization, 64 scope, 67 Prefetch directives, 69 Preprocessor cpp, 5 Fortran, 5 Proprietary environment variables FORTRAN_OPT, 91, 93 GMON_OUT_PREFIX, 91, 91 MANPATH, 91 MP_BIND, 91 MP_BLIST, 91 MP_SPIN, 91 MP_WARN, 91 NCPUS, 91 NCPUS_MAX, 91 NO_STOP_MESSAGE, 92 PGI, 92 PGI_CONTINUE, 92 PGI_OBJSUFFIX, 92 PGI_STACK_USAGE, 92 PGI_TERM, 92 PGI_TERM_DEBUG, 92, 92 STATIC_RANDOM_SEED, 92 TMP, 92 TMPDIR, 92 R Return values character, 113 Run-time Environment, 271 S Shared object files, 76 SUA/SFU, 11 Header Files, 11 Parallelization, 11 T Timing CPU_CLOCK, 43 execution, 43 SYSTEM_CLOCK, 43 Tools PGDBG, xxvi PGPROF, xxvi, xxvi V Vectorization, 28, 237 SSE instructions, 238 W Win32 Calling Conventions C, 120, 121 Default, 120, 121 STDCALL, 120, 121 symbol name construction, 121 UNIX-style, 120, 122368 About This Guide 
   
  "! "# $% $&'(    "$) * + ",* .-0/
1
2# , 34 !5
 + 3,*76 8 & 3,&'9:; 8 < 8$,(;= 
$(# $ 3! 3,1# $> " @?BADC+FEBG    ",&(* .$,= Related Publications
D!H/&'1 $1,IJ, !H 3,@
,K,* #
$ ! 8L MONKPQR,SITVU3WFXY"Z.PW[QB\1]SIP1Y^;SVTIU`_NKX^;aKbKUZ1U3WcU3Y deUgfh\1Y;i \1T M jlkXN4mn)jlo[U3W%NKPQQB\1Y p,o`b4UZ1U3WcU3Y deUgfh\Y i \T M jlkXN4mn^ SITIUF^;P1W3Q\]oF\1Y;pqn R,U3deSV\1T^ SITIU[o`bKUZ1U[W8U[Y;d"U5fh\1Y i;\1T M jlkXN4mnn r,o3]U3Qts;SVuW[\W[SIU3oFb4UZ1U3WcU3Y deUgfh\Y i \T MONKW3\1rvNw[NKxx2b4UZ1U3WcU3Y deUgfh\Y i \T MONK^;y,zDN4P1QBQ\1Y;p,o`\1Y p{BSVW8U[de]SI|,U[o`bKUZ1U[W8U[Y;d"U4fh\1Y;i \1T MO^ PW[]HW[\Ys;\1Y}i \8}0UFbKUZ1U3WcU3Y deUgfh\1Y;i \1TI~K0P1TIi QBU�? MO^ PW[]HW[\Ys;\1Y}i \8}0UFbKUZ1U3WcU3Y deUgfh\1Y;i \1TI~K0P1TIi QBU�? MO^ PW[]HW[\Ys;\1Y}i \8}0UFbKUZ1U3WcU3Y deUgfh\1Y;i \1TI~K0P1TIi QBU�? MOmBRU3Yf??@^ PW[]HW[\Y???R,R,TVSIde\1]SIP1Y? W8Pc}W3\1Q?XY;]U[WZ1\1deU Obtaining Publications
 jlo3U[WF?i u?TISId"\]SVPY o`N4\1]\1TIPc} $(1 " $
,:;,, *7,?1 $4 !5, + 3,* ?K$($, 81
@
, 8/, c,?( !/, 8 
,K, 8#,:0, ?1 ($ 3,= + (  ">/
7(  .1 3 D
 + "*C!H 3�?+?4CH! 3?K 8&; 3,�1,,1. 
B! 3,@
 +?4C!H 3�(* .$)= ? 8 `v  1 $-?1,???,?,??,???,?,?,?0= G2?> 3,
1>$7  * $$�,* ($?$$1 8 �? orderdsk@sgi.com??ACH? (*;(¡ ($ 3.?e= + (  ">/
7(  .1 3 D
 +?4C!H 3� 8& ",¡2 8 $ �(!H/, 8 8$,(  ¢ ,& $$ 8,I* *' .0&
 Order Cray Software ?K= 004–2182–003 xvii Scienti?c Libraries User’s Guide 004–2151–002© 1996, 1999 Silicon Graphics, Inc. All Rights Reserved. This manual or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Silicon Graphics, Inc. LIMITED AND RESTRICTED RIGHTS LEGEND Use, duplication, or disclosure by the Government is subject to restrictions as set forth in the Rights in Data clause at FAR 52.227-14 and/or in similar or successor clauses in the FAR, or in the DOD, DOE or NASA FAR Supplements. Unpublished rights reserved under the Copyright Laws of the United States. Contractor/manufacturer is Silicon Graphics, Inc., 1600 Amphitheatre Pkwy., Mountain View, CA 94043-1351. Autotasking, CF77, Cray, Cray Ada, CraySoft, Cray Y-MP, Cray-1, CRInform, CRI/TurboKiva, HSX, LibSci, MPP Apprentice, SSD, SUPERCLUSTER, UNICOS, X-MP EA, and UNICOS/mk are federally registered trademarks and Because no workstation is an island, CCI, CCMT, CF90, CFT, CFT2, CFT77, ConCurrent Maintenance Tools, COS, Cray Animation Theater, Cray APP, Cray C90, Cray C90D, Cray C++ Compiling System, CrayDoc, Cray EL, Cray J90, Cray J90se, CrayLink, Cray NQS, Cray/REELlibrarian, Cray S-MP, Cray SSD-T90, Cray SV1, Cray T90, Cray T3D, Cray T3E, CrayTutor, Cray X-MP, Cray XMS, Cray-2, CSIM, CVT, Delivering the power . . ., DGauss, Docview, EMDS, GigaRing, HEXAR, IOS, ND Series Network Disk Array, Network Queuing Environment, Network Queuing Tools, OLNET, RQS, SEGLDR, SMARTE, SUPERLINK, System Maintenance and Remote Testing Environment, Trusted UNICOS, and UNICOS MAX are trademarks of Cray Research, L.L.C., a wholly owned subsidiary of Silicon Graphics, Inc. SGI is a trademark of Silicon Graphics, Inc. Silicon Graphics, the Silicon Graphics logo, and IRIS are registered trademarks, and CASEVision, IRIS 4D, IRIS Power Series, IRIX, Origin2000, and POWER CHALLENGE are trademarks of Silicon Graphics, Inc. MIPS, R4000, R4400, and R8000 are registered trademarks and MIPSpro and R10000 are trademarks of MIPS Technologies, Inc. UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company, Ltd. VMS and VAX are trademarks of Digital Equipment Corporation. Portions of this product and document are derived from material copyrighted by Kuck and Associates, Inc. DynaText and DynaWeb are registered trademarks of Inso Corporation. Silicon Graphics and the Silicon Graphics logo are registered trademarks of Silicon Graphics, Inc. UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company Limited. X/Open is a trademark of X/Open Company Ltd. The X device is a trademark of the Open Group. The UNICOS operating system is derived from UNIX® System V. The UNICOS operating system is also based in part on the Fourth Berkeley Software Distribution (BSD) under license from The Regents of the University of California. St. Peter’s Basilica image courtesy of ENEL SpA and InfoByte SpA. Disk Thrower image courtesy of Xavier Berenguer, Animatica.New Features Scienti?c Libraries User’s Guide 004–2151–002 This guide has been expanded to include appendixes that describe the implementation of version 2 of the Math library (libm), used on UNICOS systems, as well as describing the algorithms used in that library.Record of Revision Version Description 2.0 July 1996. Draft printing to support the Programming Environment 2.0.1 release. 3.0 June 1997. Printing to support the Programming Environment 3.0 release. 3.3 July 1999. Printing to support the Programming Environment 3.3 release. 004–2151–002 iContents Page About This Guide xi Related publications . . . . . . . . . . . . . . . . . . . . . . . xi Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . xiv Obtaining Publications . . . . . . . . . . . . . . . . . . . . . . . xv Reader Comments . . . . . . . . . . . . . . . . . . . . . . . . xv Introduction [1] 1 Parallel Processing Issues [2] 3 Parallel Processing Overview . . . . . . . . . . . . . . . . . . . . . 3 Parallel Processing: Hardware Level . . . . . . . . . . . . . . . . . . 3 Parallel Processing: Operating System Level . . . . . . . . . . . . . . . 4 Parallel Processing: Code Level . . . . . . . . . . . . . . . . . . . 5 Costs and Bene?ts of Parallel Processing . . . . . . . . . . . . . . . . . 5 Bene?ts of Parallel Processing . . . . . . . . . . . . . . . . . . . . 6 Parallel Processing Overhead . . . . . . . . . . . . . . . . . . . . 6 Parallelism and Vectorization . . . . . . . . . . . . . . . . . . . . 7 Parallelism and Load Balancing . . . . . . . . . . . . . . . . . . . 8 Performance Issues . . . . . . . . . . . . . . . . . . . . . . . . 8 Calculating Speedups in Multitasked Programs . . . . . . . . . . . . . . 8 Amdahl’s Law . . . . . . . . . . . . . . . . . . . . . . . 9 Determining Ef?ciency . . . . . . . . . . . . . . . . . . . . . . 10 Estimating Percentage of Parallelism . . . . . . . . . . . . . . . . . . 11 Parallel Processing Environments . . . . . . . . . . . . . . . . . . . 11 Environment Variables . . . . . . . . . . . . . . . . . . . . . . 12 004–2151–002 iiiScienti?c Libraries User’s Guide Page The Dedicated Environment . . . . . . . . . . . . . . . . . . . . 13 The Multiuser Environment . . . . . . . . . . . . . . . . . . . . 13 Measuring Scienti?c Library Performance . . . . . . . . . . . . . . . . . 14 Simple Measurements . . . . . . . . . . . . . . . . . . . . . . 14 Determining Multitasking of Routines . . . . . . . . . . . . . . . . . 15 Parallel Processing Strategies . . . . . . . . . . . . . . . . . . . . . 20 Problem Sizes . . . . . . . . . . . . . . . . . . . . . . . . . 20 Strategies for a Dedicated Environment . . . . . . . . . . . . . . . . . 21 Strategies for a Multiuser Environment . . . . . . . . . . . . . . . . . 21 LAPACK [3] 25 LAPACK in the Scienti?c Library . . . . . . . . . . . . . . . . . . . 25 Types of Problems Solved by LAPACK . . . . . . . . . . . . . . . . . . 26 Solving Linear Systems . . . . . . . . . . . . . . . . . . . . . . 27 Factoring a Matrix . . . . . . . . . . . . . . . . . . . . . . . 29 Example 1: LU factorization . . . . . . . . . . . . . . . . . . . 30 Example 2: Symmetric inde?nite matrix factorization . . . . . . . . . . . 31 Error Codes . . . . . . . . . . . . . . . . . . . . . . . . . 33 Example 3: Error conditions . . . . . . . . . . . . . . . . . . . 33 Solving from the Factored Form . . . . . . . . . . . . . . . . . . . . 34 Condition Estimation . . . . . . . . . . . . . . . . . . . . . . 36 Example 4: Roundoff errors . . . . . . . . . . . . . . . . . . . 37 Use in Error Bounds . . . . . . . . . . . . . . . . . . . . . . 37 Equilibration . . . . . . . . . . . . . . . . . . . . . . . . . 39 Iterative Re?nement . . . . . . . . . . . . . . . . . . . . . . . 41 Example 5: Hilbert matrix . . . . . . . . . . . . . . . . . . . 41 Error Bounds . . . . . . . . . . . . . . . . . . . . . . . . 43 Inverting a Matrix . . . . . . . . . . . . . . . . . . . . . . . 44 Solving Least Squares Problems . . . . . . . . . . . . . . . . . . . . 45 iv 004–2151–002Contents Page Orthogonal Factorizations . . . . . . . . . . . . . . . . . . . . . 45 Example 6: Orthogonal factorization . . . . . . . . . . . . . . . . 46 Multiplying by the Orthogonal Matrix . . . . . . . . . . . . . . . . . 48 Generating the Orthogonal Matrix . . . . . . . . . . . . . . . . . . 49 Comparing Answers . . . . . . . . . . . . . . . . . . . . . . . 50 Using Sparse Linear Solvers [4] 53 Sparse Matrices . . . . . . . . . . . . . . . . . . . . . . . . . 53 Solution Techniques . . . . . . . . . . . . . . . . . . . . . . . 54 Direct Methods . . . . . . . . . . . . . . . . . . . . . . . . 55 Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . 56 Sparse Solvers . . . . . . . . . . . . . . . . . . . . . . . . . 57 Data Structures for General Sparse Matrices . . . . . . . . . . . . . . . 57 Direct Solvers . . . . . . . . . . . . . . . . . . . . . . . . . 58 Iterative Solvers . . . . . . . . . . . . . . . . . . . . . . . . 59 Other Solvers . . . . . . . . . . . . . . . . . . . . . . . . . 60 Choosing a Solver . . . . . . . . . . . . . . . . . . . . . . . . 62 Using Sparse Solvers . . . . . . . . . . . . . . . . . . . . . . 62 Tridiagonal Systems . . . . . . . . . . . . . . . . . . . . . . . 62 General-patterned Sparse Linear Systems . . . . . . . . . . . . . . . . 63 Choosing a Method Based on Problem Type . . . . . . . . . . . . . . . 64 Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . 65 Preconditioning . . . . . . . . . . . . . . . . . . . . . . . 65 Direct General Sparse Solvers . . . . . . . . . . . . . . . . . . . . 67 Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . 67 Parallel Processing . . . . . . . . . . . . . . . . . . . . . . . 67 Reusing Information . . . . . . . . . . . . . . . . . . . . . . 68 Reuse of Structure . . . . . . . . . . . . . . . . . . . . . . 68 Multiple Right-hand Sides . . . . . . . . . . . . . . . . . . . . 68 004–2151–002 vScienti?c Libraries User’s Guide Page Reuse of Values . . . . . . . . . . . . . . . . . . . . . . . 68 Save/restart . . . . . . . . . . . . . . . . . . . . . . . . 69 SITRSOL Tuning Issues . . . . . . . . . . . . . . . . . . . . . 69 Direct Solver Tuning Issues . . . . . . . . . . . . . . . . . . . . 70 SITRSOL Quick Reference . . . . . . . . . . . . . . . . . . . . . 72 Usage Examples . . . . . . . . . . . . . . . . . . . . . . . . . 75 Example 7: General symmetric positive de?nite . . . . . . . . . . . . . . 75 Example 8: General unsymmetric . . . . . . . . . . . . . . . . . . 79 Example 9: Reuse of structure . . . . . . . . . . . . . . . . . . . 84 Example 10: Multiple right-hand sides . . . . . . . . . . . . . . . . 89 Example 11: Save/restart . . . . . . . . . . . . . . . . . . . . 93 Out-of-core Linear Algebra Software [5] 101 Out-of-core Routines . . . . . . . . . . . . . . . . . . . . . . . 101 Virtual Matrices . . . . . . . . . . . . . . . . . . . . . . . . . 102 Unit Numbers . . . . . . . . . . . . . . . . . . . . . . . . 102 File Format . . . . . . . . . . . . . . . . . . . . . . . . . 103 Leading Virtual Dimension . . . . . . . . . . . . . . . . . . . . 104 De?nition and Rede?nition of Elements . . . . . . . . . . . . . . . . 104 File Size . . . . . . . . . . . . . . . . . . . . . . . . . . 104 Packed Storage Mode . . . . . . . . . . . . . . . . . . . . . . 105 Page Size . . . . . . . . . . . . . . . . . . . . . . . . . . 106 Subroutine Types . . . . . . . . . . . . . . . . . . . . . . . . 106 Complex Routines . . . . . . . . . . . . . . . . . . . . . . . 106 Summary of Routines . . . . . . . . . . . . . . . . . . . . . . 107 Initialization and Termination Subroutines . . . . . . . . . . . . . . . . 109 Virtual Copy Subroutines . . . . . . . . . . . . . . . . . . . . . 109 Virtual LAPACK Subroutines . . . . . . . . . . . . . . . . . . . . 110 Virtual BLAS Subroutines . . . . . . . . . . . . . . . . . . . . . 111 vi 004–2151–002Contents Page Using Strassen’s algorithm . . . . . . . . . . . . . . . . . . . . 111 Lower-level Routines . . . . . . . . . . . . . . . . . . . . . . 112 Examples of Out-of-core Subroutine Use . . . . . . . . . . . . . . . . . 113 Example 12: Creating a virtual matrix . . . . . . . . . . . . . . . . . 113 Example 13: Multiplying a virtual matrix . . . . . . . . . . . . . . . 113 Example 14: Example of protocol usage . . . . . . . . . . . . . . . . 114 UNICOS Environment Variables . . . . . . . . . . . . . . . . . . . . 115 Multitasking . . . . . . . . . . . . . . . . . . . . . . . . . 116 Error Reporting . . . . . . . . . . . . . . . . . . . . . . . . . 116 Performance Measurement and Tuning . . . . . . . . . . . . . . . . . . 117 Page-Buffer Space . . . . . . . . . . . . . . . . . . . . . . . 118 Memory Usage Guidelines . . . . . . . . . . . . . . . . . . . . . 118 Memory Requirement for VSGETRF and VCGETRF Routines . . . . . . . . . . 119 Sample Performance Statistics . . . . . . . . . . . . . . . . . . . . 119 Appendix A Appendix A: libm Version 2 123 Overview of Math Libraries . . . . . . . . . . . . . . . . . . . . . 123 The 1 ULP Criterion . . . . . . . . . . . . . . . . . . . . . . . 124 Numerical Methods . . . . . . . . . . . . . . . . . . . . . . . 125 Side Effects . . . . . . . . . . . . . . . . . . . . . . . . . 128 Appendix B Appendix B: Math Algorithms 129 Single-precision Real Logarithm Functions ln(x) and log(x) . . . . . . . . . . . 129 Procedure 1: ln(x) and log(x) . . . . . . . . . . . . . . . . . . . 129 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Single-precision Real Logarithm Functions ALOG(x) and ALOG10(x) . . . . . . . . . 132 Procedure 2: ALOG(x) and ALOG10(x) . . . . . . . . . . . . . . . . . 132 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 134 Single-precision Real ASIN(x) and ACOS(x) Functions . . . . . . . . . . . . . 135 004–2151–002 viiScienti?c Libraries User’s Guide Page Procedure 3: ASIN(x) . . . . . . . . . . . . . . . . . . . . . . 135 Procedure 4: ACOS(x) . . . . . . . . . . . . . . . . . . . . . . 136 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 138 Single-precision Real ATAN(x) Function . . . . . . . . . . . . . . . . . . 138 Procedure 5: ATAN(x) . . . . . . . . . . . . . . . . . . . . . . 138 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Single-precision ATAN(y,x) Function . . . . . . . . . . . . . . . . . . . 139 Procedure 6: ATAN(y,x) . . . . . . . . . . . . . . . . . . . . . 139 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Single-precision Real CBRT(x) Function . . . . . . . . . . . . . . . . . . 143 Procedure 7: CBRT(x) . . . . . . . . . . . . . . . . . . . . . . 143 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 145 Single-precision Real Exponential Function E x . . . . . . . . . . . . . . . 145 Procedure 8: e x . . . . . . . . . . . . . . . . . . . . . . . . 145 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Single-precision Real Power Function x y . . . . . . . . . . . . . . . . . 147 Procedure 9: x y . . . . . . . . . . . . . . . . . . . . . . . . 147 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 150 Single-precision Real SIN(x) and COS(x) Functions . . . . . . . . . . . . . . 151 Procedure 10: SIN(x) and COS(x) . . . . . . . . . . . . . . . . . . 151 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Single-precision Real COSH(x) and SINH(x) Functions . . . . . . . . . . . . . 155 Procedure 11: COSH(x) and SINH(x) . . . . . . . . . . . . . . . . . 155 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 156 Single-precision Real SQRT(x) Function . . . . . . . . . . . . . . . . . . 156 Procedure 12: SQRT(x) . . . . . . . . . . . . . . . . . . . . . 156 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 158 Single-precision TAN(x) and COT(x) Functions . . . . . . . . . . . . . . . . 158 viii 004–2151–002Contents Page Procedure 13: TAN(x) and COT(x) . . . . . . . . . . . . . . . . . . 158 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 161 Single-precision Real TANH(x) Function . . . . . . . . . . . . . . . . . . 162 Procedure 14: TANH(x) . . . . . . . . . . . . . . . . . . . . . 162 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Glossary 165 Index 171 Figures Figure 1. Pipelining in add operation . . . . . . . . . . . . . . . . . . 4 Figure 2. Pipelining and chaining . . . . . . . . . . . . . . . . . . . 4 Figure 3. Cost/robustness: general symmetric sparse solvers . . . . . . . . . . 63 Figure 4. Cost/robustness: general unsymmetric sparse solvers . . . . . . . . . 64 Figure 5. In-memory to virtual matrix copy . . . . . . . . . . . . . . . . 110 Figure 6. Layered software design . . . . . . . . . . . . . . . . . . . 112 Tables Table 1. Relative cost for SGER . . . . . . . . . . . . . . . . . . . . 23 Table 2. Factorization forms . . . . . . . . . . . . . . . . . . . . 30 Table 3. Solves times: LAPACK and solver routines . . . . . . . . . . . . . 35 Table 4. Veri?cation tests for LAPACK (all should be O(1)) . . . . . . . . . . . 50 Table 5. Summary of tridiagonal solvers . . . . . . . . . . . . . . . . 61 Table 6. SITRSOL argument summary . . . . . . . . . . . . . . . . . 72 Table 7. iparam summary . . . . . . . . . . . . . . . . . . . . . 73 Table 8. rparam summary . . . . . . . . . . . . . . . . . . . . . 74 Table 9. Summary of out-of-core routines for linear algebra . . . . . . . . . . . 107 004–2151–002 ixAbout This Guide This publication describes the Scienti?c Libraries (libsci) which run on UNICOS systems. The information in this manual supplements the man pages provided with the Scienti?c Library. This document is a user’s guide for programmers. Readers should have a working knowledge of the UNICOS operating system, have an understanding of the Fortran programming language, and have a working familiarity with scienti?c and mathematical theories. Related publications The following publications provide information related to the Scienti?c Library: • UNICOS User Commands Reference Manual • UNICOS System Libraries Reference Manual • Scienti?c Library Reference Manual • Optimizing Application Code on UNICOS Systems • CF90 Commands and Directives Reference Manual • Fortran Language Reference Manual, Volume 1 • Fortran Language Reference Manual, Volume 2 • Fortran Language Reference Manual, Volume 3 • LINPACK User’s Guide • LAPACK User’s Guide The following publications provide detailed information about the topics discussed in this manual. In many cases, these documents are referenced speci?cally in this manual. • Anderson, E., Z. Bai, C. Bischof, J. Demmel, J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, A. McKenney, S. Ostrouchov, and D. Sorensen. LAPACK User’s Guide. Philadelphia SIAM, 1992. 004–2151–002 xiScienti?c Libraries User’s Guide • Anderson, Edward, Jack Dongarra, and Susan Ostrouchov. Installation guide for LAPACK. LAPACK Working Note 41, Technical Report CS-91-138. University of Tennessee (Feb. 1992). • Argham, Nicolas J. Accuracy and Stability of Numeric Algorithms. Philadelphia SIAM, 1996. • Arioli, M., J. W. Demmel, and I. S. Duff. Solving sparse linear systems with sparse backward error. SIAM J. Matrix Anal. Appl. 10 (1989). • Ashcraft, Cleve. A vector implementation of the multifrontal method for large sparse, symmetric positive de?nite linear systems. Technical Report ETA-TR-51. Boeing Computer Services, 1987. • Duff, I. S., A. M. Erisman, and J. K. Reid. Direct Methods for Sparse Matrices. Monographs on Numerical Analysis. New York: Oxford University Press, 1986. • Duff, Iain, Michele Marrone, and Giuseppe Radicati. A proposal for user-level sparse BLAS. Technical Report TR/PA/92/85. CERFACS (Dec. 1992). • George, Alan and Joseph W-H Liu. Computer Solution of Large Sparse Positive De?nite Systems. Prentice-Hall Series in Computational Mathematics. Englewood Cliffs, NJ: Prentice-Hall, Inc., 1981. • Golub, Gene and James M. Ortega. Scienti?c Computing: An Introduction with Parallel Computing. Boston: Academic Press, 1993. • Golub, Gene H. and Charles F. Van Loan. Matrix Computations. 2nd edition. Baltimore, Maryland: Johns Hopkins University Press, 1989. • Hageman, Louis A. and David M. Young. Applied Iterative Methods. Computer Science and Applied Mathematics. New York and London: Academic Press, 1981. • Heroux, Michael A. A reverse communication interface for ‘‘matrix-free’’ preconditioned iterative solvers. Edited by C.A. Brebbia, D. Howard, and A. Peters In Applications of Supercomputers in Engineering II, 207-213. Boston: Computational Mechanics Publications, 1991. • Heroux, Michael A. A proposal for a sparse BLAS toolkit. Technical Report TR/PA/92/90. CERFACS (Dec. 1992). • Heroux, Michael A., Phuong Vu, and Chao Wu Yang. A parallel preconditioned conjugate gradient package for solving sparse linear systems on a Cray Y-MP. Applied Numerical Mathematics, 8 (1991). xii 004–2151–002About This Guide • Hestenes, M. R. and E. Stiefel. Methods of conjugate gradients for solving linear systems. J. Res. National Bureau of Standards 49 (1952): 409-436. • Kincaid, David R., Thomas C. Oppe, John R. Respess, and David M. Young. ITPACKV 2C User’s Guide. Technical Report CNA-191. The University of Texas at Austin: Center for Numerical Analysis, (Nov. 1984). • Manteuffel, T. A. An incomplete factorization technique for positive de?nite linear systems. Math. Comp. 34 (1980): 473-497. • Oppe, Thomas C., Wayne D. Joubert, and David R. Kincaid. NSPCG User’s Guide. The University of Texas at Austin: Center for Numerical Analysis, (Dec. 1988). • Reid, J. K., editor. On the Method of Conjugate Gradients for the Solution of Large Sparse Systems of Linear Equations. Large Sparse Sets of Linear Equations, Academic Press, 1971. • Saad, Youcef. SPARSKIT: a basic tool kit for sparse matrix computations. Preliminary Version. • Saad, Youcef. Practical use of polynomial preconditionings for the conjugate gradient method., 6(4) (Oct. 1985): 865-881. • Saad, Youcef and Martin H. Schultz. GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM Journal of Scienti?c and Statistical Computing, 7(3) (Jul. 1986): 856-869. • Sonneveld, Peter. CGS, a fast lanczos-type solver for nonsymmetric linear systems. SIAM Journal of Scienti?c and Statistical Computing, 10(1) (Jan. 1989): 36-52. • Stewart, G. W. Introduction to Matrix Computations. Orlando, Florida: Academic Press, 1973. • Wilkinson, J. H. The Algebraic Eigenvalue Problem. Oxford, England: Oxford University Press, 1965. • Yang, Chao W. A parallel multifrontal method for sparse symmetric de?nite linear systems on the Cray Y-MP. Proceedings of the Fifth SIAM Conference on Parallel Processing for Scienti?c Computing. Houston, Texas (Apr. 1992). You can ?nd a good general reference on the solution of sparse linear systems in Golub and Van Loan. You can ?nd a good introduction to direct and iterative methods, as well as methods for special linear systems, in these texts. See the special section of the November 1989 issue of the SIAM Journal of Scienti?c and Statistical Computing, pages 1135-1232 for an updated general reference. 004–2151–002 xiiiScienti?c Libraries User’s Guide See George and Liu, Duff and Erisman, and Reid for classical references that give a thorough and in-depth treatment of sparse direct solvers. Another common reference is Ashcraft. The original conjugate gradient algorithm was presented in Hestenes and Stiefel; however, Reid presented the ?rst practical application. A classical text in iterative methods is that of Hageman and Young. You can ?nd good discussions of the biconjugate gradient and biconjugate gradient squared methods in Sonneveld. GMRES is presented by Saad and Schultz. You can ?nd some references on data structures in SPARSKIT and in the proposals for sparse BLAS. Three articles that deal directly with the Cray Research libsci sparse solvers are Yang on direct solvers, Heroux (1991), and Heroux, Vu, and Yang on SITRSOL. Conventions The following conventions are used throughout this documentation: command This ?xed-space font denotes literal items, such as pathnames, man page names, commands, and programming language structures. variable Italic typeface denotes variable entries and words or concepts being de?ned. [ ] Brackets enclose optional portions of a command line. In addition to these formatting conventions, several naming conventions are used throughout the documentation. “Cray PVP systems” denotes all con?gurations of Cray parallel vector processing (PVP) systems which run the UNICOS operating system. “Cray MPP systems” denotes all con?gurations of the Cray T3E series which runs the UNICOS/mk operating system. “ IRIX systems” denotes SGI platforms which run the IRIX operating system. The default shell in the UNICOS and UNICOS/mk operating systems, referred to as the standard shell, is a version of the Korn shell that conforms to the following standards: • Institute of Electrical and Electronics Engineers (IEEE) Portable Operating System Interface (POSIX) Standard 1003.2–1992 • X/Open Portability Guide, Issue 4 (XPG4) xiv 004–2151–002About This Guide The UNICOS and UNICOS/mk operating systems also support the optional use of the C shell. Cray UNICOS Version 10.0 is an X/Open Base 95 branded product. Obtaining Publications The User Publications Catalog describes the availability and content of all Cray Research hardware and software documents that are available to customers. Customers who subscribe to the Cray Inform (CRInform) program can access this information on the CRInform system. To order a document, call +1 651 683 5907. SGI employees may send electronic mail to orderdsk@sgi.com (UNIX system users). Customers who subscribe to the CRInform program can order software release packages electronically by using the Order Cray Software option. Customers outside of the United States and Canada should contact their local service organization for ordering and documentation information. Reader Comments If you have comments about the technical accuracy, content, or organization of this document, please tell us. Be sure to include the title and part number of the document with your comments. You can contact us in any of the following ways: • Send e-mail to the following address: techpubs@sgi.com • Send a fax to the attention of “Technical Publications” at: +1 650 932 0801. • Use the Feedback option on the Technical Publications Library World Wide Web page: http://techpubs.sgi.com • Call the Technical Publications Group, through the Technical Assistance Center, using one of the following numbers: For SGI IRIX based operating systems: 1 800 800 4SGI 004–2151–002 xvScienti?c Libraries User’s Guide For UNICOS or UNICOS/mk based operating systems or Cray Origin 2000 systems: 1 800 950 2729 (toll free from the United States and Canada) or +1 651 683 5600 • Send mail to the following address: Technical Publications SGI 1600 Amphitheatre Pkwy. Mountain View, California 94043–1351 We value your comments and will respond to them promptly. xvi 004–2151–002Introduction [1] This manual describes the Scienti?c Libraries which run on UNICOS systems. The information in this manual supplements the man pages provided with the Scienti?c Library and provides details about the implementation and usage of these library routines on UNICOS systems. This manual includes the following sections: • Chapter 2, page 3, discusses parallel processing environments, ways to measure parallel processing performance, and the implementation strategies used in the Scienti?c Library routines. • Chapter 3, page 25, discusses dense linear algebra problems, including systems of linear equations, linear least squares problems, eigenvalue problems, and singular value problems. • Chapter 4, page 53, discusses sparse matrices and solution techniques for sparse linear systems. • Chapter 5, page 101, discusses the out-of-core routines, virtual matrices, and subroutines used with out-of-core routines. • Appendix A, page 123, discusses libm Version 2, the default UNICOS math library. • Appendix B, page 129, discusses the algorithms used in libm. 004–2151–002 1Parallel Processing Issues [2] Parallel processing is a method of splitting a computational task into subtasks, and then simultaneously performing the subtasks. This section discusses the different ways parallel processing strategies used on UNICOS systems, and what effects those implementations have on the performance of your code. 2.1 Parallel Processing Overview Parallel processing is performed at the hardware level, the operating system level, and the code level. This subsection brie?y discusses these different types of parallel processing. 2.1.1 Parallel Processing: Hardware Level At the hardware level, parallel processing is accomplished by using the following methods: • parallel instruction execution, which is the execution of one instruction per clock period, even those instructions that take several clock periods to complete execution. • vectorization, which is a form of parallel processing that uses instruction segmenting and vector registers. • I/O subsystems or foreground processors, which is the execution of operations in parallel with processes running in the main processors that perform I/O. • time slicing, in which the system works on several jobs or processes simultaneously. • pipelining, which allows each step of an operation to pass its result to the next step after only one clock period (see Figure 1). • chaining, which allows the movement of elements to continue from one vector operation to another, so that a process including more than one vector operation is executed as one long vectorized operation (see Figure 2). 004–2151–002 3Scienti?c Libraries User’s Guide A B Operands 1 2 3 4 5 6 7 Clock periods C Result a10035 Figure 1. Pipelining in add operation Chaining Movement of array elements Pipelining Vector load 1 CP For illustration only. Functional units are not physically arranged as shown. Multiply Vector store a10036 Figure 2. Pipelining and chaining 2.1.2 Parallel Processing: Operating System Level At the operating system level, parallel processing is accomplished by using the following methods: • multiprogramming, which occurs when a processor switches between the jobs or processes in the system. • multiprocessing, which occurs when processors simultaneously work on as many programs as there are processors. 4 004–2151–002Parallel Processing Issues [2] • multitasking, which is a general term describing more than one processor working to complete a single program. This term has become synonymous with parallel processing. 2.1.3 Parallel Processing: Code Level There are several ways to alter user code to take advantage of parallel processing: • macrotasking lets you take advantage of multitasking at the subroutine level by inserting library calls to express parallelism. This works well on programs that can be explicitly partitioned into tasks and can be simultaneously executed on multiple processors. • microtasking allows you to insert directives at the loop or block level of the code to indicate sections that can be executed on multiple CPUs. Macrotasking and microtasking are seldom used anymore. Autotasking is the preferred method to use for parallel processing at the code level. Like microtasking, Autotasking exploits parallelism at the loop or block level of code. It has lower overhead than microtasking, and it can be made fully automatic. It does not require any direct user intervention, but users can interact with the system via directives and switches. The Scienti?c Library routines were designed to be fully optimized to take advantage of parallelism and to automatically use Autotasking during execution. When using the CF90 compiler, you can use the f90 -O3 command. This command invokes aggressive optimization and Autotasking. One of the optimization actions the compiler then performs is to substitute Scienti?c Library routines (where appropriate) in the code. The routines are called at an entry point where they will use less compiling time. See the CF90 Commands and Directives Reference Manual or the f90(1) man page for more information about using the f90 command. 2.2 Costs and Bene?ts of Parallel Processing Parallel processing can eliminate idle CPU time because the workload is divided among all CPUs; therefore, the amount of work performed per unit time (the throughput) increases. However, parallel processing also introduces some overhead into program execution. In some cases, you may be able to 004–2151–002 5Scienti?c Libraries User’s Guide reduce wall-clock time, but at the cost of extra CPU time which increases because more machine resources are used. This subsection discusses these bene?ts and some of the costs of using parallel processing. 2.2.1 Bene?ts of Parallel Processing By using parallel processing, you can alleviate some of the following common problems: • Maximum-memory jobs: if the memory is occupied by a few large-memory jobs, one or more of the CPUs might be idle even though there are other jobs to run. • Dedicated machine: if the computer is running a single job, then all other CPUs are idle. • Light workload: if the amount of jobs waiting for a CPU is less than the total number of CPUs, then one or more of the CPUs becomes idle. With parallel processing, the additional CPUs reduce the wall-clock time instead of sitting idle. Even when very little idle time exists, using additional CPUs can still lead to bene?ts. 2.2.2 Parallel Processing Overhead Parallel processing introduces some overhead into program execution. This subsection discusses some of the common types of overhead introduced by parallel processing: • Multitasked programs require more memory than unitasked programs, and they can contain more code, more temporary variables, and can require additional stack space. • Multitasked jobs can be swapped more often, and remain swapped longer, on a heavily loaded production system. • Processors are forced to wait on semaphores during the process of synchronization. • Overhead is incurred when slave processors are acquired (on entry to a parallel region) and at synchronization points within parallel regions. Tests show that the overhead of executing extra Autotasking code adds a nominal 0% to 5% to the overall execution time. 6 004–2151–002Parallel Processing Issues [2] • If inner-loop Autotasking is used, vector performance can decrease because of shorter vector lengths and more vector loop startups. • Processors are sometimes held for the next parallel region to improve ef?ciency. While holding a processor can save time, it also costs time to acquire and hold them. Because overhead is associated with work distribution, jobs with large granularity have less partitioning than smaller jobs. Large jobs, however, may have problems with load balancing. 2.2.3 Parallelism and Vectorization A multitasked program usually has the same amount of work for the CPUs to perform as does a similar unitasked program. However, when the work is spread across many processors, the wall-clock time required to complete the work is usually less. Vectorization is a form of parallel processing in which array elements are processed by groups. Vectorization usually decreases both CPU time and wall-clock time; multitasking decreases only wall-clock time. Thus, vectorization can give you large gains in terms of saving time and have relatively low costs associated with it. If you make direct calls to Scienti?c Library routines, your CPU time or wall-clock time could also be affected. Scienti?c Library routines take advantage of vectorization and multitasking; because of the costs associated with multitasking, you may see an increase in CPU time when using these routines. You can control this increase somewhat by using the NCPUS environment variable to control the maximum number of CPUs to be used on a job. See Section 2.4.1, page 12, for details about using environment variables. When using multitasking, the timing results in a nondedicated environment are not always reproducible. Several factors can affect the timings obtained from multitasked routines, including some of the following: • System load • I/O performed by other jobs • Memory contention If accurate timing results are desired, it is best to run a job in a dedicated environment where it is the only job being run. 004–2151–002 7Scienti?c Libraries User’s Guide 2.2.4 Parallelism and Load Balancing The parallelism for any region is determined by the number of partitions, or chunks, of independent work the region contains. If an autotasked loop has N iterations, N is the extent of parallelism for that loop. This means that the loop can effectively be broken into N independent chunks of work. For autotasked inner loops, the extent of parallelism is the number of iterations divided by the vector length when both Autotasking and vectorization can be used. Load balancing is the process of dividing work done by each available processor into approximately equal amounts. In a multiuser environment, the number of available processors is constantly changing. When Autotasking is used, it automatically tries to perform dynamic load balancing by creating small-granularity parallelism. For example, if a DO loop is autotasked, work is allocated by default to each task one iteration at a time. A direct relationship exists between load balancing and the extent of parallelism: • The higher the extent of parallelism, the easier it is to balance the workload evenly across the processors. • Small-granularity parallelism is easier to balance across available processors than large granularity parallelism. • Small-granularity parallelism generates more overhead than large granularity parallelism. Synchronization is required each time a chunk of work is allocated to a processor. 2.3 Performance Issues This subsection discusses the following different aspects used to de?ne the performance of parallel processing: speedup, ef?ciency, and percentage of parallelism in a program. 2.3.1 Calculating Speedups in Multitasked Programs On a dedicated (nonproduction) system, the speedup ratio for a multitasked program can be calculated in the following manner, where T1 is the wall-clock execution time on a single-processor and Tp is the wall-clock execution time on p processors: Speedup ratio = T1 Tp 8 004–2151–002Parallel Processing Issues [2] (2.1) With p CPUs, a speedup ratio as close as possible to p is desired. If the program were completely parallel and no overhead existed, the speedup would be equal to p (for details about the overhead associated with multitasking, see Section 2.2.2, page 6). For example, suppose a job takes 8.7 seconds to run on a single processor. When the job is rerun using four processors, the execution time decreases to 2.5 seconds; the speedup is the following: s = 8:7 2:5 = 3:48 (2.2) The speedup ratio of multitasked code has two limitations: Amdahl’s Law (representing the speedup related to the sequential portion of the code), and multitasking overhead (discussed in Section 2.2.2, page 6). 2.3.1.1 Amdahl’s Law Some sections of programs, known as single-threaded code segments, must use a single processor. Amdahl’s Law for parallel processing illustrates the effect of single-threaded code segments on multitasking performance. Amdahl’s Law is shown in the following equation: s = 1 (1 f) + f p (2.3) The following components appear in this equation: • s: Maximum expected speedup from multitasking • p: Number of processors available for parallel execution • f: Fraction of a program that can execute in parallel (the parallel fraction) For example, suppose that code is 98% parallel, implying that 2% of the code runs in serial mode. Suppose that 64 processors are available. According to Amdahl’s Law, the maximum speedup that you can expect is: 004–2151–002 9 PGI ® User’s Guide Parallel Fortran, C and C++ for Scientists and Engineers The Portland Group™ STMicroelectronics Two Centerpointe Drive Lake Oswego, OR 97035 While every precaution has been taken in the preparation of this document, The Portland Group™, a wholly-owned subsidiary of STMicroelectronics, makes no warranty for the use of its products and assumes no responsibility for any errors that may appear, or for damages resulting from the use of the information contained herein. The Portland Group retains the right to make changes to this information at any time, without notice. The software described in this document is distributed under license from STMicroelectronics and may be used or copied only in accordance with the terms of the license agreement. No part of this document may be reproduced or transmitted in any form or by any means, for any purpose other than the purchaser's personal use without the express written permission of The Portland Group. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this manual, The Portland Group was aware of a trademark claim. The designations have been printed in caps or initial caps. Thanks is given to the Parallel Tools Consortium and, in particular, to the High Performance Debugging Forum for their efforts. PGF95, PGF90, PGC++, Cluster Development Kit, CDK and The Portland Group are trademarks and PGI, PGHPF, PGF77, PGCC, PGPROF, and PGDBG are registered trademarks of STMicroelectronics, Inc. Other brands and names are the property of their respective owners. The use of STLport, a C++ Library, is licensed separately and license, distribution and copyright notice can be found in the online documentation for a given release of the PGI compilers and tools. PGI User's Guide Copyright © 1998 – 2000, The Portland Group, Inc. Copyright © 2000 – 2005, STMicroelectronics, Inc. All rights reserved. Printed in the United States of America Part Number: 2030-990-888-0603 First Printing: Release 1.7, Jun 1998 Sixth Printing: Release 5.0, Jun 2003 Second Printing: Release 3.0, Jan 1999 Seventh Printing: Release 5.1, Nov 2003 Third Printing: Release 3.1, Sep 1999 Eight Printing: Release 5.2, Jun 2004 Fourth Printing: Release 3.2, Sep 2000 Ninth Printing: Release 6.0, Mar 2005 Fifth Printing: Release 4.0, May 2002 Tenth Printing: Release 6.1, Dec 2005 Technical support: trs@pgroup.com Sales: sales@pgroup.com Web: http://www.pgroup.com/Table of Contents 3 Table of Contents PREFACE......................................................................................................................................13 AUDIENCE DESCRIPTION..............................................................................................................13 COMPATIBILITY AND CONFORMANCE TO STANDARDS.................................................................13 ORGANIZATION............................................................................................................................14 HARDWARE AND SOFTWARE CONSTRAINTS ................................................................................15 CONVENTIONS .............................................................................................................................16 RELATED PUBLICATIONS .............................................................................................................19 GETTING STARTED ..................................................................................................................21 1.1 OVERVIEW .............................................................................................................................21 1.2 INVOKING THE COMMAND-LEVEL PGI COMPILERS................................................................22 1.2.1 Command-line Syntax....................................................................................................22 1.2.2 Command-line Options ..................................................................................................23 1.2.3 Fortran Directives and C/C++ Pragmas ........................................................................23 1.3 FILENAME CONVENTIONS ......................................................................................................24 1.3.1 Input Files ......................................................................................................................24 1.3.2 Output Files....................................................................................................................25 1.4 PARALLEL PROGRAMMING USING THE PGI COMPILERS ........................................................27 1.4.1 Running SMP Parallel Programs....................................................................................28 1.4.2 Running Data Parallel HPF Programs............................................................................28 1.5 USING THE PGI COMPILERS ON LINUX...................................................................................29 1.5.1 Linux Header Files.........................................................................................................294 Table of Contents 1.5.2 Running Parallel Programs on Linux............................................................................. 29 1.6 USING THE PGI COMPILERS ON WINDOWS ............................................................................ 30 OPTIMIZATION & PARALLELIZATION ............................................................................. 33 2.1 OVERVIEW OF OPTIMIZATION................................................................................................ 33 2.2 GETTING STARTED WITH OPTIMIZATIONS.............................................................................. 35 2.3 LOCAL AND GLOBAL OPTIMIZATION USING -O .................................................................... 37 2.3.1 Scalar SSE Code Generation.......................................................................................... 39 2.4 LOOP UNROLLING USING -MUNROLL ................................................................................... 40 2.5 VECTORIZATION USING -MVECT .......................................................................................... 41 2.5.1 Vectorization Sub-options.............................................................................................. 41 2.5.1.1 Assoc Option...........................................................................................................42 2.5.1.2 Cachesize Option .................................................................................................... 42 2.5.1.3 SSE Option ............................................................................................................. 43 2.5.1.4 Prefetch Option ....................................................................................................... 43 2.5.2 Vectorization Example Using SSE/SSE2 Instructions................................................... 44 2.6 AUTO-PARALLELIZATION USING -MCONCUR ....................................................................... 47 2.6.1 Auto-parallelization Sub-options ................................................................................... 47 2.6.1.1 Altcode Option........................................................................................................ 47 2.6.1.2 Dist Option..............................................................................................................48 2.6.1.3 Cncall Option ..........................................................................................................48 2.6.2 Loops That Fail to Parallelize ........................................................................................ 48 2.6.2.1 Innermost Loops ..................................................................................................... 49 2.6.2.2 Timing Loops.......................................................................................................... 49 2.6.2.3 Scalars..................................................................................................................... 49 2.6.2.4 Scalar Last Values................................................................................................... 50Table of Contents 5 2.7 INTER-PROCEDURAL ANALYSIS AND OPTIMIZATION USING –MIPA .......................................52 2.7.1 Building a Program Without IPA – Single Step.............................................................52 2.7.2 Building a Program Without IPA - Several Steps ..........................................................52 2.7.3 Building a Program Without IPA Using Make ..............................................................53 2.7.4 Building a Program with IPA.........................................................................................53 2.7.5 Building a Program with IPA - Single Step ...................................................................54 2.7.6 Building a Program with IPA - Several Steps ................................................................55 2.7.7 Building a Program with IPA Using Make ....................................................................56 2.7.8 Questions about IPA ......................................................................................................56 2.8 PROFILE-FEEDBACK OPTIMIZATION USING –MPFI/–MPFO.....................................................58 2.9 DEFAULT OPTIMIZATION LEVELS ..........................................................................................58 2.10 LOCAL OPTIMIZATION USING DIRECTIVES AND PRAGMAS ..................................................59 2.11 EXECUTION TIMING AND INSTRUCTION COUNTING .............................................................59 COMMAND LINE OPTIONS.....................................................................................................61 3.1 GENERIC PGI COMPILER OPTIONS .........................................................................................67 3.2 C AND C++ -SPECIFIC COMPILER OPTIONS..........................................................................122 FUNCTION INLINING .............................................................................................................129 4.1 INVOKING FUNCTION INLINING............................................................................................129 4.1.1 Using an Inline Library ................................................................................................130 4.2 CREATING AN INLINE LIBRARY............................................................................................131 4.2.1 Working with Inline Libraries......................................................................................131 4.2.2 Updating Inline Libraries - Makefiles ..........................................................................132 4.3 ERROR DETECTION DURING INLINING..................................................................................133 4.4 EXAMPLES ...........................................................................................................................1336 Table of Contents 4.5 RESTRICTIONS ON INLINING................................................................................................. 133 OPENMP DIRECTIVES FOR FORTRAN ............................................................................. 135 5.1 PARALLELIZATION DIRECTIVES ........................................................................................... 135 5.2 PARALLEL ... END PARALLEL ...................................................................................... 136 5.3 CRITICAL ... END CRITICAL .......................................................................................... 139 5.4 MASTER ... END MASTER............................................................................................... 140 5.5 SINGLE ... END SINGLE................................................................................................... 141 5.6 DO ... END DO ................................................................................................................... 142 5.7 WORKSHARE ... END WORKSHARE............................................................................. 145 5.8 BARRIER ........................................................................................................................... 145 5.9 DOACROSS ....................................................................................................................... 146 5.10 PARALLEL DO................................................................................................................ 147 5.11 PARALLEL WORKSHARE ............................................................................................ 147 5.12 SECTIONS…END SECTIONS ...................................................................................... 148 5.13 PARALLEL SECTIONS .................................................................................................. 149 5.14 ORDERED........................................................................................................................ 150 5.15 ATOMIC........................................................................................................................... 150 5.16 FLUSH.............................................................................................................................. 151 5.17 THREADPRIVATE.......................................................................................................... 151 5.18 RUN-TIME LIBRARY ROUTINES.......................................................................................... 151 5.19 ENVIRONMENT VARIABLES................................................................................................ 154 OPENMP PRAGMAS FOR C AND C++ ................................................................................. 155 6.1 PARALLELIZATION PRAGMAS .............................................................................................. 155 6.2 OMP PARALLEL..................................................................................................................... 156Table of Contents 7 6.3 OMP CRITICAL ......................................................................................................................159 6.4 OMP MASTER ........................................................................................................................160 6.5 OMP SINGLE..........................................................................................................................160 6.6 OMP FOR...............................................................................................................................161 6.7 OMP BARRIER .......................................................................................................................164 6.8 OMP PARALLEL FOR..............................................................................................................164 6.9 OMP SECTIONS......................................................................................................................165 6.10 OMP PARALLEL SECTIONS...................................................................................................165 6.11 OMP ORDERED ....................................................................................................................166 6.12 OMP ATOMIC ......................................................................................................................166 6.13 OMP FLUSH.........................................................................................................................167 6.14 OMP THREADPRIVATE.........................................................................................................167 6.15 RUN-TIME LIBRARY ROUTINES ..........................................................................................168 6.16 ENVIRONMENT VARIABLES................................................................................................171 OPTIMIZATION DIRECTIVES AND PRAGMAS ...............................................................173 7.1 ADDING DIRECTIVES TO FORTRAN.......................................................................................173 7.2 FORTRAN DIRECTIVE SUMMARY..........................................................................................174 7.3 SCOPE OF DIRECTIVES AND COMMAND LINE OPTIONS .........................................................180 7.4 ADDING PRAGMAS TO C AND C++......................................................................................181 7.5 C/C++ PRAGMA SUMMARY ................................................................................................182 7.6 SCOPE OF C/C++ PRAGMAS AND COMMAND LINE OPTIONS ...............................................185 7.7 PREFETCH DIRECTIVES ........................................................................................................188 LIBRARIES AND ENVIRONMENT VARIABLES ...............................................................191 8.1 USING BUILTIN MATH FUNCTIONS IN C/C++.......................................................................1918 Table of Contents 8.2 CREATING AND USING SHARED OBJECT FILES ON LINUX.................................................... 191 8.3 CREATING AND USING DYNAMIC-LINK LIBRARIES ON WIN32 ............................................ 193 8.4 CREATING AND USING DYNAMIC-LINK LIBRARIES ON WIN64 ............................................ 198 8.5 USING LIB3F....................................................................................................................... 206 8.6 LAPACK, THE BLAS AND FFTS......................................................................................... 206 8.7 THE C++ STANDARD TEMPLATE LIBRARY ......................................................................... 207 8.8 ENVIRONMENT VARIABLES ................................................................................................. 207 FORTRAN, C AND C++ DATA TYPES .................................................................................. 211 9.1 FORTRAN DATA TYPES ........................................................................................................ 211 9.1.1 Fortran Scalars ............................................................................................................. 211 9.1.2 FORTRAN 77 Aggregate Data Type Extensions ........................................................ 213 9.1.3 Fortran 90 Aggregate Data Types (Derived Types) ..................................................... 214 9.2 C AND C++ DATA TYPES .................................................................................................... 215 9.2.1 C and C++ Scalars ...................................................................................................... 215 9.2.2 C and C++ Aggregate Data Types .............................................................................. 217 9.2.3 Class and Object Data Layout...................................................................................... 217 9.2.4 Aggregate Alignment................................................................................................... 218 9.2.5 Bit-field Alignment...................................................................................................... 219 9.2.6 Other Type Keywords in C and C++ .......................................................................... 220 INTER-LANGUAGE CALLING.............................................................................................. 221 10.1 OVERVIEW OF CALLING CONVENTIONS ............................................................................. 221 10.2 INTER-LANGUAGE CALLING CONSIDERATIONS.................................................................. 221 10.3 FUNCTIONS AND SUBROUTINES ......................................................................................... 223 10.4 UPPER AND LOWER CASE CONVENTIONS, UNDERSCORES ................................................. 223Table of Contents 9 10.5 COMPATIBLE DATA TYPES.................................................................................................223 10.5.1 Fortran Named Common Blocks................................................................................225 10.6 ARGUMENT PASSING AND RETURN VALUES ......................................................................225 10.6.1 Passing by Value (%VAL)...........................................................................................226 10.6.2 Character Return Values ............................................................................................226 10.6.3 Complex Return Values .............................................................................................227 10.7 ARRAY INDICES .................................................................................................................227 10.8 EXAMPLE - FORTRAN CALLING C ......................................................................................228 10.9 EXAMPLE - C CALLING FORTRAN ......................................................................................229 10.10 EXAMPLE - C ++ CALLING C............................................................................................230 10.11 EXAMPLE - C CALLING C++ ............................................................................................231 10.12 EXAMPLE - FORTRAN CALLING C++...............................................................................232 10.13 EXAMPLE - C++ CALLING FORTRAN ...............................................................................234 10.14 WIN32 CALLING CONVENTIONS ......................................................................................235 10.14.1 Win32 Fortran Calling Conventions ........................................................................235 10.14.2 Symbol Name Construction and Calling Example...................................................237 10.14.3 Using the Default Calling Convention .....................................................................238 10.14.4 Using the STDCALL Calling Convention ...............................................................238 10.14.5 Using the C Calling Convention ..............................................................................239 10.14.6 Using the UNIX Calling Convention .......................................................................239 C++ NAME MANGLING ..........................................................................................................241 11.1 TYPES OF MANGLING.........................................................................................................242 11.2 MANGLING SUMMARY .......................................................................................................243 11.2.1 Type Name Mangling ................................................................................................243 11.2.2 Nested Class Name Mangling ....................................................................................243 11.2.3 Local Class Name Mangling ......................................................................................24310 Table of Contents 11.2.4 Template Class Name Mangling................................................................................ 244 RUN-TIME ENVIRONMENT.................................................................................................. 245 A1 LINUX86 AND WIN32 PROGRAMMING MODEL..................................................................... 245 A1.1 Function Calling Sequence .......................................................................................... 245 A1.2 Function Return Values ............................................................................................... 248 A1.3 Argument Passing ........................................................................................................ 250 A2 LINUX86-64 PROGRAMMING MODEL................................................................................... 253 A2.1 Function Calling Sequence .......................................................................................... 253 A2.2 Function Return Values ............................................................................................... 256 A2.3 Argument Passing ........................................................................................................ 257 A2.4 Linux86-64 Fortran Supplement.................................................................................. 262 A2.4.1 Fortran Fundamental Types .................................................................................. 262 A2.4.2 Naming Conventions............................................................................................. 263 A2.4.3 Argument Passing and Return Conventions.......................................................... 263 A2.4.4 Inter-language Calling........................................................................................... 264 A3 WIN64 PROGRAMMING MODEL ........................................................................................... 267 A3.1 Function Calling Sequence .......................................................................................... 267 A3.2 Function Return Values ............................................................................................... 269 A3.3 Argument Passing ........................................................................................................ 270 A3.4 Win64 Fortran Supplement.......................................................................................... 273 A3.4.1 Fortran Fundamental Types .................................................................................. 274 A3.4.2 Fortran Naming Conventions ................................................................................ 275 A3.4.3 Fortran Argument Passing and Return Conventions ............................................. 275 A3.4.4 Interlanguage Calling ............................................................................................ 275Table of Contents 11 MESSAGES.................................................................................................................................279 B.1 DIAGNOSTIC MESSAGES ......................................................................................................279 B.2 PHASE INVOCATION MESSAGES ..........................................................................................280 B.3 FORTRAN COMPILER ERROR MESSAGES .............................................................................280 B.3.1 Message Format...........................................................................................................280 B.3.2 Message List................................................................................................................280 B.4 FORTRAN RUNTIME ERROR MESSAGES ...............................................................................316 B.4.1 Message Format...........................................................................................................316 B.4.2 Message List................................................................................................................317 C++ DIALECT SUPPORTED ...................................................................................................321 C.1 ANACHRONISMS ACCEPTED ................................................................................................321 C.2 NEW LANGUAGE FEATURES ACCEPTED ..............................................................................323 C.3 THE FOLLOWING LANGUAGE FEATURES ARE NOT ACCEPTED...............................................325 C.4 EXTENSIONS ACCEPTED IN NORMAL C++ MODE ...............................................................325 C.5 CFRONT 2.1 COMPATIBILITY MODE .....................................................................................326 C.6 CFRONT 2.1/3.0 COMPATIBILITY MODE ...............................................................................329 INDEX..........................................................................................................................................33112 Table of ContentsPreface 13 Preface This guide is part of a set of manuals that describe how to use The Portland Group (PGI) Fortran, C, and C++ compilers and program development tools. In particular, these include the PGF77, PGF95, PGHPF, PGC++, and PGCC ANSI C compilers, the PGPROF profiler, and the PGDBG debugger. These compilers and tools work in conjunction with an x86 or x64 assembler and linker. You can use the PGI compilers and tools to compile, debug, optimize and profile serial and parallel applications for x86 (Intel Pentium II/III/4/M, Intel Centrino, Intel Xeon, AMD Athlon XP/MP) or x64 (AMD Athlon64/Opteron/Turion, Intel EM64T) processor-based systems. This PGI User's Guide provides operating instructions for the command-level compilation environment and general information about PGI’s implementation of the Fortran, C, and C++ languages. This guide does not teach the Fortran, C, or C++ programming languages. Audience Description This guide is intended for scientists and engineers using the PGI compilers. To use these compilers, you should be aware of the role of high-level languages (e.g. Fortran, C, C++) and assembly-language in the software development process and should have some level of understanding of programming. The PGI compilers are available on a variety of x86 or x64 hardware platforms and operating systems. You need to be familiar with the basic commands available on your system. Finally, your system needs to be running a properly installed and configured version of the compilers. For information on installing PGI compilers and tools, refer to the installation instructions. Compatibility and Conformance to Standards For further information, refer to the following: • American National Standard Programming Language FORTRAN, ANSI X3. -1978 (1978). • ISO/IEC 1539 : 1991, Information technology – Programming Languages – Fortran, Geneva, 1991 (Fortran 90). • ISO/IEC 1539 : 1997, Information technology – Programming Languages – Fortran, Geneva, 1997 (Fortran 95). • Fortran 95 Handbook Complete ISO/ANSI Reference, Adams et al, The MIT Press, Cambridge, Mass, 1997. 14 Preface • High Performance Fortran Language Specification, Revision 1.0, Rice University, Houston, Texas (1993), http://www.crpc.rice.edu/HPFF. • High Performance Fortran Language Specification, Revision 2.0, Rice University, Houston, Texas (1997), http://www.crpc.rice.edu/HPFF. • OpenMP Fortran Application Program Interface, Version 1.1, November 1999, http://www.openmp.org. • OpenMP C and C++ Application Program Interface, Version 1.0, October 1998, http://www.openmp.org. • Programming in VAX Fortran, Version 4.0, Digital Equipment Corporation (September, 1984). • IBM VS Fortran, IBM Corporation, Rev. GC26-4119. • Military Standard, Fortran, DOD Supplement to American National Standard Programming Language Fortran, ANSI x.3-1978, MIL-STD-1753 (November 9, 1978). • American National Standard Programming Language C, ANSI X3.159-1989. Organization This manual is divided into the following chapters and appendices: Chapter 1 Getting Started provides an introduction to the PGI compilers and describes their use and overall features. Chapter 2 Optimization & Parallelization describes standard optimization techniques that, with little effort, allow users to significantly improve the performance of programs. Chapter 3 Command Line Options provides a detailed description of each command-line option. Chapter 4 Function Inlining describes how to use function inlining and shows how to create an inline library. Chapter 5 OpenMP Directives for Fortran provides a description of the OpenMP Fortran parallelization directives and shows examples of their use. Chapter 6 OpenMP Pragmas for C and C++ provides a description of the OpenMP C and C++ parallelization pragmas and shows examples of their use. Preface 15 Chapter 7 Optimization Directives and Pragmas provides a description of each Fortran optimization directive and C/C++ optimization pragma, and shows examples of their use. Chapter 8 Libraries and Environment Variables discusses PGI support libraries, shared object files, and environment variables that affect the behavior of the PGI compilers. Chapter 9 Fortran, C and C++ Data Types describes the data types that are supported by the PGI Fortran, C, and C++ compilers. Chapter 10 Inter-language Calling provides examples showing how to place C Language calls in a Fortran program and Fortran Language calls in a C program. Chapter 11 C++ Name Mangling describes the name mangling facility and explains the transformations of names of entities to names that include information on aspects of the entity’s type and a fully qualified name. Appendix A Run-time Environment describes the assembly language calling conventions and examples of assembly language calls. Appendix B Messages provides a list of compiler error messages. Appendix C C++ Dialect Supported lists more details of the version of the C++ language that PGC++ supports. Hardware and Software Constraints This guide describes versions of the PGI compilers that produce assembly code for x86 and x64 processor-based systems. Details concerning environment-specific values and defaults and system-specific features or limitations are presented in the release notes sent with the PGI compilers. 16 Preface Conventions The PGI User's Guide uses the following conventions: italic is used for commands, filenames, directories, arguments, options and for emphasis. Constant Width is used in examples and for language statements in the text, including assembly language statements. [ item1 ] square brackets indicate optional items. In this case item1 is optional. { item2 | item 3} braces indicate that a selection is required. In this case, you must select either item2 or item3. filename ... ellipsis indicate a repetition. Zero or more of the preceding item may occur. In this example, multiple filenames are allowed. FORTRAN Fortran language statements are shown in the text of this guide using upper-case characters and a reduced point size. The PGI compilers and tools are supported on both 32-bit and 64-bit variants of the Linux and Windows operating systems on a variety of x86-compatible processors. There are a wide variety of releases and distributions of each of these types of operating systems. The PGI User’s Guide defines the following terms with respect to these platforms: x86 a processor designed to be binary compatible with i386/i486 and previous generation processors from Intel* Corporation. Used to refer collectively to such processors up to and including 32-bit variants. IA32 an Intel Architecture 32-bit processor designed to be binary compatible with x86 processors, but incorporating new features such as streaming SIMD extensions (SSE) for improved performance. AMD64 a 64-bit processor from AMD designed to be binary compatible with IA32 processors, and incorporating new features such as additional registers and 64-bit addressing support for improved performance and greatly increased memory range. EM64T a 64-bit IA32 processor with Extended Memory 64-bit Technology extensions that are binary compatible with AMD64 processors. x64 collectively, all AMD64 and EM64T processors supported by the PGI compilers. Preface 17 linux86 32-bit Linux operating system running on an x86 or x64 processorbased system, with 32-bit GNU tools, utilities and libraries used by the PGI compilers to assemble and link for 32-bit execution. linux86-64 64-bit Linux operating system running on an x64 processor-based system, with 64-bit and 32-bit GNU tools, utilities and libraries used by the PGI compilers to assemble and link for execution in either linux86 or linux86-64 environments. The 32-bit development tools and execution environment under linux86-64 are considered a cross development environment for x86 processor-based applications. Win32 any of the 32-bit Microsoft Windows Operating Systems (XP/2000/Server 2003) running on an x86 or x64 processor-based system. On these targets, the PGI compiler products include additional tools and libraries needed to build executables for 32-bit Windows systems. Win64 any of the 64-bit Microsoft Windows Operating Systems (XP Professional /Windows Server 2003 x64 Editions) running on an x64 processor-based system. On these targets, the PGI compiler products require the co-installation of a Microsoft Platform SDK to build executables for 64-bit Windows systems. Windows collectively, all Win32 and Win64 platforms supported by the PGI compilers. The following table lists the PGI compilers and tools and their corresponding commands: Table P-1: PGI Compilers and Commands Compiler or Tool Language or Function Command PGF77 FORTRAN 77 pgf77 PGF95 Fortran 90/95 pgf95 PGHPF High Performance Fortran pghpf PGCC C ANSI C99 and K&R C pgcc PGC++ ANSI C++ with cfront features pgCC 18 Preface Compiler or Tool Language or Function Command PGDBG Source code debugger pgdbg PGPROF Performance profiler pgprof In general, the designation PGF95 is used to refer to The Portland Group’s Fortran 90/95 compiler, and pgf95 is used to refer to the command that invokes the compiler. A similar convention is used for each of the PGI compilers and tools. For simplicity, examples of command-line invocation of the compilers generally reference the pgf95 command and most source code examples are written in Fortran. Usage of the PGF77 compiler, whose features are a subset of PGF95, is similar. Usage of PGHPF, PGC++, and PGCC ANSI C99 is consistent with PGF95 and PGF77, but there are command-line options and features of these compilers that do not apply to PGF95 and PGF77 (and vice versa). There are a wide variety of x86-compatible processors in use. All are supported by the PGI compilers and tools. Most of these processors are forward-compatible, but not backwardcompatible. That means code compiled to target a given processor will not necessarily execute correctly on a previous-generation processor. The most important processor types, along with a list of the features utilized by the PGI compilers that distinguish them from a compatibility standpoint, are listed in Table P-2: Table P-2: Processor Options Processor Prefetch SSE1 SSE2 SSE3 32-bit 64-bit Scalar FP Default AMD Athlon X x87 AMD Athlon XP/MP X X X x87 AMD Athlon64 X X X X X SSE AMD Opteron X X X X X SSE AMD Opteron Rev E X X X X X X SSE AMD Turion X X X X X X SSE Intel Celeron X x87 Intel Pentium II X x87 Intel Pentium III X X X x87 Preface 19 Processor Prefetch SSE1 SSE2 SSE3 32-bit 64-bit Scalar FP Default Intel Pentium 4 X X X X SSE Intel Pentium M X X X X SSE Intel Centrino X X X X SSE Intel Pentium 4 EM64T X X X X X X SSE Intel Xeon EM64T X X X X X X SSE In this manual, the convention is to use “x86” to specify the group of processors in Table P-2 that are listed “32-bit” but not “64-bit.” The convention is to use x64 to specify the group of processors that are listed as both “32-bit” and “64-bit.” x86 processor-based systems can run only under 32-bit operating systems. x64 processor-based systems can run either 32-bit or 64-bit operating systems, and can execute all 32-bit x86 binaries in either case. x64 processors have additional registers and 64-bit addressing capabilities that are utilized by the PGI compilers and tools when operating under a 64-bit operating system. The prefetch, SSE1, SSE2 and SSE3 processor features further distinguish the various processors. Where such distinctions are important with respect to a given compiler option or feature, it is explicitly noted in this manual. Note that the default for performing scalar floating-point arithmetic is to use SSE instructions on targets that support SSE1 and SSE2. See section 2.3.1, Scalar SSE Code Generation, for a detailed discussion of this topic. Related Publications The following documents contain additional information related to the x86 and x64 architectures, and the compilers and tools available from The Portland Group. • PGI Fortran Reference manual describes the FORTRAN 77, Fortran 90/95, and HPF statements, data types, input/output format specifiers, and additional reference material related to use of the PGI Fortran compilers. • System V Application Binary Interface Processor Supplement by AT&T UNIX System Laboratories, Inc. (Prentice Hall, Inc.). • System V Application Binary Interface X86-64 Architecture Processor Supplement, http://www.x86-64.org/abi.pdf. • Fortran 95 Handbook Complete ISO/ANSI Reference, Adams et al, The MIT Press, Cambridge, Mass, 1997. 20 Preface • Programming in VAX Fortran, Version 4.0, Digital Equipment Corporation (September, 1984). • IBM VS Fortran, IBM Corporation, Rev. GC26-4119. • The C Programming Language by Kernighan and Ritchie (Prentice Hall). • C: A Reference Manual by Samuel P. Harbison and Guy L. Steele Jr. (Prentice Hall, 1987). • The Annotated C++ Reference Manual by Margaret Ellis and Bjarne Stroustrup, AT&T Bell Laboratories, Inc. (Addison-Wesley Publishing Co., 1990). • OpenMP Application Program Interface , Version 2.5 May 2005 (OpenMP Architecture Review Board, 1997-2005). Getting Started 21 Chapter 1 Getting Started This chapter describes how to use the PGI compilers. The command used to invoke a compiler, for example the pgf95 command, is called a compiler driver. The compiler driver controls the following phases of compilation: preprocessing, compiling, assembling, and linking. Once a file is compiled and an executable file is produced, you can execute, debug, or profile the program on your system. Executables produced by the PGI compilers are unconstrained, meaning they can be executed on any compatible x86 or x64 processor-based system regardless of whether the PGI compilers are installed on that system. 1.1 Overview In general, using a PGI compiler involves three steps: 1. Produce a program in a file containing a .f extension or another appropriate extension (see Section 1.3.1 Input Files). This may be a program that you have written or a program that you are modifying. 2. Compile the program using the appropriate compiler command. 3. Execute, debug, or profile the executable file on your system. The PGI compilers allow many variations on these general program development steps. These variations include the following: • Stop the compilation after preprocessing, compiling or assembling to save and examine intermediate results. • Provide options to the driver that control compiler optimization or that specify various features or limitations. • Include as input intermediate files such as preprocessor output, compiler output, or assembler output. 22 Chapter 1 1.2 Invoking the Command-level PGI Compilers To translate and link a Fortran, C, or C++ language program, the pgf77, pgf95, pghpf, pgcc, and pgCC commands do the following: • Preprocess the source text file • Check the syntax of the source text • Generate an assembly language file • Pass control to the subsequent assembly and linking steps For example, if you enter the following simple Fortran program in the file hello.f: print *, “hello” end You can compile it from a shell prompt using the default pgf95 driver options. PGI$ pgf95 hello.f Linking: PGI$ By default, the executable output is placed in the file a.out (a.exe on Win32 platforms, and a filename based on the name of the first source or object file on the command line on Win64). Use the –o option to specify an output file name. To place the executable output in the file hello: PGI$ pgf95 –o hello hello.f Linking: PGI$ To execute the resulting program, simply type the filename at the command prompt and press the Return or Enter key on your keyboard: PGI$ hello hello PGI$ 1.2.1 Command-line Syntax The command-line syntax, using pgf95 as an example, is: pgf95 [options] [ path] filename [...]Getting Started 23 Where: options is one or more command-line options, all of which are described in detail in Chapter 3, Command Line Options. Case is significant for options and their arguments. The compiler drivers recognize characters preceded by a hyphen (-) as command-line options. For example, the –Mlist option specifies that the compiler creates a listing file (in the text of this manual we show command-line options using a dash instead of a hyphen, for example –Mlist). In addition, the pgCC command recognizes a group of characters preceded by a plus sign (+) as command-line options. The order of options and the filename is not fixed. That is, you can place options before and after the filename argument on the command line. However, the placement of some options is significant, for example the –l option. Note: If two or more options contradict each other, the last one in the command line takes precedence. path is the pathname to the directory containing the file named by filename. If you do not specify path for a filename, the compiler uses the current directory. You must specify path separately for each filename not in the current directory. filename is the name of a source file, assembly-language file, object file, or library to be processed by the compilation system. You can specify more than one [path]f ilename. 1.2.2 Command-line Options The command-line options control various aspects of the compilation process. For a complete alphabetical listing and a description of all the command-line options, refer to Chapter 3, Command Line Options. 1.2.3 Fortran Directives and C/C++ Pragmas Fortran directives or C/C++ pragmas inserted in program source code allow you to alter the effects of certain command-line options and control various aspects of the compilation process for a specific routine or a specific program loop. For a complete alphabetical listing and a description of all the Fortran directives and C/C++ pragmas, refer to Chapter 5, OpenMP Directives for 24 Chapter 1 Fortran, Chapter 6, OpenMP Pragmas for C and C++, and Chapter 7, Optimization Directives and Pragmas. 1.3 Filename Conventions The PGI compilers use the filenames that you specify on the command line to find and to create input and output files. This section describes the input and output filename conventions for the phases of the compilation process. 1.3.1 Input Files You can specify assembly-language files, preprocessed source files, Fortran/C/C++ source files, object files, and libraries as inputs on the command line. The compiler driver determines the type of each input file by examining the filename extensions. The drivers use the following conventions: filename.f indicates a Fortran source file. filename.F indicates a Fortran source file that can contain macros and preprocessor directives (to be preprocessed). filename.F95 indicates a Fortran 90/95 source file that can contain macros and preprocessor directives (to be preprocessed). filename.f90 indicates a Fortran 90/95 source file that is in freeform format. filename.f95 indicates a Fortran 90/95 source file that is in freeform format. filename.hpf indicates an HPF source file. filename.c indicates a C source file that can contain macros and preprocessor directives (to be preprocessed). filename.i indicates a pre-processed C or C++ source file. filename.C indicates a C++ source file that can contain macros and preprocessor directives (to be preprocessed). filename.cc indicates a C++ source file that can contain macros and preprocessor directives (to be preprocessed). filename.s indicates an assembly-language file. filename.o indicates an object file. filename.a indicates a library of object files. Getting Started 25 filename.so (Linux systems only) indicates a library of shared object files. filename.dll (Windows systems only) indicates a dynamically linked library (DLL) of object files. The driver passes files with .s extensions to the assembler and files with .o, .so, .a and .dll extensions to the linker. Input files with unrecognized extensions, or no extension, are also passed to the linker. Files with a .F (Capital F) suffix are first preprocessed by the Fortran compilers and the output is passed to the compilation phase. The Fortran preprocessor functions similar to cpp for C/C++ programs, but is built in to the Fortran compilers rather than implemented through an invocation of cpp. This ensures consistency in the pre-processing step regardless of the type or revision of operating system under which you’re compiling. Any input files not needed for a particular phase of processing are not processed. For example, if on the command line you use an assembly-language file (filename.s) and the –S option to stop before the assembly phase, the compiler takes no action on the assembly-language file. Processing stops after compilation and the assembler does not run (in this case compilation must have been completed in a previous pass which created the .s file). Refer to the following section, Output Files, for a description of the –S option. In addition to specifying primary input files on the command line, code within other files can be compiled as part of “include” files using the INCLUDE statement in a Fortran source file or the preprocessor #include directive in Fortran source files that use a .F extension or C and C++ source files. When linking a program with a library, the linker extracts only those library components that the program needs. The compiler drivers link in several libraries by default. For more information about libraries, refer to Chapter 8, Libraries. 1.3.2 Output Files By default, an executable output file produced by one of the PGI compilers is placed in the file a.out (a.exe on Win32 platforms, and a filename based on the name of the first source or object file on the command line on Win64). As shown in the preceding section, you can use the –o option to specify the output file name. If you use one of the options: –F (Fortran only), –P (C/C++ only), –S or –c, the compiler produces a file containing the output of the last phase that completes for each input file, as specified by the option supplied. The output file will be a preprocessed source file, an assemblylanguage file, or an unlinked object file respectively. Similarly, the –E option does not produce a file, but displays the preprocessed source file on the standard output. Using any of these options, 26 Chapter 1 the –o option is valid only if you specify a single input file. If no errors occur during processing, you can use the files created by these options as input to a future invocation of any of the PGI compiler drivers. The following table lists the stop after options and the output files that the compilers create when you use these options. Table 1-1: Stop after Options, Inputs and Outputs Option Stop after Input Output –E preprocessing Source files (must have .F extension for Fortran) preprocessed file to standard out –F preprocessing Source files (must have .F extension, this option is not valid for pgcc or pgCC) preprocessed file – .f –P preprocessing Source files (this option is not valid for pgf77, pgf95 or pghpf) preprocessed file – .i –S compilation Source files or preprocessed files assembly-language file – .s –c assembly Source files, preprocessed files or assembly-language files unlinked object file – .o none linking Source files, preprocessed files, assembly-language files, object files or libraries executable files a.out If you specify multiple input files or do not specify an object filename, the compiler uses the input filenames to derive corresponding default output filenames of the following form, where filename is the input filename without its extension: filename.f indicates a preprocessed file (if you compiled a Fortran file using the –F option). filename.lst indicates a listing file from the –Mlist option. filename.o indicates an object file from the –c option. filename.s indicates an assembly-language file from the –S option. Note: Unless you specify otherwise, the destination directory for any output file is the current working directory. If the file exists in the destination directory, the compiler overwrites it. Getting Started 27 The following example demonstrates the use of output filename extensions. $ pgf95 –c proto.f proto1.F This produces the output files proto.o and proto1.o, both of which are binary object files. Prior to compilation, the file proto1.F is pre-processed because it has a .F filename extension. 1.4 Parallel Programming Using the PGI Compilers The PGI compilers support three styles of parallel programming: • Automatic shared-memory parallel programs compiled using the -Mconcur option to pgf77, pgf95, pgcc, or pgCC — parallel programs of this variety can be run on shared-memory parallel (SMP) systems such as dual-core or multi-processor workstations. • OpenMP shared-memory parallel programs compiled using the -mp option to pgf77, pgf95, pgcc, or pgCC — parallel programs of this variety can be run on SMP systems. Carefully coded user-directed parallel programs using OpenMP directives can often achieve significant speed-ups on dual-core workstations or large numbers of processors on SMP server systems. Chapter 5, OpenMP Directives for Fortran, and Chapter 6, OpenMP Pragmas for C and C++, contain complete descriptions of user-directed parallel programming. • Data parallel shared- or distributed-memory parallel programs compiled using the PGHPF High Performance Fortran compiler — parallel programs of this variety can be run on SMP workstations or servers, distributed-memory clusters of workstations, or clusters of SMP workstations or servers. Coding a data parallel version of an application can be more work than using OpenMP directives, but has the advantage that the resulting executable is usable on all types of parallel systems regardless of whether shared memory is available. See the PGHPF User’s Guide for a complete description of how to build and execute data parallel HPF programs. In this manual, the first two types of parallel programs are collectively referred to as SMP parallel programs. The third type is referred to as a data parallel program, or simply as an HPF program. Some newer CPUs incorporate two or more complete processor cores (functional units, registers, level 1 cache, level 2 cache, etc) on a single silicon die. These are referred to as multi-core processors. For purposes of OpenMP, threads, or HPF parallelism, these cores function as 2 or more distinct processors. However, the processing cores are on a single chip occupying a single socket on a system motherboard. For purposes of PGI software licensing, a multi-core processor is treated as a single CPU. 28 Chapter 1 1.4.1 Running SMP Parallel Programs When you execute an SMP parallel program, by default it will use only 1 processor. To run on more than one processor, set the NCPUS environment variable to the desired number of processors (subject to a maximum of 4 for PGI’s workstation-class products). You can set this environment variable by issuing the following command: % setenv NCPUS in a shell command window under csh, or with % NCPUS=; export NCPUS in sh, ksh, or BASH command window. Note: If you set NCPUS to a number larger than the number of physical processors, your program will execute very slowly. 1.4.2 Running Data Parallel HPF Programs When you execute an HPF program, by default it will use only one processor. If you wish to run on more than one processor, use the -pghpf -np runtime option. For example, to compile and run the hello.f example defined above on one processor, you would issue the following commands: % pghpf –o hello hello.f Linking: % hello hello % To execute it on two processors, you would issue the following commands: % hello -pghpf -np 2 hello % Note: If you specify a number larger than the number of physical processors, your program will execute very slowly. Note that you still only see a single “hello” printed to your screen. This is because HPF is a single-threaded model, meaning that all statements execute with the same semantics as if they Getting Started 29 were running in serial. However, parallel statements or constructs operating on explicitly distributed data are in fact executed in parallel. The programmer must manually insert compiler directives to cause data to be distributed to the available processors. See the PGHPF User’s Guide and The High Performance Fortran Handbook for more details on constructing and executing data parallel programs on shared-memory or distributed-memory cluster systems using PGHPF. 1.5 Using the PGI Compilers on Linux 1.5.1 Linux Header Files The Linux system header files contain many GNU gcc extensions. Many of these extensions are supported. This should allow the PGCC C and C++ compilers to compile most programs compilable with the GNU compilers. A few header files not interoperable with previous revisions of the PGI compilers have been rewritten and are included in $PGI/linux86/include. These files are: sigset.h, asm/byteorder.h, stddef.h, asm/posix_types.h and others. Also, PGI’s version of stdarg.h should support changes in newer versions of Linux. If you are using the PGCC C or C++ compilers, please make sure that the supplied versions of these include files are found before the system versions. This will happen by default unless you explicitly add a –I option that references one of the system include directories. 1.5.2 Running Parallel Programs on Linux You may encounter difficulties running auto-parallel or OpenMP programs on Linux systems when the per-thread stack size is set to the default (2MB). If you have unexplained failures, please try setting the environment variable MPSTKZ to a larger value, such as 8MB. This can be accomplished with the command: % setenv MPSTKZ 8M in csh, or with % MPSTKZ=8M; export MPSTKZ in bash, sh, or ksh. If your program is still failing, you may be encountering the hard 8 MB limit on main process stack sizes in Linux. You can work around the problem by issuing the command: % limit stacksize unlimited in csh, or 30 Chapter 1 % ulimit -s unlimited in bash, sh, or ksh. 1.6 Using the PGI Compilers on Windows On Windows platforms, the tools that ship with the PGI Workstation or PGI Server commandlevel compilers include a full-featured shell command environment. After installation, you should have a PGI icon on your Windows desktop. Double-left-click on this icon to cause an instance of the BASH command shell to appear on your screen. Working within BASH is very much like working within the sh or ksh shells on a Linux system, but in addition BASH has a command history feature similar to csh and several other unique features. Shell programming is fully supported. A complete BASH User’s Guide is available through the PGI online manual set. Select “PGI Workstation” under Start->Programs and double-left-click on the documentation icon to see the online manual set. You must have a web browser installed on your system in order to read the online manuals. The BASH shell window is pre-initialized for usage of the PGI compilers, so there is no need to set environment variables or modify your command path when the command window comes up. In addition to the PGI compiler commands referenced above, within BASH you have access to over 100 common commands and utilities, including but not limited to the following: vi emacs make tar / untar gzip / gunzip ftp sed grep / egrep / fgrep awk cat cksum cp date diff du find kill ls more / less mv printenv / env rm / rmdir touch wc If you are familiar with program development in a Linux environment, editing, compiling, and executing programs within BASH will be very comfortable. If you have not previously used such an environment, you should take time to familiarize yourself with either the vi or emacs editors and with makefiles. The emacs editor has an extensive online tutorial, which you can start by bringing up emacs and selecting the appropriate option under the pull-down help menu. You can Getting Started 31 get a thorough introduction to the construction and use of makefiles through the online Makefile User’s Guide. Optimization & Parallelization 33 Chapter 2 Optimization & Parallelization Source code that is readable, maintainable, and produces correct results is not always organized for efficient execution. Normally, the first step in the program development process involves producing code that executes and produces the correct results. This first step usually involves compiling without much worry about optimization. After code is compiled and debugged, code optimization and parallelization become an issue. Invoking one of the PGI compiler commands with certain options instructs the compiler to generate optimized code. Optimization is not always performed since it increases compilation time and may make debugging difficult. However, optimization produces more efficient code that usually runs significantly faster than code that is not optimized. The compilers optimize code according to the specified optimization level. Using the –O, –Mvect, –Mipa and –Mconcur options, you can specify the optimization levels. In addition, several additional –M switches can be used to control specific types of optimization and parallelization. This chapter describes the optimization options and describes how to choose optimization options to use with the PGI compilers. Chapter 4, Function Inlining, describes how to use the function inlining options. 2.1 Overview of Optimization In general, optimization involves using transformations and replacements that generate more efficient code. This is done by the compiler and involves replacements that are independent of the particular target processor’s architecture as well as replacements that take advantage of the x86 or x64 architecture, instruction set and registers. For the discussion in this and the following chapters, optimization is divided into the following categories: Local Optimization This optimization is performed on a block-by-block basis within a program’s basic blocks. A basic block is a sequence of statements, in which the flow of control enters at the beginning and leaves at the end without the possibility of branching, except at the end. The PGI compilers perform many types of local optimization including: algebraic identity removal, constant folding, common sub-expression elimination, pipelining, redundant load and store elimination, scheduling, strength reduction, and peephole optimizations. 34 Chapter 2 Global Optimization This optimization is performed on a program unit over all its basic blocks. The optimizer performs control-flow and data-flow analysis for an entire program unit. All loops, including those formed by IFs and GOTOs are detected and optimized. Global optimization includes: constant propagation, copy propagation, dead store elimination, global register allocation, invariant code motion, and induction variable elimination. Loop Optimization: Unrolling, Vectorization, and Parallelization The performance of certain classes of loops may be improved through vectorization or unrolling options. Vectorization transforms loops to improve memory access performance and make use of packed SSE instructions which perform the same operation on multiple data items concurrently. Unrolling replicates the body of loops to reduce loop branching overhead and provide better opportunities for local optimization, vectorization and scheduling of instructions. Performance for loops on systems with multiple processors may also improve using the parallelization features of the PGI compilers. Inter-Procedural Analysis and Optimization (IPA) Interprocedural analysis allows use of information across function call boundaries to perform optimizations that would otherwise be unavailable. For example, if the actual argument to a function is in fact a constant in the caller, it may be possible to propagate that constant into the callee and perform optimizations that are not valid if the dummy argument is treated as a variable. A wide range of optimizations are enabled or improved by using IPA, including but not limited to data alignment optimizations, argument removal, constant propagation, pointer disambiguation, pure function detection, F90/F95 array shape propagation, data placement, vestigial function removal, automatic function inlining, inlining of functions from pre-compiled libraries, and interprocedural optimization of functions from pre-compiled libraries. Function Inlining This optimization allows a call to a function to be replaced by a copy of the body of that function. This optimization will sometimes speed up execution by eliminating the function call and return overhead. Function inlining may also create opportunities for other types of optimization. Function inlining is not always beneficial. When used improperly it may increase code size and generate less efficient code. Profile-Feedback Optimization (PFO) Profile-feedback optimization makes use of information from a trace file produced by specially instrumented executables which capture and save information on branch frequency, function and subroutine call frequency, semi-invariant values, loop index ranges, and other input data Optimization & Parallelization 35 dependent information that can only be collected dynamically during execution of a program. By definition, use of profile-feedback optimization is a two-phase process: compilation and execution of a specially-instrumented executable, followed by a subsequent compilation which reads a trace file generated during the first phase and uses the information in the trace file to guide compiler optimizations. 2.2 Getting Started with Optimizations Your first concern should be getting your program to execute and produce correct results. To get your program running, start by compiling and linking without optimization. Use the optimization level –O0 or select –g to perform minimal optimization. At this level, you will be able to debug your program easily and isolate any coding errors exposed during porting to x86 or x64 platforms. If you want to get started quickly with optimization, a good set of options to use with any of the PGI compilers is –fastsse –Mipa=fast. For example: $ pgf95 –fastsse –Mipa=fast prog.f For all of the PGI Fortran, C, and C++ compilers, this option will generally produce code that is well-optimized without the possibility of significant slowdowns due to pathological cases. The -fastsse option is an aggregate option that includes a number of individual PGI compiler options; which PGI compiler options are included depends on the target for which compilation is performed. The –Mipa=fast option invokes interprocedural analysis including several IPA suboptions. For C++ programs, add -Minline=levels:10 --no_exceptions: $ pgCC –fastsse –Mipa=fast –Minline=levels:10 ––no_exceptions prog.cc Note: a C++ program compiled with ––no_execptions will fail if the program uses exception handling. By experimenting with individual compiler options on a file-by-file basis, further significant performance gains can sometimes be realized. However, individual optimizations can sometimes cause slowdowns depending on coding style and must be used carefully to ensure performance improvements result. In addition to –fastsse, the optimization flags most likely to further improve performance are –O3, –Mpfi/–Mpfo, –Minline, and on targets with multiple processors –Mconcur. In addition, the –Msafeptr option can significantly improve performance of C/C++ programs in which there is known to be no pointer aliasing. However, for obvious reasons this command-line option must be used carefully. 36 Chapter 2 Three other options which are extremely useful are –help, –Minfo, and –dryrun. You can see a specification of any command-line option by invoking any of the PGI compilers with –help in combination with the option in question, without specifying any input files. For example: $ pgf95 –help –fastsse Reading rcfile /usr/pgi_rel/linux86-64/6.0/bin/.pgf95rc -fastsse == -fast -Mvect=sse -Mcache_align –Mflushz -fast Common optimizations: -O2 -Munroll=c:1 -Mnoframe -Mlre . . . Or to see the full functionality of –help itself, which can return information on either an individual option or groups of options by type: $ pgf95 –help –help Reading rcfile /usr/pgi_rel/linux86-64/6.0/bin/.pgf95rc -help[=groups|asm|debug|language|linker|opt|other|overall| phase|prepro|suffix|switch|target|variable] The –Minfo option can be used to display compile-time optimization listings. When this option is used, the PGI compilers will issue informational messages to stdout as compilation proceeds. From these messages, you can determine which loops are optimized using unrolling, SSE instructions, vectorization, parallelization, interprocedural optimizations and various miscellaneous optimizations. You can also see where and whether functions are inlined. The –Mneginfo option can be used to display informational messages listing why certain optimizations are inhibited. The –dryrun option can be useful as a diagnostic tool if you need to see the steps used by the compiler driver to pre-process, compile, assemble and link in the presence of a given set of command line inputs. When you specify the –dryrun option, these steps will be printed to stdout but will not actually be performed. For example, this allows inspection of the default and userspecified libraries that are searched during the link phase, and the order in which they are searched by the linker. The remainder of this chapter describes the –O options, the loop unroller option –Munroll, the vectorizer option –Mvect, the auto-parallelization option –Mconcur, and the inter-procedural analysis optimization –Mipa, and the profile-feedback instrumentation (–Mpfi) and optimization (–Mpfo) options. Usually, you should be able to get very near optimal compiled performance using some combination of these switches. The following overview will help if you are just getting started with one of the PGI compilers, or wish to experiment with individual optimizations. Complete specifications of each of these options are listed in Chapter 3, Command Line Options.Optimization & Parallelization 37 The chapters that follow provide more detailed information on other –M options that control specific optimizations, including function inlining. Explicit parallelization through the use of OpenMP directives or pragmas is invoked using the –mp option, described in detail in Chapter 5, OpenMP Directives for Fortran, and Chapter 6, OpenMP Pragmas for C and C++. 2.3 Local and Global Optimization using -O Using the PGI compiler commands with the –Olevel option, you can specify any of the following optimization levels (the capital O is for Optimize): –O0 level-zero specifies no optimization. A basic block is generated for each Fortran, C or C++ statement. –O1 level-one specifies local optimization. Scheduling of basic blocks is performed. Register allocation is performed. –O2 level-two specifies global optimization. This level performs all level-one local optimization as well as level-two global optimization. –O3 level-three specifies aggressive global optimization. This level performs all level-one and level-two optimizations and enables more aggressive hoisting and scalar replacement optimizations that may or may not be profitable. Level-zero optimization specifies no optimization (–O0). At this level, the compiler generates a basic block for each statement. This level is useful for the initial execution of a program. Performance will almost always be slowest using this optimization level. Level-zero is useful for debugging since there is a direct correlation between the program text and the code generated. Level-one optimization specifies local optimization (–O1). The compiler performs scheduling of basic blocks as well as register allocation. This optimization level is a good choice when the code is very irregular; that is it contains many short statements containing IF statements and the program does not contain loops (DO or DO WHILE statements). For certain types of code, this optimization level may perform better than level-two (–O2) although this case rarely occurs. The PGI compilers perform many different types of local optimizations, including but not limited to: • Algebraic identity removal • Constant folding • Common subexpression elimination • Local register optimization 38 Chapter 2 • Peephole optimizations • Redundant load and store elimination • Strength reductions Level-two optimization (–O2 or –O) specifies global optimization. The –fast option generally will specify global optimization; however, the –fast switch will vary from release to release depending on a reasonable selection of switches for any one particular release. The –O or –O2 level performs all level-one local optimizations as well as global optimizations. Control flow analysis is applied and global registers are allocated for all functions and subroutines. Loop regions are given special consideration. This optimization level is a good choice when the program contains loops, the loops are short, and the structure of the code is regular. The PGI compilers perform many different types of global optimizations, including but not limited to: • Branch to branch elimination • Constant propagation • Copy propagation • Dead store elimination • Global register allocation • Invariant code motion • Induction variable elimination You select the optimization level on the command line. For example, level-two optimization results in global optimization, as shown below: $ pgf95 –O2 prog.f Specifying –O on the command-line without a level designation is equivalent to –O2. The default optimization level changes depending on which options you select on the command line. For example, when you select the –g debugging option, the default optimization level is set to levelzero (–O0). However, you can override this default by placing –Olevel option after –g on the command-line if you need to debug optimized code. Refer to Section 2.8, Default Optimization Levels, for a description of the default levels. Optimization & Parallelization 39 As noted above, the –fast option includes –O2 on all x86 and x64 targets. If you wish to override this with –O3 while maintaining all other elements of –fast, simply compile as follows: $ pgf95 -fast –O3 prog.f 2.3.1 Scalar SSE Code Generation For all processors prior to Intel Pentium 4 and AMD Opteron/Athlon64, for example Intel Pentium III and AMD AthlonXP/MP processors, scalar floating-point arithmetic as generated by the PGI Workstation compilers is performed using x87 floating-point stack instructions. With the advent of SSE/SSE2 instructions on Intel Pentium 4/Xeon and AMD Opteron/Athlon64, it is possible to perform all scalar floating-point arithmetic using SSE/SSE2 instructions. In most cases, this is beneficial from a performance standpoint. The default on 32-bit Intel Pentium II/III (–tp p6, –tp piii, etc) or AMD AthlonXP/MP (–tp k7) is to use x87 instructions for scalar floating-point arithmetic. The default on Intel Pentium 4/Xeon or Intel EM64T running a 32-bit operating system (–tp p7), AMD Opteron/Athlon64 running a 32-bit operating system (–tp k8-32), or AMD Opteron/Athlon64 or Intel EM64T processors running a 64-bit operating system (–tp k8-64 and –tp p7-64 respectively) is to use SSE/SSE2 instructions for scalar floating-point arithmetic. The only way to override this default on AMD Opteron/Athlon64 or Intel EM64T processors running a 64-bit operating system is to specify an older 32-bit target (for example –tp k7 or –tp piii). Note that there can be significant arithmetic differences between calculations performed using x87 instructions versus SSE/SSE2. By default, all floating-point data is promoted to IEEE 80-bit format when stored on the x87 floating-point stack, and all x87 operations are performed registerto-register in this same format. Values are converted back to IEEE 32-bit or IEEE 64-bit when stored back to memory (for REAL/float and DOUBLE PRECISION/double data respectively). The default precision of the x87 floating-point stack can be reduced to IEEE 32-bit or IEEE 64-bit globally by compiling the main program with the –pc {32 | 64} option to the PGI Workstation compilers, which is described in detail in Chapter 3, Command Line Options. However, there is no way to ensure that operations performed in mixed precision will match those produced on a traditional load-store RISC/UNIX system which implements IEEE 64-bit and IEEE 32-bit registers and associated floating-point arithmetic instructions. In contrast, arithmetic results produced on Intel Pentium 4/Xeon, AMD Opteron/Athlon64 or Intel EM64T processors will usually closely match or be identical to those produced on a traditional RISC/UNIX system if all scalar arithmetic is performed using SSE/SSE2 instructions. You should keep this in mind when porting applications to and from systems which support both x87 and full SSE/SSE2 floating-point arithmetic. Many subtle issues can arise which affect your numerical results, sometimes to several digits of accuracy. 40 Chapter 2 2.4 Loop Unrolling using -Munroll This optimization unrolls loops, executing multiple instances of the loop during each iteration. This reduces branch overhead, and can improve execution speed by creating better opportunities for instruction scheduling. A loop with a constant count may be completely unrolled or partially unrolled. A loop with a non-constant count may also be unrolled. A candidate loop must be an innermost loop containing one to four blocks of code. The following shows the use of the –Munroll option: $ pgf95 –Munroll prog.f The –Munroll option is included as part of –fast and –fastsse on all x86 and x64 targets. The loop unroller expands the contents of a loop and reduces the number of times a loop is executed. Branching overhead is reduced when a loop is unrolled two or more times, since each iteration of the unrolled loop corresponds to two or more iterations of the original loop; the number of branch instructions executed is proportionately reduced. When a loop is unrolled completely, the loop’s branch overhead is eliminated altogether. Loop unrolling may be beneficial for the instruction scheduler. When a loop is completely unrolled or unrolled two or more times, opportunities for improved scheduling may be presented. The code generator can take advantage of more possibilities for instruction grouping or filling instruction delays found within the loop. Examples 2-1 and 2-2 show the effect of code unrolling on a segment that computes a dot product. REAL*4 A(100), B(100), Z INTEGER I DO I=1, 100 Z = Z + A(i) * B(i) END DO END Example 2-1: Dot Product Code REAL*4 A(100), B(100), Z INTEGER I DO I=1, 100, 2 Z = Z + A(i) * B(i) Z = Z + A(i+1) * B(i+1) END DO END Example 2-2: Unrolled Dot Product Code Optimization & Parallelization 41 Using the –Minfo option, the compiler informs you when a loop is being unrolled. For example, a message indicating the line number, and the number of times the code is unrolled, similar to the following will display when a loop is unrolled: dot: 5, Loop unrolled 5 times Using the c: and n: sub-options to –Munroll, or using –Mnounroll, you can control whether and how loops are unrolled on a file-by-file basis. Using directives or pragmas as specified in Chapter 7, Optimization Directives and Pragmas, you can precisely control whether and how a given loop is unrolled. See Chapter 3, Command Line Options, for a detailed description of the –Munroll option. 2.5 Vectorization using -Mvect The –Mvect option is included as part of –fastsse on all x86 and x64 targets. If your program contains computationally intensive loops, the –Mvect option may be helpful. If in addition you specify –Minfo, and your code contains loops that can be vectorized, the compiler reports relevant information on the optimizations applied. When a PGI compiler command is invoked with the –Mvect option, the vectorizer scans code searching for loops that are candidates for high-level transformations such as loop distribution, loop interchange, cache tiling, and idiom recognition (replacement of a recognizable code sequence, such as a reduction loop, with optimized code sequences or function calls). When the vectorizer finds vectorization opportunities, it internally rearranges or replaces sections of loops (the vectorizer changes the code generated; your source code’s loops are not altered). In addition to performing these loop transformations, the vectorizer produces extensive data dependence information for use by other phases of compilation and detects opportunities to use vector or packed Streaming SIMD Extensions (SSE) instructions on processors where these are supported. The –Mvect option can speed up code which contains well-behaved countable loops which operate on large REAL, REAL*4, REAL*8, INTEGER*4, COMPLEX or COMPLEX DOUBLE arrays in Fortran and their C/C++ counterparts. However, it is possible that some codes will show a decrease in performance when compiled with –Mvect due to the generation of conditionally executed code segments, inability to determine data alignment, and other code generation factors. For this reason, it is recommended that you check carefully whether particular program units or loops show improved performance when compiled with this option enabled. 2.5.1 Vectorization Sub-options The vectorizer performs high-level loop transformations on countable loops. A loop is countable if the number of iterations is set only before loop execution and cannot be modified during loop 42 Chapter 2 execution. Some of the vectorizer transformations can be controlled by arguments to the –Mvect command line option. The following sections describe the arguments that affect the operation of the vectorizer. In addition, some of these vectorizer operations can be controlled from within code using directives and pragmas. For details on the use of directives and pragmas, refer to Chapter 7, Optimization Directives and Pragmas. The vectorizer performs the following operations: • Loop interchange • Loop splitting • Loop fusion • Memory-hierarchy (cache tiling) optimizations • Generation of SSE instructions on processors where these are supported • Generation of prefetch instructions on processors where these are supported • Loop iteration peeling to maximize vector alignment • Alternate code generation By default, –Mvect without any sub-options is equivalent to: –Mvect=assoc,cachesize:262144 This enables the options for nested loop transformation and various other vectorizer options. These defaults may vary depending on the target system. 2.5.1.1 Assoc Option The option –Mvect=assoc instructs the vectorizer to perform associativity conversions that can change the results of a computation due to roundoff error (–Mvect=noassoc disables this option). For example, a typical optimization is to change one arithmetic operation to another arithmetic operation that is mathematically correct, but can be computationally different and generate faster code. This option is provided to enable or disable this transformation, since roundoff error for such associativity conversions may produce unacceptable results. 2.5.1.2 Cachesize Option The option –Mvect=cachesize:n instructs the vectorizer to tile nested loop operations assuming a data cache size of n bytes. By default, the vectorizer attempts to tile nested loop operations, such Optimization & Parallelization 43 as matrix multiply, using multi-dimensional strip-mining techniques to maximize re-use of items in the data cache. 2.5.1.3 SSE Option The option –Mvect=sse instructs the vectorizer to automatically generate packed SSE, SSE2 (streaming SIMD extensions) and prefetch instructions when vectorizable loops are encountered. SSE instructions, first introduced on Pentium III and AthlonXP processors, operate on singleprecision floating-point data, and hence apply only to vectorizable loops that operate on singleprecision floating-point data. SSE2 instructions, first introduced on Pentium 4, Xeon and Opteron processors, operate on double-precision floating-point data. Prefetch instructions, first introduced on Pentium III and AthlonXP processors, can be used to improve the performance of vectorizable loops that operate on either 32-bit or 64-bit floating-point data. See table P-2 for a concise list of processors that support SSE, SSE2 and prefetch instructions. Note: Programs units compiled with –Mvect=sse will not execute on Pentium, Pentium Pro, Pentium II or first generation AMD Athlon processors. They will only execute correctly on Pentium III, Pentium 4, Xeon, EM64T, AthlonXP, Athlon64 and Opteron systems running an SSE-enabled operating system. 2.5.1.4 Prefetch Option The option –Mvect=prefetch instructs the vectorizer to automatically generate prefetch instructions when vectorizable loops are encountered, even in cases where SSE or SSE2 instructions are not generated. Usually, explicit prefetching is not necessary on Pentium 4, Xeon and Opteron because these processors support hardware prefetching; nonetheless, it sometimes can be worthwhile to experiment with explicit prefetching. Prefetching can be controlled on a loop-by-loop level using prefetch directives, which are described in detail in section 7.7, Prefetch Directives. Note: Program units compiled with –Mvect=prefetch will not execute correctly on Pentium, Pentium Pro, or Pentium II processors. They will execute correctly only on Pentium III, Pentium 4, Xeon, EM64T, AthlonXP, Athlon64 or Opteron systems. In addition, the prefetchw instruction is only supported on AthlonXP, Athlon64 or Opteron systems and can cause instruction faults on non-AMD processors. For this reason, the PGI compilers do not generate prefetchw instructions by default on any target. In addition to these sub-options to –Mvect, several other sub-options are supported. See the description of –Mvect in Chapter 3, Command Line Options, for a detailed description of all available sub-options. 44 Chapter 2 2.5.2 Vectorization Example Using SSE/SSE2 Instructions One of the most important vectorization options is –Mvect=sse. This section contains an example of the use and potential effects of –Mvect=sse. When the compiler switch –Mvect=sse is used, the vectorizer in the PGI Workstation compilers automatically uses SSE and SSE2 instructions where possible when targeting processors where these are supported. This capability is supported by all of the PGI Fortran, C and C++ compilers. See table P-2 for a complete specification of which x86 and x64 processors support SSE and SSE2 instructions. Using –Mvect=sse, performance improvements of up to two times over equivalent scalar code sequences are possible. In the program in example 2-3, the vectorizer recognizes the vector operation in subroutine 'loop' when the compiler switch –Mvect=sse is used. This example shows the compilation, informational messages, and runtime results using the SSE instructions on an AMD Opteron processor-based system, along with issues that affect SSE performance. First note that the arrays in Example 2-3 are single-precision and that the vector operation is done using a unit stride loop. Thus, this loop can potentially be vectorized using SSE instructions on any processor that supports SSE or SSE2 instructions. SSE operations can be used to operate on pairs of single-precision floating-point numbers, and do not apply to double-precision floating-point numbers. SSE2 instructions can be used to operate on quads of single-precision floating-point numbers or on pairs of double-precision floating-point numbers. Loops vectorized using SSE or SSE2 instructions operate much more efficiently when processing vectors that are aligned to a cache-line boundary. You can cause unconstrained data objects of size 16 bytes or greater to be cache-aligned by compiling with the –Mcache_align switch. An unconstrained data object is a data object that is not a common block member and not a member of an aggregate data structure. Note: In order for stack-based local variables to be properly aligned, the main program or function must be compiled with –Mcache_align. The –Mcache_align switch has no effect on the alignment of Fortran allocatable or automatic arrays. If you have arrays that are constrained, for example vectors that are members of Fortran common blocks, you must specifically pad your data structures to ensure proper cache alignment; –Mcache_align causes only the beginning address of each common block to be cache-aligned. The following examples show results of compiling the example code with and without –Mvect=sse. Optimization & Parallelization 45 program vector_op parameter (N = 9999) real*4 x(n),y(n),z(n),w(n) do i = 1,n y(i) = i z(i) = 2*i w(i) = 4*i enddo do j = 1, 200000 call loop(x,y,z,w,1.0e0,n) enddo print*,x(1),x(771),x(3618),x(6498),x(9999) end subroutine loop(a,b,c,d,s,n) integer i,n real*4 a(n),b(n),c(n),d(n),s do i = 1,n a(i) = b(i) + c(i) – s * d(i) enddo end Example 2-3: Vector operation using SSE instructions Assume the above program is compiled as follows: % pgf95 -fast -Minfo vadd.f vector_op: 4, Loop unrolled 4 times loop: 18, Loop unrolled 4 times Following is the result if the generated executable is run and timed on a standalone AMD Opteron 2.2 Ghz system: % /bin/time a.out -1.000000 -771.000 -3618.000 -6498.00 -9999.00 5.15user 0.00system 0:05.16 elapsed 99%CPU 46 Chapter 2 Now, recompile with SSE vectorization enabled: % pgf95 -fast –Mvect=sse -Minfo vadd.f vector_op: 4, Unrolling inner loop 8 times Loop unrolled 7 times (completely unrolled) loop: 18, Generating vector sse code for inner loop Generated 3 prefetch instructions for this loop Note the informational message indicating that the loop has been vectorized and SSE instructions have been generated. The second part of the informational message notes that prefetch instructions have been generated for 3 loads to minimize latency of transfers of data from main memory. Executing again, you should see results similar to the following: % /bin/time a.out -1.000000 -771.000 -3618.00 -6498.00 -9999.0 3.55user 0.00system 0:03.56elapsed 99%CPU The result is a speed-up of 45% over the equivalent scalar (i.e. non-SSE) version of the program. Speed-up realized by a given loop or program can vary widely based on a number of factors: • Performance improvement using vector SSE or SSE2 instructions is most effective when the vectors of data are resident in the data cache. • If data is aligned properly, performance will be better in general than when using vector SSE operations on unaligned data. • If the compiler can guarantee that data is aligned properly, even more efficient sequences of SSE instructions can be generated. • SSE2 vector instructions can operate on 4 single-precision elements concurrently, but only 2 double-precision elements. As a result, the efficiency of loops that operate on singleprecision data can be higher. Note: Compiling with –Mvect=sse can result in numerical differences from the generated executable. Certain vectorizable operations, for example dot products, are sensitive to order of operations and the associative transformations necessary to enable vectorization (or parallelization). Optimization & Parallelization 47 2.6 Auto-Parallelization using -Mconcur With the -Mconcur option the compiler scans code searching for loops that are candidates for auto-parallelization. –Mconcur must be used at both compile-time and link-time. When the parallelizer finds opportunities for auto-parallelization, it parallelizes loops and you are informed of the line or loop being parallelized if the -Minfo option is present on the compile line. See Chapter 3, Command Line Options, for a complete specification of -Mconcur. A loop is considered parallelizable if doesn't contain any cross-iteration data dependencies. Crossiteration dependencies from reductions and expandable scalars are excluded from consideration, enabling more loops to be parallelizable. In general, loops with calls are not parallelized due to unknown side effects. Also, loops with low trip counts are not parallelized since the overhead in setting up and starting a parallel loop will likely outweigh the potential benefits. In addition, the default is to not parallelize innermost loops, since these often by definition are vectorizable using SSE instructions and it is seldom profitable to both vectorize and parallelize the same loop, especially on multi-core processors. Compiler switches and directives are available to let you override most of these restrictions on auto-parallelization. 2.6.1 Auto-parallelization Sub-options The parallelizer performs various operations that can be controlled by arguments to the –Mconcur command line option. The following sections describe these arguments that affect the operation of the vectorizer. In addition, these vectorizer operations can be controlled from within code using directives and pragmas. For details on the use of directives and pragmas, refer to Chapter 7, Optimization Directives and Pragmas. By default, –Mconcur without any sub-options is equivalent to: –Mconcur=dist:block This enables parallelization of loops with blocked iteration allocation across the available threads of execution. These defaults may vary depending on the target system. 2.6.1.1 Altcode Option The option –Mconcur=altcode instructs the parallelizer to generate alternate serial code for parallelized loops. If altcode is specified without arguments, the parallelizer determines an appropriate cutoff length and generates serial code to be executed whenever the loop count is less than or equal to that length. If altcode:n is specified, the serial altcode is executed whenever the loop count is less than or equal to n. If noaltcode is specified, no alternate serial code is generated. 48 Chapter 2 2.6.1.2 Dist Option The option –Mconcur=dist:{block|cyclic} option specifies whether to assign loop iterations to the available threads in blocks or in a cyclic (round-robin) fashion. Block distribution is the default. If cyclic is specified, iterations are allocated to processors cyclically. That is, processor 0 performs iterations 0, 3, 6, etc.; processor 1 performs iterations 1, 4, 7, etc.; and processor 2 performs iterations 2, 5, 8, etc. 2.6.1.3 Cncall Option The option –Mconcur=cncall specifies that it is safe to parallelize loops that contain subroutine or function calls. By default, such loops are excluded from consideration for auto-parallelization. Also, no minimum loop count threshold must be satisfied before parallelization will occur, and last values of scalars are assumed to be safe. The environment variable NCPUS is checked at runtime for a parallel program. If NCPUS is set to 1, a parallel program runs serially, but will use the parallel routines generated during compilation. If NCPUS is set to a value greater than 1, the specified number of processors will be used to execute the program. Setting NCPUS to a value exceeding the number of physical processors can produce inefficient execution. Executing a program on multiple processors in an environment where some of the processors are being time-shared with another executing job can also result in inefficient execution. As with the vectorizer, the -Mconcur option can speed up code if it contains well-behaved countable loops and/or computationally intensive nested loops that operate on arrays. However, it is possible that some codes will show a decrease in performance on multi-processor systems when compiled with -Mconcur due to parallelization overheads, memory bandwidth limitations in the target system, false-sharing of cache lines, or other architectural or code-generation factors. For this reason, it is recommended that you check carefully whether particular program units or loops show improved performance when compiled using this option. If the compiler is not able to successfully auto-parallelize your application, you should refer to Chapter 5, OpenMP Directives for Fortran, or Chapter 6, OpenMP Pragmas for C and C++, to see if insertion of explicit parallelization directives or pragmas and use of the –mp compiler option enables the application to run in parallel. 2.6.2 Loops That Fail to Parallelize In spite of the sophisticated analysis and transformations performed by the compiler, programmers will often note loops that are seemingly parallel, but are not parallelized. In this subsection, we’ll look at some examples of common situations where parallelization does not occur. Optimization & Parallelization 49 2.6.2.1 Innermost Loops As noted earlier in this chapter, the PGI compilers will not parallelize innermost loops by default, because it is usually not profitable. You can override this default using the command-line option –Mconcur=innermost. 2.6.2.2 Timing Loops Often, loops will occur in programs that are similar to timing loops. The outer loop in the following example is one such loop. do 1 j = 1, 2 do 1 i = 1, n a(i) = b(i) + c(i) 1 continue The outer loop above is not parallelized because the compiler detects a cross-iteration dependence in the assignment to a(i). Suppose the outer loop were parallelized. Then both processors would simultaneously attempt to make assignments into a(1:n). Now in general the values computed by each processor for a(1:n) will differ, so that simultaneous assignment into a(1:n) will produce values different from sequential execution of the loops. In this example, values computed for a(1:n) don’t depend on j, so that simultaneous assignment by both processors will not yield incorrect results. However, it is beyond the scope of the compilers’ dependence analysis to determine that values computed in one iteration of a loop don’t differ from values computed in another iteration. So the worst case is assumed, and different iterations of the outer loop are assumed to compute different values for a(1:n). Is this assumption too pessimistic? If j doesn’t occur anywhere within a loop, the loop exists only to cause some delay, most probably to improve timing resolution. And, it’s not usually valid to parallelize timing loops; to do so would distort the timing information for the inner loops. 2.6.2.3 Scalars Quite often, scalars will inhibit parallelization of non-innermost loops. There are two separate cases that present problems. In the first case, scalars appear to be expandable, but appear in noninnermost loops, as in the following example. do 1 j = 1, n x = b(j) do 1 i = 1, n a(i,j) = x + c(i,j) 1 continue 50 Chapter 2 There are a number of technical problems to be resolved in order to recognize expandable scalars in non-innermost loops. Until this generalization occurs, scalars like x above will inhibit parallelization of loops in which they are assigned. In the following example, scalar k is not expandable, and it is not an accumulator for a reduction. k = 1 do 3 i = 1, n do 1 j = 1, n 1 a(j,i) = b(k) * x k = i 2 if (i .gt. n/2) k = n - (i - n/2) 3 continue If the outer loop is parallelized, conflicting values will be stored into k by the various processors. The variable k cannot be made local to each processor because the value of k must remain coherent among the processors. It is possible the loop could be parallelized if all assignments to k are placed in critical sections. However, it is not clear where critical sections should be introduced because in general the value for k could depend on another scalar (or on k itself), and code to obtain the value of other scalars must reside in the same critical section. In the example above, the assignment to k within a conditional at label 2 prevents k from being recognized as an induction variable. If the conditional statement at label 2 is removed, k would be an induction variable whose value varies linearly with j, and the loop could be parallelized. 2.6.2.4 Scalar Last Values During parallelization, scalars within loops often need to be privatized; that is, each execution thread will have its own independent copy of the scalar. Problems can arise if a privatized scalar is accessed outside the loop. For example, consider the following loop: for (i = 1; i 5.0 ) t = x[i]; } v = t; The value of t may not be computed on the last iteration of the loop. Normally, if a scalar is assigned within a loop and used following the loop, the PGI compilers save the last value of the scalar. However, if the loop is parallelized and the scalar is not assigned on every iteration, it may be difficult (without resorting to costly critical sections) to determine on what iteration t is last assigned. Analysis allows the compiler to determine that a scalar is assigned on each iteration and hence that the loop is safe to parallelize if the scalar is used later. Optimization & Parallelization 51 For example: for ( i = 1; i < n; i++){ if ( x[i] > 0.0 ) { t = 2.0; } else { t = 3.0; y[i] = ...t; } } v = t where t is assigned on every iteration of the loop. However, there are cases where a scalar may be privatizable, but if it is used after the loop, it is unsafe to parallelize. Examine this loop: for ( i = 1; i < N; i++ ){ if( x[i] > 0.0 ){ t = x[i]; ... ... y[i] = ...t; } } v = t; where each use of t within the loop is reached by a definition from the same iteration. Here t is privatizable, but the use of t outside the loop may yield incorrect results since the compiler may not be able to detect on which iteration of the parallelized loop t is last assigned. The compiler detects the above cases. Where a scalar is used after the loop but is not defined on every iteration of the loop, parallelization will not occur. When the programmer knows that the scalar is assigned on the last iteration of the loop, the programmer may use a directive or pragma to let the compiler know the loop is safe to parallelize. The Fortran directive which tells the compiler that for a given loop the last value computed for all scalars make it safe to parallelize the loop is: cpgi$l safe_lastval In addition, a command-line option, –Msafe_lastval, provides this information for all loops within the routines being compiled (essentially providing global scope). 52 Chapter 2 2.7 Inter-Procedural Analysis and Optimization using –Mipa The PGI Fortran, C and C++ compilers use interprocedural analysis (IPA) that results in minimal changes to makefiles and the standard edit-build-run application development cycle. Other than adding –Mipa to the command line, no other changes are required. For reference and background, the process of building a program without IPA is described below, followed by the minor modifications required to use IPA with the PGI compilers. While the PGCC compiler is used here to show how IPA works, similar capabilities apply to each of the PGI Fortran, C and C++ compilers. 2.7.1 Building a Program Without IPA – Single Step Using the PGCC command-level C compiler driver, three (for example) source files can be compiled and linked into a single executable with one command: % pgcc –o a.out file1.c file2.c file3.c In actuality, the pgcc driver executes several steps to produce the assembly code and object files corresponding to each source file, and subsequently to link the object files together into a single executable file. Thus, the command above is roughly equivalent to the following commands performed individually: % pgcc -S -o file1.s file1.c % as -o file1.o file1.s % pgcc -S -o file2.s file2.c % as -o file2.o file2.s % pgcc -S -o file3.s file3.c % as -o file3.o file3.s % pgcc -o a.out file1.o file2.o file3.o If any of the three source files is edited, the executable can be rebuilt with the same command line: % pgcc -o a.out file1.c file2.c file3.c This always works as intended, but has the side-effect of recompiling all of the source files, even if only one has changed. For applications with a large number of source files, this can be timeconsuming and inefficient. 2.7.2 Building a Program Without IPA - Several Steps It is also possible to use individual pgcc commands to compile each source file into a corresponding object file, and one to link the resulting object files into an executable: Optimization & Parallelization 53 % pgcc -c file1.c % pgcc -c file2.c % pgcc -c file3.c % pgcc -o a.out file1.o file2.o file3.o The pgcc driver invokes the compiler and assembler as required to process each source file, and invokes the linker for the final link command. If you modify one of the source files, the executable can be rebuilt by compiling just that file and then relinking: % pgcc -c file1.c % pgcc -o a.out file1.o file2.o file3.o 2.7.3 Building a Program Without IPA Using Make The program compilation and linking process can be simplified greatly using the make utility on systems where it is supported. Using a file makefile containing the following lines: a.out: file1.o file2.o file3.o pgcc $(OPT) -o a.out file1.o file2.o file3.o file1.o: file1.c pgcc $(OPT) -c file1.c file2.o: file2.c pgcc $(OPT) -c file2.c file3.o: file3.c pgcc $(OPT) -c file3.c It is possible to type a single make command: % make The make utility determines which object files are out of date with respect to their corresponding source files, and invokes pgcc to recompile only those source files and to relink the executable. If you subsequently edit one or more source files, the executable can be rebuilt with the minimum number of recompilations using the same single make command. 2.7.4 Building a Program with IPA Interprocedural analysis and optimization (IPA) by the PGI compilers is designed to alter the standard and make utility command-level interfaces outlined above as little as possible. IPA occurs in three phases: 54 Chapter 2 • Collection: Create a summary of each function or procedure, collecting the useful information for interprocedural optimizations. This is done during the compile step if the –Mipa switch is present on the command line; summary information is collected and stored in the object file. • Propagation: Processing all the object files to propagate the interprocedural summary information across function and file boundaries. This is done during the link step, when all the object files are combined, if the –Mipa switch is present on the link command line. • Recompile/Optimization: Each of the object files is recompiled with the propagated interprocedural information, producing a specialized object file. This is also done during the link step when the –Mipa switch is present on the link command line. When linking with –Mipa, the PGI compilers automatically regenerate IPA-optimized versions of each object file, essentially recompiling each file. If there are IPA-optimized objects from a previous build, the compilers will minimize the recompile time by reusing those objects if they are still valid. They will still be valid if the IPA-optimized object is newer than the original object file, and the propagated IPA information for that file has not changed since it was optimized. After each object file has been recompiled, the regular linker is invoked to build the application with the IPA-optimized object files. The IPA-optimized object files are saved in the same directory as the original object files, for use in subsequent program builds. 2.7.5 Building a Program with IPA - Single Step By adding the –Mipa command line switch, several source files can be compiled and linked with interprocedural optimizations with one command: % pgcc -Mipa=fast -o a.out file1.c file2.c file3.c Just like compiling without –Mipa, the driver executes several steps to produce the assembly and object files, to create the executable: % pgcc -Mipa=fast -S -o file1.s file1.c % as -o file1.o file1.s % pgcc -Mipa=fast -S -o file2.s file2.c % as -o file2.o file2.s % pgcc -Mipa=fast -S -o file3.s file3.c % as -o file3.o file3.s % pgcc -Mipa=fast -o a.out file1.o file2.o file3.o Optimization & Parallelization 55 In the last step, an IPA linker is invoked to read all the IPA summary information and perform the interprocedural propagation. The IPA linker reinvokes the compiler on each of the object files to recompile them with interprocedural information. This creates three new objects with mangled names: file1_ipa5_a.out.o, file2_ipa5_a.out.o, file2_ipa5_a.out.o The system linker is then invoked to link these IPA-optimized objects into the final executable. Later, if one of the three source files is edited, the executable can be rebuilt with the same command line: % pgcc -Mipa=fast -o a.out file1.c file2.c file3.c This will work, but again has the side-effect of compiling each source file, and recompiling each object file at link time. 2.7.6 Building a Program with IPA - Several Steps Just by adding the –Mipa command-line switch, it is possible to use individual pgcc commands to compile each source file, followed by a command to link the resulting object files into an executable: % pgcc -Mipa=fast -c file1.c % pgcc -Mipa=fast -c file2.c % pgcc -Mipa=fast -c file3.c % pgcc -Mipa=fast -o a.out file1.o file2.o file3.o The pgcc driver invokes the compiler and assembler as required to process each source file, and invokes the IPA linker for the final link command. If you modify one of the source files, the executable can be rebuilt by compiling just that file and then relinking: % pgcc -c file1.c % pgcc -o a.out file1.o file2.o file3.o When the IPA linker is invoked, it will determine that the IPA-optimized object for file1.o (file1_ipa5_a.out.o) is stale, since it is older than the object file1.o, and hence will need to be rebuilt, and will reinvoke the compiler to generate it. In addition, depending on the nature of the changes to the source file file1.c, the interprocedural optimizations previously performed for file2 and file3 may now be inaccurate. For instance, IPA may have propagated a constant argument value in a call from a function in file1.c to a function in file2.c; if the value of the argument has changed, any optimizations based on that constant value are invalid. The IPA linker will determine which, if any, of any previously created IPA-optimized objects need to be regenerated, and will reinvoke the compiler as appropriate to regenerate them. Only those objects that are stale 56 Chapter 2 or which have new or different IPA information will be regenerated, which saves on compile time. 2.7.7 Building a Program with IPA Using Make As in the previous two sections, programs can be built with IPA using the make utility, just by adding the –Mipa command-line switch: OPT=-Mipa=fast a.out: file1.o file2.o file3.o pgcc $(OPT) -o a.out file1.o file2.o file3.o file1.o: file1.c pgcc $(OPT) -c file1.c file2.o: file2.c pgcc $(OPT) -c file2.c file3.o: file3.c pgcc $(OPT) -c file3.c The single command: % make will invoke the compiler to generate any object files that are out-of-date, then invoke pgcc to link the objects into the executable; at link time, pgcc will call the IPA linker to regenerate any stale or invalid IPA-optimized objects. 2.7.8 Questions about IPA • Why is the object file so large? An object file created with –Mipa contains several additional sections. One is the summary information used to drive the interprocedural analysis. In addition, the object file contains the compiler internal representation of the source file, so the file can be recompiled at link time with interprocedural optimizations. There may be additional information when inlining is enabled. The total size of the object file may be 5-10 times its original size. The extra sections are not added to the final executable. • What if I compile with –Mipa and link without –Mipa? The PGI compilers generate a legal object file, even when the source file is compiled with –Mipa. If you compile with –Mipa and link without –Mipa, the linker is invoked on the original object files. A legal executable will be generated; while this will not have the benefit of interprocedural optimizations, any other optimizations will apply. • What if I compile without –Mipa and link with –Mipa? Optimization & Parallelization 57 At link time, the IPA linker must have summary information about all the functions or routines used in the program. This information is created only when a file is compiled with –Mipa. If you compile a file without –Mipa and then try to get interprocedural optimizations by linking with –Mipa, the IPA linker will issue a message that some routines have no IPA summary information, and will proceed to run the system linker using the original object files. If some files were compiled with –Mipa and others were not, it will determine the safest approximation of the IPA summary information for those files not compiled with –Mipa, and use that to recompile the other files using interprocedural optimizations. • Can I build multiple applications in the same directory with –Mipa? Yes. Suppose you have three source files: main1.c, main2.c, sub.c, where sub.c is shared between the two applications. When you build the first application with –Mipa: % pgcc -o app1 main1.c sub.c the IPA linker will create two IPA-optimized object files: main1_ipa4_app1.oo sub_ipa4_app1.oo and use them to build the first application. When you build the second application: % pgcc -o app2 main2.c sub.c the IPA linker will create two more IPA-optimized object files: main2_ipa4_app2.oo sub_ipa4_app2.oo Note there are now three object files for sub.c: the original sub.o, and two IPA-optimized objects, one for each application in which it appears. • How is the mangled name for the IPA-optimized object files generated? The mangled name has '_ipa' appended, followed by the decimal number of the length of the executable file name, followed by an underscore and the executable file name itself. The suffix is changed to .oo so linking *.o does not pull in the IPA-optimized objects. If the IPA linker determines that the file would not benefit from any interprocedural optimizations, it does not have to recompile the file at link time, and will use the original object. 58 Chapter 2 2.8 Profile-Feedback Optimization using –Mpfi/–Mpfo The PGI compilers support many common profile-feedback optimizations, including semiinvariant value optimizations and block placement. These are performed under control of the –Mpfi/–Mpfo command-line options. When invoked with the –Mpfi option, the PGI compilers instrument the generated executable for collection of profile and data feedback information. This information can be used in subsequent compilations that include the –Mpfo optimization option. –Mpfi must be used at both compiletime and link-time. Programs compiled with –Mpfi include extra code to collect run-time statistics and write them out to a trace file. When the resulting program is executed, a profile feedback trace file pgfi.out is generated in the current working directory. Note: programs compiled and linked with –Mpfi will execute more slowly due to the instrumentation and data collection overhead. You should use executables compiled with –Mpfi only for execution of training runs. When invoked with the –Mpfo option, the PGI compilers use data from a pgfi.out profile feedback tracefile to enable or enhance certain performance optimizations. Use of this option requires the presence of a pgfi.out trace file in the current working directory. 2.9 Default Optimization Levels Table 2-1 shows the interaction between the –O, –g and –M options. In the table, level can be 0, 1, 2 or 3, and can be vect, unroll or ipa. The default optimization level is dependent upon these command-line options. Table 2-1: Optimization and –O, –g and –M Options Optimize Option Debug Option –M Option Optimization Level none none none 1 none none –M 2 none –g none 0 –O none or –g none 2 –Olevel none or –g none level –Olevel <= 2 none or –g –M 2 –O3 none or –g none 3 Optimization & Parallelization 59 Unoptimized code compiled using the option –O0 can be significantly slower than code generated at other optimization levels. The –M option, where is vect, concur, unroll or ipa, sets the optimization level to level-2 if no –O options are supplied. The –fast and –fastsse options set the optimization level to a target-dependent optimization level if no –O options are supplied. 2.10 Local Optimization Using Directives and Pragmas Command-line options let you specify optimizations for an entire source file. Directives supplied within a Fortran source file, and pragmas supplied within a C or C++ source file, provide information to the compiler and alter the effects of certain command-line options or default behavior of the compiler (many directives have a corresponding command-line option). While a command line option affects the entire source file that is being compiled, directives and pragmas let you do the following: • Apply, or disable, the effects of a particular command-line option to selected subprograms or to selected loops in the source file (for example, an optimization). • Globally override command-line options. • Tune selected routines or loops based on your knowledge or on information obtained through profiling. Chapter 7, Optimization Directives and Pragmas, provides details on how to add directives and pragmas to your source files. 2.11 Execution Timing and Instruction Counting As this chapter shows, once you have a program that compiles, executes and gives correct results, you may optimize your code for execution efficiency. Selecting the correct optimization level requires some thought and may require that you compare several optimization levels before arriving at the best solution. To compare optimization levels, you need to measure the execution time for your program. There are several approaches you can take for timing execution. You can use shell commands that provide execution time statistics, you can include function calls in your code that provides timing information, or you can profile sections of code. Timing functions available with the PGI compilers include 3F timing routines, the SECNDS pre-declared function in PGF77 or PGF95, or the SYSTEM_CLOCK or CPU_CLOCK intrinsics in PGF95 or PGHPF. In general, when timing a program one should try to eliminate or reduce the amount of system level activities such as program loading, I/O and task switching. Example 2-4 shows a fragment that indicates how to use SYSTEM_CLOCK effectively within either an HPF or F90/F95 program unit. 60 Chapter 2 . . . integer :: nprocs, hz, clock0, clock1 real :: time integer, allocatable :: t(:) !hpf$ distribute t(cyclic) #if defined (HPF) allocate (t(number_of_processors())) #elif defined (_OPENMP) allocate (t(OMP_GET_NUM_THREADS())) #else allocate (t(1)) #endif call system_clock (count_rate=hz) ! call system_clock(count=clock0) < do work> call system_clock(count=clock1) ! t = (clock1 - clock0) time = real (sum(t)) / (real(hz) * size(t)) . . . Example 2-4: Using SYSTEM_CLOCK Command-line Options 61 Chapter 3 Command Line Options This chapter describes the syntax and operation of each compiler option. The options are arranged in alphabetical order. On a command-line, options need to be preceded by a hyphen (-). If the compiler does not recognize an option, it passes the option to the linker. This chapter uses the following notation: [item] Square brackets indicate that the enclosed item is optional. {item | item} Braces indicate that you must select one and only one of the enclosed items. A vertical bar (|) separates the choices. ... Horizontal ellipses indicate that zero or more instances of the preceding item are valid. Note: Some options do not allow a space between the option and its argument or within an argument. This fact is noted in the syntax section of the respective option. Table 3-1: Generic PGI Compiler Options Option Description –# Display invocation information. –### Show but do not execute the driver commands (same as –dryrun). –byteswapio (Fortran only) Swap bytes from big-endian to little-endian or vice versa on input/output of unformatted data –C Instrument the generated executable to perform array bounds checking at runtime. –c Stops after the assembly phase and saves the object code in filename.o. -cyglibs (Win32 only) link against the Cygnus libraries and use the Cygnus include files. You must have the full Cygwin32 environment installed in order to use this switch. –D Defines a preprocessor macro. 62 Chapter 3 Option Description -d r y r u n Show but do not execute driver commands. -E Stops after the preprocessing phase and displays the preprocessed file on the standard output. -F Stops after the preprocessing phase and saves the preprocessed file in filename.f (this option is only valid for the PGI Fortran compilers). -fast Generally optimal set of flags for the target. -fastsse Generally optimal set of flags for targets that include SSE/SSE2 capability. -flags Display valid driver options. –fpic (Linux only) Generate position-independent code. -fPIC (Linux only) Equivalent to -fpic. -g Includes debugging information in the object module. -g77libs (Linux only) Allow object files generated by g77 to be linked into PGI main programs. -gopt Includes debugging information in the object module, but forces assembly code generation identical to that obtained when -g is not present on the command line. -help Display driver help message. -I Adds a directory to the search path for #include files. –i2 Treat INTEGER variables as 2 bytes. –i4 Treat INTEGER variables as 4 bytes. –i8 Treat INTEGER and LOGICAL variables as 8 bytes and use 64-bits for INTEGER*8 operations. -K Requests special compilation semantics with regard to conformance to IEEE 754. -L Specifies a library directory. –l Loads a library. -M Selects variations for code generation and optimization. –m Displays a link map on the standard output. Command-line Options 63 Option Description -m c m o d e l = m e d i u m (-tp k8-64 and –tp p7-64 targets only) Generate code which supports the medium memory model in the linux86-64 environment. –module (F90/F95/HPF only) Save/search for module files in directory . -mp Interpret and process user-inserted shared-memory parallel programming directives (see Chapters 5 and 6). -mslibs (Win32 only) use the Microsoft linker and include files, and link against the Microsoft Visual C++ libraries. Microsoft Visual C++ must be installed in order to use this switch. -msvcrt (Win32 only) use Microsoft’s msvcrt.dll at runtime rather than the default crtdll.dll. –O Specifies code optimization level where is 0, 1, 2 or 3. -o Names the object file. -pc (–tp px/p5/p6/piii targets only) Set precision globablly for x87 floating-point calculations; must be used when compiling the main program. may be one of 32, 64 or 80. –pg Instrument the generated executable to produce a gprof-style gmon.out sample-based profiling trace file (-qp is also supported, and is equivalent). -pgf77libs Append PGF77 runtime libraries to the link line. –pgf90libs Append PGF90/PGF95 runtime libraries to the link line. -Q Selects variations for compiler steps. -R (Linux only) Passed to the Linker. Hard code into the search path for shared object files. –r Creates a relocatable object file. –r4 Interpret DOUBLE PRECISION variables as REAL. –r8 Interpret REAL variables as DOUBLE PRECISION. –rc file Specifies the name of the driver's startup file. -S Stops after the compiling phase and saves the assembly– language code in filename.s. 64 Chapter 3 Option Description –s Strips the symbol-table information from the object file. -s h a r e d (Linux only) Passed to the linker. Instructs the linker to generate a shared object file. Implies –fpic. –show Display driver's configuration parameters after startup. –silent Do not print warning messages. –time Print execution times for the various compilation steps. -t p Specify the type of the target processor. –U Undefine a preprocessor macro. –u Initializes the symbol table with , which is undefined for the linker. An undefined symbol triggers loading of the first member of an archive library. -V [ r e l e a s e _ n u m b e r ] Displays the version messages and other information, or allows invocation of a version of the compiler other than the default. -v Displays the compiler, assembler, and linker phase invocations. -W Passes arguments to a specific phase. -w Do not print warning messages. There are a large number of compiler options specific to the PGCC and PGC++ compilers, especially PGC++. Table 3-2 lists several of these options, but is not exhaustive. For a complete list of available options, including an exhaustive list of PGC++ options, use the –help commandline option. For further detail on a given option, use –help and specify the option explicitly. Table 3-2: C and C++ -specific Compiler Options Option Description -A (pgCC only) Accept proposed ANSI C++.Command-line Options 65 Option Description --no_alternative_tokens (pgCC only) Enable/disable recognition of alternative tokens. These are tokens that make it possible to write C++ without the use of the , , [, ], #, &, and ^ and characters. The alternative tokens include the operator keywords (e.g., and, bitand, etc.) and digraphs. The default is -–no_alternative_tokens. –B Allow C++ comments (using //) in C source -b (pgCC only) Compile with cfront 2.1 compatibility. This accepts constructs and a version of C++ that is not part of the language definition but is accepted by cfront. -b3 (pgCC only) Compile with cfront 3.0 compatibility. See -b above. --bool (pgCC only) Enable or disable recognition of bool. The default value is ––bool. ––[no]builtin Do/don’t compile with math subroutine builtin support, which causes selected math library routines to be inlined. The default is ––builtin. - -c f r o n t _ 2 . 1 (pgCC only) Enable compilation of C++ with compatibility with cfront version 2.1. ––cfront_3.0 (pgCC only) Enable compilation of C++ with compatibility with cfront version 3.0. --create_pch filename (pgCC only) Create a precompiled header file with the name filename. --dependencies ( see -M ) (pgCC only) Print makefile dependencies to stdout. --dependencies_to_file filename (pgCC only) Print makefile dependencies to file filename. --diag_error tag (pgCC only) Override the normal error severity of the specified diagnostic messages. --diag_remark tag (pgCC only) Override the normal error severity of the specified diagnostic messages. --diag_suppress tag (pgCC only) Override the normal error severity of the specified diagnostic messages. --diag_warning tag (pgCC only) Override the normal error severity of the specified diagnostic messages. 66 Chapter 3 Option Description --display_error_number (pgCC only) Display the error message number in any diagnostic messages that are generated. –e (pgCC only) Set the C++ front-end error limit to the specified . --[no_]exceptions (pgCC only) Disable/enable exception handling support. The default is ––exceptions ––gnu_extensions (pgCC only) Allow GNU extensions like “include next” which are required to compile Linux system header files. --[no]llalign (pgCC only) Do/don’t align long long integers on integer boundaries. The default is – –llalign. -M Generate make dependence lists. -MD Generate make dependence lists. -MD,filename (pgCC only) Generate make dependence lists and print them to file filename. --optk_allow_dollar_in_id_chars (pgCC only) Accept dollar signs in identifiers. --pch (pgCC only) Automatically use and/or create a precompiled header file. --pch_dir directoryname (pgCC only) The directory dirname in which to search for and/or create a precompiled header file. --[no_]pch_messages (pgCC only) Enable/ disable the display of a message indicating that a precompiled header file was created or used. +p (pgCC only) Disallow all anachronistic constructs. -P Stops after the preprocessing phase and saves the preprocessed file in filename.i. --preinclude= (pgCC only) Specify file to be included at the beginning of compilation; to set systemdependent macros, types, etc -t Control instantiation of template functions. --use_pch filename (pgCC only) Use a precompiled header file of the specified name as part of the current compilation. Command-line Options 67 Option Description --[no_]using_std (pgCC only) Enable/disable implicit use of the std namespace when standard header files are included. –X (pgCC only) Generate cross-reference information and place output in specified file. –Xm (pgCC only) Allow $ in names. –xh (pgCC only) Enable exception handling. –suffix (see –P) (pgCC only) Use with –E, –F, or –P to save intermediate file in a file with the specified suffix. 3.1 Generic PGI Compiler Options -# Use the –# option to display the invocations of the compiler, assembler and linker. These invocations are command-lines created by the driver from your command-line input and the default values. Default: The compiler does not display individual phase invocations. Usage: The following command-line requests verbose invocation information. $ pgf95 -# prog.f Cross-reference: –Minfo, –V, –v. -# # # Use the –### option to display the invocations of the compiler, assembler and linker but do not execute them. These invocations are command lines created by the compiler driver from the PGIRC files and the command-line options. Default: The compiler does not display individual phase invocations. Usage: The following command-line requests verbose invocation information. 68 Chapter 3 $ pgf95 -### myprog.f Cross-reference: –Minfo, –V, –dryrun. -b y t e s w a p i o Use the –byteswapio option to swap the byte-order of data in unformatted Fortran data files on input/output. When this option is used, the order of bytes is swapped in both the data and record control words (the latter occurs in unformatted sequential files). Specifically, this option can be used to convert big-endian format data files produced by most RISC workstations and high-end servers to the little-endian format used on x86 or x64 systems on the fly during file reads/writes. This option assumes that the record layouts of unformatted sequential access and direct access files are the same on the systems. Also, the assumption is that the IEEE representation is used for floating-point numbers. In particular, the format of unformatted data files produced by PGI Fortran compilers is identical to the format used on Sun and SGI workstations, that allows you to read and write unformatted Fortran data files produced on those platforms from a program compiled for an x86 or x64 platform using the –byteswapio option. Default: The compiler does not byte-swap data on input/output. Usage: The following command-line requests byte-swapping are performed on input/output. $ pgf95 -byteswapio myprog.f -C Enables array bounds checking. If an array is an assumed size array, the bounds checking only applies to the lower bound. If an array bounds violation occurs during execution, an error message describing the error is printed and the program terminates. The text of the error message includes the name of the array, the location where the error occurred (the source file and the line number in the source), and information about the out of bounds subscript (its value, its lower and upper bounds, and its dimension). Default: The compiler does not enable array bounds checking. Usage: In this example, the compiler instruments the executable produced from myprog.f to perform array bounds checking at runtime: $ pgf95 -C myprog.f Cross-reference: –Mbounds. Command-line Options 69 -c Stops after the assembling phase. Use the –c option to halt the compilation process after the assembling phase and write the object code to the file filename.o, where the input file is filename.f. Default: The compiler produces an executable file (does not use the –c option). Usage: In this example, the compiler produces the object file myprog.o in the current directory. $ pgf95 -c myprog.f Cross-reference: –E, –Mkeepasm, –o, and –S. -c y g l i b s (Win32 only) link against the Cygnus libraries and use the Cygnus include files. You must have the full Cygwin32 environment installed in order to use this switch. Default: The compiler does not link against the Cygnus libraries. -D Defines a preprocessor macro. Use the –D option to create a macro with a given value. The value must be either an integer or a character string. You can use the –D option more than once on a compiler command line. The number of active macro definitions is limited only by available memory. You can use macros with conditional compilation to select source text during preprocessing. A macro defined in the compiler invocation remains in effect for each module on the command line, unless you remove the macro with an #undef preprocessor directive or with the –U option. The compiler processes all of the –U options in a command line after processing the –D options. Syntax: –Dname[=value] Where name is the symbolic name and value is either an integer value or a character string. 70 Chapter 3 Default: If you define a macro name without specifying a value the preprocessor assigns the string 1 to the macro name. Usage: In the following example, the macro PATHLENGTH has the value 256 until a subsequent compilation. If the –D option is not used, PATHLENGTH’s value is set to 128. $ pgf95 –DPATHLENGTH=256 myprog.F Where the source text is: #ifndef PATHLENGTH #define PATHLENGTH 128 #endif SUBROUTINE SUB CHARACTER*PATHLENGTH path ... END Cross-reference: –U -d r y r u n Use the –dryrun option to display the invocations of the compiler, assembler and linker but do not execute them. These invocations are command lines created by the compiler driver from the PGIRC file and the command-line supplied with –dryrun. Default: The compiler does not display individual phase invocations. Usage: The following command-line requests verbose invocation information. $ pgf95 -dryrun myprog.f Cross-reference: –Minfo, –V, –### -E Stops after the preprocessing phase. Use the –E option to halt the compilation process after the preprocessing phase and display the preprocessed output on the standard output. Default: The compiler produces an executable file. Usage: In the following example the compiler displays the preprocessed myprog.f on the standard output. Command-line Options 71 $ pgf95 -E myprog.f Cross-reference: See the options –C, –c, –Mkeepasm, –o, –F, –S. -F Stops compilation after the preprocessing phase. Use the –F option to halt the compilation process after preprocessing and write the preprocessed output to the file filename.f where the input file is filename .F. Default: The compiler produces an executable file. Usage: In the following example the compiler produces the preprocessed file myprog.f in the current directory. $ pgf95 -F myprog.F Cross-reference: –c,–E, –Mkeepasm, –o, –S -f a s t A generally optimal set of options is chosen depending on the target system. In addition, the appropriate –tp option is automatically included to enable generation of code optimized for the type of system on which compilation is performed. Note: Auto-selection of the appropriate –tp option means that programs built using the –fast option on a given system are not necessarily backward-compatible with older systems. Cross-reference: –O, –Munroll, –Mnoframe, –Mvect, –tp, –Mscalarsse -f a s t s s e A generally optimal set of options is chosen for targets that support SSE capability. In addition, the appropriate –tp option is automatically included to enable generation of code optimized for the type of system on which compilation is performed. Note: Auto-selection of the appropriate –tp option means that programs built using the –fastsse option on a given system are not necessarily 72 Chapter 3 backward-compatible with older systems. Cross-reference: –O, –Munroll, –Mnoframe, –Mscalarsse, –Mvect, –Mcache_align, –tp -f l a g s Displays driver options on the standard output. Use this option with –v to list options that are recognized and ignored, as well as the valid options. Cross-reference: –#, –###, –v -f p i c (Linux only) Generate position-independent code suitable for inclusion in shared object (dynamically linked library) files. Cross-reference: –shared, –G, –R -f P I C (Linux only) Equivalent to -fpic. Provided for compatibility with other compilers. Cross-reference: –fpic, –shared, –G, –R -G (Linux only) Passed to the linker. Instructs the linker to produce a shared object file. Cross-reference: –fpic, –shared, –R -g The –g option instructs the compiler to include symbolic debugging information in the object module. Debuggers, such as PGDBG, require symbolic debugging information in the object Command-line Options 73 module to display and manipulate program variables and source code. Note that including symbolic debugging information increases the size of the object module. If you specify the –g option on the command-line, the compiler sets the optimization level to –O0 (zero), unless you specify the –O option. For more information on the interaction between the –g and –O options, see the –O entry. Symbolic debugging may give confusing results if an optimization level other than zero is selected. Default: The compiler does not put debugging information into the object module. Usage: In the following example, the object file a.out contains symbolic debugging information. $ pgf95 -g myprog.f -g o p t Use of –g alters how optimized code is generated in ways that are intended to enable or improve debugging of optimized code. The –gopt option instructs the compiler to include symbolic debugging information in the object file, and to generate optimized code identical to that generated when –g is not specified. Default: The compiler does not put debugging information into the object module. Usage: In the following example, the object file a.out contains symbolic debugging information. $ pgf95 -gopt myprog.f -g 7 7 l i b s (Linux only) Use the –g77libs option on the link line if you are linking g77-compiled program units into a pgf95-compiled main program using the pgf95 driver. When this option is present, the pgf95 driver will search the necessary g77 support libraries to resolve references specific to g77 compiled program units. The g77 compiler must be installed on the system on which linking occurs in order for this option to function correctly. Default: The compiler does not search g77 support libraries to resolve references at link time. Usage: The following command-line requests that g77 support libraries be searched at link time: $ pgf95 -g77libs myprog.f g77_object.o 74 Chapter 3 -h e l p Used with no other options, –help displays options recognized by the driver on the standard output. When used in combination with one or more additional options, usage information for those options is displayed to standard output. Usage: In the following example, usage information for –Minline is printed to standard output. $ pgcc -help -Minline -Minline[=lib:||except:| name:|size:|levels:] Enable function inlining lib: Use extracted functions from extlib Inline function func except: Do not inline function func name: Inline function func size: Inline only functions smaller than n levels: Inline n levels of functions -Minline Inline all functions that were extracted In the following example, usage information for –help shows how groups of options can be listed or examined according to function $ pgcc -help -help -help[=groups|asm|debug|language|linker|opt|other| overall|phase|prepro|suffix|switch|target|variable] Show compiler switches Cross-reference: –#, –###, –show, –V, –flags -I Adds a directory to the search path for files that are included using the INCLUDE statement or the preprocessor directive #include. Use the –I option to add a directory to the list of where to search for the included files. The compiler searches the directory specified by the –I option before the default directories. Command-line Options 75 Syntax: –Idirectory Where directory is the name of the directory added to the standard search path for include files. Usage: The Fortran INCLUDE statement directs the compiler to begin reading from another file. The compiler uses two rules to locate the file: 1. If the file name specified in the INCLUDE statement includes a path name, the compiler begins reading from the file it specifies. 2. If no path name is provided in the INCLUDE statement, the compiler searches (in order): ? any directories specified using the –I option (in the order specified.) ? the directory containing the source file ? the current directory For example, the compiler applies rule (1) to the following statements: INCLUDE '/bob/include/file1' (absolute path name) INCLUDE '../../file1' (relative path name) and rule (2) to this statement: INCLUDE 'file1' Cross-reference: –Mnostdinc -i 2 , -i 4 a n d -i 8 Treat INTEGER and LOGICAL variables as either two, four, or eight bytes. INTEGER*8 values not only occupy 8 bytes of storage, but operations use 64 bits, instead of 32 bits. 76 Chapter 3 -K < f l a g > Requests that the compiler provide special compilation semantics. Syntax: –K Where flag is one of the following: ieee Perform floating-point operations in strict conformance with the IEEE 754 standard. Some optimizations are disabled, and on some systems a more accurate math library is linked if –Kieee is used during the link step. noieee Use the fastest available means to perform floating-point operations, link in faster non-IEEE libraries if available, and disable underflow traps. PIC (Linux only) Generate position-independent code. Equivalent to –fpic. Provided for compatibility with other compilers. pic (Linux only) Generate position-independent code. Equivalent to –fpic. Provided for compatibility with other compilers. trap=option[,option]... Controls the behavior of the processor when floating-point exceptions occur. Possible options include: fp align (ignored) inv denorm divz ovf unf inexact –Ktrap is only processed by the compilers when compiling main functions/programs. The options inv, denorm, divz, ovf, unf, and inexact correspond to the processor’s exception mask bits invalid operation, denormalized operand, divide-by-zero, overflow, underflow, and precision, respectively. Normally, the processor’s exception mask bits are on (floating-point exceptions are masked—the processor recovers from the exceptions and continues). If a floating-point exception occurs and its corresponding mask bit is off (or “unmasked”), execution terminates with an arithmetic exception (C's SIGFPE signal). -Ktrap=fp is equivalent to Command-line Options 77 -Ktrap=inv,divz,ovf. Default: The default is -Knoieee. -L Specifies a directory to search for libraries. Use –L to add directories to the search path for library files. Multiple –L options are valid. However, the position of multiple –L options is important relative to –l options supplied. Syntax: –Ldirectory Where directory is the name of the library directory. Default: Search the standard library directory. Usage: In the following example, the library directory is /lib and the linker links in the standard libraries required by PGF95 from /lib. $ pgf95 –L/lib myprog.f In the following example, the library directory /lib is searched for the library file libx.a and both the directories /lib and /libz are searched for liby.a. $ pgf95 –L/lib –lx –L/libz –ly myprog.f -l < l i b r a r y > Loads a library. The linker searches in addition to the standard libraries. Libraries specified with –l are searched in order of appearance and before the standard libraries. Syntax: –llibrary Where library is the name of the library to search. The compiler prepends the characters lib to the library name and adds the .a extension following the library name. Usage: In the following example, if the standard library directory is /lib the linker loads the library /lib/libmylib.a, in addition to the standard libraries. $ pgf95 myprog.f –lmylib 78 Chapter 3 -M < p g f l a g > Selects options for code generation. The options are divided into the following categories: • Code generation • Environment • Inlining • Fortran Language Controls • C/C++ Language Controls • Optimization • Miscellaneous Table 3-3 lists and briefly describes the options alphabetically and includes a field showing the category. Table 3-3: –M Options Summary pgflag Description Category anno annotate the assembly code with source code. Miscellaneous [no]asmkeyword specifies whether the compiler allows the asm keyword in C/C++ source files (pgcc and pgCC only). C/C++ Language [no]backslash determines how the backslash character is treated in quoted strings (pgf77, pgf95, and pghpf only). Fortran Language [no]bounds specifies whether array bounds checking is enabled or disabled. Miscellaneous [no]builtin Do/don’t compile with math subroutine builtin support, which causes selected math library routines to be inlined (pgcc and pgCC only). Optimization byteswapio Swap byte-order (big-endian to little-endian or vice versa) during I/O of Fortran unformatted data. Miscellaneous Command-line Options 79 pgflag Description Category cache_align where possible, align data objects of size greater than or equal to 16 bytes on cache-line boundaries. Optimization chkfpstk check for internal consistency of the x87 FP stack in the prologue of a function and after returning from a function or subroutine call (–tp px/p5/p6/piii targets only). Miscellaneous chkptr check for NULL pointers (pgf95 and pghpf only). Miscellaneous chkstk check the stack for available space upon entry to and before the start of a parallel region. Useful when many private variables are declared. Miscellaneous concur enable auto-concurrentization of loops. Multiple processors or cores will be used to execute parallelizable loops. Optimization cpp run the PGI cpp-like pre-processor without performing subsequent compilation steps. Miscellaneous cray Force Cray Fortran (CF77) compatibility (pgf77, pgf95, and pghpf only). Optimization [no]daz Do/don’t treat denormalized numbers as zero. Code Generation [no]dclchk determines whether all program variables must be declared (pgf77, pgf95, and pghpf only). Fortran Language [no]defaultunit determines how the asterisk character ("*") is treated in relation to standard input and standard output (regardless of the status of I/O units 5 and 6, pgf77, pgf95, and pghpf only). Fortran Language [no]depchk checks for potential data dependencies. Optimization [no]dlines determines whether the compiler treats lines containing the letter "D" in column one as executable statements (pgf77, pgf95, and pghpf only). Fortran Language dll Link with the DLL version of the runtime libraries (Windows only). Miscellaneous dollar specifies the character to which the compiler maps the dollar sign code (pgf77, pgf95, and pghpf only). Fortran Language dwarf1 when used with –g, generate DWARF1 format debug information. Code Generation 80 Chapter 3 pgflag Description Category dwarf2 when used with –g, generate DWARF2 format debug information. Code Generation dwarf3 when used with –g, generate DWARF3 format debug information. Code Generation extend the compiler accepts 132-column source code; otherwise it accepts 72-column code (pgf77, pgf95, and pghpf only). Fortran Language extract invokes the function extractor. Inlining fcon instructs the compiler to treat floating-point constants as float data types (pgcc and pgCC only). C/C++ Language fixed the compiler assumes F77-style fixed format source code (pgf95 and pghpf only). Fortran Language [no]flushz do/don’t set SSE flush-to-zero mode Code Generation [no]fprelaxed Perform certain floating point intrinsic functions using relaxed precision. Optimization free the compiler assumes F90-style free format source code (pgf95 and pghpf only). Fortran Language func32 the compiler aligns all functions to 32-byte boundaries. Code Generation gccbug[s] match behavior of certain gcc bugs Miscellaneous noi4 determines how the compiler treats INTEGER variables (pgf77, pgf95, and pghpf only). Optimization info prints informational messages regarding optimization and code generation to standard output as compilation proceeds. Miscellaneous inform specifies the minimum level of error severity that the compiler displays. Miscellaneous inline invokes the function inliner. Inlining [no]ipa invokes inter-procedural analysis and optimization. Optimization [no]iomutex determines whether critical sections are generated around Fortran I/O calls (pgf77, pgf95, and pghpf only). Fortran Language [no]largearrays enable support for 64-bit indexing and single static data objects of size larger than 2GB. Code Generation Command-line Options 81 pgflag Description Category lfs link in libraries that allow file I/O to files of size larger than 2GB on 32-bit systems (32-bit Linux only). Environment [no]lre Disable/enable loop-carried redundancy elimination. Optimization keepasm instructs the compiler to keep the assembly file. Miscellaneous [no]list specifies whether the compiler creates a listing file. Miscellaneous makedll Generate a dynamic link library (DLL) (Windows only). Miscellaneous [no]movnt (disable) force generation of non-temporal moves and prefetching. Code Generation neginfo instructs the compiler to produce information on why certain optimizations are not performed. Miscellaneous [no]frame eliminates operations that set up a true stack frame pointer for functions. Optimization nomain when the link step is called, don’t include the object file that calls the Fortran main program (pgf77, pgf95, and pghpf only). Code Generation noopenmp when used in combination with the -mp option, causes the compiler to ignore OpenMP parallelization directives or pragmas, but still process SGI-style parallelization directives or pragmas. Miscellaneous nopgdllmain do not link the module containing the default DllMain() into the DLL (Windows only). Miscellaneous nosgimp when used in combination with the -mp option, causes the compiler to ignore SGI-style parallelization directives or pragmas, but still process OpenMP directives or pragmas. Miscellaneous nostartup do not link in the standard startup routine (pgf77, pgf95, and pghpf only). Environment nostddef instructs the compiler to not recognize the standard preprocessor macros. Environment nostdinc instructs the compiler to not search the standard location for include files. Environment nostdlib instructs the linker to not link in the standard libraries. Environment 82 Chapter 3 pgflag Description Category noonetrip determines whether each DO loop executes at least once (pgf77, pgf95, and pghpf only). Language novintr disable idiom recognition and generation of calls to optimized vector functions. Optimization pfi instrument the generated code and link in libraries for dynamic collection of profile and data information at runtime. Optimization pfo read a pgfi.out trace file and use the information to enable or guide optimizations. Optimization [no]prefetch (disable) enable generation of prefetch instructions. Optimization preprocess perform cpp-like preprocessing on assembly language and Fortran input source files. Miscellaneous prof set profile options; function-level and line-level profiling are supported. Code Generation [no]r8 determines whether the compiler promotes REAL variables and constants to DOUBLE PRECISION (pgf77, pgf95, and pghpf only). Optimization [no]r8intrinsics determines how the compiler treats the intrinsics CMPLX and REAL (pgf77, pgf95, and pghpf only). Optimization [no]recursive allocate (do not allocate) local variables on the stack, this allows recursion. SAVEd, datainitialized, or namelist members are always allocated statically, regardless of the setting of this switch (pgf77, pgf95, and pghpf only). Code Generation [no]reentrant specifies whether the compiler avoids optimizations that can prevent code from being reentrant. Code Generation [no]ref_externals do/don’t force references to names appearing in EXTERNAL statements (pgf77, pgf95, and pghpf only). Code Generation safeptr instructs the compiler to override data dependencies between pointers and arrays (pgcc and pgCC only). Optimization Command-line Options 83 pgflag Description Category safe_lastval In the case where a scalar is used after a loop, but is not defined on every iteration of the loop, the compiler does not by default parallelize the loop. However, this option tells the compiler it safe to parallelize the loop. For a given loop, the last value computed for all scalars make it safe to parallelize the loop. Code Generation [no]save determines whether the compiler assumes that all local variables are subject to the SAVE statement (pgf77, pgf95, and pghpf only). Fortran Language [no]scalarsse do/don’t use SSE/SSE2 instructions to perform scalar floating-point arithmetic. Optimization schar specifies signed char for characters (pgcc and pgCC only - also see uchar). C/C++ Language [no]second_underscore do/don’t add the second underscore to the name of a Fortran global if its name already contains an underscore (pgf77, pgf95, and pghpf only). Code Generation [no]signextend do/don’t extend the sign bit, if it is set. Code Generation [no]single do/don’t convert float parameters to double parameter characters (pgcc and pgCC only). C/C++ Language [no]smart do/don’t enable optional AMD64-specific postpass assembly optimizer. Optimization standard causes the compiler to flag source code that does not conform to the ANSI standard (pgf77, pgf95, and pghpf only). Fortran Language nostride0 the compiler generates (does not generate) alternate code for a loop that contains an induction variable whose increment may be zero (pgf77, pgf95, and pghpf only). Code Generation uchar specifies unsigned char for characters (pgcc and pgCC only - also see schar). C/C++ Language unix uses UNIX calling and naming conventions for Fortran subprograms (pgf77, pgf95, and pghpf for Win32 only). Code Generation 84 Chapter 3 pgflag Description Category [no]unixlogical determines whether logical .TRUE. and .FALSE. are determined by non-zero (TRUE) and zero (FALSE) values for unixlogical. With nounixlogical, the default, -1 values are TRUE and 0 values are FALSE (pgf77, pgf95, and pghpf only). Fortran Language [no]unroll controls loop unrolling. Optimization [no]upcase determines whether the compiler allows uppercase letters in identifiers (pgf77, pgf95, and pghpf only). Fortran Language varargs force Fortran program units to assume calls are to C functions with a varargs type interface (pgf77 and pgf95 only). Code Generation [no]vect do/don’t invoke the code vectorizer. Optimization Following are detailed descriptions of several, but not all, of the –M options outlined in the table above. These options are grouped according the category that appears in column 3 of the table above, and are listed with exact syntax, defaults, and notes concerning similar or related options. For the latest information and description of a given option, or to see all available options, use the –help command-line option to any of the PGI compilers. -M < p g f l a g > C o d e G e n e r a t i o n C o n t r o l s Syntax: -Mdaz Set IEEE denormalized input values to zero; there is a performance benefit but misleading results can occur, such as when dividing a small normalized number by a denormalized number. -Mnodaz Do not treat denormalized numbers as zero. -Mdwarf1 Generate DWARF1 format debug information; must be used in combination with –g. Command-line Options 85 -Mdwarf2 Generate DWARF2 format debug information; must be used in combination with –g. -Mdwarf3 Generate DWARF3 format debug information; must be used in combination with –g. -Mflushz Set SSE flush-to-zero mode; if a floating-point underflow occurs, the value is set to zero. -Mnoflushz Do not set SSE flush-to-zero mode; generate underflows. -Mfunc32 Align functions on 32-byte boundaries. -Mlarge_arrays Enable support for 64-bit indexing and single static data objects larger than 2GB in size. This option is default in the presence of –mcmodel=medium. Can be used separately together with the default small memory model for certain 64-bit applications that manage their own memory space. -Mnolarge_arrays Disable support for 64-bit indexing and single static data objects larger than 2GB in size. When placed after –mcmodel=medium on the command line, disables use of 64-bit indexing for applications that have no single data object larger than 2GB. -Mnomain instructs the compiler not to include the object file that calls the Fortran main program as part of the link step. This option is useful for linking programs in which the main program is written in C/C++ and one or more subroutines are written in Fortran (pgf77, pgf95, and pghpf only). -M[no]movnt instructs the compiler to generate nontemporal move and prefetch instructions even in cases where the compiler cannot determine statically at compile-time that these instructions will be beneficial. 86 Chapter 3 -Mprof [=option[, option,...]] Set profile options. option can be any of the following: dwarf generate limited DWARF information to enable source correlation by 3 rd -party profiling tools. func perform PGI-style function-level profiling. hwcts Use PAPI-based profiling with hardware counters (linux86-64 platforms only). lines perform PGI-style line-level profiling. mpi perform MPI profiling (available only in PGI CDK Cluster Development Kit configurations). time Sample-based instruction-level profiling. -Mrecursive instructs the compiler to allow Fortran subprograms to be called recursively. -Mnorecursive Fortran subprograms may not be called recursively. -Mref_externals force references to names appearing in EXTERNAL statements (pgf77, pgf95, and pghpf only). -Mnoref_externals do not force references to names appearing in EXTERNAL statements (pgf77, pgf95, and pghpf only). -Mreentrant instructs the compiler to avoid optimizations that can prevent code from being reentrant. -Mnoreentrant instructs the compiler not to avoid optimizations that can prevent code from being reentrant. -Msecond_underscore instructs the compiler to add a second underscore to the name of a Fortran global symbol if its name already contains an underscore. This option is useful for maintaining compatibility with object code compiled using g77, which uses this convention by default (pgf77, pgf95, and pghpf only). -Mnosecond_underscore instructs the compiler not to add a second underscore to the name of a Command-line Options 87 Fortran global symbol if its name already contains an underscore (pgf77, pgf95, and pghpf only). -Msignextend instructs the compiler to extend the sign bit that is set as a result of converting an object of one data type to an object of a larger signed data type. -Mnosignextend instructs the compiler not to extend the sign bit that is set as the result of converting an object of one data type to an object of a larger data type. -Msafe_lastval In the case where a scalar is used after a loop, but is not defined on every iteration of the loop, the compiler does not by default parallelize the loop. However, this option tells the compiler it’s safe to parallelize the loop. For a given loop the last value computed for all scalars make it safe to parallelize the loop. -Mstride0 instructs the compiler to inhibit certain optimizations and to allow for stride 0 array references. This option may degrade performance and should only be used if zero-stride induction variables are possible. -Mnostride0 instructs the compiler to perform certain optimizations and to disallow for stride 0 array references. -Munix use UNIX symbol and parameter passing conventions for Fortran subprograms (pgf77, pgf95, and pghpf for Win32 only). -Mvarargs force Fortran program units to assume procedure calls are to C functions with a varargs-type interface (pgf77 and pgf95 only). Default: For arguments that you do not specify, the default code generation controls are as follows: nodaz noflushz norecursive nostride0 noreentrant nosecond_underscore noref_externals signextend 88 Chapter 3 -M < p g f l a g > E n v i r on m e n t C o n t r o l s Syntax: -Mlfs (32-bit Linux only) link in libraries that enable file I/O to files larger than 2GB (Large File Support). -Mnostartup instructs the linker not to link in the standard startup routine that contains the entry point (_ start) for the program. Note: If you use the –Mnostartup option and do not supply an entry point, the linker issues the following error message: Warning: cannot find entry symbol _ start -Mnostddef instructs the compiler not to predefine any macros to the preprocessor when compiling a C program. -Mnostdlib instructs the linker not to link in the standard libraries libpgftnrtl.a, libm.a, libc.a and libpgc.a in the library directory lib within the standard directory. You can link in your own library with the –l option or specify a library directory with the –L option. Default: For arguments that you do not specify, the default environment option depends on your configuration. Cross-reference: –D, –I, –L, –l, –U -M < p g f l a g > I n l i n in g C o n t r o l s This section describes the –M options that control function inlining. Syntax: -Mextract[=option[, option,...]] Extracts functions from the file indicated on the command line and creates or appends to the specified extract directory where option can be any of: name:func instructs the extractor to extract function func from the file. size:number instructs the extractor to extract functions with number or fewer, statements from the file. Command-line Options 89 lib:filename.ext Use directory filename.ext as the extract directory (required in order to save and re-use inline libraries). If you specify both name and size, the compiler extracts functions that match func, or that have number or fewer statements. For examples of extracting functions, see Chapter 4, Function Inlining. -Minline[=option[, option,...]] This passes options to the function inliner where option can be any of: except:func instructs the inliner to inline all eligible functions except func, a function in the source text. Multiple functions can be listed, comma-separated. [name:]func instructs the inliner to inline the function func. The func name should be a non-numeric string that does not contain a period. You can also use a name: prefix followed by the function name. If name: is specified, what follows is always the name of a function. [lib:]filename.ext instructs the inliner to inline the functions within the library file filename.ext. The compiler assumes that a filename.ext option containing a period is a library file. Create the library file using the –Mextract option. You can also use a lib: prefix followed by the library name. If lib: is specified, no period is necessary in the library name. Functions from the specified library are inlined. If no library is specified, functions are extracted from a temporary library created during an extract prepass. [size:]number instructs the inliner to inline functions with number or fewer statements. You can also use a size: prefix followed by a number. If size: is specified, what follows is always taken as a number. levels:number instructs the inliner to perform number levels of inlining. The default number is 1. If you specify both func and number, the compiler inlines functions that match the function name or have number or fewer statements. For examples of inlining functions, see Chapter 4, Function Inlining.90 Chapter 3 Usage: In the following example, the compiler extracts functions that have 500 or fewer statements from the source file myprog.f and saves them in the file extract.il. $ pgf95 –Mextract=500 –oextract.il myprog.f In the following example, the compiler inlines functions with fewer than approximately 100 statements in the source file myprog.f and writes the executable code in the default output file a.out. $ pgf95 –Minline=size:100 myprog.f Cross-reference: –o -M < p g f l a g > F o r t r a n L a n g u a g e C o n t r o l s This section describes the –M options that affect Fortran language interpretations by the PGI Fortran compilers. These options are only valid to the pgf77, pgf95, and pghpf compiler drivers. Syntax: -Mbackslash the compiler treats the backslash as a normal character, and not as an escape character in quoted strings. -Mnobackslash the compiler recognizes a backslash as an escape character in quoted strings (in accordance with standard C usage). -Mdclchk the compiler requires that all program variables be declared. -Mnodclchk the compiler does not require that all program variables be declared. -Mdefaultunit the compiler treats "*" as a synonym for standard input for reading and standard output for writing. -Mnodefaultunit the compiler treats "*" as a synonym for unit 5 on input and unit 6 on output. -Mdlines the compiler treats lines containing "D" in column 1 as executable statements (ignoring the "D"). -Mnodlines the compiler does not treat lines containing "D" in column 1 as executable statements (does not ignore the "D"). -Mdollar,char char specifies the character to which the compiler maps the dollar sign. The compiler allows the dollar sign in names. Command-line Options 91 -Mextend with –Mextend, the compiler accepts 132-column source code; otherwise it accepts 72-column code. -Mfixed with –Mfixed, the compiler assumes input source files are in FORTRAN 77-style fixed form format. -Mfree with –Mfree, the compiler assumes the input source files are in Fortran 90/95 freeform format. -Miomutex the compiler generates critical section calls around Fortran I/O statements. -Mnoiomutex the compiler does not generate critical section calls around Fortran I/O statements. -Monetrip the compiler forces each DO loop to execute at least once. -Mnoonetrip the compiler does not force each DO loop to execute at least once. This option is useful for programs written for earlier versions of Fortran. -Msave the compiler assumes that all local variables are subject to the SAVE statement. Note that this may allow older Fortran programs to run, but it can greatly reduce performance. -Mnosave the compiler does not assume that all local variables are subject to the SAVE statement. -Mstandard the compiler flags non-ANSI–conforming source code. -Munixlogical directs the compiler to treat logical values as true if the value is non-zero and false if the value is zero (UNIX F77 convention.) When –Munixlogical is enabled, a logical value or test that is non-zero is .TRUE., and a value or test that is zero is .FALSE.. In addition, the value of a logical expression is guaranteed to be one (1) when the result is .TRUE.. -Mnounixlogical directs the compiler to use the VMS convention for logical values for true and false. Even values are true and odd values are false. -Mupcase the compiler allows uppercase letters in identifiers. With –Mupcase, the identifiers "X" and "x" are different, and keywords must be in lower case. This selection affects the linking process: if you compile and link the same source code using –Mupcase on one occasion and –Mnoupcase on another, you may get two different executables (depending on whether the source contains uppercase letters). The standard 92 Chapter 3 libraries are compiled using the default –Mnoupcase. -Mnoupcase the compiler converts all identifiers to lower case. This selection affects the linking process: If you compile and link the same source code using –Mupcase on one occasion and –Mnoupcase on another, you may get two different executables (depending on whether the source contains uppercase letters). The standard libraries are compiled using –Mnoupcase. Default: For arguments that you do not specify, the defaults are as follows: nobackslash noiomutex nodclchk noonetrip nodefaultunit nosave nodlines nounixlogical dollar,_ noupcase -M < p g f l a g > C / C + + L a n g u a g e C o n t r o l s This section describes the –M options that affect C/C++ language interpretations by the PGI C and C++ compilers. These options are only valid to the pgcc and pgCC compiler drivers. Syntax: -Masmkeyword instructs the compiler to allow the asm keyword in C source files. The syntax of the asm statement is as follows: asm("statement"); Where statement is a legal assembly-language statement. The quote marks are required. -Mnoasmkeyword instructs the compiler not to allow the asm keyword in C source files. If you use this option and your program includes the asm keyword, unresolved references will be generated -Mdollar,char char specifies the character to which the compiler maps the dollar sign ($). The PGCC compiler allows the dollar sign in names; ANSI C does not allow the dollar sign in names. -Mfcon instructs the compiler to treat floating-point constants as float data types, instead of double data types. This option can improve the Command-line Options 93 performance of single-precision code. -Mschar specifies signed char characters. The compiler treats "plain" char declarations as signed char. -Msingle do not to convert float parameters to double parameters in nonprototyped functions. This option can result in faster code if your program uses only float parameters. However, since ANSI C specifies that routines must convert float parameters to double parameters in nonprototyped functions, this option results in non-ANSI conformant code. -Mnosingle instructs the compiler to convert float parameters to double parameters in non-prototyped functions. -Muchar instructs the compiler to treat "plain" char declarations as unsigned char. Default: For arguments that you do not specify, the defaults are as follows: noasmkeyword nosingle dollar,_ schar Usage: In this example, the compiler allows the asm keyword in the source file. $ pgcc -Masmkeyword myprog.c In the following example, the compiler maps the dollar sign to the dot character. $ pgcc -Mdollar,. myprog.c In the following example, the compiler treats floating-point constants as float values. $ pgcc -Mfcon myprog.c In the following example, the compiler does not convert float parameters to double parameters. $ pgcc -Msingle myprog.c Without –Muchar or with –Mschar, the variable ch is a signed character: char ch; signed char sch; 94 Chapter 3 If –Muchar is specified on the command line: $ pgcc -Muchar myprog.c char ch above is equivalent to: unsigned char ch; -M < p g f l a g > O p t i m i z a t i o n C o n t r o l s Syntax: -Mcache_align Align unconstrained objects of length greater than or equal to 16 bytes on cache-line boundaries. An unconstrained object is a data object that is not a member of an aggregate structure or common block. This option does not affect the alignment of allocatable or automatic arrays. Note: To effect cache-line alignment of stack-based local variables, the main program or function must be compiled with –Mcache_align. -Mconcur[=option [,option,...]] Instructs the compiler to enable auto-concurrentization of loops. If -Mconcur is specified, multiple processors will be used to execute loops that the compiler determines to be parallelizable. Where option is one of the following: [no]altcode:n Instructs the parallelizer to generate alternate serial code for parallelized loops. If altcode is specified without arguments, the parallelizer determines an appropriate cutoff length and generates serial code to be executed whenever the loop count is less than or equal to that length. If altcode:n is specified, the serial altcode is executed whenever the loop count is less than or equal to n. If noaltcode is specified, the parallelized version of the loop is always executed regardless of the loop count. cncall Calls in parallel loops are safe to parallelize. Loops containing calls are candidates for parallelization. Also, no minimum loop count threshold must be satisfied before parallelization will Command-line Options 95 occur, and last values of scalars are assumed to be safe. dist:block Parallelize with block distribution (this is the default). Contiguous blocks of iterations of a parallelizable loop are assigned to the available processors. dist:cyclic Parallelize with cyclic distribution. The outermost parallelizable loop in any loop nest is parallelized. If a parallelized loop is innermost, its iterations are allocated to processors cyclically. For example, if there are 3 processors executing a loop, processor 0 performs iterations 0, 3, 6, etc.; processor 1 performs iterations 1, 4, 7, etc.; and processor 2 performs iterations 2, 5, 8, etc. [no]innermost Enable parallelization of innermost loops. The default is to not parallelize innermost loops, since it is usually not profitable on dual-core processors. noassoc Disables parallelization of loops with reductions. When linking, the -Mconcur switch must be specified or unresolved references will result. The NCPUS environment variable controls how many processors or cores are used to execute parallelized loops. Note: this option applies only on shared-memory multi-processor (SMP) or dual-core processor-based systems. -Mcray[=option[,option,...]] (pgf77 and pgf95 only) Force Cray Fortran (CF77) compatibility with respect to the listed options. Possible values of option include: pointer for purposes of optimization, it is assumed that pointerbased variables do not overlay the storage of any other variable. -Mdepchk instructs the compiler to assume unresolved data dependencies actually conflict. -Mnodepchk instructs the compiler to assume potential data dependencies do not conflict. However, if data dependencies exist, this option can produce 96 Chapter 3 incorrect code. -Mfprelaxed instructs the compiler to use relaxed precision in the calculation of some intrinsic functions. Can result in improved performance at the expense of numerical accuracy. -Mnofprelaxed (default) instructs the compiler not to use relaxed precision in the calculation of intrinsic functions. -Mi4 (pgf77 and pgf95 only) the compiler treats INTEGER variables as INTEGER*4. -Mipa=[,[,…]] Pass options to the interprocedural analyzer. Note: –Mipa implies –O2, and the minimum optimization level that can be specified in combination with –Mipa is –O2. For example, if you specify –Mipa –O1 on the command line, the optimization level will automatically be elevated to –O2 by the compiler driver. It is typical and recommended to use – Mipa=fast. Many of the following sub-options can be prefaced with no, which reverses or disables the effect of the sub-option if it’s included in an aggregate sub-option like –Mipa=fast. The choices of option are: [no]align recognize when targets of a pointer dummy are aligned; default is noalign. [no]arg remove arguments replaced by const, ptr; default is noarg. [no]cg generate call graph information for viewing using the pgicg command-line utility; default is nocg. [no]const perform interprocedural constant propagation; default is const. Command-line Options 97 except: used with inline to specify functions which should not be inlined; default is to inline all eligible functions according to internally defined heuristics. [no]f90ptr F90/F95 pointer disambiguation across calls; default is nof90ptr fast choose IPA options generally optimal for the target. Use –help to see the settings for –Mipa=fast on a given target. force force all objects to re-compile regardless of whether IPA information has changed. [no]globals optimize references to global variables; default is noglobals. inline[:n] perform automatic function inlining. If the optional :n is provided, limit inlining to at most n levels. IPA-based function inlining is performed from leaf routines upward. ipofile save IPA information in a .ipo file rather than incorporating it into the object file. [no]keepobj keep the optimized object files, using file name mangling, to reduce re-compile time in subsequent builds default is keepobj. [no]libinline allow inlining of routines from libraries; implies –Mipa=inline; default is nolibinline. [no]libopt allow recompiling and optimization of routines from libraries using IPA information; default is nolibopt. [no]localarg equivalent to arg plus externalization of local pointer targets; default is nolocalarg. main: specify a function to appear as a global entry point; may appear multiple times; disables linking. [no]ptr enable pointer disambiguation across procedure calls; default is noptr. 98 Chapter 3 [no]pure pure function detection; default is nopure. required return an error condition if IPA is inhibited for any reason, rather than the default behavior of linking without IPA optimization. safe:[|] declares that the named function, or all functions in the named library, are safe; a safe procedure does not call back into the known procedures and does not change any known global variables. Without –Mipa=safe, any unknown procedures will cause IPA to fail. [no]safeall declares that all unknown procedures are safe; see –Mipa=safe; default is nosafeall. [no]shape perform Fortran 90 array shape propagation; default is noshape. summary only collect IPA summary information when compiling; this prevents IPA optimization of this file, but allows optimization for other files linked with this file. [no]vestigial remove uncalled (vestigial) functions; default is novestigial. -Mlre[=array | assoc | noassoc] Enables loop-carried redundancy elimination, an optimization that can reduce the number of arithmetic operations and memory references in loops. array treat individual array element references as candidates for possible loop-carried redundancy elimination. The default is to eliminate only redundant expressions involving two or more operands. assoc allow expression re-association; specifying this sub-option can increase opportunities for loop-carried redundancy elimination Command-line Options 99 but may alter numerical results. noassoc disallow expression re-association. -Mnolre Disables loop-carried redundancy elimination. -Mnoframe Eliminates operations that set up a true stack frame pointer for every function. With this option enabled, you cannot perform a traceback on the generated code and you cannot access local variables. -Mnoi4 (pgf77 and pgf95 only) the compiler treats INTEGER variables as INTEGER*2. -Mpfi generate profile-feedback instrumentation; this includes extra code to collect run-time statistics and dump them to a trace file for use in a subsequent compilation. –Mpfi must also appear when the program is linked. When the resulting program is executed, a profile feedback trace file pgfi.out is generated in the current working directory; see –Mpfo. NOTE: compiling and linking with –Mpfi adds significant runtime overhead to almost any executable; you should use executables compiled with –Mpfi only for execution of training runs. -Mpfo enable profile-feedback optimizations; requires the presence of a pgfi.out profile-feedback trace file in the current working directory. See –Mpfi. -Mprefetch[=option [,option...]] enables generation of prefetch instructions on processors where they are supported. Possible values for option include: d:m set the fetch-ahead distance for prefetch instructions to m cache lines. n:p set the maximum number of prefetch instructions to generate for a given loop to p. nta use the prefetchnta instruction. plain use the prefetch instruction (default). t0 use the prefetcht0 instruction. w use the AMD-specific prefetchw instruction. -Mnoprefetch Disables generation of prefetch instructions. 100 Chapter 3 -Mr8 (pgf77, pgf95 and pghpf only) the compiler promotes REAL variables and constants to DOUBLE PRECISION variables and constants, respectively. DOUBLE PRECISION elements are 8 bytes in length. -Mnor8 (pgf77, pgf95 and pghpf only) the compiler does not promote REAL variables and constants to DOUBLE PRECISION. REAL variables will be single precision (4 bytes in length). -Mr8intrinsics (pgf77, and pgf95 only) the compiler treats the intrinsics CMPLX and REAL as DCMPLX and DBLE, respectively. -Mnor8intrinsics (pgf77, and pgf95 only) the compiler does not promote the intrinsics CMPLX and REAL to DCMPLX and DBLE, respectively. -Msafeptr[=option[,option,...]] (pgcc and pgCC only) instructs the C/C++ compiler to override data dependencies between pointers of a given storage class. Possible values of option include: all assume all pointers and arrays are independent and safe for aggressive optimizations, and in particular that no pointers or arrays overlap or conflict with each other. arg instructs the compiler that arrays and pointers are treated with the same copyin and copyout semantics as Fortran dummy arguments. global instructs the compiler that global or external pointers and arrays do not overlap or conflict with each other and are independent. local/auto instructs the compiler that local pointers and arrays do not overlap or conflict with each other and are independent. static instructs the compiler that static pointers and arrays do not overlap or conflict with each other and are independent. -Mscalarsse Use SSE/SSE2 instructions to perform scalar floating-point arithmetic (this option is valid only on –tp {p7 | k8-32 | k8-64} targets). -Mnoscalarsse Do not use SSE/SSE2 instructions to perform scalar floating-point arithmetic; use x87 instructions instead (this option is not valid in Command-line Options 101 combination with the –tp k8-64 option). -Msmart instructs the compiler driver to invoke an AMD64-specific post-pass assembly optimization utility. -Mnosmart instructs the compiler not to invoke an AMD64-specific post-pass assembly optimization utility. -Munroll[=option [,option...]] invokes the loop unroller. This also sets the optimization level to 2 if the level is set to less than 2. The option is one of the following: c:m instructs the compiler to completely unroll loops with a constant loop count less than or equal to m, a supplied constant. If this value is not supplied, the m count is set to 4. n:u instructs the compiler to unroll u times, a loop that is not completely unrolled, or has a non-constant loop count. If u is not supplied, the unroller computes the number of times a candidate loop is unrolled. -Mnounroll instructs the compiler not to unroll loops. -M[no]vect [=option [,option,...]] (disable) enable the code vectorizer, where option is one of the following: altcode Instructs the vectorizer to generate alternate code (altcode) for vectorized loops when appropriate. For each vectorized loop the compiler decides whether to generate altcode and what type or types to generate, which may be any or all of: altcode without iteration peeling, altcode with non-temporal stores and other data cache optimizations, and altcode based on array alignments calculated dynamically at runtime. The compiler also determines suitable loop count and array alignment conditions for executing the alcode. This option is enabled by default. noaltcode This disables alternate code generation for vectorized loops. assoc Instructs the vectorizer to enable certain associativity conversions that can change the results of a computation 102 Chapter 3 due to roundoff error. A typical optimization is to change an arithmetic operation to an arithmetic operation that is mathematically correct, but can be computationally different, due to round-off error noassoc Instructs the vectorizer to disable associativity conversions. cachesize:n Instructs the vectorizer, when performing cache tiling optimizations, to assume a cache size of n. The default is n = 262144. nosizelimit Generate vector code for all loops where possible regardless of the number of statements in the loop. This overrides a heuristic in the vectorizer that ordinarily prevents vectorization of loops with a number of statements that exceeds a certain threshold. smallvect[:n] Instructs the vectorizer to assume that the maximum vector length is less than or equal to n. The vectorizer uses this information to eliminate generation of the stripmine loop for vectorized loops wherever possible. If the size n is omitted, the default is 100. Note: No space is allowed on either side of the colon (:). sse Instructs the vectorizer to search for vectorizable loops and, where possible, make use of SSE, SSE2 and prefetch instructions. -Mnovect instructs the compiler not to perform vectorization; can be used to override a previous instance of –Mvect on the command-line, in particular for cases where –Mvect is included in an aggregate option such as –fastsse. -Mnovintr instructs the compiler not to perform idiom recognition or introduce calls to hand-optimized vector functions.Command-line Options 103 Default: For arguments that you do not specify, the default optimization control options are as follows: depchk i4 nofprelaxed noipa nolre noprefetch nounroll novect nor8 nor8intrinsics If you do not supply an option to –Mvect, the compiler uses defaults that are dependent upon the target system. Usage: In this example, the compiler invokes the vectorizer with use of packed SSE instructions enabled. $ pgf95 –Mvect=sse –Mcache_align myprog.f Cross-reference: –g, –O -M < p g f l a g > M i s c e l l an e o u s C o n t r o l s Syntax: -Manno annotate the generated assembly code with source code when either the –S or –Mkeepasm options are used. -Mbounds enables array bounds checking. If an array is an assumed size array, the bounds checking only applies to the lower bound. If an array bounds violation occurs during execution, an error message describing the error is printed and the program terminates. The text of the error message includes the name of the array, the location where the error occurred (the source file and the line number in the source), and information about the out of bounds subscript (its value, its lower and upper bounds, and its dimension). For example: PGFTN-F-Subscript out of range for array a (a.f: 2) subscript=3, lower bound=1, upper bound=2, dimension=2 -Mnobounds disables array bounds checking. -Mbyteswapio swap byte-order from big-endian to little-endian or vice versa upon input/output of Fortran unformatted data files. 104 Chapter 3 -Mchkfpstk instructs the compiler to check for internal consistency of the x87 floatingpoint stack in the prologue of a function and after returning from a function or subroutine call. Floating-point stack corruption may occur in many ways, one of which is Fortran code calling floating-point functions as subroutines (i.e., with the CALL statement). If the PGI_CONTINUE environment variable is set upon execution of a program compiled with – Mchkfpstk, the stack will be automatically cleaned up and execution will continue. There is a performance penalty associated with the stack cleanup. If PGI_CONTINUE is set to verbose, the stack will be automatically cleaned up and execution will continue after printing of a warning message. -Mchkptr instructs the compiler to check for pointers that are de-referenced while initialized to NULL (pgf95 and pghpf only). -Mchkstk instructs the compiler to check the stack for available space in the prologue of a function and before the start of a parallel region. Prints a warning message and aborts the program gracefully if stack space is insufficient. Useful when many local and private variables are declared in an OpenMP program. –Mcpp[=option [,option,...]] run the PGI cpp-like pre-processor without execution of any subsequent compilation steps. This option is useful for generating dependence information to be included in makefiles. option is one or more of the following (Note: only one of the m, md, mm or mmd options can be present; if multiple of these options are listed, the last one listed is accepted and the others are ignored): m print makefile dependencies to stdout. md print makefile dependencies to filename.d, where filename is the root name of the input file being processed. mm print makefile dependencies to stdout, ignoring system include files. mmd print makefile dependencies to filename.d, where filename is the root name of the input file being processed, ignoring system include files. Command-line Options 105 [no]comment (don’t) retain comments in preprocessed output. [suffix:] use as the suffix of the output file containing makefile dependencies. -Mdll (Windows only) link with the DLL versions of the runtime libraries. This flag is required when linking with any DLL built by any of The Portland Group compilers. -Mgccbug[s] match the behavior of certain gcc bugs. -Minfo[=option [,option,...]] instructs the compiler to produce information on standard error, where option is one of the following: all instructs the compiler to produce all available -Minfo information. inline instructs the compiler to display information about extracted or inlined functions. This option is not useful without either the –Mextract or –Minline option. ipa instructs the compiler to display information about interprocedural optimizations. loop instructs the compiler to display information about loops, such as information on vectorization. opt instructs the compiler to display information about optimization. mp instructs the compiler to display information about parallelization. time instructs the compiler to display compilation statistics. unroll instructs the compiler to display information about loop unrolling. -Mneginfo[=option [,option,...]] instructs the compiler to produce information on standard error, where option is one of the following: 106 Chapter 3 all instructs the compiler to produce all available information on why various optimizations are not performed. concur instructs the compiler to produce all available information on why loops are not automatically parallelized. In particular, if a loop is not parallelized due to potential data dependence, the variable(s) that cause the potential dependence will be listed in the -Mneginfo messages. loop instructs the compiler to produce information on why memory hierarchy optimizations on loops are not performed. -Minform,level instructs the compiler to display error messages at the specified and higher levels, where level is one of the following: fatal instructs the compiler to display fatal error messages. severe instructs the compiler to display severe and fatal error messages. warn instructs the compiler to display warning, severe and fatal error messages. inform instructs the compiler to display all error messages (inform, warn, severe and fatal). -Mkeepasm instructs the compiler to keep the assembly file as compilation continues. Normally, the assembler deletes this file when it is finished. The assembly file has the same filename as the source file, but with a .s extension. -Mlist instructs the compiler to create a listing file. The listing file is filename.lst, where the name of the source file is filename.f. -Mnolist the compiler does not create a listing file. This is the default. -Mmakedll (Windows only) generate a dynamic link library (DLL). -Mnoopenmp when used in combination with the -mp option, causes the compiler to ignore OpenMP parallelization directives or pragmas, but still process SGI-style parallelization directives or pragmas. -Mnosgimp when used in combination with the -mp option, causes the compiler to ignore SGI-style parallelization directives or pragmas, but still process Command-line Options 107 OpenMP parallelization directives or pragmas. -Mnopgdllmain (Windows only) do not link the module containing the default DllMain() into the DLL. This flag applies to building DLLs with the PGF95 and PGHPF compilers. If you want to replace the default DllMain() routine with a custom DllMain(), use this flag and add the object containing the custom DllMain() to the link line. The latest version of the default DllMain() used by PGF95 and PGHPF is included in the Release Notes for each release; the PGF95- and PGHPF-specific code in this routine must be incorporated into the custom version of DllMain() to ensure the appropriate function of your DLL. -Mpreprocess perform cpp-like pre-processing on assembly and Fortran input source files. Default: For arguments that you do not specify, the default miscellaneous options are as follows: inform warn nolist nobounds Usage: In the following example, the compiler includes Fortran source code with the assembly code. $ pgf95 –Manno –S myprog.f In the following example, the compiler displays information about inlined functions with fewer than approximately 20 source lines in the source file myprog.f. $ pgf95 –Minfo=inline –Minline=20 myprog.f In the following example, the assembler does not delete the assembly file myprog.s after the assembly pass. $ pgf95 –Mkeepasm myprog.f In the following example, the compiler creates the listing file myprog.lst. $ pgf95 –Mlist myprog.f In the following example, array bounds checking is enabled. 108 Chapter 3 $ pgf95 –Mbounds myprog.f Cross-reference: –m, –S, –V, –v -m c m o d e l = m e d i u m (For use only on 64-bit Linux targets) Generate code for the medium memory model in the linux86-64 execution environment. Implies –Mlarge_arrays. The default small memory model of the linux86-64 environment limits the combined area for a user’s object or executable to 1GB, with the Linux kernel managing usage of the second 1GB of address for system routines, shared libraries, stacks, etc. Programs are started at a fixed address, and the program can use a single instruction to make most memory references. The medium memory model allows for larger than 2GB data areas, or .bss sections. Program units compiled using either –mcmodel=medium or –fpic require additional instructions to reference memory. The effect on performance is a function of the data-use of the application. The –mcmodel=medium switch must be used at both compile time and link time to create 64-bit executables. Program units compiled for the default small memory model can be linked into medium memory model executables as long as they are compiled –fpic, or position-independent. The linux86-64 environment provides static libxxx.a archive libraries that are built with and without –fpic, and dynamic libxxx.so shared object libraries that are compiled –fpic. The –mcmodel=medium linkswitch implies the –fpic switch and will utilize the shared libraries by default. Similarly, the $PGI/linux86-64//lib directory contains the libraries for building small memory model codes, and the $PGI/linux86-64//libso directory contains shared libraries for building –mcmodel=medium and –fpic executables. Note: It appears from the GNU tools and documentation that creation of medium memory model shared libraries is not supported. However, you can create static archive libraries (.a) that are –fpic. Default: The compiler generates code for the small memory model. Usage: The following command line requests position independent code be generated, and the –mcmodel=medium option be passed to the assembler and linker: $ pgf95 –mcmodel=medium myprog.f -m o d u l e < m o d u l e d i r > Use the -module option to specify a particular directory in which generated intermediate .modCommand-line Options 109 files should be placed. If the -module option is present, and USE statements are present in a compiled program unit, will search for .mod intermediate files prior to the search in the default (local) directory. Default: The compiler places .mod files in the current working directory, and searches only in the current working directory for pre-compiled intermediate .mod files. Usage: The following command line requests that any intermediate module file produced during compilation of myprog.f be placed in the directory mymods (in particular, the file ./mymods/myprog.mod will be used): $ pgf95 -module mymods myprog.f -m p [ = a l i g n ] Use the -mp option to instruct the compiler to interpret user-inserted OpenMP shared-memory parallel programming directives and generate an executable file which will utilize multiple processors in a shared-memory parallel system. See Chapter 5, OpenMP Directives for Fortran and Chapter 6, OpenMP Pragmas for C and C++, for a detailed description of this programming model and the associated directives and pragmas. The align sub-option forces loop iterations to be allocated to OpenMP processes using an algorithm that maximizes alignment of vector subsections in loops that are both parallelized and vectorized for SSE. This can improve performance in program units that include many such loops. It can result in load-balancing problems that significantly decrease performance in program units with relatively short loops that contain a large amount of work in each iteration. Default: The compiler ignores user-inserted shared-memory parallel programming directives and pragmas. Usage: The following command line requests processing of any shared-memory directives present in myprog.f: $ pgf95 -mp myprog.f Cross-reference: –Mconcur and –Mvect -m s l i b s (Win32 only) Use the -mslibs option to instruct the compiler to use the Microsoft linker and include files, and link against the Microsoft Visual C++ libraries. Microsoft Visual C++ must be 110 Chapter 3 installed in order to use this switch. This switch can be used to link Visual C++-compiled program units into PGI main programs on Windows. Default: The compiler uses the PGI-supplied linker and include files and links against PGIsupplied libraries. Cross-reference: –msvcrt -m s v c r t (Win32 only) Use the -msvcrt option to instruct the compiler to use Microsoft’s msvcrt.dll at runtime rather than the default crtdll.dll. These files contain the Microsoft C runtime library and the default mingw32 C runtime library respectively. It is recommended that you use the –msvcrt option in combination with the –mslibs option. Default: The compiler uses crtdll.dll at runtime. Cross-reference: –mslibs -O < l e v e l > Invokes code optimization at the specified level. Syntax: –O [level] Where level is one of the following: 0 creates a basic block for each statement. Neither scheduling nor global optimization is done. To specify this level, supply a 0 (zero) argument to the –O option. 1 schedules within basic blocks and performs some register allocations, but does no global optimization. 2 performs all level-1 optimizations, and also performs global scalar optimizations such as induction variable elimination and loop invariant movement. 3 level-three specifies aggressive global optimization. This level performs all level-one and level-two optimizations and enables more aggressive Command-line Options 111 hoisting and scalar replacement optimizations that may or may not be profitable. Default: Table 3-4 shows the interaction between the –O option, –g option, and –Mvect options. Table 3-4: Optimization and –O, –g, –Mvect, and –Mconcur Options Optimize Option Debug Option –M Option Optimization Level none none none 1 none none –Mvect 2 none none –Mconcur 2 none –g none 0 –O none or –g none 2 –Olevel none or –g none level –Olevel < 2 none or –g –Mvect 2 –Olevel < 2 none or –g –Mconcur 2 Unoptimized code compiled using the option –O0 can be significantly slower than code generated at other optimization levels. Like the –Mvect option, the –Munroll option sets the optimization level to level-2 if no –O or –g options are supplied. For more information on optimization, see Chapter 2, Optimization & Parallelization. Usage: In the following example, since no optimization level is specified and a –O option is specified, the compiler sets the optimization to level-2. $ pgf95 -O myprog.f Cross-reference: –g, –M -o Names the executable file. Use the –o option to specify the filename of the compiler object file. The final output is the result of linking. Syntax: –o filename112 Chapter 3 Where filename is the name of the file for the compilation output. The filename must not have a .f extension. Default: The compiler creates executable filenames as needed. If you do not specify the –o option, the default filename is the linker output file a.out. Usage: In the following example, the executable file is myprog instead of the default a.out. $ pgf95 myprog.f -o myprog Cross-reference: –c ,–E, –F, –S -p c (–tp px/p5/p6/piii targets only) The –pc option can be used to control the precision of operations performed using the x87 floating point unit, and their representation on the x87 floating point stack. Syntax: –pc { 32 | 64 | 80 } The x87 architecture implements a floating-point stack using 8 80-bit registers. Each register uses bits 0-63 as the significand, bits 64-78 for the exponent, and bit 79 is the sign bit. This 80-bit real format is the default format (called the extended format). When values are loaded into the floating point stack they are automatically converted into extended real format. The precision of the floating point stack can be controlled, however, by setting the precision control bits (bits 8 and 9) of the floating control word appropriately. In this way, you can explicitly set the precision to standard IEEE double-precision using 64 bits, or to single precision using 32 bits. * The default precision is system dependent. To alter the precision in a given program unit, the main program must be compiled with the same -pc option. The command line option –pc val lets the programmer set the compiler’s precision preference. Valid values for val are: 32 single precision 64 double precision 80 extended precision * According to Intel documentation, this only affects the x87 operations of add, subtract, multiply, divide, and square root. In particular, it does not appear to affect the x87 transcendental instructions. Command-line Options 113 Extended Precision Option – Operations performed exclusively on the floating-point stack using extended precision, without storing into or loading from memory, can cause problems with accumulated values within the extra 16 bits of extended precision values. This can lead to answers, when rounded, that do not match expected results. For example, if the argument to sin is the result of previous calculations performed on the floating-point stack, then an 80-bit value used instead of a 64-bit value can result in slight discrepancies. Results can even change sign due to the sin curve being too close to an x-intercept value when evaluated. To maintain consistency in this case, you can assure that the compiler generates code that calls a function. According to the x86 ABI, a function call must push its arguments on the stack (in this way memory is guaranteed to be accessed, even if the argument is an actual constant.) Thus, even if the called function simply performs the inline expansion, using the function call as a wrapper to sin has the effect of trimming the argument precision down to the expected size. Using the -Mnobuiltin option on the command line for C accomplishes this task by resolving all math routines in the library libm, performing a function call of necessity. The other method of generating a function call for math routines, but one that may still produce the inline instructions, is by using the -Kieee switch. A second example illustrates the precision control problem using a section of code to determine machine precision: program find_precision w = 1.0 100 w=w+w y=w+1 z=y-w if (z .gt. 0) goto 100 C now w is just big enough that |((w+1)-w)-1| >= 1 ... print*,w end In this case, where the variables are implicitly real*4, operations are performed on the floatingpoint stack where optimization removed unnecessary loads and stores from memory. The general case of copy propagation being performed follows this pattern: a = x y = 2.0 + a Instead of storing x into a, then loading a to perform the addition, the value of x can be left on the floating-point stack and added to 2.0. Thus, memory accesses in some cases can be avoided, leaving answers in the extended real format. If copy propagation is disabled, stores of all left-hand 114 Chapter 3 sides will be performed automatically and reloaded when needed. This will have the effect of rounding any results to their declared sizes. For the above program, w has a value of 1.8446744E+19 when executed using default (extended) precision. If, however, -Kieee is set, the value becomes 1.6777216E+07 (single precision.) This difference is due to the fact that -Kieee disables copy propagation, so all intermediate results are stored into memory, then reloaded when needed. Copy propagation is only disabled for floatingpoint operations, not integer. With this particular example, setting the -pc switch will also adjust the result. The switch -Kieee also has the effect of making function calls to perform all transcendental operations. Although the function still produces the x86 machine instruction for computation (unless in C the -Mnobuiltin switch is set), arguments are passed on the stack, which results in a memory store and load. Finally, -Kieee also disables reciprocal division for constant divisors. That is, for a/b with unknown a and constant b, the expression is usually converted at compile time to a*(1/b), thus turning an expensive divide into a relatively fast scalar multiplication. However, numerical discrepancies can occur when this optimization is used. Understanding and correctly using the -pc, -Mnobuiltin, and -Kieee switches should enable you to produce the desired and expected precision for calculations which utilize floating-point operations. Usage: $ pgf95 –pc 64 myprog.c -p g (Linux only) Instructs the compiler to instrument the generated executable for gprof-style samplebased profiling. Must be used at both the compile and link steps. A gmon.out style trace is generated when the resulting program is executed, and and can be analyzed using gprof or pgprof. Syntax: –pg Default: The compiler does not instrument the generated executable for gprof-style profiling. Command-line Options 115 -Q Selects variations for compilation. There are four uses for the –Q option. Syntax: –Qdirdirectory The first variety, using the dir keyword, lets you supply a directory parameter that indicates the directory where the compiler driver is located. –Qoptionprog,opt The second variety, using the option keyword, lets you supply the option opt to the program prog. The prog parameter can be one of pgftn, as, or ld. –Qpathpathname The third –Q variety, using the path keyword, lets you supply an additional pathname to the search path for the compiler’s required .o files. –Qproducesourcetype The fourth –Q variety, using the produce keyword, lets you choose a stop-after location for the compilation based on the supplied sourcetype parameter. Valid sourcetypes are: .i, .c, .s and .o. These indicate respectively, stop-after preprocessing, compiling, assembling, or linking. Usage: The following examples show the different –Q options. $ pgf95 –Qproduce .s hello.f $ pgf95 –Qoption ld,-s hello.f $ pgf95 –Qpath /home/test hello.f $ pgf95 –Qdir /home/comp/new hello.f Cross-reference: –p -R < d i r e c t o r y > Valid only on Linux and is passed to the linker. Instructs the linker to hard-code the pathname 116 Chapter 3 into the search path for generated shared object (dynamically linked library) files. Note that there cannot be a space between R and . Cross-reference: –fpic, –shared, –G -r 4 a n d -r 8 Interpret DOUBLE PRECISION variables as REAL (–r4) or REAL variables as DOUBLE PRECISION (–r8). Usage: $ pgf95 –r4 myprog.f Cross-reference: –i2, –i4, -i8, -nor8 -r c Specifies the name of the driver startup configuration file. If the file or pathname supplied is not a full pathname, the path for the configuration file loaded is relative to the $DRIVER path (the path of the currently executing driver). If a full pathname is supplied, that file is used for the driver configuration file. Syntax: –rc [path] filename Where path is either a relative pathname, relative to the value of $DRIVER, or a full pathname beginning with "/". Filename is the driver configuration file. Default: The driver uses the configuration file .pgirc. Usage: In the following example, the file .pgf95rctest, relative to /usr/pgi/linux86/bin, the value of $DRIVER, is the driver configuration file. $ pgf95 -rc .pgf95rctest myprog.f Cross-reference: –showCommand-line Options 117 -S Stops compilation after the compiling phase and writes the assembly-language output to the file filename.s, where the input file is filename.f. Default: The compiler produces an executable file. Usage: In this example, pgf95 produces the file myprog.s in the current directory. $ pgf95 -S myprog.f Cross-reference: –c, –E, –F, –Mkeepasm, –o -s h a r e d Valid only on Linux and is passed to the linker. Instructs the linker to produce a shared object (dynamically linked library) file. Cross-reference: –fpic, –G, –R -s h o w Produce driver help information describing the current driver configuration. Usage: In the following example, the driver displays configuration information to the standard output after processing the driver configuration file. $ pgf95 -show myprog.f Cross-reference: –V , –v, –###, –help, –rc -s i l e n t Do not print warning messages. Usage: In the following example, the driver does not display warning messages. $ pgf95 -silent myprog.f Cross-reference: -v, -V, -w118 Chapter 3 -t i m e Print execution times for various compilation steps. Usage: In the following example, pgf95 prints the execution times for the various compilation steps. $ pgf95 -time myprog.f Cross-reference: –# -t p Set the target architecture. By default, the PGI compilers produce code specifically targeted to the type of processor on which the compilation is performed. In particular, the default is to use all supported instructions wherever possible when compiling on a given system. As a result, executables created on a given system may not be useable on previous generation systems (for example, executables created on a Pentium 4 may fail to execute on a Pentium III or Pentium II). Processor-specific optimizations can be specified or limited explicitly by using the -tp option. In this way, it is possible to create executables that are usable on previous generation systems. With the exception of k8-64, k8-64e, p7-64, and x64, any of these sub-options are valid on any x86 or x64 processor-based system. The k8-64, k8-64e, p7-64 and x64 options are valid only on x64 processor-based systems. The –tp x64 option is used to generate unified binary object and executable files. The –tp k8-64 and –tp k8-64e options result in generation of code supported on and optimized for AMD x64 processors, while the –tp p7-64 option results in generation of code that is supported on and optimized for Intel x64 processors. Performance of k8-64 or k8-64e code executed on Intel x64 processors, or of p7-64 code executed on AMD x64 processors, can often be significantly less than that obtained with a native binary. The –tp x64 option results in generation of unified binary object and executable files which are supported on and include optimized code sequences for both AMD and Intel x64 processors. Following is a list of possible sub-options to –tp and the processors they are intended to target: k8-32 generate 32-bit code for AMD Athlon64, AMD Opteron and compatible processors. k8-64 generate 64-bit code for AMD Athlon64, AMD Opteron and compatible Command-line Options 119 processors. k8-64e generate 64-bit code for AMD Opteron Revision E, AMD Turion, and compatible processors. p6 generate 32-bit code for Pentium Pro/II/III and AthlonXP compatible processors. p7 generate 32-bit code for Pentium 4 and compatible processors. p7-64 generate 64-bit code for Intel P4/Xeon EM64T and compatible processors. piii generate 32-bit code for Pentium III and compatible processors, including support for single-precision vector code using SSE instructions. px generate 32-bit code that is useable on any x86 processor-based system. x64 generate 64-bit unified binary code including full optimizations and support for both AMD and Intel x64 processors. See Table P-2 for a concise list of the features of these processors that distinguish them as separate targets when using the PGI compilers and tools. Syntax: -tp {k8-32 | k8-64 | k8-64e | p6 | p7 | p7-64 | piii | px | x64} Usage: In the following example, pgf95 sets the target architecture to Pentium 4 EM64T: $ pgf95 -tp p7-64 myprog.f Default: The default style of code generation is auto-selected depending on the type of processor on which compilation is performed. The –tp x64 style of unified binary code generation is only enabled by an explicit –tp x64 option. -U Undefines a preprocessor macro. Use the –U option or the #undef preprocessor directive to undefine macros. Syntax: –Usymbol 120 Chapter 3 Where symbol is a symbolic name. Usage: The following examples undefine the macro test. $ pgf95 –Utest myprog.F $ pgf95 –Dtest –Utest myprog.F Cross-reference: –D,–Mnostdde. -V [ r e l e a s e _ n u m b e r ] Displays additional information, including version messages. If a release_number is appended, the compiler driver will attempt to compile using the specified release instead of the default release. There can be no space between –V and release_number. The specified release must be co-installed with the default release, and must have a release number greater than or equal to 4.1 (the first release for which this functionality is supported). Usage: The following command-line shows the output using the -V option. % pgf95 -V myprog.f The following command-line causes PGF95 to compile using the 5.2 release instead of the default: % pgcc –V5.2 myprog.c Cross-reference: –Minfo, –v -v Use the –v option to display the invocations of the compiler, assembler, and linker. These invocations are command lines created by the compiler driver from the files and the –W options you specify on the compiler command-line. Default: The compiler does not display individual phase invocations. Cross-reference: –Minfo, –V Command-line Options 121 -W Passes arguments to a specific phase. Use the –W option to specify options for the assembler, compiler or linker. Note: A given PGI compiler command invokes the compiler driver, which parses the commandline and generates the appropriate commands for the compiler, assembler and linker. Syntax: –W {0 | a | l },option[, option...] Where: 0 (the number zero) specifies the compiler. a specifies the assembler. l (lowercase letter l) specifies the linker. option is a string that is passed to and interpreted by the compiler, assembler or linker. Options separated by commas are passed as separate command line arguments. Note: You cannot have a space between the –W and the single-letter pass identifier, between the identifier and the comma, or between the comma and the option. Usage: In the following example the linker loads the text segment at address 0xffc00000 and the data segment at address 0xffe00000. $ pgf95 -Wl,-k,-t,0xffc00000,-d,0xffe00000 myprog.f -w Do not print warning messages. 122 Chapter 3 3.2 C and C++ -specific Compiler Options The following options are specific to PGCC C and/or C++. -A (pgCC only) Using this option, the PGC++ compiler accepts code conforming to the proposed ANSI C++ standard. It issues errors for non-conforming code. Default: By default, the compiler accepts code conforming to the standard C++ Annotated Reference Manual. Usage: The following command-line requests ANSI conforming C++. $ pgCC –A hello.cc Cross-references: –b and +p. - -[ n o _ ] a l t e r n at i v e _ t o k e n s (pgCC only) Enable or disable recognition of alternative tokens. These are tokens that make it possible to write C++ without the use of the , , [, ], #, &, , ^, and characters. The alternative tokens include the operator keywords (e.g., and, bitand, etc.) and digraphs. The default behavior is --no_alternative_tokens. -B (pgcc and pgCC only) Enable use of C++ style comments starting with // in C program units. Default: The PGCC ANSI and K&R C compiler does not allow C++ style comments. Usage: In the following example the compiler accepts C++ style comments. $ pgcc -B myprog.cc Command-line Options 123 -b (pgCC only) Enable compilation of C++ with cfront 2.1 compatibility. This causes the compiler to accept language constructs that, while not part of the C++ language definition, are accepted by the AT&T C++ Language System (cfront release 2.1). This option also enables acceptance of anachronisms. Default: The compiler does not accept cfront language constructs that are not part of the C++ language definition. Usage: In the following example the compiler accepts cfront constructs. $ pgCC -b myprog.cc Cross-references: ––cfront2.1, –b3 , ––cfront3.0, +p, –A -b 3 (pgCC only) Enable compilation of C++ with cfront 3.0 compatibility. This causes the compiler to accept language constructs that, while not part of the C++ language definition, are accepted by the AT&T C++ Language System (cfront release 3.0). This option also enables acceptance of anachronisms. Default: The compiler does not accept cfront language constructs that are not part of the C++ language definition. Usage: In the following example, the compiler accepts cfront constructs. $ pgCC -b3 myprog.cc Cross-references: ––cfront2.1, –b , ––cfront3.0 , +p, –A - -[ n o _ ] b o o l (pgCC only) Enable or disable recognition of bool. The default value is --bool. 124 Chapter 3 - -c f r o n t _ 2 . 1 (pgCC only) Enable compilation of C++ with cfront 2.1 compatibility. This causes the compiler to accept language constructs that, while not part of the C++ language definition, are accepted by the AT&T C++ Language System (cfront release 2.1). This option also enables acceptance of anachronisms. Default: The compiler does not accept cfront language constructs that are not part of the C++ language definition. Usage: In the following example, the compiler accepts cfront constructs. $ pgCC --cfront_2.1 myprog.cc Cross-references: –b, –b3 , ––cfront3.0, +p, –A - -c f r o n t _ 3 . 0 (pgCC only) Enable compilation of C++ with cfront 3.0 compatibility. This causes the compiler to accept language constructs that, while not part of the C++ language definition, are accepted by the AT&T C++ Language System (cfront release 3.0). This option also enables acceptance of anachronisms. Default: The compiler does not accept cfront language constructs that are not part of the C++ language definition. Usage: In the following example, the compiler accepts cfront constructs. $ pgCC ––cfront_3.0 myprog.cc Cross-references: ––cfront2.1, –b , –b3 , +p, –A - -c r e a t e _ p c h f i l e n a m e (pgCC only) If other conditions are satisfied, create a precompiled header file with the specified name. If --pch (automatic PCH mode) appears on the command line following this option, its effect is erased. Command-line Options 125 - -d i a g _ s u p p r e s s t a g (pgCC only) Override the normal error severity of the specified diagnostic messages. The message(s) may be specified using a mnemonic error tag or using an error number. - -d i a g _ r e m a r k t a g (pgCC only) Override the normal error severity of the specified diagnostic messages. The message(s) may be specified using a mnemonic error tag or using an error number. - -d i a g _ w a r n i n g t a g (pgCC only) Override the normal error severity of the specified diagnostic messages. The message(s) may be specified using a mnemonic error tag or using an error number. - -d i a g _ e r r o r t a g (pgCC only) Override the normal error severity of the specified diagnostic messages. The message(s) may be specified using a mnemonic error tag or using an error number. - -d i s p l a y _ e r r o r _ n u m b e r (pgCC only) Display the error message number in any diagnostic messages that are generated. The option may be used to determine the error number to be used when overriding the severity of a diagnostic message. - -[ n o _ ] e x c e p t i o n s (pgCC only) Enable/disable exception handling support. The default is --exceptions. 126 Chapter 3 - -[ n o ] l l a l i g n (pgCC only) Do/don’t align long long integers on long long boundaries. The default is --llalign. -M Generate a list of make dependencies and print them to stdout. Compilation stops after the preprocessing phase. -M D Generate a list of make dependencies and print them to the file .d, where is the name of the file under compilation. - -o p t k _ a l l o w _ d o l l ar _ i n _ i d _ c h a r s (pgCC only) Accept dollar signs ($) in identifiers. -P Stops compilation after the preprocessing phase. Use the –P option to halt the compilation process after preprocessing and write the preprocessed output to the file filename.i, where the input file is filename.c or filename.cc. Use the –suffix option with this option to save the intermediate file in a file with the specified suffix. Default: The compiler produces an executable file. Usage: In the following example, the compiler produces the preprocessed file myprog.i in the current directory. $ pgCC -P myprog.cc Cross-references: –C,–c,–E, –Mkeepasm, –o, –S Command-line Options 127 - -p c h (pgCC only) Automatically use and/or create a precompiled header file. If --use_pch or --create_pch (manual PCH mode) appears on the command line following this option, its effect is erased. - -p c h _ d i r d i r e c t o r y n a m e (pgCC only) The directory in which to search for and/or create a precompiled header file. This option may be used with automatic PCH mode (--pch) or manual PCH mode (--create_pch or --use_pch). - -[ n o _ ] p c h _ m e s s a g e s (pgCC only) Enable or disable the display of a message indicating that a precompiled header file was created or used in the current compilation. - -p r e i n c l u d e = < f i l e n a m e > (pgCC only) Specifies the name of a file to be included at the beginning of the compilation. This option can be used to set system-dependent macros and types, for example. - -u s e _ p c h f i l e n a m e (pgCC only) Use a precompiled header file of the specified name as part of the current compilation. If --pch (automatic PCH mode) appears on the command line following this option, its effect is erased. 128 Chapter 3 - -[ n o _ ] u s i n g _ s t d (pgCC only) Enable or disable implicit use of the std namespace when standard header files are included. Default: The default is --using_std. Usage: The following command-line disables implicit use of the std namespace: $ pgCC –-no_using_std hello.cc -t (pgCC only) Control instantiation of template functions. Syntax: –t [arg] where arg is one of the following: all Instantiates all functions whether or not they are used. local Instantiates only the functions that are used in this compilation, and forces those functions to be local to this compilation. Note: This may cause multiple copies of local static variables. If this occurs, the program may not execute correctly. none Instantiates no functions. (this is the default) used Instantiates only the functions that are used in this compilation. Usage: In the following example, all templates are instantiated. $ pgCC -tall myprog.cc Function Inlining 129 Chapter 4 Function Inlining Function inlining replaces a call to a function or a subroutine with the body of the function or subroutine. This can speed up execution by eliminating parameter passing and function/subroutine call and return overhead. It also allows the compiler to optimize the function with the rest of the code. Note that using function inlining indiscriminately can result in much larger code size and no increase in execution speed. The PGI compilers provide two categories of inlining: • Automatic inlining - During the compilation process, a hidden pass precedes the compilation pass. This hidden pass extracts functions that are candidates for inlining. The inlining of functions occurs as the source files are compiled. • Inline libraries - You create inline libraries, for example using the pgf95 command and the –Mextract and –o options. There is no hidden extract pass but you must ensure that any files that depend on the inline library use the latest version of the inline library. There are important restrictions on inlining. Inlining only applies to certain types of functions. Refer to Section 4.5 Restrictions on Inlining, at the end of this chapter for more details on function inlining limitations. 4.1 Invoking Function Inlining To invoke the function inliner, use the –Minline option. If you do not specify an inline library, the compiler performs a special prepass on all source files named on the compiler command line before it compiles any of them. This pass extracts functions that meet the requirements for inlining and puts them in a temporary inline library for use by the compilation pass. Several –Minline options let you determine the selection criteria for functions to be inlined. These selection criteria include: except:func Inline all eligible functions except func, a function in the source text. Multiple functions can be listed, comma-separated. [name:]func A function name, which is a string matching func, a function in the source text. 130 Chapter 4 [size:]n A size, which instructs the compiler to select functions with a statement count less than or equal to n, the specified size. Note: the size n may not exactly equal the number of statements in a selected function (the size parameter is used as a rough gauge). levels:n A level number, which represents the number of function calling levels to be inlined. The default number is one (1). Using a level greater than one indicates that function calls within inlined functions may be replaced with inlined code. This allows the function inliner to automatically perform a sequence of inline and extract processes. [lib:]file.ext A library file name. This instructs the inliner to inline the functions within the library file file.ext. Create the library file using the –Mextract option. If no inline library is specified, functions are extracted from a temporary library created during an extract prepass. If you specify both a function name and a size n, the compiler inlines functions that match the function name or have n or fewer statements. If a keyword name:, lib: or size: is omitted, then a name with a period is assumed to be an inline library, a number is assumed to be a size, and a name without a period is assumed to be a function name. In the following example, the compiler inlines functions with fewer than approximately 100 statements in the source file myprog.f and writes the executable code in the default output file a.out. $ pgf95 -Minline=size:100 myprog.f Refer to Chapter 3, Command Line Options, for more information on the –Minline options. 4.1.1 Using an Inline Library If you specify one or more inline libraries on the command line with the –Minline option, the compiler does not perform an initial extract pass. The compiler selects functions to inline from the specified inline library. If you also specify a size or function name, all functions in the inline library meeting the selection criteria are selected for inline expansion at points in the source text where they are called. If you do not specify a function name or a size limitation for the –Minline option, the compiler inlines every function in the inline library that matches a function in the source text. Function Inlining 131 In the following example, the compiler inlines the function proc from the inline library lib.il and writes the executable code in the default output file a.out. $ pgf95 -Minline=name:proc,lib:lib.il myprog.f The following command line is equivalent to the line above, the only difference in this example is that the name: and lib: inline keywords are not used. The keywords are provided so you can avoid name conflicts if you use an inline library name that does not contain a period. Otherwise, without the keywords, a period lets the compiler know that the file on the command line is an inline library. $ pgf95 -Minline=proc,lib.il myprog.f 4.2 Creating an Inline Library You can create or update an inline library using the –Mextract command-line option. If you do not specify a selection criteria along with the –Mextract option, the compiler attempts to extract all subprograms. When you use the –Mextract option, only the extract phase is performed; the compile and link phases are not performed. The output of an extract pass is a library of functions available for inlining. It is placed in the inline library file specified on the command line with the –o filename specification. If the library file exists, new information is appended to it. If the file does not exist, it is created. You can use the –Minline option with the –Mextract option. In this case, the extracted library of functions can have other functions inlined into the library. Using both options enables you to obtain more than one level of inlining. In this situation, if you do not specify a library with the –Minline option, the inline process consists of two extract passes. The first pass is a hidden pass implied by the –Minline option, during which the compiler extracts functions and places them into a temporary library. The second pass uses the results of the first pass but puts its results into the library that you specify with the –o option. 4.2.1 Working with Inline Libraries An inline library is implemented as a directory with each inline function in the library stored as a file using an encoded form of the inlinable function. A special file named TOC in the inline library directory serves as a table of contents for the inline library. This is a printable, ASCII file which can be examined to find out information about the library contents, such as names and sizes of functions, the source file from which they were extracted, the version number of the extractor which created the entry, etc. Libraries and their elements can be manipulated using ordinary system commands. 132 Chapter 4 • Inline libraries can be copied or renamed. • Elements of libraries can be deleted or copied from one library to another. • The ls command can be used to determine the last-change date of a library entry. Dependencies in Makefiles–When a library is created or updated using one of the PGI compilers, the last-change date of the library directory is updated. This allows a library to be listed as a dependence in a makefile (and ensures that the necessary compilations will be performed when a library is changed). 4.2.2 Updating Inline Libraries - Makefiles If you use inline libraries you need to be certain that they remain up to date with the source files into which they are inlined. One way to assure inline libraries are updated is to include them in a makefile. The makefile fragment in Example 4-1 assumes the file utils.f contains a number of small functions used in the files parser.f and alloc.f. The makefile also maintains the inline library utils.il. The makefile updates the library whenever you change utils.f or one of the include files it uses. In turn, the makefile compiles parser.f and alloc.f whenever you update the library. SRC = mydir FC = pgf95 FFLAGS = –O2 main.o: $(SRC)/main.f $(SRC)/global.h $(FC) $(FFLAGS) -c $(SRC)/main.f utils.o: $(SRC)/utils.f $(SRC)/global.h $(SRC)/utils.h $(FC) $(FFLAGS) -c $(SRC)/utils.f utils.il: $(SRC)/utils.f $(SRC)global.h $(SRC)/utils.h $(FC) $(FFLAGS) -Mextract=15 -o utils.il parser.o: $(SRC)/parser.f $(SRC)/global.h utils.il $(FC) $(FFLAGS) -Minline=utils.il -c $(SRC)/parser.f alloc.o: $(SRC)/alloc.f $(SRC)/global.h utils.il $(FC) $(FFLAGS) -Minline=utils.il -c $(SRC)/alloc.f myprog: main.o utils.o parser.o alloc.o $(FC) -o myprog main.o utils.o parser.o alloc.o Example 4-1: Sample Makefile Function Inlining 133 4.3 Error Detection during Inlining To request inlining information from the compiler when you invoke the inliner, specify the –Minfo=inline option. For example: $ pgf95 -Minline=mylib.il -Minfo=inline myext.f 4.4 Examples Assume the program dhry consists of a single source file dhry.f. The following command line builds an executable file for dhry in which proc7 is inlined wherever it is called: $ pgf95 dhry.f -Minline=proc7 The following command lines build an executable file for dhry in which proc7 plus any functions of approximately 10 or fewer statements are inlined (one level only). Note that the specified functions are inlined only if they are previously placed in the inline library, temp.il, during the extract phase. $ pgf95 dhry.f -Mextract -o temp.il $ pgf95 dhry.f -Minline=10,Proc7,temp.il Assume the program fibo.f contains a single function fibo that calls itself recursively. The following command line creates the file fibo.o in which fibo is inlined into itself: $ pgf95 fibo.f -c –Mrecursive -Minline=fibo Because this version of fibo recurses only half as deeply, it executes noticeably faster. Using the same source file dhry.f, the following example builds an executable for dhry in which all functions of roughly ten or fewer statements are inlined. Two levels of inlining are performed. This means that if function A calls function B, and B calls C, and both B and C are inlinable, then the version of B which is inlined into A will have had C inlined into it. $ pgf95 dhry.f -Minline=size:10,levels:2 4.5 Restrictions on Inlining The following Fortran subprograms cannot be extracted: • Main or BLOCK DATA programs. • Subprograms containing alternate return, assigned GO TO, DATA, SAVE, or EQUIVALENCE statements. 134 Chapter 4 • Subprograms containing FORMAT statements. • Subprograms containing multiple entries. A Fortran subprogram is not inlined if any of the following applies: • It is referenced in a statement function. • A common block mismatch exists; i.e., the caller must contain all common blocks specified in the callee, and elements of the common blocks must agree in name, order, and type (except that the caller's common block can have additional members appended to the end of the common block). • An argument mismatch exists; i.e., the number and type (size) of actual and formal parameters must be equal. • A name clash exists; e.g., a call to subroutine xyz in the extracted subprogram and a variable named xyz in the caller. The following types of C and C++ functions cannot be inlined: • Functions whose return type is a struct data type, or functions which have a struct argument. This limitation applies only to x86 targets. • Functions containing switch statements • Functions which reference a static variable whose definition is nested within the function • Function which accept a variable number of arguments Certain C/C++ functions can only be inlined into the file that contains their definition: • Static functions • Functions which call a static function • Functions which reference a static variable OpenMP Directives for Fortran 135 Chapter 5 OpenMP Directives for Fortran The PGF77 and PGF95 Fortran compilers support the OpenMP Fortran Application Program Interface. The OpenMP shared-memory parallel programming model is defined by a collection of compiler directives, library routines, and environment variables that can be used to specify sharedmemory parallelism in Fortran, C and C++ programs. The directives include a parallel region construct for writing coarse grain SPMD programs, work-sharing constructs which specify that DO loop iterations should be split among the available threads of execution, and synchronization constructs. The data environment is controlled using clauses on the directives or with additional directives. Run-time library routines are provided to query the parallel runtime environment, for example to determine how many threads are participating in execution of a parallel region. Finally, environment variables are provided to control the execution behavior of parallel programs. For more information on OpenMP, see http://www.openmp.org. For an introduction to how to execute programs that use multiple processors along with some pointers to example code, see Section 1.4 Parallel Programming Using the PGI Compilers. The file ftp://ftp.pgroup.com/pub/SMP/fftpde.tar.gz contains a more advanced self-guided tutorial on how to parallelize the NAS FT fast Fourier transform benchmark using OpenMP directives. You can retrieve it using a web browser, and unpack it using the following commands within a shell command window: % gunzip fftpde.tar.gz % tar xvf fftpde.tar Follow the instructions in the README file to work through the tutorial. 5.1 Parallelization Directives Parallelization directives are comments in a program that are interpreted by the PGI Fortran compilers when the option -mp is specified on the command line. The form of a parallelization directive is: sentinel directive_name [clauses] With the exception of the SGI-compatible DOACROSS directive, the sentinel must be !$OMP, C$OMP, or *$OMP, must start in column 1 (one), and must appear as a single word without embedded white space. The sentinel marking a DOACROSS directive is C$. Standard Fortran syntax restrictions (line length, case insensitivity, etc.) apply to the directive line. Initial directive lines 136 Chapter 5 must have a space or zero in column six and continuation directive lines must have a character other than space or zero in column six. Continuation lines for C$DOACROSS directives are specified using the C$& sentinel. The order in which clauses appear in the parallelization directives is not significant. Commas separate clauses within the directives, but commas are not allowed between the directive name and the first clause. Clauses on directives may be repeated as needed subject to the restrictions listed in the description of each clause. The compiler option -mp enables recognition of the parallelization directives. The use of this option also implies: -Mreentrant local variables are placed on the stack and optimizations that may result in non-reentrant code are disabled (e.g., -Mnoframe); -Miomutex critical sections are generated around Fortran I/O statements. Many of the directives are presented in pairs and must be used in pairs. In the examples given with each section, the routines omp_get_num_threads() and omp_get_thread_num() are used, refer to Section 5.18 Run-time Library Routines for more information. They return the number of threads currently in the team executing the parallel region and the thread number within the team, respectively. 5.2 PARALLEL ... END PARALLEL The OpenMP PARALLEL END PARALLEL directive is supported using the following syntax. Syntax: !$OMP PARALLEL [Clauses] < Fortran code executed in body of parallel region > !$OMP END PARALLEL Clauses: PRIVATE(list) SHARED(list) DEFAULT(PRIVATE | SHARED | NONE) FIRSTPRIVATE(list) REDUCTION([{operator | intrinsic}:] list) COPYIN(list) IF(scalar_logical_expression) NUM_THREADS(scalar_integer_expression) OpenMP Directives for Fortran 137 This directive pair declares a region of parallel execution. It directs the compiler to create an executable in which the statements between PARALLEL and END PARALLEL are executed by multiple lightweight threads. The code that lies between PARALLEL and END PARALLEL is called a parallel region. The OpenMP parallelization directives support a fork/join execution model in which a single thread executes all statements until a parallel region is encountered. At the entrance to the parallel region, a system-dependent number of symmetric parallel threads begin executing all statements in the parallel region redundantly. These threads share work by means of work-sharing constructs such as parallel DO loops (see below). The number of threads in the team is controlled by the OMP_NUM_THREADS environment variable. If OMP_NUM_THREADS is not defined, the program will execute parallel regions using only one processor. Branching into or out of a parallel region is not supported. All other shared-memory parallelization directives must occur within the scope of a parallel region. Nested PARALLEL ... END PARALLEL directive pairs are not supported and are ignored. The END PARALLEL directive denotes the end of the parallel region, and is an implicit barrier. When all threads have completed execution of the parallel region, a single thread resumes execution of the statements that follow. Note: By default, there is no work distribution in a parallel region. Each active thread executes the entire region redundantly until it encounters a directive that specifies work distribution. For work distribution, see the DO, PARALLEL DO, or DOACROSS directives. PROGRAM WHICH_PROCESSOR_AM_I INTEGER A(0:1) INTEGER omp_get_thread_num A(0) = -1 A(1) = -1 !$OMP PARALLEL A(omp_get_thread_num()) = omp_get_thread_num() !$OMP END PARALLEL PRINT *, “A(0)=”,A(0), “ A(1)=”,A(1) END The variables specified in a PRIVATE list are private to each thread in a team. In effect, the compiler creates a separate copy of each of these variables for each thread in the team. When an assignment to a private variable occurs, each thread assigns to its local copy of the variable. When operations involving a private variable occur, each thread performs the operations using its local copy of the variable. 138 Chapter 5 Important points about private variables are: • Variables declared private in a parallel region are undefined upon entry to the parallel region. If the first use of a private variable within the parallel region is in a right-hand-side expression, the results of the expression will be undefined (i.e., this is probably a coding error). • Likewise, variables declared private in a parallel region are undefined when serial execution resumes at the end of the parallel region. The variables specified in a SHARED list are shared between all threads in a team, meaning that all threads access the same storage area for SHARED data. The DEFAULT clause lets you specify the default attribute for variables in the lexical extent of the parallel region. Individual clauses specifying PRIVATE, SHARED, etc. status override the declared DEFAULT. Specifying DEFAULT(NONE) declares that there is no implicit default, and in this case, each variable in the parallel region must be explicitly listed with an attribute of PRIVATE, SHARED, FIRSTPRIVATE, LASTPRIVATE, or REDUCTION. Variables that appear in the list of a FIRSTPRIVATE clause are subject to the same semantics as PRIVATE variables, but in addition, are initialized from the original object existing prior to entering the parallel region. Variables that appear in the list of a REDUCTION clause must be SHARED. A private copy of each variable in list is created for each thread as if the PRIVATE clause had been specified. Each private copy is initialized according to the operator as specified in Table 5-1: Table 5-1: Initialization of REDUCTION Variables Operator / Intrinsic Initialization + 0 * 1 - 0 .AND. .TRUE. .OR. .FALSE. .EQV. .TRUE. .NEQV. .FALSE. MAX Smallest Representable Number MIN Largest Representable Number IAND All bits on IOR 0 IEOR 0 OpenMP Directives for Fortran 139 At the end of the parallel region, a reduction is performed on the instances of variables appearing in list using operator or intrinsic as specified in the REDUCTION clause. The initial value of each REDUCTION variable is included in the reduction operation. If the {operator | intrinsic}: portion of the REDUCTION clause is omitted, the default reduction operator is “+” (addition). The COPYIN clause applies only to THREADPRIVATE common blocks. In the presence of the COPYIN clause, data from the master thread’s copy of the common block is copied to the threadprivate copies upon entry to the parallel region. In the presence of an IF clause, the parallel region will be executed in parallel only if the corresponding scalar_logical_expression evaluates to .TRUE.. Otherwise, the code within the region will be executed by a single processor regardless of the value of the environment variable OMP_NUM_THREADS. If the NUM_THREADS clause is present, the corresponding scalar_integer_expression must evaluate to a positive integer value. This value sets the maximum number of threads used during execution of the parallel region. A NUM_THREADS clause overrides either a previous call to the library routine omp_set_num_threads() or the setting of the OMP_NUM_THREADS environment variable. 5.3 CRITICAL ... END CRITICAL The OpenMP END CRITICAL directive uses the following syntax. !$OMP CRITICAL [(name)] < Fortran code executed in body of critical section > !$OMP END CRITICAL [(name)] Within a parallel region, you may have code that will not execute properly when multiple threads act upon the same sub-region of code. This is often due to a shared variable that is written and then read again. The CRITICAL ... END CRITICAL directive pair defines a subsection of code within a parallel region, referred to as a critical section, which will be executed one thread at a time. The optional name argument identifies the critical section. The first thread to arrive at a critical section will be the first to execute the code within the section. The second thread to arrive will not begin execution of statements in the critical section until the first thread has exited the critical section. Likewise each of the remaining threads will wait its turn to execute the statements in the critical section. Critical sections cannot be nested, and any such specifications are ignored. Branching into or out of a critical section is illegal. If a name argument appears on a CRITICAL directive, the same name must appear on the END CRITICAL directive. 140 Chapter 5 PROGRAM CRITICAL_USE REAL A(100,100), MX, LMX INTEGER I, J MX = -1.0 LMX = -1.0 CALL RANDOM_SEED() CALL RANDOM_NUMBER(A) !$OMP PARALLEL PRIVATE(I), FIRSTPRIVATE(LMX) !$OMP DO DO J=1,100 DO I=1,100 LMX = MAX(A(I,J), LMX) ENDDO ENDDO !$OMP CRITICAL MX = MAX(MX, LMX) !$OMP END CRITICAL !$OMP END PARALLEL PRINT *, “MAX VALUE OF A IS “, MX END Note that this program could also be implemented without the critical region by declaring MX as a reduction variable and performing the MAX calculation in the loop using MX directly rather than using LMX. See Sections 5.2 PARALLEL ... END PARALLEL and 5.6 DO ... END DO for more information on how to use the REDUCTION clause on a parallel DO loop. 5.4 MASTER ... END MASTER The OpenMP END MASTER directive uses the following syntax. !$OMP MASTER < Fortran code in body of MASTER section > !$OMP END MASTER In a parallel region of code, there may be a sub-region of code that should execute only on the master thread. Instead of ending the parallel region before this subregion and then starting it up again after this subregion, the MASTER ... END MASTER directive pair let you conveniently designate code that executes on the master thread and is skipped by the other threads. There is no implied barrier on entry to or exit from a MASTER ... END MASTER section of code. Nested master sections are ignored. Branching into or out of a master section is not supported. OpenMP Directives for Fortran 141 PROGRAM MASTER_USE INTEGER A(0:1) INTEGER omp_get_thread_num A=-1 !$OMP PARALLEL A(omp_get_thread_num()) = omp_get_thread_num() !$OMP MASTER PRINT *, “YOU SHOULD ONLY SEE THIS ONCE” !$OMP END MASTER !$OMP END PARALLEL PRINT *, “A(0)=“, A(0), “ A(1)=“, A(1) END 5.5 SINGLE ... END SINGLE The OpenMP SINGLE END SINGLE directive uses the following syntax. !$OMP SINGLE [Clauses] < Fortran code in body of SINGLE processor section > !$OMP END SINGLE [NOWAIT] Clauses: PRIVATE(list) FIRSTPRIVATE(list) COPYPRIVATE(list) In a parallel region of code, there may be a sub-region of code that will only execute correctly on a single thread. Instead of ending the parallel region before this subregion and then starting it up again after this subregion, the SINGLE ... END SINGLE directive pair lets you conveniently designate code that executes on a single thread and is skipped by the other threads. There is an implied barrier on exit from a SINGLE ... END SINGLE section of code unless the optional NOWAIT clause is specified. Nested single process sections are ignored. Branching into or out of a single process section is not supported. PROGRAM SINGLE_USE INTEGER A(0:1) INTEGER omp_get_thread_num() !$OMP PARALLEL 142 Chapter 5 A(omp_get_thread_num()) = omp_get_thread_num() !$OMP SINGLE PRINT *, "YOU SHOULD ONLY SEE THIS ONCE" !$OMP END SINGLE !$OMP END PARALLEL PRINT *, "A(0)=", A(0), " A(1)=", A(1) END The PRIVATE and FIRSTPRIVATE clauses are as described in Section 5.2 PARALLEL ... END PARALLEL. The COPYPRIVATE clause causes the variables in list to be copied from the private copies in the single thread that executes the SINGLE region to the other copies in all other threads of the team at the end of the SINGLE region. The COPYPRIVATE clause must not be used with NOWAIT. 5.6 DO ... END DO The OpenMP DO END DO directive uses the following syntax. Syntax: !$OMP DO [Clauses ] < Fortran DO loop to be executed in parallel > !$OMP END DO [NOWAIT] Clauses: PRIVATE(list) FIRSTPRIVATE(list) LASTPRIVATE(list) REDUCTION({operator | intrinsic } : list) SCHEDULE (type [, chunk]) ORDERED The real purpose of supporting parallel execution is the distribution of work across the available threads. You can explicitly manage work distribution with constructs such as: IF (omp_get_thread_num() .EQ. 0) THEN … ELSE IF (omp_get_thread_num() .EQ. 1) THEN … ENDIF OpenMP Directives for Fortran 143 However, these constructs are not in the form of directives. The DO ... END DO directive pair provides a convenient mechanism for the distribution of loop iterations across the available threads in a parallel region. Items to note about clauses are: • Variables declared in a PRIVATE list are treated as private to each processor participating in parallel execution of the loop, meaning that a separate copy of the variable exists on each processor. • Variables declared in a FIRSTPRIVATE list are PRIVATE, and in addition are initialized from the original object existing before the construct. • Variables declared in a LASTPRIVATE list are PRIVATE, and in addition the thread that executes the sequentially last iteration updates the version of the object that existed before the construct. • The REDUCTION clause is as described in Section 5.2 PARALLEL ... END PARALLEL. • The SCHEDULE clause is explained in the following section. • If ORDERED code blocks are contained in the dynamic extent of the DO directive, the ORDERED clause must be present. For more information on ORDERED code blocks, see Section 5.14 ORDERED. The DO ... END DO directive pair directs the compiler to distribute the iterative DO loop immediately following the !$OMP DO directive across the threads available to the program. The DO loop is executed in parallel by the team that was started by an enclosing parallel region. If the !$OMP END DO directive is not specified, the !$OMP DO is assumed to end with the enclosed DO loop. DO ... END DO directive pairs may not be nested. Branching into or out of a !$OMP DO loop is not supported. By default, there is an implicit barrier after the end of the parallel loop; the first thread to complete its portion of the work will wait until the other threads have finished their portion of work. If NOWAIT is specified, the threads will not synchronize at the end of the parallel loop. Other items to note about !$OMP DO loops: • The DO loop index variable is always private. • !$OMP DO loops must be executed by all threads participating in the parallel region or none at all. • The END DO directive is optional, but if it is present it must appear immediately after the end of the enclosed DO loop. PROGRAM DO_USE REAL A(1000), B(1000) 144 Chapter 5 DO I=1,1000 B(I) = FLOAT(I) ENDDO !$OMP PARALLEL !$OMP DO DO I=1,1000 A(I) = SQRT(B(I)); ENDDO … !$OMP END PARALLEL … END The SCHEDULE clause specifies how iterations of the DO loop are divided up between processors. Given a SCHEDULE (type [, chunk]) clause, type can be STATIC, DYNAMIC, GUIDED, or RUNTIME. These are defined as follows: • When SCHEDULE (STATIC, chunk) is specified, iterations are allocated in contiguous blocks of size chunk. The blocks of iterations are statically assigned to threads in a roundrobin fashion in order of the thread ID numbers. The chunk must be a scalar integer expression. If chunk is not specified, a default chunk size is chosen equal to: (number_of_iterations + omp_num_threads() - 1) / omp_num_threads() • When SCHEDULE (DYNAMIC, chunk) is specified, iterations are allocated in contiguous blocks of size chunk. As each thread finishes a piece of the iteration space, it dynamically obtains the next set of iterations. The chunk must be a scalar integer expression. If no chunk is specified, a default chunk size is chosen equal to 1. • When SCHEDULE (GUIDED, chunk) is specified, the chunk size is reduced in an exponentially decreasing manner with each dispatched piece of the iteration space. Chunk specifies the minimum number of iterations to dispatch each time, except when there are less than chunk iterations remaining to be processed, at which point all remaining iterations are assigned. If no chunk is specified, a default chunk size is chosen equal to 1. • When SCHEDULE (RUNTIME) is specified, the decision regarding iteration scheduling is deferred until runtime. The schedule type and chunk size can be chosen at runtime by setting the OMP_SCHEDULE environment variable. If this environment variable is not set, the resulting schedule is equivalent to SCHEDULE(STATIC). OpenMP Directives for Fortran 145 5.7 WORKSHARE ... END WORKSHARE The OpenMP WORKSHARE … END WORKSHARE directive pair uses the following syntax. Syntax: !$OMP WORKSHARE < Fortran structured block to be executed in parallel > !$OMP END WORKSHARE [NOWAIT] The Fortran structured block enclosed by the WORKSHARE … END WORKSHARE directive pair can consist only of the following types of statements and constructs: • Array assignments • Scalar assignments • FORALL statements or constructs • WHERE statements or constructs • OpenMP ATOMIC , CRITICAL or PARALLEL constructs The work implied by the above statements and constructs is split up between the threads executing the WORKSHARE construct in a way that is guaranteed to maintain standard Fortran semantics. The goal of the WORKSHARE construct is to effect parallel execution of non-iterative but implicitly data parallel array assignments, FORALL, and WHERE statements and constructs intrinsic to the Fortran language beginning with Fortran 90. The Fortran structured block contained within a WORKSHARE construct must not contain any user-defined function calls unless the function is ELEMENTAL. 5.8 BARRIER The OpenMP BARRIER directive uses the following syntax. !$OMP BARRIER There may be occasions in a parallel region, when it is necessary that all threads complete work to that point before any thread is allowed to continue. The BARRIER directive synchronizes all threads at such a point in a program. Multiple barrier points are allowed within a parallel region. The BARRIER directive must either be executed by all threads executing the parallel region or by none of them. 146 Chapter 5 5.9 DOACROSS The C$DOACROSS directive is not part of the OpenMP standard, but is supported for compatibility with programs parallelized using legacy SGI-style directives. Syntax: C$DOACROSS [ Clauses ] < Fortran DO loop to be executed in parallel > Clauses: [ {PRIVATE | LOCAL} (list) ] [ {SHARED | SHARE} (list) ] [ MP_SCHEDTYPE={SIMPLE | INTERLEAVE} ] [ CHUNK= ] [ IF (logical_expression) ] The C$DOACROSS directive has the effect of a combined parallel region and parallel DO loop applied to the loop immediately following the directive. It is very similar to the OpenMP PARALLEL DO directive, but provides for backward compatibility with codes parallelized for SGI systems prior to the OpenMP standardization effort. The C$DOACROSS directive must not appear within a parallel region. It is a shorthand notation that tells the compiler to parallelize the loop to which it applies, even though that loop is not contained within a parallel region. While this syntax is more convenient, it should be noted that if multiple successive DO loops are to be parallelized it is more efficient to define a single enclosing parallel region and parallelize each loop using the OpenMP DO directive. A variable declared PRIVATE or LOCAL to a C$DOACROSS loop is treated the same as a private variable in a parallel region or DO (see above). A variable declared SHARED or SHARE to a C$DOACROSS loop is shared among the threads, meaning that only 1 copy of the variable exists to be used and/or modified by all of the threads. This is equivalent to the default status of a variable that is not listed as PRIVATE in a parallel region or DO (this same default status is used in C$DOACROSS loops as well). OpenMP Directives for Fortran 147 5.10 PARALLEL DO The OpenMP PARALLEL DO directive uses the following syntax. Syntax: !$OMP PARALLEL DO [CLAUSES] < Fortran DO loop to be executed in parallel > [!$OMP END PARALLEL DO] Clauses: PRIVATE(list) SHARED(list) DEFAULT(PRIVATE | SHARED | NONE) FIRSTPRIVATE(list) LASTPRIVATE(list) REDUCTION({operator | intrinsic} : list) COPYIN (list) IF(scalar_logical_expression) NUM_THREADS(scalar_integer_expression) SCHEDULE (type [, chunk]) ORDERED The semantics of the PARALLEL DO directive are identical to those of a parallel region containing only a single parallel DO loop and directive. Note that the END PARALLEL DO directive is optional. The available clauses are as defined in Section 5.2 PARALLEL ... END PARALLEL and Section 5.6 DO ... END DO. 5.11 PARALLEL WORKSHARE The OpenMP PARALLEL WORKSHARE directive uses the following syntax. Syntax: !$OMP PARALLEL WORKSHARE [CLAUSES] < Fortran structured block to be executed in parallel > [!$OMP END PARALLEL WORKSHARE] Clauses: PRIVATE(list) SHARED(list) 148 Chapter 5 DEFAULT(PRIVATE | SHARED | NONE) FIRSTPRIVATE(list) LASTPRIVATE(list) REDUCTION({operator | intrinsic} : list) COPYIN (list) IF(scalar_logical_expression) NUM_THREADS(scalar_integer_expression) SCHEDULE (type [, chunk]) ORDERED The semantics of the PARALLEL WORKSHARE directive are identical to those of a parallel region containing a single WORKSHARE construct. Note that the END PARALLEL WORKSHARE directive is optional, and that NOWAIT may not be specified on an END PARALLEL WORKSHARE directive. The available clauses are as defined in Section 5.2 PARALLEL ... END PARALLEL. 5.12 SECTIONS … END SECTIONS The OpenMP SECTIONS / END SECTIONS directive pair uses the following syntax: Syntax: !$OMP SECTIONS [ Clauses ] [!$OMP SECTION] < Fortran code block executed by processor i > [!$OMP SECTION] < Fortran code block executed by processor j > … !$OMP END SECTIONS [NOWAIT] Clauses: PRIVATE (list) FIRSTPRIVATE (list) LASTPRIVATE (list) REDUCTION({operator | intrinsic} : list) The SECTIONS / END SECTIONS directives define a non-iterative work-sharing construct within a parallel region. Each section is executed by a single processor. If there are more processors than sections, some processors will have no work and will jump to the implied barrier at the end of the construct. If there are more sections than processors, one or more processors will execute more than one section. OpenMP Directives for Fortran 149 A SECTION directive may only appear within the lexical extent of the enclosing SECTIONS / END SECTIONS directives. In addition, the code within the SECTIONS / END SECTIONS directives must be a structured block, and the code in each SECTION must be a structured block. The available clauses are as defined in Section 5.2 PARALLEL ... END PARALLEL and Section 5.6 DO ... END DO. 5.13 PARALLEL SECTIONS The OpenMP PARALLEL SECTIONS / END SECTIONS directive pair uses the following syntax: Syntax: !$OMP PARALLEL SECTIONS [CLAUSES] [!$OMP SECTION] < Fortran code block executed by processor i > [!$OMP SECTION] < Fortran code block executed by processor j > … !$OMP END SECTIONS [NOWAIT] Clauses: PRIVATE(list) SHARED(list) DEFAULT(PRIVATE | SHARED | NONE) FIRSTPRIVATE(list) LASTPRIVATE(list) REDUCTION({operator | intrinsic} : list) COPYIN (list) IF(scalar_logical_expression) NUM_THREADS(scalar_integer_expression) The PARALLEL SECTIONS / END SECTIONS directives define a non-iterative work-sharing construct without the need to define an enclosing parallel region. Each section is executed by a single processor. If there are more processors than sections, some processors will have no work and will jump to the implied barrier at the end of the construct. If there are more sections than processors, one or more processors will execute more than one section. A SECTION directive may only appear within the lexical extent of the enclosing PARALLEL SECTIONS / END SECTIONS directives. In addition, the code within the PARALLEL SECTIONS / END SECTIONS directives must be a structured block, and the code in each SECTION must be a structured block. 150 Chapter 5 The available clauses are as defined in Section 5.2 PARALLEL ... END PARALLEL and Section 5.6 DO ... END DO. 5.14 ORDERED The OpenMP ORDERED directive is supported using the following syntax: !$OMP ORDERED < Fortran code block executed by processor > !$OMP END ORDERED The ORDERED directive can appear only in the dynamic extent of a DO or PARALLEL DO directive that includes the ORDERED clause. The code block between the ORDERED / END ORDERED directives is executed by only one thread at a time, and in the order of the loop iterations. This sequentializes the ordered code block while allowing parallel execution of statements outside the code block. The following additional restrictions apply to the ORDERED directive: • The ORDERED code block must be a structured block. It is illegal to branch into or out of the block. • A given iteration of a loop with a DO directive cannot execute the same ORDERED directive more than once, and cannot execute more than one ORDERED directive. 5.15 ATOMIC The OpenMP ATOMIC directive uses following syntax: !$OMP ATOMIC The ATOMIC directive is semantically equivalent to enclosing the following single statement in a CRITICAL / END CRITICAL directive pair. The statement must be of one of the following forms: • x = x operator expr • x = expr operator x • x = intrinsic (x, expr) • x = intrinsic (expr, x) where x is a scalar variable of intrinsic type, expr is a scalar expression that does not reference x, intrinsic is one of MAX, MIN, IAND, IOR, or IEOR, and operator is one of +, *, -, /, .AND., .OR., .EQV., or .NEQV..OpenMP Directives for Fortran 151 5.16 FLUSH The OpenMP FLUSH directive uses the following syntax: !$OMP FLUSH [(list)] The FLUSH directive ensures that all processor-visible data items, or only those specified in list when it’s present, are written back to memory at the point at which the directive appears. 5.17 THREADPRIVATE The OpenMP THREADPRIVATE directive uses the following syntax: !$OMP THREADPRIVATE (list) Where list is a comma-separated list of named variables to be made private to each thread or named common blocks to be made private to each thread but global within the thread . Common block names must appear between slashes (i.e. /common_blockn/). This directive must appear in the declarations section of a program unit after the declaration of any common blocks or variables listed. On entry to a parallel region, data in a THREADPRIVATE common block or variable is undefined unless COPYIN is specified on the PARALLEL directive. When a common block or variable that is initialized using DATA statements appears in a THREADPRIVATE directive, each thread’s copy is initialized once prior to its first use. The following restrictions apply to the THREADPRIVATE directive: • The THREADPRIVATE directive must appear after every declaration of a thread private common block. • Only named common blocks can be made thread private • It is illegal for a THREADPRIVATE common block or its constituent variables to appear in any clause other than a COPYIN clause. • A variable can appear in a THREADRIVATE directive only in the scope in which it is declared. It must not be an element of a common block or be declared in an EQUIVALENCE statement. • A variable that appears in a THREADPRIVATE directive and is not declared in the scope of a module must have the SAVE attribute. 5.18 Run-time Library Routines User-callable functions are available to the Fortran programmer to query and alter the parallel execution environment. 152 Chapter 5 integer omp_get_num_threads() returns the number of threads in the team executing the parallel region from which it is called. When called from a serial region, this function returns 1. A nested parallel region is the same as a single parallel region. By default, the value returned by this function is equal to the value of the environment variable OMP_NUM_THREADS or to the value set by the last previous call to the omp_set_num_threads() subroutine defined below. subroutine omp_set_num_threads(scalar_integer_exp) sets the number of threads to use for the next parallel region. This subroutine can only be called from a serial region of code. If it is called from within a parallel region, or within a subroutine or function that is called from within a parallel region, the results are undefined. This subroutine has precedence over the OMP_NUM_THREADS environment variable. integer omp_get_thread_num() returns the thread number within the team. The thread number lies between 0 and omp_get_num_threads()-1. When called from a serial region, this function returns 0. A nested parallel region is the same as a single parallel region. integer function omp_get_max_threads() returns the maximum value that can be returned by calls to omp_get_num_threads(). If omp_set_num_threads() is used to change the number of processors, subsequent calls to omp_get_max_threads() will return the new value. This function returns the maximum value whether executing from a parallel or serial region of code. integer function omp_get_num_procs() returns the number of processors that are available to the program. logical function omp_in_parallel() returns .TRUE. if called from within a parallel region and .FALSE. if called outside of a parallel region. When called from within a parallel region that is serialized, for example in the presence of an IF clause evaluating .FALSE., the function will return .FALSE.. subroutine omp_set_dynamic(scalar_logical_exp) is designed to allow automatic dynamic adjustment of the number of threads used for execution of parallel regions. This function is recognized, but currently has no effect. logical function omp_get_dynamic() OpenMP Directives for Fortran 153 is designed to allow the user to query whether automatic dynamic adjustment of the number of threads used for execution of parallel regions is enabled. This function is recognized, but currently always returns .FALSE.. subroutine omp_set_nested(scalar_logical_exp) is designed to allow enabling/disabling of nested parallel regions. This function is recognized, but currently has no effect. logical function omp_get_nested() is designed to allow the user to query whether dynamic adjustment of the number of threads available for execution of parallel regions is enabled. This function is recognized, but currently always returns .FALSE. double precision function omp_get_wtime() returns the elapsed wall clock time in seconds as a DOUBLE PRECISION value. Times returned are per-thread times, and are not necessarily globally consistent across all threads. double precision function omp_get_wtick() returns the resolution of omp_get_wtime(), in seconds, as a DOUBLE PRECISION value. subroutine omp_init_lock(integer_var) initializes a lock associated with the variable integer_var for use in subsequent calls to lock routines. This initial state of integer_var is unlocked. It is illegal to make a call to this routine if integer_var is already associated with a lock. subroutine omp_destroy_lock(integer_var) disassociates a lock associated with the variable integer_var. subroutine omp_set_lock(integer_var) causes the calling thread to wait until the specified lock is available. The thread gains ownership of the lock when it is available. It is illegal to make a call to this routine if integer_var has not been associated with a lock. subroutine omp_unset_lock(integer_var) causes the calling thread to release ownership of the lock associated with integer_var. It is illegal to make a call to this routine if integer_var has not been associated with a lock. 154 Chapter 5 logical function omp_test_lock(integer_var) causes the calling thread to try to gain ownership of the lock associated with integer_var. The function returns .TRUE. if the thread gains ownership of the lock, and .FALSE. otherwise. It is illegal to make a call to this routine if integer_var has not been associated with a lock. 5.19 Environment Variables OMP_NUM_THREADS - specifies the number of threads to use during execution of parallel regions. The default value for this variable is 1. For historical reasons, the environment variable NCPUS is supported with the same functionality. In the event that both OMP_NUM_THREADS and NCPUS are defined, the value of OMP_NUM_THREADS takes precedence. Note: OMP_NUM_THREADS threads will be used to execute the program regardless of the number of physical processors available in the system. As a result, you can run programs using more threads than physical processors and they will execute correctly. However, performance of programs executed in this manner can be unpredictable, and oftentimes will be inefficient OMP_SCHEDULE - specifies the type of iteration scheduling to use for DO and PARALLEL DO loops which include the SCHEDULE(RUNTIME) clause. The default value for this variable is “STATIC”. If the optional chunk size is not set, a chunk size of 1 is assumed except in the case of a STATIC schedule. For a STATIC schedule, the default is as defined in Section 5.6 DO ... END DO. Examples of the use of OMP_SCHEDULE are as follows: $ setenv OMP_SCHEDULE “STATIC, 5” $ setenv OMP_SCHEDULE “GUIDED, 8” $ setenv OMP_SCHEDULE “DYNAMIC” OMP_DYNAMIC - currently has no effect. OMP_NESTED - currently has no effect. MPSTKZ - increase the size of the stacks used by threads executing in parallel regions. It is for use with programs that utilize large amounts of thread-local storage in the form of private variables or local variables in functions or subroutines called within parallel regions. The value should be an integer concatenated with M or m to specify stack sizes of n megabytes. For example: $ setenv MPSTKZ 8MOpenMP Pragmas for C and C++ 155 Chapter 6 OpenMP Pragmas for C and C++ The PGCC ANSI C and C++ compilers support the OpenMP C/C++ Application Program Interface. The OpenMP shared-memory parallel programming model is defined by a collection of compiler directives or pragmas, library routines, and environment variables that can be used to specify shared-memory parallelism in Fortran, C and C++ programs. The OpenMP C/C++ pragmas include a parallel region construct for writing coarse grain SPMD programs, worksharing constructs which specify that C/C++ for loop iterations should be split among the available threads of execution, and synchronization constructs. The data environment is controlled using clauses on the pragmas or with additional pragmas. Run-time library functions are provided to query the parallel runtime environment, for example to determine how many threads are participating in execution of a parallel region. Finally, environment variables are provided to control the execution behavior of parallel programs. For more information on OpenMP, and a complete copy of the OpenMP C/C++ API specification, see http://www.openmp.org. 6.1 Parallelization Pragmas Parallelization pragmas are #pragma statements in a C or C++ program that are interpreted by the PGCC C and C++ compilers when the option -mp is specified on the command line. The form of a parallelization pragma is: #pragma omp pragmas_name [clauses] The pragmas follow the conventions of the C and C++ standards. Whitespace can appear before and after the #. Preprocessing tokens following the #pragma omp are subject to macro replacement. The order in which clauses appear in the parallelization pragmas is not significant. Spaces separate clauses within the pragmas. Clauses on pragmas may be repeated as needed subject to the restrictions listed in the description of each clause. For the purposes of the OpenMP pragmas, a C/C++ structured block is defined to be a statement or compound statement (a sequence of statements beginning with { and ending with }) that has a single entry and a single exit. No statement or compound statement is a C/C++ structured block if there is a jump into or out of that statement. The compiler option -mp enables recognition of the parallelization pragmas. The use of this option also implies: -Mreentrant local variables are placed on the stack and optimizations that may result in 156 Chapter 6 non-reentrant code are disabled (e.g., -Mnoframe) Also, note that calls to I/O library functions are system-dependent and are not necessarily guaranteed to be thread-safe. I/O library calls within parallel regions should be protected by critical regions (see below) to ensure they function correctly on all systems. In the examples given with each section, the functions omp_get_num_threads() and omp_get_thread_num() are used (refer to Section 6.15 Run-time Library Routines.) They return the number of threads currently in the team executing the parallel region and the thread number within the team, respectively. 6.2 omp parallel The OpenMP omp parallel pragma uses the following syntax: Syntax: #pragma omp parallel [clauses] < C/C++ structured block > Clauses: private(list) shared(list) default(shared | none) firstprivate(list) reduction(operator: list) copyin (list) if (scalar_expression) num_threads(scalar_integer_expression) This pragma declares a region of parallel execution. It directs the compiler to create an executable in which the statements within the following C/C++ structured block are executed by multiple lightweight threads. The code that lies within the structured block is called a parallel region. The OpenMP parallelization pragmas support a fork/join execution model in which a single thread executes all statements until a parallel region is encountered. At the entrance to the parallel region, a system-dependent number of symmetric parallel threads begin executing all statements in the parallel region redundantly. These threads share work by means of work-sharing constructs such as parallel for loops (see the next example). The number of threads in the team is controlled by the OMP_NUM_THREADS environment variable. If OMP_NUM_THREADS is not defined, the program will execute parallel regions using only one processor. Branching into or out of a parallel region is not supported. OpenMP Pragmas for C and C++ 157 All other shared-memory parallelization pragmas must occur within the scope of a parallel region. Nested omp parallel pragmas are not supported and are ignored. There is an implicit barrier at the end of a parallel region. When all threads have completed execution of the parallel region, a single thread resumes execution of the statements that follow. It should be emphasized that by default there is no work distribution in a parallel region. Each active thread executes the entire region redundantly until it encounters a directive that specifies work distribution. For work distribution, see the omp for pragma. #include #include main(){ int a[2]={-1,-1}; #pragma omp parallel { a[omp_get_thread_num()] = omp_get_thread_num(); } printf(“a[0] = %d, a[1] = %d”,a[0],a[1]); } The variables specified in a private list are private to each thread in a team. In effect, the compiler creates a separate copy of each of these variables for each thread in the team. When an assignment to a private variable occurs, each thread assigns to its local copy of the variable. When operations involving a private variable occur, each thread performs the operations using its local copy of the variable. Other important points to note about private variables are the following: • Variables declared private in a parallel region are undefined upon entry to the parallel region. If the first use of a private variable within the parallel region is in a right-hand-side expression, the results of the expression will be undefined (i.e., this is probably a coding error). • Likewise, variables declared private in a parallel region are undefined when serial execution resumes at the end of the parallel region. The variables specified in a shared list are shared between all threads in a team, meaning that all threads access the same storage area for shared data. The default clause allows the user to specify the default attribute for variables in the lexical extent of the parallel region. Individual clauses specifying private, shared, etc status override the declared default. Specifying default(none) declares that there is no implicit default, and in this case each variable in the parallel region must be explicitly listed with an attribute of private, shared, firstprivate, or reduction. 158 Chapter 6 Variables that appear in the list of a firstprivate clause are subject to the same semantics as private variables, but in addition are initialized from the original object existing prior to entering the parallel region. Variables that appear in the list of a reduction clause must be shared. A private copy of each variable in list is created for each thread as if the private clause had been specified. Each private copy is initialized according to the operator as specified in Table 6-1: Table 6-1: Initialization of Reduction Variables Operator Initialization + 0 * 1 - 0 & ~0 | 0 ^ 0 && 1 || 0 • At the end of the parallel region, a reduction is performed on the instances of variables appearing in list using operator as specified in the reduction clause. The initial value of each reduction variable is included in the reduction operation. If the operator: portion of the reduction clause is omitted, the default reduction operator is “+” (addition). • The copyin clause applies only to threadprivate variables. In the presence of the copyin clause, data from the master thread’s copy of the threadprivate variable is copied to the thread private copies upon entry to the parallel region. • In the presence of an if clause, the parallel region will be executed in parallel only if the corresponding scalar_expression evaluates to a non-zero value. Otherwise, the code within the region will be executed by a single processor regardless of the value of the environment variable OMP_NUM_THREADS. If the num_threads clause is present, the corresponding scalar_integer_expression must evaluate to a positive integer value. This value sets the maximum number of threads used during execution of the parallel region. A num_threads clause overrides either a OpenMP Pragmas for C and C++ 159 previous call to the library routine omp_set_num_threads() or the setting of the OMP_NUM_THREADS environment variable. 6.3 omp critical The OpenMP omp critical pragma uses the following syntax: #pragma omp critical [(name)] < C/C++ structured block > Within a parallel region, there may exist subregions of code that will not execute properly when executed by multiple threads simultaneously. This is often due to a shared variable that is written and then read again. The omp critical pragma defines a subsection of code within a parallel region, referred to as a critical section, which will be executed one thread at a time. The first thread to arrive at a critical section will be the first to execute the code within the section. The second thread to arrive will not begin execution of statements in the critical section until the first thread has exited the critical section. Likewise, each of the remaining threads will wait its turn to execute the statements in the critical section. An optional name may be used to identify the critical region. Names used to identify critical regions have external linkage and are in a name space separate from the name spaces used by labels, tags, members, and ordinary identifiers. Critical sections cannot be nested, and any such specifications are ignored. Branching into or out of a critical section is illegal. #include main(){ int a[100][100], mx=-1, lmx=-1, i, j; for (j=0; j<100; j++) for (i=0; i<100; i++) a[i][j]=1+(int)(10.0*rand()/(RAND_MAX+1.0)); #pragma omp parallel private(i) firstprivate(lmx) { #pragma omp for for (j=0; j<100; j++) for (i=0; i<100; i++) lmx = (lmx > a[i][j]) ? lmx : a[i][j]; #pragma omp critical mx = (mx > lmx) ? mx : lmx; } 160 Chapter 6 printf ("max value of a is %d\n",mx); } 6.4 omp master The OpenMP omp master pragma uses the following syntax: #pragma omp master < C/C++ structured block > In a parallel region of code, there may be a sub-region of code that should execute only on the master thread. Instead of ending the parallel region before this subregion, and then starting it up again after this subregion, the omp master pragma allows the user to conveniently designate code that executes on the master thread and is skipped by the other threads. There is no implied barrier on entry to or exit from a master section. Nested master sections are ignored. Branching into or out of a master section is not supported. #include #include main(){ int a[2]={-1,-1}; #pragma omp parallel { a[omp_get_thread_num()] = omp_get_thread_num(); #pragma omp master printf(“YOU SHOULD ONLY SEE THIS ONCE\n”); } printf(“a[0]=%d, a[1]=%d\n”,a[0],a[1]); } 6.5 omp single The OpenMP omp single pragma uses the following syntax: #pragma omp single [Clauses] < C/C++ structured block > Clauses: private(list) firstprivate(list) OpenMP Pragmas for C and C++ 161 copyprivate(list) nowait In a parallel region of code, there may be a subregion of code that will only execute correctly on a single thread. Instead of ending the parallel region before this subregion, and then starting it up again after this subregion, the omp single pragma allows the user to conveniently designate code that executes on a single thread and is skipped by the other threads. There is an implied barrier on exit from a single process section unless the optional nowait clause is specified. Nested single process sections are ignored. Branching into or out of a single process section is not supported. The private and firstprivate clauses are as described in Section 6.2 omp parallel. The copyprivate clause causes the variables in list to be copied from the private copies in the single thread that executes the single region to the other copies in all other threads of the team at the end of the single region. The copyprivate clause must not be used with nowait. 6.6 omp for The OpenMP omp for pragma uses the following syntax: #pragma omp for [Clauses] < C/C++ for loop to be executed in parallel > Clauses: private(list) firstprivate(list) lastprivate(list) reduction(operator: list) schedule (kind[, chunk]) ordered nowait The real purpose of supporting parallel execution is the distribution of work across the available threads. You can explicitly manage work distribution with constructs such as: if (omp_get_thread_num() == 0) { … } else if (omp_get_thread_num() == 1) { 162 Chapter 6 … } However, these constructs are not in the form of pragmas. The omp for pragma provides a convenient mechanism for the distribution of loop iterations across the available threads in a parallel region. The following variables can be used: • Variables declared in a private list are treated as private to each processor participating in parallel execution of the loop, meaning that a separate copy of the variable exists on each processor. • Variables declared in a firstprivate list are private, and in addition are initialized from the original object existing before the construct. • Variables declared in a lastprivate list are private, and in addition the thread that executes the sequentially last iteration updates the version of the object that existed before the construct. • The reduction clause is as described in Section 6.2 omp parallel. The schedule clause is explained below. • If ordered code blocks are contained in the dynamic extent of the for directive, the ordered clause must be present. See Section 6.11 omp ordered for more information on ordered code blocks. The omp for pragma directs the compiler to distribute the iterative for loop immediately following across the threads available to the program. The for loop is executed in parallel by the team that was started by an enclosing parallel region. Branching into or out of an omp for loop is not supported, and omp for pragmas may not be nested. By default, there is an implicit barrier after the end of the parallel loop; the first thread to complete its portion of the work will wait until the other threads have finished their portion of work. If nowait is specified, the threads will not synchronize at the end of the parallel loop. Other items to note about omp for loops: • The for loop index variable is always private and must be a signed integer. • omp for loops must be executed by all threads participating in the parallel region or none at all. • The for loop must be a structured block and its execution must not be terminated by break. • Values of the loop control expressions and the chunk expressions must be the same for all threads executing the loop. OpenMP Pragmas for C and C++ 163 #include #include main(){ float a[1000], b[1000]; int i; for (i=0; i<1000; i++) b[i] = i; #pragma omp parallel { #pragma omp for for (i=0; i<1000; i++) a[i] = sqrt(b[i]); … } … } The schedule clause specifies how iterations of the for loop are divided up between processors. Given a schedule (kind[, chunk]) clause, kind can be static, dynamic, guided, or runtime. These are defined as follows: • When schedule (static, chunk) is specified, iterations are allocated in contiguous blocks of size chunk. The blocks of iterations are statically assigned to threads in a round-robin fashion in order of the thread ID numbers. The chunk must be a scalar integer expression. If chunk is not specified, a default chunk size is chosen equal to: (number_of_iterations + omp_num_threads() - 1) / omp_num_threads() • When schedule (dynamic, chunk) is specified, iterations are allocated in contiguous blocks of size chunk. As each thread finishes a piece of the iteration space, it dynamically obtains the next set of iterations. The chunk must be a scalar integer expression. If no chunk is specified, a default chunk size is chosen equal to 1. • When schedule (guided, chunk) is specified, the chunk size is reduced in an exponentially decreasing manner with each dispatched piece of the iteration space. Chunk specifies the minimum number of iterations to dispatch each time, except when there are less than chunk iterations remaining to be processed, at which point all remaining iterations are assigned. If no chunk is specified, a default chunk size is chosen equal to 1. • When schedule (runtime) is specified, the decision regarding iteration scheduling is deferred until runtime. The schedule type and chunk size can be chosen at runtime by 164 Chapter 6 setting the OMP_SCHEDULE environment variable. If this environment variable is not set, the resulting schedule is equivalent to schedule(static). 6.7 omp barrier The OpenMP omp barrier pragma uses the following syntax: #pragma omp barrier There may be occasions in a parallel region when it is necessary that all threads complete work to that point before any thread is allowed to continue. The omp barrier pragma synchronizes all threads at such a point in a program. Multiple barrier points are allowed within a parallel region. The omp barrier pragma must either be executed by all threads executing the parallel region or by none of them. 6.8 omp parallel for The omp parallel for pragma uses the following syntax. #pragma omp parallel for [clauses] < C/C++ for loop to be executed in parallel > Clauses: private(list) shared(list) default(shared | none) firstprivate(list) lastprivate(list) reduction(operator: list) copyin (list) if (scalar_expression) ordered schedule (kind[, chunk]) num_threads(scalar_integer_expression) The semantics of the omp parallel for pragma are identical to those of a parallel region containing only a single parallel for loop and pragma. The available clauses are as defined in Sections 6.2 omp parallel and 6.6 omp for. OpenMP Pragmas for C and C++ 165 6.9 omp sections The omp sections pragma uses the following syntax: #pragma omp sections [ Clauses ] { [#pragma omp section] < C/C++ structured block executed by processor i > [#pragma omp section] < C/C++ structured block executed by processor j > … } Clauses: private (list) firstprivate (list) lastprivate (list) reduction(operator: list) nowait The omp sections pragma defines a non-iterative work-sharing construct within a parallel region. Each section is executed by a single thread. If there are more threads than sections, some threads will have no work and will jump to the implied barrier at the end of the construct. If there are more sections than threads, one or more threads will execute more than one section. An omp section pragma may only appear within the lexical extent of the enclosing omp sections pragma. In addition, the code within the omp sections pragma must be a structured block, and the code in each omp section must be a structured block. The available clauses are as defined in Sections 6.2 omp parallel and 6.6 omp for. 6.10 omp parallel sections The omp parallel sections pragma uses the following syntax: #pragma omp parallel sections [clauses] { [#pragma omp section] < C/C++ structured block executed by processor i > [#pragma omp section] < C/C++ structured block executed by processor j > 166 Chapter 6 … } Clauses: private(list) shared(list) default(shared | none) firstprivate(list) lastprivate (list) reduction({operator: list) copyin (list) if (scalar_expression) num_threads(scalar_integer_expression) nowait The omp parallel sections pragma defines a non-iterative work-sharing construct without the need to define an enclosing parallel region. Semantics are identical to a parallel region containing only an omp sections pragma and the associated structured block. 6.11 omp ordered The OpenMP ordered pragma uses the following syntax: #pragma omp ordered < C/C++ structured block > The ordered pragma can appear only in the dynamic extent of a for or parallel for pragma that includes the ordered clause. The structured code block appearing after the ordered pragma is executed by only one thread at a time, and in the order of the loop iterations. This sequentializes the ordered code block while allowing parallel execution of statements outside the code block. The following additional restrictions apply to the ordered pragma: • The ordered code block must be a structured block. It is illegal to branch into or out of the block. • A given iteration of a loop with a DO directive cannot execute the same ORDERED directive more than once, and cannot execute more than one ORDERED directive. 6.12 omp atomic The omp atomic pragma uses the following syntax: OpenMP Pragmas for C and C++ 167 #pragma omp atomic < C/C++ expression statement > The omp atomic pragma is semantically equivalent to subjecting the following single C/C++ expression statement to an omp critical pragma. The expression statement must be of one of the following forms: • x = expr • x++ • ++x • x-- • --x where x is a scalar variable of intrinsic type, expr is a scalar expression that does not reference x, is not overloaded and is one of +, *, -, /, &, ^, |, << or >>. 6.13 omp flush The omp flush pragma uses the following syntax: #pragma omp flush [(list)] The omp flush pragma ensures that all processor-visible data items, or only those specified in list when it’s present, are written back to memory at the point at which the directive appears. 6.14 omp threadprivate The omp threadprivate pragma uses the following syntax: #pragma omp threadprivate (list) Where list is a list of variables to be made private to each thread but global within the thread. This pragma must appear in the declarations section of a program unit after the declaration of any variables listed. On entry to a parallel region, data in a threadprivate variable is undefined unless copyin is specified on the omp parallel pragma. When a variable appears in an omp threadprivate pragma, each thread’s copy is initialized once at an unspecified point prior to its first use as the master copy would be initialized in a serial execution of the program. The following restrictions apply to the omp threadprivate pragma: • The omp threadprivate pragma must appear after the declaration of every threadprivate variable included in list. 168 Chapter 6 • It is illegal for an omp threadprivate variable to appear in any clause other than a copyin, schedule or if clause. • If a variable is specified in an omp threadprivate pragma in one translation unit, it must be specified in an omp threadprivate pragma in every translation unit in which it appears. • The address of an omp threadprivate variable is not an address constant. • An omp threadprivate variable must not have an incomplete type or a reference type. 6.15 Run-time Library Routines User-callable functions are available to the OpenMP C/C++ programmer to query and alter the parallel execution environment. Any program unit that invokes these functions should include the statement #include . The omp.h include file contains definitions for each of the C/C++ library routines and two required type definitions. #include int omp_get_num_threads(void); returns the number of threads in the team executing the parallel region from which it is called. When called from a serial region, this function returns 1. A nested parallel region is the same as a single parallel region. By default, the value returned by this function is equal to the value of the environment variable OMP_NUM_THREADS or to the value set by the last previous call to the omp_set_num_threads() function defined below. #include void omp_set_num_threads(int num_threads); sets the number of threads to use for the next parallel region. This function can only be called from a serial region of code. If it is called from within a parallel region, or within a function that is called from within a parallel region, the results are undefined. This function has precedence over the OMP_NUM_THREADS environment variable. #include int omp_get_thread_num(void); returns the thread number within the team. The thread number lies between 0 and omp_get_num_threads()-1. When called from a serial region, this function returns 0. A nested parallel region is the same as a single parallel region. #include OpenMP Pragmas for C and C++ 169 int omp_get_max_threads(void); returns the maximum value that can be returned by calls to omp_get_num_threads(). If omp_set_num_threads() is used to change the number of processors, subsequent calls to omp_get_max_threads() will return the new value. This function returns the maximum value whether executing from a parallel or serial region of code. #include int omp_get_num_procs(void); returns the number of processors that are available to the program. #include int omp_in_parallel(void); returns non-zero if called from within a parallel region and zero if called outside of a parallel region. When called from within a parallel region that is serialized, for example in the presence of an if clause evaluating to zero, the function will return zero. #include void omp_set_dynamic(int dynamic_threads); is designed to allow automatic dynamic adjustment of the number of threads used for execution of parallel regions. This function is recognized, but currently has no effect. #include int omp_get_dynamic(void); is designed to allow the user to query whether automatic dynamic adjustment of the number of threads used for execution of parallel regions is enabled. This function is recognized, but currently always returns zero. #include void omp_set_nested(int nested); is designed to allow enabling/disabling of nested parallel regions. This function is recognized, but currently has no effect. #include int omp_get_nested(void); is designed to allow the user to query whether dynamic adjustment of the number of threads available for execution of parallel regions is enabled. This function is recognized, but currently always returns zero. 170 Chapter 6 #include double omp_get_wtime() returns the elapsed wall clock time in seconds as a floating-point double value. Times returned are per-thread times, and are not necessarily globally consistent across all threads. #include double omp_get_wtick() returns the resolution of omp_get_wtime(), in seconds, as a floating-point double value. #include void omp_init_lock(omp_lock_t *lock); void omp_init_nest_lock(omp_nest_lock_t *lock); initializes a lock associated with the variable lock for use in subsequent calls to lock routines. This initial state of lock is unlocked. It is illegal to make a call to this routine if lock is already associated with a lock. #include void omp_destroy_lock(omp_lock_t *lock); void omp_destroy_nest_lock(omp_nest_lock_t *lock); disassociates a lock associated with the variable lock. #include void omp_set_lock(omp_lock_t *lock); void omp_set_nest_lock(omp_nest_lock_t *lock); causes the calling thread to wait until the specified lock is available. The thread gains ownership of the lock when it is available. It is illegal to make a call to this routine if lock has not been associated with a lock. #include void omp_unset_lock(omp_lock_t *lock); void omp_unset_nest_lock(omp_nest_lock_t *lock); causes the calling thread to release ownership of the lock associated with lock. It is illegal to make a call to this routine if lock has not been associated with a lock. #include int omp_test_lock(omp_lock_t *lock); int omp_test_nest_lock(omp_nest_lock_t *lock); OpenMP Pragmas for C and C++ 171 causes the calling thread to try to gain ownership of the lock associated with lock. The function returns non-zero if the thread gains ownership of the lock, and zero otherwise. It is illegal to make a call to this routine if lock has not been associated with a lock. 6.16 Environment Variables OMP_NUM_THREADS - specifies the number of threads to use during execution of parallel regions. The default value for this variable is 1. For historical reasons, the environment variable NCPUS is supported with the same functionality. In the event that both OMP_NUM_THREADS and NCPUS are defined, the value of OMP_NUM_THREADS takes precedence. Note: OMP_NUM_THREADS threads will be used to execute the program regardless of the number of physical processors available in the system. As a result, you can run programs using more threads than physical processors and they will execute correctly. However, performance of programs executed in this manner can be unpredictable, and oftentimes will be inefficient OMP_SCHEDULE - specifies the type of iteration scheduling to use for omp for and omp parallel for loops that include the schedule(runtime) clause. The default value for this variable is “static”. If the optional chunk size is not set, a chunk size of 1 is assumed except in the case of a static schedule. For a static schedule, the default is as defined in Section 6.6 omp for. Examples of the use of OMP_SCHEDULE are as follows: $ setenv OMP_SCHEDULE “static, 5” $ setenv OMP_SCHEDULE “guided, 8” $ setenv OMP_SCHEDULE “dynamic” OMP_DYNAMIC - currently has no effect. OMP_NESTED - currently has no effect. MPSTKZ - increase the size of the stacks used by threads executing in parallel regions. For use with programs that utilize large amounts of thread-local storage in the form of private variables or local variables in functions or subroutines called within parallel regions. The value should be an integer concatenated with M or m to specify stack sizes of n megabytes. For example: $ setenv MPSTKZ 8M Optimization Directives and Pragmas 173 Chapter 7 Optimization Directives and Pragmas Directives are Fortran comments that the user may supply in a Fortran source file to provide information to the compiler. Directives alter the effects of certain command line options or default behavior of the compiler. While a command line option affects the entire source file that is being compiled, directives apply, or disable, the effects of a command line option to selected subprograms or to selected loops in the source file (for example, an optimization). Use directives to tune selected routines or loops. 7.1 Adding Directives to Fortran Directives may have any of the following forms: cpgi$g directive cpgi$r directive cpgi$l directive cpgi$ directive The C must be in column 1. Either * or ! is allowed in place of C. The scope indicator occurs after the $; this indicator controls the scope of the directive. Some directives ignore the scope indicator. The valid scopes, as shown above, are: g (global) indicates the directive applies to the end of the source file. r (routine) indicates the directive applies to the next subprogram. l (loop) indicates the directive applies to the next loop (but not to any loop contained within the loop body). Loop-scoped directives are only applied to DO loops. blank indicates that the default scope for the directive is applied. The body of the directive may immediately follow the scope indicator. Alternatively, any number of blanks may precede the name of the directive. Any names in the body of the directive, including the directive name, may not contain embedded blanks. Blanks may surround any special characters, such as a comma or an equal sign. 174 Chapter 7 The directive name, including the directive prefix, may contain upper or lower case letters (case is not significant). Case is significant for any variable names that appear in the body of the directive if the command line option –Mupcase is selected. For compatibility with other vendors’ directives, the prefix cpgi$ may be substituted with cdir$ or cvd$. 7.2 Fortran Directive Summary Table 7-1 summarizes the supported Fortran directives. The scope entry indicates the allowed scope indicators for each directive; the default scope is surrounded by parentheses. The system field indicates the target system type for which the pragma applies. Many of the directives can be preceded by NO. The default entry in the table indicates the default of the directive; n/a appears if a default does not apply. The name of a directive may also be prefixed with –M; for example, the directive –Mbounds is equivalent to bounds and –Mopt is equivalent to opt. Table 7-1: Fortran Optimization Directive Summary Directive Function Default Scope altcode noaltcode Do/don’t generate alternate code for vectorized and parallelized loops altcode (l)rg assoc noassoc Do/don’t perform associative transformations assoc (l)rg bounds nobounds Do/don’t perform array bounds checking nobounds (r)g* cncall nocncall Loops are considered for parallelization, even if they contain calls to user-defined subroutines or functions, or if their loop counts do not exceed usual thresholds. nocncall (l)rg concur noconcur Do/don’t enable auto-concurrentization of loops concur (l)rg depchk nodepchk Do/don’t ignore potential data dependencies depchk (l)rg eqvchk noeqvchk Do/don’t check EQUIVALENCE s for data dependencies. eqvchk (l)rg invarif noinvarif Do/don’t remove invariant if constructs from loops. invarif (l)rg ivdep Ignore potential data dependencies depchk (l)rg lstval nolstval Do/don’t compute last values. lstval (l)rg opt Select optimization level. N/A (r)g Optimization Directives and Pragmas 175 Directive Function Default Scope safe_lastval Parallelize when loop contains a scalar used outside of loop. not enabled (l) unroll nounroll Do/don’t unroll loops. nounroll (l)rg vector novector Do/don't perform vectorizations. vector (l)rg vintr novintr Do/don’t recognize vector intrinsics. vintr (l)rg In the case of the vector/novector directive, the scope is the code following the directive until the end of the routine for r-scoped directives (as opposed to the entire routine), or until the end of the file for g-scoped directives (as opposed to the entire file). altcode (noaltcode) Instructs the compiler to generate alternate code for vectorized or parallelized loops. The noaltcode directive disables generation of alternate code. This directive affects the compiler only when –Mvect or –Mconcur is enabled on the command line. cpgi$ altcode Enables alternate code (altcode) generation for vectorized loops. For each loop the compiler decides whether to generate altcode and what type(s) to generate, which may be any or all of: altcode without iteration peeling, altcode with non-temporal stores and other data cache optimizations, and altcode based on array alignments calculated dynamically at runtime. The compiler also determines suitable loop count and array alignment conditions for executing the alternate code. cpgi$ altcode alignment For a vectorized loop, if possible generate an alternate vectorized loop containing additional aligned moves which is executed if a runtime array alignment test is passed. cpgi$ altcode [(n)] concur For each auto-parallelized loop, generate an alternate serial loop to be executed if the loop count is less than or equal to n. If n is omitted or n is 0, the compiler determines a suitable value of n for each loop. cpgi$ altcode [(n)] concurreduction 176 Chapter 7 This directive sets the loop count threshold for parallelization of reduction loops to n. For each auto-parallelized reduction loop, generate an alternate serial loop to be executed if the loop count is less than or equal to n. If n is omitted or n is 0, the compiler determines a suitable value of n for each loop. cpgi$ altcode [(n)] nontemporal For a vectorized loop, if possible generate an alternate vectorized loop containing non-temporal stores and other cache optimizations to be executed if the loop count is greater than n. If n is omitted or n is 1, the compiler determines a suitable value of n for each loop. The alternate code is optimized for the case when the data referenced in the loop does not all fit in level 2 cache. cpgi$ altcode [(n)] nopeel For a vectorized loop where iteration peeling is performed by default, if possible generate an alternate vectorized loop without iteration peeling to be executed if the loop count is less than or equal to n. If n is omitted or n is 1, the compiler determines a suitable value of n for each loop, and in some cases it may decide not to generate an alternate unpeeled loop. cpgi$ altcode [(n)] vector For each vectorized loop, generate an alternate scalar loop to be executed if the loop count is less than or equal to n. If n is omitted or n is 1, the compiler determines a suitable value of n for each loop. cpgi$ noaltcode This directive sets the loop count thresholds for parallelization of all innermost loops to 0, and disables alternate code generation for vectorized loops. assoc (noassoc) This directive toggles the effects of the –Mvect=noassoc command-line option (an Optimization –M control). By default, when scalar reductions are present the vectorizer may change the order of operations so that it can generate better code (e.g., dot product). Such transformations change the result of the computation due to roundoff error. The noassoc directive disables these transformations. This directive affects the compiler only when –Mvect is enabled on the command line. bounds (nobounds) Optimization Directives and Pragmas 177 This directive alters the effects of the –Mbounds command line option. This directive enables the checking of array bounds when subscripted array references are performed. By default, array bounds checking is not performed. cncall (nocncall) Loops within the specified scope are considered for parallelization, even if they contain calls to user-defined subroutines or functions, or if their loop counts do not exceed the usual thresholds. A nocncall directive cancels the effect of a previous cncall. concur (noconcur) This directive alters the effects of the –Mconcur command-line option. The directive instructs the auto-parallelizer to enable auto-concurrentization of loops. If concur is specified, multiple processors will be used to execute loops which the auto-parallelizer determines to be parallelizable. The noconcur directive disables these transformations. This directive affects the compiler only when –Mconcur is enabled on the command line. depchk (nodepchk) This directive alters the effects of the –Mdepchk command line option. When potential data dependencies exist, the compiler, by default, assumes that there is a data dependence that in turn may inhibit certain optimizations or vectorizations. nodepchk directs the compiler to ignore unknown data dependencies. eqvchk (noeqvchk) When examining data dependencies, noeqvchk directs the compiler to ignore any dependencies between variables appearing in EQUIVALENCE statements. invarif (noinvarif) There is no command-line option corresponding to this directive. Normally, the compiler removes certain invariant if constructs from within a loop and places them outside of the loop. The directive noinvarif directs the compiler to not move such constructs. The directive invarif toggles a previous noinvarif. ivdep The ivdep directive is equivalent to the directive nodepchk. opt The syntax of this directive is: cpgi$ opt= where, the optional is r or g and is an integer constant representing the optimization level to be used when compiling a subprogram (routine scope) or all subprograms in 178 Chapter 7 a file (global scope). The opt directive overrides the value specified by the command line option –On. lstval (nolstval) There is no command line option corresponding to this directive. The compiler determines whether the last values for loop iteration control variables and promoted scalars need to be computed. In certain cases, the compiler must assume that the last values of these variables are needed and therefore computes their last values. The directive nolstval directs the compiler not to compute the last values for those cases. safe_lastval During parallelization scalars within loops need to be privatized. Problems are possible if a scalar is accessed outside the loop. For example: do i = 1, N if( f(x(i)) > 5.0 ) t = x(i) enddo v = t creates a problem since the value of t may not be computed on the last iteration of the loop. If a scalar assigned within a loop is used outside the loop, we normally save the last value of the scalar. Essentially the value of the scalar on the "last iteration" is saved, in this case when i = N. If the loop is parallelized and the scalar is not assigned on every iteration, it may be difficult to determine on what iteration t is last assigned, without resorting to costly critical sections. Analysis allows the compiler to determine if a scalar is assigned on every iteration, thus the loop is safe to parallelize if the scalar is used later. An example loop is: do i = 1, N if( x(i) > 0.0 ) t = 2.0 else t = 3.0 endif y(i) = ...t... enddo v = t where t is assigned on every iteration of the loop. However, there are cases where a scalar may be privatizable. If it is used after the loop, it is unsafe to parallelize. Examine this loop: Optimization Directives and Pragmas 179 do i = 1,N if( x(i) > 0.0 ) t = x(i) ... ... y(i) = ...t.. endif enddo v = t where each use of t within the loop is reached by a definition from the same iteration. Here t is privatizable, but the use of t outside the loop may yield incorrect results since the compiler may not be able to detect on which iteration of the parallelized loop t is assigned last. The compiler detects the above cases. Where a scalar is used after the loop, but is not defined on every iteration of the loop, parallelization will not occur. If you know that the scalar is assigned on the last iteration of the loop, making it safe to parallelize the loop, a pragma is available to let the compiler know the loop is safe to parallelize. Use the following C pragma to tell the compiler that for a given loop the last value computed for all scalars make it safe to parallelize the loop: cpgi$l safe_lastval In addition, a command-line option, -Msafe_lastval, provides this information for all loops within the routines being compiled (essentially providing global scope.) unroll (nounroll) The directive nounroll is used to disable loop unrolling and unroll to enable unrolling. The directive takes arguments c and n. A c specifies that c (complete unrolling should be turned on or off) An n specifies that n (count) unrolling should be turned on or off. In addition, the following arguments may be added to the unroll directive: cpgi$ unroll = c:v This sets the threshold to which c unrolling applies; v is a constant; a loop whose constant loop count is <= v is completely unrolled. cpgi$ unroll = n:v This adjusts threshold to which n unrolling applies; v is a constant; a loop to which n unrolling applies is unrolled v times. 180 Chapter 7 The directives unroll and nounroll only apply if –Munroll is selected on the command line. vector (novector) The directive novector is used to disable vectorization. The directive vector is used to reenable vectorization after a previous novector directive. The directives vector and novector only apply if –Mvect has been selected on the command line. vintr (novintr) The directive novintr directs the vectorizer to disable recognition of vector intrinsics. The directive vintr is used to re-enable recognition of vector intrinsics after a previous novintr directive. The directives vintr and novintr only apply if –Mvect has been selected on the command line. 7.3 Scope of Directives and Command Line options This section presents several examples showing the effect of directives and the scope of directives. Remember that during compilation, the effect of a directive may be to either turn an option on, or turn an option off. Directives apply to the section of code following the directive, corresponding to the specified scope (that is, the following loop, the following routine, or the rest of the program). Consider the following code: integer maxtime, time parameter (n = 1000, maxtime = 10) double precision a(n,n), b(n,n), c(n,n) do time = 1, maxtime do i = 1, n do j = 1, n c(i,j) = a(i,j) + b(i,j) enddo enddo enddo end When compiled with –Mvect, both interior loops are interchanged with the outer loop. $ pgf95 -Mvect dirvect1.f Directives alter this behavior either globally or on a routine or loop by loop basis. To assure that vectorization is not applied, use the novector directive with global scope. Optimization Directives and Pragmas 181 cpgi$g novector integer maxtime, time parameter (n = 1000, maxtime = 10) double precision a(n,n), b(n,n), c(n,n) do time = 1, maxtime do i = 1, n do j = 1, n c(i,j) = a(i,j) + b(i,j) enddo enddo enddo end In this version, the compiler disables vectorization for the entire source file. Another use of the directive scoping mechanism turns an option on or off locally, either for a specific procedure or for a specific loop: integer maxtime, time parameter (n = 1000, maxtime = 10) double precision a(n,n), b(n,n), c(n,n) cpgi$l novector do time = 1, maxtime do i = 1, n do j = 1, n c(i,j) = a(i,j) + b(i,j) enddo enddo enddo end Loop level scoping does not apply to nested loops. That is, the directive only applies to the following loop. In this example, the directive turns off vector transformations for the top-level loop. If the outer loop were a timing loop, this would be a practical use for a loop-scoped directive. 7.4 Adding Pragmas to C and C++ Pragmas may be supplied in a C/C++ source file to provide information to the compiler. Like directives in Fortran, pragmas alter the effects of certain command-line options or default behavior of the compiler (many pragmas have a corresponding command-line option). While a commandline option affects the entire source file that is being compiled, pragmas apply the effects of a particular command-line option to selected functions or to selected loops in the source file. 182 Chapter 7 Pragmas may also toggle an option, selectively enabling and disabling the option. Pragmas let you tune selected functions or loops based on your knowledge of the code. The general syntax of a pragma is: #pragma [ scope ] pragma-body The optional scope field is an indicator for the scope of the pragma; some pragmas ignore the scope indicator. The valid scopes are: global indicates the pragma applies to the entire source file. routine indicates the pragma applies to the next function. loop indicates the pragma applies to the next loop (but not to any loop contained within the loop body). Loop-scoped pragmas are only applied to for and while loops. If a scope indicator is not present, the default scope, if any, is applied. Whitespace must appear after the pragma keyword and between the scope indicator and the body of the pragma. Whitespace may also surround any special characters, such as a comma or an equal sign. Case is significant for the names of the pragmas and any variable names that appear in the body of the pragma. 7.5 C/C++ Pragma Summary Table 7-2 summarizes the supported pragmas. The scope entry in the table indicates the permitted scope indicators for each pragma: the letters L, R, and G indicate loop, routine, and global scope, respectively. The default scope is surrounded by parentheses. The "*" in the scope field indicates that the scope is the code following the pragma until the end of the routine for R-scoped pragmas, as opposed to the entire routine, or until the end of the file for G-scoped pragmas, as opposed to the entire file. Many of the pragmas can be preceded by no. The default entry in the table indicates the default of the pragma; N/A appears if a default does not apply. The name of any pragma may be prefixed with -M; for example, –Mnoassoc is equivalent to noassoc and –Mvintr is equivalent to vintr. The section following the table provides brief descriptions of the pragmas that are unique to C/C++. Pragmas that have a corresponding directive in Fortran are described in Section 7.2. Optimization Directives and Pragmas 183 Table 7-2: C/C++ Pragma Summary Pragma Function Default Scope altcode noaltcode Do/don’t generate alternate code for vectorized and parallelized loops altcode (L)RG assoc noassoc Do/don’t perform associative transformations. assoc (L)RG bounds nobounds Do/don’t perform array bounds checking. nobounds (R)G concur noconcur Do/don’t enable auto-concurrentization of loops. concur (L)RG depchk nodepchk Do/don’t ignore potential data dependencies. depchk (L)RG fcon nofcon Do/don’t assume unsuffixed real constants are single precision. nofcon (R)G invarif noinvarif Do/don’t remove invariant if constructs from loops. invarif (L)RG lstval nolstval Do/don’t compute last values. lstval (L)RG opt Select optimization level. N/A (R)G safe nosafe Do/don’t treat pointer arguments as safe. safe (R)G safe_lastval Parallelize when loop contains a scalar used outside of loop. not enabled (L) safeptr nosafeptr Do/don’t ignore potential data dependencies to pointers. nosafeptr L(R)G single nosingle Do/don’t convert float parameters to double. nosingle (R)G* 184 Chapter 7 Pragma Function Default Scope unroll nounroll Do/don’t unroll loops. nounroll (L)RG vector novector Do/don’t perform vectorizations. vector (L)RG vintr novintr Do/don’t recognize vector intrinsics. vintr (L)RG fcon (nofcon) This pragma alters the effects of the –Mfcon command-line option (a –M Language control). The pragma instructs the compiler to treat non-suffixed floating-point constants as float rather than double. By default, all non-suffixed floating-point constants are treated as double. safe (nosafe) By default, the compiler assumes that all pointer arguments are unsafe. That is, the storage located by the pointer can be accessed by other pointers. The forms of the safe pragma are: #pragma [scope] [no]safe #pragma safe (variable [, variable]...) where scope is either global or routine. When the pragma safe is not followed by a variable name (or name list), all pointer arguments appearing in a routine (if scope is routine) or all routines (if scope is global) will be treated as safe. If variable names occur after safe, each name is the name of a pointer argument in the current function. The named argument is considered to be safe. Note that if just one variable name is specified, the surrounding parentheses may be omitted. There is no command-line option corresponding to this pragma. safeptr (nosafeptr) The pragma safeptr directs the compiler to treat pointer variables of the indicated storage class as safe. The pragma nosafeptr directs the compiler to treat pointer variables of the indicated storage class as unsafe. This pragma alters the effects of the –Msafeptr command-line option. Optimization Directives and Pragmas 185 The syntax of this pragma is: #pragma [scope] value where value is: [no]safeptr={arg|local|auto|global|static|all},... Note that the values local and auto are equivalent. For example, in a file containing multiple functions, the command-line option –Msafeptr might be helpful for one function, but can’t be used because another function in the file would produce incorrect results. In such a file, the safeptr pragma, used with routine scope could improve performance and produce correct results. single (nosingle) The pragma single directs the compiler not to convert float parameters to double in nonprototyped functions. This can result in faster code if the program uses only float parameters. NOTE: Since ANSI C specifies that routines must convert float parameters to double in non-prototyped functions, this pragma results in non-ANSI conforming code. 7.6 Scope of C/C++ Pragmas and Command Line Options This section presents several examples showing the effect of pragmas and the use of the pragma scope indicators. Note during compilation a pragma either turns an option on or turns an option off. Pragmas apply to the section of code corresponding to the specified scope (that is, the entire file, the following loop, or the following or current routine). For pragmas that have only routine and global scope, there are two rules for determining the scope of a pragma. We cover these special scope rules at the end of this section. In all cases, pragmas override a corresponding command-line option. 186 Chapter 7 Consider the program: main() { float a[100][100], b[100][100], c[100][100]; int time, maxtime, n, i, j; maxtime=10; n=100; for (time=0; time[,[,...]] where is any valid variable or array element reference. NOTE: the sentinel for prefetch directives is cmem$, which is distinct from the cpgi$ sentinel used for optimization directives. Any prefetch directives that use the cpgi$ sentinel will be ignored by the PGI compilers. The "c" must be in column 1. Either * or ! is allowed in place of c. Any scope indicator that occurs after the $ (g, r or l) is ignored. The directive name, including the directive prefix, may contain upper or lower case letters (case is not significant). Case is significant for any variable names that appear in the body of the directive if the command line option –Mupcase is selected. An example using prefetch directives to prefetch data in a matrix multiplication inner loop where a row of one source matrix has been gathered into a contiguous vector might look as follows: real*8 a(m,n), b(n,p), c(m,p), arow(n) ... do j = 1, p cmem$ prefetch arow(1),b(1,j) cmem$ prefetch arow(5),b(5,j) cmem$ prefetch arow(9),b(9,j) do k = 1, n, 4 cmem$ prefetch arow(k+12),b(k+12,j) c(i,j) = c(i,j) + arow(k) * b(k,j) c(i,j) = c(i,j) + arow(k+1) * b(k+1,j) c(i,j) = c(i,j) + arow(k+2) * b(k+2,j) c(i,j) = c(i,j) + arow(k+3) * b(k+3,j) enddo enddo This pattern of prefetch directives will cause the compiler to emit prefetch instructions whereby elements of arow and b are fetched into the data cache starting 4 iterations prior to first use. By varying the prefetch distance in this way, it is possible in some cases to reduce the effects of main memory latency and improve performance. 190 Chapter 7 Libraries and Environment Variables 191 Chapter 8 Libraries and Environment Variables This chapter discusses issues related to PGI-supplied compiler libraries. It also addresses the use of C/C++ builtin functions in place of the corresponding libc routines, creation of dynamically linked libraries (also known as shared objects or shared libraries), and math libraries. 8.1 Using builtin Math Functions in C/C++ The name of the math header file is math.h. Include the math header file in all of your source files that use a math library routine as in the following example, which calculates the inverse cosine of p/3. #include #define PI 3.1415926535 main() { double x, y; x = PI/3.0; y = acos(x); } Including math.h will cause PGCC C and C++ to use builtin functions, which are much more efficient than library calls. In particular, the following intrinsics calls will be processed using builtins if you include math.h: atan2 cos fabs exp atan sin log pow tan sqrt log10 8.2 Creating and Using Shared Object Files on Linux All of the PGI Fortran, C, and C++ compilers support creation of shared object files. Unlike statically linked object and library files, shared object files link and resolve references with an executable at runtime via a dynamic linker supplied with your operating system. The PGI compilers must generate position independent code to support creation of shared objects by the linker. This is not the default, use the steps that follow to create object files with position 192 Chapter 8 independent code and shared object files that are to include them. The following steps describe how to create and use a shared object file. Step 1 - To create an object file with position independent code, compile it with the appropriate PGI compiler using the -fpic option (the -fPIC, -Kpic, and -KPIC options are supported for compatibility with other systems you may have used, and are equivalent to-fpic). For example, use the following command to create an object file with position independent code using pgf95: % pgf95 -c -fpic tobeshared.f Step 2 - To produce a shared object file, use the appropriate PGI compiler to invoke the linker supplied with your system. It is customary to name such files using a .so filename extension. On Linux, this is done by passing the -shared option to the linker: % pgf95 -shared -o tobeshared.so tobeshared.o Note that compilation and generation of the shared object can be performed in one step using both the -fpic option and the appropriate option for generation of a shared object file. Step 3 - To use a shared object file, compile and link the program which will reference functions or subroutines in the shared object file using the appropriate PGI compiler and listing the shared object on the link line: % pgf95 -o myprog myprof.f tobeshared.so Step 4 - You now have an executable myprog which does not include any code from functions or subroutines in tobeshared.so, but which can be executed and dynamically linked to that code. By default, when the program is linked to produce myprog, no assumptions are made on the location of tobeshared.so. In order for myprog to execute correctly, you must initialize the environment variable LD_LIBRARY_PATH to include the directory containing tobeshared.so. If LD_LIBRARY_PATH is already initialized, it is important not to overwrite its contents. Assuming you have placed tobeshared.so in a directory /home/myusername/bin, you can initialize LD_LIBRARY_PATH to include that directory and preserve its existing contents as follows: % setenv LD_LIBRARY_PATH “$LD_LIBRARY_PATH”:/home/myusername/bin If you know that tobeshared.so will always reside in a specific directory, you can create the executable myprog in a form that assumes this using the -R link-time option. For example, you can link as follows: % pgf95 -o myprog myprof.f tobeshared.so -R/home/myusername/bin Libraries and Environment Variables 193 Note that there is no space between -R and the directory name. As with the -L option, no space can be present. If the -R option is used, it is not necessary to initialize LD_LIBRARY_PATH. In the example above, the dynamic linker will always look in /home/myusername/bin to resolve references to tobeshared.so. By default, if the LD_LIBRARY_PATH environment variable is not set, the linker will only search /usr/lib for shared objects. The command ldd is a useful tool when working with shared object files and executables that reference them. When applied to an executable as follows: % ldd myprog ldd lists all shared object files referenced in the executable along with the pathname of the directory from which they will be extracted. If the pathname is not hard-coded using the -R option, and if LD_LIBRARY_PATH is not initialized, the pathname is listed as “not found”. See the online man page for ldd for more information on options and usage. 8.3 Creating and Using Dynamic-Link Libraries on Win32 Some of the PGI compiler runtime libraries are available in both static library and dynamic-link library (DLL) form for Windows. There are several differences between these two types of libraries. Both libraries are used when resolving external references when linking an executable, but the process differs for each type of library. When linking with a static library, the code needed from the library is incorporated into the executable. Once the executable has been built, the library is no longer needed; the executable does not rely on the static library at runtime. When linking with a DLL, external references are resolved using the DLL's import library, not the DLL itself. The code in the DLL associated with the external references does not become a part of the executable. The DLL is loaded when the executable that needs it is run. For the DLL to be loaded in this manner, the DLL must be in your path, Windows directory, or Windows systems directory. Static libraries and DLLs also handle global data differently. If two static libraries contain global data with the same name, and both libraries are linked to the executable, the global data item in the libraries will be resolved to the same memory location. If this situation occurs with two DLLs, however, the global data items in each DLL are resolved to separate memory locations. In short, global data in a DLL cannot be directly accessed from outside the DLL. The PGI runtime DLLs can be used to create both executables and other DLLs. 194 Chapter 8 The following switches apply: –Mdll Link with the DLL version of the runtime libraries. This flag is required when linking with any DLL built by the PGI compilers. –Mmakedll Generate a dynamic-link library or DLL. –Mnopgdllmain Do not link the module containing the default DllMain() into the DLL. This flag applies to building DLLs with the PGF95 and PGHPF compilers. If you want to replace the default DllMain() routine with a custom DllMain(), use this flag and add the object containing the custom DllMain() to the link line. The latest version of the default DllMain() is included in the Release Notes; the code in this routine specific to PGF95 and PGHPF must be incorporated into the custom version of DllMain() to ensure the appropriate function of your DLL. –o Passed to the linker. Name the DLL . ––output-def Passed to linker. Generate a .def named for the DLL. The .def file contains the symbols exported by the DLL. Generating a .def file is not required when building a DLL but can be a useful debugging tool if the DLL does not contain the symbols that you expect it to contain. You can also create your own .def file, containing the symbols you want to export to the DLL. To use your .def file, add it to the link line and omit ––output-def. ––out-implib Passed to linker. Generate an import library named for the DLL. A DLL’s import library is the interface used when linking an executable that depends on routines in a DLL. ––export-all-symbols Passed to linker. Use this flag to export all global and weak defined symbols to the DLL. Even with this flag, some symbols are not exported; see ––no-default-excludes. Libraries and Environment Variables 195 ––no-default-excludes Passed to linker. When ––export-all-symbols is used, there are still some special symbols (i.e., DllMain@12) that are not exported. Use ––no-default-excludes to export these symbols to the DLL. To use the PGI compilers to create an executable that links to the DLL form of the runtime, use the compiler flag –Mdll. The executable built will be smaller than one built without –Mdll; the PGI runtime DLLs, however, must be available when the executable is run. The –Mdll flag must be used when an executable is linked against a DLL built by the PGI compilers. Each PGI compiler can also create DLLs for Windows. The following examples outline how to use –Mmakedll to do so. Example 1: Build a DLL out of two source files, object1.f and object2.f, and use it to build the main source file, prog1.f. Step 1: object1.f: subroutine subf1 (n) integer n n=1 print *,"n=",n return end object2.f: function funf2 () real funf2 funf2 = 2.0 return end 196 Chapter 8 prog1.f: program test external subf1 real funf2, val integer n call subf1(n) val = funf2() write (*,*) 'val = ', val stop end Step 2: Create the DLL obj12.dll and its import library obj12.lib using the following series of commands: % pgf95 -c object1.f object2.f % pgf95 object1.o object2.o -Mmakedll -o obj12.dll \ --out-implib obj12.lib Step 3: Compile the main program: % pgf95 -Mdll -o prog1 prog1.f -L. -lobj12 The –Mdll switch causes the compiler to link against the PGI runtime DLLs instead of the PGI runtime static libraries. The –Mdll switch is required when linking against any PGI-compiled DLL such as obj12.dll. The –l switch is used to specify that obj12.lib, the DLL’s import library, will be used to resolve the calls to subf1 and funf2 in prog1.f. Step 4: Ensure that obj12.dll is in your path, then run the executable 'prog1' to determine if the DLL was successfully created and linked: % prog1 n=1 val = 2.000000 FORTRAN STOP Libraries and Environment Variables 197 Should you wish to change obj12.dll without changing the subroutine or function interfaces, no rebuilding of prog1 is necessary. Just recreate obj12.dll, and the new obj12.dll will be loaded at runtime. Example 2: Build two DLLs when each DLL is dependent on the other, and use them to build the main program. In the following source files, object3.c makes calls to routines defined in object4.c, and vice versa. This situation of mutual imports requires two steps to build each DLL. object3.c: extern void func_4b(void); void func_3a(void) { printf("func_3a, calling a routine in obj4.dll\n"); func_4b(); } void func_3b(void) { printf("func_3b\n"); } object4.c: extern void func_3b(void); void func_4a(void) { printf("func_4a, calling a routine in obj3.dll\n"); func_3b(); } void func_4b(void) { printf("func_4b\n"); } prog2.c: extern void func_3a(void); extern void func_4a(void); int main() { func_3a(); func_4a(); } 198 Chapter 8 Step 1: To make obj3.dll and obj4.dll, first compile the source and create an import library for each DLL that will be built. % pgcc -c object3.c % pgcc object3.o -Mmakedll -o obj3.dll --out-implib obj3.lib Creating library file: obj3.lib object3.o(.text+0x24):object3.c: undefined reference to `func_4b' % pgcc -c object4.c % pgcc object4.o -Mmakedll -o obj4.dll --out-implib obj4.lib Creating library file: obj4.lib object4.o(.text+0x24):object4.c: undefined reference to `func_3b' The undefined reference errors are to be expected in the first step of building DLLs with mutual imports. These errors will be resolved in the next step where each DLL is built and linked against the import library previously created for the other DLL. % pgcc object3.o -Mmakedll -o obj3.dll -L. -lobj4 % pgcc object4.o -Mmakedll -o obj4.dll -L. -lobj3 Step 2: Compile the main program and link against the import libraries for obj3.dll and obj4.dll. % pgcc -Mdll -o prog2 prog2.c -L. -lobj3 -lobj4 Step 3: Execute prog2 to ensure that the DLLs were created properly: % prog2 func_3a, calling a routine in obj4.dll func_4b func_4a, calling a routine in obj3.dll func_3b 8.4 Creating and Using Dynamic-Link Libraries on Win64 Some of the PGI compiler runtime libraries are available in both static library and dynamic-link library (DLL) form for Win64. The static libraries are still the default during linking. To use the Libraries and Environment Variables 199 PGI Workstation C and Fortran compilers to create an executable that links to the runtime DLLs, use the compiler flag –Mdll at the link step. There are several differences between static and dynamic-link libraries. Both libraries are used when resolving external references when linking an executable, but the process differs for each type of library. When linking with a static library, the code needed from the library is incorporated into the executable. When linking with a DLL, external references are resolved using the DLL's import library, not the DLL itself. The code in the DLL associated with the external references does not become a part of the executable. The DLL is loaded when the executable that needs it is run. For the DLL to be loaded in this manner, the DLL must be in your path. Static libraries and DLLs also handle global data differently. Global data in static libraries is automatically accessible to other objects linked into an executable. Global data in a DLL can only be accessed from outside the DLL if the DLL exports the data and the image that uses the data imports it. To this end the C compilers support the Microsoft storage class extensions __declspec(dllimport) and __declspec(dllexport). These extensions may appear as storage class modifiers and enable functions and data to be imported and exported: extern int __declspec(dllimport) intfunc(); float __declspec(dllexport) fdata; The Fortran compilers support the DEC ATTRIBUTES extensions DLLIMPORT and DLLEXPORT: cDEC$ ATTRIBUTES DLLEXPORT :: object [,object] ... cDEC$ ATTRIBUTES DLLIMPORT :: object [,object] ... c is one of C, c, !, or *. object is the name of the subprogram or common block that is exported or imported. Note that common block names are enclosed within slashes (/). In example: cDEC$ ATTRIBUTES DLLIMPORT :: intfunc !DEC$ ATTRIBUTES DLLEXPORT :: /fdata/ The Examples in this section further illustrate the use of these extensions. The PGI runtime DLLs can be used to create both executables and other DLLs. The following switches apply: –Mdll Link with the DLL version of the runtime libraries. This flag is required when linking with any DLL built by the PGI compilers. –Mmakedll Generate a dynamic-link library or DLL. –Mmakeimplib Generate an import library without generating a DLL. Use this flag when you want to generate an import library for a DLL but are not yet ready to build the 200 Chapter 8 DLL itself. This situation might arise, for example, when building DLLs with mutual imports (see Example 32 below). –o Passed to the linker. Name the DLL or import library . –def When used with –Mmakedll, this flag is passed to the linker and a .def file named is generated for the DLL. The .def file contains the symbols exported by the DLL. Generating a .def file is not required when building a DLL but can be a useful debugging tool if the DLL does not contain the symbols that you expect it to contain. When used with –Mmakeimplib, this flag is passed to lib which requires a .def file to create an import library. The .def file can be empty if the list of symbols to export are passed to lib on the command line or explicitly marked as dllexport in the source code. –implib Passed to linker. Generate an import library named for the DLL. A DLL’s import library is the interface used when linking an executable that depends on routines in a DLL. To use the PGI compilers to create an executable that links to the DLL form of the runtime, use the compiler flag –Mdll. The executable built will be smaller than one built without –Mdll; the PGI runtime DLLs, however, must be available when the executable is run. The –Mdll flag must be used when an executable is linked against a DLL built by the PGI compilers. The following examples outline how to use –Mmakedll and –Mmakeimplib to build and use DLLs with the PGI compilers. Example 1: Build a DLL out of a single source file, object1.f, which exports data and a subroutine using DLLEXPORT. Build the main source file, prog1.f, which uses DLLIMPORT to import the data and subroutine from the DLL. object1.f: subroutine sub1(i) !DEC$ ATTRIBUTES DLLEXPORT :: sub1 integer i common /acommon/ adata integer adata !DEC$ ATTRIBUTES DLLEXPORT :: /acommon/ print *, "sub1 adata", adata print *, "sub1 i ", i adata = i Libraries and Environment Variables 201 end prog1.f: program prog1 common /acommon/ adata integer adata external sub1 !DEC$ ATTRIBUTES DLLIMPORT:: sub1, /acommon/ adata = 11 call sub1(12) print *, "main adata", adata end Step 1: Create the DLL obj1.dll and its import library obj1.lib using the following series of commands: % pgf95 -c object1.f % pgf95 –Mmakedll object1.obj -o obj1.dll Step 2: Compile the main program: % pgf95 -Mdll -o prog1 prog1.f -defaultlib:obj1 The –Mdll switch causes the compiler to link against the PGI runtime DLLs instead of the PGI runtime static libraries. The –Mdll switch is required when linking against any PGI-compiled DLL such as obj1.dll. The -defaultlib: switch is used to specify that obj1.lib, the DLL’s import library, should be used to resolve imports. Step 3: Ensure that obj1.dll is in your path, then run the executable prog1 to determine if the DLL was successfully created and linked: % prog1 sub1 adata 11 sub1 i 12 main adata 12 Should you wish to change obj1.dll without changing the subroutine or function interfaces, no rebuilding of prog1 is necessary. Just recreate obj1.dll and the new obj1.dll will be loaded at runtime. Example 2: Build a DLL out of a single source file, object2.c, which exports data and a subroutine using __declspec(dllexport). Build the main source file, prog2.c, which uses __declspec(dllimport) to import the data and subroutine from the DLL. 202 Chapter 8 object2.c: int __declspec(dllexport) data; void __declspec(dllexport) func2() { printf("in func2, data == %d\n", data); } prog2.c: int __declspec(dllimport) data; void __declspec(dllimport) func2(); int main() { data = 11; func2(); return 0; } Step 1: Create the DLL obj2.dll and its import library obj2.lib using the following series of commands: % pgcc -c object2.c % pgcc –Mmakedll object2.obj -o obj2.dll Step 2: Compile the main program: % pgcc -Mdll -o prog2 prog2.c -defaultlib:obj2 The –Mdll switch causes the compiler to link against the PGI runtime DLLs instead of the PGI runtime static libraries. The –Mdll switch is required when linking against any PGI-compiled DLL such as obj2.dll. The -defaultlib: switch is used to specify that obj2.lib, the DLL’s import library, should be used to resolve the imported data and subroutine in prog2.c. Step 3: Ensure that obj2.dll is in your path, then run the executable prog2 to determine if the DLL was successfully created and linked: % prog2 in func2, data == 11 Libraries and Environment Variables 203 Should you wish to change obj2.dll without changing the subroutine or function interfaces, no rebuilding of prog2 is necessary. Just recreate obj2.dll and the new obj2.dll will be loaded at runtime. Example 3: Build two DLLs when each DLL is dependent on the other, and use them to build the main program. In the following source files, object3.c makes calls to routines defined in object4.c, and vice versa. This situation of mutual imports requires two steps to build each DLL. object3.c: void __declspec(dllimport) func_4b(void); void __declspec(dllexport) func_3a(void) { printf("func_3a, calling a routine in obj4.dll\n"); func_4b(); } void __declspec(dllexport) func_3b(void) { printf("func_3b\n"); } object4.c: void __declspec(dllimport) func_3b(void); void __declspec(dllexport) func_4a(void) { printf("func_4a, calling a routine in obj3.dll\n"); func_3b(); } void __declspec(dllexport) func_4b(void) { printf("func_4b\n"); } prog3.c: void __declspec(dllimport) func_3a(void); void __declspec(dllimport) func_4a(void); int main() { 204 Chapter 8 func_3a(); func_4a(); return 0; } Step 1: To make obj3.dll and obj4.dll, first compile the source and create an import library for each DLL that will be built. The PGI drivers call the Microsoft lib tool to create import libraries. The lib tool will only create an import library if a module-definition (.def) file is provided. A .def file contains symbols to export. In this example, the symbols to be exported are already marked as such by __declspec(dllexport) so the .def file should be empty. % pgcc -c object3.c % pgcc -Mmakeimplib -o obj3.lib object3.obj -def obj3.def % pgcc -c object4.c % pgcc -Mmakeimplib -o obj4.lib object4.obj -def obj4.def Step 2: Create the DLLs using the import libraries obj3.lib and obj4.lib. % pgcc -Mmakedll -o obj3.dll object3.obj -defaultlib:obj4 % pgcc -Mmakedll -o obj4.dll object4.obj -defaultlib:obj3 Step 3: Compile the main program and link against the import libraries for obj3.dll and obj4.dll. % pgcc -Mdll prog3.c -o prog3 -defaultlib:obj3 -defaultlib:obj4 Step 4: Execute prog3 to ensure that the DLLs were created properly: % prog3 func_3a, calling a routine in obj4.dll func_4b func_4a, calling a routine in obj3.dll func_3b Example 2: Build two DLLs when each DLL is dependent on the other, and use them to build the main program. In the following source files, object2.f95 makes calls to routines defined in object3.f95, and vice versa. This situation of mutual imports requires two steps to build each DLL. object2.f95: subroutine func_2a external func_3b !DEC$ ATTRIBUTES DLLEXPORT :: func_2a !DEC$ ATTRIBUTES DLLIMPORT :: func_3b print*,"func_2a, calling a routine in obj3.dll” call func_3b() end subroutine Libraries and Environment Variables 205 subroutine func_2b !DEC$ ATTRIBUTES DLLEXPORT :: func_2b print*,"func_2b” end subroutine object3.f95 subroutine func_3a external func_2b !DEC$ ATTRIBUTES DLLEXPORT :: func_3a !DEC$ ATTRIBUTES DLLIMPORT :: func_2b print*,"func_3a, calling a routine in obj2.dll” call func_2b() end subroutine subroutine func_3b !DEC$ ATTRIBUTES DLLEXPORT :: func_3b print*,"func_3b” end subroutine prog2.f95: program prog2 external func_2a external func_3a !DEC$ ATTRIBUTES DLLIMPORT :: func_2a !DEC$ ATTRIBUTES DLLIMPORT :: func_3a call func_2a() call func_3a() end program Step 1: To make obj2.dll and obj3.dll, first compile the source and create an import library for each DLL that will be built. The PGI drivers call the Microsoft lib tool to create import libraries. The lib tool will only create an import library if a module-definition (.def) file is provided. A .def file contains symbols to export. In this example, the symbols to be exported are already marked as such by the DLLIMPORT statements, so the .def file should be empty. % touch obj2.def % pgf95 -c object2.f95 % pgf95 -Mmakeimplib -o obj2.lib object2.obj -def obj2.def % touch obj3.def % pgf95 -c object3.f95 % pgf95 -Mmakeimplib -o obj3.lib object3.obj -def obj3.def Step 2: Create the DLLs using the import libraries obj3.lib and obj4.lib. 206 Chapter 8 % pgf95 -Mmakedll -o obj2.dll object2.obj -defaultlib:obj3 % pgf95 -Mmakedll -o obj3.dll object3.obj -defaultlib:obj2 Step 3: Compile the main program and link against the import libraries for obj3.dll and obj4.dll. % pgf95 -Mdll prog2.f95 -o prog2 -defaultlib:obj2 -defaultlib:obj3 Step 4: Execute prog2 to ensure that the DLLs were created properly: % prog2 func_2a, calling a routine in obj3.dll func_3b func_3a, calling a routine in obj2.dll func_2b 8.5 Using LIB3F The PGI Fortran compilers include complete support for the de facto standard LIB3F library routines on both Linux and Windows operating systems. See the PGI Fortran Reference manual for a complete list of available routines in the PGI implementation of LIB3F. 8.6 LAPACK, the BLAS and FFTs Pre-compiled versions of the public domain LAPACK and BLAS libraries are included with the PGI compilers on Linux and Windows systems in the files $PGI//lib/lapack.a and $PGI//lib/blas.a respectively, where is replaced with the appropriate target name (linux86, linux86-64, win64, or nt86). To use these libraries, simply link them in using the -l option when linking your main program: % pgf95 myprog.f -lblas -llapack Highly optimized assembly-coded versions of the BLAS and certain FFT routines may be available for your platform. In some cases, these are shipped with the PGI compilers. See the current release notes for the PGI compilers you are using to determine if these optimized libraries exist, where they can be downloaded (if necessary), and how to incorporate them into your installation as the default. Libraries and Environment Variables 207 8.7 The C++ Standard Template Library The PGC++ compiler includes a bundled copy of the STLPort Standard C++ Library. See the online Standard C++ Library tutorial and reference manual at http://www.stlport.com for further details and licensing. 8.8 Environment Variables Several environment variables can be used to alter the default behavior of the PGI compilers and the executables which they generate. Many of these environment variables are documented in context in other sections of the PGI User’s Guide. They are gathered here for easy reference. Specifically excluded are environment variables specific to OpenMP which are used to control the behavior of OpenMP programs. See section 5.17, Environment Variables, for a list and description of environment variables that affect the execution of Fortran OpenMP programs. See section 6.16, Environment Variables, for a list and description of environment variables that affect the execution of C and C++ OpenMP programs. Also excluded are environment variables that control the behavior of the PGDBG debugger or PGPROF profiler. See the PGI Tools Guide for a description of environment variables that affect these tools. FORTRAN_OPT - If this variable exists and contains the value vaxio, the record length in the open statement is in units of 4-byte words, and the $ edit descriptor only has an effect for lines beginning with a space or +. If this variable exists and contains the value format_relaxed, an I/O item corresponding to a numerical edit descriptor (F, E, I, etc.) is not required to be a type implied by the descriptor. For example: $ setenv FORTRAN_OPT vaxio will cause the PGI Fortran compilers to use VAX I/O conventions as defined above. MPSTKZ - increase the size of the stacks used by threads executing in parallel regions. It is for use with programs that utilize large amounts of thread-local storage in the form of private variables or local variables in functions or subroutines called within parallel regions. The value should be an integer concatenated with M or m to specify stack sizes of n megabytes. For example: $ setenv MPSTKZ 8M MP_BIND - the MP_BIND environment variable can be set to yes or y to bind processes or threads executing in a parallel region to physical processors, or to no or n to disable such binding. The default is to not bind processes to processors. This is an execution time environment variable 208 Chapter 8 interpreted by the PGI runtime support libraries. It does not affect the behavior of the PGI compilers in any way. Note: the MP_BIND environment variable is not supported on all platforms. MP_BLIST - In addition to the MP_BIND variable, it is possible to define the thread-CPU relationship. For example, setting MP_BLIST=3,2,1,0 maps CPUs 3, 2, 1 and 0 to threads 0, 1, 2 and 3 respectively. MP_SPIN - When a thread executing in a parallel region enters a barrier, it spins on a semaphore. MP_SPIN can be used to specify the number of times it checks the semaphore before calling sched_yield() (on linux) or _sleep() (on Windows). These calls cause the thread to be rescheduled, allowing other processes to run. The default values are 100 (Linux) and 10000 (Windows). MP_WARN - By default, a warning will be printed to stderr if you execute an OpenMP or autoparallelized program with NCPUS or OMP_NUM_THREADS set to a value larger than the number of physical processors in the system. For example, if you produce a parallelized executable a.out and execute as follows on a system with only one processor: % setenv NCPUS 2 % a.out Warning: OMP_NUM_THREADS or NCPUS (2) greater than available cpus (1) FORTRAN STOP Setting MP_WARN to no will eliminate these warning messages. NCPUS - The NCPUS environment variable can be used to set the number of processes or threads used in parallel regions. The default is to use only one process or thread (serial mode). If both OMP_NUM_THREADS and NCPUS are set, the value of OMP_NUM_THREADS takes precedence. Warning: setting NCPUS to a value larger than the number of physical processors or cores in your system can cause parallel programs to run very slowly. NCPUS_MAX - The NCPUS_MAX environment variable can be used to limit the maximum number of processes or threads used in a parallel program. Attempts to dynamically set the number of processes or threads to a higher value, for example using set_omp_num_threads(), will cause the number of processes or threads to be set at the value of NCPUS_MAX rather than the value specified in the function call. NO_STOP_MESSAGE - If this variable exists, the execution of a plain STOP statement does not produce the message FORTRAN STOP. The default behavior of the PGI Fortran compilers is to issue this message. PGI - The PGI environment variable specifies the root directory where the PGI compilers and tools are installed. The default value of this variable is /usr/pgi. In most cases, the name of this root directory is derived dynamically by the PGI compilers and tools through determination of the Libraries and Environment Variables 209 path to the instance of the compiler or tool that has been invoked. However, there are still some dependences on the PGI environment variable, and it can be used as a convenience when initializing your environment for use of the PGI compilers and tools. For example, assuming you use csh and want the 64-bit linux86-64 versions of the PGI compilers and tools to be default: % setenv PGI /usr/pgi % setenv MANPATH "$MANPATH":$PGI/linux86/6.0/man % setenv LM_LICENSE_FILE $PGI/license.dat % set path = ($PGI/linux86-64/6.0/bin $path) PGI_CONTINUE - If the PGI_CONTINUE environment variable is set upon execution of a program compiled with –Mchkfpstk, the stack will be automatically cleaned up and execution will continue. There is a performance penalty associated with the stack cleanup. If PGI_CONTINUE is set to verbose, the stack will be automatically cleaned up and execution will continue after printing of a warning message. STATIC_RANDOM_SEED - The first call to the Fortran 90/95 RANDOM_SEED intrinsic without arguments will reset the random seed to a default value, then advance the seed by a variable amount based on time. Subsequent calls to RANDOM_SEED without arguments will reset the random seed to the same initial value as the first call. Unless the time is exactly the same, each time a program is run a different random number sequence will be generated. You can force the seed returned by RANDOM_SEED to be constant, thereby generating the same sequence of random numbers at each execution of the program, by setting the environment variable STATIC_RANDOM_SEED to yes. TMPDIR - Can be used to specify the directory that should be used for placement of any temporary files created during execution of the PGI compilers and tools. TZ - Can be used to explicitly set the time zone, and is used in some contexts by the PGC++ compiler. For more information on the possible settings for TZ, use the tzselect utility on Linux for a detailed description of possible settings and step-by-step instructions for setting the value of TZ for a given time zone. Fortran, C and C++ Data Types 211 Chapter 9 Fortran, C and C++ Data Types This chapter describes the scalar and aggregate data types recognized by the PGI Fortran, C, and C++ compilers, the format and alignment of each type in memory, and the range of values each type can take on x86 or x64 processor-based systems running a 32-bit operating system. For more information on x86-specific data representation, refer to the System V Application Binary Interface, Processor Supplement, listed in the . This chapter specifically does not address x64 processor-based systems running a 64-bit operating system, because the application binary interface (ABI) for those systems is still evolving. See http://www.x86-64.org/abi.pdf for the latest version of this ABI. 9.1 Fortran Data Types 9.1.1 Fortran Scalars A scalar data type holds a single value, such as the integer value 42 or the real value 112.6. Table 9-1 lists scalar data types, their size, format and range. Table 9-2 shows the range and approximate precision for Fortran real data types. Table 9-3 shows the alignment for different scalar data types. The alignments apply to all scalars, whether they are independent or contained in an array, a structure or a union. Table 9-1: Representation of Fortran Data Types Fortran Data Type Format Range INTEGER 2's complement integer -2 31 to 2 31 -1 INTEGER*2 2's complement integer -32768 to 32767 INTEGER*4 same as INTEGER INTEGER*8 same as INTEGER -2 63 to 2 63 -1 LOGICAL same as INTEGER true or false LOGICAL*1 8 bit value true or false LOGICAL*2 16 bit value true or false LOGICAL*4 same as INTEGER true or false 212 Chapter 9 Fortran Data Type Format Range LOGICAL*8 same as INTEGER true or false BYTE 2's complement -128 to 127 REAL Single-precision floating point 10 -37 to 10 38 (1) REAL*4 Single-precision floating point 10 -37 to 10 38 (1) REAL*8 Double-precision floating point 10 -307 to 10 308 (1) DOUBLE PRECISION Double-precision floating point 10 -307 to 10 308 (1) COMPLEX See REAL See REAL DOUBLE COMPLEX See DOUBLE PRECISION See DOUBLE PRECISION COMPLEX*16 Same as above Same as above CHARACTER*n Sequence of n bytes (1) Approximate value The logical constants .TRUE. and .FALSE. are all ones and all zeroes, respectively. Internally, the value of a logical variable is true if the least significant bit is one and false otherwise. When the option –Munixlogical is set, a logical variable with a non-zero value is true and with a zero value is false. Table 9-2: Real Data Type Ranges Data Type Binary Range Decimal Range Digits of Precision REAL 2 -126 to 2 128 10 -37 to 10 38 7-8 REAL*8 2 -1022 to 2 1024 10 -307 to 10 308 15-16 Fortran, C and C++ Data Types 213 Table 9-3: Scalar Type Alignment Type Is Aligned on a LOGICAL*1 1-byte boundary LOGICAL*2 2-byte boundary LOGICAL*4 4-byte boundary LOGICAL*8 8-byte boundary BYTE 1-byte boundary INTEGER*2 2-byte boundary INTEGER*4 4-byte boundary INTEGER*8 8-byte boundary REAL*4 4-byte boundary REAL*8 8-byte boundary COMPLEX*8 4-byte boundary COMPLEX*16 8-byte boundary 9.1.2 FORTRAN 77 Aggregate Data Type Extensions The PGF77 compiler supports de facto standard extensions to FORTRAN 77 that allow for aggregate data types. An aggregate data type consists of one or more scalar data type objects. You can declare the following aggregate data types: array consists of one or more elements of a single data type placed in contiguous locations from first to last. structure is a structure that can contain different data types. The members are allocated in the order they appear in the definition but may not occupy contiguous locations. union is a single location that can contain any of a specified set of scalar or aggregate data types. A union can have only one value at a time. The data type of the union member to which data is assigned determines the data type of the union after that assignment. 214 Chapter 9 The alignment of an array, a structure or union (an aggregate) affects how much space the object occupies and how efficiently the processor can address members. Arrays use the alignment of their members. Array types align according to the alignment of the array elements. For example, an array of INTEGER*2 data aligns on a 2 byte boundary. Structures and Unions align according to the alignment of the most restricted data type of the structure or union. In the next example, the union aligns on a 4-byte boundary since the alignment of C, the most restrictive element, is four. STRUCTURE /ASTR/ UNION MAP INTEGER*2 A ! 2 bytes END MAP MAP BYTE B ! 1 byte END MAP MAP INTEGER*4 C ! 4 bytes END MAP END UNION END STRUCTURE Structure alignment can result in unused space called padding. Padding between members of the structure is called internal padding. Padding between the last member and the end of the space is called tail padding. The offset of a structure member from the beginning of the structure is a multiple of the member’s alignment. For example, since an INTEGER*2 aligns on a 2-byte boundary, the offset of an INTEGER*2 member from the beginning of a structure is a multiple of two bytes. 9.1.3 Fortran 90 Aggregate Data Types (Derived Types) The Fortran 90 standard added formal support for aggregate data types. The TYPE statement begins a derived type data specification or declares variables of a specified user-defined type. For example, the following would define a derived type ATTENDEE: TYPE ATTENDEE CHARACTER(LEN=30) NAME Fortran, C and C++ Data Types 215 CHARACTER(LEN=30) ORGANIZATION CHARACTER (LEN=30) EMAIL END TYPE ATTENDEE In order to declare a variable of type ATTENDEE and access the contents of such a variable, code such as the following would be used: TYPE (ATTENDEE) ATTLIST(100) . . . ATTLIST(1)%NAME = ‘JOHN DOE’ 9.2 C and C++ Data Types 9.2.1 C and C++ Scalars Table 9-4 lists C and C++ scalar data types, their size and format. The alignment of a scalar data type is equal to its size. Table 9-5 shows scalar alignments that apply to individual scalars and to scalars that are elements of an array or members of a structure or union. Wide characters are supported (character constants prefixed with an L). The size of each wide character is 4 bytes. Table 9-4: C/C++ Scalar Data Types Data Type Size (bytes) Format Range unsigned char 1 ordinal 0 to 255 [signed] char 1 two's-complement integer -128 to 127 unsigned short 2 ordinal 0 to 65535 [signed] short 2 two's-complement integer -32768 to 32767 unsigned int 4 ordinal 0 to 2 32 -1 [signed] int [signed] long int 4 two's-complement integer -2 31 to 2 31 -1 unsigned long int 4 ordinal 0 to 2 32 -1 [signed] long long [int] 8 two's-complement integer -2 63 to 2 63 -1 216 Chapter 9 Data Type Size (bytes) Format Range unsigned long long [int] 8 ordinal 0 to 2 64 -1 float 4 IEEE single-precision floating-point 10 -37 to 10 38 (1) double 8 IEEE doubleprecision floating-point 10 -307 to 10 308 (1) long double 8 IEEE doubleprecision floating-point 10 -307 to 10 308 (1) bit field (2) (unsigned value) 1 to 32 bits ordinal 0 to 2 size -1, where size is the number of bits in the bit field bit field (2) (signed value) 1 to 32 bits two's complement integer -2 size-1 to 2 size-1 -1, where size is the number of bits in the bit field pointer 4 address 0 to 2 32 -1 enum 4 two's complement integer -2 31 to 2 31 -1 (1) Approximate value (2) Bit fields occupy as many bits as you assign them, up to 4 bytes, and their length need not be a multiple of 8 bits (1 byte) Table 9-5: Scalar Alignment Data Type Alignment char is aligned on a 1-byte boundary. * short is aligned on a 2-byte boundary. * [long] int is aligned on a 4-byte boundary. * enum is aligned on a 4-byte boundary. pointer is aligned on a 4-byte boundary. float is aligned on a 4-byte boundary. double is aligned on an 8-byte boundary. long double is aligned on an 8-byte boundary. (*) signed or unsignedFortran, C and C++ Data Types 217 9.2.2 C and C++ Aggregate Data Types An aggregate data type consists of one or more scalar data type objects. You can declare the following aggregate data types: array consists of one or more elements of a single data type placed in contiguous locations from first to last. class (C++ only) is a class that defines an object and its member functions. The object can contain fundamental data types or other aggregates including other classes. The class members are allocated in the order they appear in the definition but may not occupy contiguous locations. struct is a structure that can contain different data types. The members are allocated in the order they appear in the definition but may not occupy contiguous locations. When a struct is defined with member functions, its alignment issues are the same as those for a class. union is a single location that can contain any of a specified set of scalar or aggregate data types. A union can have only one value at a time. The data type of the union member to which data is assigned determines the data type of the union after that assignment. 9.2.3 Class and Object Data Layout Class and structure objects with no virtual entities and with no base classes, that is just direct data field members, are laid out in the same manner as C structures. The following section describes the alignment and size of these C-like structures. C++ classes (and structures as a special case of a class) are more difficult to describe. Their alignment and size is determined by compiler generated fields in addition to user-specified fields. The following paragraphs describe how storage is laid out for more general classes. The user is warned that the alignment and size of a class (or structure) is dependent on the existence and placement of direct and virtual base classes and of virtual function information. The information that follows is for informational purposes only, reflects the current implementation, and is subject to change. Do not make assumptions about the layout of complex classes or structures. All classes are laid out in the same general way, using the following pattern (in the sequence indicated): • First, storage for all of the direct base classes (which implicitly includes storage for nonvirtual indirect base classes as well): ? When the direct base class is also virtual, only enough space is set aside for a pointer to the actual storage, which appears later. 218 Chapter 9 ? In the case of a non-virtual direct base class, enough storage is set aside for its own non-virtual base classes, its virtual base class pointers, its own fields, and its virtual function information, but no space is allocated for its virtual base classes. • Next, storage for the class’s own fields. • Next, storage for virtual function information (typically, a pointer to a virtual function table). • Finally, storage for its virtual base classes, with space enough in each case for its own nonvirtual base classes, virtual base class pointers, fields, and virtual function information. 9.2.4 Aggregate Alignment The alignment of an array, a structure or union (an aggregate) affects how much space the object occupies and how efficiently the processor can address members. Arrays use the alignment of their members. Arrays align according to the alignment of the array elements. For example, an array of short data type aligns on a 2-byte boundary. Structures and Unions align according to the most restrictive alignment of the enclosing members. For example the union un1 below aligns on a 4-byte boundary since the alignment of c, the most restrictive element, is four: union un1 { short a; /* 2 bytes */ char b; /* 1 byte */ int c; /* 4 bytes */ }; Structure alignment can result in unused space, called padding. Padding between members of a structure is called internal padding. Padding between the last member and the end of the space occupied by the structure is called tail padding. Figure 9-1 illustrates structure alignment. Consider the following structure: struct strc1 { char a; /* occupies byte 0 */ short b; /* occupies bytes 2 and 3 */ char c; /* occupies byte 4 */ int d; /* occupies bytes 8 through 11 */ }; Fortran, C and C++ Data Types 219 b byte 0 xxxx byte 4 byte 8 d xxxx a c Figure 9-1: Internal Padding in a Structure Figure 9-2 shows how tail padding is applied to a structure aligned on a doubleword boundary. struct strc2{ int m1[4]; /* occupies bytes 0 through 15 */ double m2; /* occupies bytes 16 through 23 */ short m3; /* occupies bytes 24 and 25 */ } st; 9.2.5 Bit-field Alignment Bit-fields have the same size and alignment rules as other aggregates, with several additions to these rules: • Bit-fields are allocated from right to left. • A bit-field must entirely reside in a storage unit appropriate for its type. Bit-fields never cross unit boundaries. • Bit-fields may share a storage unit with other structure/union members, including members that are not bit-fields. • Unnamed bit-field's types do not affect the alignment of a structure or union. • Items of [signed/unsigned] long long type may not appear in field declarations. 220 Chapter 9 st.m1[0] st.m1[1] st.m1[2] st.m1[3] m2 m2 xxxx m3 xxxx byte 0 byte 4 byte 8 byte 12 byte 16 byte 20 byte 24 byte 28 Figure 9-2: Tail Padding in a Structure 9.2.6 Other Type Keywords in C and C++ The void data type is neither a scalar nor an aggregate. You can use void or void* as the return type of a function to indicate the function does not return a value, or as a pointer to an unspecified data type, respectively. The const and volatile type qualifiers do not in themselves define data types, but associate attributes with other types. Use const to specify that an identifier is a constant and is not to be changed. Use volatile to prevent optimization problems with data that can be changed from outside the program, such as memory-mapped I/O buffers. Inter-language Calling 221 Chapter 10 Inter-language Calling This chapter describes inter-language calling conventions for C, C++, and Fortran programs using the PGI compilers. The following sections describe how to call a Fortran function or subroutine from a C or C++ program and how to call a C or C++ function from a Fortran program. For information on calling assembly language programs, refer to Appendix A, Run-time Environment. 10.1 Overview of Calling Conventions This chapter includes information on the following topics: • Functions and subroutines in Fortran, C, and C++ • Naming and case conversion conventions • Compatible data types • Argument passing and special return values • Arrays and Indexes • Win32 calling conventions Default Fortran calling conventions under Win32 differ from those used under Linux and Win64 operating systems. Win32 programs compiled using the -Munix Fortran command-line option use the Linux/Win64 convention rather than the default Win32 convention. Sections 6.1 through 6.13 describe how to perform inter-language calling using the Linux/Win64 convention. All information in those sections pertaining to compatibility of arguments applies to Win32 as well. See Section 10.14 Win32 Calling Conventions for details on the symbol name and argument passing conventions used on Win32 platforms. 10.2 Inter-language Calling Considerations In general, when argument data types and function return values agree you can call a C or C++ function from Fortran and likewise, you can call a Fortran function from C or C++. You may need to develop special procedures in cases where data types for arguments do not agree. For example, the Fortran COMPLEX type does not have a matching type in C, it is still possible to provide inter-222 Chapter 10 language calls but there are no general calling conventions for such cases. In this instance, you need to develop a special procedure. Follow these guidelines: • Note that if a C++ function contains objects with constructors and destructors, calling such a function from either C or Fortran will not be possible unless the initialization in the main program is performed from a C++ program where constructors and destructors are properly initialized. • In general, you can call a C function from C++ without problems as long as you use the extern "C" keyword to declare the C function in the C++ program. This prevents name mangling for the C function name. If you want to call a C++ function from C, likewise you have to use the extern "C" keyword to declare the C++ function. This keeps the C++ compiler from mangling the name of the function. • You can use the __cplusplus macro to allow a program or header file to work for both C and C++. For example, the following defines in the header file stdio.h allow this file to work for both C and C++. #ifndef _STDIO_H #define _STDIO_H #ifdef __cplusplus extern "C" { #endif /* __cplusplus */ . . /* Functions and data types defined... */ . #ifdef __cplusplus } #endif /* __cplusplus */ #endif C++ member functions cannot be declared extern, as their names will always be mangled. Therefore, C++ member functions cannot be called from C or Fortran. Inter-language Calling 223 10.3 Functions and Subroutines Fortran, C, and C++ define functions and subroutines differently. For a Fortran program calling a C or C++ function, observe the following return value convention: • When the C or C++ function returns a value, call it from Fortran as a function, and otherwise call it as a subroutine. For a C/C++ program calling a Fortran function, the call should return a similar type. Table 10-1 lists compatible types. If the call is to a Fortran subroutine, a Fortran CHARACTER function, or a Fortran COMPLEX function, call it from C/C++ as a function that returns void. The exception to this convention is when a Fortran subroutine has alternate returns; call such a subroutine from C/C++ as a function returning int whose value is the value of the integer expression specified in the alternate RETURN statement. 10.4 Upper and Lower Case Conventions, Underscores By default on Linux systems, all Fortran symbol names are converted to lower case. C and C++ are case sensitive, so upper-case function names stay upper-case. When you use inter-language calling, you can either name your C/C++ functions with lower-case names, or invoke the Fortran compiler command with the option –Mupcase, in which case it will not convert symbol names to lower-case. When programs are compiled using one of the PGI Fortran compilers on Linux systems, an underscore is appended to Fortran global names (names of functions, subroutines and common blocks). This mechanism distinguishes Fortran name space from C/C++ name space. Use these naming conventions: • If you call a C/C++ function from Fortran, you should rename the C/C++ function by appending an underscore (or use C$PRAGMA C in the Fortran program, refer to Chapter 7, Optimization Directives and Pragmas, for details on C$PRAGMA C). • If you call a Fortran function from C/C++, you should append an underscore to the Fortran function name in the calling program. 10.5 Compatible Data Types Table 10-1 shows compatible data types between Fortran and C/C++. Table 10-2 shows how the Fortran COMPLEX type may be represented in C/C++. If you can make your function/subroutine parameters and return values match types, you should be able to use inter-language calling. 224 Chapter 10 Table 10-1: Fortran and C/C++ Data Type Compatibility Fortran Type (lower case) C/C++ Type Size (bytes) character x char x 1 character*n x char x[n] n real x float x 4 real*4 x float x 4 real*8 x double x 8 double precision double x 8 integer x int x 4 integer*1 x signed char x 1 integer*2 x short x 2 integer*4 x int x 4 integer*8 x long long x 8 logical x int x 4 logical*1 x char x 1 logical*2 x short x 2 logical*4 int x 4 logical*8 long long x 8 Table 10-2: Fortran and C/C++ Representation of the COMPLEX Type Fortran Type (lower case) C/C++ Type Size (bytes) complex x struct {float r,i;} x; 8 complex*8 x struct {float r,i;} x; 8 double complex x struct {double dr,di;} x; 16 Inter-language Calling 225 10.5.1 Fortran Named Common Blocks A named Fortran common block can be represented in C/C++ by a structure whose members correspond to the members of the common block. The name of the structure in C/C++ must have the added underscore. For example, the Fortran common block: INTEGER I COMPLEX C DOUBLE COMPLEX CD DOUBLE PRECISION D COMMON /COM/ i, c, cd, d is represented in C with the following equivalent: extern struct { int i; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; and in C++ with the following equivalent: extern “C” struct { int i; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; 10.6 Argument Passing and Return Values In Fortran, arguments are passed by reference (i.e. the address of the argument is passed, rather than the argument itself). In C/C++, arguments are passed by value, except for strings and arrays, which are passed by reference. Due to the flexibility provided in C/C++, you can work around these differences. Solving the parameter passing differences generally involves intelligent use of the & and * operators in argument passing when C/C++ calls Fortran and in argument declarations when Fortran calls C/C++. For strings declared in Fortran as type CHARACTER, an argument representing the length of the string is passed to a calling function. On Linux systems, or when using the UNIX calling 226 Chapter 10 convention on Windows (-Munix), the compiler places the length argument(s) at the end of the parameter list, following the other formal arguments. The length argument is passed by value, not by reference. 10.6.1 Passing by Value (%VAL) When passing parameters from a Fortran subprogram to a C/C++ function, it is possible to pass by value using the %VAL function. If you enclose a Fortran parameter with %VAL(), the parameter is passed by value. For example, the following call passes the integer I and the logical BVAR by value. INTEGER*1 I LOGICAL*1 BVAR CALL CVALUE (%VAL(I), %VAL(BVAR)) 10.6.2 Character Return Values Section 10.3 Functions and Subroutines describes the general rules for return values for C/C++ and Fortran inter-language calling. There is a special return value to consider. When a Fortran function returns a character, two arguments need to be added at the beginning of the C/C++ calling function’s argument list: • the address of the return character or characters • the length of the return character Example 10-1 illustrates the extra parameters, tmp and 10, supplied by the caller: CHARACTER*(*) FUNCTION CHF( C1, I) CHARACTER*(*) C1 INTEGER I END extern void chf_(); char tmp[10]; char c1[9]; int i; chf_(tmp, 10, c1, &i, 9); Example 10-1: Character Return Parameters Inter-language Calling 227 If the Fortran function is declared to return a character value of constant length, for example CHARACTER*4 FUNCTION CHF(), the second extra parameter representing the length must still be supplied, but is not used. Note: The value of the character function is not automatically NULL-terminated. 10.6.3 Complex Return Values When a Fortran function returns a complex value, an argument needs to be added at the beginning of the C/C++ calling function’s argument list; this argument is the address of the complex return value. Example 10-2 illustrates the extra parameter, cplx, supplied by the caller: COMPLEX FUNCTION CF(C, I) INTEGER I . . . END extern void cf_(); typedef struct {float real, imag;} cplx; cplx c1; int i; cf_(&c1, &i); Example 10-2: COMPLEX Return Values 10.7 Array Indices C/C++ arrays and Fortran arrays use different default initial array index values. By default, C/C++ arrays start at 0 and Fortran arrays start at 1. If you adjust your array comparisons so that a Fortran second element is compared to a C/C++ first element, and adjust similarly for other elements, you should not have problems working with this difference. If this is not satisfactory, you can declare your Fortran arrays to start at zero. Another difference between Fortran and C/C++ arrays is the storage method used. Fortran uses column-major order and C/C++ use row-major order. For one-dimensional arrays, this poses no problems. For two-dimensional arrays, where there are an equal number of rows and columns, row and column indexes can simply be reversed. For arrays other than single dimensional arrays, and square two-dimensional arrays, inter-language function mixing is not recommended. 228 Chapter 10 10.8 Example - Fortran Calling C Example 10-4 shows a C function that is called by the Fortran main program shown in Example 10-3. Notice that each argument is defined as a pointer, since Fortran passes by reference. Also notice that the C function name uses all lower-case and a trailing "_". logical*1 bool1 character letter1 integer*4 numint1, numint2 real numfloat1 double precision numdoub1 integer*2 numshor1 external cfunc call cfunc (bool1, letter1, numint1, numint2, & numfloat1, numdoub1, numshor1) write( *, "(L2, A2, I5, I5, F6.1, F6.1, I5)") & bool1, letter1, numint1, numint2, numfloat1, & numdoub1, numshor1 end Example 10-3: Fortran Main Program fmain.f #define TRUE 0xff #define FALSE 0 void cfunc_( bool1, letter1, numint1, numint2, numfloat1,\ numdoub1, numshor1, len_letter1) char *bool1, *letter1; int *numint1, *numint2; float *numfloat1; double *numdoub1; short *numshor1; int len_letter1; { *bool1 = TRUE; *letter1 = 'v'; *numint1 = 11; *numint2 = -44; *numfloat1 = 39.6 ; *numdoub1 = 39.2 ; *numshor1 = 981; } Example 10-4: C function cfunc_Inter-language Calling 229 Compile and execute the program fmain.f with the call to cfunc_ using the following command lines: $ pgcc -c cfunc.c $ pgf95 cfunc.o fmain.f Executing the a.out file should produce the following output: T v 11 -44 39.6 39.2 981 10.9 Example - C Calling Fortran Example 10-6 shows a C main program that calls the Fortran subroutine shown in Example 10-5. Notice that each call uses the & operator to pass by reference. Also notice that the call to the Fortran subroutine uses all lower-case and a trailing "_". subroutine forts ( bool1, letter1, numint1, & numint2, numfloat1, numdoub1, numshor1) logical*1 bool1 character letter1 integer numint1, numint2 double precision numdoub1 real numfloat1 integer*2 numshor1 bool1 = .true. letter1 = "v" numint1 = 11 numint2 = -44 numdoub1 = 902 numfloat1 = 39.6 numshor1 = 299 return end Example 10-5: Fortran Subroutine forts.f230 Chapter 10 main () { char bool1, letter1; int numint1, numint2; float numfloat1; double numdoub1; short numshor1; extern void forts_ (); forts_(&bool1,&letter1,&numint1,&numint2,&numfloat1, &numdoub1,&numshor1, 1); printf(" %s %c %d %d %3.1f %.0f %d\n", bool1?"TRUE":"FALSE",letter1,numint1, numint2, numfloat1, numdoub1, numshor1); } Example 10-6: C Main Program cmain.c To compile this Fortran subroutine and C program, use the following commands: $ pgcc -c cmain.f $ pgf95 -Mnomain cmain.o forts.f Executing the resulting a.out file should produce the following output: TRUE v 11 -44 39.6 902 299 10.10 Example - C ++ Calling C void cfunc(num1, num2, res) int num1, num2, *res; { printf("func: a = %d b = %d ptr c = %x\n",num1,num2,res); *res=num1/num2; printf("func: res = %d\n",*res); } Example 10-7: Simple C Function cfunc.cInter-language Calling 231 xtern "C" void cfunc(int n, int m, int *p); #include main() { int a,b,c; a=8; b=2; cout << "main: a = "< extern "C" { extern void forts_(char *,char *,int *,int *, float *,double *,short *); } main () { char bool1, letter1; int numint1, numint2; float numfloat1; double numdoub1; short numshor1; forts_(&bool1,&letter1,&numint1,&numint2,&numfloat1, &numdoub1,&numshor1); cout << “ bool1 = “; bool1?cout << “TRUE “:cout << “FALSE “; cout << endl; cout << “ letter1 = “ << letter1 << endl; cout << “ numint1 = “ << numint1 << endl; cout << “ numint2 = “ << numint2 << endl; cout << “ numfloat1 = “ << numfloat1 << endl; cout << “ numdoub1 = “ << numdoub1 << endl; Inter-language Calling 235 cout << “ numshor1 = “ << numshor1 << endl; } Example 10-14: C++ main program cpmain.C To compile this Fortran subroutine and C++ program, use the following command lines: $ pgf95 -c forts.f $ pgCC forts.o cpmain.C -lpgf95 -lpgf95_rpm1 -lpgf952 \ -lpgf95rtl -lpgftnrtl Executing this C++ main should produce the following output: bool1 = TRUE letter1 = v numint1 = 11 numint2 = -44 numfloat1 = 39.6 numdoub1 = 902 numshor1 = 299 Note that you must explicitly link in the PGF95 runtime support libraries when linking pgf95- compiled program units into C or C++ main programs. When linking pgf77 -compiled program units into C or C++ main programs, you need only link in –lpgftnrtl. 10.14 Win32 Calling Conventions Aside from name-mangling considerations in C++, the calling convention (i.e., the symbol name to which the subroutine or function name is mapped and the means by which arguments are passed) for C/C++ is identical between most compilers on Win32 and Linux/Win64. However, Fortran calling conventions vary widely between legacy Win32 Fortran compilers and Linux/Win64 Fortran compilers. 10.14.1 Win32 Fortran Calling Conventions Four styles of calling conventions are supported using the PGI Fortran compilers for Win32: Default, C, STDCALL, and UNIX. • Default - Default is the method used in the absence of compilation flags or directives to alter the default. 236 Chapter 10 • C or STDCALL - The C or STDCALL conventions are used if an appropriate compiler directive is placed in a program unit containing the call. The C and STDCALL conventions are typically used to call routines coded in C or assembly language that depend on these conventions. • UNIX - The UNIX convention is used in any Fortran program unit compiled using the -Munix compilation flag. Table 10-3 outlines each of these calling conventions. Table 10-3: Calling Conventions Supported by the PGI Fortran Compilers Convention Default STDCALL C UNIX Case of symbol name Upper Lower Lower Lower Leading underscore Yes Yes Yes Yes Trailing underscore No No No Yes Argument byte count added Yes Yes No No Arguments passed by reference Yes No* No* Yes Character argument byte counts passed After each char argument No No End of argument list Character strings truncated to first character and passed by value No Yes Yes No varargs support No No Yes No Caller cleans stack No No Yes Yes * Except arrays, which are always passed by reference even in the STDCALL and C conventions Inter-language Calling 237 Note: While it is compatible with the Fortran implementations of Microsoft and several other vendors, the C calling convention supported by the PGI Fortran compilers for Windows is not strictly compatible with the C calling convention used by most C/C++ compilers. In particular, symbol names produced by PGI Fortran compilers using the C convention are all lower case. The standard C convention is to preserve mixed-case symbol names. You can cause any of the PGI Fortran compilers to preserve mixed-case symbol names using the -Mupcase option, but be aware that this could have other ramifications on your program. 10.14.2 Symbol Name Construction and Calling Example This section presents an example of the rules outlined in table 10-3. In the pseudocode used below, %addr refers to the address of a data item while %val refers to the value of that data item. Subroutine and function names are converted into symbol names according to the rules outlined in table 10-3. Consider the following subroutine call: call work ( ‘ERR’, a, b, n) where a is a double precision scalar, b is a real vector of size n, and n is an integer. • Default - The symbol name for the subroutine is constructed by pre-pending an underscore, converting to all upper case, and appending an @ sign followed by an integer indicating the total number of bytes occupied by the argument list. Byte counts for character arguments appear immediately following the corresponding argument in the argument list. The following is an example of the pseudo-code for the above call using Default conventions: call _WORK@20 (%addr(‘ERR’), 3, %addr(a), %addr(b), %addr(n)) • STDCALL - The symbol name for the subroutine is constructed by pre-pending an underscore, converting to all lower case, and appending an @ sign followed by an integer indicating the total number of bytes occupied by the argument list. Character strings are truncated to the first character in the string, which is passed by value as the first byte in a 4-byte word. The following is an example of the pseudo-code for the above call using STDCALL conventions: call _work@20 (%val(‘E’), %val(a), %addr(b), %val(n)) 238 Chapter 10 Note that in this case there are still 20 bytes in the argument list. However, rather than 5 4-byte quantities as in the Default convention, there are 3 4-byte quantities and 1 8-byte quantity (the double precision value of a). • C - The symbol name for the subroutine is constructed by pre-pending an underscore and converting to all lower case. Character strings are truncated to the first character in the string, which is passed by value as the first byte in a 4-byte word. The following is an example of the pseudo-code for the above call using C conventions: call _work (%val(‘E’), %val (a), %addr(b), %val(n)) • UNIX - The symbol name for the subroutine is constructed by pre-pending an underscore, converting to all lower case, and appending an underscore. Byte counts for character strings appear in sequence following the last argument in the argument list. The following is an example of the pseudo-code for the above call using UNIX conventions: call _work_ (%addr(‘ERR’), %addr (a), %addr(b), %addr(n), 3) 10.14.3 Using the Default Calling Convention Using the Default calling convention is straightforward. Use the default convention if no directives are inserted to modify calling conventions and if the -Munix compilation flag is not used. See the previous section for a complete description of the Default convention. 10.14.4 Using the STDCALL Calling Convention Using the STDCALL calling convention requires the insertion of a compiler directive into the declarations section of any Fortran program unit which calls the STDCALL program unit. This directive has no effect when the -Munix compilation flag is used, meaning you cannot mix UNIXstyle argument passing and STDCALL calling conventions within the same file. Syntax for the directive is as follows: !MS$ATTRIBUTES STDCALL :: work Where work is the name of the subroutine to be called using STDCALL conventions. More than one subroutine may be listed, separated by commas. See Section 10.14.2 Symbol Name Construction and Calling Example for a complete description of the implementation of STDCALL. Inter-language Calling 239 Note: The directive prefix !DEC$ is also supported, but requires a space between the prefix and the directive keyword ATTRIBUTES. The ! must begin the prefix when compiling using Fortran 90 freeform format. The characters C or * can be used in place of ! in either form of the prefix when compiling used fixed-form (F77-style) format. The directives are completely case insensitive. 10.14.5 Using the C Calling Convention Using the C calling convention requires the insertion of a compiler directive into the declarations section of any Fortran program unit which calls the C program unit. This directive has no effect when the -Munix compilation flag is used, meaning you cannot mix UNIX-style argument passing and C calling conventions within the same file. Syntax for the directive is as follows: !MS$ATTRIBUTES C :: work Where work is the name of the subroutine to be called using C conventions. More than one subroutine may be listed, separated by commas. See above for a complete description of the implementation of the C calling convention. Note: The directive prefix !DEC$ is also supported, but requires a space between the prefix and the directive keyword ATTRIBUTES. The ! must begin the prefix when compiling using Fortran 90 freeform format. The characters C or * can be used in place of ! in either form of the prefix when compiling used fixed-form (F77-style) format. The directives are completely case insensitive. 10.14.6 Using the UNIX Calling Convention Using the UNIX calling convention is straightforward. Any program unit compiled using -Munix compilation flag will use the UNIX convention. C++ Name Mangling 241 Chapter 11 C++ Name Mangling Name mangling transforms the names of entities so that the names include information on aspects of the entity’s type and fully qualified name. This is necessary since the intermediate language into which a program is translated contains fewer and simpler name spaces than there are in the C++ language. Specifically: • Overloaded function names are not allowed in the intermediate language. • Classes have their own scopes in C++, but not in the generated intermediate language. For example, an entity x from inside a class must not conflict with an entity x from the file scope. • External names in the object code form a completely flat name space. The names of entities with external linkage must be projected onto that name space so that they do not conflict with one another. A function f from a class A, for example, must not have the same external name as a function f from class B. • Some names are not names in the conventional sense of the word, they're not strings of alphanumeric characters, for example operator=. We can see that there are two problems here: 1. Generating external names that will not clash. 2. Generating alphanumeric names for entities with strange names in C++. Name mangling solves these problems by generating external names that will not clash, and alphanumeric names for entities with strange names in C++. It also solves the problem of generating hidden names for some behind-the-scenes language support in such a way that they will match up across separate compilations. You will see mangled names if you view files that are translated by PGC++, and you do not use tools that demangle the C++ names. Intermediate files that use mangled names include the assembly and object files created by the pgCC command and the C-like file that can be viewed as output from pgCC using the +i command-line option The name mangling algorithm for the PGC++ compiler is the same as that for cfront, and also matches the description in Section 7.2, Function Name Encoding, of The Annotated C++ 242 Chapter 11 Reference Manual (except for some minor details). Refer to the ARM for a complete description of name mangling. 11.1 Types of Mangling The following entity names are mangled: • Function names including non-member function names are mangled, to deal with overloading. Names of functions with extern "C" linkage are not mangled. • Mangled function names have the function name followed by __ followed by F followed by the mangled description of the types of the parameters of the function. If the function is a member function, the mangled form of the class name precedes the F. If the member function is static, an S also precedes the F. int f(float); // f__Ff class A int f(float); // f__1AFf static int g(float); // g__1ASFf ; • Special and operator function names, like constructors and operator=(). The encoding is similar to that for normal functions, but a coded name is used instead of the routine name: class A int operator+(float); // __pl__1Aff A(float); // __ct__1Aff ; int operator+(A, float); // __pl__F1Af • Static data member names. The mangled form is the member name followed by __ followed by the mangled form of the class name: class A static int i; // i__1A ; • Names of variables generated for virtual function tables. These have names like vtblmangled-class-name or vtblmangled-base-class-namemangled-classname.C++ Name Mangling 243 • Names of variables generated to contain runtime type information. These have names like Ttype-encoding and TIDtype-encoding. 11.2 Mangling Summary This section lists some of the C++ entities that are mangled and provides some details on the mangling algorithm. For more details, refer to The Annotated C++ Reference Manual. 11.2.1 Type Name Mangling Using PGC++, each type has a corresponding mangled encoding. For example, a class type is represented as the class name preceded by the number of characters in the class name, as in 5abcde for abcde. Simple types are encoded as lower-case letters, as in i for int or f for float. Type modifiers and declarators are encoded as upper-case letters preceding the types they modify, as in U for unsigned or P for pointer. 11.2.2 Nested Class Name Mangling Nested class types are encoded as a Q followed by a digit indicating the depth of nesting, followed by a _, followed by the mangled-form names of the class types in the fully-qualified name of the class, from outermost to innermost: class A class B // Q2_1A1B ; ; 11.2.3 Local Class Name Mangling The name of the nested class itself is mangled to the form described above with a prefix __, which serves to make the class name distinct from all user names. Local class names are encoded as L followed by a number (which has no special meaning; it’s just an identifying number assigned to the class) followed by __ followed by the mangled name of the class (this is not in the ARM, and cfront encodes local class names slightly differently): void f() class A // L1__1A} ; ; This form is used when encoding the local class name as a type. It’s not necessary to mangle the name of the local class itself unless it's also a nested class. 244 Chapter 11 11.2.4 Template Class Name Mangling Template classes have mangled names that encode the arguments of the template: template class abc ; abc x; abc__pt__3_ii This describes two template arguments of type int with the total length of template argument list string, including the underscore, and a fixed string, indicates parameterized type as well, the name of the class template. Run-time Environment 245 Appendix A Run-time Environment This appendix describes the programming model supported for compiler code generation, including register conventions and calling conventions for x86 and x64 processor-based systems. Section A1 addresses these conventions for processors running linux86 or Win32 operating systems, section A2 for processors running linux86-64 operating systems, and section A3 for processors running Win64 operating systems. A1 Linux86 and Win32 Programming Model This section defines compiler and assembly language conventions for the use of certain aspects of an x86 processor running a linux86 or Win32 operating system. These standards must be followed to guarantee that compilers, application programs, and operating systems written by different people and organizations will work together. The conventions supported by the PGCC ANSI C compiler implement the application binary interface (ABI) as defined in the System V Application Binary Interface: Intel Processor Supplement and the System V Application Binary Interface, listed in the “Related Publications” section in the Preface. A1.1 Function Calling Sequence This section describes the standard function calling sequence, including the stack frame, register usage, and parameter passing. Register Usage Conventions Table A-1 defines the standard for register allocation. The 32-bit x86 Architecture provides a number of registers. All the integer registers and all the floating-point registers are global to all procedures in a running program. Table A-1: Register Allocation Type Name Purpose General %eax integer return value %edx dividend register (for divide operations) %ecx count register (shift and string operations) %ebx local register variable 246 Appendix A Type Name Purpose %ebp optional stack frame pointer %esi local register variable %edi local register variable %esp stack pointer Floating-point %st(0) floating-point stack top, return value %st(1) floating-point next to stack top %st(...) %st(7) floating-point stack bottom In addition to the registers, each function has a frame on the run-time stack. This stack grows downward from high addresses. Table A-2 shows the stack frame organization. Table A-2: Standard Stack Frame Position Contents Frame 4n+8 (%ebp) argument word n previous 8 (%ebp) argument word 0 4 (%ebp) return address 0 (%ebp) caller's %ebp current -4 (%ebp) n bytes of local -n (%ebp) variables and temps Several key points concerning the stack frame: • The stack is kept double word aligned • Argument words are pushed onto the stack in reverse order (i.e., the rightmost argument in C call syntax has the highest address) A dummy word may be pushed ahead of the rightmost argument in order to preserve doubleword alignment. All incoming arguments appear on the stack, residing in the stack frame of the caller. • An argument’s size is increased, if necessary, to make it a multiple of words. This may require tail padding, depending on the size of the argument. All registers on an x86 system are global and thus visible to both a calling and a called function. Registers %ebp, %ebx, %edi, %esi, and %esp are non-volatile across function calls. Therefore, Run-time Environment 247 a function must preserve these registers’ values for its caller. Remaining registers are volatile (scratch). If a calling function wants to preserve such a register value across a function call, it must save its value explicitly. Some registers have assigned roles in the standard calling sequence: %esp The stack pointer holds the limit of the current stack frame, which is the address of the stack’s bottom-most, valid word. At all times, the stack pointer should point to a word-aligned area. %ebp The frame pointer holds a base address for the current stack frame. Consequently, a function has registers pointing to both ends of its frame. Incoming arguments reside in the previous frame, referenced as positive offsets from %ebp, while local variables reside in the current frame, referenced as negative offsets from %ebp. A function must preserve this register value for its caller. %eax Integral and pointer return values appear in %eax. A function that returns a structure or union value places the address of the result in %eax. Otherwise, this is a scratch register. %esi, %edi These local registers have no specified role in the standard calling sequence. Functions must preserve their values for the caller. %ecx, %edx Scratch registers have no specified role in the standard calling sequence. Functions do not have to preserve their values for the caller. %st(0) Floating-point return values appear on the top of the floating point register stack; there is no difference in the representation of single or doubleprecision values in floating point registers. If the function does not return a floating point value, then the stack must be empty. %st(1) - %st(7) Floating point scratch registers have no specified role in the standard calling sequence. These registers must be empty before entry and upon exit from a function. EFLAGS The flags register contains the system flags, such as the direction flag and the carry flag. The direction flag must be set to the “forward” (i.e., zero) direction before entry and upon exit from a function. Other user flags have no specified role in the standard calling sequence and are not reserved. 248 Appendix A Floating Point Control Word The control word contains the floating-point flags, such as the rounding mode and exception masking. This register is initialized at process initialization time and its value must be preserved. Signals can interrupt processes. Functions called during signal handling have no unusual restriction on their use of registers. Moreover, if a signal handling function returns, the process resumes its original execution path with registers restored to their original values. Thus, programs and compilers may freely use all registers without danger of signal handlers changing their values. A1.2 Function Return Values Functions Returning Scalars or No Value • A function that returns an integral or pointer value places its result in register %eax. • A function that returns a long long integer value places its result in the registers %edx and %eax. The most significant word is placed in %edx and the least significant word is placed in %eax. • A floating-point return value appears on the top of the floating point stack. The caller must then remove the value from the floating point stack, even if it does not use the value. Failure of either side to meet its obligations leads to undefined program behavior. The standard calling sequence does not include any method to detect such failures nor to detect return value type mismatches. Therefore, the user must declare all functions properly. There is no difference in the representation of single-, double- or extendedprecision values in floating-point registers. • Functions that return no value (also called procedures or void functions) put no particular value in any register. • A call instruction pushes the address of the next instruction (the return address) onto the stack. The return instruction pops the address off the stack and effectively continues execution at the next instruction after the call instruction. A function that returns a scalar or no value must preserve the caller's registers as described above. Additionally, the called function must remove the return address from the stack, leaving the stack pointer (%esp) with the value it had before the call instruction was executed. Functions Returning Structures or Unions If a function returns a structure or union, then the caller provides space for the return value and places its address on the stack as argument word zero. In effect, this address becomes a hidden first argument. Run-time Environment 249 A function that returns a structure or union also sets %eax to the value of the original address of the caller's area before it returns. Thus, when the caller receives control again, the address of the returned object resides in register %eax and can be used to access the object. Both the calling and the called functions must cooperate to pass the return value successfully: • The calling function must supply space for the return value and pass its address in the stack frame; • The called function must use the address from the frame and copy the return value to the object so supplied; • The called function must remove this address from the stack before returning. Failure of either side to meet its obligation leads to undefined program behavior. The standard function calling sequence does not include any method to detect such failures nor to detect structure and union type mismatches. Therefore, you must declare the function properly. Table A-3 illustrates the stack contents when the function receives control, after the call instruction, and when the calling function again receives control, after the ret instruction. Table A-3: Stack Contents for Functions Returning struct/union Position After Call After Return Position 4n+8 (%esp) argument word n argument word n 4n-4 (%esp) 8 (%esp) argument word 1 argument word 1 0 (%esp) 4 (%esp) value address undefined 0 (%esp) return address The following sections of this appendix describe where arguments appear on the stack. The examples are written as if the function prologue described above had been used. 250 Appendix A A1.3 Argument Passing Integral and Pointer Arguments As mentioned, a function receives all its arguments through the stack; the last argument is pushed first. In the standard calling sequence, the first argument is at offset 8(%ebp), the second argument is at offset 12(%ebp), etc., as previously shown in Table A-3. Functions pass all integer-valued arguments as words, expanding or padding signed or unsigned bytes and halfwords as needed. Table A-4: Integral and Pointer Arguments Call Argument Stack Address g(1, 2, 3, (void *)0); 1 8 (%ebp) 2 12 (%ebp) 3 16 (%ebp) (void *) 0 20 (%ebp) Floating-Point Arguments The stack also holds floating-point arguments: single-precision values use one word and doubleprecision use two. The example below uses only double-precision arguments. Table A-5: Floating-point Arguments Call Argument Stack Address h(1.414, 1, 2.998e10); word 0, 1.414 8 (%ebp) word 1, 1.414 12 (%ebp) 1 16 (%ebp) word 0 2.998e10 20 (%ebp) word 1, 2.998e10 24 (%ebp) Run-time Environment 251 Structure and Union Arguments Structures and unions can have byte, halfword, or word alignment, depending on the constituents. An argument’s size is increased, if necessary, to make it a multiple of words. This may require tail padding, depending on the size of the argument. Structure and union arguments are pushed onto the stack in the same manner as integral arguments, described above. This provides call-by-value semantics, letting the called function modify its arguments without affecting the calling function’s object. In the example below, the argument, s, is a structure consisting of more than 2 words. Table A-6: Structure and Union Arguments Call Argument Stack Address i(1,s); 1 8 (%ebp) word 0, s 12 (%ebp) word 1, s 16 (%ebp) ... ... Implementing a Stack In general, compilers and programmers must maintain a software stack. Register %esp is the stack pointer. Register %esp is set by the operating system for the application when the program is started. The stack must be a grow-down stack. A separate frame pointer enables calls to routines that change the stack pointer to allocate space on the stack at run-time (e.g. alloca). Some languages can also return values from a routine allocated on stack space below the original top-of-stack pointer. Such a routine prevents the calling function from using %esp-relative addressing to get at values on the stack. If the compiler does not call routines that leave %esp in an altered state when they return, a frame pointer is not needed and is not used if the compiler option –Mnoframe is specified. Although not required, the stack should be kept aligned on 8-byte boundaries so that 8-byte locals are favorably aligned with respect to performance. PGI's compilers allocate stack space for each routine in multiples of 8 bytes. Variable Length Parameter Lists Parameter passing in registers can handle a variable number of parameters. The C language uses a special method to access variable-count parameters. The stdarg.h and varargs.h files define several functions to access these parameters. A C routine with variable parameters must use the va_start macro to set up a data structure before the parameters can be used. The va_arg macro must be used to access the successive parameters. 252 Appendix A C Parameter Conversion In C, for a called prototyped function, the parameter type in the called function must match the argument type in the calling function. If the called function is not prototyped, the calling convention uses the types of the arguments but promotes char or short to int, and unsigned char or unsigned short to unsigned int and promotes float to double, unless you use the --Msingle option. For more information on the –Msingle option, refer to Chapter 3. If the called function is prototyped, the unused bits of a register containing a char or short parameter are undefined and the called function must extend the sign of the unused bits when needed. Calling Assembly Language Programs /* File: testmain.c */ main(){ long l_para1 = 0x3f800000; float f_para2 = 1.0; double d_para3 = 0.5; float f_return; extern float sum_3 (long para1, float para2, double para3); f_return = sum_3(l_para1, f_para2, d_para3); printf("Parameter one, type long = %08x\n", l_para1); printf("Parameter two, type float = %f\n", f_para2); printf("Parameter three, type double = %g\n", d_para3); printf("The sum after conversion = %f\n", f_return); } Run-time Environment 253 # File: sum_3.s # Computes ( para1 + para2 ) + para3 .text .align 4 .long .EN1-sum_3+0xc8000000 .align 16 .globl sum_3 sum_3: pushl %ebp movl %esp,%ebp subl $8,%esp ..EN1: fildl 8(%ebp) fadds 12(%ebp) faddl 16(%ebp) fstps -4(%ebp) flds -4(%ebp) leave ret .type sum_3,@function .size sum_3,.-sum_3 Example A-1: C Program Calling an Assembly-language Routine A2 Linux86-64 Programming Model This section defines compiler and assembly language conventions for the use of certain aspects of an x64 processor running a linux86-64 operating system. These standards must be followed to guarantee that compilers, application programs, and operating systems written by different people and organizations will work together. The conventions supported by the PGCC ANSI C compiler implement the application binary interface (ABI) as defined in the System V Application Binary Interface: AMD64 Architecture Processor Supplement and the System V Application Binary Interface, listed in the “Related Publications” section in the Preface. A2.1 Function Calling Sequence This section describes the standard function calling sequence, including the stack frame, register usage, and parameter passing. 254 Appendix A Register Usage Conventions Table A-7 defines the standard for register allocation. The x64 Architecture provides a variety of registers. All the general purpose registers, XMM registers, and x87 registers are global to all procedures in a running program. Table A-7: Register Allocation Type Name Purpose General %rax 1 st return register %rbx callee-saved; optional base pointer %rcx pass 4 th argument to functions %rdx pass 3 rd argument to functions; 2 nd return register %rsp stack pointer %rbp callee-saved; optional stack frame pointer %rsi pass 2 nd argument to functions %rdi pass 1 st argument to functions %r8 pass 5 th argument to functions %r9 pass 6 th argument to functions %r10 temporary register; pass a function’s static chain pointer %r11 temporary register %r12-r15 callee-saved registers XMM %xmm0-%xmm1 pass and return floating point arguments %xmm2-%xmm7 pass floating point arguments %xmm8-%xmm15 temporary registers x87 %st(0) temporary register; return long double arguments %st(1) temporary register; return long double arguments %st(2) - %st(7) temporary registers In addition to the registers, each function has a frame on the run-time stack. This stack grows downward from high addresses. Table A-8 shows the stack frame organization. Run-time Environment 255 Table A-8: Standard Stack Frame Position Contents Frame 8n+16 (%rbp) argument eightbyte n previous . . . 16 (%rbp) argument eightbyte 0 8 (%rbp) return address current 0 (%rbp) caller's %rbp current -8 (%rbp) unspecified . . . 0 (%rsp) variable size -128 (%rsp) red zone Key points concerning the stack frame: • The end of the input argument area is aligned on a 16-byte boundary. • The 128-byte area beyond the location of %rsp is called the red zone and can be used for temporary local data storage. This area is not modified by signal or interrupt handlers. • A call instruction pushes the address of the next instruction (the return address) onto the stack. The return instruction pops the address off the stack and effectively continues execution at the next instruction after the call instruction. A function must preserve nonvolatile registers (described below). Additionally, the called function must remove the return address from the stack, leaving the stack pointer (%rsp) with the value it had before the call instruction was executed. All registers on an x64 system are global and thus visible to both a calling and a called function. Registers %rbx, %rsp, %rbp, %r12, %r13, %r14, and %r15 are non-volatile across function calls. Therefore, a function must preserve these registers’ values for its caller. Remaining registers are volatile (scratch). If a calling function wants to preserve such a register value across a function call, it must save its value explicitly. Registers are used extensively in the standard calling sequence. The first six integer and pointer arguments are passed in these registers (listed in order): %rdi, %rsi, %rdx, %rcx, %r8, %r9. The first eight floating point arguments are passed in the first eight XMM registers: %xmm0, %xmm1, …, %xmm7. The registers %rax and %rdx are used to return integer and pointer values. The registers %xmm0 and %xmm1 are used to return floating point values. Additional registers with assigned roles in the standard calling sequence: 256 Appendix A %rsp The stack pointer holds the limit of the current stack frame, which is the address of the stack’s bottom-most, valid word. The stack must be 16-byte aligned. %rbp The frame pointer holds a base address for the current stack frame. Consequently, a function has registers pointing to both ends of its frame. Incoming arguments reside in the previous frame, referenced as positive offsets from %rbp, while local variables reside in the current frame, referenced as negative offsets from %rbp. A function must preserve this register value for its caller. RFLAGS The flags register contains the system flags, such as the direction flag and the carry flag. The direction flag must be set to the “forward” (i.e., zero) direction before entry and upon exit from a function. Other user flags have no specified role in the standard calling sequence and are not preserved. Floating Point Control Word The control word contains the floating-point flags, such as the rounding mode and exception masking. This register is initialized at process initialization time and its value must be preserved. Signals can interrupt processes. Functions called during signal handling have no unusual restriction on their use of registers. Moreover, if a signal handling function returns, the process resumes its original execution path with registers restored to their original values. Thus, programs and compilers may freely use all registers without danger of signal handlers changing their values. A2.2 Function Return Values Functions Returning Scalars or No Value • A function that returns an integral or pointer value places its result in the next available register of the sequence %rax, %rdx. • A function that returns a floating point value that fits in the XMM registers returns this value in the next available XMM register of the sequence %xmm0, %xmm1. • An X87 floating-point return value appears on the top of the floating point stack in %st(0) as an 80-bit X87 number. If this X87 return value is a complex number, the real part of the value is returned in %st(0) and the imaginary part in %st(1). • A function that returns a value in memory also returns the address of this memory in %rax. Run-time Environment 257 • Functions that return no value (also called procedures or void functions) put no particular value in any register. Functions Returning Structures or Unions A function can use either registers or memory to return a structure or union. The size and type of the structure or union determine how it is returned. If a structure or union is larger than 16 bytes, it is returned in memory allocated by the caller. To determine whether a 16-byte or smaller structure or union can be returned in one or more return registers, examine the first eight bytes of the structure or union. The type or types of the structure or union’s fields making up these eight bytes determine how these eight bytes will be returned. If the eight bytes contain at least one integral type, the eight bytes will be returned in %rax even if non-integral types are also present in the eight bytes. If the eight bytes only contain floating point types, these eight bytes will be returned in %xmm0. If the structure or union is larger than eight bytes but smaller than 17 bytes, examine the type or types of the fields making up the second eight bytes of the structure or union. If these eight bytes contain at least one integral type, these eight bytes will be returned in %rdx even if non-integral types are also present in the eight bytes. If the eight bytes only contain floating point types, these eight bytes will be returned in %xmm1. If a structure or union is returned in memory, the caller provides the space for the return value and passes its address to the function as a “hidden” first argument in %rdi. This address will also be returned in %rax. A2.3 Argument Passing Integral and Pointer Arguments Integral and pointer arguments are passed to a function using the next available register of the sequence %rdi, %rsi, %rdx, %rcx, %r8, %r9. After this list of registers has been exhausted, all remaining integral and pointer arguments are passed to the function via the stack. Floating-Point Arguments Float and double arguments are passed to a function using the next available XMM register taken in the order from %xmm0 to %xmm7. After this list of registers has been exhausted, all remaining float and double arguments are passed to the function via the stack. Structure and Union Arguments Structure and union arguments can be passed to a function in either registers or on the stack. The size and type of the structure or union determine how it is passed. If a structure or union is larger than 16 bytes, it is passed to the function in memory. 258 Appendix A To determine whether a 16-byte or smaller structure or union can be passed to a function in one or two registers, examine the first eight bytes of the structure or union. The type or types of the structure or union’s fields making up these eight bytes determine how these eight bytes will be passed. If the eight bytes contain at least one integral type, the eight bytes will be passed in the first available general purpose register of the sequence %rdi, %rsi, %rdx, %rcx, %r8, %r9 even if non-integral types are also present in the eight bytes. If the eight bytes only contain floating point types, these eight bytes will be passed in the first available XMM register of the sequence from %xmm0 to %xmm7. If the structure or union is larger than eight bytes but smaller than 17 bytes, examine the type or types of the fields making up the second eight bytes of the structure or union. If the eight bytes contain at least one integral type, the eight bytes will be passed in the next available general purpose register of the sequence %rdi, %rsi, %rdx, %rcx, %r8, %r9 even if non-integral types are also present in the eight bytes. If these eight bytes only contain floating point types, these eight bytes will be passed in the next available XMM register of the sequence from %xmm0 to %xmm7. If the first or second eight bytes of the structure or union cannot be passed in a register for some reason, the entire structure or union must be passed in memory. Passing Arguments on the Stack If there are arguments left after every argument register has been allocated, the remaining arguments are passed to the function on the stack. The unassigned arguments are pushed on the stack in reverse order, with the last argument pushed first. Table A-9 shows the register allocation and stack frame offsets for the function declaration and call shown in Example A-2. Both table and example are adapted from System V Application Binary Interface: AMD64 Architecture Processor Supplement. Run-time Environment 259 typedef struct { int a, b; double d; } structparm; structparm s; int e,f,g,h,i,j,k; float flt; double m,n; extern void func (int e, int f, structparm s, int g, int h, float flt, double m, double n, int i, int j, int k); func (e, f, s, g, h, flt, m, n, i, j, k); Example A-2: Parameter Passing Table A-9: Register Allocation for Example A-2 General Purpose Registers Floating Point Registers Stack Frame Offset %rdi: e %xmm0: s.d 0: j %rsi: f %xmm1: flt 8: k %rdx: s.a,s.b %xmm2: m %rcx: g %xmm3: n %r8: h %r9: i Implementing a Stack In general, compilers and programmers must maintain a software stack. The stack pointer, register %rsp, is set by the operating system for the application when the program is started. The stack must grow downwards from high addresses. A separate frame pointer enables calls to routines that change the stack pointer to allocate space on the stack at run-time (e.g. alloca). Some languages can also return values from a routine allocated on stack space below the original top-of-stack pointer. Such a routine prevents the calling function from using %rsp-relative addressing for values on the stack. If the compiler does not call routines that leave %rsp in an altered state when they return, a frame pointer is not needed and may not be used if the compiler option –Mnoframe is specified. The stack must be kept aligned on 16-byte boundaries. 260 Appendix A Variable Length Parameter Lists Parameter passing in registers can handle a variable number of parameters. The C language uses a special method to access variable-count parameters. The stdarg.h and varargs.h files define several functions to access these parameters. A C routine with variable parameters must use the va_start macro to set up a data structure before the parameters can be used. The va_arg macro must be used to access the successive parameters. For calls that use varargs or stdargs, the register %rax acts as a hidden argument whose value is the number of XMM registers used in the call. C Parameter Conversion In C, for a called prototyped function, the parameter type in the called function must match the argument type in the calling function. If the called function is not prototyped, the calling convention uses the types of the arguments but promotes char or short to int, and unsigned char or unsigned short to unsigned int and promotes float to double, unless you use the --Msingle option. For more information on the –Msingle option, refer to Chapter 3. Run-time Environment 261 Calling Assembly Language Programs /* File: testmain.c */ main() { long l_para1 = 0x3f800000; float f_para2 = 1.0; double d_para3 = 0.5; float f_return; extern float sum_3 (long para1, float para2, double para3); f_return = sum_3(l_para1, f_para2, d_para3); printf("Parameter one, type long = %08x\n", l_para1); printf("Parameter two, type float = %f\n", f_para2); printf("Parameter three, type double = %g\n", d_para3); printf("The sum after conversion = %f\n", f_return); } # File: sum_3.s # Computes ( para1 + para2 ) + para3 .text .align 16 .globl sum_3 sum_3: pushq %rbp movq %rsp, %rbp cvtsi2ssq %rdi, %xmm2 addss %xmm0, %xmm2 cvtss2sd %xmm2, %xmm2 addsd %xmm1, %xmm2 cvtsd2ss %xmm2, %xmm2 movaps %xmm2, %xmm0 popq %rbp ret .type sum_3,@function .size sum_3,.-sum_3 Example A-3: C Program Calling an Assembly-language Routine 262 Appendix A A2.4 Linux86-64 Fortran Supplement Sections A2.4.1 through A2.4.4 define the Fortran supplement to the ABI for x64 Linux. The register usage conventions set forth in that document remain the same for Fortran. A2.4.1 Fortran Fundamental Types Table A-10: Linux86-64 Fortran Fundamental Types Fortran Type Size (bytes) Alignment (bytes) INTEGER 4 4 INTEGER*1 1 1 INTEGER*2 2 2 INTEGER*4 4 4 INTEGER*8 8 8 LOGICAL 4 4 LOGICAL*1 1 1 LOGICAL*2 2 2 LOGICAL*4 4 4 LOGICAL*8 8 8 BYTE 1 1 CHARACTER*n n 1 REAL 4 4 REAL*4 4 4 REAL*8 8 8 DOUBLE PRECISION 8 8 COMPLEX 8 4 COMPLEX*8 8 4 COMPLEX*16 16 8 DOUBLE COMPLEX 16 8 A logical constant is one of: .TRUE. .FALSE. Run-time Environment 263 The logical constants .TRUE. and .FALSE. are defined to be the four-byte values -1 and 0 respectively. A logical expression is defined to be .TRUE. if its least significant bit is 1 and .FALSE. otherwise. Note that the value of a character is not automatically NULL-terminated. A2.4.2 Naming Conventions By default, all globally visible Fortran symbol names (subroutines, functions, common blocks) are converted to lower-case. In addition, an underscore is appended to Fortran global names to distinguish the Fortran name space from the C/C++ name space. A2.4.3 Argument Passing and Return Conventions Arguments are passed by reference (i.e. the address of the argument is passed, rather than the argument itself). In contrast, C/C++ arguments are passed by value. When passing an argument declared as Fortran type CHARACTER, an argument representing the length of the CHARACTER argument is also passed to the function. This length argument is a four-byte integer passed by value, and is passed at the end of the parameter list following the other formal arguments. A length argument is passed for each CHARACTER argument; the length arguments are passed in the same order as their respective CHARACTER arguments. A Fortran function, returning a value of type CHARACTER, adds two arguments to the beginning of its argument list. The first additional argument is the address of the area created by the caller for the return value; the second additional argument is the length of the return value. If a Fortran function is declared to return a character value of constant length, for example CHARACTER*4 FUNCTION CHF(), the second extra parameter representing the length of the return value must still be supplied. A Fortran complex function returns its value in memory. The caller provides space for the return value and passes the address of this storage as if it were the first argument to the function. Alternate return specifiers of a Fortran function are not passed as arguments by the caller. The alternate return function passes the appropriate return value back to the caller in %rax. The handling of the following Fortran 90 features is implementation-defined: internal procedures, pointer arguments, assumed-shape arguments, functions returning arrays, and functions returning derived types. 264 Appendix A A2.4.4 Inter-language Calling Inter-language calling between Fortran and C/C++ is possible if function/subroutine parameters and return values match types. If a C/C++ function returns a value, call it from Fortran as a function, otherwise, call it as a subroutine. If a Fortran function has type CHARACTER or COMPLEX, call it from C/C++ as a void function. If a Fortran subroutine has alternate returns, call it from C/C++ as a function returning int; the value of such a subroutine is the value of the integer expression specified in the alternate RETURN statement. If a Fortran subroutine does not contain alternate returns, call it from C/C++ as a void function. The following table provides the C/C++ data type corresponding to each Fortran data type. Table A-11: Fortran and C/C++ Data Type Compatibility Fortran Type C/C++ Type Size (bytes) CHARACTER*n x char x[n] n REAL x float x 4 REAL*4 x float x 4 REAL*8 x double x 8 DOUBLE PRECISION x double x 8 INTEGER x int x 4 INTEGER*1 x signed char x 1 INTEGER*2 x short x 2 INTEGER*4 x int x 4 INTEGER*8 x long x, or long long x 8 8 LOGICAL x int x 4 LOGICAL*1 x char x 1 LOGICAL*2 x short x 2 LOGICAL*4 x int x 4 LOGICAL*8 x long x, or long long x 8 8 Run-time Environment 265 Table A-12: Fortran and C/C++ Representation of the COMPLEX Type Fortran Type C/C++ Type Size (bytes) COMPLEX x struct {float r, I;} x; 8 COMPLEX*8 x struct {float r, I;} x; 8 COMPLEX*16 x struct {double dr,di;} x; 16 DOUBLE COMPLEX x struct {double dr,di;} x; 16 Arrays C/C++ arrays and Fortran arrays use different default initial array index values. By default, C/C++ arrays start at 0 and Fortran arrays start at 1. A Fortran array can be declared to start at zero. Another difference between Fortran and C/C++ arrays is the storage method used. Fortran uses column-major order and C/C++ use row-major order. For one-dimensional arrays, this poses no problems. For two-dimensional arrays, where there are an equal number of rows and columns, row and column indexes can simply be reversed. Inter-language function mixing is not recommended for arrays other than single dimensional arrays and square two-dimensional arrays. Structures, Unions, Maps, and Derived Types Fields within Fortran structures and derived types, and multiple map declarations within a Fortran union, conform to the same alignment requirements used by C structures. Common Blocks A named Fortran common block can be represented in C/C++ by a structure whose members correspond to the members of the common block. The name of the structure in C/C++ must have the added underscore. For example, the Fortran common block: INTEGER I, J COMPLEX C DOUBLE COMPLEX CD DOUBLE PRECISION D COMMON /COM/ i, j, c, cd, d 266 Appendix A is represented in C with the following equivalent: extern struct { int i; int j; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; and in C++ with the following equivalent: extern "C" struct { int i; int j; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; Note that the compiler-provided name of the BLANK COMMON block is implementation specific. Calling Fortran COMPLEX and CHARACTER functions from C/C++ is not as straightforward as calling other types of Fortran functions. Additional arguments must be passed to the Fortran function by the C/C++ caller. A Fortran COMPLEX function returns its value in memory; the first argument passed to the function must contain the address of the storage for this value. A Fortran CHARACTER function adds two arguments to the beginning of its argument list. The following example of calling a Fortran CHARACTER function from C/C++ illustrates these caller-provided extra parameters: CHARACTER*(*) FUNCTION CHF(C1, I) CHARACTER*(*) C1 INTEGER I END extern void chf_(); char tmp[10]; char c1[9]; int i; chf_(tmp, 10, c1, &i, 9); The extra parameters tmp and 10 are supplied for the return value, while 9 is supplied as the length of c1. Refer to Section 2.8, Argument Passing and Return Conventions, for additional information. Run-time Environment 267 A3 Win64 Programming Model This section defines compiler and assembly language conventions for the use of certain aspects of an x64 processor running a Win64 operating system. These standards must be followed to guarantee that compilers, application programs, and operating systems written by different people and organizations will work together. The conventions supported by the PGCC ANSI C compiler implement the application binary interface (ABI) as defined in the AMD64 Software Conventions document. A3.1 Function Calling Sequence This section describes the standard function calling sequence, including the stack frame, register usage, and parameter passing. Register Usage Conventions Table A-10 defines the standard for register allocation. The 64-bit AMD64 Architecture (AMD64) provides a number of registers. All the general purpose registers, XMM registers, and x87 registers are global to all procedures in a running program. Table A-10: Register Allocation Type Name Purpose General %rax return value register %rbx callee-saved %rcx pass 1 st argument to functions %rdx pass 2 nd argument to functions %rsp stack pointer %rbp callee-saved; optional stack frame pointer %rsi callee-saved %rdi callee-saved %r8 pass 3 rd argument to functions %r9 pass 4 th argument to functions %r10-%r11 temporary registers; used in syscall/sysret instructions %r12-r15 callee-saved registers XMM %xmm0 pass 1 st floating point argument; return value register %xmm1 pass 2 nd floating point argument %xmm2 pass 3 rd floating point argument 268 Appendix A Type Name Purpose %xmm3 pass 4 th floating point argument %xmm4-%xmm5 temporary registers %xmm6-%xmm15 callee-saved registers In addition to the registers, each function has a frame on the run-time stack. This stack grows downward from high addresses. Table A-11 shows the stack frame organization. Table A-11: Standard Stack Frame Position Contents Frame 8n-120 (%rbp) argument eightbyte n previous . . . -80 (%rbp) argument eightbyte 5 -88 (%rbp) %r9 home -96 (%rbp) %r8 home -104 (%rbp) %rdx home -112 (%rbp) %rcx home -120 (%rbp) return address current -128 (%rbp) caller's %rbp . . . 0 (%rsp) variable size Key points concerning the stack frame: • The parameter area at the bottom of the stack must contain enough space to hold all the parameters needed by any function call. Space must be set aside for the four register parameters to be “homed” to the stack even if there are less than four register parameters used in a given call. • Sixteen-byte alignment of the stack is required except within a function’s prolog and within leaf functions. All registers on an x64 system are global and thus visible to both a calling and a called function. Registers %rbx, %rsp, %rbp, %rsi, %rdi, %r12, %r13, %r14, and %r15 are non-volatile. Therefore, a called function must preserve these registers’ values for its caller. Remaining registers are scratch. If a calling function wants to preserve such a register value across a function call, it must save a value in its local stack frame. Run-time Environment 269 Registers are used in the standard calling sequence. The first four arguments are passed in registers. Integral and pointer arguments are passed in these general purpose registers (listed in order): %rcx, %rdx, %r8, %r9. Floating point arguments are passed in the first four XMM registers: %xmm0, %xmm1, %xmm2, %xmm3. Registers are assigned using the argument’s ordinal position in the argument list. For example, if a function’s first argument is an integral type and its second argument is a floating-point type, the first argument will be passed in the first general purpose register (%rcx) and the second argument will be passed in the second XMM register (%xmm1); the first XMM register and second general purpose register are ignored. Arguments after the first four are passed on the stack. Integral and pointer type return values are returned in %rax. Floating point return values are returned in %xmm0. Additional registers with assigned roles in the standard calling sequence: %rsp The stack pointer holds the limit of the current stack frame, which is the address of the stack’s bottom-most, valid word. The stack pointer should point to a 16-byte aligned area unless in the prolog or a leaf function. %rbp The frame pointer, if used, can provide a way to reference the previous frame on the stack. Details are implementation dependent. A function must preserve this register value for its caller. MXCSR The flags register MXCSR contains the system flags, such as the direction flag and the carry flag. The six status flags (MXCSR[0:5]) are volatile; the remainder of the register is nonvolatile. x87 Floating Point Control Word (FPCSR) The control word contains the floating-point flags, such as the rounding mode and exception masking. This register is initialized at process initialization time and its value must be preserved. Signals can interrupt processes. Functions called during signal handling have no unusual restriction on their use of registers. Moreover, if a signal handling function returns, the process resumes its original execution path with registers restored to their original values. Thus, programs and compilers may freely use all registers without danger of signal handlers changing their values. A3.2 Function Return Values Functions Returning Scalars or No Value • A function that returns an integral or pointer value that fits in 64 bits places its result in %rax. 270 Appendix A • A function that returns a floating point value that fits in the XMM registers returns this value in %xmm0. • A function that returns a value in memory via the stack places the address of this memory (passed to the function as a “hidden” first argument in %rcx) in %rax. • Functions that return no value (also called procedures or void functions) put no particular value in any register. • A call instruction pushes the address of the next instruction (the return address) onto the stack. The return instruction pops the address off the stack and effectively continues execution at the next instruction after the call instruction. A function that returns a scalar or no value must preserve the caller's registers as described above. Additionally, the called function must remove the return address from the stack, leaving the stack pointer (%rsp) with the value it had before the call instruction was executed. Functions Returning Structures or Unions A function can use either registers or the stack to return a structure or union. The size and type of the structure or union determine how it is returned. A structure or union is returned in memory if it is larger than 8 bytes or if its size is 3, 5, 6, or 7 bytes. A structure or union is returned in %rax if its size is 1, 2, 4, or 8 bytes. If a structure or union is to be returned in memory, the caller provides space for the return value and passes its address to the function as a “hidden” first argument in %rcx. This address will also be returned in %rax. A3.3 Argument Passing Integral and Pointer Arguments Integral and pointer arguments are passed to a function using the next available register of the sequence %rcx, %rdx, %r8, %r9. After this list of registers has been exhausted, all remaining integral and pointer arguments are passed to the function via the stack. Floating-Point Arguments Float and double arguments are passed to a function using the next available XMM register of the sequence %xmm0, %xmm1, %xmm2, %xmm3. After this list of registers has been exhausted, all remaining XMM floating-point arguments are passed to the function via the stack. Array, Structure, and Union Arguments Arrays and strings are passed to functions using a pointer to caller-allocated memory. Run-time Environment 271 Structure and union arguments of size 1, 2, 4, or 8 bytes will be passed as if they were integers of the same size. Structures and unions of other sizes will be passed as a pointer to a temporary, allocated by the caller, and whose value contains the value of the argument. The caller-allocated temporary memory used for arguments of aggregate type must be 16-byte aligned. Passing Arguments on the Stack Registers are assigned using the argument’s ordinal position in the argument list. For example, if a function’s first argument is an integral type and its second argument is a floating-point type, the first argument will be passed in the first general purpose register (%rcx) and the second argument will be passed in the second XMM register (%xmm1); the first XMM register and second general purpose register are ignored. Arguments after the first four are passed on the stack; they are pushed on the stack in reverse order, with the last argument pushed first. Table A-12 shows the register allocation and stack frame offsets for the function declaration and call shown in Example A-4. typedef struct { int i; float f; } struct1; int i; float f; double d; long l; long long ll; struct1 s1; extern void func (int i, float f, struct1 s1, double d, long long ll, long l); func (i, f, s1, d, ll, l); Example A-4: Parameter Passing Table A-12: Register Allocation for Example A-4 General Purpose Registers Floating Point Registers Stack Frame Offset %rcx: i %xmm0: 32: ll %rdx: %xmm1: f 40: l %r8: s1.i, s1.f %xmm2: 272 Appendix A General Purpose Registers Floating Point Registers Stack Frame Offset %r9: %xmm3: d Implementing a Stack In general, compilers and programmers must maintain a software stack. The stack pointer, register %rsp, is set by the operating system for the application when the program is started. The stack must grow downwards from high addresses. A separate frame pointer enables calls to routines that change the stack pointer to allocate space on the stack at run-time (e.g. alloca). Some languages can also return values from a routine allocated on stack space below the original top-of-stack pointer. Such a routine prevents the calling function from using %rsp-relative addressing to get at values on the stack. If the compiler does not call routines that leave %rsp in an altered state when they return, a frame pointer is not needed and is not used if the compiler option –Mnoframe is specified. The stack must always be 16-byte aligned except within the prolog and within leaf functions. Variable Length Parameter Lists Parameter passing in registers can handle a variable number of parameters. The C language uses a special method to access variable-count parameters. The stdarg.h and varargs.h files define several functions to access these parameters. A C routine with variable parameters must use the va_start macro to set up a data structure before the parameters can be used. The va_arg macro must be used to access the successive parameters. For unprototyped functions or functions that use varargs, floating-point arguments passed in registers must be passed in both an XMM register and its corresponding general purpose register. C Parameter Conversion In C, for a called prototyped function, the parameter type in the called function must match the argument type in the calling function. If the called function is not prototyped, the calling convention uses the types of the arguments but promotes char or short to int, and unsigned char or unsigned short to unsigned int and promotes float to double, unless you use the –Msingle option. For more information on the –Msingle option, refer to Chapter 3. If the called function is prototyped, the unused bits of a register containing a char or short parameter are undefined and the called function must extend the sign of the unused bits when needed. Calling Assembly Language Programs /* File: testmain.c */ main() { long l_para1 = 0x3f800000; Run-time Environment 273 float f_para2 = 1.0; double d_para3 = 0.5; float f_return; extern float sum_3 (long para1, float para2, double para3); f_return = sum_3(l_para1, f_para2, d_para3); printf("Parameter one, type long = %08x\n", l_para1); printf("Parameter two, type float = %f\n", f_para2); printf("Parameter three, type double = %g\n", d_para3); printf("The sum after conversion = %f\n", f_return); } # File: sum_3.s # Computes ( para1 + para2 ) + para3 .text .align 16 .globl sum_3 sum_3: pushq %rbp leaq 128(%rsp), %rbp cvtsi2ss %ecx, %xmm0 addss %xmm1, %xmm0 cvtss2sd %xmm0, %xmm0 addsd %xmm2, %xmm0 cvtsd2ss %xmm0, %xmm0 popq %rbp ret .type sum_3,@function .size sum_3,.-sum_3 Example A-5: C Program Calling an Assembly-language Routine A3.4 Win64 Fortran Supplement Sections A3.4.1 through A3.4.4 define the Fortran supplement to the AMD64 Software Conventions for Win64. The register usage conventions set forth in that document remain the same for Fortran. 274 Appendix A A3.4.1 Fortran Fundamental Types Table A-13: Win64 Fortran Fundamental Types Fortran Type Size (bytes) Alignment (bytes) INTEGER 4 4 INTEGER*1 1 1 INTEGER*2 2 2 INTEGER*4 4 4 INTEGER*8 8 8 LOGICAL 4 4 LOGICAL*1 1 1 LOGICAL*2 2 2 LOGICAL*4 4 4 LOGICAL*8 8 8 BYTE 1 1 CHARACTER*n n 1 REAL 4 4 REAL*4 4 4 REAL*8 8 8 DOUBLE PRECISION 8 8 COMPLEX 8 4 COMPLEX*8 8 4 COMPLEX*16 16 8 DOUBLE COMPLEX 16 8 A logical constant is one of: .TRUE. .FALSE. The logical constants .TRUE. and .FALSE. are defined to be the four-byte values -1 and 0 respectively. A logical expression is defined to be .TRUE. if its least significant bit is 1 and .FALSE. otherwise. Note that the value of a character is not automatically NULL-terminated. Run-time Environment 275 A3.4.2 Fortran Naming Conventions By default, all globally visible Fortran symbol names (subroutines, functions, common blocks) are converted to lower-case. In addition, an underscore is appended to Fortran global names to distinguish the Fortran name space from the C/C++ name space. A3.4.3 Fortran Argument Passing and Return Conventions Arguments are passed by reference (i.e. the address of the argument is passed, rather than the argument itself). In contrast, C/C++ arguments are passed by value. When passing an argument declared as Fortran type CHARACTER, an argument representing the length of the CHARACTER argument is also passed to the function. This length argument is a four-byte integer passed by value, and is passed at the end of the parameter list following the other formal arguments. A length argument is passed for each CHARACTER argument; the length arguments are passed in the same order as their respective CHARACTER arguments. A Fortran function, returning a value of type CHARACTER, adds two arguments to the beginning of its argument list. The first additional argument is the address of the area created by the caller for the return value; the second additional argument is the length of the return value. If a Fortran function is declared to return a character value of constant length, for example CHARACTER*4 FUNCTION CHF(), the second extra parameter representing the length of the return value must still be supplied. A Fortran complex function returns its value in memory. The caller provides space for the return value and passes the address of this storage as if it were the first argument to the function. Alternate return specifiers of a Fortran function are not passed as arguments by the caller. The alternate return function passes the appropriate return value back to the caller in %rax. The handling of the following Fortran 90 features is implementation-defined: internal procedures, pointer arguments, assumed-shape arguments, functions returning arrays, and functions returning derived types. A3.4.4 Interlanguage Calling Inter-language calling between Fortran and C/C++ is possible if function/subroutine parameters and return values match types. If a C/C++ function returns a value, call it from Fortran as a function, otherwise, call it as a subroutine. If a Fortran function has type CHARACTER or COMPLEX, call it from C/C++ as a void function. If a Fortran subroutine has alternate returns, call it from C/C++ as a function returning int; the value of such a subroutine is the value of the integer expression specified in the alternate RETURN statement. If a Fortran subroutine does not contain alternate returns, call it from C/C++ as a void function. The following table provides the C/C++ data type corresponding to each Fortran data type. 276 Appendix A Table A-14: Fortran and C/C++ Data Type Compatibility Fortran Type C/C++ Type Size (bytes) CHARACTER*n x char x[n] n REAL x float x 4 REAL*4 x float x 4 REAL*8 x double x 8 DOUBLE PRECISION x double x 8 INTEGER x int x 4 INTEGER*1 x signed char x 1 INTEGER*2 x short x 2 INTEGER*4 x int x 4 INTEGER*8 x long long x 8 LOGICAL x int x 4 LOGICAL*1 x char x 1 LOGICAL*2 x short x 2 LOGICAL*4 x int x 4 LOGICAL*8 x long long x 8 Table A-15: Fortran and C/C++ Representation of the COMPLEX Type Fortran Type C/C++ Type Size (bytes) COMPLEX x struct {float r, I;} x; 8 COMPLEX*8 x struct {float r, I;} x; 8 COMPLEX*16 x struct {double dr,di;} x; 16 DOUBLE COMPLEX x struct {double dr,di;} x; 16 Run-time Environment 277 Arrays C/C++ arrays and Fortran arrays use different default initial array index values. By default, C/C++ arrays start at 0 and Fortran arrays start at 1. A Fortran array can be declared to start at zero. Another difference between Fortran and C/C++ arrays is the storage method used. Fortran uses column-major order and C/C++ use row-major order. For one-dimensional arrays, this poses no problems. For two-dimensional arrays, where there are an equal number of rows and columns, row and column indexes can simply be reversed. Inter-language function mixing is not recommended for arrays other than single dimensional arrays and square two-dimensional arrays. Structures, Unions, Maps, and Derived Types Fields within Fortran structures and derived types, and multiple map declarations within a Fortran union, conform to the same alignment requirements used by C structures. Common Blocks A named Fortran common block can be represented in C/C++ by a structure whose members correspond to the members of the common block. The name of the structure in C/C++ must have the added underscore. For example, the Fortran common block: INTEGER I, J COMPLEX C DOUBLE COMPLEX CD DOUBLE PRECISION D COMMON /COM/ i, j, c, cd, d is represented in C with the following equivalent: extern struct { int i; int j; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; 278 Appendix A and in C++ with the following equivalent: extern "C" struct { int i; int j; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; Note that the compiler-provided name of the BLANK COMMON block is implementation specific. Calling Fortran COMPLEX and CHARACTER functions from C/C++ is not as straightforward as calling other types of Fortran functions. Additional arguments must be passed to the Fortran function by the C/C++ caller. A Fortran COMPLEX function returns its value in memory; the first argument passed to the function must contain the address of the storage for this value. A Fortran CHARACTER function adds two arguments to the beginning of its argument list. The following example of calling a Fortran CHARACTER function from C/C++ illustrates these caller-provided extra parameters: CHARACTER*(*) FUNCTION CHF(C1, I) CHARACTER*(*) C1 INTEGER I END extern void chf_(); char tmp[10]; char c1[9]; int i; chf_(tmp, 10, c1, &i, 9); The extra parameters tmp and 10 are supplied for the return value, while 9 is supplied as the length of c1. Refer to Section 10.6, Argument Passing and Return Values, for additional information. Messages 279 Appendix B Messages This appendix describes the various messages that the compiler produces. These messages include the sign-on message and diagnostic messages for remarks, warnings, and errors. The compiler always displays any error messages, along with the erroneous source line, on the screen. If you specify the –Mlist option, the compiler places any error messages in the listing file. You can also use the –v option to display more information about the compiler, assembler, and linker invocations and about the host system. For more information on the –Mlist and –v options, refer to Chapter 3, Command Line Options. B.1 Diagnostic Messages Diagnostic messages provide syntactic and semantic information about your source text. Syntactic information includes information such as syntax errors. Semantic includes information includes such as unreachable code. You can specify that the compiler displays error messages at a certain level with the –Minform option. The compiler messages refer to a severity level, a message number, and the line number where the error occurs. The compiler can also display internal error messages on standard errors. If your compilation produces any internal errors, contact you’re The Portland Group’s technical reporting service by sending e-mail to trs@pgroup.com. If you use the listing file option –Mlist, the compiler places diagnostic messages after the source lines in the listing file, in the following format: PGFTN-etype-enum-message (filename: line) Where: etype is a character signifying the severity level enum is the error number message is the error message 280 Appendix B filename is the source filename line is the line number where the compiler detected an error. B.2 Phase Invocation Messages You can display compiler, assembler, and linker phase invocations by using the –v command line option. For further information about this option, see Chapter 3, Command Line Options. B.3 Fortran Compiler Error Messages This section presents the error messages generated by the PGF77 and PGF95 compilers. The compilers display error messages in the program listing and on standard output; and can also display internal error messages on standard error. B.3.1 Message Format Each message is numbered. Each message also lists the line and column number where the error occurs. A dollar sign ($) in a message represents information that is specific to each occurrence of the message. B.3.2 Message List Error message severities: I - informative. W - warning. S - severe error. F - fatal error. V - variable. V000 Internal compiler error. $ $ This message indicates an error in the compiler, rather than a user error – although it may be possi- ble for a user error to cause an internal error. The severity may vary; if it is informative or warning, correct object code was probably generated, but it is not safe to rely on this. Regardless of the severity or cause, internal errors should be reported to trs@pgroup.com. Messages 281 F001 Source input file name not specified On the command line, source file name should be specified either before all the switches, or after them. F002 Unable to open source input file: $ Source file name misspelled, file not in current working directory, or file is read protected. F003 Unable to open listing file Probably, user does not have write permission for the current working directory. F004 $ $ Generic message for file errors. F005 Unable to open temporary file Compiler uses directory "/usr/tmp" or "/tmp" in which to create temporary files. If neither of these directories is available on the node on which the compiler is being used, this error will occur. S006 Input file empty Source input file does not contain any Fortran statements other than comments or compiler directives. F007 Subprogram too large to compile at this optimization level $ Internal compiler data structure overflow, working storage exhausted, or some other nonrecoverable problem related to the size of the subprogram. If this error occurs at opt 2, reducing the opt level to 1 may work around the problem. Moving the subprogram being compiled to its own source file may eliminate the problem. If this error occurs while compiling a subprogram of fewer than 2000 statements it should be reported to the compiler maintenance group as a possible compiler problem. F008 Error limit exceeded The compiler gives up because too many severe errors were issued; the error limit can be reset on the command line. F009 Unable to open assembly file Probably, user does not have write permission for the current working directory. 282 Appendix B F010 File write error occurred $ Probably, file system is full. S011 Unrecognized command line switch: $ Refer to PDS reference document for list of allowed compiler switches. S012 Value required for command line switch: $ Certain switches require an immediately following value, such as "-opt 2". S013 Unrecognized value specified for command line switch: $ S014 Ambiguous command line switch: $ Too short an abbreviation was used for one of the switches. W015 Hexadecimal or octal constant truncated to fit data type I016 Identifier, $, truncated to 31 chars An identifier may be at most 31 characters in length; characters after the 31st are ignored. S017 Unable to open include file: $ File is missing, read protected, or maximum include depth (10) exceeded. Remember that the file name should be enclosed in quotes. S018 Illegal label $ $ Used for label ’field’ errors or illegal values. E.g., in fixed source form, the label field (first five characters) of the indicated line contains a non-numeric character. S019 Illegally placed continuation line A continuation line does not follow an initial line, or more than 99 continuation lines were specified. S020 Unrecognized compiler directive Refer to user’s manual for list of allowed compiler directives. S021 Label field of continuation line is not blank The first five characters of a continuation line must be blank. Messages 283 S022 Unexpected end of file - missing END statement S023 Syntax error - unbalanced $ Unbalanced parentheses or brackets. W024 CHARACTER or Hollerith constant truncated to fit data type A character or hollerith constant was converted to a data type that was not large enough to contain all of the characters in the constant. This type conversion occurs when the constant is used in an arithmetic expression or is assigned to a non-character variable. The character or hollerith constant is truncated on the right, that is, if 4 characters are needed then the first 4 are used and the remaining characters are discarded. W025 Illegal character ($) - ignored The current line contains a character, possibly non-printing, which is not a legal Fortran character (characters inside of character or Hollerith constants cannot cause this error). As a general rule, all non-printing characters are treated as white space characters (blanks and tabs); no error message is generated when this occurs. If for some reason, a non-printing character is not treated as a white space character, its hex representation is printed in the form dd where each d is a hex digit. S026 Unmatched quote S027 Illegal integer constant: $ Integer constant is too large for 32 bit word. S028 Illegal real or double precision constant: $ S029 Illegal $ constant: $ Illegal hexadecimal, octal, or binary constant. A hexadecimal constant consists of digits 0..9 and letters A..F or a..f; any other character in a hexadecimal constant is illegal. An octal constant consists of digits 0..7; any other digit or character in an octal constant is illegal. A binary constant consists of digits 0 or 7; any other digit or character in a binary constant is illegal. S030 Explicit shape must be specified for $ S031 Illegal data type length specifier for $ The data type length specifier (e.g. 4 in INTEGER*4) is not a constant expression that is a member of the set of allowed values for this particular data type. 284 Appendix B W032 Data type length specifier not allowed for $ The data type length specifier (e.g. 4 in INTEGER*4) is not allowed in the given syntax (e.g. DIMENSION A(10)*4). S033 Illegal use of constant $ A constant was used in an illegal context, such as on the left side of an assignment statement or as the target of a data initialization statement. S034 Syntax error at or near $ I035 Predefined intrinsic $ loses intrinsic property An intrinsic name was used in a manner inconsistent with the language definition for that intrinsic. The compiler, based on the context, will treat the name as a variable or an external function. S036 Illegal implicit character range First character must alphabetically precede second. S037 Contradictory data type specified for $ The indicated identifier appears in more than one type specification statement and different data types are specified for it. S038 Symbol, $, has not been explicitly declared The indicated identifier must be declared in a type statement; this is required when the IMPLICIT NONE statement occurs in the subprogram. W039 Symbol, $, appears illegally in a SAVE statement $ An identifier appearing in a SAVE statement must be a local variable or array. S040 Illegal common variable $ Indicated identifier is a dummy variable, is already in a common block, or has previously been defined to be something other than a variable or array. W041 Illegal use of dummy argument $ This error can occur in several situations. It can occur if dummy arguments were specified on a PROGRAM statement. It can also occur if a dummy argument name occurs in a DATA, COMMON, SAVE, or EQUIVALENCE statement. A program statement must have an empty argument list. Messages 285 S042 $ is a duplicate dummy argument S043 Illegal attempt to redefine $ $ An attempt was made to define a symbol in a manner inconsistent with an earlier definition of the same symbol. This can happen for a number of reasons. The message attempts to indicate the situation that occurred. intrinsic - An attempt was made to redefine an intrinsic function. A symbol that represents an intrinsic function may be redefined if that symbol has not been previously verified to be an intrinsic function. For example, the intrinsic sin can be defined to be an integer array. If a symbol is verified to be an intrinsic function via the INTRINSIC statement or via an intrinsic function reference then it must be referred to as an intrinsic function for the remainder of the program unit. symbol - An attempt was made to redefine a symbol that was previously defined. An example of this is to declare a symbol to be a PARAMETER which was previously declared to be a subprogram argument. S044 Multiple declaration for symbol $ A redundant declaration of a symbol has occurred. For example, an attempt was made to declare a symbol as an ENTRY when that symbol was previously declared as an ENTRY. S045 Data type of entry point $ disagrees with function $ The current function has entry points with data types inconsistent with the data type of the current function. For example, the function returns type character and an entry point returns type complex. S046 Data type length specifier in wrong position The CHARACTER data type specifier has a different position for the length specifier from the other data types. Suppose, we want to declare arrays ARRAYA and ARRAYB to have 8 elements each having an element length of 4 bytes. The difference is that ARRAYA is character and ARRAYB is integer. The declarations would be CHARACTER ARRAYA(8)*4 and INTEGER ARRAYB*4(8). S047 More than seven dimensions specified for array S048 Illegal use of ’*’ in declaration of array $ An asterisk may be used only as the upper bound of the last dimension. S049 Illegal use of ’*’ in non-subroutine subprogram 286 Appendix B The alternate return specifier ’*’ is legal only in the subroutine statement. Programs, functions, and block data are not allowed to have alternate return specifiers. S050 Assumed size array, $, is not a dummy argument S051 Unrecognized built-in % function The allowable built-in functions are %VAL, %REF, %LOC, and %FILL. One was encountered that did not match one of these allowed forms. S052 Illegal argument to %VAL or %LOC S053 %REF or %VAL not legal in this context The built-in functions %REF and %VAL can only be used as actual parameters in procedure calls. W054 Implicit character $ used in a previous implicit statement An implicit character has been given an implied data type more than once. The implied data type for the implicit character is changed anyway. W055 Multiple implicit none statements The IMPLICIT NONE statement can occur only once in a subprogram. W056 Implicit type declaration The -dclchk switch and an implicit declaration following an IMPLICIT NONE statement will produce a warning message for IMPLICIT statements. S057 Illegal equivalence of dummy variable, $ Dummy arguments may not appear in EQUIVALENCE statements. S058 Equivalenced variables $ and $ not in same common block A common block variable must not be equivalenced with a variable in another common block. S059 Conflicting equivalence between $ and $ The indicated equivalence implies a storage layout inconsistent with other equivalences. S060 Illegal equivalence of structure variable, $ STRUCTURE and UNION variables may not appear in EQUIVALENCE statements. Messages 287 S061 Equivalence of $ and $ extends common block backwards W062 Equivalence forces $ to be unaligned EQUIVALENCE statements have defined an address for the variable which has an alignment not optimal for variables of its data type. This can occur when INTEGER and CHARACTER data are equivalenced, for instance. I063 Gap in common block $ before $ S064 Illegal use of $ in DATA statement implied DO loop The indicated variable is referenced where it is not an active implied DO index variable. S065 Repeat factor less than zero S066 Too few data constants in initialization statement S067 Too many data constants in initialization statement S068 Numeric initializer for CHARACTER $ out of range 0 through 255 A CHARACTER*1 variable or character array element can be initialized to an integer, octal, or hexadecimal constant if that constant is in the range 0 through 255. S069 Illegal implied DO expression The only operations allowed within an implied DO expression are integer +, -, *, and /. S070 Incorrect sequence of statements $ The statement order is incorrect. For instance, an IMPLICIT NONE statement must precede a specification statement which in turn must precede an executable statement. S071 Executable statements not allowed in block data S072 Assignment operation illegal to $ $ The destination of an assignment operation must be a variable, array reference, or vector reference. The assignment operation may be by way of an assignment statement, a data statement, or the index variable of an implied DO-loop. The compiler has determined that the identifier used as the destination, is not a storage location. The error message attempts to indicate the type of entity used. 288 Appendix B entry point - An assignment to an entry point that was not a function procedure was attempted. external procedure - An assignment to an external procedure or a Fortran intrinsic name was attempted. if the identifier is the name of an entry point that is not a function, an external procedure... S073 Intrinsic or predeclared, $, cannot be passed as an argument S074 Illegal number or type of arguments to $ $ The indicated symbol is an intrinsic or generic function, or a predeclared subroutine or function, requiring a certain number of arguments of a fixed data type. S075 Subscript, substring, or argument illegal in this context for $ This can happen if you try to doubly index an array such as ra(2)(3). This also applies to substring and function references. S076 Subscripts specified for non-array variable $ S077 Subscripts omitted from array $ S078 Wrong number of subscripts specified for $ S079 Keyword form of argument illegal in this context for $$ S080 Subscript for array $ is out of bounds S081 Illegal selector $ $ S082 Illegal substring expression for variable $ Substring expressions must be of type integer and if constant must be greater than zero. S083 Vector expression used where scalar expression required A vector expression was used in an illegal context. For example, iscalar = iarray, where a scalar is assigned the value of an array. Also, character and record references are not vectorizable. S084 Illegal use of symbol $ $ This message is used for many different errors. Messages 289 S085 Incorrect number of arguments to statement function $ S086 Dummy argument to statement function must be a variable S087 Non-constant expression where constant expression required S088 Recursive subroutine or function call of $ A function may not call itself. S089 Illegal use of symbol, $, with character length = * Symbols of type CHARACTER*(*) must be dummy variables and must not be used as statement function dummy parameters and statement function names. Also, a dummy variable of type CHARACTER*(*) cannot be used as a function. S090 Hollerith constant more than 4 characters In certain contexts, Hollerith constants may not be more than 4 characters long. S091 Constant expression of wrong data type S092 Illegal use of variable length character expression A character expression used as an actual argument, or in certain contexts within I/O statements, must not consist of a concatenation involving a passed length character variable. W093 Type conversion of expression performed An expression of some data type appears in a context which requires an expression of some other data type. The compiler generates code to convert the expression into the required type. S094 Variable $ is of wrong data type $ The indicated variable is used in a context which requires a variable of some other data type. S095 Expression has wrong data type An expression of some data type appears in a context which requires an expression of some other data type. S096 Illegal complex comparison The relations .LT., .GT., .GE., and .LE. are not allowed for complex values. 290 Appendix B S097 Statement label $ has been defined more than once More than one statement with the indicated statement number occurs in the subprogram. S098 Divide by zero S099 Illegal use of $ Aggregate record references may only appear in aggregate assignment statements, unformatted I/O statements, and as parameters to subprograms. They may not appear, for example, in expressions. Also, records with differing structure types may not be assigned to one another. S100 Expression cannot be promoted to a vector An expression was used that required a scalar quantity to be promoted to a vector illegally. For example, the assignment of a character constant string to a character array. Records, too, cannot be promoted to vectors. S101 Vector operation not allowed on $ Record and character typed entities may only be referenced as scalar quantities. S102 Arithmetic IF expression has wrong data type The parenthetical expression of an arithmetic if statement must be an integer, real, or double precision scalar expression. S103 Type conversion of subscript expression for $ The data type of a subscript expression must be integer. If it is not, it is converted. S104 Illegal control structure $ This message is issued for a number of errors involving IF-THEN statements and DO loops. If the line number specified is the last line (END statement) of the subprogram, the error is probably an unterminated DO loop or IF-THEN statement. S105 Unmatched ELSEIF, ELSE or ENDIF statement An ELSEIF, ELSE, or ENDIF statement cannot be matched with a preceding IF-THEN statement. S106 DO index variable must be a scalar variable The DO index variable cannot be an array name, a subscripted variable, a PARAMETER name, a function name, a structure name, etc. Messages 291 S107 Illegal assigned goto variable $ S108 Illegal variable, $, in NAMELIST group $ A NAMELIST group can only consist of arrays and scalars which are not dummy arguments and pointer-based variables. I109 Overflow in $ constant $, constant truncated at left A non-decimal (hexadecimal, octal, or binary) constant requiring more than 64-bits produces an overflow. The constant is truncated at left (e.g. ’1234567890abcdef1’x will be ’234567890abcdef1’x). I110 I111 Underflow of real or double precision constant I112 Overflow of real or double precision constant S113 Label $ is referenced but never defined S114 Cannot initialize $ W115 Assignment to DO variable $ in loop S116 Illegal use of pointer-based variable $ $ S117 Statement not allowed within a $ definition The statement may not appear in a STRUCTURE or derived type definition. S118 Statement not allowed in DO, IF, or WHERE block I119 Redundant specification for $ Data type of indicated symbol specified more than once. I120 Label $ is defined but never referenced I121 Operation requires logical or integer data types 292 Appendix B An operation in an expression was attempted on data having a data type incompatible with the operation. For example, a logical expression can consist of only logical elements of type integer or logical. Real data would be invalid. I122 Character string truncated Character string or Hollerith constant appearing in a DATA statement or PARAMETER statement has been truncated to fit the declared size of the corresponding identifier. W123 Hollerith length specification too big, reduced The length specifier field of a hollerith constant specified more characters than were present in the character field of the hollerith constant. The length specifier was reduced to agree with the number of characters present. S124 Relational expression mixes character with numeric data A relational expression is used to compare two arithmetic expressions or two character expressions. A character expression cannot be compared to an arithmetic expression. I125 Dummy procedure $ not declared EXTERNAL A dummy argument which is not declared in an EXTERNAL statement is used as the subprogram name in a CALL statement, or is called as a function, and is therefore assumed to be a dummy procedure. This message can result from a failure to declare a dummy array. I126 Name $ is not an intrinsic function I127 Optimization level for $ changed to opt 1 $ W128 Integer constant truncated to fit data type: $ An integer constant will be truncated when assigned to data types smaller than 32-bits, such as a BYTE. I129 Floating point overflow. Check constants and constant expressions I130 Floating point underflow. Check constants and constant expressions I131 Integer overflow. Check floating point expressions cast to integer Messages 293 I132 Floating pt. invalid oprnd. Check constants and constant expressions I133 Divide by 0.0. Check constants and constant expressions S134 Illegal attribute $ $ W135 Missing STRUCTURE name field A STRUCTURE name field is required on the outermost structure. W136 Field-namelist not allowed The field-namelist field of the STRUCTURE statement is disallowed on the outermost structure. W137 Field-namelist is required in nested structures W138 Multiply defined STRUCTURE member name $ A member name was used more than once within a structure. W139 Structure $ in RECORD statement not defined A RECORD statement contains a reference to a STRUCTURE that has not yet been defined. S140 Variable $ is not a RECORD S141 RECORD required on left of $ S142 $ is not a member of this RECORD S143 $ requires initializer W144 NEED ERROR MESSAGE $ $ This is used as a temporary message for compiler development. W145 %FILL only valid within STRUCTURE block The %FILL special name was used outside of a STRUCTURE multiline statement. It is only valid when used within a STRUCTURE multiline statement even though it is ignored. S146 Expression must be character type 294 Appendix B S147 Character expression not allowed in this context S148 Reference to $ required An aggregate reference to a record was expected during statement compilation but another data type was found instead. S149 Record where arithmetic value required An aggregate record reference was encountered when an arithmetic expression was expected. S150 Structure, Record, derived type, or member $ not allowed in this context A structure, record, or member reference was found in a context which is not supported. For example, the use of structures, records, or members within a data statement is disallowed. S151 Empty TYPE, STRUCTURE, UNION, or MAP TYPE - ENDTYPE, STRUCTURE - ENDSTRUCTURE, UNION - ENDUNION MAP - ENDMAP declaration contains no members. S152 All dimension specifiers must be ’:’ S153 Array objects are not conformable $ S154 DISTRIBUTE target, $, must be a processor S155 $ $ S156 Number of colons and triplets must be equal in ALIGN $ with $ S157 Illegal subscript use of ALIGN dummy $ - $ S158 Alternate return not specified in SUBROUTINE or ENTRY An alternate return can only be used if alternate return specifiers appeared in the SUBROUTINE or ENTRY statements. S159 Alternate return illegal in FUNCTION subprogram An alternate return cannot be used in a FUNCTION. S160 ENDSTRUCTURE, ENDUNION, or ENDMAP does not match top Messages 295 S161 Vector subscript must be rank-one array W162 Not equal test of loop control variable $ replaced with < or > test. S163 S164 Overlapping data initializations of $ An attempt was made to data initialize a variable or array element already initialized. S165 $ appeared more than once as a subprogram A subprogram name appeared more than once in the source file. The message is applicable only when an assembly file is the output of the compiler. S166 $ cannot be a common block and a subprogram A name appeared as a common block name and a subprogram name. The message is applicable only when an assembly file is the output of the compiler. I167 Inconsistent size of common block $ A common block occurs in more than one subprogram of a source file and its size is not identical. The maximum size is chosen. The message is applicable only when an assembly file is the output of the compiler. S168 Incompatible size of common block $ A common block occurs in more than one subprogram of a source file and is initialized in one subprogram. Its initialized size was found to be less than its size in the other subprogram(s). The message is applicable only when an assembly file is the output of the compiler. W169 Multiple data initializations of common block $ A common block is initialized in more than one subprogram of a source file. Only the first set of initializations apply. The message is applicable only when an assembly file is the output of the compiler. W170 F90 extension: $ $ Use of a nonstandard feature. A description of the feature is provided. W171 F90 extension: nonstandard statement type $ 296 Appendix B W172 F90 extension: numeric initialization of CHARACTER $ A CHARACTER*1 variable or array element was initialized with a numeric value. W173 F90 extension: nonstandard use of data type length specifier W174 F90 extension: type declaration contains data initialization W175 F90 extension: IMPLICIT range contains nonalpha characters W176 F90 extension: nonstandard operator $ W177 F90 extension: nonstandard use of keyword argument $ W178 W179 F90 extension: use of structure field reference $ W180 F90 extension: nonstandard form of constant W181 F90 extension: & alternate return W182 F90 extension: mixed non-character and character elements in COMMON $ W183 F90 extension: mixed non-character and character EQUIVALENCE ($,$) W184 Mixed type elements (numeric and/or character types) in COMMON $ W185 Mixed numeric and/or character type EQUIVALENCE ($,$) S186 Argument missing for formal argument $ S187 Too many arguments specified for $ S188 Argument number $ to $: type mismatch S189 Argument number $ to $: association of scalar actual argument to array dummy argument S190 Argument number $ to $: non-conformable arrays Messages 297 S191 Argument number $ to $ cannot be an assumed-size array S192 Argument number $ to $ must be a label W193 Argument number $ to $ does not match INTENT (OUT) W194 INTENT(IN) argument cannot be defined - $ S195 Statement may not appear in an INTERFACE block $ S196 Deferred-shape specifiers are required for $ S197 Invalid qualifier or qualifier value (/$) in OPTIONS statement An illegal qualifier was found or a value was specified for a qualifier which does not expect a value. In either case, the qualifier for which the error occurred is indicated in the error message. S198 $ $ in ALLOCATE/DEALLOCATE W199 Unaligned memory reference A memory reference occurred whose address does not meet its data alignment requirement. S200 Missing UNIT/FILE specifier S201 Illegal I/O specifier - $ S202 Repeated I/O specifier - $ S203 FORMAT statement has no label S204 $ $ Miscellaneous I/O error. S205 Illegal specification of scale factor The integer following + or - has been omitted, or P does not follow the integer value. S206 Repeat count is zero S207 Integer constant expected in edit descriptor 298 Appendix B S208 Period expected in edit descriptor S209 Illegal edit descriptor S210 Exponent width not used in the Ew.dEe or Gw.dEe edit descriptors S211 Internal I/O not allowed in this I/O statement S212 Illegal NAMELIST I/O Namelist I/O cannot be performed with internal, unformatted, formatted, and list-directed I/O. Also, I/O lists must not be present. S213 $ is not a NAMELIST group name S214 Input item is not a variable reference S215 Assumed sized array name cannot be used as an I/O item or specifier An assumed sized array was used as an item to be read or written or as an I/O specifier (i.e., FMT = array-name). In these contexts the size of the array must be known. S216 STRUCTURE/UNION cannot be used as an I/O item S217 ENCODE/DECODE buffer must be a variable, array, or array element S218 Statement labeled $ $ S219 S220 Redefining predefined macro $ S221 #elif after #else A preprocessor #elif directive was found after a #else directive; only #endif is allowed in this context. S222 #else after #else A preprocessor #else directive was found after a #else directive; only #endif is allowed in this context. Messages 299 S223 #if-directives too deeply nested Preprocessor #if directive nesting exceeded the maximum allowed (currently 10). S224 Actual parameters too long for $ The total length of the parameters in a macro call to the indicated macro exceeded the maximum allowed (currently 2048). W225 Argument mismatch for $ The number of arguments supplied in the call to the indicated macro did not agree with the number of parameters in the macro’s definition. F226 Can’t find include file $ The indicated include file could not be opened. S227 Definition too long for $ The length of the macro definition of the indicated macro exceeded the maximum allowed (currently 2048). S228 EOF in comment The end of a file was encountered while processing a comment. S229 EOF in macro call to $ The end of a file was encountered while processing a call to the indicated macro. S230 EOF in string The end of a file was encountered while processing a quoted string. S231 Formal parameters too long for $ The total length of the parameters in the definition of the indicated macro exceeded the maximum allowed (currently 2048). S232 Identifier too long The length of an identifier exceeded the maximum allowed (currently 2048). S233 300 Appendix B W234 Illegal directive name The sequence of characters following a # sign was not an identifier. W235 Illegal macro name A macro name was not an identifier. S236 Illegal number $ The indicated number contained a syntax error. F237 Line too long The input source line length exceeded the maximum allowed (currently 2048). W238 Missing #endif End of file was encountered before a required #endif directive was found. W239 Missing argument list for $ A call of the indicated macro had no argument list. S240 Number too long The length of a number exceeded the maximum allowed (currently 2048). W241 Redefinition of symbol $ The indicated macro name was redefined. I242 Redundant definition for symbol $ A definition for the indicated macro name was found that was the same as a previous definition. F243 String too long The length of a quoted string exceeded the maximum allowed (currently 2048). S244 Syntax error in #define, formal $ not identifier A formal parameter that was not an identifier was used in a macro definition. W245 Syntax error in #define, missing blank after name or arglist There was no space or tab between a macro name or argument list and the macro’s definition. Messages 301 S246 Syntax error in #if A syntax error was found while parsing the expression following a #if or #elif directive. S247 Syntax error in #include The #include directive was not correctly formed. W248 Syntax error in #line A #line directive was not correctly formed. W249 Syntax error in #module A #module directive was not correctly formed. W250 Syntax error in #undef A #undef directive was not correctly formed. W251 Token after #ifdef must be identifier The #ifdef directive was not followed by an identifier. W252 Token after #ifndef must be identifier The #ifndef directive was not followed by an identifier. S253 Too many actual parameters to $ The number of actual arguments to the indicated macro exceeded the maximum allowed (currently 31). S254 Too many formal parameters to $ The number of formal arguments to the indicated macro exceeded the maximum allowed (currently 31). F255 Too much pushback The preprocessor ran out of space while processing a macro expansion. The macro may be recursive. W256 Undefined directive $ The identifier following a # was not a directive name. 302 Appendix B S257 EOF in #include directive End of file was encountered while processing a #include directive. S258 Unmatched #elif A #elif directive was encountered with no preceding #if or #elif directive. S259 Unmatched #else A #else directive was encountered with no preceding #if or #elif directive. S260 Unmatched #endif A #endif directive was encountered with no preceding #if, #ifdef, or #ifndef directive. S261 Include files nested too deeply The nesting depth of #include directives exceeded the maximum (currently 20). S262 Unterminated macro definition for $ A newline was encountered in the formal parameter list for the indicated macro. S263 Unterminated string or character constant A newline with no preceding backslash was found in a quoted string. I264 Possible nested comment The characters /* were found within a comment. S265 S266 S267 W268 Cannot inline subprogram; common block mismatch W269 Cannot inline subprogram; argument type mismatch This message may be Severe if have gone too far to undo inlining process. F270 Missing -exlib option Messages 303 W271 Can’t inline $ - wrong number of arguments I272 Argument of inlined function not used S273 Inline library not specified on command line (-inlib switch) F274 Unable to access file $/TOC S275 Unable to open file $ while extracting or inlining F276 Assignment to constant actual parameter in inlined subprogram I277 Inlining of function $ may result in recursion S278 W279 Possible use of $ before definition in $ The optimizer has detected the possibility that a variable is used before it has been assigned a value. The names of the variable and the function in which the use occurred are listed. The line number, if specified, is the line number of the basic block containing the use of the variable. W280 Syntax error in directive $ messages 280-300 rsvd for directive handling W281 Directive ignored - $ $ S300 Too few data constants in initialization of derived type $ S301 $ must be TEMPLATE or PROCESSOR S302 Unmatched END$ statement S303 END statement for $ required in an interface block S304 EXIT/CYCLE statement must appear in a DO/DOWHILE loop$$ S305 $ cannot be named, $ S306 $ names more than one construct 304 Appendix B S307 $ must have the construct name $ S308 DO may not terminate at an EXIT, CYCLE, RETURN, STOP, GOTO, or arithmetic IF S309 Incorrect name, $, specified in END statement S310 $ $ Generic message for MODULE errors. W311 Non-replicated mapping for $ array, $, ignored W312 Array $ should be declared SEQUENCE W313 Subprogram $ called within INDEPENDENT loop not PURE E314 IPA: actual argument $ is a label, but dummy argument $ is not an asterisk The call passes a label to the subprogram; the corresponding dummy argument in the subprogram should be an asterisk to declare this as the alternate return. I315 IPA: routine $, $ constant dummy arguments This many dummy arguments are being replaced by constants due to interprocedural analysis. I316 IPA: routine $, $ INTENT(IN) dummy arguments This many dummy arguments are being marked as INTENT(IN) due to interprocedural analysis. I317 IPA: routine $, $ array alignments propagated This many array alignments were propagated by interprocedural analysis. I318 IPA: routine $, $ distribution formats propagated This many array distribution formats were propagated by interprocedural analysis. I319 IPA: routine $, $ distribution targets propagated This many array distribution targets were propagated by interprocedural analysis. I320 IPA: routine $, $ common blocks optimized Messages 305 This many mapped common blocks were optimized by interprocedural analysis. I321 IPA: routine $, $ common blocks not optimized This many mapped common blocks were not optimized by interprocedural analysis, either because they were declared differently in different routines, or they did not appear in the main program. I322 IPA: analyzing main program $ Interprocedural analysis is building the call graph and propagating information with the named main program. I323 IPA: collecting information for $ Interprocedural analysis is saving information for the current subprogram for subsequent analysis and propagation. W324 IPA file $ appears to be out of date W325 IPA file $ is for wrong subprogram: $ W326 Unable to open file $ to propagate IPA information to $ I327 IPA: $ subprograms analyzed I328 IPA: $ dummy arguments replaced by constants I329 IPA: $ INTENT(IN) dummy arguments should be INTENT(INOUT) I330 IPA: $ dummy arguments changed to INTENT(IN) I331 IPA: $ inherited array alignments replaced I332 IPA: $ transcriptive distribution formats replaced I333 IPA: $ transcriptive distribution targets replaced I334 IPA: $ descriptive/prescriptive array alignments verified I335 IPA: $ descriptive/prescriptive distribution formats verified I336 IPA: $ descriptive/prescriptive distribution targets verified 306 Appendix B I337 IPA: $ common blocks optimized I338 IPA: $ common blocks not optimized S339 Bad IPA contents file: $ S340 Bad IPA file format: $ S341 Unable to create file $ while analyzing IPA information S342 Unable to open file $ while analyzing IPA information S343 Unable to open IPA contents file $ S344 Unable to create file $ while collecting IPA information F345 Internal error in $: table overflow Analysis failed due to a table overflowing its maximum size. W346 Subprogram $ appears twice The subprogram appears twice in the same source file; IPA will ignore the first appearance. F347 Missing -ipalib option Interprocedural analysis, enabled with the -ipacollect, -ipaanalyze, or -ipapropagate options, requires the -ipalib option to specify the library directory. W348 Common /$/ $ has different distribution target The array was declared in a common block with a different distribution target in another subprogram. W349 Common /$/ $ has different distribution format The array was declared in a common block with a different distribution format in another subprogram. W350 Common /$/ $ has different alignment The array was declared in a common block with a different alignment in another subprogram. W351 Wrong number of arguments passed to $ Messages 307 The subroutine or function statement for the given subprogram has a different number of dummy arguments than appear in the call. W352 Wrong number of arguments passed to $ when bound to $ The subroutine or function statement for the given subprogram has a different number of dummy arguments than appear in the call to the EXTERNAL name given. W353 Subprogram $ is missing A call to a subroutine or function with this name appears, but it could not be found or analyzed. I354 Subprogram $ is not called No calls to the given subroutine or function appear anywhere in the program. W355 Missing argument in call to $ A nonoptional argument is missing in a call to the given subprogram. I356 Array section analysis incomplete Interprocedural analysis for array section arguments is incomplete; some information may not be available for optimization. I357 Expression analysis incomplete Interprocedural analysis for expression arguments is incomplete; some information may not be available for optimization. W358 Dummy argument $ is EXTERNAL, but actual is not subprogram The call statement passes a scalar or array to a dummy argument that is declared EXTERNAL. W359 SUBROUTINE $ passed to FUNCTION dummy argument $ The call statement passes a subroutine name to a dummy argument that is used as a function. W360 FUNCTION $ passed to FUNCTION dummy argument $ with different result type The call statement passes a function argument to a function dummy argument, but the dummy has a different result type. W361 FUNCTION $ passed to SUBROUTINE dummy argument $ 308 Appendix B The call statement passes a function name to a dummy argument that is used as a subroutine. W362 Argument $ has a different type than dummy argument $ The type of the actual argument is different than the type of the corresponding dummy argument. W363 Dummy argument $ is a POINTER but actual argument $ is not The dummy argument is a pointer, so the actual argument must be also. W364 Array or array expression passed to scalar dummy argument $ The actual argument is an array, but the dummy argument is a scalar variable. W365 Scalar or scalar expression passed to array dummy argument $ The actual argument is a scalar variable, but the dummy argument is an array. F366 Internal error: interprocedural analysis fails An internal error occurred during interprocedural analysis; please report this to the compiler maintenance group. If user errors were reported when collecting IPA information or during IPA analysis, correcting them may avoid this error. I367 Array $ bounds cannot be matched to formal argument Passing a nonsequential array to a sequential dummy argument may require copying the array to sequential storage. The most common cause is passing an ALLOCATABLE array or array expression to a dummy argument that is declared with explicit bounds. Declaring the dummy argument as assumed shape, with bounds (:,:,:), will remove this warning. W368 Array-valued expression passed to scalar dummy argument $ The actual argument is an array-valued expression, but the dummy argument is a scalar variable. W369 Dummy argument $ has different rank than actual argument The actual argument is an array or array-valued expression with a different rank than the dummy argument. W370 Dummy argument $ has different shape than actual argument The actual argument is an array or array-valued expression with a different shape than the dummy argument; this may require copying the actual argument into sequential storage. W371 Dummy argument $ is INTENT(IN) but may be modified Messages 309 The dummy argument was declared as INTENT(IN), but analysis has found that the argument may be modified; the INTENT(IN) declaration should be changed. W372 Cannot propagate alignment from $ to $ The most common cause is when passing an array with an inherited alignment to a dummy argument with non- inherited alignment. I373 Cannot propagate distribution format from $ to $ The most common cause is when passing an array with a transcriptive distribution format to a dummy argument with prescriptive or descriptive distribution format. I374 Cannot propagate distribution target from $ to $ The most common cause is when passing an array with a transcriptive distribution target to a dummy argument with prescriptive or descriptive distribution target. I375 Distribution format mismatch between $ and $ Usually this arises when the actual and dummy arguments are distributed in different dimensions. I376 Alignment stride mismatch between $ and $ This may arise when the actual argument has a different stride in its alignment to its template than does the dummy argument. I377 Alignment offset mismatch between $ and $ This may arise when the actual argument has a different offset in its alignment to its template than does the dummy argument. I378 Distribution target mismatch between $ and $ This may arise when the actual and dummy arguments have different distribution target sizes. I379 Alignment of $ is too complex The alignment specification of the array is too complex for interprocedural analysis to verify or propagate; the program will work correctly, but without the benefit of IPA. I380 Distribution format of $ is too complex The distribution format specification of the array is too complex for interprocedural analysis to verify or propagate; the program will work correctly, but without the benefit of IPA. 310 Appendix B I381 Distribution target of $ is too complex The distribution target specification of the array is too complex for interprocedural analysis to verify or propagate; the program will work correctly, but without the benefit of IPA. I382 IPA: $ subprograms analyzed Interprocedural analysis succeeded in finding and analyzing this many subprograms in the whole program. I383 IPA: $ dummy arguments replaced by constants Interprocedural analysis has found this many dummy arguments in the whole program that can be replaced by constants. I384 IPA: $ dummy arguments changed to INTENT(IN) Interprocedural analysis has found this many dummy arguments in the whole program that are not modified and can be declared as INTENT(IN). W385 IPA: $ INTENT(IN) dummy arguments should be INTENT(INOUT) Interprocedural analysis has found this many dummy arguments in the whole program that were declared as INTENT(IN) but should be INTENT(INOUT). I386 IPA: $ array alignments propagated Interprocedural analysis has found this many array dummy arguments that could have the inherited array alignment replaced by a descriptive alignment. I387 IPA: $ array alignments verified Interprocedural analysis has verified that the prescriptive or descriptive alignments of this many array dummy arguments match the alignments of the actual argument. I388 IPA: $ array distribution formats propagated Interprocedural analysis has found this many array dummy arguments that could have the transcriptive distribution format replaced by a descriptive format. I389 IPA: $ array distribution formats verified Interprocedural analysis has verified that the prescriptive or descriptive distribution formats of this many array dummy arguments match the formats of the actual argument. I390 IPA: $ array distribution targets propagated Messages 311 Interprocedural analysis has found this many array dummy arguments that could have the transcriptive distribution target replaced by a descriptive target. I391 IPA: $ array distribution targets verified Interprocedural analysis has verified that the prescriptive or descriptive distribution targets of this many array dummy arguments match the targets of the actual argument. I392 IPA: $ common blocks optimized Interprocedural analysis has found this many common blocks that could be optimized. I393 IPA: $ common blocks not optimized Interprocedural analysis has found this many common blocks that could not be optimized, either because the common block was not declared in the main program, or because it was declared differently in different subprograms. I394 IPA: $ replaced by constant value The dummy argument was replaced by a constant as per interprocedural analysis. I395 IPA: $ changed to INTENT(IN) The dummy argument was changed to INTENT(IN) as per interprocedural analysis. I396 IPA: array alignment propagated to $ The template alignment for the dummy argument was changed as per interprocedural analysis. I397 IPA: distribution format propagated to $ The distribution format for the dummy argument was changed as per interprocedural analysis. I398 IPA: distribution target propagated to $ The distribution target for the dummy argument was changed as per interprocedural analysis. I399 IPA: common block $ not optimized The given common block was not optimized by interprocedural analysis either because it was not declared in the main program, or because it was declared differently in different subprograms. E400 IPA: dummy argument $ is an asterisk, but actual argument is not a label 312 Appendix B The subprogram expects an alternate return label for this argument. E401 Actual argument $ is a subprogram, but Dummy argument $ is not declared EXTERNAL The call statement passes a function or subroutine name to a dummy argument that is a scalar variable or array. E402 Actual argument $ is illegal E403 Actual argument $ and formal argument $ have different ranks The actual and formal array arguments differ in rank, which is allowed only if both arrays are declared with the HPF SEQUENCE attribute. E404 Sequential array section of $ in argument $ is not contiguous When passing an array section to a formal argument that has the HPF SEQUENCE attribute, the actual argument must be a whole array with the HPF SEQUENCE attribute, or an array section of such an array where the section is a contiguous sequence of elements. E405 Array expression argument $ may not be passed to sequential dummy argument $ When the dummy argument has the HPF SEQUENCE attribute, the actual argument must be a whole array with the HPF SEQUENCE attribute or a contiguous array section of such an array, unless an INTERFACE block is used. E406 Actual argument $ and formal argument $ have different character lengths The actual and formal array character arguments have different character lengths, which is allowed only if both character arrays are declared with the HPF SEQUENCE attribute, unless an INTERFACE block is used. W407 Argument $ has a different character length than dummy argument $ The character length of the actual argument is different than the length specified for the corresponding dummy argument. W408 Specified main program $ is not a PROGRAM The main program specified on the command line is a subroutine, function, or block data subprogram. Messages 313 W409 More than one main program in IPA directory: $ and $ There is more than one main program analyzed in the IPA directory shown. The first one found is used. W410 No main program found; IPA analysis fails. The main program must appear in the IPA directory for analysis to proceed. W411 Formal argument $ is DYNAMIC but actual argument is an expression W412 Formal argument $ is DYNAMIC but actual argument $ is not I413 Formal argument $ has two reaching distributions and may be a candidate for cloning I414 $ and $ may be aliased and one of them is assigned Interprocedural analysis has determined that two formal arguments because the same variable is passed in both argument positions, or one formal argument and a global or COMMON variable may be aliased, because the global or COMMON variable is passed as an actual argument. If either alias is assigned in the subroutine, unexpected results may occur; this message alerts the user that this situation is disallowed by the Fortran standard. F415 IPA fails: incorrect IPA file Interprocedural analysis saves its information in special IPA files in the specified IPA directory. One of these files has been renamed or corrupted. This can arise when there are two files with the same prefix, such as ’a.hpf’ and ’a.f90’. E416 Argument $ has the SEQUENCE attribute, but the dummy parameter $ does not When an actual argument is an array with the SEQUENCE attribute, the dummy parameter must have the SEQUENCE attribute or an INTERFACE block must be used. E417 Interface block for $ is a SUBROUTINE but should be a FUNCTION E418 Interface block for $ is a FUNCTION but should be a SUBROUTINE E419 Interface block for $ is a FUNCTION has wrong result type W420 Earlier $ directive overrides $ directive 314 Appendix B W421 $ directive can only appear in a function or subroutine E422 Nonconstant DIM= argument is not supported E423 Constant DIM= argument is out of range E424 Equivalence using substring or vector triplets is not allowed E425 A record is not allowed in this context E426 WORD type cannot be converted E427 Interface block for $ has wrong number of arguments E428 Interface block for $ should have $ E429 Interface block for $ should not have $ E430 Interface block for $ has wrong $ W431 Program is too large for Interprocedural Analysis to complete W432 Illegal type conversion $ E433 Subprogram $ called within INDEPENDENT loop not LOCAL W434 Incorrect home array specification ignored S435 Array declared with zero size An array was declared with a zero or negative dimension bound, as ’real a(-1)’, or an upper bound less than the lower bound, as ’real a(4:2)’. W436 Independent loop not parallelized$ W437 Type $ will be mapped to $ Where DOUBLE PRECISION is not supported, it is mapped to REAL, and similarly for COMPLEX(16) or COMPLEX*32. E438 $ $ not supported on this platform Messages 315 This construct is not supported by the compiler for this target. S439 An internal subprogram cannot be passed as argument - $ S440 Defined assignment statements may not appear in WHERE statement or WHERE block S441 $ may not appear in a FORALL block E442 Adjustable-length character type not supported on this host - $ $ S443 EQUIVALENCE of derived types not supported on this host - $ S444 Derived type in EQUIVALENCE statement must have SEQUENCE attribute - $ A variable or array with derived type appears in an EQUIVALENCE statement. The derived type must have the SEQUENCE attribute, but does not. E445 Array bounds must be integer $ $ The expressions in the array bounds must be integer. S446 Argument number $ to $: rank mismatch The number of dimensions in the array or array expression does not match the number of dimensions in the dummy argument. S447 Argument number $ to $ must be a subroutine or function name S448 Argument number $ to $ must be a subroutine name S449 Argument number $ to $ must be a function name S450 Argument number $ to $: kind mismatch S451 Arrays of derived type with a distributed member are not supported S452 Assumed length character, $, is not a dummy argument S453 Derived type variable with pointer member not allowed in IO - $ $ 316 Appendix B S454 Subprogram $ is not a module procedure Only names of module procedures declared in this module or accessed through USE association can appear in a MODULE PROCEDURE statement. S455 A derived type array section cannot appear with a member array section - $ A reference like A(:)%B(:), where ’A’ is a derived type array and ’B’ is a member array, is not allowed; a section subscript may appear after ’A’ or after ’B’, but not both. S456 Unimplemented for data type for MATMUL S457 Illegal expression in initialization S458 Argument to NULL() must be a pointer S459 Target of NULL() assignment must be a pointer S460 ELEMENTAL procedures cannot be RECURSIVE S461 Dummy arguements of ELEMENATAL procedures must be scalar S462 Arguments and return values of ELEMENATAL procedures cannot have the POINTER attribute S463 Arguments of ELEMENATAL procedures cannot be procedures S464 An ELEMENTAL procedure cannot be passed as argument - $ B.4 Fortran Runtime Error Messages This section presents the error messages generated by the runtime system. The runtime system displays error messages on standard output. B.4.1 Message Format The messages are numbered but have no severity indicators because they all terminate program execution. Messages 317 B.4.2 Message List Here are the runtime error messages: 201 illegal value for specifier An improper specifier value has been passed to an I/O runtime routine. Example: within an OPEN statement, form='unknown'. 202 conflicting specifiers Conflicting specifiers have been passed to an I/O runtime routine. Example: within an OPEN statement, form='unformatted', blank='null'. 203 record length must be specified A recl specifier required for an I/O runtime routine has not been passed. Example: within an OPEN statement, access='direct' has been passed, but the record length has not been specified (recl=specifier). 204 illegal use of a readonly file Self explanatory. Check file and directory modes for readonly status. 205 'SCRATCH' and 'SAVE'/'KEEP' both specified In an OPEN statement, a file disposition conflict has occurred. Example: within an OPEN statement, status='scratch' and dispose='keep' have been passed. 206 attempt to open a named file as 'SCRATCH' 207 file is already connected to another unit 208 'NEW' specified for file that already exists 209 'OLD' specified for file that does not exist 210 dynamic memory allocation failed Memory allocation operations occur only in conjunction with namelist I/O. The most probable cause of fixed buffer overflow is exceeding the maximum number of simultaneously open file units. 318 Appendix B 211 invalid file name 212 invalid unit number A file unit number less than or equal to zero has been specified. 215 formatted/unformatted file conflict Formatted/unformatted file operation conflict. 217 attempt to read past end of file 219 attempt to read/write past end of record For direct access, the record to be read/written exceeds the specified record length. 220 write after last internal record 221 syntax error in format string A runtime encoded format contains a lexical or syntax error. 222 unbalanced parentheses in format string 223 illegal P or T edit descriptor - value missing 224 illegal Hollerith or character string in format An unknown token type has been found in a format encoded at run-time. 225 lexical error -- unknown token type 226 unrecognized edit descriptor letter in format An unexpected Fortran edit descriptor (FED) was found in a runtime format item. 228 end of file reached without finding group 229 end of file reached while processing group Messages 319 230 scale factor out of range -128 to 127 Fortran P edit descriptor scale factor not within range of -128 to 127. 231 error on data conversion 233 too many constants to initialize group item 234 invalid edit descriptor An invalid edit descriptor has been found in a format statement. 235 edit descriptor does not match item type Data types specified by I/O list item and corresponding edit descriptor conflict. 236 formatted record longer than 2000 characters 237 quad precision type unsupported 238 tab value out of range A tab value of less than one has been specified. 239 entity name is not member of group 242 illegal operation on direct access file 243 format parentheses nesting depth too great 244 syntax error - entity name expected 245 syntax error within group definition 246 infinite format scan for edit descriptor 248 illegal subscript or substring specification 249 error in format - illegal E, F, G or D descriptor 320 Appendix B 250 error in format - number missing after '.', '-', or '+' 251 illegal character in format string 252 operation attempted after end of file 253 attempt to read non-existent record (direct access) 254 illegal repeat count in format C++ Dialect Supported 321 Appendix C C++ Dialect Supported The PGC++ compiler accepts the C++ language as defined by The Annotated C++ Reference Manual (ARM) by Ellis and Stroustrup, Addison-Wesley, 1990, including templates, exceptions, and support for the anachronisms described in section 18 of the ARM. This is the same language defined by the language reference for ATT’s cfront version 3.0.1, with the addition of exceptions. PGC++ optionally accepts a number of features erroneously accepted by cfront version 2.1. Using the –b option, PGC++ accepts these features, which may never have been legal C++ but have found their way into some user’s code. Command-line options provide full support of many C++ variants, including strict standard conformance. PGC++ provides command line options that enable the user to specify whether anachronisms and/or cfront 2.1 compatibility features should be accepted. Refer to Section C.4 for details on features that are not part of the ARM but are part of the ANSI C++ working draft X3J16/WG21. C.1 Anachronisms Accepted The following anachronisms are accepted when anachronisms are enabled (when the +p option is not used): • overload is allowed in function declarations. It is accepted and ignored. • Definitions are not required for static data members that can be initialized using default initialization. This anachronism does not apply to static data members of template classes; they must always be defined. • The number of elements in an array may be specified in an array delete operation. The value is ignored. • A single operator++() and operator--() function can be used to overload both prefix and postfix operations. • The base class name may be omitted in a base class initializer if there is only one immediate base class. 322 Appendix C • Assignment to this in constructors and destructors is allowed. This is allowed only if anachronisms are enabled and the assignment to this configuration parameter is enabled. • A bound function pointer (a pointer to a member function for a given object) can be cast to a pointer to a function. • A nested class name may be used as a non-nested class name provided no other class of that name has been declared. This anachronism is not applied to template classes. • A reference to a non-const type may be initialized from a value of a different type. A temporary is created, it is initialized from the (converted) initial value, and the reference is set to the temporary. • A reference to a non-const class type may be initialized from an rvalue of the class type or a derived class thereof. No (additional) temporary is used. • A function with old-style parameter declarations is allowed and may participate in function overloading as though it was prototyped. Default argument promotion is not applied to parameter types of such functions when the check for compatibility is done, so that the following declares the overloading of two functions named f: int f(int); int f(x) char x; return x; It will be noted that in C, this code is legal but has a different meaning: a tentative declaration of f is followed by its definition. • When --nonconst_ref_anachronism is enabled, a reference to a nonconst class can be bound to a class rvalue of the same type or a derived type thereof. struct A { A(int); A operator=(A&); A operator+(const A&); }; main () { A b(1); b = A(1) + A(2); // Allowed as anachronism } C++ Dialect Supported 323 C.2 New Language Features Accepted The following features not in the ARM but in the X3J16/WG21 Working paper are accepted: • The dependent statement of an if, while, do-while, or for is considered to be a scope, and the restriction on having such a dependent statement be a declaration is removed. • The expression tested in an if, while, do-while, or for, as the first operand of a ''?'' operator, or as an operand of the "&&", "::", or "!" operators may have a pointer-to-member type or a class type that can be converted to a pointer-to-member type in addition to the scalar cases permitted by the ARM. • Qualified names are allowed in elaborated type specifiers. • Use of a global-scope qualifier in member references of the form x.::A::B and p->::A::B. • The precedence of the third operand of the ``?'' operator is changed. • If control reaches the end of the main() routine, and main() has an integral return type, it is treated as if a return 0; statement were executed. • Pointers to arrays with unknown bounds as parameter types are diagnosed as errors. • A functional-notation cast of the form A() can be used even if A is a class without a (nontrivial) constructor. The temporary created gets the same default initialization to zero as a static object of the class type. • A cast can be used to select one out of a set of overloaded functions when taking the address of a function. • Template friend declarations and definitions are permitted in class definitions and class template definitions. • Type template parameters are permitted to have default arguments. • Function templates may have nontype template parameters. • A reference to const volatile cannot be bound to an rvalue. • Qualification conversions, such as conversion from T** to T const * const * are allowed. • Digraphs are recognized. • Operator keywords (e.g., and, bitand, etc.) are recognized. 324 Appendix C • Static data member declarations can be used to declare member constants. • wchar_t is recognized as a keyword and a distinct type. • bool is recognized. • RTTI (runtime type identification), including dynamic_cast and the typeid operator, are implemented. • Declarations in tested conditions (in if, switch, for, and while statements) are supported. • Array new and delete are implemented. • New-style casts (static_cast, reinterpret_cast, and const_cast) are implemented. • Definition of a nested class outside its enclosing class is allowed. • mutable is accepted on nonstatic data member declarations. • Namespaces are implemented, including using declarations and directives. Access declarations are broadened to match the corresponding using declarations. • Explicit instantiation of templates is implemented. • typename keyword is implemented. • explicit is accepted to declare Non-converting constructors . • The scope of a variable declared in a for-init-statement of a loop is the scope of the loop, not the surrounding scope. • Member templates are implemented. • The new specialization syntax (using "template<>") is implemented. • Cv-qualifiers are retained on rvalues (in particular, on function return values). • The distinction between trivial and nontrivial constructors has been implemented, as has the distinction between PODs and non-PODs with trivial constructors. • The linkage specification is treated as part of the function type (affecting function overloading and implicit conversions). • extern inline functions are supported, and the default linkage for inline functions is external. • A typedef name may be used in an explicit destructor call. • Placement delete is implemented. C++ Dialect Supported 325 • An array allocated via a placement new can be deallocated via delete. • Covariant return types on overriding virtual functions are supported. • enum types are considered to be non-integral types. • Partial specialization of class templates is implemented. • Partial ordering of function templates is implemented. • Function declarations that match a function template are regarded as independent functions, not as “guiding declarations” that are instances of the template. • It is possible to overload operators using functions that take enum types and no class types. • Explicit specification of function template arguments is supported. • Unnamed template parameters are supported. • The new lookup rules for member references of the form x.A::PB and p->A::B are supported. • The notation :: template (and ->template, etc.) is supported. C.3 The following language features are not accepted The following feature of the ISO/IEC 14882:1998 C++ standard is not supported: • Exported templates are not implemented C.4 Extensions Accepted in Normal C++ Mode The following extensions are accepted in all modes (except when strict ANSI violations are diagnosed as errors, see the –A option): • A friend declaration for a class may omit the class keyword: class A { friend B; // Should be "friend class B" }; • Constants of scalar type may be defined within classes: class A { const int size = 10; int a[size]; }; 326 Appendix C • In the declaration of a class member, a qualified name may be used: struct A{ int A::f(); // Should be int f(); } • The preprocessing symbol c_plusplus is defined in addition to the standard __cplusplus. • An assignment operator declared in a derived class with a parameter type matching one of its base classes is treated as a "default'' assignment operator --- that is, such a declaration blocks the implicit generation of a copy assignment operator. (This is cfront behavior that is known to be relied upon in at least one widely used library.) Here's an example: struct A { } ; struct B : public A { B& operator=(A&); }; By default, as well as in cfront-compatibility mode, there will be no implicit declaration of B::operator=(const B&), whereas in strict-ANSI mode B::operator=(A&) is not a copy assignment operator and B::operator=(const B&) is implicitly declared. • Implicit type conversion between a pointer to an extern “C” function and a pointer to an extern “C++” function is permitted. Here’s an example: extern “C” void f(); // f’s type has extern “C” linkage void (*pf) () // pf points to an extern “C++” function = &f; // error unless implicit conv is allowed This extension is allowed in environments where C and C++ functions share the same calling conventions (though it is pointless unless DEFAULT_C_AND_CPP_FUNTION_TYPES_ARE_DISTINCT is TRUE). When DEFAULT_IMPL_CONV_BETWEEN_C_AND_CPP_FUNCTION_PTRS_ALLOWED is set, it is enabled by default; it can also be enabled in cfront-compatibility mode or with command-line option –implicit_extern_c_type_conversion. It is disabled in strict-ANSI mode. C.5 cfront 2.1 Compatibility Mode The following extensions are accepted in cfront 2.1 compatibility mode in addition to the extensions listed in the following 2.1/3.0 section (i.e., these are things that were corrected in the 3.0 release of cfront): C++ Dialect Supported 327 • The dependent statement of an if, while, do-while, or for is not considered to define a scope. The dependent statement may not be a declaration. Any objects constructed within the dependent statement are destroyed at exit from the dependent statement. • Implicit conversion from integral types to enumeration types is allowed. • A non-const member function may be called for a const object. A warning is issued. • A const void * value may be implicitly converted to a void * value, e.g., when passed as an argument. • When, in determining the level of argument match for overloading, a reference parameter is initialized from an argument that requires a non-class standard conversion, the conversion counts as a user-defined conversion. (This is an outright bug, which unfortunately happens to be exploited in the NIH class libraries.) • When a builtin operator is considered alongside overloaded operators in overload resolution, the match of an operand of a builtin type against the builtin type required by the builtin operator is considered a standard conversion in all cases (e.g., even when the type is exactly right without conversion). • A reference to a non-const type may be initialized from a value that is a const-qualified version of the same type, but only if the value is the result of selecting a member from a const class object or a pointer to such an object. • A cast to an array type is allowed; it is treated like a cast to a pointer to the array element type. A warning is issued. • When an array is selected from a class, the type qualifiers on the class object (if any) are not preserved in the selected array. (In the normal mode, any type qualifiers on the object are preserved in the element type of the resultant array.) • An identifier in a function is allowed to have the same name as a parameter of the function. A warning is issued. • An expression of type void may be supplied on the return statement in a function with a void return type. A warning is issued. • cfront has a bug that causes a global identifier to be found when a member of a class or one of its base classes should actually be found. This bug is not emulated in cfront compatibility mode. • A parameter of type "const void *'' is allowed on operator delete; it is treated as equivalent to "void *". 328 Appendix C • A period (".") may be used for qualification where "::" should be used. Only "::'' may be used as a global qualifier. Except for the global qualifier, the two kinds of qualifier operators may not be mixed in a given name (i.e., you may say A::B::C or A.B.C but not A::B.C or A.B::C). A period may not be used in a vacuous destructor reference nor in a qualifier that follows a template reference such as A::B. • cfront 2.1 does not correctly look up names in friend functions that are inside class definitions. In this example function f should refer to the functions and variables (e.g., f1 and a1) from the class declaration. Instead, the global definitions are used. int a1; int e1; void f1(); class A { int a1; void f1(); friend void f() { int i1 = a1; // cfront uses global a1 f1(); // cfront uses global f1 } }; Only the innermost class scope is (incorrectly) skipped by cfront as illustrated in the following example. int a1; int b1; struct A { static int a1; class B { static int b1; friend void f() { int i1 = a1; // cfront uses A::a1 int j1 = b1; // cfront uses global b1 } }; }; • operator= may be declared as a nonmember function. (This is flagged as an anachronism by cfront 2.1) C++ Dialect Supported 329 • A type qualifier is allowed (but ignored) on the declaration of a constructor or destructor. For example: class A { A() const; // No error in cfront 2.1 mode }; C.6 cfront 2.1/3.0 Compatibility Mode The following extensions are accepted in both cfront 2.1 and cfront 3.0 compatibility mode (i.e., these are features or problems that exist in both cfront 2.1 and 3.0): • Type qualifiers on the this parameter may to be dropped in contexts such as this example: struct A { void f() const; }; void (A::*fp)() = &A::f; This is actually a safe operation. A pointer to a const function may be put into a pointer to non-const, because a call using the pointer is permitted to modify the object and the function pointed to will actually not modify the object. The opposite assignment would not be safe. • Conversion operators specifying conversion to void are allowed. • A nonstandard friend declaration may introduce a new type. A friend declaration that omits the elaborated type specifier is allowed in default mode, but in cfront mode the declaration is also allowed to introduce a new type name. struct A { friend B; }; • The third operator of the ? operator is a conditional expression instead of an assignment expression as it is in the current X3J16/WG21 Working Paper. • A reference to a pointer type may be initialized from a pointer value without use of a temporary even when the reference pointer type has additional type qualifiers above those present in the pointer value. For example, int *p; const int *&r = p; // No temporary used 330 Appendix C • A reference may be initialized with a null. Index 331 Index A Auto-parallelization .................................. 47 B Basic block................................................ 33 Bounds checking..................................... 103 C C/C++ Builtin Functions......................... 191 C/C++ Math Header File......................... 191 C++ Name Mangling .............................. 241 C++ Standard Template Library ............. 207 Cache tiling with -Mvect......................................... 101 Cache tiling failed cache tiling ................................ 105 Command-line Options ....................... 23, 61 -# .......................................................... 67 -### ....................................................... 67 --[no]llalign ......................................... 126 --[no_]alternative_tokens .................... 122 --[no_]bool .......................................... 123 --[no_]exceptions ................................ 125 --[no_]pch_messages .......................... 127 --[no_]using_std .................................. 128 -A ........................................................ 122 -b ................................................ 122, 123 -b3 ....................................................... 123 -byteswapio ........................................... 68 -c .......................................................... 69 -C .......................................................... 68 --cfront_2.1 ......................................... 124 --cfront_3.0 ......................................... 124 --create_pch......................................... 124 -cyglibs.................................................. 69 -D .......................................................... 69 --diag_error ......................................... 125 --diag_remark...................................... 125 --diag_suppress ...................................125 --diag_warning ....................................125 --display_error_number.......................125 -dryrun...................................................70 -E ...........................................................70 -F ...........................................................71 -fast .......................................................71 -flags......................................................72 -fpic .......................................................72 -fPIC......................................................72 -g ...........................................................72 -G ..........................................................72 -g77libs..................................................73 -gopt ......................................................73 -help ......................................................74 -I ...........................................................74 -i2, -i4 and -i8........................................75 -Kflag ....................................................76 -l ...........................................................77 -L ...........................................................77 -M........................................................126 -M ...........................................78 -Manno ................................................103 -Masmkeyword .....................................92 -Mbackslash ..........................................90 -Mbounds ............................................103 -Mbyteswapio......................................103 -Mcache_align.......................................94 -Mchkfpstk ..........................................104 -Mchkptr..............................................104 -Mchkstk .............................................104 -mcmodel=medium .............................108 -Mconcur ...............................................94 -Mcpp ..................................................104 -Mcray...................................................95 -MD.....................................................126 -Mdaz ....................................................84 -Mdclchk ...............................................90 -Mdefaultunit ........................................90 332 Index -Mdepchk .............................................. 95 -Mdlines................................................ 90 -Mdll ................................................... 105 -Mdollar .......................................... 90, 92 -Mdwarf1 .............................................. 84 -Mdwarf2 .............................................. 85 -Mdwarf3 .............................................. 85 -Mextend............................................... 91 -Mextract............................................... 88 -Mfcon .................................................. 92 -Mfixed ................................................. 91 -Mflushz................................................ 85 -Mfprelaxed .......................................... 96 -Mfree ................................................... 91 -Mfunc32 .............................................. 85 -Mgccbugs .......................................... 105 -Mi4 ...................................................... 96 -Minfo ................................................. 105 -Minform............................................. 106 -Minline ................................................ 89 -Miomutex ............................................ 91 -Mipa..................................................... 96 -Mkeepasm.......................................... 106 -Mlarge_arrays...................................... 85 -Mlfs ..................................................... 88 -Mlist................................................... 106 -Mlre ..................................................... 98 -Mmakedll........................................... 106 -Mneginfo ........................................... 105 -Mnoasmkeyword ................................. 92 -Mnobackslash ...................................... 90 -Mnobounds ........................................ 103 -Mnodaz................................................ 84 -Mnodclchk........................................... 90 -Mnodefaultunit .................................... 90 -Mnodepchk .......................................... 95 -Mnodlines ............................................ 90 -Mnoflushz............................................ 85 -Mnofprelaxed ...................................... 96 -Mnoframe ............................................ 99 -Mnoi4 .................................................. 99 -Mnoiomutex ........................................ 91 -Mnolarge_arrays .................................. 85 -Mnolist............................................... 106 -Mnolre ................................................. 99 -Mnomain.............................................. 85 -Mnontemporal ..................................... 85 -Mnoonetrip .......................................... 91 -Mnoopenmp....................................... 106 -Mnopgdllmain ................................... 107 -Mnoprefetch......................................... 99 -Mnor8 ................................................ 100 -Mnor8intrinsics.................................. 100 -Mnorecursive ....................................... 86 -Mnoreentrant ....................................... 86 -Mnoref_externals................................. 86 -Mnosave............................................... 91 -Mnoscalarsse ..................................... 100 -Mnosecond_underscore ....................... 86 -Mnosgimp.......................................... 106 -Mnosignextend .................................... 87 -Mnosingle ............................................ 93 -Mnosmart........................................... 101 -Mnostartup........................................... 88 -Mnostddef ............................................ 88 -Mnostdlib............................................. 88 -Mnostride0........................................... 87 -Mnounixlogical.................................... 91 -Mnounroll .......................................... 101 -Mnoupcase........................................... 92 -Mnovect............................................. 102 -Mnovintr ............................................ 102 -module ............................................... 108 -Monetrip .............................................. 91 -mp ...................................................... 109 -Mpfi ..................................................... 99 -Mpfo .................................................... 99 -Mprefetch............................................. 99 -Mpreprocess....................................... 107 -Mprof ................................................... 86 -Mr8 .................................................... 100 -Mr8intrinsics...................................... 100 -Mrecursive ........................................... 86 -Mreentrant ........................................... 86 Index 333 -Mref_externals..................................... 86 -Msafe_lastval....................................... 87 -Msafeptr............................................. 100 -Msave .................................................. 91 -Mscalarsse ......................................... 100 -Mschar ................................................. 93 -Msecond_underscore ........................... 86 -Msignextend ........................................ 87 -Msingle ................................................ 93 -mslibs................................................. 109 -Msmart............................................... 101 -Mstandard ............................................ 91 -Mstride0............................................... 87 -msvcrt ................................................ 110 -Muchar................................................. 93 -Munix .................................................. 87 -Munixlogical........................................ 91 -Munroll.............................................. 101 -Mupcase............................................... 91 -Mvarargs.............................................. 87 -Mvect................................................. 101 -o ................................................ 111, 114 -O ........................................................ 110 --optk_allow_dollar_in_id_chars ........ 126 -P ........................................................ 126 -pc ....................................................... 112 --pch .................................................... 127 --pch_dir.............................................. 127 --preinclude ......................................... 127 -Q ........................................................ 115 -R ........................................................ 115 -r4 and -r8 ........................................... 116 -rc ........................................................ 116 -S ........................................................ 117 -shared................................................. 117 -show................................................... 117 -silent .................................................. 117 syntax .................................................... 22 -t ........................................................ 128 -time .................................................... 118 -tp ........................................................ 118 -U ........................................................ 119 --use_pch.............................................127 -v .........................................................120 -V ........................................................120 -w ........................................................121 -W .......................................................121 Compilation driver ....................................21 Compilers Invoke at command level ......................22 PGC++ ...........................................17, 20 PGCC ANSI C................................17, 20 PGF77.............................................17, 20 PGF95.............................................17, 20 PGHPF............................................17, 20 cpp.............................................................25 D Data Types ..............................................211 bitfields................................................219 C/C++ aggregate alignment................218 C/C++ scalar data types ......................215 C/C++ struct.......................................217 C/C++ void.........................................220 C++ class and object layout................217 C++ classes.........................................217 DEC structures ....................................213 DEC Unions ........................................213 F90 derived types ................................214 Fortran.................................................211 internal padding...................................218 tail padding..........................................218 Directives Fortran...................................................23 optimization ........................................173 Parallelization......................................135 prefetch................................................188 scope....................................................180 E Environment variables ............................207 FORTRAN_OPT ..................................207 MP_BIND............................................207 MP_BLIST .........................................208 334 Index MP_SPIN............................................ 208 MP_WARN............................................ 208 MPSTKZ................................................ 29 NCPUS ................................................ 208 NCPUS_MAX....................................... 208 NO_STOP_MESSAGE......................... 208 PGI..................................................... 208 PGI_CONTINUE................................ 209 STATIC_RANDOM_SEED ................. 209 TMPDIR.............................................. 209 TZ ....................................................... 209 F Filename Conventions .............................. 24 extensions ............................................. 24 Input Files ............................................. 24 Output Files........................................... 25 Floating-point stack ................................ 112 Fortran directive summary............................... 174 named common blocks ....................... 225 Function Inlining inlining and makefiles......................... 132 inlining examples ................................ 133 inlining restrictions ............................. 133 I Inter-language Calling ............................ 221 %VAL................................................. 226 arguments and return values ............... 225 array indices........................................ 227 C calling C++...................................... 231 C calling Fortran ................................. 229 C++ calling C...................................... 230 C++ calling Fortran............................ 234 character case conventions.................. 223 character return values ........................ 226 compatible data types.................. 223, 224 Fortran calling C................................. 228 Fortran calling C++............................ 232 underscores ......................................... 223 L Language options ...................................... 92 Libraries BLAS .................................................. 206 FFTs .................................................... 206 LAPACK............................................. 206 LIB3F.................................................. 206 shared object files................................ 191 Linux......................................................... 29 Header Files .......................................... 29 Parallelization ....................................... 29 Listing Files ............................ 103, 105, 106 Loop unrolling .......................................... 40 Loops failed auto-parallelization...................... 48 innermost............................................... 49 scalars.................................................... 49 timing .................................................... 49 M Command-line Options ........................... 109 O OpenMP C/C++ Pragmas........................ 155 atomic.................................................. 166 barrier.................................................. 164 critical ................................................. 159 flush .................................................... 167 for........................................................ 161 master.................................................. 160 ordered ................................................ 166 parallel ................................................ 156 parallel for ........................................... 164 parallel sections................................... 165 sections................................................ 165 single................................................... 160 threadprivate ....................................... 167 OpenMP C/C++ Support Routines omp_destroy_lock() ............................ 170 omp_get _thread_num() ...................... 168 omp_get_dynamic() ............................ 169 omp_get_max_threads()...................... 169 Index 335 omp_get_nested()................................ 169 omp_get_num_procs() ........................ 169 omp_get_num_threads() ..................... 168 omp_get_wtick() ................................. 170 omp_get_wtime() ................................ 170 omp_in_parallel()................................ 169 omp_init_lock()................................... 170 omp_set_dynamic()............................. 169 omp_set_lock() ................................... 170 omp_set_nested() ................................ 169 omp_set_num_threads()...................... 168 omp_test_lock() .................................. 171 omp_unset_lock()................................ 170 OpenMP environment variables MPSTKZ.............................. 154, 171, 207 OMP_DYNAMIC .......................... 154, 171 OMP_NESTED............................. 154, 171 OMP_NUM_THREADS................. 154, 171 OMP_SCHEDULE................................ 154 OpenMP Fortran Directives .................... 135 ATOMIC............................................. 150 BARRIER ........................................... 145 CRITICAL .......................................... 139 DO....................................................... 142 FLUSH................................................ 151 MASTER ............................................ 140 ORDERED.......................................... 150 PARALLEL ........................................ 136 PARALLEL DO ................................. 147 PARALLEL SECTIONS .................... 149 PARALLEL WORKSHARE.............. 147 SECTIONS ......................................... 148 SINGLE .............................................. 141 THREADPRIVATE............................ 151 WORKSHARE ................................... 145 OpenMP Fortran Support Routines omp_destroy_lock() ............................ 153 omp_get_dynamic() ............................ 153 omp_get_max_threads()...................... 152 omp_get_nested()................................ 153 omp_get_num_procs() ........................ 152 omp_get_num_threads() ..................... 151 omp_get_thread_num() .......................152 omp_get_wtick() .................................153 omp_get_wtime() ................................153 omp_in_parallel() ................................152 omp_init_lock()...................................153 omp_set_dynamic().............................152 omp_set_lock()....................................153 omp_set_nested() ................................153 omp_set_num_threads() ......................152 omp_test_lock()...................................154 omp_unset_lock()................................153 Optimization............................................173 C/C++ pragmas .............................59, 181 C/C++ pragmas scope .........................185 cache tiling ..........................................101 default optimization levels ....................58 Fortran directives ..........................59, 173 Fortran directives scope ......................180 function inlining ............................34, 129 global optimization..........................34, 38 inline libraries......................................130 Inter-Procedural Analysis................34, 52 IPA ........................................................34 PFO .......................................................34 local optimization..................................33 loop optimization ..................................34 loop unrolling .......................... 34, 40, 101 loops ................................................98, 99 -O ........................................................110 -O0 ........................................................37 -O1 ........................................................37 -O2 ........................................................37 -O3 ........................................................37 -Olevel...................................................37 parallelization..................................34, 47 pointers................................................100 prefetching ............................................99 profile-feedback (PFO) .........................58 Profile-Feedback Optimization .............34 vectorization....................................34, 41 336 Index P Parallelization ........................................... 47 auto-parallelization ............................... 47 failed auto-parallelization ............. 48, 105 -Mconcur auto-parallelization............... 94 NCPUS environment variable............... 48 safe_lastval............................................ 50 user-directed........................................ 109 Parallelization Directives ........................ 135 Parallelization Pragmas........................... 155 Pragmas C/C++ .................................................. 23 optimization ........................................ 182 scope ................................................... 185 Prefetch directives................................... 188 Preprocessor cpp......................................................... 25 Fortran................................................... 25 R Run-time Environment............................ 245 S Shared object files................................... 191 T Timing CPU_CLOCK ....................................... 59 execution............................................... 59 SYSTEM_CLOCK ............................... 59 Tools PGDBG........................................... 18, 20 PGPROF......................................... 18, 20 V Vectorization..................................... 41, 101 SSE instructions.................................. 102 W Win32 Calling Conventions C ................................................ 235, 238 Default ........................................ 235, 237 STDCALL................................... 235, 237 symbol name construction................... 237 UNIX-style.................................. 235, 238 PGI ® User’s Guide Parallel Fortran, C and C++ for Scientists and Engineers The Portland Group™ STMicroelectronics Two Centerpointe Drive Lake Oswego, OR 97035ii While every precaution has been taken in the preparation of this document, The Portland Group™, a wholly-owned subsidiary of STMicroelectronics, makes no warranty for the use of its products and assumes no responsibility for any errors that may appear, or for damages resulting from the use of the information contained herein. The Portland Group retains the right to make changes to this information at any time, without notice. The software described in this document is distributed under license from STMicroelectronics and may be used or copied only in accordance with the terms of the license agreement. No part of this document may be reproduced or transmitted in any form or by any means, for any purpose other than the purchaser's personal use without the express written permission of The Portland Group. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this manual, The Portland Group was aware of a trademark claim. The designations have been printed in caps or initial caps. Thanks is given to the Parallel Tools Consortium and, in particular, to the High Performance Debugging Forum for their efforts. PGF95, PGF90, PGC++, Cluster Development Kit, CDK, PGI Unified Binary, PGI Visual Fortran, PVF and The Portland Group are trademarks and PGI, PGHPF, PGF77, PGCC, PGPROF, and PGDBG are registered trademarks of STMicroelectronics, Inc. Other brands and names are the property of their respective owners. The use of STLport, a C++ Library, is licensed separately and license, distribution and copyright notice can be found in the online documentation for a given release of the PGI compilers and tools. PGI User's Guide Copyright © 1998 – 2000 The Portland Group, Inc. Copyright © 2000 – 2007 STMicroelectronics, Inc. All rights reserved. Printed in the United States of America First Printing: Release 1.7, Jun 1998 Second Printing: Release 3.0, Jan 1999 Third Printing: Release 3.1, Sep 1999 Fourth Printing: Release 3.2, Sep 2000 Fifth Printing: Release 4.0, May 2002 Sixth Printing: Release 5.0, Jun 2003 Seventh Printing: Release 5.1, Nov 2003 Eight Printing: Release 5.2, Jun 2004 Ninth Printing: Release 6.0, Mar 2005 Tenth Printing: Release 6.1, Dec 2005 Eleventh Printing: Release 6.2, Aug 2006 Twelfth Printing: Release 7.0-1, December, 2006 Thirteenth Printing: Release 7.0-2, February, 2007 Technical support: http://www.pgroup.com/support/ Sales: sales@pgroup.com Web: http://www.pgroup.com/iii Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xv Audience Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Compatibility and Conformance to Standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvi Hardware and Software Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xvii Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii Related Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxii 1 Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Invoking the Command-level PGI Compilers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Command-line Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Command-line Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Fortran Directives and C/C++ Pragmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 Filename Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 Input Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 Output Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 Parallel Programming Using the PGI Compilers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 Running SMP Parallel Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 Running Data Parallel HPF Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 Using the PGI Compilers on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10 Linux Header Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10 Running Parallel Programs on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10 Using the PGI Compilers on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 BASH Shell Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 Windows Command Prompt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 Using the PGI Compilers on SUA and SFU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 Subsystem for Unix Applications (SUA and SFU) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 SUA/SFU Header Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 Running Parallel Programs on SUA and SFU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 Customizing the Compilers with siterc and User rc Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 2 Optimization & Parallelization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Overview of Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 Getting Started with Optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 Local and Global Optimization using -O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19iv Scalar SSE Code Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Loop Unrolling using -Munroll . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Vectorization using -Mvect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Vectorization Sub-options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Assoc Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Cachesize Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 SSE Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Prefetch Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Vectorization Example Using SSE/SSE2 Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Auto-Parallelization using -Mconcur . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Auto-parallelization Sub-options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Altcode Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Dist Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Cncall Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Loops That Fail to Parallelize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Innermost Loops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Timing Loops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Scalars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Scalar Last Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Processor-Specific Optimization and the Unified Binary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Inter-Procedural Analysis and Optimization using –Mipa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Building a Program Without IPA – Single Step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Building a Program Without IPA - Several Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Building a Program Without IPA Using Make . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Building a Program with IPA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Building a Program with IPA - Single Step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Building a Program with IPA - Several Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Building a Program with IPA Using Make . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Questions about IPA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Profile-Feedback Optimization using –Mpfi/–Mpfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Default Optimization Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Local Optimization Using Directives and Pragmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Execution Timing and Instruction Counting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Portability of Multi-Threaded Programs on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 libpgbind . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 libnuma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3 Command Line Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47 Generic PGI Compiler Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55v C and C++ -specific Compiler Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115 4 Function Inlining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Invoking Function Inlining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .123 Using an Inline Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .124 Creating an Inline Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125 Working with Inline Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125 Updating Inline Libraries - Makefiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .126 Error Detection during Inlining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .127 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .127 Restrictions on Inlining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .128 5 OpenMP Directives for Fortran . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Parallelization Directives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .131 PARALLEL ... END PARALLEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .132 CRITICAL ... END CRITICAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .136 MASTER ... END MASTER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .137 SINGLE ... END SINGLE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .138 DO ... END DO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .139 WORKSHARE ... END WORKSHARE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .141 BARRIER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .142 DOACROSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .142 PARALLEL DO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .143 PARALLEL WORKSHARE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .144 SECTIONS … END SECTIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .145 PARALLEL SECTIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .145 ORDERED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .146 ATOMIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .147 FLUSH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .147 THREADPRIVATE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .148 Run-time Library Routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .148 Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .151 6 OpenMP Pragmas for C and C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 Parallelization Pragmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .155 omp parallel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .156 omp critical . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .159 omp master . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .160 omp single . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .161 omp for . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .161vi omp barrier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 omp parallel for . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 omp sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 omp parallel sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 omp ordered . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 omp atomic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 omp flush . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 omp threadprivate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 Run-time Library Routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 7 Directives and Pragmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .175 PGI Proprietary Fortran Directives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 PGI Proprietary Fortran Directive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 Scope of Directives and Command Line options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 !DEC$ Directives for Windows Fortran . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 Adding Pragmas to C and C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 C/C++ Pragma Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 Scope of C/C++ Pragmas and Command Line Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 Prefetch Directives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 8 Libraries and Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .197 Using builtin Math Functions in C/C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 Creating and Using Shared Object Files on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 Creating and Using Dynamic-Link Libraries on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 Using LIB3F . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 LAPACK, the BLAS and FFTs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 The C++ Standard Template Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 Stack Traceback and JIT Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 9 Fortran, C and C++ Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .215 Fortran Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 Fortran Scalars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 FORTRAN 77 Aggregate Data Type Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Fortran 90 Aggregate Data Types (Derived Types) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 C and C++ Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 C and C++ Scalars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 C and C++ Aggregate Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 Class and Object Data Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224vii Aggregate Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .224 Bit-field Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .226 Other Type Keywords in C and C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .226 10 Programming Considerations for 64-Bit Environments . . . . . . . . . . . . . . . . . . 229 Data Types in the 64-Bit Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .229 C/C++ Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .229 Fortran Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .230 Large Static Data in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .230 Large Dynamically Allocated Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .230 64-Bit Array Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .231 Compiler Options for 64-bit Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .231 Practical Limitations of Large Array Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .234 Example: Medium Memory Model and Large Array in C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .234 Example: Medium Memory Model and Large Array in Fortran . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .236 Example: Large Array and Small Memory Model in Fortran . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 11 Inter-language Calling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 Overview of Calling Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .239 Inter-language Calling Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .239 Functions and Subroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .240 Upper and Lower Case Conventions, Underscores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .241 Compatible Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .241 Fortran Named Common Blocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .243 Argument Passing and Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .244 Passing by Value (%VAL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .244 Character Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .244 Complex Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .245 Array Indices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .246 Example - Fortran Calling C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .246 Example - C Calling Fortran . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .247 Example - C ++ Calling C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .248 Example - C Calling C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .249 Example - Fortran Calling C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .250 Example - C++ Calling Fortran . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .251 Win32 Calling Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .253 Win32 Fortran Calling Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .253 Symbol Name Construction and Calling Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .255 Using the Default Calling Convention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .256 Using the STDCALL Calling Convention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .256viii Using the C Calling Convention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 Using the UNIX Calling Convention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 12 C/C++ Inline Assembly and Intrinsics 259 Inline Assembly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 Extended Inline Assembly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 Output Operands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 Input Operands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264 Clobber List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266 Additional Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268 Simple Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 Machine Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 Multiple Alternative Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274 Constraint Modifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 Operand Aliases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 Assembly String Modifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278 Extended Asm Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280 Intrinsics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281 13 C++ Name Mangling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .283 Types of Mangling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 Mangling Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 Type Name Mangling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 Nested Class Name Mangling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 Local Class Name Mangling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 Template Class Name Mangling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 Appendix A. Run-time Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .287 Linux86 and Win32 Programming Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 Function Calling Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 Register Usage Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 Function Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 Argument Passing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 Linux86-64 Programming Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 Function Calling Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 Function Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 Argument Passing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 Linux86-64 Fortran Supplement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306 Fortran Fundamental Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307ix Naming Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .308 Argument Passing and Return Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .308 Inter-language Calling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .309 Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .311 Common Blocks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .311 Win64/SUA64 Programming Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .313 Function Calling Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .313 Function Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .317 Argument Passing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .317 Win64/SUA64 Fortran Supplement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .321 Fortran Fundamental Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .322 Fortran Naming Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .323 Fortran Argument Passing and Return Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .323 Interlanguage Calling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .324 Appendix B. Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329 Diagnostic Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .329 Phase Invocation Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .330 Fortran Compiler Error Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .330 Message Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .330 Message List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .330 Fortran Runtime Error Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .378 Message Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .378 Message List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .378 Appendix C. C++ Dialect Supported . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 Anachronisms Accepted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .383 New Language Features Accepted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .384 The following language features are not accepted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .387 Extensions Accepted in Normal C++ Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .387 cfront 2.1 Compatibility Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .389 cfront 2.1/3.0 Compatibility Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .391 Appendix D. C/C++ MMX/SSE Inline Intrinsics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393xxi Tables Table 1-1: Stop after Options, Inputs and Outputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 Table 1-2: Examples of Using siterc and User rc Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14 Table 2-1: Optimization and –O, –g and –M Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43 Table 3-1: Generic PGI Compiler Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .48 Table 3-2: C and C++ -specific Compiler Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52 Table 3-3: –M Options Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .66 Table 3-4: Optimization and –O, –g, –Mvect, and –Mconcur Options. . . . . . . . . . . . . . . . . . . . . . . . . . . .104 Table 5-1: Initialization of REDUCTION Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .135 Table 6-1: Initialization of Reduction Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .158 Table 7-1: Proprietary Fortran Directive Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .177 Table 7-2: C/C++ Pragma Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .189 Table 8-1: Supported PGI_TERM Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .212 Table 9-1: Representation of Fortran Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .216 Table 9-2: Real Data Type Ranges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .218 Table 9-3: Scalar Type Alignment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .218 Table 9-4: C/C++ Scalar Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .221 Table 9-5: Scalar Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .223 Table 10-1: 64-bit Compiler Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .232 Table 10-2: Effects of Options on Memory and Array Sizes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .233 Table 10-3: 64-Bit Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .234 Table 11-1: Fortran and C/C++ Data Type Compatibility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 Table 11-2: Fortran and C/C++ Representation of the COMPLEX Type . . . . . . . . . . . . . . . . . . . . . . . . . . . .242 Table 11-3: Calling Conventions Supported by the PGI Fortran Compilers . . . . . . . . . . . . . . . . . . . . . . . . .254 Table 12-1: Simple Constraints. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .269 Table 12-2: x86/x86_64 Machine Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .271 Table 12-3: Multiple Alternative Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .274 Table 12-4: Constraint Modifier Characters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .275 Table 12-5: Assembly String Modifier Characters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .278 Table 12-6: Intrinsic Header File Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .282 Table A-1: Register Allocation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .288 Table A-2: Standard Stack Frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .289 Table A-3: Stack Contents for Functions Returning struct/union. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .292 Table A-4: Integral and Pointer Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .293 Table A-5: Floating-point Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .294 Table A-6: Structure and Union Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .295 Table A-7: Register Allocation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .298xii Table A-8: Standard Stack Frame. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 Table A-9: Register Allocation for Example A-2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 Table A-10: Linux86-64 Fortran Fundamental Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 Table A-11: Fortran and C/C++ Data Type Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310 Table A-12: Fortran and C/C++ Representation of the COMPLEX Type . . . . . . . . . . . . . . . . . . . . . . . . . . . 310 Table A-13: Register Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314 Table A-14: Standard Stack Frame. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 Table A-15: Register Allocation for Example A-4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 Table A-16: Win64/SUA64 Fortran Fundamental Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322 Table A-17: Fortran and C/C++ Data Type Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 Table A-18: Fortran and C/C++ Representation of the COMPLEX Type . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 Table D-1: MMX Intrinsics (mmintrin.h) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 Table D-2: SSE Intrinsics (xmmintrin.h) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 Table D-3: SSE2 Intrinsics (emmintrin.h). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 Table D-4: SSE3 Intrinsics (pmmintrin.h) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400xiii Figures Figure 9-1: Internal Padding in a Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .225 Figure 9-2: Tail Padding in a Structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .226xivAudience Description xv Preface This guide is part of a set of manuals that describe how to use The Portland Group (PGI) Fortran, C, and C++ compilers and program development tools. In particular, these include the PGF77, PGF95, PGHPF, PGC++, and PGCC ANSI C compilers, the PGPROF profiler, and the PGDBG debugger. These compilers and tools work in conjunction with an x86 or x64 assembler and linker. You can use the PGI compilers and tools to compile, debug, optimize and profile serial and parallel applications for x86 (Intel Pentium II/III/4/M, Intel Centrino, Intel Xeon, AMD Athlon XP/MP) or x64 (AMD Athlon64/Opteron/Turion, Intel EM64T, Intel Core Duo, Intel Core 2 Duo) processor-based systems. The PGI User's Guide provides operating instructions for the PGI command-level development environment. It also contains details concerning the PGI compilers' interpretation of the Fortran language, implementation of Fortran language extensions, and command-level compilation. Previous experience with or knowledge of the Fortran programming language is assumed. Audience Description This manual is intended for scientists and engineers using the PGI compilers. To use these compilers, you should be aware of the role of high-level languages (e.g. Fortran, C, C++) and assembly-language in the software development process and should have some level of understanding of programming. The PGI compilers are available on a variety of x86 or x64 hardware platforms and operating systems. You need to be familiar with the basic commands available on your system. Finally, your system needs to be running a properly installed and configured version of the compilers. For information on installing PGI compilers and tools, refer to the Release and Installation notes included with your software. Compatibility and Conformance to Standards For further information, refer to the following: • American National Standard Programming Language FORTRAN, ANSI X3. -1978 (1978). • ISO/IEC 1539 : 1991, Information technology – Programming Languages – Fortran, Geneva, 1991 (Fortran 90).xvi • ISO/IEC 1539 : 1997, Information technology – Programming Languages – Fortran, Geneva, 1997 (Fortran 95). • Fortran 95 Handbook Complete ISO/ANSI Reference, Adams et al, The MIT Press, Cambridge, Mass, 1997. • High Performance Fortran Language Specification, Revision 1.0, Rice University, Houston, Texas (1993), http://www.crpc.rice.edu/HPFF. • High Performance Fortran Language Specification, Revision 2.0, Rice University, Houston, Texas (1997), http://www.crpc.rice.edu/HPFF. • OpenMP Application Program Interface, Version 2.5, May 2005, http://www.openmp.org. • Programming in VAX Fortran, Version 4.0, Digital Equipment Corporation (September, 1984). • IBM VS Fortran, IBM Corporation, Rev. GC26-4119. • Military Standard, Fortran, DOD Supplement to American National Standard Programming Language Fortran, ANSI x.3-1978, MIL-STD-1753 (November 9, 1978). • American National Standard Programming Language C, ANSI X3.159-1989. • ISO/IEC 9899:1999, Information technology – Programming Languages – C, Geneva, 1999 (C99). Organization This manual is divided into the following chapters and appendices: Chapter 1, “Getting Started” provides an introduction to the PGI compilers and describes their use and overall features. Chapter 2, “Optimization & Parallelization” describes standard optimization techniques that, with little effort, allow users to significantly improve the performance of programs. Chapter 3, “Command Line Options” provides a detailed description of each command-line option. Chapter 4, “Function Inlining” describes how to use function inlining and shows how to create an inline library. Chapter 5, “OpenMP Directives for Fortran” provides a description of the OpenMP Fortran parallelization directives and shows examples of their use.Hardware and Software Constraints xvii Chapter 6, “OpenMP Pragmas for C and C++” provides a description of the OpenMP C and C++ parallelization pragmas and shows examples of their use. Chapter 7, “Directives and Pragmas” provides a description of each Fortran optimization directive and C/C++ optimization pragma, and shows examples of their use. Chapter 8, “Libraries and Environment Variables” discusses PGI support libraries, shared object files, and environment variables that affect the behavior of the PGI compilers. Chapter 9, “Fortran, C and C++ Data Types” describes the data types that are supported by the PGI Fortran, C, and C++ compilers. Chapter 10, “Programming Considerations for 64-Bit Environments” discusses issues that programmers should be aware of when targeting 64-bit processors. Chapter 11, “Inter-language Calling” provides examples showing how to place C Language calls in a Fortran program and Fortran Language calls in a C program. Chapter 12, “C/C++ Inline Assembly and Intrinsics” describes how to use inline assembly code in C and C++ programs, as well as how to use intrinsic functions that map directly to x86 and x64 machine instructions. Chapter 13, “C++ Name Mangling” describes the name mangling facility and explains the transformations of names of entities to names that include information on aspects of the entity’s type and a fully qualified name. Chapter Appendix A., “Run-time Environment” describes the assembly language calling conventions and examples of assembly language calls. Chapter Appendix B., “Messages” provides a list of compiler error messages. Chapter Appendix C., “C++ Dialect Supported” lists more details of the version of the C++ language that PGC++ supports. Hardware and Software Constraints This guide describes versions of the PGI compilers that produce assembly code for x86 and x64 processor-based systems. Details concerning environment-specific values and defaults and systemspecific features or limitations are presented in the release notes delivered with the PGI compilers.xviii Conventions The PGI User's Guide uses the following conventions: italic is used for commands, filenames, directories, arguments, options and for emphasis. Constant Width is used in examples and for language statements in the text, including assembly language statements. [ item1 ] square brackets indicate optional items. In this case item1 is optional. { item2 | item 3} braces indicate that a selection is required. In this case, you must select either item2 or item3. filename... ellipsis indicate a repetition. Zero or more of the preceding item may occur. In this example, multiple filenames are allowed. FORTRAN Fortran language statements are shown in the text of this guide using upper-case characters and a reduced point size. The PGI compilers and tools are supported on both 32-bit and 64-bit variants of the Linux and Windows operating systems on a variety of x86-compatible processors. There are a wide variety of releases and distributions of each of these types of operating systems. The PGI User’s Guide defines the following terms with respect to these platforms: x86 a processor designed to be binary compatible with i386/i486 and previous generation processors from Intel* Corporation. Used to refer collectively to such processors up to and including 32-bit variants. IA32 an Intel Architecture 32-bit processor designed to be binary compatible with x86 processors, but incorporating new features such as streaming SIMD extensions (SSE) for improved performance. AMD64 a 64-bit processor from AMD designed to be binary compatible with IA32 processors, and incorporating new features such as additional registers and 64-bit addressing support for improved performance and greatly increased memory range. EM64T a 64-bit IA32 processor with Extended Memory 64-bit Technology extensions that are binary compatible with AMD64 processors. Conventions xix x64 collectively, all AMD64 and EM64T processors supported by the PGI compilers. linux86 32-bit Linux operating system running on an x86 or x64 processor-based system, with 32-bit GNU tools, utilities and libraries used by the PGI compilers to assemble and link for 32-bit execution. linux86-64 64-bit Linux operating system running on an x64 processor-based system, with 64-bit and 32-bit GNU tools, utilities and libraries used by the PGI compilers to assemble and link for execution in either linux86 or linux86-64 environments. The 32-bit development tools and execution environment under linux86-64 are considered a cross development environment for x86 processor-based applications. SFU Services for Unix, a 32-bit-only predecessor of SUA, the Subsystem for Unix Applications. See SUA. SUA Subsystem for UNIX-based Applications (SUA) is source-compatibility subsystem for compiling and running custom UNIX-based applications on a computer running 32-bit or 64-bit Windows server-class operating system. It provides an operating system for Portable Operating System Interface (POSIX) processes. SUA supports a package of support utilities (including shells and >300 Unix commands), case-sensitive file names, and job control. The subsystem installs separately from the Windows kernel to support UNIX functionality without any emulation. Win32 any of the 32-bit Microsoft Windows Operating Systems (XP/2000/Server 2003) running on an x86 or x64 processor-based system. On these targets, the PGI compiler products include all of the tools and libraries needed to build executables for 32-bit Windows systems. Win64 any of the 64-bit Microsoft Windows Operating Systems (XP Professional / Windows Server 2003 x64 Editions) running on an x64 processor-based system. On these targets, the PGI compiler products include all of the tools and libraries needed to build executables for 32-bit Windows systems. Windows collectively, all Win32 and Win64 platforms supported by the PGI compilers. The following table lists the PGI compilers and tools and their corresponding commands:xx Table P-1: PGI Compilers and Commands In general, the designation PGF95 is used to refer to The Portland Group’s Fortran 90/95 compiler, and pgf95 is used to refer to the command that invokes the compiler. A similar convention is used for each of the PGI compilers and tools. For simplicity, examples of command-line invocation of the compilers generally reference the pgf95 command and most source code examples are written in Fortran. Usage of the PGF77 compiler, whose features are a subset of PGF95, is similar. Usage of PGHPF, PGC++, and PGCC ANSI C99 is consistent with PGF95 and PGF77, but there are command-line options and features of these compilers that do not apply to PGF95 and PGF77 (and vice versa). There are a wide variety of x86-compatible processors in use. All are supported by the PGI compilers and tools. Most of these processors are forward-compatible, but not backward-compatible. That means code compiled to target a given processor will not necessarily execute correctly on a previous-generation processor. The most important processor types, along with a list of the features utilized by the PGI compilers that distinguish them from a compatibility standpoint, are listed in the following table: Compiler or Tool Language or Function Command PGF77 FORTRAN 77 pgf77 PGF95 Fortran 90/95 pgf95 PGHPF High Performance Fortran pghpf PGCC C ANSI C99 and K&R C pgcc PGC++ ANSI C++ with cfront features pgcpp (pgCC) PGDBG Source code debugger pgdbg PGPROF Performance profiler pgprofConventions xxi Table P-2: Processor Options In this manual, the convention is to use “x86” to specify the group of processors in the previous table that are listed as “32-bit” but not “64-bit.” The convention is to use x64 to specify the group of processors that are listed as both “32-bit” and “64-bit.” x86 processor-based systems can run only 32-bit Processor Prefetch SSE1 SSE2 SSE3 32-bit 64-bit Scalar FP Default AMD Athlon N N N N Y N x87 AMD Athlon XP/MP Y Y N N Y N x87 AMD Athlon64 Y Y Y N Y Y SSE AMD Opteron Y Y Y N Y Y SSE AMD Opteron Rev E Y Y Y Y Y Y SSE AMD Opteron Rev F Y Y Y Y Y Y SSE AMD Turion Y Y Y Y Y Y SSE Intel Celeron N N N N Y N x87 Intel Pentium II N N N N Y N x87 Intel Pentium III Y Y N N Y N x87 Intel Pentium 4 Y Y Y N Y N SSE Intel Pentium M Y Y Y N Y N SSE Intel Centrino Y Y Y N Y N SSE Intel Pentium 4 EM64T Y Y Y Y Y Y SSE Intel Xeon EM64T Y Y Y Y Y Y SSE Intel Core Duo EM64T Y Y Y Y Y Y SSE Intel Core 2 Duo EM64T Y Y Y Y Y Y SSExxii operating systems. x64 processor-based systems can run either 32-bit or 64-bit operating systems, and can execute all 32-bit x86 binaries in either case. x64 processors have additional registers and 64-bit addressing capabilities that are utilized by the PGI compilers and tools when running on a 64-bit operating system. The prefetch, SSE1, SSE2 and SSE3 processor features further distinguish the various processors. Where such distinctions are important with respect to a given compiler option or feature, it is explicitly noted in this manual. Note that the default for performing scalar floating-point arithmetic is to use SSE instructions on targets that support SSE1 and SSE2. See section 2.3.1, Scalar SSE Code Generation, for a detailed discussion of this topic. Related Publications The following documents contain additional information related to the x86 and x64 architectures, and the compilers and tools available from The Portland Group. • PGI Fortran Reference manual describes the FORTRAN 77, Fortran 90/95, and HPF statements, data types, input/output format specifiers, and additional reference material related to use of the PGI Fortran compilers. • System V Application Binary Interface Processor Supplement by AT&T UNIX System Laboratories, Inc. (Prentice Hall, Inc.). • System V Application Binary Interface X86-64 Architecture Processor Supplement, http:// www.x86-64.org/abi.pdf. • Fortran 95 Handbook Complete ISO/ANSI Reference, Adams et al, The MIT Press, Cambridge, Mass, 1997. • Programming in VAX Fortran, Version 4.0, Digital Equipment Corporation (September, 1984). • IBM VS Fortran, IBM Corporation, Rev. GC26-4119. • The C Programming Language by Kernighan and Ritchie (Prentice Hall). • C: A Reference Manual by Samuel P. Harbison and Guy L. Steele Jr. (Prentice Hall, 1987). • The Annotated C++ Reference Manual by Margaret Ellis and Bjarne Stroustrup, AT&T Bell Laboratories, Inc. (Addison-Wesley Publishing Co., 1990).Related Publications xxiii • OpenMP Application Program Interface , Version 2.5 May 2005 (OpenMP Architecture Review Board, 1997-2005).xxivOverview 1 1 Getting Started This chapter describes how to use the PGI compilers. The command used to invoke a compiler, for example the pgf95 command, is called a compiler driver. The compiler driver controls the following phases of compilation: preprocessing, compiling, assembling, and linking. Once a file is compiled and an executable file is produced, you can execute, debug, or profile the program on your system. Executables produced by the PGI compilers are unconstrained, meaning they can be executed on any compatible x86 or x64 processor-based system regardless of whether the PGI compilers are installed on that system. Overview In general, using a PGI compiler involves three steps: 1. Produce a program in a file containing a .f extension or another appropriate extension (see “Input Files” on page 4). This may be a program that you have written or a program that you are modifying. 2. Compile the program using the appropriate compiler command. 3. Execute, debug, or profile the executable file on your system. The PGI compilers allow many variations on these general program development steps. These variations include the following: • Stop the compilation after preprocessing, compiling or assembling to save and examine intermediate results. • Provide options to the driver that control compiler optimization or that specify various features or limitations. • Include as input intermediate files such as preprocessor output, compiler output, or assembler output. Invoking the Command-level PGI Compilers To translate and link a Fortran, C, or C++ program, the pgf77, pgf95, pghpf, pgcc, and pgcpp commands do the following:Getting Started 2 • Preprocess the source text file • Check the syntax of the source text • Generate an assembly language file • Pass control to the subsequent assembly and linking steps For example, if you enter the following simple Fortran program in the file hello.f: print *, "hello" end You can compile it from a shell prompt using the default pgf95 driver options. PGI$ pgf95 hello.f Linking: PGI$ By default, the executable output is placed in the file a.out (or, on Windows platforms, a filename based on the name of the first source or object file on the command line). Use the –o option to specify an output file name. To place the executable output in the file hello: PGI$ pgf95 -o hello.f Linking: PGI$ To execute the resulting program, simply type the filename at the command prompt and press the Return or Enter key on your keyboard: PGI$ hello hello PGI$ Command-line Syntax The command-line syntax, using pgf95 as an example, is: pgf95 [options] [path]filename [...] Where:Invoking the Command-level PGI Compilers 3 options is one or more command-line options, all of which are described in detail in Chapter 3, “Command Line Options”. Case is significant for options and their arguments. The compiler drivers recognize characters preceded by a hyphen (-) as command-line options. For example, the –Mlist option specifies that the compiler creates a listing file (in the text of this manual we show command-line options using a dash instead of a hyphen, for example – Mlist). In addition, the pgcpp command recognizes a group of characters preceded by a plus sign (+) as command-line options. The order of options and the filename is not fixed. That is, you can place options before and after the filename argument on the command line. However, the placement of some options is significant, for example the –l option. Note If two or more options contradict each other, the last one in the command line takes precedence. path is the pathname to the directory containing the file named by filename. If you do not specify path for a filename, the compiler uses the current directory. You must specify path separately for each filename not in the current directory. filename is the name of a source file, assembly-language file, object file, or library to be processed by the compilation system. You can specify more than one [path]filename. Command-line Options The command-line options control various aspects of the compilation process. For a complete alphabetical listing and a description of all the command-line options, refer to Chapter 3, “Command Line Options”.Getting Started 4 Fortran Directives and C/C++ Pragmas Fortran directives or C/C++ pragmas inserted in program source code allow you to alter the effects of certain command-line options and control various aspects of the compilation process for a specific routine or a specific program loop. For a complete alphabetical listing and a description of all the Fortran directives and C/C++ pragmas, refer to 5, “OpenMP Directives for Fortran”, 6, “OpenMP Pragmas for C and C++”, and 7, “Directives and Pragmas”. Filename Conventions The PGI compilers use the filenames that you specify on the command line to find and to create input and output files. This section describes the input and output filename conventions for the phases of the compilation process. Input Files You can specify assembly-language files, preprocessed source files, Fortran/C/C++ source files, object files, and libraries as inputs on the command line. The compiler driver determines the type of each input file by examining the filename extensions. The drivers use the following conventions: filename.f indicates a Fortran source file. filename.F indicates a Fortran source file that can contain macros and preprocessor directives (to be preprocessed). filename.FOR indicates a Fortran source file that can contain macros and preprocessor directives (to be preprocessed). filename.F95 indicates a Fortran 90/95 source file that can contain macros and preprocessor directives (to be preprocessed). filename.f90 indicates a Fortran 90/95 source file that is in freeform format. filename.f95 indicates a Fortran 90/95 source file that is in freeform format. filename.hpf indicates an HPF source file. filename.c indicates a C source file that can contain macros and preprocessor directives (to be preprocessed). filename.i indicates a pre-processed C or C++ source file.Filename Conventions 5 filename.C indicates a C++ source file that can contain macros and preprocessor directives (to be preprocessed). filename.cc indicates a C++ source file that can contain macros and preprocessor directives (to be preprocessed). filename.s indicates an assembly-language file. filename.o (Linux systems only) indicates an object file. filename.obj (Windows systems only) indicates an object file. filename.a (Linux systems only) indicates a library of object files. filename.lib (Windows systems only) indicates a library of object files. filename.so (Linux systems only) indicates a library of shared object files. filename.dll (Windows systems only) indicates a library of shared object files. The driver passes files with .s extensions to the assembler and files with .o, .so, .a and .lib extensions to the linker. Input files with unrecognized extensions, or no extension, are also passed to the linker. Files with a .F (Capital F) or .FOR suffix are first preprocessed by the Fortran compilers and the output is passed to the compilation phase. The Fortran preprocessor functions similar to cpp for C/C++ programs, but is built in to the Fortran compilers rather than implemented through an invocation of cpp. This ensures consistency in the pre-processing step regardless of the type or revision of operating system under which you’re compiling. Any input files not needed for a particular phase of processing are not processed. For example, if on the command line you use an assembly-language file (filename.s) and the –S option to stop before the assembly phase, the compiler takes no action on the assembly-language file. Processing stops after compilation and the assembler does not run (in this case compilation must have been completed in a previous pass which created the .s file). Refer to the following section, Output Files, for a description of the –S option. In addition to specifying primary input files on the command line, code within other files can be compiled as part of “include” files using the INCLUDE statement in a Fortran source file or the preprocessor #include directive in Fortran source files that use a .F extension or C and C++ source files.Getting Started 6 When linking a program with a library, the linker extracts only those library components that the program needs. The compiler drivers link in several libraries by default. For more information about libraries, refer to 8, “Libraries and Environment Variables”, . Output Files By default, an executable output file produced by one of the PGI compilers is placed in the file a.out (or, on Windows, a filename based on the name of the first source or object file on the command line). As shown in the preceding section, you can use the –o option to specify the output file name. If you use one of the options: –F (Fortran only), –P (C/C++ only), –S or –c, the compiler produces a file containing the output of the last phase that completes for each input file, as specified by the option supplied. The output file will be a preprocessed source file, an assembly-language file, or an unlinked object file respectively. Similarly, the –E option does not produce a file, but displays the preprocessed source file on the standard output. Using any of these options, the –o option is valid only if you specify a single input file. If no errors occur during processing, you can use the files created by these options as input to a future invocation of any of the PGI compiler drivers. The following table lists the stop after options and the output files that the compilers create when you use these options.Filename Conventions 7 Table 1-1: Stop after Options, Inputs and Outputs If you specify multiple input files or do not specify an object filename, the compiler uses the input filenames to derive corresponding default output filenames of the following form, where filename is the input filename without its extension: filename.f indicates a preprocessed file (if you compiled a Fortran file using the –F option). filename.l stindicates a listing file from the –Mlist option. filename.o indicates an object file from the –c option. filename.s indicates an assembly-language file from the –S option. Note Unless you specify otherwise, the destination directory for any output file is the current working directory. If the file exists in the destination directory, the compiler overwrites it. The following example demonstrates the use of output filename extensions. $ pgf95 -c proto.f proto1.F Option Stop after Input Output –E preprocessing Source files (must have .F extension for Fortran) preprocessed file to standard out –F preprocessing Source files (must have .F extension, this option is not valid for pgcc or pgcpp) preprocessed file – .f –P preprocessing Source files (this option is not valid for pgf77, pgf95 or pghpf) preprocessed file – .i –S compilation Source files or preprocessed files assembly-language file – .s –c assembly Source files, preprocessed files or assemblylanguage files unlinked object file – .o none linking Source files, preprocessed files, assembly-language files, object files or libraries executable files a.outGetting Started 8 This produces the output files proto.o and proto1.o, both of which are binary object files. Prior to compilation, the file proto1.F is pre-processed because it has a .F filename extension. Parallel Programming Using the PGI Compilers The PGI compilers support three styles of parallel programming: • Automatic shared-memory parallel programs compiled using the -Mconcur option to pgf77, pgf95, pgcc, or pgcpp — parallel programs of this variety can be run on shared-memory parallel (SMP) systems such as dual-core or multi-processor workstations. • OpenMP shared-memory parallel programs compiled using the -mp option to pgf77, pgf95, pgcc, or pgcpp — parallel programs of this variety can be run on SMP systems. Carefully coded userdirected parallel programs using OpenMP directives can often achieve significant speed-ups on dual-core workstations or large numbers of processors on SMP server systems. 5, “OpenMP Directives for Fortran” and 6, “OpenMP Pragmas for C and C++” contain complete descriptions of user-directed parallel programming. • Data parallel shared- or distributed-memory parallel programs compiled using the PGHPF High Performance Fortran compiler — parallel programs of this variety can be run on SMP workstations or servers, distributed-memory clusters of workstations, or clusters of SMP workstations or servers. Coding a data parallel version of an application can be more work than using OpenMP directives, but has the advantage that the resulting executable is usable on all types of parallel systems regardless of whether shared memory is available. See the PGHPF User’s Guide for a complete description of how to build and execute data parallel HPF programs. In this manual, the first two types of parallel programs are collectively referred to as SMP parallel programs. The third type is referred to as a data parallel program, or simply as an HPF program. Some newer CPUs incorporate two or more complete processor cores (functional units, registers, level 1 cache, level 2 cache, etc) on a single silicon die. These are referred to as multi-core processors. For purposes of HPF, threads, or OpenMP parallelism, these cores function as 2 or more distinct processors. However, the processing cores are on a single chip occupying a single socket on a system motherboard. For purposes of PGI software licensing, a multi-core processor is treated as a single CPU. Parallel Programming Using the PGI Compilers 9 Running SMP Parallel Programs When you execute an SMP parallel program, by default it will use only 1 processor. To run on more than one processor, set the NCPUS environment variable to the desired number of processors (subject to a maximum of 4 for PGI’s workstation-class products). You can set this environment variable by issuing the following command: % setenv NCPUS in a Windows command prompt window % setenv NCPUS in a shell command window under csh, or with % NCPUS=; export NCPUS in sh, ksh, or BASH command window. Note If you set NCPUS to a number larger than the number of physical processors, your program may execute very slowly. Running Data Parallel HPF Programs When you execute an HPF program, by default it will use only one processor. If you wish to run on more than one processor, use the -pghpf -np runtime option. For example, to compile and run the hello.f example defined above on one processor, you would issue the following commands: % pghpf -o hello hello.f Linking: % hello hello % To execute it on two processors, you would issue the following commands: % hello -pghpf -np 2 hello %Getting Started 10 Note If you specify a number larger than the number of physical processors, your program will execute very slowly. Note that you still only see a single “hello” printed to your screen. This is because HPF is a singlethreaded model, meaning that all statements execute with the same semantics as if they were running in serial. However, parallel statements or constructs operating on explicitly distributed data are in fact executed in parallel. The programmer must manually insert compiler directives to cause data to be distributed to the available processors. See the PGHPF User’s Guide and The High Performance Fortran Handbook for more details on constructing and executing data parallel programs on shared-memory or distributed-memory cluster systems using PGHPF. Using the PGI Compilers on Linux Linux Header Files The Linux system header files contain many GNU gcc extensions. Many of these extensions are supported. This should allow the PGCC C and C++ compilers to compile most programs compilable with the GNU compilers. A few header files not interoperable with the PGI compilers have been rewritten and are included in $PGI/linux86/include. These files are: sigset.h, asm/byteorder.h, stddef.h, asm/ posix_types.h and others. Also, PGI’s version of stdarg.h should support changes in newer versions of Linux. If you are using the PGCC C or C++ compilers, please make sure that the supplied versions of these include files are found before the system versions. This will happen by default unless you explicitly add a –I option that references one of the system include directories. Running Parallel Programs on Linux You may encounter difficulties running auto-parallel or OpenMP programs on Linux systems when the per-thread stack size is set to the default (2MB). If you have unexplained failures, please try setting the environment variable MPSTKZ to a larger value, such as 8MB. This can be accomplished with the command: % setenv MPSTKZ 8M in csh, or with % MPSTKZ=8M; export MPSTKZUsing the PGI Compilers on Windows 11 in bash, sh, or ksh. If your program is still failing, you may be encountering the hard 8 MB limit on main process stack sizes in Linux. You can work around the problem by issuing the command: % limit stacksize unlimited in csh, or % ulimit -s unlimited in bash, sh, or ksh. Using the PGI Compilers on Windows BASH Shell Environment On Windows platforms, the tools that ship with the PGI Workstation or PGI Server command-level compilers include a full-featured shell command environment. After installation, you should have a PGI icon on your Windows desktop. Double-left-click on this icon to cause an instance of the BASH command shell to appear on your screen. Working within BASH is very much like working within the sh or ksh shells on a Linux system, but in addition BASH has a command history feature similar to csh and several other unique features. Shell programming is fully supported. A complete BASH User’s Guide is available through the PGI online manual set. Select “PGI Workstation” under Start->Programs and double-leftclick on the documentation icon to see the online manual set. You must have a web browser installed on your system in order to read the online manuals. The BASH shell window is pre-initialized for usage of the PGI compilers and tools, so there is no need to set environment variables or modify your command path when the command window comes up. In addition to the PGI compiler commands referenced above, within BASH you have access to over 100 common commands and utilities, including but not limited to the following: vi emacs make tar / untar gzip / gunzip ftp sed grep / egrep / fgrep awk cat cksum cpGetting Started 12 If you are familiar with program development in a Linux environment, editing, compiling, and executing programs within BASH will be very comfortable. If you have not previously used such an environment, you should take time to familiarize yourself with either the vi or emacs editors and with makefiles. The emacs editor has an extensive online tutorial, which you can start by bringing up emacs and selecting the appropriate option under the pull-down help menu. You can get a thorough introduction to the construction and use of makefiles through the online Makefile User’s Guide. Windows Command Prompt The PGI Workstation entry in the Windows Start menu contains a submenu titled PGI Workstation Tools. This submenu contains a shortcut labeled PGI Command Prompt (32-bit). The shortcut is used to launch a Windows command shell using an environment pre-initialized for the use of the 32-bit PGI compilers and tools. On x64 systems, a second shortcut labeled PGI Command Prompt (64-bit) will also be present. This shortcut launches a Windows command shell using an environment pre-initialized for use of the 64-bit PGI compilers and tools. Using the PGI Compilers on SUA and SFU Subsystem for Unix Applications (SUA and SFU) Subsystem for Unix Applications (SUA) is a source-compatibility subsystem for running Unix applications on 32-bit nd 64-bit Windows server-class operating systems. PGI Workstation for Windows includes compilers and tools for SUA and its 32-bit-only predecessor, Services For Unix (SFU). SUA provides an operating system for POSIX processes. There is a package of support utilities available for download from Microsoft that provides a more complete Unix environment, including things like shells, scripting utilities, a telnet client, development tools, and so on. date diff du find kill ls more / less mv printenv / env rm / rmdir touch wcCustomizing the Compilers with siterc and User rc Files 13 SUA/SFU Header Files The SUA/SFU system header files contain numerous non-standard extensions. Many of these extensions are supported. This should allow the PGCC C and C++ compilers to compile most programs compilable with the GNU compilers. A few header files not interoperable with the PGI compilers have been rewritten and are included in $PGI/sua32/include or $PGI/sua64/include. These files are: stdarg.h, stddef.h, and others. If you are using the PGCC C or C++ compilers, please make sure that the supplied versions of these include files are found before the system versions. This will happen by default unless you explicitly add a –I option that references one of the system include directories. Running Parallel Programs on SUA and SFU You may encounter difficulties running auto-parallel or OpenMP programs on SUA/SFU systems when the per-thread stack size is set to the default (2MB). If you have unexplained failures, please try setting the environment variable MPSTKZ to a larger value, such as 8MB. This can be accomplished with the command: % setenv MPSTKZ 8M in csh, or with % MPSTKZ=8M; export MPSTKZ in bash, sh, or ksh. Customizing the Compilers with siterc and User rc Files The PGI compiler drivers utilize a file named siterc to enable site-specific customization of the behavior of the PGI compilers. The siterc file is located in the bin subdirectory of the PGI installation directory. Using siterc, you can control how the compiler drivers invoke the various components in the compilation tool chain. In addition to the siterc file, user rc files can reside in a given user’s home directory, as specified by the user’s HOME environment variable. On Linux and SUA these files are named .mypgf77rc, .mypgf90rc, .mypgccrc, .mypgcpprc, and .mypghpfrc, and can be used to control the respective PGI compilers. On native windows, these files are named mypgf77rc, mypgf90rc, mypgccrc, mypgcpprc, and mypghpfrc. All of these files are optional.Getting Started 14 Following are some examples that show how these rc files can be used to tailor a given installation for a particular purpose. Table 1-2: Examples of Using siterc and User rc Files Make the libraries found in /opt/ newlibs/64 available to all linux86-64 compilations Add the line: set SITELIB=/opt/newlibs/64; to /opt/pgi/linux86-64/6.2/bin/siterc Make the libraries found in /opt/ newlibs/32 available to all linux86 compilations. Add the line: set SITELIB=/opt/newlibs/32; to /opt/pgi/linux86/6.2/bin/siterc Add a new library path /opt/local/ fast to all linux86-64 compilations. Add the line: append SITELIB=/opt/local/fast;to /opt/pgi/linux86-64/6.2/bin/ siterc Make the include path /opt/acml/ include available to all compilations; –I/opt/acml/include. Add the line: set SITEINC=/opt/acml/include; to /opt/pgi/linux86/6.2/bin/siterc and/opt/pgi/linux86-64/6.2/ bin/siterc Change –Mmpi to link in/opt/ mympi/64/libmpix.a with linux86-64 compilations. Add the lines: set MPILIBDIR=/opt/mympi/64; set MPILIBNAME=mpix;to /opt/pgi/linux86-64/6.2/bin/siterc Have linux86-64 compilations always add–DIS64BIT –DAMD Add the line: set SITEDEF=IS64BIT AMD; to /opt/pgi/linux86-64/6.2/bin/siterc A user wishes to build an F90 executable for linux86-64 or linux86 that resolves PGI shared objects in the relative directory ./REDIST Add the line: set RPATH=./REDIST; to ~/.mypgf95rc. NOTE: this will onlyaffect the behavior of PGF95 for thegiven user.Overview of Optimization 15 2 Optimization & Parallelization Source code that is readable, maintainable, and produces correct results is not always organized for efficient execution. Normally, the first step in the program development process involves producing code that executes and produces the correct results. This first step usually involves compiling without much worry about optimization. After code is compiled and debugged, code optimization and parallelization become an issue. Invoking one of the PGI compiler commands with certain options instructs the compiler to generate optimized code. Optimization is not always performed since it increases compilation time and may make debugging difficult. However, optimization produces more efficient code that usually runs significantly faster than code that is not optimized. The compilers optimize code according to the specified optimization level. Using the –O, –Mvect, –Mipa and –Mconcur options, you can specify the optimization levels. In addition, several –M switches can be used to control specific types of optimization and parallelization. This chapter describes the optimization options and describes how to choose optimization options to use with the PGI compilers. Chapter 4, “Function Inlining”, describes how to use the function inlining options. Overview of Optimization In general, optimization involves using transformations and replacements that generate more efficient code. This is done by the compiler and involves replacements that are independent of the particular target processor’s architecture as well as replacements that take advantage of the x86 or x64 architecture, instruction set and registers. For the discussion in this and the following chapters, optimization is divided into the following categories: Local Optimization This optimization is performed on a block-by-block basis within a program’s basic blocks. A basic block is a sequence of statements, in which the flow of control enters at the beginning and leaves at the end without the possibility of branching, except at the end. The PGI compilers perform many types of local optimization including: algebraic identity removal, constant folding, common sub-expression elimination, pipelining, redundant load and store elimination, scheduling, strength reduction, and peephole optimizations.Optimization & Parallelization 16 Global Optimization This optimization is performed on a program unit over all its basic blocks. The optimizer performs control-flow and data-flow analysis for an entire program unit. All loops, including those formed by IFs and GOTOs are detected and optimized. Global optimization includes: constant propagation, copy propagation, dead store elimination, global register allocation, invariant code motion, and induction variable elimination. Loop Optimization: Unrolling, Vectorization,and Parallelization The performance of certain classes of loops may be improved through vectorization or unrolling options. Vectorization transforms loops to improve memory access performance and make use of packed SSE instructions which perform the same operation on multiple data items concurrently. Unrolling replicates the body of loops to reduce loop branching overhead and provide better opportunities for local optimization, vectorization and scheduling of instructions. Performance for loops on systems with multiple processors may also improve using the parallelization features of the PGI compilers. Inter-Procedural Analysis and Optimization (IPA) Interprocedural analysis allows use of information across function call boundaries to perform optimizations that would otherwise be unavailable. For example, if the actual argument to a function is in fact a constant in the caller, it may be possible to propagate that constant into the callee and perform optimizations that are not valid if the dummy argument is treated as a variable. A wide range of optimizations are enabled or improved by using IPA, including but not limited to data alignment optimizations, argument removal, constant propagation, pointer disambiguation, pure function detection, F90/F95 array shape propagation, data placement, vestigial function removal, automatic function inlining, inlining of functions from pre-compiled libraries, and interprocedural optimization of functions from pre-compiled libraries. Function Inlining This optimization allows a call to a function to be replaced by a copy of the body of that function. This optimization will sometimes speed up execution by eliminating the function call and return overhead. Function inlining may also create opportunities for other types of optimization. Function inlining is not always beneficial. When used improperly it may increase code size and generate less efficient code. Getting Started with Optimizations 17 Profile-Feedback Optimization (PFO) Profile-feedback optimization makes use of information from a trace file produced by specially instrumented executables which capture and save information on branch frequency, function and subroutine call frequency, semi-invariant values, loop index ranges, and other input data dependent information that can only be collected dynamically during execution of a program. By definition, use of profile-feedback optimization is a two-phase process: compilation and execution of a speciallyinstrumented executable, followed by a subsequent compilation which reads a trace file generated during the first phase and uses the information in the trace file to guide compiler optimizations. Getting Started with Optimizations Your first concern should be getting your program to execute and produce correct results. To get your program running, start by compiling and linking without optimization. Use the optimization level –O0 or select –g to perform minimal optimization. At this level, you will be able to debug your program easily and isolate any coding errors exposed during porting to x86 or x64 platforms. If you want to get started quickly with optimization, a good set of options to use with any of the PGI compilers is –fast –Mipa=fast. For example: $ pgf95 -fast -Mipa=fast prog.f For all of the PGI Fortran, C, and C++ compilers, this option will generally produce code that is welloptimized without the possibility of significant slowdowns due to pathological cases. The -fast option is an aggregate option that includes a number of individual PGI compiler options; which PGI compiler options are included depends on the target for which compilation is performed. The –Mipa=fast option invokes interprocedural analysis including several IPA suboptions. For C++ programs, add -Minline=levels:10 --no_exceptions: $ pgcpp -fast -Mipa=fast -Minline=levels:10 --no_exceptions prog.cc Note: a C++ program compiled with --no_execptions will fail if the program uses exception handling. By experimenting with individual compiler options on a file-by-file basis, further significant performance gains can sometimes be realized. However, individual optimizations can sometimes cause slowdowns depending on coding style and must be used carefully to ensure performance improvements result. In addition to –fastsse, the optimization flags most likely to further improve performance are – O3, –Mpfi/–Mpfo, –Minline, and on targets with multiple processors –Mconcur. Optimization & Parallelization 18 In addition, the –Msafeptr option can significantly improve performance of C/C++ programs in which there is known to be no pointer aliasing. However, for obvious reasons this command-line option must be used carefully. Three other options which are extremely useful are –help, –Minfo, and –dryrun. You can see a specification of any command-line option by invoking any of the PGI compilers with –help in combination with the option in question, without specifying any input files. For example: $ pgf95 -help -fast Reading rcfile /usr/pgi_rel/linux86-64/6.0/bin/.pgf95rc -fastsse == -fast -Mvect=sse -Mcache_align -Mflushz -fast Common optimizations: -O2 -Munroll=c:1 -Mnoframe -Mlre . . . Or to see the full functionality of –help itself, which can return information on either an individual option or groups of options by type: $ pgf95 -help -help Reading rcfile /usr/pgi_rel/linux86-64/6.0/bin/.pgf95rc -help[=groups|asm|debug|language|linker|opt|other|overall| phase|prepro|suffix|switch|target|variable] The –Minfo option can be used to display compile-time optimization listings. When this option is used, the PGI compilers will issue informational messages to stdout as compilation proceeds. From these messages, you can determine which loops are optimized using unrolling, SSE instructions, vectorization, parallelization, interprocedural optimizations and various miscellaneous optimizations. You can also see where and whether functions are inlined. The –Mneginfo option can be used to display informational messages listing why certain optimizations are inhibited. The –dryrun option can be useful as a diagnostic tool if you need to see the steps used by the compiler driver to pre-process, compile, assemble and link in the presence of a given set of command line inputs. When you specify the –dryrun option, these steps will be printed to stdout but will not actually be performed. For example, this allows inspection of the default and user-specified libraries that are searched during the link phase, and the order in which they are searched by the linker. Local and Global Optimization using -O 19 The remainder of this chapter describes the –O options, the loop unroller option –Munroll, the vectorizer option –Mvect, the auto-parallelization option –Mconcur, and the inter-procedural analysis optimization –Mipa, and the profile-feedback instrumentation (–Mpfi) and optimization (–Mpfo) options. Usually, you should be able to get very near optimal compiled performance using some combination of these switches. The following overview will help if you are just getting started with one of the PGI compilers, or wish to experiment with individual optimizations. Complete specifications of each of these options are listed in Chapter 3, “Command Line Options” . The chapters that follow provide more detailed information on other –M options that control specific optimizations, including function inlining. Explicit parallelization through the use of OpenMP directives or pragmas is invoked using the –mp option, described in detail in Chapter 5, “OpenMP Directives for Fortran” and Chapter 6, “OpenMP Pragmas for C and C++”. Local and Global Optimization using -O Using the PGI compiler commands with the –Olevel option, you can specify any of the following optimization levels (the capital O is for Optimize): –O0 level-zero specifies no optimization. A basic block is generated for each language statement. –O1 level-one specifies local optimization. Scheduling of basic blocks is performed. Register allocation is performed. –O2 level-two specifies global optimization. This level performs all level-one local optimization as well as level-two global optimization. –O3 level-three specifies aggressive global optimization. This level performs all level-one and level-two optimizations and enables more aggressive hoisting and scalar replacement optimizations that may or may not be profitable. –O4 level-four performs all level-one, level-two, and level-three optimizations and enables hoisting of guarded invariant floating point expressions. Level-zero optimization specifies no optimization (–O0). At this level, the compiler generates a basic block for each statement. This level is useful for the initial execution of a program. Performance will almost always be slowest using this optimization level. Level-zero is useful for debugging since there is a direct correlation between the program text and the code generated.Optimization & Parallelization 20 Level-one optimization specifies local optimization (–O1). The compiler performs scheduling of basic blocks as well as register allocation. This optimization level is a good choice when the code is very irregular; that is it contains many short statements containing IF statements and the program does not contain loops (DO or DO WHILE statements). For certain types of code, this optimization level may perform better than level-two (–O2) although this case rarely occurs. The PGI compilers perform many different types of local optimizations, including but not limited to: • Algebraic identity removal • Constant folding • Common subexpression elimination • Local register optimization • Peephole optimizations • Redundant load and store elimination • Strength reductions Level-two optimization (–O2 or –O) specifies global optimization. The –nfast option generally will specify global optimization; however, the –nfast switch will vary from release to release depending on a reasonable selection of switches for any one particular release. The –O or –O2 level performs all levelone local optimizations as well as global optimizations. Control flow analysis is applied and global registers are allocated for all functions and subroutines. Loop regions are given special consideration. This optimization level is a good choice when the program contains loops, the loops are short, and the structure of the code is regular. The PGI compilers perform many different types of global optimizations, including but not limited to: • Branch to branch elimination • Constant propagation • Copy propagation • Dead store elimination • Global register allocation • Invariant code motionLocal and Global Optimization using -O 21 • Induction variable elimination You select the optimization level on the command line. For example, level-two optimization results in global optimization, as shown below: $ pgf95 -O2 prog.f Specifying –O on the command-line without a level designation is equivalent to –O2. The default optimization level changes depending on which options you select on the command line. For example, when you select the –g debugging option, the default optimization level is set to level-zero (–O0). However, you can use the -gopt option to generate debug information without perturbing optimization if you need to debug optimized code. Refer to Section 2.8, Default Optimization Levels, for a description of the default levels. As noted above, the –nfast option includes –O2 on all x86 and x64 targets. If you wish to override this with –O3 while maintaining all other elements of –nfast, simply compile as follows: $ pgf95 -nfast -O3 prog.f Note Beginning in release 7.0, the –fast became synonymous with the –fastsse option, and the optimizations performed by –fast in previous releases were placed under the –nfast option. Use –nfast for older x86 processors, as described in the following section. Scalar SSE Code Generation For all processors prior to Intel Pentium 4 and AMD Opteron/Athlon64, for example Intel Pentium III and AMD AthlonXP/MP processors, scalar floating-point arithmetic as generated by the PGI Workstation compilers is performed using x87 floating-point stack instructions. With the advent of SSE/SSE2 instructions on Intel Pentium 4/Xeon and AMD Opteron/Athlon64, it is possible to perform all scalar floating-point arithmetic using SSE/SSE2 instructions. In most cases, this is beneficial from a performance standpoint. The default on 32-bit Intel Pentium II/III (–tp p6, –tp piii, etc) or AMD AthlonXP/MP (–tp k7) is to use x87 instructions for scalar floating-point arithmetic. The default on Intel Pentium 4/Xeon or Intel EM64T running a 32-bit operating system (–tp p7), AMD Opteron/Athlon64 running a 32-bit operating system (–tp k8-32), or AMD Opteron/Athlon64 or Intel EM64T processors running a 64-bit operating Optimization & Parallelization 22 system (–tp k8-64 and –tp p7-64 respectively) is to use SSE/SSE2 instructions for scalar floating-point arithmetic. The only way to override this default on AMD Opteron/Athlon64 or Intel EM64T processors running a 64-bit operating system is to specify an older 32-bit target (for example –tp k7 or –tp piii). Note that there can be significant arithmetic differences between calculations performed using x87 instructions versus SSE/SSE2. By default, all floating-point data is promoted to IEEE 80-bit format when stored on the x87 floating-point stack, and all x87 operations are performed register-to-register in this same format. Values are converted back to IEEE 32-bit or IEEE 64-bit when stored back to memory (for REAL/float and DOUBLE PRECISION/double data respectively). The default precision of the x87 floatingpoint stack can be reduced to IEEE 32-bit or IEEE 64-bit globally by compiling the main program with the –pc {32 | 64} option to the PGI Workstation compilers, which is described in detail in Chapter 3, “Command Line Options”. However, there is no way to ensure that operations performed in mixed precision will match those produced on a traditional load-store RISC/UNIX system which implements IEEE 64-bit and IEEE 32-bit registers and associated floating-point arithmetic instructions. In contrast, arithmetic results produced on Intel Pentium 4/Xeon, AMD Opteron/Athlon64 or Intel EM64T processors will usually closely match or be identical to those produced on a traditional RISC/ UNIX system if all scalar arithmetic is performed using SSE/SSE2 instructions. You should keep this in mind when porting applications to and from systems which support both x87 and full SSE/SSE2 floating-point arithmetic. Many subtle issues can arise which affect your numerical results, sometimes to several digits of accuracy. Loop Unrolling using -Munroll This optimization unrolls loops, executing multiple instances of the loop during each iteration. This reduces branch overhead, and can improve execution speed by creating better opportunities for instruction scheduling. A loop with a constant count may be completely unrolled or partially unrolled. A loop with a non-constant count may also be unrolled. A candidate loop must be an innermost loop containing one to four blocks of code. The following shows the use of the –Munroll option: $ pgf95 -Munroll prog.f The –Munroll option is included as part of –fast and –fastsse on all x86 and x64 targets. The loop unroller expands the contents of a loop and reduces the number of times a loop is executed. Branching overhead is reduced when a loop is unrolled two or more times, since each iteration of the unrolled loop corresponds to two or more iterations of the original loop; the number of branch instructions executed is proportionately reduced. When a loop is unrolled completely, the loop’s branch overhead is eliminated altogether.Loop Unrolling using -Munroll 23 Loop unrolling may be beneficial for the instruction scheduler. When a loop is completely unrolled or unrolled two or more times, opportunities for improved scheduling may be presented. The code generator can take advantage of more possibilities for instruction grouping or filling instruction delays found within the loop. Examples 2-1 and 2-2 show the effect of code unrolling on a segment that computes a dot product. Example 2-1: Dot Product Code REAL*4 A(100), B(100), Z INTEGER I DO I=1, 100 Z = Z + A(i) * B(i) END DO END Example 2-2: Unrolled Dot Product Code REAL*4 A(100), B(100), Z INTEGER I DO I=1, 100, 2 Z = Z + A(i) * B(i) Z = Z + A(i+1) * B(i+1) END DO END Using the –Minfo option, the compiler informs you when a loop is being unrolled. For example, a message indicating the line number, and the number of times the code is unrolled, similar to the following will display when a loop is unrolled: dot: 5, Loop unrolled 5 times Using the c: and n: sub-options to –Munroll, or using –Mnounroll, you can control whether and how loops are unrolled on a file-by-file basis. Using directives or pragmas as specified in Chapter 7, “Directives and Pragmas”, you can precisely control whether and how a given loop is unrolled. See Chapter 3, “Command Line Options”, for a detailed description of the –Munroll option.Optimization & Parallelization 24 Vectorization using -Mvect The –Mvect option is included as part of –fastsse on all x86 and x64 targets. If your program contains computationally intensive loops, the –Mvect option may be helpful. If in addition you specify –Minfo, and your code contains loops that can be vectorized, the compiler reports relevant information on the optimizations applied. When a PGI compiler command is invoked with the –Mvect option, the vectorizer scans code searching for loops that are candidates for high-level transformations such as loop distribution, loop interchange, cache tiling, and idiom recognition (replacement of a recognizable code sequence, such as a reduction loop, with optimized code sequences or function calls). When the vectorizer finds vectorization opportunities, it internally rearranges or replaces sections of loops (the vectorizer changes the code generated; your source code’s loops are not altered). In addition to performing these loop transformations, the vectorizer produces extensive data dependence information for use by other phases of compilation and detects opportunities to use vector or packed Streaming SIMD Extensions (SSE) instructions on processors where these are supported. The –Mvect option can speed up code which contains well-behaved countable loops which operate on large REAL, REAL*4, REAL*8, INTEGER*4, COMPLEX or COMPLEX DOUBLE arrays in Fortran and their C/C++ counterparts. However, it is possible that some codes will show a decrease in performance when compiled with –Mvect due to the generation of conditionally executed code segments, inability to determine data alignment, and other code generation factors. For this reason, it is recommended that you check carefully whether particular program units or loops show improved performance when compiled with this option enabled. Vectorization Sub-options The vectorizer performs high-level loop transformations on countable loops. A loop is countable if the number of iterations is set only before loop execution and cannot be modified during loop execution. Some of the vectorizer transformations can be controlled by arguments to the –Mvect command line option. The following sections describe the arguments that affect the operation of the vectorizer. In addition, some of these vectorizer operations can be controlled from within code using directives and pragmas. For details on the use of directives and pragmas, refer to Chapter 7, “Directives and Pragmas” . The vectorizer performs the following operations: • Loop interchange • Loop splittingVectorization using -Mvect 25 • Loop fusion • Memory-hierarchy (cache tiling) optimizations • Generation of SSE instructions on processors where these are supported • Generation of prefetch instructions on processors where these are supported • Loop iteration peeling to maximize vector alignment • Alternate code generation By default, –Mvect without any sub-options is equivalent to: -Mvect=assoc,cachesize:262144 This enables the options for nested loop transformation and various other vectorizer options. These defaults may vary depending on the target system. Assoc Option The option –Mvect=assoc instructs the vectorizer to perform associativity conversions that can change the results of a computation due to roundoff error (–Mvect=noassoc disables this option). For example, a typical optimization is to change one arithmetic operation to another arithmetic operation that is mathematically correct, but can be computationally different and generate faster code. This option is provided to enable or disable this transformation, since roundoff error for such associativity conversions may produce unacceptable results. Cachesize Option The option –Mvect=cachesize:n instructs the vectorizer to tile nested loop operations assuming a data cache size of n bytes. By default, the vectorizer attempts to tile nested loop operations, such as matrix multiply, using multi-dimensional strip-mining techniques to maximize re-use of items in the data cache. SSE Option The option –Mvect=sse instructs the vectorizer to automatically generate packed SSE, SSE2 (streaming SIMD extensions) and prefetch instructions when vectorizable loops are encountered. SSE instructions, first introduced on Pentium III and AthlonXP processors, operate on single-precision floating-point data, and hence apply only to vectorizable loops that operate on single-precision floating-point data. Optimization & Parallelization 26 SSE2 instructions, first introduced on Pentium 4, Xeon and Opteron processors, operate on doubleprecision floating-point data. Prefetch instructions, first introduced on Pentium III and AthlonXP processors, can be used to improve the performance of vectorizable loops that operate on either 32-bit or 64-bit floating-point data. See table P-2 for a concise list of processors that support SSE, SSE2 and prefetch instructions. Note Programs units compiled with –Mvect=sse will not execute on Pentium, Pentium Pro, Pentium II or first generation AMD Athlon processors. They will only execute correctly on Pentium III, Pentium 4, Xeon, EM64T, AthlonXP, Athlon64 and Opteron systems running an SSE-enabled operating system. Prefetch Option The option –Mvect=prefetch instructs the vectorizer to automatically generate prefetch instructions when vectorizable loops are encountered, even in cases where SSE or SSE2 instructions are not generated. Usually, explicit prefetching is not necessary on Pentium 4, Xeon and Opteron because these processors support hardware prefetching; nonetheless, it sometimes can be worthwhile to experiment with explicit prefetching. Prefetching can be controlled on a loop-by-loop level using prefetch directives, which are described in detail in “Prefetch Directives” on page 194. Note Program units compiled with –Mvect=prefetch will not execute correctly on Pentium, Pentium Pro, or Pentium II processors. They will execute correctly only on Pentium III, Pentium 4, Xeon, EM64T, AthlonXP, Athlon64 or Opteron systems. In addition, the prefetchw instruction is only supported on AthlonXP, Athlon64 or Opteron systems and can cause instruction faults on non-AMD processors. For this reason, the PGI compilers do not generate prefetchw instructions by default on any target. In addition to these sub-options to –Mvect, several other sub-options are supported. See the description of –Mvect in Chapter 3, “Command Line Options”, for a detailed description of all available sub-options. Vectorization Example Using SSE/SSE2 Instructions One of the most important vectorization options is –Mvect=sse. This section contains an example of the use and potential effects of –Mvect=sse.Vectorization using -Mvect 27 When the compiler switch –Mvect=sse is used, the vectorizer in the PGI Workstation compilers automatically uses SSE and SSE2 instructions where possible when targeting processors where these are supported. This capability is supported by all of the PGI Fortran, C and C++ compilers. See table P-2 for a complete specification of which x86 and x64 processors support SSE and SSE2 instructions. Using – Mvect=sse, performance improvements of up to two times over equivalent scalar code sequences are possible. In the program in Example 2-3, “Vector operation using SSE instructions”, the vectorizer recognizes the vector operation in subroutine 'loop' when the compiler switch –Mvect=sse is used. This example shows the compilation, informational messages, and runtime results using the SSE instructions on an AMD Opteron processor-based system, along with issues that affect SSE performance. First note that the arrays in Example 2-3 , “Vector operation using SSE instructions” are single-precision and that the vector operation is done using a unit stride loop. Thus, this loop can potentially be vectorized using SSE instructions on any processor that supports SSE or SSE2 instructions. SSE operations can be used to operate on pairs of single-precision floating-point numbers, and do not apply to double-precision floating-point numbers. SSE2 instructions can be used to operate on quads of singleprecision floating-point numbers or on pairs of double-precision floating-point numbers. Loops vectorized using SSE or SSE2 instructions operate much more efficiently when processing vectors that are aligned to a cache-line boundary. You can cause unconstrained data objects of size 16 bytes or greater to be cache-aligned by compiling with the –Mcache_align switch. An unconstrained data object is a data object that is not a common block member and not a member of an aggregate data structure. Note In order for stack-based local variables to be properly aligned, the main program or function must be compiled with –Mcache_align. The –Mcache_align switch has no effect on the alignment of Fortran allocatable or automatic arrays. If you have arrays that are constrained, for example vectors that are members of Fortran common blocks, you must specifically pad your data structures to ensure proper cache alignment; –Mcache_align causes only the beginning address of each common block to be cache-aligned. The following examples show results of compiling the example code with and without –Mvect=sse.Optimization & Parallelization 28 Example 2-3: Vector operation using SSE instructions program vector_op parameter (N = 9999) real*4 x(n),y(n),z(n),w(n) do i = 1,n y(i) = i z(i) = 2*i w(i) = 4*i enddo do j = 1, 200000 call loop(x,y,z,w,1.0e0,n) enddo print*,x(1),x(771),x(3618),x(6498),x(9999) end subroutine loop(a,b,c,d,s,n) integer i,n real*4 a(n),b(n),c(n),d(n),s do i = 1,n a(i) = b(i) + c(i) - s * d(i) enddo end Assume the above program is compiled as follows: % pgf95 -fast -Minfo vadd.f vector_op: 4, Loop unrolled 4 times loop: 18, Loop unrolled 4 times Following is the result if the generated executable is run and timed on a standalone AMD Opteron 2.2 Ghz system: % /bin/time a.out -1.000000 -771.000 -3618.000 -6498.00 -9999.00 5.15user 0.00system 0:05.16 elapsed 99%CPU Now, recompile with SSE vectorization enabled:Vectorization using -Mvect 29 % pgf95 -fast –Mvect=sse -Minfo vadd.f vector_op: 4, Unrolling inner loop 8 times Loop unrolled 7 times (completely unrolled) loop: 18, Generating vector sse code for inner loop Generated 3 prefetch instructions for this loop Note the informational message indicating that the loop has been vectorized and SSE instructions have been generated. The second part of the informational message notes that prefetch instructions have been generated for 3 loads to minimize latency of transfers of data from main memory. Executing again, you should see results similar to the following: % /bin/time a.out -1.000000 -771.000 -3618.00 -6498.00 -9999.0 3.55user 0.00system 0:03.56elapsed 99%CPU The result is a speed-up of 45% over the equivalent scalar (i.e. non-SSE) version of the program. Speedup realized by a given loop or program can vary widely based on a number of factors: • Performance improvement using vector SSE or SSE2 instructions is most effective when the vectors of data are resident in the data cache. • If data is aligned properly, performance will be better in general than when using vector SSE operations on unaligned data. • If the compiler can guarantee that data is aligned properly, even more efficient sequences of SSE instructions can be generated. • SSE2 vector instructions can operate on 4 single-precision elements concurrently, but only 2 double-precision elements. As a result, the efficiency of loops that operate on single-precision data can be higher. Note Compiling with –Mvect=sse can result in numerical differences from the executables generated with less optimization. Certain vectorizable operations, for example dot products, are sensitive to order of operations and the associative transformations necessary to enable vectorization (or parallelization).Optimization & Parallelization 30 Auto-Parallelization using -Mconcur With the -Mconcur option the compiler scans code searching for loops that are candidates for autoparallelization. –Mconcur must be used at both compile-time and link-time. When the parallelizer finds opportunities for auto-parallelization, it parallelizes loops and you are informed of the line or loop being parallelized if the -Minfo option is present on the compile line. See Chapter 3, “Command Line Options”, for a complete specification of -Mconcur. A loop is considered parallelizable if doesn't contain any cross-iteration data dependencies. Crossiteration dependencies from reductions and expandable scalars are excluded from consideration, enabling more loops to be parallelizable. In general, loops with calls are not parallelized due to unknown side effects. Also, loops with low trip counts are not parallelized since the overhead in setting up and starting a parallel loop will likely outweigh the potential benefits. In addition, the default is to not parallelize innermost loops, since these often by definition are vectorizable using SSE instructions and it is seldom profitable to both vectorize and parallelize the same loop, especially on multi-core processors. Compiler switches and directives are available to let you override most of these restrictions on auto-parallelization. Auto-parallelization Sub-options The parallelizer performs various operations that can be controlled by arguments to the –Mconcur command line option. The following sections describe these arguments that affect the operation of the vectorizer. In addition, these vectorizer operations can be controlled from within code using directives and pragmas. For details on the use of directives and pragmas, refer to Chapter 7, “Directives and Pragmas”. By default, –Mconcur without any sub-options is equivalent to: -Mconcur=dist:block This enables parallelization of loops with blocked iteration allocation across the available threads of execution. These defaults may vary depending on the target system. Auto-Parallelization using -Mconcur 31 Altcode Option The option –Mconcur=altcode instructs the parallelizer to generate alternate serial code for parallelized loops. If altcode is specified without arguments, the parallelizer determines an appropriate cutoff length and generates serial code to be executed whenever the loop count is less than or equal to that length. If altcode:n is specified, the serial altcode is executed whenever the loop count is less than or equal to n. If noaltcode is specified, no alternate serial code is generated. Dist Option The option –Mconcur=dist:{block|cyclic} option specifies whether to assign loop iterations to the available threads in blocks or in a cyclic (round-robin) fashion. Block distribution is the default. If cyclic is specified, iterations are allocated to processors cyclically. That is, processor 0 performs iterations 0, 3, 6, etc.; processor 1 performs iterations 1, 4, 7, etc.; and processor 2 performs iterations 2, 5, 8, etc. Cncall Option The option –Mconcur=cncall specifies that it is safe to parallelize loops that contain subroutine or function calls. By default, such loops are excluded from consideration for auto-parallelization. Also, no minimum loop count threshold must be satisfied before parallelization will occur, and last values of scalars are assumed to be safe. The environment variable NCPUS is checked at runtime for a parallel program. If NCPUS is set to 1, a parallel program runs serially, but will use the parallel routines generated during compilation. If NCPUS is set to a value greater than 1, the specified number of processors will be used to execute the program. Setting NCPUS to a value exceeding the number of physical processors can produce inefficient execution. Executing a program on multiple processors in an environment where some of the processors are being time-shared with another executing job can also result in inefficient execution. As with the vectorizer, the -Mconcur option can speed up code if it contains well-behaved countable loops and/or computationally intensive nested loops that operate on arrays. However, it is possible that some codes will show a decrease in performance on multi-processor systems when compiled with -Mconcur due to parallelization overheads, memory bandwidth limitations in the target system, false-sharing of cache lines, or other architectural or code-generation factors. For this reason, it is recommended that you check carefully whether particular program units or loops show improved performance when compiled using this option.Optimization & Parallelization 32 If the compiler is not able to successfully auto-parallelize your application, you should refer to Chapter 5, “OpenMP Directives for Fortran”, or Chapter 6, “OpenMP Pragmas for C and C++”, to see if insertion of explicit parallelization directives or pragmas and use of the –mp compiler option enables the application to run in parallel. Loops That Fail to Parallelize In spite of the sophisticated analysis and transformations performed by the compiler, programmers will often note loops that are seemingly parallel, but are not parallelized. In this subsection, we’ll look at some examples of common situations where parallelization does not occur. Innermost Loops As noted earlier in this chapter, the PGI compilers will not parallelize innermost loops by default, because it is usually not profitable. You can override this default using the command-line option – Mconcur=innermost. Timing Loops Often, loops will occur in programs that are similar to timing loops. The outer loop in the following example is one such loop. do 1 j = 1, 2 do 1 i = 1, n a(i) = b(i) + c(i) 1continue The outer loop above is not parallelized because the compiler detects a cross-iteration dependence in the assignment to a(i). Suppose the outer loop were parallelized. Then both processors would simultaneously attempt to make assignments into a(1:n). Now in general the values computed by each processor for a(1:n) will differ, so that simultaneous assignment into a(1:n) will produce values different from sequential execution of the loops. In this example, values computed for a(1:n) don’t depend on j, so that simultaneous assignment by both processors will not yield incorrect results. However, it is beyond the scope of the compilers’ dependence analysis to determine that values computed in one iteration of a loop don’t differ from values computed in another iteration. So the worst case is assumed, and different iterations of the outer loop are assumed to compute different values for a(1:n). Is this assumption too pessimistic? If j doesn’t occur anywhere Auto-Parallelization using -Mconcur 33 within a loop, the loop exists only to cause some delay, most probably to improve timing resolution. And, it’s not usually valid to parallelize timing loops; to do so would distort the timing information for the inner loops. Scalars Quite often, scalars will inhibit parallelization of non-innermost loops. There are two separate cases that present problems. In the first case, scalars appear to be expandable, but appear in non-innermost loops, as in the following example. do 1 j = 1, n x = b(j) do 1 i = 1, n a(i,j) = x + c(i,j) 1 continue There are a number of technical problems to be resolved in order to recognize expandable scalars in non-innermost loops. Until this generalization occurs, scalars like x above will inhibit parallelization of loops in which they are assigned. In the following example, scalar k is not expandable, and it is not an accumulator for a reduction. k = 1 do 3 i = 1, n do 1 j = 1, n 1 a(j,i) = b(k) * x k = i 2if (i .gt. n/2) k = n - (i - n/2) 3 continue If the outer loop is parallelized, conflicting values will be stored into k by the various processors. The variable k cannot be made local to each processor because the value of k must remain coherent among the processors. It is possible the loop could be parallelized if all assignments to k are placed in critical sections. However, it is not clear where critical sections should be introduced because in general the value for k could depend on another scalar (or on k itself), and code to obtain the value of other scalars must reside in the same critical section. In the example above, the assignment to k within a conditional at label 2 prevents k from being recognized as an induction variable. If the conditional statement at label 2 is removed, k would be an induction variable whose value varies linearly with j, and the loop could be parallelized.Optimization & Parallelization 34 Scalar Last Values During parallelization, scalars within loops often need to be privatized; that is, each execution thread will have its own independent copy of the scalar. Problems can arise if a privatized scalar is accessed outside the loop. For example, consider the following loop: for (i = 1; i 5.0 ) t = x[i]; } v = t; The value of t may not be computed on the last iteration of the loop. Normally, if a scalar is assigned within a loop and used following the loop, the PGI compilers save the last value of the scalar. However, if the loop is parallelized and the scalar is not assigned on every iteration, it may be difficult (without resorting to costly critical sections) to determine on what iteration t is last assigned. Analysis allows the compiler to determine that a scalar is assigned on each iteration and hence that the loop is safe to parallelize if the scalar is used later. For example: for ( i = 1; i < n; i++){ if ( x[i] > 0.0 ) { t = 2.0; } else { t = 3.0; y[i] = ...t; } } v = t where t is assigned on every iteration of the loop. However, there are cases where a scalar may be privatizable, but if it is used after the loop, it is unsafe to parallelize. Examine this loop: for ( i = 1; i < N; i++ ){ if( x[i] > 0.0 ){ t = x[i]; ... ...Processor-Specific Optimization and the Unified Binary 35 y[i] = ...t; } } v = t; where each use of t within the loop is reached by a definition from the same iteration. Here t is privatizable, but the use of t outside the loop may yield incorrect results since the compiler may not be able to detect on which iteration of the parallelized loop t is last assigned. The compiler detects the above cases. Where a scalar is used after the loop but is not defined on every iteration of the loop, parallelization will not occur. When the programmer knows that the scalar is assigned on the last iteration of the loop, the programmer may use a directive or pragma to let the compiler know the loop is safe to parallelize. The Fortran directive which tells the compiler that for a given loop the last value computed for all scalars make it safe to parallelize the loop is: cpgi$l safe_lastval In addition, a command-line option, –Msafe_lastval, provides this information for all loops within the routines being compiled (essentially providing global scope). Processor-Specific Optimization and the Unified Binary Different processors have subtle and not-so-subtle differences in hardware features such as instruction sets and cache size. The compilers make architecture-specific decisions about such things as instruction selection, instruction scheduling, and vectorization. By default, the PGI compilers produce code specifically targeted to the type of processor on which the compilation is performed. In particular, the default is to use all supported instructions wherever possible when compiling on a given system. As a result, executables created on a given system may not be useable on previous generation systems (for example, executables created on a Pentium 4 may fail to execute on a Pentium III or Pentium II). The – tp option can be used to control this behavior by specifying the processor or processors with which the generated code is compatible. See “-tp [,target...]” on page 111 for more information. The 64-bit PGI compilers have the capability of generating unified binaries, which provide a lowoverhead means for generating a single executable that is compatibie with and gets good performance on more than one hardware platform. Executable size is automatically controlled via unified binary culling. Only those functions and subroutines where the target affects the generated code will have unique binary images, resulting in a code-size savings of 10-90% compared to generating full copies of code for each target.Optimization & Parallelization 36 Programs can use PGI Unified Binary even if all of the object files and libraries are not compiled as unified binaries. PGI Unified Binary object files can be use to create programs or libraries like any other object file. No special start up code is needed; support is linked in from the PGI libraries. The -Mpfi option disables generation of PGI Unified Binary. Instead, the default target auto-detect rules for the host are used to select the target processor. Inter-Procedural Analysis and Optimization using –Mipa The PGI Fortran, C and C++ compilers use interprocedural analysis (IPA) that results in minimal changes to makefiles and the standard edit-build-run application development cycle. Other than adding –Mipa to the command line, no other changes are required. For reference and background, the process of building a program without IPA is described below, followed by the minor modifications required to use IPA with the PGI compilers. While the PGCC compiler is used here to show how IPA works, similar capabilities apply to each of the PGI Fortran, C and C++ compilers. Note that the examples use Linux file naming conventions. On Windows, ‘.o’ files would be ‘.obj’ files, and ‘a.out’ files would be ‘.exe’ files. Building a Program Without IPA – Single Step Using the PGCC command-level compiler driver, three (for example) source files can be compiled and linked into a single executable with one command: % pgcc -o a.out file1.c file2.c file3.c In actuality, the pgcc driver executes several steps to produce the assembly code and object files corresponding to each source file, and subsequently to link the object files together into a single executable file. Thus, the command above is roughly equivalent to the following commands performed individually: % pgcc -S -o file1.s file1.c % as -o file1.o file1.s % pgcc -S -o file2.s file2.c % as -o file2.o file2.s % pgcc -S -o file3.s file3.c % as -o file3.o file3.s % pgcc -o a.out file1.o file2.o file3.o If any of the three source files is edited, the executable can be rebuilt with the same command line:Inter-Procedural Analysis and Optimization using –Mipa 37 % pgcc -o a.out file1.c file2.c file3.c This always works as intended, but has the side-effect of recompiling all of the source files, even if only one has changed. For applications with a large number of source files, this can be time-consuming and inefficient. Building a Program Without IPA - Several Steps It is also possible to use individual pgcc commands to compile each source file into a corresponding object file, and one to link the resulting object files into an executable: % pgcc -c file1.c % pgcc -c file2.c % pgcc -c file3.c % pgcc -o a.out file1.o file2.o file3.o The pgcc driver invokes the compiler and assembler as required to process each source file, and invokes the linker for the final link command. If you modify one of the source files, the executable can be rebuilt by compiling just that file and then relinking: % pgcc -c file1.c % pgcc -o a.out file1.o file2.o file3.o Building a Program Without IPA Using Make The program compilation and linking process can be simplified greatly using the make utility on systems where it is supported. Using a file makefile containing the following lines: a.out: file1.o file2.o file3.o pgcc $(OPT) -o a.out file1.o file2.o file3.o file1.o: file1.c pgcc $(OPT) -c file1.c file2.o: file2.c pgcc $(OPT) -c file2.c file3.o: file3.c pgcc $(OPT) -c file3.c It is possible to type a single make command: % makeOptimization & Parallelization 38 The make utility determines which object files are out of date with respect to their corresponding source files, and invokes the compiler to recompile only those source files and to relink the executable. If you subsequently edit one or more source files, the executable can be rebuilt with the minimum number of recompilations using the same single make command. Building a Program with IPA Interprocedural analysis and optimization (IPA) by the PGI compilers is designed to alter the standard and make utility command-level interfaces outlined above as little as possible. IPA occurs in three phases: • Collection: Create a summary of each function or procedure, collecting the useful information for interprocedural optimizations. This is done during the compile step if the –Mipa switch is present on the command line; summary information is collected and stored in the object file. • Propagation: Processing all the object files to propagate the interprocedural summary information across function and file boundaries. This is done during the link step, when all the object files are combined, if the –Mipa switch is present on the link command line. • Recompile/Optimization: Each of the object files is recompiled with the propagated interprocedural information, producing a specialized object file. This is also done during the link step when the –Mipa switch is present on the link command line. When linking with –Mipa, the PGI compilers automatically regenerate IPA-optimized versions of each object file, essentially recompiling each file. If there are IPA-optimized objects from a previous build, the compilers will minimize the recompile time by reusing those objects if they are still valid. They will still be valid if the IPA-optimized object is newer than the original object file, and the propagated IPA information for that file has not changed since it was optimized. After each object file has been recompiled, the regular linker is invoked to build the application with the IPA-optimized object files. The IPA-optimized object files are saved in the same directory as the original object files, for use in subsequent program builds. Building a Program with IPA - Single Step By adding the –Mipa command line switch, several source files can be compiled and linked with interprocedural optimizations with one command: % pgcc -Mipa=fast -o a.out file1.c file2.c file3.cInter-Procedural Analysis and Optimization using –Mipa 39 Just like compiling without –Mipa, the driver executes several steps to produce the assembly and object files, to create the executable: % pgcc -Mipa=fast -S -o file1.s file1.c % as -o file1.o file1.s % pgcc -Mipa=fast -S -o file2.s file2.c % as -o file2.o file2.s % pgcc -Mipa=fast -S -o file3.s file3.c % as -o file3.o file3.s % pgcc -Mipa=fast -o a.out file1.o file2.o file3.o In the last step, an IPA linker is invoked to read all the IPA summary information and perform the interprocedural propagation. The IPA linker reinvokes the compiler on each of the object files to recompile them with interprocedural information. This creates three new objects with mangled names: file1_ipa5_a.out.o, file2_ipa5_a.out.o, file2_ipa5_a.out.o The system linker is then invoked to link these IPA-optimized objects into the final executable. Later, if one of the three source files is edited, the executable can be rebuilt with the same command line: % pgcc -Mipa=fast -o a.out file1.c file2.c file3.c This will work, but again has the side-effect of compiling each source file, and recompiling each object file at link time. Building a Program with IPA - Several Steps Just by adding the –Mipa command-line switch, it is possible to use individual pgcc commands to compile each source file, followed by a command to link the resulting object files into an executable: % pgcc -Mipa=fast -c file1.c % pgcc -Mipa=fast -c file2.c % pgcc -Mipa=fast -c file3.c % pgcc -Mipa=fast -o a.out file1.o file2.o file3.o The pgcc driver invokes the compiler and assembler as required to process each source file, and invokes the IPA linker for the final link command. If you modify one of the source files, the executable can be rebuilt by compiling just that file and then relinking:Optimization & Parallelization 40 % pgcc -c file1.c % pgcc -o a.out file1.o file2.o file3.o When the IPA linker is invoked, it will determine that the IPA-optimized object for file1.o (file1_ipa5_a.out.o) is stale, since it is older than the object file1.o, and hence will need to be rebuilt, and will reinvoke the compiler to generate it. In addition, depending on the nature of the changes to the source file file1.c, the interprocedural optimizations previously performed for file2 and file3 may now be inaccurate. For instance, IPA may have propagated a constant argument value in a call from a function in file1.c to a function in file2.c; if the value of the argument has changed, any optimizations based on that constant value are invalid. The IPA linker will determine which, if any, of any previously created IPA-optimized objects need to be regenerated, and will reinvoke the compiler as appropriate to regenerate them. Only those objects that are stale or which have new or different IPA information will be regenerated, which saves on compile time. Building a Program with IPA Using Make As in the previous two sections, programs can be built with IPA using the make utility, just by adding the –Mipa command-line switch: OPT=-Mipa=fast a.out: file1.o file2.o file3.o pgcc $(OPT) -o a.out file1.o file2.o file3.o file1.o: file1.c pgcc $(OPT) -c file1.c file2.o: file2.c pgcc $(OPT) -c file2.c file3.o: file3.c pgcc $(OPT) -c file3.c The single command: % make will invoke the compiler to generate any object files that are out-of-date, then invoke pgcc to link the objects into the executable; at link time, pgcc will call the IPA linker to regenerate any stale or invalid IPA-optimized objects.Inter-Procedural Analysis and Optimization using –Mipa 41 Questions about IPA Why is the object file so large? An object file created with –Mipa contains several additional sections. One is the summary information used to drive the interprocedural analysis. In addition, the object file contains the compiler internal representation of the source file, so the file can be recompiled at link time with interprocedural optimizations. There may be additional information when inlining is enabled. The total size of the object file may be 5-10 times its original size. The extra sections are not added to the final executable. What if I compile with –Mipa and link without –Mipa? The PGI compilers generate a legal object file, even when the source file is compiled with –Mipa. If you compile with –Mipa and link without –Mipa, the linker is invoked on the original object files. A legal executable will be generated; while this will not have the benefit of interprocedural optimizations, any other optimizations will apply. What if I compile without –Mipa and link with –Mipa? At link time, the IPA linker must have summary information about all the functions or routines used in the program. This information is created only when a file is compiled with –Mipa. If you compile a file without –Mipa and then try to get interprocedural optimizations by linking with –Mipa, the IPA linker will issue a message that some routines have no IPA summary information, and will proceed to run the system linker using the original object files. If some files were compiled with –Mipa and others were not, it will determine the safest approximation of the IPA summary information for those files not compiled with –Mipa, and use that to recompile the other files using interprocedural optimizations. Can I build multiple applications in the same directory with –Mipa? Yes. Suppose you have three source files: main1.c, main2.c, sub.c, where sub.c is shared between the two applications. When you build the first application with –Mipa: % pgcc -o app1 main1.c sub.c the IPA linker will create two IPA-optimized object files: main1_ipa4_app1.oo sub_ipa4_app1.oo and use them to build the first application. When you build the second application: % pgcc -o app2 main2.c sub.cOptimization & Parallelization 42 the IPA linker will create two more IPA-optimized object files: main2_ipa4_app2.oo sub_ipa4_app2.oo Note there are now three object files for sub.c: the original sub.o, and two IPA-optimized objects, one for each application in which it appears. How is the mangled name for the IPA-optimized object files generated? The mangled name has '_ipa' appended, followed by the decimal number of the length of the executable file name, followed by an underscore and the executable file name itself. The suffix is changed to .oo (on Linux) or .oobj (on Windows) so linking *.o or *.obj does not pull in the IPA-optimized objects. If the IPA linker determines that the file would not benefit from any interprocedural optimizations, it does not have to recompile the file at link time, and will use the original object. Profile-Feedback Optimization using –Mpfi/–Mpfo The PGI compilers support many common profile-feedback optimizations, including semi-invariant value optimizations and block placement. These are performed under control of the –Mpfi/–Mpfo command-line options. When invoked with the –Mpfi option, the PGI compilers instrument the generated executable for collection of profile and data feedback information. This information can be used in subsequent compilations that include the –Mpfo optimization option. –Mpfi must be used at both compile-time and link-time. Programs compiled with –Mpfi include extra code to collect run-time statistics and write them out to a trace file. When the resulting program is executed, a profile feedback trace file pgfi.out is generated in the current working directory. Note Programs compiled and linked with –Mpfi will execute more slowly due to the instrumentation and data collection overhead. You should use executables compiled with – Mpfi only for execution of training runs. When invoked with the –Mpfo option, the PGI compilers use data from a pgfi.out profile feedback tracefile to enable or enhance certain performance optimizations. Use of this option requires the presence of a pgfi.out trace file in the current working directory.Default Optimization Levels 43 Default Optimization Levels The following table shows the interaction between the –O, –g and –M options. In the table, level can be 0, 1, 2 or 3, and can be vect, unroll or ipa. The default optimization level is dependent upon these command-line options. Table 2-1: Optimization and –O, –g and –M Options Unoptimized code compiled using the option –O0 can be significantly slower than code generated at other optimization levels. The –M option, where is vect, concur, unroll or ipa, sets the optimization level to level-2 if no –O options are supplied. The –fast and –fastsse options set the optimization level to a target-dependent optimization level if no –O options are supplied. Local Optimization Using Directives and Pragmas Command-line options let you specify optimizations for an entire source file. Directives supplied within a Fortran source file, and pragmas supplied within a C or C++ source file, provide information to the compiler and alter the effects of certain command-line options or default behavior of the compiler (many directives have a corresponding command-line option). While a command line option affects the entire source file that is being compiled, directives and pragmas let you do the following: Optimize Option Debug Option –M Option Optimization Level none none none 1 none none –M 2 none –g none 0 –O none or –g none 2 –Olevel none or –g none level –Olevel <= 2 none or –g –M 2 –O3 none or –g none 3Optimization & Parallelization 44 • Apply, or disable, the effects of a particular command-line option to selected subprograms or to selected loops in the source file (for example, an optimization). • Globally override command-line options. • Tune selected routines or loops based on your knowledge or on information obtained through profiling. Chapter 7, “Directives and Pragmas” provides details on how to add directives and pragmas to your source files. Execution Timing and Instruction Counting As this chapter shows, once you have a program that compiles, executes and gives correct results, you may optimize your code for execution efficiency. Selecting the correct optimization level requires some thought and may require that you compare several optimization levels before arriving at the best solution. To compare optimization levels, you need to measure the execution time for your program. There are several approaches you can take for timing execution. You can use shell commands that provide execution time statistics, you can include function calls in your code that provides timing information, or you can profile sections of code. Timing functions available with the PGI compilers include 3F timing routines, the SECNDS pre-declared function in PGF77 or PGF95, or the SYSTEM_CLOCK or CPU_CLOCK intrinsics in PGF95 or PGHPF. In general, when timing a program one should try to eliminate or reduce the amount of system level activities such as program loading, I/O and task switching. The following example shows a fragment that indicates how to use SYSTEM_CLOCK effectively within either an HPF or F90/F95 program unit. Example 2-4: Using SYSTEM_CLOCK . . . integer :: nprocs, hz, clock0, clock1 real :: time integer, allocatable :: t(:) !hpf$ distribute t(cyclic) #if defined (HPF) allocate (t(number_of_processors())) #elif defined (_OPENMP) allocate (t(OMP_GET_NUM_THREADS())) #elsePortability of Multi-Threaded Programs on Linux 45 allocate (t(1)) #endif call system_clock (count_rate=hz) ! call system_clock(count=clock0) < do work> call system_clock(count=clock1) ! t = (clock1 - clock0) time = real (sum(t)) / (real(hz) * size(t)) . . . Portability of Multi-Threaded Programs on Linux PGI has created two libraries - libpgbind and libnuma - to handle the variations between various implementations of Linux. Some older versions of Linux are lacking certain features that support multi-processor and multi-core systems, in particular, the system call 'sched_setaffinity' and the numa library libnuma. The PGI runtime library uses these features to implement some -Mconcur and -mp operations. These variations have led to the creation of two PGI libraries, libpgbind and libnuma. These libraries are used on all 32-bit and 64-bit Linux systems. These libraries are not needed on Windows. When a program is linked with the system libnuma library, the program depends on the libnuma library in order to run. On systems without a system libnuma library, the PGI version of libnuma provides the required stubs so that the program links and executes properly. If the program is linked with libpgbind and libnuma, the differences between systems is masked by the different versions of libpgbind and libnuma. In particular, PGI provides two versions of libpgbind - one for systems with working support for sched_setaffinity and another for systems that do not. When a program is deployed to the target system, the proper set of libraries, real or stub, should be deployed with the program. This facility requires that the program be dynamically linked with libpgbind and libnuma. libpgbind On some versions of Linux, the system call sched_setaffinity does not exist or does not work. The library libpgbind is used to work around this problem.Optimization & Parallelization 46 During installation, a small test program is compiled, linked, and executed. If the test program compiles, links, and executes successfully, the installed version of libpgbind calls the system sched_setaffinity, otherwise the stub version is installed. libnuma Not all systems have libnuma. Typically, only numa systems will have this library. PGI supplies a stub version of libnuma which satisfies the calls from the PGI runtime to libnuma. Note that libnuma is a shared library that is linked dynamically at runtime. The reason to have a numa library on all systems is to allow multi-threaded programs (e.g. compiled with –mp or –Mconcur) to be compiled, linked, and executed without regard to whether the host or target systems has a numa library. When the numa library is not available, a multi-threaded program still runs because the calls to the numa library are satisfied by the PGI stub library. During installation, the installation procedure checks for the existence of a real libnuma amoung the system libraries. If the real library is not found, the PGI stub version is substituted.47 3 Command Line Options This chapter describes the syntax and operation of each compiler option. The options are arranged in alphabetical order. On a command-line, options need to be preceded by a hyphen (-). If the compiler does not recognize an option, it passes the option to the linker. This chapter uses the following notation: [item] Square brackets indicate that the enclosed item is optional. {item | item} Braces indicate that you must select one and only one of the enclosed items. A vertical bar (|) separates the choices. ... Horizontal ellipses indicate that zero or more instances of the preceding item are valid. NOTE Some options do not allow a space between the option and its argument or within an argument. This fact is noted in the syntax section of the respective option.Command Line Options 48 Table 3-1: Generic PGI Compiler Options Option Description -# Display invocation information. -### Show but do not execute the driver commands (same as – dryrun). –-byteswapio (Fortran only) Swap bytes from big-endian to little-endian or vice versa on input/output of unformatted data -C Instrument the generated executable to perform array bounds checking at runtime. -c Stops after the assembly phase and saves the object code in filename.o. -D Defines a preprocessor macro. -d Print additional information from the preprocessor. -dryrun Show but do not execute driver commands. -E Stops after the preprocessing phase and displays the preprocessed file on the standard output. -F Stops after the preprocessing phase and saves the preprocessed file in filename.f (this option is only valid for the PGI Fortran compilers). -nfast Generally optimal set of flags for the target. -fastsse Generally optimal set of flags for targets that include SSE/SSE2 capability. ---flagcheck Simply return zero status if flags are correct. -flags Display valid driver options. -fpic (Linux only) Generate position-independent code. -fPIC (Linux only) Equivalent to -fpic.49 -g Includes debugging information in the object module. -g77libs (Linux only) Allow object files generated by g77 to be linked into PGI main programs. -gopt Includes debugging information in the object module, but forces assembly code generation identical to that obtained when is not present on the command line. -help Display driver help message. -I Adds a directory to the search path for #include files. -i2, -i4 and -i8 Treat INTEGER variables as 2 bytes. -i2, -i4 and -i8 Treat INTEGER variables as 4 bytes. -i2, -i4 and -i8 Treat INTEGER and LOGICAL variables as 8 bytes and use 64-bits for INTEGER*8 operations. -K Requests special compilation semantics with regard to conformance to IEEE 754. --keeplnk If the compiler generates a temporary indirect file for a long linker command, preserves the temporary file instead of deleting it. -L Specifies a library directory. -I Loads a library. -M Selects variations for code generation and optimization. –m Displays a link map on the standard output. -mcmodel=medium (-tp k8-64 and –tp p7-64 targets only) Generate code which supports the medium memory model in the linux86-64 environment. -module (F90/F95/HPF only) Save/search for module files in directory . Option DescriptionCommand Line Options 50 -mp[=align,[no]numa] Interpret and process user-inserted shared-memory parallel programming directives (see Chapters 5 and 6). -nfast Generally optimal set of flags for the target. Doesn’t use SSE. -O Specifies code optimization level where is 0, 1, 2, 3, or 4. -o Names the object file. -pc (–tp px/p5/p6/piii targets only) Set precision globablly for x87 floating-point calculations; must be used when compiling the main program. may be one of 32, 64 or 80. -pg Instrument the generated executable to produce a gprof-style gmon.out sample-based profiling trace file (-qp is also supported, and is equivalent). -pgf77libs Append PGF77 runtime libraries to the link line. –pgf90libs Append PGF90/PGF95 runtime libraries to the link line. -Q Selects variations for compiler steps. -R (Linux only) Passed to the Linker. Hard code into the search path for shared object files. –r Creates a relocatable object file. -r4 and -r8 Interpret DOUBLE PRECISION variables as REAL. -r4 and -r8 Interpret REAL variables as DOUBLE PRECISION. -rc file Specifies the name of the driver's startup file. -S Stops after the compiling phase and saves the assembly–language code in filename.s. –s Strips the symbol-table information from the object file. -shared (Linux only) Passed to the linker. Instructs the linker to generate a shared object file. Implies –fpic. Option Description51 There are a large number of compiler options specific to the PGCC and PGC++ compilers, especially PGC++. The next table lists several of these options, but is not exhaustive. For a complete list of available options, including an exhaustive list of PGC++ options, use the –help command-line option. For further detail on a given option, use –help and specify the option explicitly. -show Display driver's configuration parameters after startup. -soname Do not print warning messages. -soname Pass the soname option and its argument to the linker. -time Print execution times for the various compilation steps. -tp [,target...] Specify the type(s) of the target processor(s). -U Undefine a preprocessor macro. –u Initializes the symbol table with , which is undefined for the linker. An undefined symbol triggers loading of the first member of an archive library. -V[release_number] Displays the version messages and other information, or allows invocation of a version of the compiler other than the default. -v Displays the compiler, assembler, and linker phase invocations. -W Passes arguments to a specific phase. -w Do not print warning messages. Option DescriptionCommand Line Options 52 Table 3-2: C and C++ -specific Compiler Options Option Description -A (pgcpp only) Accept proposed ANSI C++. --[no_]alternative_tokens (pgcpp only) Enable/disable recognition of alternative tokens. These are tokens that make it possible to write C++ without the use of the , , [, ], #, &, and ^ and characters. The alternative tokens include the operator keywords (e.g., and, bitand, etc.) and digraphs. The default is -–no_alternative_tokens. -B Allow C++ comments (using //) in C source. -b (pgcpp only) Compile with cfront 2.1 compatibility. This accepts constructs and a version of C++ that is not part of the language definition but is accepted by cfront. EDG option. -b3 (pgcpp only) Compile with cfront 3.0 compatibility. See -b above. --[no_]bool (pgcpp only) Enable or disable recognition of bool. The default value is ––bool. ––[no]??? Do/don’t compile with math subroutine builtin support, which causes selected math library routines to be inlined. The default is ––builtin. --cfront_2.1 (pgcpp only) Enable compilation of C++ with compatibility with cfront version 2.1. --cfront_3.0 (pgcpp only) Enable compilation of C++ with compatibility with cfront version 3.0. --create_pch filename (pgcpp only) Create a precompiled header file with the name filename. --dependencies ( see -M ) (pgcpp only) Print makefile dependencies to stdout. 53 --dependencies_to_file filename (pgcpp only) Print makefile dependencies to file filename. --diag_error tag (pgcpp only) Override the normal error severity of the specified diagnostic messages. --diag_remark tag (pgcpp only) Override the normal error severity of the specified diagnostic messages. --diag_suppress tag (pgcpp only) Override the normal error severity of the specified diagnostic messages. --diag_warning tag (pgcpp only) Override the normal error severity of the specified diagnostic messages. --display_error_number (pgcpp only) Display the error message number in any diagnostic messages that are generated. –e (pgcpp only) Set the C++ front-end error limit to the specified . --[no_]exceptions (pgcpp only) Disable/enable exception handling support. The default is ––exceptions ––gnu_extensions (pgcpp only) Allow GNU extensions like “include next” which are required to compile Linux system header files. --[no]llalign (pgcpp only) Do/don’t align long long integers on integer boundaries. The default is ––llalign. -M Generate make dependence lists. -MD Generate make dependence lists. -MD,filename (pgcpp only) Generate make dependence lists and print them to file filename. --optk_allow_dollar_in_id_chars (pgcpp only) Accept dollar signs in identifiers. Option DescriptionCommand Line Options 54 --pch (pgcpp only) Automatically use and/or create a precompiled header file. --pch_dir directoryname (pgcpp only) The directory dirname in which to search for and/or create a precompiled header file. --[no_]pch_messages (pgcpp only) Enable/ disable the display of a message indicating that a precompiled header file was created or used. +p (pgcpp only) Disallow all anachronistic constructs. cfront option -P Stops after the preprocessing phase and saves the preprocessed file in filename.i. --preinclude= (pgcpp only) Specify file to be included at the beginning of compilation; to set system-dependent macros, types, etc -t Control instantiation of template functions. EDG option --use_pch filename (pgcpp only) Use a precompiled header file of the specified name as part of the current compilation. --[no_]using_std (pgcpp only) Enable/disable implicit use of the std namespace when standard header files are included. –X (pgcpp only) Generate cross-reference information and place output in specified file. EDG option. –Xm (pgcpp only) Allow $ in names. –xh (pgcpp only) Enable exception handling. EDG option. -suffix (see –P) (pgcpp only) Use with –E, –F, or –P to save intermediate file in a file with the specified suffix. Option DescriptionGeneric PGI Compiler Options 55 Generic PGI Compiler Options -# Use the –# option to display the invocations of the compiler, assembler and linker. These invocations are command-lines created by the driver from your command-line input and the default values. Default: The compiler does not display individual phase invocations. Usage: The following command-line requests verbose invocation information. $ pgf95 -# prog.f Cross-reference: –Minfo, –V, –v. -### Use the –### option to display the invocations of the compiler, assembler and linker but do not execute them. These invocations are command lines created by the compiler driver from the PGIRC files and the command-line options. Default: The compiler does not display individual phase invocations. Usage: The following command-line requests verbose invocation information. $ pgf95 -### myprog.f Cross-reference: –Minfo, –V, –dryrun. -byteswapio Use the –byteswapio option to swap the byte-order of data in unformatted Fortran data files on input/ output. When this option is used, the order of bytes is swapped in both the data and record control words (the latter occurs in unformatted sequential files). Specifically, this option can be used to convert big-endian format data files produced by most RISC workstations and high-end servers to the little-endian format used on x86 or x64 systems on the fly during file reads/writes. This option assumes that the record layouts of unformatted sequential access and direct access files are the same on the systems. Also, the assumption is that the IEEE representation is used for floating-point numbers. In particular, the format of unformatted data files produced by PGI Fortran compilers is identical to the format used on Sun and SGI workstations, that allows you to read and write unformatted Fortran data files produced on those platforms from a program compiled for an x86 or x64 platform using the –byteswapio option.Command Line Options 56 Default: The compiler does not byte-swap data on input/output. Usage: The following command-line requests byte-swapping are performed on input/output. $ pgf95 -byteswapio myprog.f -C Enables array bounds checking. If an array is an assumed size array, the bounds checking only applies to the lower bound. If an array bounds violation occurs during execution, an error message describing the error is printed and the program terminates. The text of the error message includes the name of the array, the location where the error occurred (the source file and the line number in the source), and information about the out of bounds subscript (its value, its lower and upper bounds, and its dimension). Default: The compiler does not enable array bounds checking. Usage: In this example, the compiler instruments the executable produced from myprog.f to perform array bounds checking at runtime: $ pgf95 -C myprog.f Cross-reference: –Mbounds. -c Stops after the assembling phase. Use the –c option to halt the compilation process after the assembling phase and write the object code to the file filename.o, where the input file is filename.f. Default: The compiler produces an executable file (does not use the –c option). Usage: In this example, the compiler produces the object file myprog.o in the current directory. $ pgf95 -c myprog.f Cross-reference: –E, –Mkeepasm, –o, and –S. -d Print additional information from the preprocessor. Syntax: -d[D|I|M|N]Generic PGI Compiler Options 57 where D, I, M, and N are: -dD Print macros and values from source files -dI Print include file names -dM Print macros and values, including predefined and command-line macros -dN Print macro names from source files Cross-reference: –E, –D, –U. -D Defines a preprocessor macro. Use the –D option to create a macro with a given value. The value must be either an integer or a character string. You can use the –D option more than once on a compiler command line. The number of active macro definitions is limited only by available memory. You can use macros with conditional compilation to select source text during preprocessing. A macro defined in the compiler invocation remains in effect for each module on the command line, unless you remove the macro with an #undef preprocessor directive or with the –U option. The compiler processes all of the –U options in a command line after processing the –D options. Syntax: -Dname[=value] Where name is the symbolic name and value is either an integer value or a character string. Default: If you define a macro name without specifying a value the preprocessor assigns the string 1 to the macro name. Usage: In the following example, the macro PATHLENGTH has the value 256 until a subsequent compilation. If the –D option is not used, PATHLENGTH’s value is set to 128. $ pgf95 -DPATHLENGTH=256 myprog.F Where the source text is: #ifndef PATHLENGTH #define PATHLENGTH 128 #endifCommand Line Options 58 SUBROUTINE SUB CHARACTER*PATHLENGTH path ... END Cross-reference: –U -dryrun Use the –dryrun option to display the invocations of the compiler, assembler and linker but do not execute them. These invocations are command lines created by the compiler driver from the PGIRC file and the command-line supplied with –dryrun. Default: The compiler does not display individual phase invocations. Usage: The following command-line requests verbose invocation information. $ pgf95 -dryrun myprog.f Cross-reference: –Minfo, –V, –### -E Stops after the preprocessing phase. Use the –E option to halt the compilation process after the preprocessing phase and display the preprocessed output on the standard output. Default: The compiler produces an executable file. Usage: In the following example the compiler displays the preprocessed myprog.f on the standard output. $ pgf95 -E myprog.f Cross-reference: See the options –C, –c, –Mkeepasm, –o, –F, –S. -F Stops compilation after the preprocessing phase. Use the –F option to halt the compilation process after preprocessing and write the preprocessed output to the file filename.f where the input file is filename .F. Default: The compiler produces an executable file.Generic PGI Compiler Options 59 Usage: In the following example the compiler produces the preprocessed file myprog.f in the current directory. $ pgf95 -F myprog.F Cross-reference: –c,–E, –Mkeepasm, –o, –S -fast A generally optimal set of options is chosen for targets that support SSE capability. In addition, the appropriate –tp option is automatically included to enable generation of code optimized for the type of system on which compilation is performed. Note Auto-selection of the appropriate –tp option means that programs built using the –fastsse option on a given system are not necessarily backward-compatible with older systems. Cross-reference: –O, –Munroll, –Mnoframe, –Mscalarsse, –Mvect, –Mcache_align, –tp -fastsse Synonymous with -fast. --flagcheck Causes the compiler to check that flags are correct then exit. If flags are all correct then the compiler returns a zero status. -flags Displays driver options on the standard output. Use this option with –v to list options that are recognized and ignored, as well as the valid options. Cross-reference: –#, –###, –v -fpic (Linux only) Generate position-independent code suitable for inclusion in shared object (dynamically linked library) files.Command Line Options 60 Cross-reference: –shared, –G, –R -fPIC (Linux only) Equivalent to -fpic. Provided for compatibility with other compilers. Cross-reference: –fpic, –shared, –G, –R -G (Linux only) Passed to the linker. Instructs the linker to produce a shared object file. Cross-reference: –fpic, –shared, –R -g The –g option instructs the compiler to include symbolic debugging information in the object module. Debuggers, such as PGDBG, require symbolic debugging information in the object module to display and manipulate program variables and source code. Note that including symbolic debugging information increases the size of the object module. If you specify the –g option on the command-line, the compiler sets the optimization level to –O0 (zero), unless you specify the –O option. For more information on the interaction between the –g and –O options, see the –O entry. Symbolic debugging may give confusing results if an optimization level other than zero is selected. Default: The compiler does not put debugging information into the object module. Usage: In the following example, the object file a.out contains symbolic debugging information. $ pgf95 -g myprog.f -gopt Use of –g alters how optimized code is generated in ways that are intended to enable or improve debugging of optimized code. The –gopt option instructs the compiler to include symbolic debugging information in the object file, and to generate optimized code identical to that generated when –g is not specified. Default: The compiler does not put debugging information into the object module. Usage: In the following example, the object file a.out contains symbolic debugging information.Generic PGI Compiler Options 61 $ pgf95 -gopt myprog.f -g77libs (Linux only) Use the –g77libs option on the link line if you are linking g77-compiled program units into a pgf95-compiled main program using the pgf95 driver. When this option is present, the pgf95 driver will search the necessary g77 support libraries to resolve references specific to g77 compiled program units. The g77 compiler must be installed on the system on which linking occurs in order for this option to function correctly. Default: The compiler does not search g77 support libraries to resolve references at link time. Usage: The following command-line requests that g77 support libraries be searched at link time: $ pgf95 -g77libs myprog.f g77_object.o -help Used with no other options, –help displays options recognized by the driver on the standard output. When used in combination with one or more additional options, usage information for those options is displayed to standard output. Usage: In the following example, usage information for –Minline is printed to standard output. $ pgcc -help -Minline -Minline[=lib:||except:| name:|size:|levels:] Enable function inlining lib: Use extracted functions from extlib Inline function func except: Do not inline function func name: Inline function func size: Inline only functions smaller than n levels: Inline n levels of functions -Minline Inline all functions that were extracted In the following example, usage information for –help shows how groups of options can be listed or examined according to function $ pgcc -help -help -help[=groups|asm|debug|language|linker|opt|other| overall|phase|prepro|suffix|switch|target|variable] Show compiler switches Command Line Options 62 Cross-reference: –#, –###, –show, –V, –flags -I Adds a directory to the search path for files that are included using the INCLUDE statement or the preprocessor directive #include. Use the –I option to add a directory to the list of where to search for the included files. The compiler searches the directory specified by the –I option before the default directories.stdinc Syntax: -Idirectory Where directory is the name of the directory added to the standard search path for include files. Usage: The Fortran INCLUDE statement directs the compiler to begin reading from another file. The compiler uses two rules to locate the file: 1. If the file name specified in the INCLUDE statement includes a path name, the compiler begins reading from the file it specifies. 2. If no path name is provided in the INCLUDE statement, the compiler searches (in order): • any directories specified using the –I option (in the order specified.) • the directory containing the source file • the current directory For example, the compiler applies rule (1) to the following statements: INCLUDE '/bob/include/file1' (absolute path name) INCLUDE '../../file1' (relative path name) and rule (2) to this statement: INCLUDE 'file1' Cross-reference: –MnostdincGeneric PGI Compiler Options 63 -i2, -i4 and -i8 Treat INTEGER and LOGICAL variables as either two, four, or eight bytes. INTEGER*8 values not only occupy 8 bytes of storage, but operations use 64 bits, instead of 32 bits. -K Requests that the compiler provide special compilation semantics. Syntax: –K Where flag is one of the following: ieee Perform floating-point operations in strict conformance with the IEEE 754 standard. Some optimizations are disabled, and on some systems a more accurate math library is linked if –Kieee is used during the link step. noieee Use the fastest available means to perform floating-point operations, link in faster non-IEEE libraries if available, and disable underflow traps. PIC (Linux only) Generate position-independent code. Equivalent to –fpic. Provided for compatibility with other compilers. pic (Linux only) Generate position-independent code. Equivalent to –fpic. Provided for compatibility with other compilers. trap=option[,option]... Controls the behavior of the processor when floating-point exceptions occur. Possible options include: • fp • align (ignored) • inv • denorm • divz • ovfCommand Line Options 64 • unf • inexact –Ktrap is only processed by the compilers when compiling main functions/programs. The options inv, denorm, divz, ovf, unf, and inexact correspond to the processor’s exception mask bits invalid operation, denormalized operand, divide-by-zero, overflow, underflow, and precision, respectively. Normally, the processor’s exception mask bits are on (floating-point exceptions are masked—the processor recovers from the exceptions and continues). If a floating-point exception occurs and its corresponding mask bit is off (or “unmasked”), execution terminates with an arithmetic exception (C's SIGFPE signal). -Ktrap=fp is equivalent to -Ktrap=inv,divz,ovf. Default: The default is -Knoieee. --keeplnk If the compiler generates a temporary indirect file for a long linker command, preserve the temporary file instead of deleting it. (Windows only.) -L Specifies a directory to search for libraries. Use –L to add directories to the search path for library files. Multiple –L options are valid. However, the position of multiple –L options is important relative to –l options supplied. Syntax: -Ldirectory Where directory is the name of the library directory. Default: Search the standard library directory. Usage: In the following example, the library directory is /lib and the linker links in the standard libraries required by PGF95 from /lib. $ pgf95 -L/lib myprog.fGeneric PGI Compiler Options 65 In the following example, the library directory /lib is searched for the library file libx.a and both the directories /lib and /libz are searched for liby.a. $ pgf95 -L/lib -lx -L/libz -ly myprog.f -l Loads a library. The linker searches in addition to the standard libraries. Libraries specified with –l are searched in order of appearance and before the standard libraries. Syntax: -llibrary Where library is the name of the library to search. The compiler prepends the characters lib to the library name and adds the .a extension following the library name. Usage: In the following example, if the standard library directory is /lib the linker loads the library / lib/libmylib.a, in addition to the standard libraries. $ pgf95 myprog.f -lmylib -M Selects options for code generation. The options are divided into the following categories: Code generation Environment Inlining Fortran Language Controls C/C++ Language Controls Optimization Miscellaneous The following table lists and briefly describes the options alphabetically and includes a field showing the category. Command Line Options 66 Table 3-3: –M Options Summary pgflag Description Category allocatable=95|03 controls whether to use Fortran 95 or Fortran 2003 semantics in allocatable array assignments. Fortran Language anno annotate the assembly code with source code. Miscellaneous [no]autoinline C/C++ when a function is declared with the inline keyword, inline it at -O2 and above. Inlining [no]asmkeyword specifies whether the compiler allows the asm keyword in C/C++ source files (pgcc and pgcpp only). C/C++ Language [no]backslash determines how the backslash character is treated in quoted strings (pgf77, pgf95, and pghpf only). Fortran Language [no]bounds specifies whether array bounds checking is enabled or disabled. Miscellaneous [no]builtin Do/don’t compile with math subroutine builtin support, which causes selected math library routines to be inlined (pgcc and pgcpp only). Optimization byteswapio Swap byte-order (big-endian to little-endian or vice versa) during I/O of Fortran unformatted data. Miscellaneous cache_align where possible, align data objects of size greater than or equal to 16 bytes on cache-line boundaries. Optimization chkfpstk check for internal consistency of the x87 FP stack in the prologue of a function and after returning from a function or subroutine call (–tp px/p5/p6/piii targets only). MiscellaneousGeneric PGI Compiler Options 67 chkptr check for NULL pointers (pgf95 and pghpf only). Miscellaneous chkstk check the stack for available space upon entry to and before the start of a parallel region. Useful when many private variables are declared. Miscellaneous concur enable auto-concurrentization of loops. Multiple processors or cores will be used to execute parallelizable loops. Optimization cpp run the PGI cpp-like pre-processor without performing subsequent compilation steps. Miscellaneous cray Force Cray Fortran (CF77) compatibility (pgf77, pgf95, and pghpf only). Optimization [no]daz Do/don’t treat denormalized numbers as zero. Code Generation [no]dclchk determines whether all program variables must be declared (pgf77, pgf95, and pghpf only). Fortran Language [no]defaultunit determines how the asterisk character ("*") is treated in relation to standard input and standard output (regardless of the status of I/O units 5 and 6, pgf77, pgf95, and pghpf only). Fortran Language [no]depchk checks for potential data dependencies. Optimization [no]dse enables [disables] dead store elimination phase for programs making extensive use of function inlining. Optimization [no]dlines determines whether the compiler treats lines containing the letter "D" in column one as executable statements (pgf77, pgf95, and pghpf only). Fortran Language pgflag Description CategoryCommand Line Options 68 dll Link with the DLL version of the runtime libraries (Windows only). Miscellaneous dollar,char specifies the character to which the compiler maps the dollar sign code (pgf77, pgf95, and pghpf only). Fortran Language dwarf1 when used with –g, generate DWARF1 format debug information. Code Generation dwarf2 when used with –g, generate DWARF2 format debug information. Code Generation dwarf3 when used with –g, generate DWARF3 format debug information. Code Generation extend the compiler accepts 132-column source code; otherwise it accepts 72-column code (pgf77, pgf95, and pghpf only). Fortran Language extract invokes the function extractor. Inlining fcon instructs the compiler to treat floating-point constants as float data types (pgcc and pgcpp only). C/C++ Language fixed the compiler assumes F77-style fixed format source code (pgf95 and pghpf only). Fortran Language [no]flushz do/don’t set SSE flush-to-zero mode Code Generation [no]fprelaxed[=option] Perform certain floating point intrinsic functions using relaxed precision. Optimization free the compiler assumes F90-style free format source code (pgf95 and pghpf only). Fortran Language func32 the compiler aligns all functions to 32-byte boundaries. Code Generation pgflag Description CategoryGeneric PGI Compiler Options 69 gccbug[s] match behavior of certain gcc bugs Miscellaneous noi4 determines how the compiler treats INTEGER variables (pgf77, pgf95, and pghpf only). Optimization info prints informational messages regarding optimization and code generation to standard output as compilation proceeds. Miscellaneous inform specifies the minimum level of error severity that the compiler displays. Miscellaneous inline invokes the function inliner. Inlining [no]ipa invokes inter-procedural analysis and optimization. Optimization [no]iomutex determines whether critical sections are generated around Fortran I/O calls (pgf77, pgf95, and pghpf only). Fortran Language [no]large_arrays enable support for 64-bit indexing and single static data objects of size larger than 2GB. Code Generation lfs link in libraries that allow file I/O to files of size larger than 2GB on 32-bit systems (32-bit Linux only). Environment [no]lre Disable/enable loop-carried redundancy elimination. Optimization keepasm instructs the compiler to keep the assembly file. Miscellaneous nolist specifies whether the compiler creates a listing file. Miscellaneous makedll Generate a dynamic link library (DLL) (Windows only). Miscellaneous pgflag Description CategoryCommand Line Options 70 [no]movnt (disable) force generation of non-temporal moves and prefetching. Code Generation neginfo instructs the compiler to produce information on why certain optimizations are not performed. Miscellaneous noframe eliminates operations that set up a true stack frame pointer for functions. Optimization nomain when the link step is called, don’t include the object file that calls the Fortran main program (pgf77, pgf95, and pghpf only). Code Generation noopenmp when used in combination with the -mp option, causes the compiler to ignore OpenMP parallelization directives or pragmas, but still process SGI-style parallelization directives or pragmas. Miscellaneous nopgdllmain do not link the module containing the default DllMain() into the DLL (Windows only). Miscellaneous norpath On Linux, do not add -rpath paths to the link line. Miscellaneous nosgimp when used in combination with the -mp option, causes the compiler to ignore SGI-style parallelization directives or pragmas, but still process OpenMP directives or pragmas. Miscellaneous nostartup do not link in the standard startup routine (pgf77, pgf95, and pghpf only). Environment nostddef instructs the compiler to not recognize the standard preprocessor macros. Environment nostdinc instructs the compiler to not search the standard location for include files. Environment pgflag Description CategoryGeneric PGI Compiler Options 71 nostdlib instructs the linker to not link in the standard libraries. Environment noonetrip determines whether each DO loop executes at least once (pgf77, pgf95, and pghpf only). Language novintr disable idiom recognition and generation of calls to optimized vector functions. Optimization pfi instrument the generated code and link in libraries for dynamic collection of profile and data information at runtime. Optimization pfo read a pgfi.out trace file and use the information to enable or guide optimizations. Optimization [no]prefetch (disable) enable generation of prefetch instructions. Optimization preprocess perform cpp-like preprocessing on assembly language and Fortran input source files. Miscellaneous prof set profile options; function-level and linelevel profiling are supported. Code Generation nor8 determines whether the compiler promotes REAL variables and constants to DOUBLE PRECISION (pgf77, pgf95, and pghpf only). Optimization nor8intrinsics determines how the compiler treats the intrinsics CMPLX and REAL (pgf77, pgf95, and pghpf only). Optimization [no]recursive allocate (do not allocate) local variables on the stack, this allows recursion. SAVEd, datainitialized, or namelist members are always allocated statically, regardless of the setting of this switch (pgf77, pgf95, and pghpf only). Code Generation pgflag Description CategoryCommand Line Options 72 [no]reentrant specifies whether the compiler avoids optimizations that can prevent code from being reentrant. Code Generation [no]ref_externals do/don’t force references to names appearing in EXTERNAL statements (pgf77, pgf95, and pghpf only). Code Generation safeptr instructs the compiler to override data dependencies between pointers and arrays (pgcc and pgcpp only). Optimization safe_lastval In the case where a scalar is used after a loop, but is not defined on every iteration of the loop, the compiler does not by default parallelize the loop. However, this option tells the compiler it safe to parallelize the loop. For a given loop, the last value computed for all scalars make it safe to parallelize the loop. Code Generation [no]nosave determines whether the compiler assumes that all local variables are subject to the SAVE statement (pgf77, pgf95, and pghpf only). Fortran Language [no]scalarsse do/don’t use SSE/SSE2 instructions to perform scalar floating-point arithmetic. Optimization schar specifies signed char for characters (pgcc and pgcpp only - also see uchar). C/C++ Language [no]second_underscore do/don’t add the second underscore to the name of a Fortran global if its name already contains an underscore (pgf77, pgf95, and pghpf only). Code Generation [no]signextend do/don’t extend the sign bit, if it is set. Code Generation pgflag Description CategoryGeneric PGI Compiler Options 73 nosingle do/don’t convert float parameters to double parameter characters (pgcc and pgcpp only). C/C++ Language [no]smart do/don’t enable optional post-pass assembly optimizer. Optimization [no]smartalloc add a call to the routine mallopt in the main routine. To be effective, this switch must be specified when compiling the file containing the Fortran, C, or C++ main program. Environment standard causes the compiler to flag source code that does not conform to the ANSI standard (pgf77, pgf95, and pghpf only). Fortran Language nostride0 the compiler generates (does not generate) alternate code for a loop that contains an induction variable whose increment may be zero (pgf77, pgf95, and pghpf only). Code Generation uchar specifies unsigned char for characters (pgcc and pgcpp only - also see schar). C/C++ Language unix uses UNIX calling and naming conventions for Fortran subprograms (pgf77, pgf95, and pghpf for Win32 only). CodeGeneration [no]nounixlogical determines whether logical .TRUE. and .FALSE. are determined by non-zero (TRUE) and zero (FALSE) values for unixlogical. With nounixlogical, the default, -1 values are TRUE and 0 values are FALSE (pgf77, pgf95, and pghpf only). Fortran Language [no]unroll controls loop unrolling. Optimization pgflag Description CategoryCommand Line Options 74 Following are detailed descriptions of several, but not all, of the –M options outlined in the table above. These options are grouped according the category that appears in column 3 of the table above, and are listed with exact syntax, defaults, and notes concerning similar or related options. For the latest information and description of a given option, or to see all available options, use the –help command-line option to any of the PGI compilers. -M Code Generation Controls Syntax: -Mdaz Set IEEE denormalized input values to zero; there is a performance benefit but misleading results can occur, such as when dividing a small normalized number by a denormalized number. This option must be set for the main program to take effect. -Mnodaz Do not treat denormalized numbers as zero. This option must be set for the main program to take effect. -Mdwarf1 Generate DWARF1 format debug information; must be used in combination with –g. -Mdwarf2 Generate DWARF2 format debug information; must be used in combination with –g. -Mdwarf3 Generate DWARF3 format debug information; must be used in combination with –g. noupcase determines whether the compiler allows uppercase letters in identifiers (pgf77, pgf95, and pghpf only). Fortran Language varargs force Fortran program units to assume calls are to C functions with a varargs type interface (pgf77 and pgf95 only). Code Generation [no]vect do/don’t invoke the code vectorizer. Optimization pgflag Description CategoryGeneric PGI Compiler Options 75 -Mflushz Set SSE flush-to-zero mode; if a floating-point underflow occurs, the value is set to zero. This option must be set for the main program to take effect. -Mnoflushz Do not set SSE flush-to-zero mode; generate underflows. This option must be set for the main program to take effect. -Mfunc32 Align functions on 32-byte boundaries. -Mlarge_arrays Enable support for 64-bit indexing and single static data objects larger than 2GB in size. This option is default in the presence of – mcmodel=medium. Can be used separately together with the default small memory model for certain 64-bit applications that manage their own memory space. See “Programming Considerations for 64-Bit Environments” on page 229 for more information -Mnolarge_arrays Disable support for 64-bit indexing and single static data objects larger than 2GB in size. When placed after –mcmodel=medium on the command line, disables use of 64-bit indexing for applications that have no single data object larger than 2GB. -Mnomain instructs the compiler not to include the object file that calls the Fortran main program as part of the link step. This option is useful for linking programs in which the main program is written in C/C++ and one or more subroutines are written in Fortran (pgf77, pgf95, and pghpf only). -M[no]movnt instructs the compiler to generate nontemporal move and prefetch instructions even in cases where the compiler cannot determine statically at compile-time that these instructions will be beneficial. -Mprof[=option[,option,...]] Set profile options. option can be any of the following: dwarf generate limited DWARF information to enable source correlation by 3rd-party profiling tools. (all platforms) func perform PGI-style function-level profiling (all platforms)Command Line Options 76 hwcts Use PAPI-based profiling with hardware counters (linux86-64 platforms only). lines perform PGI-style line-level profiling. (all platforms) mpi perform MPI profiling (available only in PGI CDK Cluster Development Kit configurations on Linux platforms). time Sample-based instruction-level profiling. (Linux only) -Mrecursive instructs the compiler to allow Fortran subprograms to be called recursively. -Mnorecursive Fortran subprograms may not be called recursively. -Mref_externals force references to names appearing in EXTERNAL statements (pgf77, pgf95, and pghpf only). -Mnoref_externals do not force references to names appearing in EXTERNAL statements (pgf77, pgf95, and pghpf only). -Mreentrant instructs the compiler to avoid optimizations that can prevent code from being reentrant. -Mnoreentrant instructs the compiler not to avoid optimizations that can prevent code from being reentrant. -Msecond_underscore instructs the compiler to add a second underscore to the name of a Fortran global symbol if its name already contains an underscore. This option is useful for maintaining compatibility with object code compiled using g77, which uses this convention by default (pgf77, pgf95, and pghpf only). -Mnosecond_underscore instructs the compiler not to add a second underscore to the name of a Fortran global symbol if its name already contains an underscore (pgf77, pgf95, and pghpf only).Generic PGI Compiler Options 77 -Msignextend instructs the compiler to extend the sign bit that is set as a result of converting an object of one data type to an object of a larger signed data type. -Mnosignextend instructs the compiler not to extend the sign bit that is set as the result of converting an object of one data type to an object of a larger data type. -Msafe_lastval In the case where a scalar is used after a loop, but is not defined on every iteration of the loop, the compiler does not by default parallelize the loop. However, this option tells the compiler it’s safe to parallelize the loop. For a given loop the last value computed for all scalars make it safe to parallelize the loop. -Mstride0 instructs the compiler to inhibit certain optimizations and to allow for stride 0 array references. This option may degrade performance and should only be used if zero-stride induction variables are possible. -Mnostride0 instructs the compiler to perform certain optimizations and to disallow for stride 0 array references. -Munix use UNIX symbol and parameter passing conventions for Fortran subprograms (pgf77, pgf95, and pghpf for Win32 only). -Mvarargs force Fortran program units to assume procedure calls are to C functions with a varargs-type interface (pgf77 and pgf95 only). Default: For arguments that you do not specify, the default code generation controls are as follows: -M Environment Controls Syntax: nodaz noflushz norecursive nostride0 noreentrant noref_externals signextend nosecond_underscoreCommand Line Options 78 -Mlfs (32-bit Linux only) link in libraries that enable file I/O to files larger than 2GB (Large File Support). -Mnostartup instructs the linker not to link in the standard startup routine that contains the entry point (_start) for the program. Note If you use the –Mnostartup option and do not supply an entry point, the linker issues the following error message: Warning: cannot find entry symbol _start -Mnostddef instructs the compiler not to predefine any macros to the preprocessor when compiling a C program. -Mnostdlib instructs the linker not to link in the standard libraries libpgftnrtl.a, libm.a, libc.a and libpgc.a in the library directory lib within the standard directory. You can link in your own library with the –l option or specify a library directory with the –L option. Default: For arguments that you do not specify, the default environment option depends on your configuration. Cross-reference: –D, –I, –L, –l, –U -M Inlining Controls This section describes the –M options that control function inlining. Syntax: -Mextract[=option[,option,...]] Extracts functions from the file indicated on the command line and creates or appends to the specified extract directory where option can be any of: name:func instructs the extractor to extract function func from the file. size:number instructs the extractor to extract functions with number or fewer, statements from the file.Generic PGI Compiler Options 79 lib:filename.ext Use directory filename.ext as the extract directory (required in order to save and re-use inline libraries). If you specify both name and size, the compiler extracts functions that match func, or that have number or fewer statements. For examples of extracting functions, see Function Inlining. -Minline[=option[,option,...]] This passes options to the function inliner where option can be any of: except:func instructs the inliner to inline all eligible functions except func, a function in the source text. Multiple functions can be listed, comma-separated. [name:]func instructs the inliner to inline the function func. The func name should be a nonnumeric string that does not contain a period. You can also use a name: prefix followed by the function name. If name: is specified, what follows is always the name of a function. [lib:]filename.ext instructs the inliner to inline the functions within the library file filename.ext. The compiler assumes that a filename.ext option containing a period is a library file. Create the library file using the –Mextract option. You can also use a lib: prefix followed by the library name. If lib: is specified, no period is necessary in the library name. Functions from the specified library are inlined. If no library is specified, functions are extracted from a temporary library created during an extract prepass.Command Line Options 80 [size:]number instructs the inliner to inline functions with number or fewer statements. You can also use a size: prefix followed by a number. If size: is specified, what follows is always taken as a number. levels:number instructs the inliner to perform number levels of inlining. The default number is 1. If you specify both func and number, the compiler inlines functions that match the function name or have number or fewer statements. For examples of inlining functions, see Function Inlining. Usage: In the following example, the compiler extracts functions that have 500 or fewer statements from the source file myprog.f and saves them in the file extract.il. $ pgf95 -Mextract=500 -oextract.il myprog.f In the following example, the compiler inlines functions with fewer than approximately 100 statements in the source file myprog.f and writes the executable code in the default output file a.out. $ pgf95 -Minline=size:100 myprog.f Cross-reference: –o, -Mextract -M Fortran Language Controls This section describes the –M options that affect Fortran language interpretations by the PGI Fortran compilers. These options are only valid to the pgf77, pgf95, and pghpf compiler drivers. Syntax: -Mallocatable=95|03 controls whether Fortran 95 or Fortran 2003 semantics are used in allocatable array assignments. -Mbackslash the compiler treats the backslash as a normal character, and not as an escape character in quoted strings. -Mnobackslash the compiler recognizes a backslash as an escape character in quoted strings (in accordance with standard C usage). Generic PGI Compiler Options 81 -Mdclchk the compiler requires that all program variables be declared. -Mnodclchk the compiler does not require that all program variables be declared. -Mdefaultunit the compiler treats "*" as a synonym for standard input for reading and standard output for writing. -Mnodefaultunit the compiler treats "*" as a synonym for unit 5 on input and unit 6 on output. -Mdlines the compiler treats lines containing "D" in column 1 as executable statements (ignoring the "D"). -Mnodlines the compiler does not treat lines containing "D" in column 1 as executable statements (does not ignore the "D"). -Mdollar,char char specifies the character to which the compiler maps the dollar sign. The compiler allows the dollar sign in names. -Mextend with –Mextend, the compiler accepts 132-column source code; otherwise it accepts 72-column code. -Mfixed with –Mfixed, the compiler assumes input source files are in FORTRAN 77-style fixed form format. -Mfree with –Mfree, the compiler assumes the input source files are in Fortran 90/95 freeform format. -Miomutex the compiler generates critical section calls around Fortran I/O statements. -Mnoiomutex the compiler does not generate critical section calls around Fortran I/O statements. -Monetrip the compiler forces each DO loop to execute at least once. -Mnoonetrip the compiler does not force each DO loop to execute at least once. This option is useful for programs written for earlier versions of Fortran. -Msave the compiler assumes that all local variables are subject to the SAVE statement. Note that this may allow older Fortran programs to run, but it can greatly reduce performance.Command Line Options 82 -Mnosave the compiler does not assume that all local variables are subject to the SAVE statement. -Mstandard the compiler flags non-ANSI–conforming source code. -Munixlogical directs the compiler to treat logical values as true if the value is nonzero and false if the value is zero (UNIX F77 convention.) When – Munixlogical is enabled, a logical value or test that is non-zero is .TRUE., and a value or test that is zero is .FALSE.. In addition, the value of a logical expression is guaranteed to be one (1) when the result is .TRUE.. -Mnounixlogical ldirects the compiler to use the VMS convention for logical values for true and false. Even values are true and odd values are false. -Mupcase the compiler allows uppercase letters in identifiers. With –Mupcase, the identifiers "X" and "x" are different, and keywords must be in lower case. This selection affects the linking process: if you compile and link the same source code using –Mupcase on one occasion and –Mnoupcase on another, you may get two different executables (depending on whether the source contains uppercase letters). The standard libraries are compiled using the default –Mnoupcase. -Mnoupcase the compiler converts all identifiers to lower case. This selection affects the linking process: If you compile and link the same source code using –Mupcase on one occasion and –Mnoupcase on another, you may get two different executables (depending on whether the source contains uppercase letters). The standard libraries are compiled using – Mnoupcase. Default: For arguments that you do not specify, the defaults are as follows: nobackslash noiomutex nodclchk noonetrip nodefaultunit nosave nodlines nounixlogicalGeneric PGI Compiler Options 83 -M C/C++ Language Controls This section describes the –M options that affect C/C++ language interpretations by the PGI C and C++ compilers. These options are only valid to the pgcc and pgcpp compiler drivers. Syntax: -Masmkeyword instructs the compiler to allow the asm keyword in C source files. The syntax of the asm statement is as follows: asm("statement"); Where statement is a legal assembly-language statement. The quote marks are required. -Mnoasmkeyword instructs the compiler not to allow the asm keyword in C source files. If you use this option and your program includes the asm keyword, unresolved references will be generated -Mdollar,char char specifies the character to which the compiler maps the dollar sign ($). The PGCC compiler allows the dollar sign in names; ANSI C does not allow the dollar sign in names. -Mfcon instructs the compiler to treat floating-point constants as float data types, instead of double data types. This option can improve the performance of single-precision code. -Mschar specifies signed char characters. The compiler treats "plain" char declarations as signed char. -Msingle do not to convert float parameters to double parameters in nonprototyped functions. This option can result in faster code if your program uses only float parameters. However, since ANSI C specifies that routines must convert float parameters to double parameters in nonprototyped functions, this option results in non-ANSI conformant code. -Mnosingle instructs the compiler to convert float parameters to double parameters in non-prototyped functions. dollar,_ noupcaseCommand Line Options 84 -Muchar instructs the compiler to treat "plain" char declarations as unsigned char. Default: For arguments that you do not specify, the defaults are as follows: Usage: In this example, the compiler allows the asm keyword in the source file. $ pgcc -Masmkeyword myprog.c In the following example, the compiler maps the dollar sign to the dot character. $ pgcc -Mdollar,. myprog.c In the following example, the compiler treats floating-point constants as float values. $ pgcc -Mfcon myprog.c In the following example, the compiler does not convert float parameters to double parameters. $ pgcc -Msingle myprog.c Without –Muchar or with –Mschar, the variable ch is a signed character: char ch; signed char sch; If –Muchar is specified on the command line: $ pgcc -Muchar myprog.c char ch above is equivalent to: unsigned char ch; -M Optimization Controls Syntax: noasmkeyword nosingle dollar,_ scharGeneric PGI Compiler Options 85 -Mcache_align Align unconstrained objects of length greater than or equal to 16 bytes on cache-line boundaries. An unconstrained object is a data object that is not a member of an aggregate structure or common block. This option does not affect the alignment of allocatable or automatic arrays. cache_align Note: To effect cache-line alignment of stack-based local variables, the main program or function must be compiled with –Mcache_align. -Mconcur[=option [,option,...]] Instructs the compiler to enable auto-concurrentization of loops. If - Mconcur is specified, multiple processors will be used to execute loops that the compiler determines to be parallelizable. Where option is one of the following: [no]altcode:n Instructs the parallelizer to generate alternate serial code for parallelized loops. If altcode is specified without arguments, the parallelizer determines an appropriate cutoff length and generates serial code to be executed whenever the loop count is less than or equal to that length. If altcode:n is specified, the serial altcode is executed whenever the loop count is less than or equal to n. If noaltcode is specified, the parallelized version of the loop is always executed regardless of the loop count. cncall Calls in parallel loops are safe to parallelize. Loops containing calls are candidates for parallelization. Also, no minimum loop count threshold must be satisfied before parallelization will occur, and last values of scalars are assumed to be safe.Command Line Options 86 dist:block Parallelize with block distribution (this is the default). Contiguous blocks of iterations of a parallelizable loop are assigned to the available processors. dist:cyclic Parallelize with cyclic distribution. The outermost parallelizable loop in any loop nest is parallelized. If a parallelized loop is innermost, its iterations are allocated to processors cyclically. For example, if there are 3 processors executing a loop, processor 0 performs iterations 0, 3, 6, etc.; processor 1 performs iterations 1, 4, 7, etc.; and processor 2 performs iterations 2, 5, 8, etc. [no]innermost Enable parallelization of innermost loops. The default is to not parallelize innermost loops, since it is usually not profitable on dual-core processors. noassoc Disables parallelization of loops with reductions. When linking, the -Mconcur switch must be specified or unresolved references will result. The NCPUS environment variable controls how many processors or cores are used to execute parallelized loops. Note This option applies only on shared-memory multi-processor (SMP) or multi-core processorbased systems. -Mcray[=option[,option,...]] (pgf77 and pgf95 only) Force Cray Fortran (CF77) compatibility with respect to the listed options. Possible values of option include: pointer for purposes of optimization, it is assumed that pointer-based variables do not overlay the storage of any other variable.Generic PGI Compiler Options 87 -Mdepchk instructs the compiler to assume unresolved data dependencies actually conflict. -Mnodepchk instructs the compiler to assume potential data dependencies do not conflict. However, if data dependencies exist, this option can produce incorrect code. -Mdse Enables a dead store elimination phase that is useful for programs that rely on extensive use of inline function calls for performance. This is disabled by default. -Mnodse (default) Disables the dead store elimination phase. -Mfprelaxed[=option] instructs the compiler to use relaxed precision in the calculation of some intrinsic functions. Can result in improved performance at the expense of numerical accuracy. The possible values for option are: div Perform divide using relaxed precision. sqrt Perform square root with relaxed precision. rsqrt Perform reciprocal square root (1/sqrt) using relaxed precision. With no options, -Mfprelaxed will choose generate relaxed precision code for those operations that generate a significant performance improvement, depending on the target processor. -Mnofprelaxed (default) instructs the compiler not to use relaxed precision in the calculation of intrinsic functions. -Mi4 (pgf77 and pgf95 only) the compiler treats INTEGER variables as INTEGER*4. -Mipa=[,[,…]] Pass options to the interprocedural analyzer. Note: –Mipa implies –O2, and the minimum optimization level that can be specified in combination with –Mipa is –O2. For example, if you specify –Mipa –O1 on the command line, the optimization level will automatically be elevated to –O2 by the compiler driver. It is typical and recommended to Command Line Options 88 use –Mipa=fast. Many of the following sub-options can be prefaced with no, which reverses or disables the effect of the sub-option if it’s included in an aggregate sub-option like –Mipa=fast. The choices of option are: [no]align recognize when targets of a pointer dummy are aligned; default is noalign. [no]arg remove arguments replaced by const, ptr; default is noarg. [no]cg generate call graph information for viewing using the pgicg command-line utility; default is nocg. [no]const perform interprocedural constant propagation; default is const. except: used with inline to specify functions which should not be inlined; default is to inline all eligible functions according to internally defined heuristics. [[no]f90ptr F90/F95 pointer disambiguation across calls; default is nof90ptr fast choose IPA options generally optimal for the target. Use –help to see the settings for –Mipa=fast on a given target. force force all objects to re-compile regardless of whether IPA information has changed. [no]globals optimize references to global variables; default is noglobals. inline[:n] perform automatic function inlining. If the optional :n is provided, limit inlining to at most n levels. IPA-based function inlining is performed from leaf routines upward.Generic PGI Compiler Options 89 ipofile save IPA information in a .ipo file rather than incorporating it into the object file. [no]keepobj keep the optimized object files, using file name mangling, to reduce re-compile time in subsequent builds default is keepobj. [no]libc optimize calls to certain standard C library routines.; default is nolibc. [no]libinline allow inlining of routines from libraries; implies –Mipa=inline; default is nolibinline. [no]libopt allow recompiling and optimization of routines from libraries using IPA information; default is nolibopt. [no]localarg equivalent to arg plus externalization of local pointer targets; default is nolocalarg. main: specify a function to appear as a global entry point; may appear multiple times; disables linking. [no]ptr enable pointer disambiguation across procedure calls; default is noptr. [no]pure pure function detection; default is nopure. required return an error condition if IPA is inhibited for any reason, rather than the default behavior of linking without IPA optimization. safe:[|] declares that the named function, or all functions in the named library, are safe; a safe procedure does not call back into the known procedures and does not Command Line Options 90 change any known global variables. Without –Mipa=safe, any unknown procedures will cause IPA to fail. [no]safeall declares that all unknown procedures are safe; see –Mipa=safe; default is nosafeall. [no]shape perform Fortran 90 array shape propagation; default is noshape. summary only collect IPA summary information when compiling; this prevents IPA optimization of this file, but allows optimization for other files linked with this file. [no]vestigial remove uncalled (vestigial) functions; default is novestigial. -Mlre[=array | assoc | noassoc] Enables loop-carried redundancy elimination, an optimization that can reduce the number of arithmetic operations and memory references in loops. array treat individual array element references as candidates for possible loop-carried redundancy elimination. The default is to eliminate only redundant expressions involving two or more operands. assoc allow expression re-association; specifying this sub-option can increase opportunities for loop-carried redundancy elimination but may alter numerical results. noassoc disallow expression re-association. -Mnolre Disables loop-carried redundancy elimination. -Mnoframe Eliminates operations that set up a true stack frame pointer for every function. With this option enabled, you cannot perform a traceback on the generated code and you cannot access local variables. Generic PGI Compiler Options 91 -Mnoi4 (pgf77 and pgf95 only) the compiler treats INTEGER variables as INTEGER*2. -Mpfi generate profile-feedback instrumentation; this includes extra code to collect run-time statistics and dump them to a trace file for use in a subsequent compilation. –Mpfi must also appear when the program is linked. When the resulting program is executed, a profile feedback trace file pgfi.out is generated in the current working directory; see –Mpfo. Note compiling and linking with –Mpfi adds significant runtime overhead to almost any executable; you should use executables compiled with –Mpfi only for execution of training runs. -Mpfo enable profile-feedback optimizations; requires the presence of a pgfi.out profile-feedback trace file in the current working directory. See –Mpfi. -Mprefetch[=option [,option...]] enables generation of prefetch instructions on processors where they are supported. Possible values for option include: d:m set the fetch-ahead distance for prefetch instructions to m cache lines. n:p set the maximum number of prefetch instructions to generate for a given loop to p. nta use the prefetchnta instruction. plain use the prefetch instruction (default). t0 use the prefetcht0 instruction. w use the AMD-specific prefetchw instruction. -Mnoprefetch Disables generation of prefetch instructions. Command Line Options 92 -Mr8 (pgf77, pgf95 and pghpf only) the compiler promotes REAL variables and constants to DOUBLE PRECISION variables and constants, respectively. DOUBLE PRECISION elements are 8 bytes in length. -Mnor8 (pgf77, pgf95 and pghpf only) the compiler does not promote REAL variables and constants to DOUBLE PRECISION. REAL variables will be single precision (4 bytes in length). -Mr8intrinsics (pgf77, and pgf95 only) the compiler treats the intrinsics CMPLX and REAL as DCMPLX and DBLE, respectively. -Mnor8intrinsics (pgf77, and pgf95 only) the compiler does not promote the intrinsics CMPLX and REAL to DCMPLX and DBLE, respectively. -Msafeptr[=option[,option,...]] (pgcc and pgcpp only) instructs the C/C++ compiler to override data dependencies between pointers of a given storage class. Possible values of option include: all assume all pointers and arrays are independent and safe for aggressive optimizations, and in particular that no pointers or arrays overlap or conflict with each other. arg instructs the compiler that arrays and pointers are treated with the same copyin and copyout semantics as Fortran dummy arguments. global instructs the compiler that global or external pointers and arrays do not overlap or conflict with each other and are independent. local/auto instructs the compiler that local pointers and arrays do not overlap or conflict with each other and are independent.Generic PGI Compiler Options 93 static instructs the compiler that static pointers and arrays do not overlap or conflict with each other and are independent. -Mscalarsse Use SSE/SSE2 instructions to perform scalar floating-point arithmetic (this option is valid only on –tp {p7 | k8-32 | k8-64} targets). -Mnoscalarsse Do not use SSE/SSE2 instructions to perform scalar floating-point arithmetic; use x87 instructions instead (this option is not valid in combination with the –tp k8-64 option). -Msmart instructs the compiler driver to invoke a post-pass assembly optimization utility. -Mnosmart instructs the compiler not to invoke an AMD64-specific post-pass assembly optimization utility. -Munroll[=option [,option...]] invokes the loop unroller. This also sets the optimization level to 2 if the level is set to less than 2. The option is one of the following: c:m instructs the compiler to completely unroll loops with a constant loop count less than or equal to m, a supplied constant. If this value is not supplied, the m count is set to 4. n:u instructs the compiler to unroll u times, a loop that is not completely unrolled, or has a non-constant loop count. If u is not supplied, the unroller computes the number of times a candidate loop is unrolled. -Mnounroll instructs the compiler not to unroll loops. -M[no]vect[=option [,option,...]] (disable) enable the code vectorizer, where option is one of the following: altcode Instructs the vectorizer to generate alternate code (altcode) for vectorized loops when appropriate. For each vectorized loop the compiler decides whether to generate altcode and what type or types to generate, which may be any or Command Line Options 94 all of: altcode without iteration peeling, altcode with non-temporal stores and other data cache optimizations, and altcode based on array alignments calculated dynamically at runtime. The compiler also determines suitable loop count and array alignment conditions for executing the alcode. This option is enabled by default. noaltcode This disables alternate code generation for vectorized loops. assoc Instructs the vectorizer to enable certain associativity conversions that can change the results of a computation due to roundoff error. A typical optimization is to change an arithmetic operation to an arithmetic operation that is mathematically correct, but can be computationally different, due to roundoff error noassoc Instructs the vectorizer to disable associativity conversions. cachesize:n Instructs the vectorizer, when performing cache tiling optimizations, to assume a cache size of n. The default is set using per-processor type, either using the -tp switch or auto-detected from the host computer. [no]sizelimit Generate vector code for all loops where possible regardless of the number of statements in the loop. This overrides a heuristic in the vectorizer that ordinarily prevents vectorization of loops with a Generic PGI Compiler Options 95 number of statements that exceeds a certain threshold. The default is nosizelimit. smallvect[:n] Instructs the vectorizer to assume that the maximum vector length is less than or equal to n. The vectorizer uses this information to eliminate generation of the stripmine loop for vectorized loops wherever possible. If the size n is omitted, the default is 100. Note: No space is allowed on either side of the colon (:). sse Instructs the vectorizer to search for vectorizable loops and, where possible, make use of SSE, SSE2 and prefetch instructions. -Mnovect instructs the compiler not to perform vectorization; can be used to override a previous instance of –Mvect on the command-line, in particular for cases where –Mvect is included in an aggregate option such as –fastsse. -Mnovintr instructs the compiler not to perform idiom recognition or introduce calls to hand-optimized vector functions. Default: For arguments that you do not specify, the default optimization control options are as follows: depchk noprefetch i4 nounroll nofprelaxed novect noipa nor8 nolre nor8intrinsicsCommand Line Options 96 If you do not supply an option to –Mvect, the compiler uses defaults that are dependent upon the target system. Usage: In this example, the compiler invokes the vectorizer with use of packed SSE instructions enabled. $ pgf95 -Mvect=sse -Mcache_align myprog.f Cross-reference: –g, –O -M Miscellaneous Controls Syntax: -Manno annotate the generated assembly code with source code when either the –S or –Mkeepasm options are used. -Mbounds enables array bounds checking. If an array is an assumed size array, the bounds checking only applies to the lower bound. If an array bounds violation occurs during execution, an error message describing the error is printed and the program terminates. The text of the error message includes the name of the array, the location where the error occurred (the source file and the line number in the source), and information about the out of bounds subscript (its value, its lower and upper bounds, and its dimension). For example: PGFTN-F-Subscript out of range for array a (a.f: 2) subscript=3, lower bound=1, upper bound=2, dimension=2 -Mnobounds disables array bounds checking. -Mbyteswapio swap byte-order from big-endian to little-endian or vice versa upon input/output of Fortran unformatted data files. -Mchkfpstk instructs the compiler to check for internal consistency of the x87 floating-point stack in the prologue of a function and after returning from a function or subroutine call. Floating-point stack corruption may occur in many ways, one of which is Fortran code calling floating-point functions as subroutines (i.e., with the CALL statement). If the PGI_CONTINUE environment variable is set upon execution of a program compiled with –Mchkfpstk, the stack will be automatically cleaned up and execution will continue. There is a performance penalty Generic PGI Compiler Options 97 associated with the stack cleanup. If PGI_CONTINUE is set to verbose, the stack will be automatically cleaned up and execution will continue after printing of a warning message. -Mchkptr instructs the compiler to check for pointers that are de-referenced while initialized to NULL (pgf95 and pghpf only). -Mchkstk instructs the compiler to check the stack for available space in the prologue of a function and before the start of a parallel region. Prints a warning message and aborts the program gracefully if stack space is insufficient. Useful when many local and private variables are declared in an OpenMP program. –Mcpp[=option [,option,...]] run the PGI cpp-like pre-processor without execution of any subsequent compilation steps. This option is useful for generating dependence information to be included in makefiles. option is one or more of the following (Note: only one of the m, md, mm or mmd options can be present; if multiple of these options are listed, the last one listed is accepted and the others are ignored): m print makefile dependencies to stdout. md print makefile dependencies to filename.d, where filename is the root name of the input file being processed. mm print makefile dependencies to stdout, ignoring system include files. mmd print makefile dependencies to filename.d, where filename is the root name of the input file being processed, ignoring system include files. [no]comment (don’t) retain comments in ed output. [suffix:] use as the suffix of the output file containing makefile dependencies.Command Line Options 98 -Mdll (Windows only) link with the DLL versions of the runtime libraries. This flag is required when linking with any DLL built by any of The Portland Group compilers. This option implies -D_DLL, which defines the preprocessor symbol _DLL. -Mgccbug[s] match the behavior of certain gcc bugs. -Minfo[=option [,option,...]] instructs the compiler to produce information on standard error, where option is one of the following: all instructs the compiler to produce all available -Minfo information. inline instructs the compiler to display information about extracted or inlined functions. This option is not useful without either the –Mextract or –Minline option. ipa instructs the compiler to display information about interprocedural optimizations. loop instructs the compiler to display information about loops, such as information on vectorization. opt instructs the compiler to display information about optimization. mp instructs the compiler to display information about parallelization. time instructs the compiler to display compilation statistics. unroll instructs the compiler to display information about loop unrolling.Generic PGI Compiler Options 99 -Mneginfo[=option [,option,...]] neginfo instructs the compiler to produce information on standard error, where option is one of the following: all instructs the compiler to produce all available information on why various optimizations are not performed. concur instructs the compiler to produce all available information on why loops are not automatically parallelized. In particular, if a loop is not parallelized due to potential data dependence, the variable(s) that cause the potential dependence will be listed in the -Mneginfo messages. loop instructs the compiler to produce information on why memory hierarchy optimizations on loops are not performed. -Minform,level instructs the compiler to display error messages at the specified and higher levels, where level is one of the following:inform fatal instructs the compiler to display fatal error messages. severe instructs the compiler to display severe and fatal error messages. warn instructs the compiler to display warning, severe and fatal error messages. inform informinstructs the compiler to display all error messages (inform, warn, severe and fatal). -Mkeepasm instructs the compiler to keep the assembly file as compilation continues. Normally, the assembler deletes this file when it is finished. The assembly file has the same filename as the source file, but with a .s extension.Command Line Options 100 -Mlist instructs the compiler to create a listing file. The listing file is filename.lst, where the name of the source file is filename.f. -Mnolist the compiler does not create a listing file. This is the default. -Mmakedll (Windows only) generate a dynamic link library (DLL). -Mnoopenmp when used in combination with the -mp option, causes the compiler to ignore OpenMP parallelization directives or pragmas, but still process SGI-style parallelization directives or pragmas. -Mnosgimp when used in combination with the -mp option, causes the compiler to ignore SGI-style parallelization directives or pragmas, but still process OpenMP parallelization directives or pragmas. -Mnopgdllmain (Windows only) do not link the module containing the default DllMain() into the DLL. This flag applies to building DLLs with the PGF95 and PGHPF compilers. If you want to replace the default DllMain() routine with a custom DllMain(), use this flag and add the object containing the custom DllMain() to the link line. The latest version of the default DllMain() used by PGF95 and PGHPF is included in the Release Notes for each release; the PGF95- and PGHPF-specific code in this routine must be incorporated into the custom version of DllMain() to ensure the appropriate function of your DLL. -Mpreprocess perform cpp-like pre-processing on assembly and Fortran input source files. Default: For arguments that you do not specify, the default miscellaneous options are as follows: Usage: In the following example, the compiler includes Fortran source code with the assembly code. $ pgf95 -Manno -S myprog.f In the following example, the compiler displays information about inlined functions with fewer than approximately 20 source lines in the source file myprog.f. inform warn nolist noboundsGeneric PGI Compiler Options 101 $ pgf95 -Minfo=inline -Minline=20 myprog.f In the following example, the assembler does not delete the assembly file myprog.s after the assembly pass. $ pgf95 -Mkeepasm myprog.f In the following example, the compiler creates the listing file myprog.lst. $ pgf95 -Mlist myprog.f In the following example, array bounds checking is enabled. $ pgf95 -Mbounds myprog.f Cross-reference: –m, –S, –V, –v -mcmodel=medium (For use only on 64-bit Linux targets) Generate code for the medium memory model in the linux86-64 execution environment. Implies –Mlarge_arrays. The default small memory model of the linux86-64 environment limits the combined area for a user’s object or executable to 1GB, with the Linux kernel managing usage of the second 1GB of address for system routines, shared libraries, stacks, etc. Programs are started at a fixed address, and the program can use a single instruction to make most memory references. The medium memory model allows for larger than 2GB data areas, or .bss sections. Program units compiled using either –mcmodel=medium or –fpic require additional instructions to reference memory. The effect on performance is a function of the data-use of the application. The – mcmodel=medium switch must be used at both compile time and link time to create 64-bit executables. Program units compiled for the default small memory model can be linked into medium memory model executables as long as they are compiled –fpic, or position-independent. The linux86-64 environment provides static libxxx.a archive libraries that are built with and without –fpic, and dynamic libxxx.so shared object libraries that are compiled –fpic. The –mcmodel=medium linkswitch implies the –fpic switch and will utilize the shared libraries by default. Similarly, the $PGI/ linux86-64//lib directory contains the libraries for building small memory model codes, and the $PGI/linux86-64//libso directory contains shared libraries for building –mcmodel=medium Command Line Options 102 and –fpic executables. Note: It appears from the GNU tools and documentation that creation of medium memory model shared libraries is not supported. However, you can create static archive libraries (.a) that are –fpic. Default: The compiler generates code for the small memory model. Usage: The following command line requests position independent code be generated, and the – mcmodel=medium option be passed to the assembler and linker: $ pgf95 -mcmodel=medium myprog.f -module Use the -module option to specify a particular directory in which generated intermediate .mod files should be placed. If the -module option is present, and USE statements are present in a compiled program unit, will search for .mod intermediate files prior to the search in the default (local) directory. Default: The compiler places .mod files in the current working directory, and searches only in the current working directory for pre-compiled intermediate .mod files. Usage: The following command line requests that any intermediate module file produced during compilation of myprog.f be placed in the directory mymods (in particular, the file ./mymods/ myprog.mod will be used): $ pgf95 -module mymods myprog.f -mp[=align,[no]numa] Use the -mp option to instruct the compiler to interpret user-inserted OpenMP shared-memory parallel programming directives and generate an executable file which will utilize multiple processors in a shared-memory parallel system. See OpenMP Directives for Fortranand OpenMP Pragmas for C and C++, for a detailed description of this programming model and the associated directives and pragmas. The align sub-option forces loop iterations to be allocated to OpenMP processes using an algorithm that maximizes alignment of vector sub-sections in loops that are both parallelized and vectorized for SSE. This can improve performance in program units that include many such loops. It can result in load-balancing problems that significantly decrease performance in program units with relatively short loops that contain a large amount of work in each iteration. The numa suboption uses libnuma on systems where it is available.Generic PGI Compiler Options 103 Default: The compiler ignores user-inserted shared-memory parallel programming directives and pragmas. Usage: The following command line requests processing of any shared-memory directives present in myprog.f: $ pgf95 -mp myprog.f Cross-reference: –Mconcur and –Mvect -nfast A generally optimal set of options is chosen depending on the target system. In addition, the appropriate –tp option is automatically included to enable generation of code optimized for the type of system on which compilation is performed. Note Auto-selection of the appropriate –tp option means that programs built using the –fast option on a given system are not necessarily backward-compatible with older systems. Cross-reference: –O, –Munroll, –Mnoframe, –Mvect, –tp, –Mscalarsse -O Invokes code optimization at the specified level. Syntax: –O [level] Where level is one of the following: 0 creates a basic block for each statement. Neither scheduling nor global optimization is done. To specify this level, supply a 0 (zero) argument to the –O option. 1 schedules within basic blocks and performs some register allocations, but does no global optimization. 2 performs all level-1 optimizations, and also performs global scalar optimizations such as induction variable elimination and loop invariant movement.Command Line Options 104 3 level-three specifies aggressive global optimization. This level performs all level-one and level-two optimizations and enables more aggressive hoisting and scalar replacement optimizations that may or may not be profitable. 4 level-four performs all level-one, level-two, and level-three optimizations and enables hoisting of guarded invariant floating point expressions. Default: This table shows the interaction between the –O option, –g option, –Mvect, and –Mconcur options. Table 3-4: Optimization and –O, –g, –Mvect, and –Mconcur Options Unoptimized code compiled using the option –O0 can be significantly slower than code generated at other optimization levels. Like the –Mvect option, the –Munroll option sets the optimization level to level-2 if no –O or –g options are supplied. The -gopt option is recommended for generation of debug information with optimized code. For more information on optimization, see Optimization & Parallelization. Usage: In the following example, since no optimization level is specified and a –O option is specified, the compiler sets the optimization to level-2. Optimize Option Debug Option –M Option Optimization Level none none none 1 none none –Mvect 2 none none –Mconcur 2 none –g none 0 –O none or –g none 2 –Olevel none or –g none level –Olevel < 2 none or –g –Mvect 2 –Olevel < 2 none or –g –Mconcur 2Generic PGI Compiler Options 105 $ pgf95 -O myprog.f Cross-reference: –g, –M, –gopt -o Names the executable file. Use the –o option to specify the filename of the compiler object file. The final output is the result of linking. Syntax: –o filename Where filename is the name of the file for the compilation output. The filename must not have a .f extension. Default: The compiler creates executable filenames as needed. If you do not specify the –o option, the default filename is the linker output file a.out. Usage: In the following example, the executable file is myprog instead of the default a.out. $ pgf95 myprog.f -o myprog Cross-reference: –c ,–E, –F, –S -pc (–tp px/p5/p6/piii targets only) The –pc option can be used to control the precision of operations performed using the x87 floating point unit, and their representation on the x87 floating point stack. Syntax: –pc { 32 | 64 | 80 } The x87 architecture implements a floating-point stack using 8 80-bit registers. Each register uses bits 0-63 as the significand, bits 64-78 for the exponent, and bit 79 is the sign bit. This 80-bit real format is the default format (called the extended format). When values are loaded into the floating point stack they are automatically converted into extended real format. The precision of the floating point stack can be controlled, however, by setting the precision control bits (bits 8 and 9) of the floating control word appropriately. In this way, you can explicitly set the precision to standard IEEE double-precision using 64 bits, or to single precision using 32 bits. 1 The default precision is system dependent. To alter Command Line Options 106 the precision in a given program unit, the main program must be compiled with the same -pc option. The command line option –pc val lets the programmer set the compiler’s precision preference. Valid values for val are: • 32 single precision • 64 double precision • 80 extended precision Extended Precision Option – Operations performed exclusively on the floating-point stack using extended precision, without storing into or loading from memory, can cause problems with accumulated values within the extra 16 bits of extended precision values. This can lead to answers, when rounded, that do not match expected results. For example, if the argument to sin is the result of previous calculations performed on the floatingpoint stack, then an 80-bit value used instead of a 64-bit value can result in slight discrepancies. Results can even change sign due to the sin curve being too close to an x-intercept value when evaluated. To maintain consistency in this case, you can assure that the compiler generates code that calls a function. According to the x86 ABI, a function call must push its arguments on the stack (in this way memory is guaranteed to be accessed, even if the argument is an actual constant.) Thus, even if the called function simply performs the inline expansion, using the function call as a wrapper to sin has the effect of trimming the argument precision down to the expected size. Using the -Mnobuiltin option on the command line for C accomplishes this task by resolving all math routines in the library libm, performing a function call of necessity. The other method of generating a function call for math routines, but one that may still produce the inline instructions, is by using the -Kieee switch. A second example illustrates the precision control problem using a section of code to determine machine precision: program find_precision w = 1.0 100 w=w+w y=w+1 z=y-w if (z .gt. 0) goto 100 1. According to Intel documentation, this only affects the x87 operations of add, subtract, multiply, divide, and square root. In particular, it does not appear to affect the x87 transcendental instructions.Generic PGI Compiler Options 107 C now w is just big enough that |((w+1)-w)-1| >= 1 ... print*,w end In this case, where the variables are implicitly real*4, operations are performed on the floating-point stack where optimization removed unnecessary loads and stores from memory. The general case of copy propagation being performed follows this pattern: a = x y = 2.0 + a Instead of storing x into a, then loading a to perform the addition, the value of x can be left on the floating-point stack and added to 2.0. Thus, memory accesses in some cases can be avoided, leaving answers in the extended real format. If copy propagation is disabled, stores of all left-hand sides will be performed automatically and reloaded when needed. This will have the effect of rounding any results to their declared sizes. For the above program, w has a value of 1.8446744E+19 when executed using default (extended) precision. If, however, -Kieee is set, the value becomes 1.6777216E+07 (single precision.) This difference is due to the fact that -Kieee disables copy propagation, so all intermediate results are stored into memory, then reloaded when needed. Copy propagation is only disabled for floating-point operations, not integer. With this particular example, setting the -pc switch will also adjust the result. The switch -Kieee also has the effect of making function calls to perform all transcendental operations. Although the function still produces the x86 machine instruction for computation (unless in C the -Mnobuiltin switch is set), arguments are passed on the stack, which results in a memory store and load. Finally, -Kieee also disables reciprocal division for constant divisors. That is, for a/b with unknown a and constant b, the expression is usually converted at compile time to a*(1/b), thus turning an expensive divide into a relatively fast scalar multiplication. However, numerical discrepancies can occur when this optimization is used. Understanding and correctly using the -pc, -Mnobuiltin, and -Kieee switches should enable you to produce the desired and expected precision for calculations which utilize floating-point operations. Usage: $ pgf95 -pc 64 myprog.cCommand Line Options 108 -pg (Linux only) Instructs the compiler to instrument the generated executable for gprof-style samplebased profiling. Must be used at both the compile and link steps. A gmon.out style trace is generated when the resulting program is executed, and and can be analyzed using gprof or pgprof. Syntax: –pg Default: The compiler does not instrument the generated executable for gprof-style profiling. -Q Selects variations for compilation. There are four uses for the –Q option. Syntax: -Qdirdirectory The first variety, using the dir keyword, lets you supply a directory parameter that indicates the directory where the compiler driver is located. -Qoptionprog,opt The second variety, using the option keyword, lets you supply the option opt to the program prog. The prog parameter can be one of pgftn, as, or ld. -Qpathpathname The third –Q variety, using the path keyword, lets you supply an additional pathname to the search path for the compiler’s required .o files. -Qproducesourcetype The fourth –Q variety, using the produce keyword, lets you choose a stop-after location for the compilation based on the supplied sourcetype parameter. Valid sourcetypes are: .i, .c, .s and .o. These indicate respectively, stop-after preprocessing, compiling, assembling, or linking. Usage: The following examples show the different –Q options.Generic PGI Compiler Options 109 $ pgf95 -Qproduce .s hello.f $ pgf95 -Qoption ld,-s hello.f $ pgf95 -Qpath /home/test hello.f $ pgf95 -Qdir /home/comp/new hello.f Cross-reference: –p -R Valid only on Linux and is passed to the linker. Instructs the linker to hard-code the pathname into the search path for generated shared object (dynamically linked library) files. Note that there cannot be a space between R and . Cross-reference: –fpic, –shared, –G -r4 and -r8 Interpret DOUBLE PRECISION variables as REAL (–r4) or REAL variables as DOUBLE PRECISION (– r8). Usage: $ pgf95 -r4 myprog.f Cross-reference: –i2, –i4, -i8, -nor8 -rc Specifies the name of the driver startup configuration file. If the file or pathname supplied is not a full pathname, the path for the configuration file loaded is relative to the $DRIVER path (the path of the currently executing driver). If a full pathname is supplied, that file is used for the driver configuration file. Syntax: –rc [path] filename Where path is either a relative pathname, relative to the value of $DRIVER, or a full pathname beginning with "/". Filename is the driver configuration file. Default: The driver uses the configuration file .pgirc.Command Line Options 110 Usage: In the following example, the file .pgf95rctest, relative to /usr/pgi/linux86/bin, the value of $DRIVER, is the driver configuration file. $ pgf95 -rc .pgf95rctest myprog.f Cross-reference: –show -S Stops compilation after the compiling phase and writes the assembly-language output to the file filename.s, where the input file is filename.f. Default: The compiler produces an executable file. Usage: In this example, pgf95 produces the file myprog.s in the current directory. $ pgf95 -S myprog.f Cross-reference: –c, –E, –F, –Mkeepasm, –o -shared Valid only on Linux and is passed to the linker. Instructs the linker to produce a shared object (dynamically linked library) file. Cross-reference: –fpic, –G, –R -show Produce driver help information describing the current driver configuration. Usage: In the following example, the driver displays configuration information to the standard output after processing the driver configuration file. $ pgf95 -show myprog.f Cross-reference: –V , –v, –###, –help, –rc -silent Do not print warning messages. Usage: In the following example, the driver does not display warning messages. $ pgf95 -silent myprog.fGeneric PGI Compiler Options 111 Cross-reference: -v, -V, -w -soname (Linux only.) The compiler recognizes the -soname option and passes it through to the linker. Usage: In the following example, the driver passes the soname option and its argument through to the linker. $ pgf95 -soname library.so myprog.f -time Print execution times for various compilation steps. Usage: In the following example, pgf95 prints the execution times for the various compilation steps. $ pgf95 -time myprog.f Cross-reference: –# -tp [,target...] Set the target architecture. By default, the PGI compilers produce code specifically targeted to the type of processor on which the compilation is performed. In particular, the default is to use all supported instructions wherever possible when compiling on a given system. As a result, executables created on a given system may not be useable on previous generation systems (for example, executables created on a Pentium 4 may fail to execute on a Pentium III or Pentium II). Processor-specific optimizations can be specified or limited explicitly by using the -tp option. In this way, it is possible to create executables that are usable on previous generation systems. With the exception of k8-64, k8-64e, p7-64, and x64, any of these sub-options are valid on any x86 or x64 processor-based system. The k8-64, k8-64e, p7-64 and x64 options are valid only on x64 processorbased systems. The –tp x64 option is used to generate unified binary object and executable files. The –tp k8-64 and – tp k8-64e options result in generation of code supported on and optimized for AMD x64 processors, while the –tp p7-64 option results in generation of code that is supported on and optimized for Intel x64 processors. Performance of k8-64 or k8-64e code executed on Intel x64 processors, or of p7-64 code executed on AMD x64 processors, can often be significantly less than that obtained with a native binary. The –tp x64 option results in generation of unified binary object and executable files which are supported on and include optimized code sequences for both AMD and Intel x64 processors. Command Line Options 112 Following is a list of possible sub-options to –tp and the processors they are intended to target: k8-32 generate 32-bit code for AMD Athlon64, AMD Opteron and compatible processors. k8-64 generate 64-bit code for AMD Athlon64, AMD Opteron and compatible processors. k8-64e generate 64-bit code for AMD Opteron Revision E, AMD Turion, and compatible processors. p6 generate 32-bit code for Pentium Pro/II/III and AthlonXP compatible processors. p7 generate 32-bit code for Pentium 4 and compatible processors. p7-64 generate 64-bit code for Intel P4/Xeon EM64T and compatible processors. core2 generate 32-bit code for Intel Core 2 Duo and compatible processors. core2-64 generate 64-bit code for Intel Core 2 Duo EM64T and compatible processors. piii generate 32-bit code for Pentium III and compatible processors, including support for single-precision vector code using SSE instructions. px generate 32-bit code that is useable on any x86 processor-based system. x64 generate 64-bit unified binary code including full optimizations and support for both AMD and Intel x64 processors. See Table 2 , “Processor Options” for a concise list of the features of these processors that distinguish them as separate targets when using the PGI compilers and tools. Syntax for 64-bit targets: -tp {k8-64 | k8-64e | p7-64 | core2-64 | x64} Syntax for 32-bit targets: -tp {k8-32 | p6 | p7 | core2 | piii | px}Generic PGI Compiler Options 113 Usage: In the following example, pgf95 sets the target architecture to EM64T: $ pgf95 -tp p7-64 myprog.f Default: The default style of code generation is auto-selected depending on the type of processor on which compilation is performed. The –tp x64 style of unified binary code generation is only enabled by an explicit –tp x64 option. Using -tp to Generate a Unified Binary All 64-bit PGI compilers can produce PGI Unified Binary programs containing code streams fully optimized and supported for both AMD64 and Intel EM64T processors using the -tp target option. The compilers generate and combine into one executable multiple binary code streams each optimized for a specific platform. At runtime, this one executable senses the environment and dynamically selects the appropriate code stream. Different processors have subtle and not-so-subtle differences in hardware features such as instruction sets and cache size. The compilers make architecture-specific decisions about such things as instruction selection, instruction scheduling, and vectorization, all of which can have significant effects on performance and compatibility. PGI unified binaries provide a low-overhead means for a single program to run well on a number of hardware platforms. The target processor switch, -tp, accepts a comma-separated list of 64-bit targets and will generate code optimized for each listed target. For example, -tp k8-64,p7-64,core2-64 generates optimized code for three targets. A special target switch, -tp x64, is the same as -tp k8-64,p7- 64. See “Processor-Specific Optimization and the Unified Binary” on page 35 for more information on unified binaries. -U Undefines a preprocessor macro. Use the –U option or the #undef preprocessor directive to undefine macros. Syntax: -Usymbol Where symbol is a symbolic name.Command Line Options 114 Usage: The following examples undefine the macro test. $ pgf95 -Utest myprog.F $ pgf95 -Dtest -Utest myprog.F Cross-reference: –D,–Mnostddef. -V[release_number] Displays additional information, including version messages. If a release_number is appended, the compiler driver will attempt to compile using the specified release instead of the default release. There can be no space between –V and release_number. The specified release must be co-installed with the default release, and must have a release number greater than or equal to 4.1 (the first release for which this functionality is supported). Usage: The following command-line shows the output using the -V option. % pgf95 -V myprog.f The following command-line causes PGF95 to compile using the 5.2 release instead of the default: % pgcc -V5.2 myprog.c Cross-reference: –Minfo, –v -v Use the –v option to display the invocations of the compiler, assembler, and linker. These invocations are command lines created by the compiler driver from the files and the –W options you specify on the compiler command-line. Default: The compiler does not display individual phase invocations. Cross-reference: –Minfo, –V -W Passes arguments to a specific phase. Use the –W option to specify options for the assembler, compiler or linker. Note: A given PGI compiler command invokes the compiler driver, which parses the command-line and generates the appropriate commands for the compiler, assembler and linker. Syntax: –W {0 | a | l },option[,option...]C and C++ -specific Compiler Options 115 Where: 0 (the number zero) specifies the compiler. a specifies the assembler. l (lowercase letter l) specifies the linker. option is a string that is passed to and interpreted by the compiler, assembler or linker. Options separated by commas are passed as separate command line arguments. Note You cannot have a space between the –W and the single-letter pass identifier, between the identifier and the comma, or between the comma and the option. Usage: In the following example the linker loads the text segment at address 0xffc00000 and the data segment at address 0xffe00000. $ pgf95 -Wl,-k,-t,0xffc00000,-d,0xffe00000 myprog.f -w Do not print warning messages. C and C++ -specific Compiler Options The following options are specific to PGCC C and/or C++. -A (pgcpp only) Using this option, the PGC++ compiler accepts code conforming to the proposed ANSI C++ standard. It issues errors for non-conforming code. Default: By default, the compiler accepts code conforming to the standard C++ Annotated Reference Manual. Usage: The following command-line requests ANSI conforming C++. $ pgcpp -A hello.cc Cross-references: –b and +p.Command Line Options 116 --[no_]alternative_tokens (pgcpp only) Enable or disable recognition of alternative tokens. These are tokens that make it possible to write C++ without the use of the , , [, ], #, &, , ^, and characters. The alternative tokens include the operator keywords (e.g., and, bitand, etc.) and digraphs. The default behavior is -- no_alternative_tokens. -B (pgcc and pgcpp only) Enable use of C++ style comments starting with // in C program units. Default: The PGCC ANSI and K&R C compiler does not allow C++ style comments. Usage: In the following example the compiler accepts C++ style comments. $ pgcc -B myprog.cc -b (pgcpp only) Enable compilation of C++ with cfront 2.1 compatibility. This causes the compiler to accept language constructs that, while not part of the C++ language definition, are accepted by the AT&T C++ Language System (cfront release 2.1). This option also enables acceptance of anachronisms. Default: The compiler does not accept cfront language constructs that are not part of the C++ language definition. Usage: In the following example the compiler accepts cfront constructs. $ pgcpp -b myprog.cc Cross-references: ––cfront2.1, –b3 , ––cfront3.0, +p, –A -b3 (pgcpp only) Enable compilation of C++ with cfront 3.0 compatibility. This causes the compiler to accept language constructs that, while not part of the C++ language definition, are accepted by the AT&T C++ Language System (cfront release 3.0). This option also enables acceptance of anachronisms.C and C++ -specific Compiler Options 117 Default: The compiler does not accept cfront language constructs that are not part of the C++ language definition. Usage: In the following example, the compiler accepts cfront constructs. $ pgcpp -b3 myprog.cc Cross-references: ––cfront2.1, –b , ––cfront3.0 , +p, –A --[no_]bool (pgcpp only) Enable or disable recognition of bool. The default value is --bool. --cfront_2.1 (pgcpp only) Enable compilation of C++ with cfront 2.1 compatibility. This causes the compiler to accept language constructs that, while not part of the C++ language definition, are accepted by the AT&T C++ Language System (cfront release 2.1). This option also enables acceptance of anachronisms. Default: The compiler does not accept cfront language constructs that are not part of the C++ language definition. Usage: In the following example, the compiler accepts cfront constructs. $ pgcpp --cfront_2.1 myprog.cc Cross-references: –b, –b3 , ––cfront3.0, +p, –A --cfront_3.0 (pgcpp only) Enable compilation of C++ with cfront 3.0 compatibility. This causes the compiler to accept language constructs that, while not part of the C++ language definition, are accepted by the AT&T C++ Language System (cfront release 3.0). This option also enables acceptance of anachronisms. Default: The compiler does not accept cfront language constructs that are not part of the C++ language definition. Usage: In the following example, the compiler accepts cfront constructs. $ pgcpp --cfront_3.0 myprog.ccCommand Line Options 118 Cross-references: ––cfront2.1, –b , –b3 , +p, –A --create_pch filename (pgcpp only) If other conditions are satisfied, create a precompiled header file with the specified name. If --pch (automatic PCH mode) appears on the command line following this option, its effect is erased. --diag_suppress tag (pgcpp only) Override the normal error severity of the specified diagnostic messages. The message(s) may be specified using a mnemonic error tag or using an error number. --diag_remark tag (pgcpp only) Override the normal error severity of the specified diagnostic messages. The message(s) may be specified using a mnemonic error tag or using an error number. --diag_warning tag (pgcpp only) Override the normal error severity of the specified diagnostic messages. The message(s) may be specified using a mnemonic error tag or using an error number. --diag_error tag (pgcpp only) Override the normal error severity of the specified diagnostic messages. The message(s) may be specified using a mnemonic error tag or using an error number. --display_error_number (pgcpp only) Display the error message number in any diagnostic messages that are generated. The option may be used to determine the error number to be used when overriding the severity of a diagnostic message. --[no_]exceptions (pgcpp only) Enable/disable exception handling support. The default is --exceptions. --[no]llalign (pgcpp only) Do/don’t align long long integers on long long boundaries. The default is --llalign.C and C++ -specific Compiler Options 119 -M Generate a list of make dependencies and print them to stdout. Compilation stops after the preprocessing phase. -MD Generate a list of make dependencies and print them to the file .d, where is the name of the file under compilation. dependencies_file --optk_allow_dollar_in_id_chars (pgcpp only) Accept dollar signs ($) in identifiers. -P Stops compilation after the preprocessing phase. Use the –P option to halt the compilation process after preprocessing and write the preprocessed output to the file filename.i, where the input file is filename.c or filename.cc. Use the -suffix option with this option to save the intermediate file in a file with the specified suffix. Default: The compiler produces an executable file. Usage: In the following example, the compiler produces the preprocessed file myprog.i in the current directory. $ pgcpp -P myprog.cc Cross-references: –C,–c,–E, –Mkeepasm, –o, –S --pch (pgcpp only) Automatically use and/or create a precompiled header file. If --use_pch or --create_pch (manual PCH mode) appears on the command line following this option, its effect is erased. --pch_dir directoryname (pgcpp only) The directory in which to search for and/or create a precompiled header file. This option may be used with automatic PCH mode (--pch) or manual PCH mode (--create_pch or --use_pch).Command Line Options 120 --[no_]pch_messages (pgcpp only) Enable or disable the display of a message indicating that a precompiled header file was created or used in the current compilation. --preinclude= (pgcpp only) Specifies the name of a file to be included at the beginning of the compilation. This option can be used to set system-dependent macros and types, for example. --use_pch filename (pgcpp only) Use a precompiled header file of the specified name as part of the current compilation. If --pch (automatic PCH mode) appears on the command line following this option, its effect is erased. --[no_]using_std (pgcpp only) Enable or disable implicit use of the std namespace when standard header files are included. Default: The default is --using_std. Usage: The following command-line disables implicit use of the std namespace: $ pgcpp --no_using_std hello.cc -t (pgcpp only) Control instantiation of template functions. Syntax: –t [arg] where arg is one of the following: all Instantiates all functions whether or not they are used. local Instantiates only the functions that are used in this compilation, and forces those functions to be local to this compilation. Note: This may cause multiple copies of local static variables. If this occurs, the program may not execute correctly.C and C++ -specific Compiler Options 121 none Instantiates no functions. (this is the default) used Instantiates only the functions that are used in this compilation. Usage: In the following example, all templates are instantiated. $ pgcpp -tall myprog.ccCommand Line Options 122Invoking Function Inlining 123 4 Function Inlining Function inlining replaces a call to a function or a subroutine with the body of the function or subroutine. This can speed up execution by eliminating parameter passing and function/subroutine call and return overhead. It also allows the compiler to optimize the function with the rest of the code. Note that using function inlining indiscriminately can result in much larger code size and no increase in execution speed. The PGI compilers provide two categories of inlining: • Automatic inlining - During the compilation process, a hidden pass precedes the compilation pass. This hidden pass extracts functions that are candidates for inlining. The inlining of functions occurs as the source files are compiled. • Inline libraries - You create inline libraries, for example using the pgf95 command and the – Mextract and –o options. There is no hidden extract pass but you must ensure that any files that depend on the inline library use the latest version of the inline library. There are important restrictions on inlining. Inlining only applies to certain types of functions. Refer to “Restrictions on Inlining” on page 128, at the end of this chapter for more details on function inlining limitations. Invoking Function Inlining To invoke the function inliner, use the –Minline option. If you do not specify an inline library, the compiler performs a special prepass on all source files named on the compiler command line before it compiles any of them. This pass extracts functions that meet the requirements for inlining and puts them in a temporary inline library for use by the compilation pass. Several –Minline options let you determine the selection criteria for functions to be inlined. These selection criteria include: except:func Inline all eligible functions except func, a function in the source text. Multiple functions can be listed, comma-separated. [name:]func A function name, which is a string matching func, a function in the source text.Function Inlining 124 [size:]n A size, which instructs the compiler to select functions with a statement count less than or equal to n, the specified size. Note: the size n may not exactly equal the number of statements in a selected function (the size parameter is used as a rough gauge). levels:n A level number, which represents the number of function calling levels to be inlined. The default number is one (1). Using a level greater than one indicates that function calls within inlined functions may be replaced with inlined code. This allows the function inliner to automatically perform a sequence of inline and extract processes. [lib:]file.ext A library file name. This instructs the inliner to inline the functions within the library file file.ext. Create the library file using the –Mextract option. If no inline library is specified, functions are extracted from a temporary library created during an extract prepass. If you specify both a function name and a size n, the compiler inlines functions that match the function name or have n or fewer statements. If a keyword name:, lib: or size: is omitted, then a name with a period is assumed to be an inline library, a number is assumed to be a size, and a name without a period is assumed to be a function name. In the following example, the compiler inlines functions with fewer than approximately 100 statements in the source file myprog.f and writes the executable code in the default output file a.out. $ pgf95 -Minline=size:100 myprog.f Refer to 3, “Command Line Options” for more information on the –Minline options. Using an Inline Library If you specify one or more inline libraries on the command line with the –Minline option, the compiler does not perform an initial extract pass. The compiler selects functions to inline from the specified inline library. If you also specify a size or function name, all functions in the inline library meeting the selection criteria are selected for inline expansion at points in the source text where they are called. If you do not specify a function name or a size limitation for the –Minline option, the compiler inlines every function in the inline library that matches a function in the source text.Creating an Inline Library 125 In the following example, the compiler inlines the function proc from the inline library lib.il and writes the executable code in the default output file a.out. $ pgf95 -Minline=name:proc,lib:lib.il myprog.f The following command line is equivalent to the line above, the only difference in this example is that the name: and lib: inline keywords are not used. The keywords are provided so you can avoid name conflicts if you use an inline library name that does not contain a period. Otherwise, without the keywords, a period lets the compiler know that the file on the command line is an inline library. $ pgf95 -Minline=proc,lib.il myprog.f Creating an Inline Library You can create or update an inline library using the –Mextract command-line option. If you do not specify a selection criteria along with the –Mextract option, the compiler attempts to extract all subprograms. When you use the –Mextract option, only the extract phase is performed; the compile and link phases are not performed. The output of an extract pass is a library of functions available for inlining. It is placed in the inline library file specified on the command line with the –o filename specification. If the library file exists, new information is appended to it. If the file does not exist, it is created. You can use the –Minline option with the –Mextract option. In this case, the extracted library of functions can have other functions inlined into the library. Using both options enables you to obtain more than one level of inlining. In this situation, if you do not specify a library with the –Minline option, the inline process consists of two extract passes. The first pass is a hidden pass implied by the – Minline option, during which the compiler extracts functions and places them into a temporary library. The second pass uses the results of the first pass but puts its results into the library that you specify with the –o option. Working with Inline Libraries An inline library is implemented as a directory with each inline function in the library stored as a file using an encoded form of the inlinable function.Function Inlining 126 A special file named TOC in the inline library directory serves as a table of contents for the inline library. This is a printable, ASCII file which can be examined to find out information about the library contents, such as names and sizes of functions, the source file from which they were extracted, the version number of the extractor which created the entry, etc. Libraries and their elements can be manipulated using ordinary system commands. • Inline libraries can be copied or renamed. • Elements of libraries can be deleted or copied from one library to another. • The ls command can be used to determine the last-change date of a library entry. Dependencies in Makefiles–When a library is created or updated using one of the PGI compilers, the last-change date of the library directory is updated. This allows a library to be listed as a dependence in a makefile (and ensures that the necessary compilations will be performed when a library is changed). Updating Inline Libraries - Makefiles If you use inline libraries you need to be certain that they remain up to date with the source files into which they are inlined. One way to assure inline libraries are updated is to include them in a makefile. The makefile fragment in the following example assumes the file utils.f contains a number of small functions used in the files parser.f and alloc.f. The makefile also maintains the inline library utils.il. The makefile updates the library whenever you change utils.f or one of the include files it uses. In turn, the makefile compiles parser.f and alloc.f whenever you update the library. Example 4-1: Sample Makefile SRC = mydir FC = pgf95 FFLAGS = -O2 main.o: $(SRC)/main.f $(SRC)/global.h $(FC) $(FFLAGS) -c $(SRC)/main.f utils.o: $(SRC)/utils.f $(SRC)/global.h $(SRC)/utils.h $(FC) $(FFLAGS) -c $(SRC)/utils.f utils.il: $(SRC)/utils.f $(SRC)global.h $(SRC)/utils.h $(FC) $(FFLAGS) -Mextract=15 -o utils.il parser.o: $(SRC)/parser.f $(SRC)/global.h utils.il $(FC) $(FFLAGS) -Minline=utils.il -c Error Detection during Inlining 127 $(SRC)/parser.f alloc.o: $(SRC)/alloc.f $(SRC)/global.h utils.il $(FC) $(FFLAGS) -Minline=utils.il -c $(SRC)/alloc.f myprog: main.o utils.o parser.o alloc.o $(FC) -o myprog main.o utils.o parser.o alloc.o Error Detection during Inlining To request inlining information from the compiler when you invoke the inliner, specify the – Minfo=inline option. For example: $ pgf95 -Minline=mylib.il -Minfo=inline myext.f Examples Assume the program dhry consists of a single source file dhry.f. The following command line builds an executable file for dhry in which proc7 is inlined wherever it is called: $ pgf95 dhry.f -Minline=proc7 The following command lines build an executable file for dhry in which proc7 plus any functions of approximately 10 or fewer statements are inlined (one level only). Note that the specified functions are inlined only if they are previously placed in the inline library, temp.il, during the extract phase. $ pgf95 dhry.f -Mextract -o temp.il $ pgf95 dhry.f -Minline=10,Proc7,temp.il Assume the program fibo.f contains a single function fibo that calls itself recursively. The following command line creates the file fibo.o in which fibo is inlined into itself: $ pgf95 fibo.f -c -Mrecursive -Minline=fibo Because this version of fibo recurses only half as deeply, it executes noticeably faster. Using the same source file dhry.f, the following example builds an executable for dhry in which all functions of roughly ten or fewer statements are inlined. Two levels of inlining are performed. This means that if function A calls function B, and B calls C, and both B and C are inlinable, then the version of B which is inlined into A will have had C inlined into it. Function Inlining 128 $ pgf95 dhry.f -Minline=size:10,levels:2 Restrictions on Inlining The following Fortran subprograms cannot be extracted: • Main or BLOCK DATA programs. • Subprograms containing alternate return, assigned GO TO, DATA, SAVE, or EQUIVALENCE statements. • Subprograms containing FORMAT statements. • Subprograms containing multiple entries. A Fortran subprogram is not inlined if any of the following applies: • It is referenced in a statement function. • A common block mismatch exists; i.e., the caller must contain all common blocks specified in the callee, and elements of the common blocks must agree in name, order, and type (except that the caller's common block can have additional members appended to the end of the common block). • An argument mismatch exists; i.e., the number and type (size) of actual and formal parameters must be equal. • A name clash exists; e.g., a call to subroutine xyz in the extracted subprogram and a variable named xyz in the caller. The following types of C and C++ functions cannot be inlined: • Functions whose return type is a struct data type, or functions which have a struct argument. This limitation applies only to x86 targets. • Functions containing switch statements • Functions which reference a static variable whose definition is nested within the function • Function which accept a variable number of arguments Certain C/C++ functions can only be inlined into the file that contains their definition: • Static functionsRestrictions on Inlining 129 • Functions which call a static function • Functions which reference a static variableFunction Inlining 130Parallelization Directives 131 5 OpenMP Directives for Fortran The PGF77 and PGF95 Fortran compilers support the OpenMP Fortran Application Program Interface. The OpenMP shared-memory parallel programming model is defined by a collection of compiler directives, library routines, and environment variables that can be used to specify shared-memory parallelism in Fortran, C and C++ programs. The directives include a parallel region construct for writing coarse grain SPMD programs, work-sharing constructs which specify that DO loop iterations should be split among the available threads of execution, and synchronization constructs. The data environment is controlled using clauses on the directives or with additional directives. Run-time library routines are provided to query the parallel runtime environment, for example to determine how many threads are participating in execution of a parallel region. Finally, environment variables are provided to control the execution behavior of parallel programs. For more information on OpenMP, see http:// www.openmp.org. For an introduction to how to execute programs that use multiple processors along with some pointers to example code, see “Parallel Programming Using the PGI Compilers” on page 8. Parallelization Directives Parallelization directives are comments in a program that are interpreted by the PGI Fortran compilers when the option -mp is specified on the command line. The form of a parallelization directive is: sentinel directive_name[clauses] With the exception of the SGI-compatible DOACROSS directive, the sentinel must be !$OMP, C$OMP, or *$OMP, must start in column 1 (one), and must appear as a single word without embedded white space. The sentinel marking a DOACROSS directive is C$. Standard Fortran syntax restrictions (line length, case insensitivity, etc.) apply to the directive line. Initial directive lines must have a space or zero in column six and continuation directive lines must have a character other than space or zero in column six. Continuation lines for C$DOACROSS directives are specified using the C$& sentinel.OpenMP Directives for Fortran 132 The order in which clauses appear in the parallelization directives is not significant. Commas separate clauses within the directives, but commas are not allowed between the directive name and the first clause. Clauses on directives may be repeated as needed subject to the restrictions listed in the description of each clause. The compiler option -mp enables recognition of the parallelization directives. The use of this option also implies: -Mreentrant local variables are placed on the stack and optimizations that may result in non-reentrant code are disabled (e.g., -Mnoframe); -Miomutex critical sections are generated around Fortran I/O statements. Many of the directives are presented in pairs and must be used in pairs. In the examples given with each section, the routines omp_get_num_threads() and omp_get_thread_num() are used, refer to “Runtime Library Routines” on page 148 for more information. They return the number of threads currently in the team executing the parallel region and the thread number within the team, respectively. PARALLEL ... END PARALLEL The OpenMP PARALLEL END PARALLEL directive is supported using the following syntax. Syntax: !$OMP PARALLEL [Clauses] < Fortran code executed in body of parallel region > !$OMP END PARALLEL Clauses: PRIVATE(list) SHARED(list) DEFAULT(PRIVATE | SHARED | NONE) FIRSTPRIVATE(list) REDUCTION([{operator | intrinsic}:] list) COPYIN(list) IF(scalar_logical_expression) NUM_THREADS(scalar_integer_expression) This directive pair declares a region of parallel execution. It directs the compiler to create an executable in which the statements between PARALLEL and END PARALLEL are executed by multiple lightweight threads. The code that lies between PARALLEL and END PARALLEL is called a parallel region. PARALLEL ... END PARALLEL 133 The OpenMP parallelization directives support a fork/join execution model in which a single thread executes all statements until a parallel region is encountered. At the entrance to the parallel region, a system-dependent number of symmetric parallel threads begin executing all statements in the parallel region redundantly. These threads share work by means of work-sharing constructs such as parallel DO loops (see below). The number of threads in the team is controlled by the OMP_NUM_THREADS environment variable. If OMP_NUM_THREADS is not defined, the program will execute parallel regions using only one processor. Branching into or out of a parallel region is not supported. All other shared-memory parallelization directives must occur within the scope of a parallel region. Nested PARALLEL ... END PARALLEL directive pairs are not supported and are ignored. The END PARALLEL directive denotes the end of the parallel region, and is an implicit barrier. When all threads have completed execution of the parallel region, a single thread resumes execution of the statements that follow. NOTE By default, there is no work distribution in a parallel region. Each active thread executes the entire region redundantly until it encounters a directive that specifies work distribution. For work distribution, see the DO, PARALLEL DO, or DOACROSS directives. PROGRAM WHICH_PROCESSOR_AM_I INTEGER A(0:1) INTEGER omp_get_thread_num A(0) = -1 A(1) = -1 !$OMP PARALLEL A(omp_get_thread_num()) = omp_get_thread_num() !$OMP END PARALLEL PRINT *, "A(0)=",A(0), " A(1)=",A(1) END The variables specified in a PRIVATE list are private to each thread in a team. In effect, the compiler creates a separate copy of each of these variables for each thread in the team. When an assignment to a private variable occurs, each thread assigns to its local copy of the variable. When operations involving a private variable occur, each thread performs the operations using its local copy of the variable. Important points about private variables are: OpenMP Directives for Fortran 134 • Variables declared private in a parallel region are undefined upon entry to the parallel region. If the first use of a private variable within the parallel region is in a right-hand-side expression, the results of the expression will be undefined (i.e., this is probably a coding error). • Likewise, variables declared private in a parallel region are undefined when serial execution resumes at the end of the parallel region. The variables specified in a SHARED list are shared between all threads in a team, meaning that all threads access the same storage area for SHARED data. The DEFAULT clause lets you specify the default attribute for variables in the lexical extent of the parallel region. Individual clauses specifying PRIVATE, SHARED, etc. status override the declared DEFAULT. Specifying DEFAULT(NONE) declares that there is no implicit default, and in this case, each variable in the parallel region must be explicitly listed with an attribute of PRIVATE, SHARED, FIRSTPRIVATE, LASTPRIVATE, or REDUCTION. Variables that appear in the list of a FIRSTPRIVATE clause are subject to the same semantics as PRIVATE variables, but in addition, are initialized from the original object existing prior to entering the parallel region. Variables that appear in the list of a REDUCTION clause must be SHARED. A private copy of each variable in list is created for each thread as if the PRIVATE clause had been specified. Each private copy is initialized according to the operator as specified in the following table:PARALLEL ... END PARALLEL 135 Table 5-1: Initialization of REDUCTION Variables At the end of the parallel region, a reduction is performed on the instances of variables appearing in list using operator or intrinsic as specified in the REDUCTION clause. The initial value of each REDUCTION variable is included in the reduction operation. If the {operator | intrinsic}: portion of the REDUCTION clause is omitted, the default reduction operator is “+” (addition). The COPYIN clause applies only to THREADPRIVATE common blocks. In the presence of the COPYIN clause, data from the master thread’s copy of the common block is copied to the threadprivate copies upon entry to the parallel region. In the presence of an IF clause, the parallel region will be executed in parallel only if the corresponding scalar_logical_expression evaluates to .TRUE.. Otherwise, the code within the region will be executed by a single processor regardless of the value of the environment variable OMP_NUM_THREADS. Operator / Intrinsic Initialization + 0 * 1 - 0 .AND. .TRUE. .OR. .FALSE. .EQV. .TRUE. .NEQV. .FALSE. MAX Smallest Representable Number MIN Largest Representable Number IAND All bits on IOR 0 IEOR 0OpenMP Directives for Fortran 136 If the NUM_THREADS clause is present, the corresponding scalar_integer_expression must evaluate to a positive integer value. This value sets the maximum number of threads used during execution of the parallel region. A NUM_THREADS clause overrides either a previous call to the library routine omp_set_num_threads() or the setting of the OMP_NUM_THREADS environment variable. CRITICAL ... END CRITICAL The OpenMP END CRITICAL directive uses the following syntax. !$OMP CRITICAL [(name)] < Fortran code executed in body of critical section > !$OMP END CRITICAL [(name)] Within a parallel region, you may have code that will not execute properly when multiple threads act upon the same sub-region of code. This is often due to a shared variable that is written and then read again. The CRITICAL ... END CRITICAL directive pair defines a subsection of code within a parallel region, referred to as a critical section, which will be executed one thread at a time. The optional name argument identifies the critical section. The first thread to arrive at a critical section will be the first to execute the code within the section. The second thread to arrive will not begin execution of statements in the critical section until the first thread has exited the critical section. Likewise each of the remaining threads will wait its turn to execute the statements in the critical section. Critical sections cannot be nested, and any such specifications are ignored. Branching into or out of a critical section is illegal. If a name argument appears on a CRITICAL directive, the same name must appear on the END CRITICAL directive. PROGRAM CRITICAL_USE REAL A(100,100), MX, LMX INTEGER I, J MX = -1.0 LMX = -1.0 CALL RANDOM_SEED() CALL RANDOM_NUMBER(A) !$OMP PARALLEL PRIVATE(I), FIRSTPRIVATE(LMX) !$OMP DO DO J=1,100 DOMASTER ... END MASTER 137 I=1,100 LMX = MAX(A(I,J), LMX) ENDDO ENDDO !$OMP CRITICAL MX = MAX(MX, LMX) !$OMP END CRITICAL !$OMP END PARALLEL PRINT *, "MAX VALUE OF A IS ", MX END Note that this program could also be implemented without the critical region by declaring MX as a reduction variable and performing the MAX calculation in the loop using MX directly rather than using LMX. See “PARALLEL ... END PARALLEL” on page 132 and “DO ... END DO” on page 139 for more information on how to use the REDUCTION clause on a parallel DO loop. MASTER ... END MASTER The OpenMP END MASTER directive uses the following syntax. !$OMP MASTER < Fortran code in body of MASTER section > !$OMP END MASTER In a parallel region of code, there may be a sub-region of code that should execute only on the master thread. Instead of ending the parallel region before this subregion and then starting it up again after this subregion, the MASTER ... END MASTER directive pair let you conveniently designate code that executes on the master thread and is skipped by the other threads. There is no implied barrier on entry to or exit from a MASTER ... END MASTER section of code. Nested master sections are ignored. Branching into or out of a master section is not supported. PROGRAM MASTER_USE INTEGER A(0:1) INTEGER omp_get_thread_num A=-1 !$OMP PARALLEL A(omp_get_thread_num()) = omp_get_thread_num() !$OMP MASTER PRINT *, "YOU SHOULD ONLYOpenMP Directives for Fortran 138 SEE THIS ONCE" !$OMP END MASTER !$OMP END PARALLEL PRINT *, "A(0)=", A(0), " A(1)=", A(1) END SINGLE ... END SINGLE The OpenMP SINGLE END SINGLE directive uses the following syntax. !$OMP SINGLE [Clauses] < Fortran code in body of SINGLE processor section > !$OMP END SINGLE [NOWAIT] Clauses: PRIVATE(list) FIRSTPRIVATE(list) COPYPRIVATE(list) In a parallel region of code, there may be a sub-region of code that will only execute correctly on a single thread. Instead of ending the parallel region before this subregion and then starting it up again after this subregion, the SINGLE ... END SINGLE directive pair lets you conveniently designate code that executes on a single thread and is skipped by the other threads. There is an implied barrier on exit from a SINGLE ... END SINGLE section of code unless the optional NOWAIT clause is specified. Nested single process sections are ignored. Branching into or out of a single process section is not supported. PROGRAM SINGLE_USE INTEGER A(0:1) INTEGER omp_get_thread_num() !$OMP PARALLEL A(omp_get_thread_num()) = omp_get_thread_num() !$OMP SINGLE PRINT *, "YOU SHOULD ONLY SEE THIS ONCE" !$OMP END SINGLE !$OMP END PARALLEL PRINT *, "A(0)=", A(0), " A(1)=", A(1) ENDDO ... END DO 139 The PRIVATE and FIRSTPRIVATE clauses are as described in “PARALLEL ... END PARALLEL” on page 132. The COPYPRIVATE clause causes the variables in list to be copied from the private copies in the single thread that executes the SINGLE region to the other copies in all other threads of the team at the end of the SINGLE region. The COPYPRIVATE clause must not be used with NOWAIT. DO ... END DO The OpenMP DO END DO directive uses the following syntax. Syntax: !$OMP DO [Clauses ] < Fortran DO loop to be executed in parallel > !$OMP END DO [NOWAIT] Clauses: PRIVATE(list) FIRSTPRIVATE(list) LASTPRIVATE(list) REDUCTION({operator | intrinsic } : list) SCHEDULE (type [, chunk]) ORDERED The real purpose of supporting parallel execution is the distribution of work across the available threads. You can explicitly manage work distribution with constructs such as: IF (omp_get_thread_num() .EQ. 0) THEN ... ELSE IF (omp_get_thread_num() .EQ. 1) THEN ... ENDIF However, these constructs are not in the form of directives. The DO ... END DO directive pair provides a convenient mechanism for the distribution of loop iterations across the available threads in a parallel region. Items to note about clauses are:OpenMP Directives for Fortran 140 Variables declared in a PRIVATE list are treated as private to each processor participating in parallel execution of the loop, meaning that a separate copy of the variable exists on each processor. Variables declared in a FIRSTPRIVATE list are PRIVATE, and in addition are initialized from the original object existing before the construct. Variables declared in a LASTPRIVATE list are PRIVATE, and in addition the thread that executes the sequentially last iteration updates the version of the object that existed before the construct. The REDUCTION clause is as described in “PARALLEL ... END PARALLEL” on page 132.The SCHEDULE clause is explained in the following section. If ORDERED code blocks are contained in the dynamic extent of the DO directive, the ORDERED clause must be present. For more information on ORDERED code blocks, see “ORDERED” on page 146 The DO ... END DO directive pair directs the compiler to distribute the iterative DO loop immediately following the !$OMP DO directive across the threads available to the program. The DO loop is executed in parallel by the team that was started by an enclosing parallel region. If the !$OMP END DO directive is not specified, the !$OMP DO is assumed to end with the enclosed DO loop. DO ... END DO directive pairs may not be nested. Branching into or out of a !$OMP DO loop is not supported. By default, there is an implicit barrier after the end of the parallel loop; the first thread to complete its portion of the work will wait until the other threads have finished their portion of work. If NOWAIT is specified, the threads will not synchronize at the end of the parallel loop. Other items to note about !$OMP DO loops: • The DO loop index variable is always private. • !$OMP DO loops must be executed by all threads participating in the parallel region or none at all. • The END DO directive is optional, but if it is present it must appear immediately after the end of the enclosed DO loop. PROGRAM DO_USE REAL A(1000), B(1000) DO I=1,1000 B(I) = FLOAT(I) ENDDO !$OMP PARALLEL !$OMP DO DO I=1,1000 A(I) = SQRT(B(I)); ENDDOWORKSHARE ... END WORKSHARE 141 ... !$OMP END PARALLEL ... END The SCHEDULE clause specifies how iterations of the DO loop are divided up between processors. Given a SCHEDULE (type [, chunk]) clause, type can be STATIC, DYNAMIC, GUIDED, or RUNTIME. These are defined as follows: • When SCHEDULE (STATIC, chunk) is specified, iterations are allocated in contiguous blocks of size chunk. The blocks of iterations are statically assigned to threads in a round-robin fashion in order of the thread ID numbers. The chunk must be a scalar integer expression. If chunk is not specified, a default chunk size is chosen equal to: (number_of_iterations + omp_num_threads() - 1) / omp_num_threads() • When SCHEDULE (DYNAMIC, chunk) is specified, iterations are allocated in contiguous blocks of size chunk. As each thread finishes a piece of the iteration space, it dynamically obtains the next set of iterations. The chunk must be a scalar integer expression. If no chunk is specified, a default chunk size is chosen equal to 1. • When SCHEDULE (GUIDED, chunk) is specified, the chunk size is reduced in an exponentially decreasing manner with each dispatched piece of the iteration space. Chunk specifies the minimum number of iterations to dispatch each time, except when there are less than chunk iterations remaining to be processed, at which point all remaining iterations are assigned. If no chunk is specified, a default chunk size is chosen equal to 1. • When SCHEDULE (RUNTIME) is specified, the decision regarding iteration scheduling is deferred until runtime. The schedule type and chunk size can be chosen at runtime by setting the OMP_SCHEDULE environment variable. If this environment variable is not set, the resulting schedule is equivalent to SCHEDULE(STATIC). WORKSHARE ... END WORKSHARE The OpenMP WORKSHARE … END WORKSHARE directive pair uses the following syntax. Syntax:OpenMP Directives for Fortran 142 !$OMP WORKSHARE < Fortran structured block to be executed in parallel > !$OMP END WORKSHARE [NOWAIT] The Fortran structured block enclosed by the WORKSHARE … END WORKSHARE directive pair can consist only of the following types of statements and constructs: • Array assignments • Scalar assignments • FORALL statements or constructs • WHERE statements or constructs • OpenMP ATOMIC , CRITICAL or PARALLEL constructs The work implied by the above statements and constructs is split up between the threads executing the WORKSHARE construct in a way that is guaranteed to maintain standard Fortran semantics. The goal of the WORKSHARE construct is to effect parallel execution of non-iterative but implicitly data parallel array assignments, FORALL, and WHERE statements and constructs intrinsic to the Fortran language beginning with Fortran 90. The Fortran structured block contained within a WORKSHARE construct must not contain any user-defined function calls unless the function is ELEMENTAL. BARRIER The OpenMP BARRIER directive uses the following syntax. !$OMP BARRIER There may be occasions in a parallel region, when it is necessary that all threads complete work to that point before any thread is allowed to continue. The BARRIER directive synchronizes all threads at such a point in a program. Multiple barrier points are allowed within a parallel region. The BARRIER directive must either be executed by all threads executing the parallel region or by none of them. DOACROSS The C$DOACROSS directive is not part of the OpenMP standard, but is supported for compatibility with programs parallelized using legacy SGI-style directives. Syntax:PARALLEL DO 143 C$DOACROSS [ Clauses ] < Fortran DO loop to be executed in parallel > Clauses: [ {PRIVATE | LOCAL} (list) ] [ {SHARED | SHARE} (list) ] [ MP_SCHEDTYPE={SIMPLE | INTERLEAVE} ] [ CHUNK= ] [ IF (logical_expression) ] The C$DOACROSS directive has the effect of a combined parallel region and parallel DO loop applied to the loop immediately following the directive. It is very similar to the OpenMP PARALLEL DO directive, but provides for backward compatibility with codes parallelized for SGI systems prior to the OpenMP standardization effort. The C$DOACROSS directive must not appear within a parallel region. It is a shorthand notation that tells the compiler to parallelize the loop to which it applies, even though that loop is not contained within a parallel region. While this syntax is more convenient, it should be noted that if multiple successive DO loops are to be parallelized it is more efficient to define a single enclosing parallel region and parallelize each loop using the OpenMP DO directive. A variable declared PRIVATE or LOCAL to a C$DOACROSS loop is treated the same as a private variable in a parallel region or DO (see above). A variable declared SHARED or SHARE to a C$DOACROSS loop is shared among the threads, meaning that only 1 copy of the variable exists to be used and/or modified by all of the threads. This is equivalent to the default status of a variable that is not listed as PRIVATE in a parallel region or DO (this same default status is used in C$DOACROSS loops as well). PARALLEL DO The OpenMP PARALLEL DO directive uses the following syntax. Syntax: !$OMP PARALLEL DO [CLAUSES] < Fortran DO loop to be executed in parallel > [!$OMP END PARALLEL DO] Clauses: PRIVATE(list) SHARED(list) DEFAULT(PRIVATE | SHARED | NONE) FIRSTPRIVATE(list)OpenMP Directives for Fortran 144 LASTPRIVATE(list) REDUCTION({operator | intrinsic} : list) COPYIN (list) IF(scalar_logical_expression) NUM_THREADS(scalar_integer_expression) SCHEDULE (type [, chunk]) ORDERED The semantics of the PARALLEL DO directive are identical to those of a parallel region containing only a single parallel DO loop and directive. Note that the END PARALLEL DO directive is optional. The available clauses are as defined in “PARALLEL ... END PARALLEL” on page 132 and “DO ... END DO” on page 139. PARALLEL WORKSHARE The OpenMP PARALLEL WORKSHARE directive uses the following syntax. Syntax: !$OMP PARALLEL WORKSHARE [CLAUSES] < Fortran structured block to be executed in parallel > [!$OMP END PARALLEL WORKSHARE] Clauses: PRIVATE(list) SHARED(list) DEFAULT(PRIVATE | SHARED | NONE) FIRSTPRIVATE(list) LASTPRIVATE(list) REDUCTION({operator | intrinsic} : list) COPYIN (list) IF(scalar_logical_expression) NUM_THREADS(scalar_integer_expression) SCHEDULE (type [, chunk]) ORDERED The semantics of the PARALLEL WORKSHARE directive are identical to those of a parallel region containing a single WORKSHARE construct. Note that the END PARALLEL WORKSHARE directive is optional, and that NOWAIT may not be specified on an END PARALLEL WORKSHARE directive. The available clauses are as defined in “PARALLEL ... END PARALLEL” on page 132.SECTIONS … END SECTIONS 145 SECTIONS … END SECTIONS The OpenMP SECTIONS / END SECTIONS directive pair uses the following syntax: Syntax: !$OMP SECTIONS [ Clauses ] [!$OMP SECTION] < Fortran code block executed by processor i > [!$OMP SECTION] < Fortran code block executed by processor j > ... !$OMP END SECTIONS [NOWAIT] Clauses: PRIVATE (list) FIRSTPRIVATE (list) LASTPRIVATE (list) REDUCTION({operator | intrinsic} : list) The SECTIONS / END SECTIONS directives define a non-iterative work-sharing construct within a parallel region. Each section is executed by a single processor. If there are more processors than sections, some processors will have no work and will jump to the implied barrier at the end of the construct. If there are more sections than processors, one or more processors will execute more than one section. A SECTION directive may only appear within the lexical extent of the enclosing SECTIONS / END SECTIONS directives. In addition, the code within the SECTIONS / END SECTIONS directives must be a structured block, and the code in each SECTION must be a structured block. The available clauses are as defined in “PARALLEL ... END PARALLEL” on page 132 and “DO ... END DO” on page 139. PARALLEL SECTIONS The OpenMP PARALLEL SECTIONS / END SECTIONS directive pair uses the following syntax: Syntax:OpenMP Directives for Fortran 146 !$OMP PARALLEL SECTIONS [CLAUSES] [!$OMP SECTION] < Fortran code block executed by processor i > [!$OMP SECTION] < Fortran code block executed by processor j > ... !$OMP END SECTIONS [NOWAIT] Clauses: PRIVATE(list) SHARED(list) DEFAULT(PRIVATE | SHARED | NONE) FIRSTPRIVATE(list) LASTPRIVATE(list) REDUCTION({operator | intrinsic} : list) COPYIN (list) IF(scalar_logical_expression) NUM_THREADS(scalar_integer_expression) The PARALLEL SECTIONS / END SECTIONS directives define a non-iterative work-sharing construct without the need to define an enclosing parallel region. Each section is executed by a single processor. If there are more processors than sections, some processors will have no work and will jump to the implied barrier at the end of the construct. If there are more sections than processors, one or more processors will execute more than one section. A SECTION directive may only appear within the lexical extent of the enclosing PARALLEL SECTIONS / END SECTIONS directives. In addition, the code within the PARALLEL SECTIONS / END SECTIONS directives must be a structured block, and the code in each SECTION must be a structured block. The available clauses are as defined in “PARALLEL ... END PARALLEL” on page 132 and “DO ... END DO” on page 139. ORDERED The OpenMP ORDERED directive is supported using the following syntax: !$OMP ORDERED < Fortran code block executed by processor > !$OMP END ORDEREDATOMIC 147 The ORDERED directive can appear only in the dynamic extent of a DO or PARALLEL DO directive that includes the ORDERED clause. The code block between the ORDERED / END ORDERED directives is executed by only one thread at a time, and in the order of the loop iterations. This sequentializes the ordered code block while allowing parallel execution of statements outside the code block. The following additional restrictions apply to the ORDERED directive: • The ORDERED code block must be a structured block. It is illegal to branch into or out of the block. • A given iteration of a loop with a DO directive cannot execute the same ORDERED directive more than once, and cannot execute more than one ORDERED directive. ATOMIC The OpenMP ATOMIC directive uses following syntax: !$OMP ATOMIC The ATOMIC directive is semantically equivalent to enclosing the following single statement in a CRITICAL / END CRITICAL directive pair. The statement must be of one of the following forms: x = x operator expr x = expr operator x x = intrinsic (x, expr) x = intrinsic (expr, x) where x is a scalar variable of intrinsic type, expr is a scalar expression that does not reference x, intrinsic is one of MAX, MIN, IAND, IOR, or IEOR, and operator is one of +, *, -, /, .AND., .OR., .EQV., or .NEQV.. FLUSH The OpenMP FLUSH directive uses the following syntax: !$OMP FLUSH [(list)] The FLUSH directive ensures that all processor-visible data items, or only those specified in list when it’s present, are written back to memory at the point at which the directive appears.OpenMP Directives for Fortran 148 THREADPRIVATE The OpenMP THREADPRIVATE directive uses the following syntax: !$OMP THREADPRIVATE (list) Where list is a comma-separated list of named variables to be made private to each thread or named common blocks to be made private to each thread but global within the thread . Common block names must appear between slashes (i.e. /common_blockn/). This directive must appear in the declarations section of a program unit after the declaration of any common blocks or variables listed. On entry to a parallel region, data in a THREADPRIVATE common block or variable is undefined unless COPYIN is specified on the PARALLEL directive. When a common block or variable that is initialized using DATA statements appears in a THREADPRIVATE directive, each thread’s copy is initialized once prior to its first use. The following restrictions apply to the THREADPRIVATE directive: • The THREADPRIVATE directive must appear after every declaration of a thread private common block. • Only named common blocks can be made thread private • It is illegal for a THREADPRIVATE common block or its constituent variables to appear in any clause other than a COPYIN clause. • A variable can appear in a THREADRIVATE directive only in the scope in which it is declared. It must not be an element of a common block or be declared in an EQUIVALENCE statement. • A variable that appears in a THREADPRIVATE directive and is not declared in the scope of a module must have the SAVE attribute. Run-time Library Routines User-callable functions are available to the Fortran programmer to query and alter the parallel execution environment. integer omp_get_num_threads()Run-time Library Routines 149 returns the number of threads in the team executing the parallel region from which it is called. When called from a serial region, this function returns 1. A nested parallel region is the same as a single parallel region. By default, the value returned by this function is equal to the value of the environment variable OMP_NUM_THREADS or to the value set by the last previous call to the omp_set_num_threads() subroutine defined below. subroutine omp_set_num_threads(scalar_integer_exp) sets the number of threads to use for the next parallel region. This subroutine can only be called from a serial region of code. If it is called from within a parallel region, or within a subroutine or function that is called from within a parallel region, the results are undefined. This subroutine has precedence over the OMP_NUM_THREADS environment variable. integer omp_get_thread_num() returns the thread number within the team. The thread number lies between 0 and omp_get_num_threads()-1. When called from a serial region, this function returns 0. A nested parallel region is the same as a single parallel region. integer function omp_get_max_threads() returns the maximum value that can be returned by calls to omp_get_num_threads(). If omp_set_num_threads() is used to change the number of processors, subsequent calls to omp_get_max_threads() will return the new value. This function returns the maximum value whether executing from a parallel or serial region of code. integer function omp_get_num_procs() returns the number of processors that are available to the program. !omp_get_stack_size interface function omp_get_stack_size () use omp_lib_kinds integer ( kind=OMP_STACK_SIZE_KIND ) :: omp_get_stack_size end function omp_get_stack_size end interface Returns the value of the OpenMP internal control variable that specifies the size that will be used to create a stack for a newly created thread. Note that this value may not be the size of the stack of the current thread.OpenMP Directives for Fortran 150 subroutine omp_set_stack_size(integer(KIND=OMP_STACK_SIZE_KIND)) Changes the value of the OpenMP internal control variable that specifies the size that will be used to create a stack for a newly created thread. The argument is the stack size in kilobytes. The size of the stack of the current thread cannot be changed. In the PGI implementation, all OpenMP or autoparallelization threads are created just prior to the first parallel region, therefore, only calls to omp_set_stack_size prior to the first region have an effect. logical function omp_in_parallel() returns .TRUE. if called from within a parallel region and .FALSE. if called outside of a parallel region. When called from within a parallel region that is serialized, for example in the presence of an IF clause evaluating .FALSE., the function will return .FALSE.. subroutine omp_set_dynamic(scalar_logical_exp) is designed to allow automatic dynamic adjustment of the number of threads used for execution of parallel regions. This function is recognized, but currently has no effect. logical function omp_get_dynamic() is designed to allow the user to query whether automatic dynamic adjustment of the number of threads used for execution of parallel regions is enabled. This function is recognized, but currently always returns .FALSE.. subroutine omp_set_nested(scalar_logical_exp) is designed to allow enabling/disabling of nested parallel regions. This function is recognized, but currently has no effect. logical function omp_get_nested() is designed to allow the user to query whether dynamic adjustment of the number of threads available for execution of parallel regions is enabled. This function is recognized, but currently always returns .FALSE. double precision function omp_get_wtime() returns the elapsed wall clock time in seconds as a DOUBLE PRECISION value. Times returned are perthread times, and are not necessarily globally consistent across all threads. double precision function omp_get_wtick()Environment Variables 151 returns the resolution of omp_get_wtime(), in seconds, as a DOUBLE PRECISION value. subroutine omp_init_lock(integer_var) initializes a lock associated with the variable integer_var for use in subsequent calls to lock routines. This initial state of integer_var is unlocked. It is illegal to make a call to this routine if integer_var is already associated with a lock. subroutine omp_destroy_lock(integer_var) disassociates a lock associated with the variable integer_var. subroutine omp_set_lock(integer_var) causes the calling thread to wait until the specified lock is available. The thread gains ownership of the lock when it is available. It is illegal to make a call to this routine if integer_var has not been associated with a lock. subroutine omp_unset_lock(integer_var) causes the calling thread to release ownership of the lock associated with integer_var. It is illegal to make a call to this routine if integer_var has not been associated with a lock. logical function omp_test_lock(integer_var) causes the calling thread to try to gain ownership of the lock associated with integer_var. The function returns .TRUE. if the thread gains ownership of the lock, and .FALSE. otherwise. It is illegal to make a call to this routine if integer_var has not been associated with a lock. Environment Variables OMP_NUM_THREADS - specifies the number of threads to use during execution of parallel regions. The default value for this variable is 1. For historical reasons, the environment variable NCPUS is supported with the same functionality. In the event that both OMP_NUM_THREADS and NCPUS are defined, the value of OMP_NUM_THREADS takes precedence. NOTE OMP_NUM_THREADS threads will be used to execute the program regardless of the number of physical processors available in the system. As a result, you can run programs using more threads than physical processors and they will execute correctly. However, performance of programs executed in this manner can be unpredictable, and oftentimes will be inefficientOpenMP Directives for Fortran 152 OMP_SCHEDULE - specifies the type of iteration scheduling to use for DO and PARALLEL DO loops which include the SCHEDULE(RUNTIME) clause. The default value for this variable is “STATIC”. If the optional chunk size is not set, a chunk size of 1 is assumed except in the case of a STATIC schedule. For a STATIC schedule, the default is as defined in “DO ... END DO” on page 139. Examples of the use of OMP_SCHEDULE are as follows: $ setenv OMP_SCHEDULE "STATIC, 5" $ setenv OMP_SCHEDULE "GUIDED, 8" $ setenv OMP_SCHEDULE "DYNAMIC" OMP_DYNAMIC - currently has no effect. OMP_NESTED - currently has no effect. MPSTKZ - increase the size of the stacks used by threads executing in parallel regions. It is for use with programs that utilize large amounts of thread-local storage in the form of private variables or local variables in functions or subroutines called within parallel regions. The value should be an integer concatenated with M or m to specify stack sizes of n megabytes. For example: $ setenv MPSTKZ 8M OMP_WAIT_POLICY (Proposed OpenMP 3.0 Feature) - The OpenMP environment variable OMP_WAIT_POLICY sets the behavior of idle threads. The values are ACTIVE and PASSIVE. This behavior is also shared by threads created by auto-parallelization. Threads are considered idle when waiting at a barrier, when waiting to enter a critical region, or unemployed between parallel regions. Threads waiting for critical sections always busy wait. Barriers always busy wait with calls to sched_yield determined by MP_SPIN. Unemployed threads during a serial region can either busy wait using the barrier (ACTIVE) or politely wait using a mutex (PASSIVE). The choice is set by OMP_WAIT_POLICY. The default is ACTIVE. When ACTIVE is set, idle threads consume 100% of their CPU allotment spinning in a busy loop waiting to restart in a parallel region. This mechanism allows for very quick entry into parallel regions which is good for programs that enter and leave parallel regions frequently. When PASSIVE is set, idle threads wait on a mutex (in the operating system) and consume no CPU time until being restarted. Passive idle is best when a program has long periods of serial activity or when the program runs on a multi-user machine or otherwise shares CPU resources.Environment Variables 153 OMP_STACK_SIZE (Proposed OpenMP 3.0 Feature) - The OpenMP environment variable OMP_STACK_SIZE overrides the default stack size for a newly created thread. The value is an decimal integer followed by an optional letter B, K, M, or G, to specify bytes, kilobytes, megabytes, and gigabytes, respectively. If no letter is used, the default is kilobytes. There is no space between the value and the letter; for example, one megabyte is specified 1M. The environment variable OMP_STACK_SIZE is read on program start-up; if the program changes its own environment, the variable is not re-checked. This environment variable takes precedence over MPSTKSZ (see above). Once a thread is created, its stack size cannot be changed. In the PGI implementation, threads are created prior to the first parallel region and persist for the life of the program. The stack size of the main program is not affected by OMP_STACK_SIZE. Its size is set at program start up. For more information on controlling the program stack size in Linux, see “Running Parallel Programs on Linux” on page 10.OpenMP Directives for Fortran 154Parallelization Pragmas 155 6 OpenMP Pragmas for C and C++ The PGCC ANSI C and C++ compilers support the OpenMP C/C++ Application Program Interface. The OpenMP shared-memory parallel programming model is defined by a collection of compiler directives or pragmas, library routines, and environment variables that can be used to specify shared-memory parallelism in Fortran, C and C++ programs. The OpenMP C/C++ pragmas include a parallel region construct for writing coarse grain SPMD programs, work-sharing constructs which specify that C/C++ for loop iterations should be split among the available threads of execution, and synchronization constructs. The data environment is controlled using clauses on the pragmas or with additional pragmas. Run-time library functions are provided to query the parallel runtime environment, for example to determine how many threads are participating in execution of a parallel region. Finally, environment variables are provided to control the execution behavior of parallel programs. For more information on OpenMP, and a complete copy of the OpenMP C/C++ API specification, see http:// www.openmp.org Parallelization Pragmas Parallelization pragmas are #pragma statements in a C or C++ program that are interpreted by the PGCC C and C++ compilers when the option -mp is specified on the command line. The form of a parallelization pragma is: #pragma omppragma_name[clauses] The pragmas follow the conventions of the C and C++ standards. Whitespace can appear before and after the #. Preprocessing tokens following the #pragma omp are subject to macro replacement. The order in which clauses appear in the parallelization pragmas is not significant. Spaces separate clauses within the pragmas. Clauses on pragmas may be repeated as needed subject to the restrictions listed in the description of each clause. For the purposes of the OpenMP pragmas, a C/C++ structured block is defined to be a statement or compound statement (a sequence of statements beginning with { and ending with }) that has a single entry and a single exit. No statement or compound statement is a C/C++ structured block if there is a jump into or out of that statement. OpenMP Pragmas for C and C++ 156 The compiler option -mp enables recognition of the parallelization pragmas. The use of this option also implies: -Mreentrant local variables are placed on the stack and optimizations that may result in non-reentrant code are disabled (e.g., -Mnoframe) Also, note that calls to I/O library functions are system-dependent and are not necessarily guaranteed to be thread-safe. I/O library calls within parallel regions should be protected by critical regions (see below) to ensure they function correctly on all systems. In the examples given with each section, the functions omp_get_num_threads() and omp_get_thread_num() are used (refer to “Run-time Library Routines” on page 169.) They return the number of threads currently in the team executing the parallel region and the thread number within the team, respectively. omp parallel The OpenMP omp parallel pragma uses the following syntax: Syntax: #pragma omp parallel [clauses] < C/C++ structured block > Clauses: private(list) shared(list) default(shared | none) firstprivate(list) reduction(operator: list) copyin (list) if (scalar_expression) num_threads(scalar_integer_expression) This pragma declares a region of parallel execution. It directs the compiler to create an executable in which the statements within the following C/C++ structured block are executed by multiple lightweight threads. The code that lies within the structured block is called a parallel region. The OpenMP parallelization pragmas support a fork/join execution model in which a single thread executes all statements until a parallel region is encountered. At the entrance to the parallel region, a system-dependent number of symmetric parallel threads begin executing all statements in the parallel omp parallel 157 region redundantly. These threads share work by means of work-sharing constructs such as parallel for loops (see the next example). The number of threads in the team is controlled by the OMP_NUM_THREADS environment variable. If OMP_NUM_THREADS is not defined, the program will execute parallel regions using only one processor. Branching into or out of a parallel region is not supported. All other shared-memory parallelization pragmas must occur within the scope of a parallel region. Nested omp parallel pragmas are not supported and are ignored. There is an implicit barrier at the end of a parallel region. When all threads have completed execution of the parallel region, a single thread resumes execution of the statements that follow. It should be emphasized that by default there is no work distribution in a parallel region. Each active thread executes the entire region redundantly until it encounters a directive that specifies work distribution. For work distribution, see the omp for pragma. #include #include main(){ int a[2]={-1,-1}; #pragma omp parallel { a[omp_get_thread_num()] = omp_get_thread_num(); } printf("a[0] = %d, a[1] = %d",a[0],a[1]); } The variables specified in a private list are private to each thread in a team. In effect, the compiler creates a separate copy of each of these variables for each thread in the team. When an assignment to a private variable occurs, each thread assigns to its local copy of the variable. When operations involving a private variable occur, each thread performs the operations using its local copy of the variable. Other important points to note about private variables are the following: • Variables declared private in a parallel region are undefined upon entry to the parallel region. If the first use of a private variable within the parallel region is in a right-hand-side expression, the results of the expression will be undefined (i.e., this is probably a coding error). • Likewise, variables declared private in a parallel region are undefined when serial execution resumes at the end of the parallel region.OpenMP Pragmas for C and C++ 158 The variables specified in a shared list are shared between all threads in a team, meaning that all threads access the same storage area for shared data. The default clause allows the user to specify the default attribute for variables in the lexical extent of the parallel region. Individual clauses specifying private, shared, etc status override the declared default. Specifying default(none) declares that there is no implicit default, and in this case each variable in the parallel region must be explicitly listed with an attribute of private, shared, firstprivate, or reduction. Variables that appear in the list of a firstprivate clause are subject to the same semantics as private variables, but in addition are initialized from the original object existing prior to entering the parallel region. Variables that appear in the list of a reduction clause must be shared. A private copy of each variable in list is created for each thread as if the private clause had been specified. Each private copy is initialized according to the operator as specified in the following table: Table 6-1: Initialization of Reduction Variables • At the end of the parallel region, a reduction is performed on the instances of variables appearing in list using operator as specified in the reduction clause. The initial value of each reduction variable is included in the reduction operation. If the operator: portion of the reduction clause is omitted, the default reduction operator is “+” (addition). Operator Initialization + 0 * 1 - 0 & ~0 | 0 ^ 0 && 1 || 0omp critical 159 • The copyin clause applies only to threadprivate variables. In the presence of the copyin clause, data from the master thread’s copy of the threadprivate variable is copied to the thread private copies upon entry to the parallel region. • In the presence of an if clause, the parallel region will be executed in parallel only if the corresponding scalar_expression evaluates to a non-zero value. Otherwise, the code within the region will be executed by a single processor regardless of the value of the environment variable OMP_NUM_THREADS. • If the num_threads clause is present, the corresponding scalar_integer_expression must evaluate to a positive integer value. This value sets the maximum number of threads used during execution of the parallel region. A num_threads clause overrides either a previous call to the library routine omp_set_num_threads() or the setting of the OMP_NUM_THREADS environment variable. omp critical The OpenMP omp critical pragma uses the following syntax: #pragma omp critical [(name)] < C/C++ structured block > Within a parallel region, there may exist subregions of code that will not execute properly when executed by multiple threads simultaneously. This is often due to a shared variable that is written and then read again. The omp critical pragma defines a subsection of code within a parallel region, referred to as a critical section, which will be executed one thread at a time. The first thread to arrive at a critical section will be the first to execute the code within the section. The second thread to arrive will not begin execution of statements in the critical section until the first thread has exited the critical section. Likewise, each of the remaining threads will wait its turn to execute the statements in the critical section. An optional name may be used to identify the critical region. Names used to identify critical regions have external linkage and are in a name space separate from the name spaces used by labels, tags, members, and ordinary identifiers. Critical sections cannot be nested, and any such specifications are ignored. Branching into or out of a critical section is illegal.OpenMP Pragmas for C and C++ 160 #include main(){ int a[100][100], mx=-1, lmx=-1, i, j; for (j=0; j<100; j++) for (i=0; i<100; i++) a[i][j]=1+(int)(10.0*rand()/(RAND_MAX+1.0)); #pragma omp parallel private(i) firstprivate(lmx) { #pragma omp for for (j=0; j<100; j++) for (i=0; i<100; i++) lmx = (lmx > a[i][j]) ? lmx : a[i][j]; #pragma omp critical mx = (mx > lmx) ? mx : lmx; } printf ("max value of a is %d\n",mx); } omp master The OpenMP omp master pragma uses the following syntax: #pragma omp master < C/C++ structured block > In a parallel region of code, there may be a sub-region of code that should execute only on the master thread. Instead of ending the parallel region before this subregion, and then starting it up again after this subregion, the omp master pragma allows the user to conveniently designate code that executes on the master thread and is skipped by the other threads. There is no implied barrier on entry to or exit from a master section. Nested master sections are ignored. Branching into or out of a master section is not supported. #include #include main(){ int a[2]={-1,-1}; #pragma omp parallel { a[omp_get_thread_num()] = omp_get_thread_num(); #pragma omp masteromp single 161 printf("YOU SHOULD ONLY SEE THIS ONCE\n"); } printf("a[0]=%d, a[1]=%d\n",a[0],a[1]); } omp single The OpenMP omp single pragma uses the following syntax: #pragma omp single [Clauses] < C/C++ structured block > Clauses: private(list) firstprivate(list) copyprivate(list) nowait In a parallel region of code, there may be a subregion of code that will only execute correctly on a single thread. Instead of ending the parallel region before this subregion, and then starting it up again after this subregion, the omp single pragma allows the user to conveniently designate code that executes on a single thread and is skipped by the other threads. There is an implied barrier on exit from a single process section unless the optional nowait clause is specified. Nested single process sections are ignored. Branching into or out of a single process section is not supported. The private and firstprivate clauses are as described in “omp parallel” on page 156. The copyprivate clause causes the variables in list to be copied from the private copies in the single thread that executes the single region to the other copies in all other threads of the team at the end of the single region. The copyprivate clause must not be used with nowait. omp for The OpenMP omp for pragma uses the following syntax: #pragma omp for [Clauses] < C/C++ for loop to be executed in parallel > Clauses:OpenMP Pragmas for C and C++ 162 private(list) firstprivate(list) lastprivate(list) reduction(operator: list) schedule (kind[, chunk]) ordered nowait The real purpose of supporting parallel execution is the distribution of work across the available threads. You can explicitly manage work distribution with constructs such as: if (omp_get_thread_num() == 0) { ... } else if (omp_get_thread_num() == 1) { ... } However, these constructs are not in the form of pragmas. The omp for pragma provides a convenient mechanism for the distribution of loop iterations across the available threads in a parallel region. The following variables can be used: • Variables declared in a private list are treated as private to each processor participating in parallel execution of the loop, meaning that a separate copy of the variable exists on each processor. • Variables declared in a firstprivate list are private, and in addition are initialized from the original object existing before the construct. • Variables declared in a lastprivate list are private, and in addition the thread that executes the sequentially last iteration updates the version of the object that existed before the construct. • The reduction clause is as described in “omp parallel” on page 156. The schedule clause is explained below. • If ordered code blocks are contained in the dynamic extent of the for directive, the ordered clause must be present. See “omp ordered” on page 167 for more information on ordered code blocks. The omp for pragma directs the compiler to distribute the iterative for loop immediately following across the threads available to the program. The for loop is executed in parallel by the team that was started by an enclosing parallel region. Branching into or out of an omp for loop is not supported, and omp for pragmas may not be nested.omp for 163 By default, there is an implicit barrier after the end of the parallel loop; the first thread to complete its portion of the work will wait until the other threads have finished their portion of work. If nowait is specified, the threads will not synchronize at the end of the parallel loop. Other items to note about omp for loops: • The for loop index variable is always private and must be a signed integer. • omp for loops must be executed by all threads participating in the parallel region or none at all. • The for loop must be a structured block and its execution must not be terminated by break. • Values of the loop control expressions and the chunk expressions must be the same for all threads executing the loop. #include #include main(){ float a[1000], b[1000]; int i; for (i=0; i<1000; i++) b[i] = i; #pragma omp parallel { #pragma omp for for (i=0; i<1000; i++) a[i] = sqrt(b[i]); ... } ... } The schedule clause specifies how iterations of the for loop are divided up between processors. Given a schedule (kind[, chunk]) clause, kind can be static, dynamic, guided, or runtime. These are defined as follows: • When schedule (static, chunk) is specified, iterations are allocated in contiguous blocks of size chunk. The blocks of iterations are statically assigned to threads in a round-robin fashion in order of the thread ID numbers. The chunk must be a scalar integer expression. If chunk is not specified, a default chunk size is chosen equal to: (number_of_iterations + omp_num_threads() - 1) / omp_num_threads()OpenMP Pragmas for C and C++ 164 • When schedule (dynamic, chunk) is specified, iterations are allocated in contiguous blocks of size chunk. As each thread finishes a piece of the iteration space, it dynamically obtains the next set of iterations. The chunk must be a scalar integer expression. If no chunk is specified, a default chunk size is chosen equal to 1. • When schedule (guided, chunk) is specified, the chunk size is reduced in an exponentially decreasing manner with each dispatched piece of the iteration space. Chunk specifies the minimum number of iterations to dispatch each time, except when there are less than chunk iterations remaining to be processed, at which point all remaining iterations are assigned. If no chunk is specified, a default chunk size is chosen equal to 1. • When schedule (runtime) is specified, the decision regarding iteration scheduling is deferred until runtime. The schedule type and chunk size can be chosen at runtime by setting the OMP_SCHEDULE environment variable. If this environment variable is not set, the resulting schedule is equivalent to schedule(static). omp barrier The OpenMP omp barrier pragma uses the following syntax: #pragma omp barrier There may be occasions in a parallel region when it is necessary that all threads complete work to that point before any thread is allowed to continue. The omp barrier pragma synchronizes all threads at such a point in a program. Multiple barrier points are allowed within a parallel region. The omp barrier pragma must either be executed by all threads executing the parallel region or by none of them. omp parallel for The omp parallel for pragma uses the following syntax. #pragma omp parallel for [clauses] < C/C++ for loop to be executed in parallel > Clauses:omp sections 165 private(list) shared(list) default(shared | none) firstprivate(list) lastprivate(list) reduction(operator: list) copyin (list) if (scalar_expression) ordered schedule (kind[, chunk]) num_threads(scalar_integer_expression) The semantics of the omp parallel for pragma are identical to those of a parallel region containing only a single parallel for loop and pragma. The available clauses are as defined in “omp parallel” on page 156 and “omp for” on page 161. omp sections The omp sections pragma uses the following syntax: #pragma omp sections [ Clauses ] { [#pragma omp section] < C/C++ structured block executed by processor i > [#pragma omp section] < C/C++ structured block executed by processor j > ... } Clauses: private (list) firstprivate (list) lastprivate (list) reduction(operator: list) nowaitOpenMP Pragmas for C and C++ 166 The omp sections pragma defines a non-iterative work-sharing construct within a parallel region. Each section is executed by a single thread. If there are more threads than sections, some threads will have no work and will jump to the implied barrier at the end of the construct. If there are more sections than threads, one or more threads will execute more than one section. An omp section pragma may only appear within the lexical extent of the enclosing omp sections pragma. In addition, the code within the omp sections pragma must be a structured block, and the code in each omp section must be a structured block. The available clauses are as defined in “omp parallel” on page 156 and “omp for” on page 161. omp parallel sections The omp parallel sections pragma uses the following syntax: #pragma omp parallel sections [clauses] { [#pragma omp section] < C/C++ structured block executed by processor i > [#pragma omp section] < C/C++ structured block executed by processor j > ... } Clauses: private(list) shared(list) default(shared | none) firstprivate(list) lastprivate (list) reduction({operator: list) copyin (list) if (scalar_expression) num_threads(scalar_integer_expression) nowaitomp ordered 167 The omp parallel sections pragma defines a non-iterative work-sharing construct without the need to define an enclosing parallel region. Semantics are identical to a parallel region containing only an omp sections pragma and the associated structured block. omp ordered The OpenMP ordered pragma uses the following syntax: #pragma omp ordered < C/C++ structured block > The ordered pragma can appear only in the dynamic extent of a for or parallel for pragma that includes the ordered clause. The structured code block appearing after the ordered pragma is executed by only one thread at a time, and in the order of the loop iterations. This sequentializes the ordered code block while allowing parallel execution of statements outside the code block. The following additional restrictions apply to the ordered pragma: • The ordered code block must be a structured block. It is illegal to branch into or out of the block. • A given iteration of a loop with a DO directive cannot execute the same ORDERED directive more than once, and cannot execute more than one ORDERED directive. omp atomic The omp atomic pragma uses the following syntax: #pragma omp atomic < C/C++ expression statement > The omp atomic pragma is semantically equivalent to subjecting the following single C/C++ expression statement to an omp critical pragma. The expression statement must be of one of the following forms: x = expr x++ ++x x-- --xOpenMP Pragmas for C and C++ 168 where x is a scalar variable of intrinsic type, expr is a scalar expression that does not reference x, is not overloaded and is one of +, *, -, /, &, ^, |, << or >>. omp flush The omp flush pragma uses the following syntax: #pragma omp flush [(list)] The omp flush pragma ensures that all processor-visible data items, or only those specified in list when it’s present, are written back to memory at the point at which the directive appears. omp threadprivate The omp threadprivate pragma uses the following syntax: #pragma omp threadprivate (list) Where list is a list of variables to be made private to each thread but global within the thread. This pragma must appear in the declarations section of a program unit after the declaration of any variables listed. On entry to a parallel region, data in a threadprivate variable is undefined unless copyin is specified on the omp parallel pragma. When a variable appears in an omp threadprivate pragma, each thread’s copy is initialized once at an unspecified point prior to its first use as the master copy would be initialized in a serial execution of the program. The following restrictions apply to the omp threadprivate pragma: • The omp threadprivate pragma must appear after the declaration of every threadprivate variable included in list. • It is illegal for an omp threadprivate variable to appear in any clause other than a copyin, schedule or if clause. • If a variable is specified in an omp threadprivate pragma in one translation unit, it must be specified in an omp threadprivate pragma in every translation unit in which it appears. • The address of an omp threadprivate variable is not an address constant. • An omp threadprivate variable must not have an incomplete type or a reference type.Run-time Library Routines 169 Run-time Library Routines User-callable functions are available to the OpenMP C/C++ programmer to query and alter the parallel execution environment. Any program unit that invokes these functions should include the statement #include . The omp.h include file contains definitions for each of the C/C++ library routines and two required type definitions. #include int omp_get_num_threads(void); returns the number of threads in the team executing the parallel region from which it is called. When called from a serial region, this function returns 1. A nested parallel region is the same as a single parallel region. By default, the value returned by this function is equal to the value of the environment variable OMP_NUM_THREADS or to the value set by the last previous call to the omp_set_num_threads() function defined below. #include void omp_set_num_threads(int num_threads); sets the number of threads to use for the next parallel region. This function can only be called from a serial region of code. If it is called from within a parallel region, or within a function that is called from within a parallel region, the results are undefined. This function has precedence over the OMP_NUM_THREADS environment variable. #include int omp_get_thread_num(void); returns the thread number within the team. The thread number lies between 0 and omp_get_num_threads()-1. When called from a serial region, this function returns 0. A nested parallel region is the same as a single parallel region. #include int omp_get_max_threads(void); returns the maximum value that can be returned by calls to omp_get_num_threads(). If omp_set_num_threads() is used to change the number of processors, subsequent calls to omp_get_max_threads() will return the new value. This function returns the maximum value whether executing from a parallel or serial region of code. #include int omp_get_num_procs(void);OpenMP Pragmas for C and C++ 170 returns the number of processors that are available to the program. #include size_t omp_get_stack_size(void); Returns the value of the OpenMP internal control variable that specifies the size that will be used to create a stack for a newly created thread. Note that this value may not be the size of the stack of the current thread. #include void omp_set_stack_size(size_t); Changes the value of the OpenMP internal control variable that specifies the size that will be used to create a stack for a newly created thread. The argument specifies the stack size in kilobytes. The size of the stack of the current thread cannot be changed. In the PGI implementation, all OpenMP or autoparallelization threads are created just prior to the first parallel region, therefore, only calls to omp_set_stack_size prior to the first region have an effect. #include int omp_in_parallel(void); returns non-zero if called from within a parallel region and zero if called outside of a parallel region. When called from within a parallel region that is serialized, for example in the presence of an if clause evaluating to zero, the function will return zero. #include void omp_set_dynamic(int dynamic_threads); is designed to allow automatic dynamic adjustment of the number of threads used for execution of parallel regions. This function is recognized, but currently has no effect. #include int omp_get_dynamic(void); is designed to allow the user to query whether automatic dynamic adjustment of the number of threads used for execution of parallel regions is enabled. This function is recognized, but currently always returns zero. #include void omp_set_nested(int nested); is designed to allow enabling/disabling of nested parallel regions. This function is recognized, but currently has no effect.Run-time Library Routines 171 #include int omp_get_nested(void); is designed to allow the user to query whether dynamic adjustment of the number of threads available for execution of parallel regions is enabled. This function is recognized, but currently always returns zero. #include double omp_get_wtime() returns the elapsed wall clock time in seconds as a floating-point double value. Times returned are perthread times, and are not necessarily globally consistent across all threads. #include double omp_get_wtick() returns the resolution of omp_get_wtime(), in seconds, as a floating-point double value. #include void omp_init_lock(omp_lock_t *lock); void omp_init_nest_lock(omp_nest_lock_t *lock); initializes a lock associated with the variable lock for use in subsequent calls to lock routines. This initial state of lock is unlocked. It is illegal to make a call to this routine if lock is already associated with a lock. #include void omp_destroy_lock(omp_lock_t *lock); void omp_destroy_nest_lock(omp_nest_lock_t *lock); disassociates a lock associated with the variable lock. #include void omp_set_lock(omp_lock_t *lock); void omp_set_nest_lock(omp_nest_lock_t *lock); causes the calling thread to wait until the specified lock is available. The thread gains ownership of the lock when it is available. It is illegal to make a call to this routine if lock has not been associated with a lock.OpenMP Pragmas for C and C++ 172 #include void omp_unset_lock(omp_lock_t *lock); void omp_unset_nest_lock(omp_nest_lock_t *lock); causes the calling thread to release ownership of the lock associated with lock. It is illegal to make a call to this routine if lock has not been associated with a lock. #include int omp_test_lock(omp_lock_t *lock); int omp_test_nest_lock(omp_nest_lock_t *lock); causes the calling thread to try to gain ownership of the lock associated with lock. The function returns non-zero if the thread gains ownership of the lock, and zero otherwise. It is illegal to make a call to this routine if lock has not been associated with a lock. Environment Variables OMP_NUM_THREADS - specifies the number of threads to use during execution of parallel regions. The default value for this variable is 1. For historical reasons, the environment variable NCPUS is supported with the same functionality. In the event that both OMP_NUM_THREADS and NCPUS are defined, the value of OMP_NUM_THREADS takes precedence. NOTE OMP_NUM_THREADS threads will be used to execute the program regardless of the number of physical processors available in the system. As a result, you can run programs using more threads than physical processors and they will execute correctly. However, performance of programs executed in this manner can be unpredictable, and oftentimes will be inefficient OMP_SCHEDULE - specifies the type of iteration scheduling to use for omp for and omp parallel for loops that include the schedule(runtime) clause. The default value for this variable is “static”. If the optional chunk size is not set, a chunk size of 1 is assumed except in the case of a static schedule. For a static schedule, the default is as defined in “omp for” on page 161. Examples of the use of OMP_SCHEDULE are as follows:Environment Variables 173 $ setenv OMP_SCHEDULE "static, 5" $ setenv OMP_SCHEDULE "guided, 8" $ setenv OMP_SCHEDULE "dynamic" OMP_DYNAMIC - currently has no effect. OMP_NESTED - currently has no effect. MPSTKZ - increase the size of the stacks used by threads executing in parallel regions. For use with programs that utilize large amounts of thread-local storage in the form of private variables or local variables in functions or subroutines called within parallel regions. The value should be an integer concatenated with M or m to specify stack sizes of n megabytes. For example: $ setenv MPSTKZ 8M OMP_WAIT_POLICY (Proposed OpenMP 3.0 Feature) - The OpenMP environment variable OMP_WAIT_POLICY sets the behavior of idle threads. The values are ACTIVE and PASSIVE. This behavior is also shared by threads created by auto-parallelization. Threads are considered idle when waiting at a barrier, when waiting to enter a critical region, or unemployed between parallel regions. Threads waiting for critical sections always busy wait. Barriers always busy wait with calls to sched_yield determined by MP_SPIN. Unemployed threads during a serial region can either busy wait using the barrier (ACTIVE) or politely wait using a mutex (PASSIVE). The choice is set by OMP_WAIT_POLICY. The default is ACTIVE. When ACTIVE is set, idle threads consume 100% of their CPU allotment spinning in a busy loop waiting to restart in a parallel region. This mechanism allows for very quick entry into parallel regions which is good for programs that enter and leave parallel regions frequently. When PASSIVE is set, idle threads wait on a mutex (in the operating system) and consume no CPU time until being restarted. Passive idle is best when a program has long periods of serial activity or when the program runs on a multi-user machine or otherwise shares CPU resources. OMP_STACK_SIZE (Proposed OpenMP 3.0 Feature) - The OpenMP environment variable OMP_STACK_SIZE overrides the default stack size for a newly created thread. The value is an decimal integer followed by an optional letter B, K, M, or G, to specify bytes, kilobytes, megabytes, and gigabytes, respectively. If no letter is used, the default is kilobytes. There is no space between the value and the letter; for example, one megabyte is specified 1M.OpenMP Pragmas for C and C++ 174 The environment variable OMP_STACK_SIZE is read on program start-up; if the program changes its own environment, the variable is not re-checked. This environment variable takes precedence over MPSTKSZ (see above). Once a thread is created, its stack size cannot be changed. In the PGI implementation, threads are created prior to the first parallel region and persist for the life of the program. The stack size of the main program is not affected by OMP_STACK_SIZE. Its size is set at program start up. For more information on controlling the program stack size in Linux, see “Running Parallel Programs on Linux” on page 10.PGI Proprietary Fortran Directives 175 7 Directives and Pragmas Directives are Fortran comments that the user may supply in a Fortran source file to provide information to the compiler. Directives alter the effects of certain command line options or default behavior of the compiler. While a command line option affects the entire source file that is being compiled, directives apply, or disable, the effects of a command line option to selected subprograms or to selected loops in the source file (for example, an optimization). Use directives to tune selected routines or loops. PGI Proprietary Fortran Directives PGI Fortran compilers support proprietary directives that may have any of the following forms: !pgi$g directive !pgi$r directive !pgi$l directive !pgi$ directive Either * or C is allowed in place of !. The scope indicator occurs after the $; this indicator controls the scope of the directive. Some directives ignore the scope indicator. The valid scopes, as shown above, are: g (global) indicates the directive applies to the end of the source file. r (routine) indicates the directive applies to the next subprogram. l (loop) indicates the directive applies to the next loop (but not to any loop contained within the loop body). Loop-scoped directives are only applied to DO loops. blank indicates that the default scope for the directive is applied. The body of the directive may immediately follow the scope indicator. Alternatively, any number of blanks may precede the name of the directive. Any names in the body of the directive, including the directive name, may not contain embedded blanks. Blanks may surround any special characters, such as a comma or an equal sign. The directive name, including the directive prefix, may contain upper or lower case letters (case is not significant). Case is significant for any variable names that appear in the body of the directive if the command line option –Mupcase is selected. For compatibility with other vendors’ directives, the prefix cpgi$ may be substituted with cdir$ or cvd$.Directives and Pragmas 176 PGI Proprietary Fortran Directive Summary The next table summarizes the supported Fortran directives. The scope entry indicates the allowed scope indicators for each directive; the default scope is surrounded by parentheses. The system field indicates the target system type for which the pragma applies. Many of the directives can be preceded by NO. The default entry in the table indicates the default of the directive; n/a appears if a default does not apply. The name of a directive may also be prefixed with –M; for example, the directive –Mbounds is equivalent to bounds and –Mopt is equivalent to opt.PGI Proprietary Fortran Directive Summary 177 Table 7-1: Proprietary Fortran Directive Summary Directive Function Default Scope altcode noaltcode Do/don’t generate alternate code for vectorized and parallelized loops altcode (l)rg assoc noassoc Do/don’t perform associative transformations assoc (l)rg bounds nobounds Do/don’t perform array bounds checking nobounds (r)g* cncall nocncall Loops are considered for parallelization, even if they contain calls to user-defined subroutines or functions, or if their loop counts do not exceed usual thresholds. nocncall (l)rg concur noconcur Do/don’t enable auto-concurrentization of loops concur (l)rg depchk nodepchk Do/don’t ignore potential data dependencies depchk (l)rg eqvchk noeqvchk Do/don’t check EQUIVALENCE s for data dependencies. eqvchk (l)rg invarif noinvarif Do/don’t remove invariant if constructs from loops. invarif (l)rg ivdep Ignore potential data dependencies depchk (l)rg lstval nolstval Do/don’t compute last values. lstval (l)rg opt Select optimization level. N/A (r)g safe_lastval Parallelize when loop contains a scalar used outside of loop. not enabled (l) tp Generate PGI Unified Binary code optimized for specified targets N/A (r)gDirectives and Pragmas 178 In the case of the vector/novector directive, the scope is the code following the directive until the end of the routine for r-scoped directives (as opposed to the entire routine), or until the end of the file for gscoped directives (as opposed to the entire file). altcode (noaltcode) Instructs the compiler to generate alternate code for vectorized or parallelized loops. The noaltcode directive disables generation of alternate code. This directive affects the compiler only when –Mvect or –Mconcur is enabled on the command line. cpgi$ altcode Enables alternate code (altcode) generation for vectorized loops. For each loop the compiler decides whether to generate altcode and what type(s) to generate, which may be any or all of: altcode without iteration peeling, altcode with nontemporal stores and other data cache optimizations, and altcode based on array alignments calculated dynamically at runtime. The compiler also determines suitable loop count and array alignment conditions for executing the alternate code. cpgi$ altcode alignment For a vectorized loop, if possible generate an alternate vectorized loop containing additional aligned moves which is executed if a runtime array alignment test is passed. cpgi$ altcode [(n)] concur For each auto-parallelized loop, generate an alternate serial loop to be executed if the loop count is less than or equal to n. If n is omitted or n is 0, the compiler determines a suitable value of n for each loop. unroll nounroll Do/don’t unroll loops. nounroll (l)rg vector novector Do/don't perform vectorizations. vector (l)rg vintr novintr Do/don’t recognize vector intrinsics. vintr (l)rg Directive Function Default ScopePGI Proprietary Fortran Directive Summary 179 cpgi$ altcode [(n)] concurreduction This directive sets the loop count threshold for parallelization of reduction loops to n. For each auto-parallelized reduction loop, generate an alternate serial loop to be executed if the loop count is less than or equal to n. If n is omitted or n is 0, the compiler determines a suitable value of n for each loop. cpgi$ altcode [(n)] nontemporal For a vectorized loop, if possible generate an alternate vectorized loop containing non-temporal stores and other cache optimizations to be executed if the loop count is greater than n. If n is omitted or n is 1, the compiler determines a suitable value of n for each loop. The alternate code is optimized for the case when the data referenced in the loop does not all fit in level 2 cache. cpgi$ altcode [(n)] nopeel For a vectorized loop where iteration peeling is performed by default, if possible generate an alternate vectorized loop without iteration peeling to be executed if the loop count is less than or equal to n. If n is omitted or n is 1, the compiler determines a suitable value of n for each loop, and in some cases it may decide not to generate an alternate unpeeled loop. For each vectorized loop, generate an alternate scalar loop to be executed if the loop count is less than or equal to n. If n is omitted or n is 1, the compiler determines a suitable value of n for each loop. cpgi$ altcode [(n)] vector cpgi$ noaltcodeThis directive sets the loop count thresholds for parallelization of all innermost loops to 0, and disables alternate code generation for vectorized loops. assoc (noassoc) This directive toggles the effects of the –Mvect=noassoc command-line option (an Optimization –M control). By default, when scalar reductions are present the vectorizer may change the order of operations so that it can generate better code (e.g., dot product). Such transformations change the result of the computation due to roundoff error. The noassoc directive disables these transformations. This directive affects the compiler only when –Mvect is enabled on the command line.Directives and Pragmas 180 bounds (nobounds) This directive alters the effects of the –Mbounds command line option. This directive enables the checking of array bounds when subscripted array references are performed. By default, array bounds checking is not performed. cncall (nocncall) Loops within the specified scope are considered for parallelization, even if they contain calls to userdefined subroutines or functions, or if their loop counts do not exceed the usual thresholds. A nocncall directive cancels the effect of a previous cncall. concur (noconcur) This directive alters the effects of the –Mconcur command-line option. The directive instructs the auto-parallelizer to enable auto-concurrentization of loops. If concur is specified, multiple processors will be used to execute loops which the auto-parallelizer determines to be parallelizable. The noconcur directive disables these transformations. This directive affects the compiler only when – Mconcur is enabled on the command line. depchk (nodepchk) This directive alters the effects of the –Mdepchk command line option. When potential data dependencies exist, the compiler, by default, assumes that there is a data dependence that in turn may inhibit certain optimizations or vectorizations. nodepchk directs the compiler to ignore unknown data dependencies. eqvchk (noeqvchk) When examining data dependencies, noeqvchk directs the compiler to ignore any dependencies between variables appearing in EQUIVALENCE statements. invarif (noinvarif) There is no command-line option corresponding to this directive. Normally, the compiler removes certain invariant if constructs from within a loop and places them outside of the loop. The directive noinvarif directs the compiler to not move such constructs. The directive invarif toggles a previous noinvarif. ivdep The ivdep directive is equivalent to the directive nodepchk.PGI Proprietary Fortran Directive Summary 181 opt The syntax of this directive is: cpgi$ opt= where, the optional is r or g and is an integer constant representing the optimization level to be used when compiling a subprogram (routine scope) or all subprograms in a file (global scope). The opt directive overrides the value specified by the command line option –On. lstval (nolstval) There is no command line option corresponding to this directive. The compiler determines whether the last values for loop iteration control variables and promoted scalars need to be computed. In certain cases, the compiler must assume that the last values of these variables are needed and therefore computes their last values. The directive nolstval directs the compiler not to compute the last values for those cases. safe_lastval During parallelization scalars within loops need to be privatized. Problems are possible if a scalar is accessed outside the loop. For example: do i = 1, N if( f(x(i)) > 5.0 ) t = x(i) enddo v = t creates a problem since the value of t may not be computed on the last iteration of the loop. If a scalar assigned within a loop is used outside the loop, we normally save the last value of the scalar. Essentially the value of the scalar on the "last iteration" is saved, in this case when i = N. If the loop is parallelized and the scalar is not assigned on every iteration, it may be difficult to determine on what iteration t is last assigned, without resorting to costly critical sections. Analysis allows the compiler to determine if a scalar is assigned on every iteration, thus the loop is safe to parallelize if the scalar is used later. An example loop is: do i = 1, N if( x(i) > 0.0 ) t = 2.0 else t = 3.0Directives and Pragmas 182 endif y(i) = ...t... enddo v = t where t is assigned on every iteration of the loop. However, there are cases where a scalar may be privatizable. If it is used after the loop, it is unsafe to parallelize. Examine this loop: do i = 1,N if( x(i) > 0.0 ) t = x(i) ... ... y(i) = ...t.. endif enddo v = t where each use of t within the loop is reached by a definition from the same iteration. Here t is privatizable, but the use of t outside the loop may yield incorrect results since the compiler may not be able to detect on which iteration of the parallelized loop t is assigned last. The compiler detects the above cases. Where a scalar is used after the loop, but is not defined on every iteration of the loop, parallelization will not occur. If you know that the scalar is assigned on the last iteration of the loop, making it safe to parallelize the loop, a pragma is available to let the compiler know the loop is safe to parallelize. Use the following C pragma to tell the compiler that for a given loop the last value computed for all scalars make it safe to parallelize the loop: cpgi$l safe_lastval In addition, a command-line option, -Msafe_lastval, provides this information for all loops within the routines being compiled (essentially providing global scope.) tp The directive tp is used to specify one or more processor targets for which to generate code. cpgi$ tp [target]...Scope of Directives and Command Line options 183 See “Processor Options” on page xxi for a list of targets that can be used as parameters to the tp directive. See “Processor-Specific Optimization and the Unified Binary” on page 35 for more information on unified binaries. unroll (nounroll) The directive nounroll is used to disable loop unrolling and unroll to enable unrolling. The directive takes arguments c and n. A c specifies that c (complete unrolling should be turned on or off) An n specifies that n (count) unrolling should be turned on or off. In addition, the following arguments may be added to the unroll directive: cpgi$ unroll = c:v This sets the threshold to which c unrolling applies; v is a constant; a loop whose constant loop count is <= v is completely unrolled. cpgi$ unroll = n:v This adjusts threshold to which n unrolling applies; v is a constant; a loop to which n unrolling applies is unrolled v times. The directives unroll and nounroll only apply if –Munroll is selected on the command line. vector (novector) The directive novector is used to disable vectorization. The directive vector is used to re-enable vectorization after a previous novector directive. The directives vector and novector only apply if – Mvect has been selected on the command line. vintr (novintr) The directive novintr directs the vectorizer to disable recognition of vector intrinsics. The directive vintr is used to re-enable recognition of vector intrinsics after a previous novintr directive. The directives vintr and novintr only apply if –Mvect has been selected on the command line. Scope of Directives and Command Line options This section presents several examples showing the effect of directives and the scope of directives. Remember that during compilation, the effect of a directive may be to either turn an option on, or turn an option off. Directives apply to the section of code following the directive, corresponding to the specified scope (that is, the following loop, the following routine, or the rest of the program).Directives and Pragmas 184 Consider the following code: integer maxtime, time parameter (n = 1000, maxtime = 10) double precision a(n,n), b(n,n), c(n,n) do time = 1, maxtime do i = 1, n do j = 1, n c(i,j) = a(i,j) + b(i,j) enddo enddo enddo end When compiled with –Mvect, both interior loops are interchanged with the outer loop. $ pgf95 -Mvect dirvect1.f Directives alter this behavior either globally or on a routine or loop by loop basis. To assure that vectorization is not applied, use the novector directive with global scope. cpgi$g novector integer maxtime, time parameter (n = 1000, maxtime = 10) double precision a(n,n), b(n,n), c(n,n) do time = 1, maxtime do i = 1, n do j = 1, n c(i,j) = a(i,j) + b(i,j) enddo enddo enddo end In this version, the compiler disables vectorization for the entire source file. Another use of the directive scoping mechanism turns an option on or off locally, either for a specific procedure or for a specific loop: integer maxtime, time parameter (n = 1000, maxtime = 10) double precision a(n,n), b(n,n), c(n,n) cpgi$l novector do time = 1, maxtime do i = 1, n do j = 1, n!DEC$ Directives for Windows Fortran 185 c(i,j) = a(i,j) + b(i,j) enddo enddo enddo end Loop level scoping does not apply to nested loops. That is, the directive only applies to the following loop. In this example, the directive turns off vector transformations for the top-level loop. If the outer loop were a timing loop, this would be a practical use for a loop-scoped directive. !DEC$ Directives for Windows Fortran PGI Fortran compilers for Microsoft Windows support several de-facto standard Fortran directives that help with interlanguage calling and importing and exporting routines to and from DLLs. These directives all take the form: !DEC$ directive Syntax: ATTRIBUTES Clause !DEC$ ATTRIBUTES where is one of: ALIAS : 'alias_name' :: routine_name Specifies an alternative name with which to resolve routine_name. C Same as STDCALL on Win64 DLLEXPORT :: name Specifies that 'name' is being exported to other applications or DLL's DLLIMPORT :: name Specifies that 'name' is being imported from other applications or DLL's REFERENCE :: name Specifies that the argument 'name' is being passed by reference. Often this attribute is used in conjuction with STDCALL, where STDCALL refers to an entire routine,then individual arguments are modified with REFERENCE.Directives and Pragmas 186 STDCALL :: routine_name Specifies that routine 'routine_name' will have its arguments passed by value. When a routine marked STDCALL is called, arguments (except arrays and characters) will be sent by value. The standard F90/F95 calling convention is by reference. VALUE :: name Specifies that the argument 'name' is being passed by value. Often used to specify that a particular argument is being passed by value. Loop Distribution Directive !DEC$ DISTRIBUTE POINT This directive is front-end based, and tells the compiler at what point within a loop to split into two loops. subroutine dist(a,b,n) integer i integer n integer a(*) integer b(*) do i = 1,n a(i) = a(i)+2 !DEC$ DISTRIBUTE POINT b(i) = b(i)*4 enddo end subroutine !DEC$ DISTRIBUTEPOINT is same as !DEC$ DISTRIBUTE POINT ALIAS Attribute !DEC$ ALIAS same as !DEC$ ATTRIBUTES ALIASAdding Pragmas to C and C++ 187 Adding Pragmas to C and C++ Pragmas may be supplied in a C/C++ source file to provide information to the compiler. Like directives in Fortran, pragmas alter the effects of certain command-line options or default behavior of the compiler (many pragmas have a corresponding command-line option). While a command-line option affects the entire source file that is being compiled, pragmas apply the effects of a particular commandline option to selected functions or to selected loops in the source file. Pragmas may also toggle an option, selectively enabling and disabling the option. Pragmas let you tune selected functions or loops based on your knowledge of the code. The general syntax of a pragma is: #pragma [ scope ] pragma-body The optional scope field is an indicator for the scope of the pragma; some pragmas ignore the scope indicator. The valid scopes are: global indicates the pragma applies to the entire source file. routine indicates the pragma applies to the next function. loop indicates the pragma applies to the next loop (but not to any loop contained within the loop body). Loop-scoped pragmas are only applied to for and while loops. If a scope indicator is not present, the default scope, if any, is applied. Whitespace must appear after the pragma keyword and between the scope indicator and the body of the pragma. Whitespace may also surround any special characters, such as a comma or an equal sign. Case is significant for the names of the pragmas and any variable names that appear in the body of the pragma. C/C++ Pragma Summary The following table summarizes the supported pragmas. The scope entry in the table indicates the permitted scope indicators for each pragma: the letters L, R, and G indicate loop, routine, and global scope, respectively. The default scope is surrounded by parentheses. The "*" in the scope field indicates that the scope is the code following the pragma until the end of the routine for R-scoped pragmas, as opposed to the entire routine, or until the end of the file for G-scoped pragmas, as opposed to the entire file. Directives and Pragmas 188 Many of the pragmas can be preceded by no. The default entry in the table indicates the default of the pragma; N/A appears if a default does not apply. The name of any pragma may be prefixed with -M; for example, –Mnoassoc is equivalent to noassoc and –Mvintr is equivalent to vintr. The section following the table provides brief descriptions of the pragmas that are unique to C/C++. Pragmas that have a corresponding directive in Fortran are described in “PGI Proprietary Fortran Directive Summary” on page 176.C/C++ Pragma Summary 189 Table 7-2: C/C++ Pragma Summary Pragma Function Default Scope altcode noaltcode Do/don’t generate alternate code for vectorized and parallelized loops altcode (L)RG assoc noassoc Do/don’t perform associative transformations. assoc (L)RG bounds nobounds Do/don’t perform array bounds checking. nobounds (R)G concur noconcur Do/don’t enable auto-concurrentization of loops. concur (L)RG depchk nodepchk Do/don’t ignore potential data dependencies. depchk (L)RG fcon nofcon Do/don’t assume unsuffixed real constants are single precision. nofcon (R)G invarif noinvarif Do/don’t remove invariant if constructs from loops. invarif (L)RG lstval nolstval Do/don’t compute last values. lstval (L)RG opt Select optimization level. N/A (R)G safe nosafe Do/don’t treat pointer arguments as safe. safe (R)G safe_lastval Parallelize when loop contains a scalar used outside of loop. not enabled (L) safeptr nosafeptr Do/don’t ignore potential data dependencies to pointers. nosafeptr L(R)G single nosingle Do/don’t convert float parameters to double. nosingle (R)G* tp Generate PGI Unified Binary code optimized for specified targets N/A (R)GDirectives and Pragmas 190 fcon (nofcon) This pragma alters the effects of the –Mfcon command-line option (a –M Language control). The pragma instructs the compiler to treat non-suffixed floating-point constants as float rather than double. By default, all non-suffixed floating-point constants are treated as double. safe (nosafe) By default, the compiler assumes that all pointer arguments are unsafe. That is, the storage located by the pointer can be accessed by other pointers. The forms of the safe pragma are: #pragma [scope] [no]safe #pragma safe (variable [, variable]...) where scope is either global or routine. When the pragma safe is not followed by a variable name (or name list), all pointer arguments appearing in a routine (if scope is routine) or all routines (if scope is global) will be treated as safe. If variable names occur after safe, each name is the name of a pointer argument in the current function. The named argument is considered to be safe. Note that if just one variable name is specified, the surrounding parentheses may be omitted. There is no command-line option corresponding to this pragma. safeptr (nosafeptr) The pragma safeptr directs the compiler to treat pointer variables of the indicated storage class as safe. The pragma nosafeptr directs the compiler to treat pointer variables of the indicated storage class as unsafe. This pragma alters the effects of the –Msafeptr command-line option. unroll nounroll Do/don’t unroll loops. nounroll (L)RG vector novector Do/don’t perform vectorizations. vector (L)RG vintr novintr Do/don’t recognize vector intrinsics. vintr (L)RG Pragma Function Default ScopeScope of C/C++ Pragmas and Command Line Options 191 The syntax of this pragma is: #pragma [scope] value where value is: [no]safeptr={arg|local|auto|global|static|all},... Note that the values local and auto are equivalent. For example, in a file containing multiple functions, the command-line option –Msafeptr might be helpful for one function, but can’t be used because another function in the file would produce incorrect results. In such a file, the safeptr pragma, used with routine scope could improve performance and produce correct results. single (nosingle) The pragma single directs the compiler not to convert float parameters to double in non-prototyped functions. This can result in faster code if the program uses only float parameters. Note Since ANSI C specifies that routines must convert float parameters to double in nonprototyped functions, this pragma results in non-ANSI conforming code. Scope of C/C++ Pragmas and Command Line Options This section presents several examples showing the effect of pragmas and the use of the pragma scope indicators. Note during compilation a pragma either turns an option on or turns an option off. Pragmas apply to the section of code corresponding to the specified scope (that is, the entire file, the following loop, or the following or current routine). For pragmas that have only routine and global scope, there are two rules for determining the scope of a pragma. We cover these special scope rules at the end of this section. In all cases, pragmas override a corresponding command-line option. Consider the program: main() { float a[100][100], b[100][100], c[100][100]; int time, maxtime, n, i, j; maxtime=10; n=100;Directives and Pragmas 192 for (time=0; time[,[,...]] where is any valid variable or array element reference. NOTE The sentinel for prefetch directives is c$mem, which is distinct from the cpgi$ sentinel used for optimization directives. Any prefetch directives that use the cpgi$ sentinel will be ignored by the PGI compilers.Prefetch Directives 195 The "c" must be in column 1. Either * or ! is allowed in place of c. The scope indicators g, r and l used with the cpgi$ sentinel are not supported. The directive name, including the directive prefix, may contain upper or lower case letters (case is not significant). Case is significant for any variable names that appear in the body of the directive if the command line option –Mupcase is selected. An example using prefetch directives to prefetch data in a matrix multiplication inner loop where a row of one source matrix has been gathered into a contiguous vector might look as follows: real*8 a(m,n), b(n,p), c(m,p), arow(n) ... do j = 1, p c$mem prefetch arow(1),b(1,j) c$mem prefetch arow(5),b(5,j) c$mem prefetch arow(9),b(9,j) do k = 1, n, 4 c$mem prefetch arow(k+12),b(k+12,j) c(i,j) = c(i,j) + arow(k) * b(k,j) c(i,j) = c(i,j) + arow(k+1) * b(k+1,j) c(i,j) = c(i,j) + arow(k+2) * b(k+2,j) c(i,j) = c(i,j) + arow(k+3) * b(k+3,j) enddo enddo This pattern of prefetch directives will cause the compiler to emit prefetch instructions whereby elements of arow and b are fetched into the data cache starting 4 iterations prior to first use. By varying the prefetch distance in this way, it is possible in some cases to reduce the effects of main memory latency and improve performance.Directives and Pragmas 196Using builtin Math Functions in C/C++ 197 8 Libraries and Environment Variables This chapter discusses issues related to PGI-supplied compiler libraries. It also addresses the use of C/ C++ builtin functions in place of the corresponding libc routines, creation of dynamically linked libraries (also known as shared objects or shared libraries), and math libraries. Using builtin Math Functions in C/C++ The name of the math header file is math.h. Include the math header file in all of your source files that use a math library routine as in the following example, which calculates the inverse cosine of pi/3. #include #define PI 3.1415926535 main() { double x, y; x = PI/3.0; y = acos(x); } Including math.h will cause PGCC C and C++ to use builtin functions, which are much more efficient than library calls. In particular, the following intrinsics calls will be processed using builtins if you include math.h: atan2 cos fabs exp atan sin log pow tan sqrt log10Libraries and Environment Variables 198 Creating and Using Shared Object Files on Linux All of the PGI Fortran, C, and C++ compilers support creation of shared object files. Unlike statically linked object and library files, shared object files link and resolve references with an executable at runtime via a dynamic linker supplied with your operating system. The PGI compilers must generate position independent code to support creation of shared objects by the linker. This is not the default, use the steps that follow to create object files with position independent code and shared object files that are to include them. The following steps describe how to create and use a shared object file. 1. To create an object file with position independent code, compile it with the appropriate PGI compiler using the -fpic option (the -fPIC, -Kpic, and -KPIC options are supported for compatibility with other systems you may have used, and are equivalent to-fpic). For example, use the following command to create an object file with position independent code using pgf95: % pgf95 -c -fpic tobeshared.f 2. To produce a shared object file, use the appropriate PGI compiler to invoke the linker supplied with your system. It is customary to name such files using a .so filename extension. On Linux, this is done by passing the -shared option to the linker: % pgf95 -shared -o tobeshared.so tobeshared.o Note that compilation and generation of the shared object can be performed in one step using both the -fpic option and the appropriate option for generation of a shared object file. 3. To use a shared object file, compile and link the program which will reference functions or subroutines in the shared object file using the appropriate PGI compiler and listing the shared object on the link line: % pgf95 -o myprog myprof.f tobeshared.so 4. You now have an executable myprog which does not include any code from functions or subroutines in tobeshared.so, but which can be executed and dynamically linked to that code. By default, when the program is linked to produce myprog, no assumptions are made on the location of tobeshared.so. In order for myprog to execute correctly, you must initialize the environment variable LD_LIBRARY_PATH to include the directory containing tobeshared.so. If LD_LIBRARY_PATH is already initialized, it is important not to overwrite its contents. Assuming you have placed tobeshared.so in a directory /home/myusername/bin, you can initialize LD_LIBRARY_PATH to include that directory and preserve its existing contents as follows: % setenv LD_LIBRARY_PATH "$LD_LIBRARY_PATH":/home/myusername/binCreating and Using Dynamic-Link Libraries on Windows 199 If you know that tobeshared.so will always reside in a specific directory, you can create the executable myprog in a form that assumes this using the -R link-time option. For example, you can link as follows: % pgf95 -o myprog myprof.f tobeshared.so -R/home/myusername/bin Note that there is no space between -R and the directory name. As with the -L option, no space can be present. If the -R option is used, it is not necessary to initialize LD_LIBRARY_PATH. In the example above, the dynamic linker will always look in /home/myusername/bin to resolve references to tobeshared.so. By default, if the LD_LIBRARY_PATH environment variable is not set, the linker will only search /usr/lib for shared objects. The command ldd is a useful tool when working with shared object files and executables that reference them. When applied to an executable as follows: % ldd myprog ldd lists all shared object files referenced in the executable along with the pathname of the directory from which they will be extracted. If the pathname is not hard-coded using the -R option, and if LD_LIBRARY_PATH is not initialized, the pathname is listed as “not found”. See the online man page for ldd for more information on options and usage. Creating and Using Dynamic-Link Libraries on Windows Some of the PGI compiler runtime libraries are available in both static library and dynamic-link library (DLL) form for Windows. The static libraries are always used by default. To use the PGI Workstation C and Fortran compilers to create an executable that links to the runtime DLLs, use the compiler flag – Mdll at the link step. There are several differences between static and dynamic-link libraries. Both libraries are used when resolving external references when linking an executable, but the process differs for each type of library. When linking with a static library, the code needed from the library is incorporated into the executable. When linking with a DLL, external references are resolved using the DLL's import library, not the DLL itself. The code in the DLL associated with the external references does not become a part of the executable. The DLL is loaded when the executable that needs it is run. For the DLL to be loaded in this manner, the DLL must be in your path. Static libraries and DLLs also handle global data differently. Global data in static libraries is automatically accessible to other objects linked into an executable. Global data in a DLL can only be accessed from outside the DLL if the DLL exports the data and the image that uses the data imports it. To Libraries and Environment Variables 200 this end the C compilers support the Microsoft storage class extensions __declspec(dllimport) and __declspec(dllexport). These extensions may appear as storage class modifiers and enable functions and data to be imported and exported: extern int __declspec(dllimport) intfunc(); float __declspec(dllexport) fdata; The Fortran compilers support the DEC ATTRIBUTES extensions DLLIMPORT and DLLEXPORT: cDEC$ ATTRIBUTES DLLEXPORT :: object [,object] ... cDEC$ ATTRIBUTES DLLIMPORT :: object [,object] ... c is one of C, c, !, or *. object is the name of the subprogram or common block that is exported or imported. Note that common block names are enclosed within slashes (/). In example: cDEC$ ATTRIBUTES DLLIMPORT :: intfunc !DEC$ ATTRIBUTES DLLEXPORT :: /fdata/ The Examples in this section further illustrate the use of these extensions. To create a DLL from the command line, use the –Mmakedll option. The following switches apply to making and using DLLs with the PGI compilers: –Mdll Link with the DLL version of the runtime libraries. This flag is required when linking with any DLL built by the PGI compilers. –Mmakedll Generate a dynamic-link library or DLL. –Mmakeimplib Generate an import library without generating a DLL. Use this flag when you want to generate an import library for a DLL but are not yet ready to build the DLL itself. This situation might arise, for example, when building DLLs with mutual imports (see Example 4 below). –o Passed to the linker. Name the DLL or import library . –def When used with –Mmakedll, this flag is passed to the linker and a .def file named is generated for the DLL. The .def file contains the symbols exported by the DLL. Generating a .def file is not required when building a DLL but can be a useful debugging tool if the DLL does not contain the symbols that you expect it to contain. Creating and Using Dynamic-Link Libraries on Windows 201 When used with –Mmakeimplib, this flag is passed to lib which requires a .def file to create an import library. The .def file can be empty if the list of symbols to export are passed to lib on the command line or explicitly marked as dllexport in the source code. –implib Passed to linker. Generate an import library named for the DLL. A DLL’s import library is the interface used when linking an executable that depends on routines in a DLL. To use the PGI compilers to create an executable that links to the DLL form of the runtime, use the compiler flag –Mdll. The executable built will be smaller than one built without –Mdll; the PGI runtime DLLs, however, must be available on the system where the executable is run. The –Mdll flag must be used when an executable is linked against a DLL built by the PGI compilers. The following examples outline how to use –Mmakedll and –Mmakeimplib to build and use DLLs with the PGI compilers. Example 1: Build a DLL out of a single source file, object1.f, which exports data and a subroutine using DLLEXPORT. Build the main source file, prog1.f, which uses DLLIMPORT to import the data and subroutine from the DLL. object1.f: subroutine sub1(i) !DEC$ ATTRIBUTES DLLEXPORT :: sub1 integer i common /acommon/ adata integer adata !DEC$ ATTRIBUTES DLLEXPORT :: /acommon/ print *, "sub1 adata", adata print *, "sub1 i ", i adata = i end prog1.f: program prog1 common /acommon/ adata integer adata external sub1 !DEC$ ATTRIBUTES DLLIMPORT:: sub1, /acommon/ adata = 11 call sub1(12) print *, "main adata", adata endLibraries and Environment Variables 202 Step 1: Create the DLL obj1.dll and its import library obj1.lib using the following series of commands: % pgf95 -c object1.f % pgf95 –Mmakedll object1.obj -o obj1.dll Step 2: Compile the main program: % pgf95 -Mdll -o prog1 prog1.f -defaultlib:obj1 The –Mdll switch causes the compiler to link against the PGI runtime DLLs instead of the PGI runtime static libraries. The –Mdll switch is required when linking against any PGI-compiled DLL such as obj1.dll. The -defaultlib: switch is used to specify that obj1.lib, the DLL’s import library, should be used to resolve imports. Step 3: Ensure that obj1.dll is in your path, then run the executable prog1 to determine if the DLL was successfully created and linked: % prog1 sub1 adata 11 sub1 i 12 main adata 12 Should you wish to change obj1.dll without changing the subroutine or function interfaces, no rebuilding of prog1 is necessary. Just recreate obj1.dll and the new obj1.dll will be loaded at runtime. Example 2: Build a DLL out of a single source file, object2.c, which exports data and a subroutine using __declspec(dllexport). Build the main source file, prog2.c, which uses __declspec(dllimport) to import the data and subroutine from the DLL. object2.c: int __declspec(dllexport) data; void __declspec(dllexport) func2(int i) { printf("func2: data == %d\n", data); printf("func2: i == %d\n", i); data = i; } prog2.c: int __declspec(dllimport) data; void __declspec(dllimport) func2(int); intCreating and Using Dynamic-Link Libraries on Windows 203 main() { data = 11; func2(12); printf("main: data == %d\n", data); return 0; } Step 1: Create the DLL obj2.dll and its import library obj2.lib using the following series of commands: % pgcc -c object2.c % pgcc -Mmakedll object2.obj -o obj2.dll Step 2: Compile the main program: % pgcc -Mdll -o prog2 prog2.c -defaultlib:obj2 The –Mdll switch causes the compiler to link against the PGI runtime DLLs instead of the PGI runtime static libraries. The –Mdll switch is required when linking against any PGI-compiled DLL such as obj2.dll. The -defaultlib: switch is used to specify that obj2.lib, the DLL’s import library, should be used to resolve the imported data and subroutine in prog2.c. Step 3: Ensure that obj2.dll is in your path, then run the executable prog2 to determine if the DLL was successfully created and linked: PGI$ prog2 func2: data == 11 func2: i == 12 main: data == 12 Should you wish to change obj2.dll without changing the subroutine or function interfaces, no rebuilding of prog2 is necessary. Just recreate obj2.dll and the new obj2.dll will be loaded at runtime. Example 3: DLLs Containing Circular Imports The two DLLs to be built in this example, obj3.dll and obj4.dll, each import a routine exported by the other. In order to link the first DLL, the import library for the second DLL must be available. Usually an import library is created when a DLL is linked. In this case, however, the second DLL cannot be linked without the import library for the first DLL. When such circular imports exist, an import library for one of the DLLs must be created in a separate step without creating the DLL. The PGI drivers call the Microsoft lib tool to create import libraries for this situation.Libraries and Environment Variables 204 object3.c: void __declspec(dllimport) func_4b(void); void __declspec(dllexport) func_3a(void) { printf("func_3a, calling a routine in obj4.dll\n"); func_4b(); } void __declspec(dllexport) func_3b(void) { printf("func_3b\n"); } object4.c: void __declspec(dllimport) func_3b(void); void __declspec(dllexport) func_4a(void) { printf("func_4a, calling a routine in obj3.dll\n"); func_3b(); } void __declspec(dllexport) func_4b(void) { printf("func_4b\n"); } prog3.c: void __declspec(dllimport) func_3a(void); void __declspec(dllimport) func_4a(void); int main() { func_3a(); func_4a(); return 0; } Step 1: Use the -Mmakelib and -def options with the PGI compilers to build an import library for the first DLL without building the DLL itself. % pgcc -c object3.c % pgcc -Mmakeimplib -o obj3.lib object3.objCreating and Using Dynamic-Link Libraries on Windows 205 The -def= option can also be used with -Mmakeimplib. Use a .def file when you need to export additional symbols from the DLL. A .def file is not needed in this example because all symbols are exported using __declspec(dllexport). Step 2: Use the import library created in Step 1, obj3.lib, to link the second DLL. % pgcc -c object4.c % pgcc -Mmakedll -o obj4.dll object4.obj -defaultlib:obj3 Step 3: Use the import library created in Step 2, obj4.lib, to link the first DLL. % pgcc -Mmakedll -o obj3.dll object3.obj -defaultlib:obj4 Step 4: Compile the main program and link against the import libraries for the two DLLs. % pgcc -Mdll prog3.c -o prog3 -defaultlib:obj3 -defaultlib:obj4 Step 5: Execute prog3.exe to ensure that the DLLs were create properly. % prog3 func_3a, calling a routine in obj4.dll func_4b func_4a, calling a routine in obj3.dll func_3b Example 5: Importing a Fortran module from a DLL. The source file my_module_def.f90 is used to create a DLL containing a Fortran module. the source file my_module_use.f90 is used to build a program that imports and uses the Fortran module from my_module_def.f90. !------------------------------------------------------------- ! my_module_def.f90 !------------------------------------------------------------- MODULE TESTM TYPE A_TYPE INTEGER :: AN_INT END TYPE A_TYPE TYPE(A_TYPE) :: A, B !DEC$ ATTRIBUTES DLLEXPORT :: A,B CONTAINS SUBROUTINE PRINT_A !DEC$ ATTRIBUTES DLLEXPORT :: PRINT_A WRITE(*,*) A%AN_INTLibraries and Environment Variables 206 END SUBROUTINE SUBROUTINE PRINT_B !DEC$ ATTRIBUTES DLLEXPORT :: PRINT_B WRITE(*,*) B%AN_INT END SUBROUTINE END MODULE !------------------------------------------------------- ! my_module_use.f90 !------------------------------------------------------ USE TESTM A%AN_INT = 1 B%AN_INT = 2 CALL PRINT_A CALL PRINT_B END Step 1: Create the DLL. % pgf90 -Mmakedll -o my_module_def.dll my_module_def.f90 Creating library my_module_def.lib and object my_module_def.exp Step 2: Create the EXE and link against the import library for the imported DLL. % pgf90 -Mdll -o my_module_use.exe my_module_use.f90 -defaultlib:my_module_def.lib Step 3: Run the EXE to ensure that the module was imported from the DLL properly. % my_module_use.exe 1 2 Using LIB3F The PGI Fortran compilers include complete support for the de facto standard LIB3F library routines on both Linux and Windows operating systems. See the PGI Fortran Reference manual for a complete list of available routines in the PGI implementation of LIB3F.LAPACK, the BLAS and FFTs 207 LAPACK, the BLAS and FFTs Pre-compiled versions of the public domain LAPACK and BLAS libraries are included with the PGI compilers on Linux and Windows systems in the files $PGI//lib/lapack.a and $PGI// lib/blas.a respectively, where is replaced with the appropriate target name (linux86, linux86- 64, win64, win32, sua32, or sua64). (Note that sua32 is used for SFU.) To use these libraries, simply link them in using the -l option when linking your main program: % pgf95 myprog.f -llapack -lblas Highly optimized assembly-coded versions of the BLAS and certain FFT routines may be available for your platform. In some cases, these are shipped with the PGI compilers. See the current release notes for the PGI compilers you are using to determine if these optimized libraries exist, where they can be downloaded (if necessary), and how to incorporate them into your installation as the default. The C++ Standard Template Library The PGC++ compiler includes a bundled copy of the STLPort Standard C++ Library. See the online Standard C++ Library tutorial and reference manual at http://www.stlport.com for further details and licensing. Environment Variables Several environment variables can be used to alter the default behavior of the PGI compilers and the executables which they generate. Many of these environment variables are documented in context in other sections of the PGI User’s Guide. They are gathered here for easy reference. Specifically excluded are environment variables specific to OpenMP which are used to control the behavior of OpenMP programs. See section 5.17, Environment Variables, for a list and description of environment variables that affect the execution of Fortran OpenMP programs. See section 6.16, Environment Variables, for a list and description of environment variables that affect the execution of C and C++ OpenMP programs. Also excluded are environment variables that control the behavior of the PGDBG debugger or PGPROF profiler. See the PGI Tools Guide for a description of environment variables that affect these tools. Libraries and Environment Variables 208 FORTRAN_OPT - If this variable exists and contains the value vaxio, the record length in the open statement is in units of 4-byte words, and the $ edit descriptor only has an effect for lines beginning with a space or +. If this variable exists and contains the value format_relaxed, an I/O item corresponding to a numerical edit descriptor (F, E, I, etc.) is not required to be a type implied by the descriptor. For example: $ setenv FORTRAN_OPT vaxio will cause the PGI Fortran compilers to use VAX I/O conventions as defined above. MPSTKZ - increase the size of the stacks used by threads executing in parallel regions. It is for use with programs that utilize large amounts of thread-local storage in the form of private variables or local variables in functions or subroutines called within parallel regions. The value should be an integer concatenated with M or m to specify stack sizes of n megabytes. For example: $ setenv MPSTKZ 8M MP_BIND - the MP_BIND environment variable can be set to yes or y to bind processes or threads executing in a parallel region to physical processors, or to no or n to disable such binding. The default is to not bind processes to processors. This is an execution time environment variable interpreted by the PGI runtime support libraries. It does not affect the behavior of the PGI compilers in any way. Note: the MP_BIND environment variable is not supported on all platforms. MP_BLIST - In addition to the MP_BIND variable, it is possible to define the thread-CPU relationship. For example, setting MP_BLIST=3,2,1,0 maps CPUs 3, 2, 1 and 0 to threads 0, 1, 2 and 3 respectively. MP_SPIN - When a thread executing in a parallel region enters a barrier, it spins on a semaphore. MP_SPIN can be used to specify the number of times it checks the semaphore before calling sched_yield() (on linux) or _sleep() (on Windows). These calls cause the thread to be re-scheduled, allowing other processes to run. The default values are 100 (Linux) and 10000 (Windows). MP_WARN - By default, a warning will be printed to stderr if you execute an OpenMP or autoparallelized program with NCPUS or OMP_NUM_THREADS set to a value larger than the number of physical processors in the system. For example, if you produce a parallelized executable a.out and execute as follows on a system with only one processor: % setenv NCPUS 2 % a.out Warning: OMP_NUM_THREADS or NCPUS (2) greater than available cpus (1) FORTRAN STOPEnvironment Variables 209 Setting MP_WARN to no will eliminate these warning messages. NCPUS - The NCPUS environment variable can be used to set the number of processes or threads used in parallel regions. The default is to use only one process or thread (serial mode). If both OMP_NUM_THREADS and NCPUS are set, the value of OMP_NUM_THREADS takes precedence. Warning: setting NCPUS to a value larger than the number of physical processors or cores in your system can cause parallel programs to run very slowly. NCPUS_MAX - The NCPUS_MAX environment variable can be used to limit the maximum number of processes or threads used in a parallel program. Attempts to dynamically set the number of processes or threads to a higher value, for example using set_omp_num_threads(), will cause the number of processes or threads to be set at the value of NCPUS_MAX rather than the value specified in the function call. NO_STOP_MESSAGE - If this variable exists, the execution of a plain STOP statement does not produce the message FORTRAN STOP. The default behavior of the PGI Fortran compilers is to issue this message. OMP_WAIT_POLICY (Proposed OpenMP 3.0 Feature) - The OpenMP environment variable OMP_WAIT_POLICY sets the behavior of idle threads. The values are ACTIVE and PASSIVE. This behavior is also shared by threads created by auto-parallelization. Threads are considered idle when waiting at a barrier, when waiting to enter a critical region, or unemployed between parallel regions. Threads waiting for critical sections always busy wait. Barriers always busy wait with calls to sched_yield determined by MP_SPIN. Unemployed threads during a serial region can either busy wait using the barrier (ACTIVE) or politely wait using a mutex (PASSIVE). The choice is set by OMP_WAIT_POLICY. The default is ACTIVE. When ACTIVE is set, idle threads consume 100% of their CPU allotment spinning in a busy loop waiting to restart in a parallel region. This mechanism allows for very quick entry into parallel regions which is good for programs that enter and leave parallel regions frequently. When PASSIVE is set, idle threads wait on a mutex (in the operating system) and consume no CPU time until being restarted. Passive idle is best when a program has long periods of serial activity or when the program runs on a multi-user machine or otherwise shares CPU resources. OMP_STACK_SIZE (Proposed OpenMP 3.0 Feature) - The OpenMP environment variable OMP_STACK_SIZE overrides the default stack size for a newly created thread. The value is an decimal integer followed by an optional letter B, K, M, or G, to specify bytes, kilobytes, megabytes, and gigabytes, respectively. If no letter is used, the default is kilobytes. There is no space between the value and the letter; for example, one megabyte is specified 1M.Libraries and Environment Variables 210 The environment variable OMP_STACK_SIZE is read on program start-up; if the program changes its own environment, the variable is not re-checked. This environment variable takes precedence over MPSTKSZ (see above). Once a thread is created, its stack size cannot be changed. In the PGI implementation, threads are created prior to the first parallel region and persist for the life of the program. The stack size of the main program is not affected by OMP_STACK_SIZE. Its size is set at program start up. For more information on controlling the program stack size in Linux, see “Running Parallel Programs on Linux” on page 10. PGI - The PGI environment variable specifies the root directory where the PGI compilers and tools are installed. The default value of this variable is /usr/pgi. In most cases, the name of this root directory is derived dynamically by the PGI compilers and tools through determination of the path to the instance of the compiler or tool that has been invoked. However, there are still some dependencies on the PGI environment variable, and it can be used as a convenience when initializing your environment for use of the PGI compilers and tools. For example, assuming you use csh and want the 64-bit linux86-64 versions of the PGI compilers and tools to be default: % setenv PGI /usr/pgi % setenv MANPATH "$MANPATH":$PGI/linux86/6.0/man % setenv LM_LICENSE_FILE $PGI/license.dat % set path = ($PGI/linux86-64/6.0/bin $path) PGI_CONTINUE - If the PGI_CONTINUE environment variable is set upon execution of a program compiled with –Mchkfpstk, the stack will be automatically cleaned up and execution will continue. There is a performance penalty associated with the stack cleanup. If PGI_CONTINUE is set to verbose, the stack will be automatically cleaned up and execution will continue after printing of a warning message. STATIC_RANDOM_SEED - The first call to the Fortran 90/95 RANDOM_SEED intrinsic without arguments will reset the random seed to a default value, then advance the seed by a variable amount based on time. Subsequent calls to RANDOM_SEED without arguments will reset the random seed to the same initial value as the first call. Unless the time is exactly the same, each time a program is run a different random number sequence will be generated. You can force the seed returned by RANDOM_SEED to be constant, thereby generating the same sequence of random numbers at each execution of the program, by setting the environment variable STATIC_RANDOM_SEED to yes. PGI_TERM - The stack traceback and just-in-time debugging functionality is controlled by the PGI_TERM environment variable. The run-time libraries use the value of PGI_TERM to determine what action to take when a progam abnormally terminates. See “Stack Traceback and JIT Debugging” on page 211 for more information.Environment Variables 211 PGI_TERM_DEBUG - The PGI_TERM_DEBUG variable may be set to override the default behavior when PGI_TERM is set to debug. TMPDIR - Can be used to specify the directory that should be used for placement of any temporary files created during execution of the PGI compilers and tools. TZ - Can be used to explicitly set the time zone, and is used in some contexts by the PGC++ compiler. For more information on the possible settings for TZ, use the tzselect utility on Linux for a detailed description of possible settings and step-by-step instructions for setting the value of TZ for a given time zone. Stack Traceback and JIT Debugging When a programming error results in a run-time error message or an application exception, a program will usually exit, perhaps with an error message. The PGI run-time library includes a mechanism to override this default action and instead print a stack traceback, start a debugger, or (on Linux) create a core file for post-mortem debugging. The stack traceback and just-in-time debugging functionality is controlled by an environment variable, PGI_TERM. The run-time libraries use the value of PGI_TERM to determine what action to take when a program abnormally terminates. When the PGI run-time library detects an error or catches a signal, it calls the routine pgi_stop_here prior to generating a stack traceback or starting the debugger. The pgi_stop_here routine is a convenient spot to set a breakpoint when debugging a program. The value of PGI_TERM is a comma-separated list of options. The format used to set the environment variable follows. in csh: % setenv PGI_TERM option[,option...] in bash or sh: $ PGI_TERM=option[,option...] $ export PGI_TERM in the Windows Command Prompt: C:\> set PGI_TERM=option[,option...]Libraries and Environment Variables 212 Supported values for option are listed in the following table. By default, all of these options are disabled. Table 8-1: Supported PGI_TERM Values A description of how to apply each of these options follows. [no]debug This enables/disables just-in-time debugging. The default is nodebug. When the debugger is invoked, the following default command is issued to start the debugger. pgdbg -text -attach pid The PGI_TERM_DEBUG environment variable may be set to override the default setting. The value of the environment variable should be set to the command line used to invoke the program. For example: gdb --quiet --pid %d The first occurrence of %d in PGI_TERM_DEBUG string will be replaced by the process id. The program named in PGI_TERM_DEBUG string must be found on the current $PATH or specified with a full path name. [no]trace This enables/disables the stack traceback. The default is notrace. [no]signal This enables/disables the establishing signal handlers for the most common signals that cause program termination. The default is nosignal; however, setting trace and debug will enable signal; override this behavior with nosignal. [no]debug Enables/disables just-in-time debugging (debugging invoked on error) [no]trace Enables/disables stack traceback on error [no]signal Enables/disables establishment of signal handlers for common signals that cause program termination [no]abort Enables/disables calling the system termination routine abort()Environment Variables 213 [no]abort This enables/disables calling the system termination routine abort(). The default is noabort. When noabort is in effect the process terminates by calling "_exit(127)". On Linux and SUA, the abort routine will create a core file and exit with code 127. On Windows, the abort routine exits with the status of the exception received; for example, if the program receives an access violation abort exits with status 0xC0000005. A few runtime errors just print an error message and call "exit(127)". These are mainly errors such as specifying an invalid environment variable value where a traceback would not be useful. If it appears that abort does not generate core files on a Linux system, be sure to unlimit the coredumpsize. For example, using csh: % limit coredumpsize unlimited % setenv PGI_TERM abort Using bash or sh: $ ulimit -c unlimited $ export PGI_TERM=abort To debug a core file with pgdbg, start pgdbg with the -core option. For example, to view a core file named “core” for a program named “a.out”: $ pgdbg -core core a.outLibraries and Environment Variables 214Fortran Data Types 215 9 Fortran, C and C++ Data Types This chapter describes the scalar and aggregate data types recognized by the PGI Fortran, C, and C++ compilers, the format and alignment of each type in memory, and the range of values each type can take on x86 or x64 processor-based systems running a 32-bit operating system. For more information on x86- specific data representation, refer to the System V Application Binary Interface, Processor Supplement, listed in the This chapter specifically does not address x64 processor-based systems running a 64-bit operating system, because the application binary interface (ABI) for those systems is still evolving. See http://www.x86-64.org/abi.pdf for the latest version of this ABI. Fortran Data Types Fortran Scalars A scalar data type holds a single value, such as the integer value 42 or the real value 112.6. The next table lists scalar data types, their size, format and range. Table 9-2 , “Real Data Type Ranges” shows the range and approximate precision for Fortran real data types. Table 9-3 , “Scalar Type Alignment” shows the alignment for different scalar data types. The alignments apply to all scalars, whether they are independent or contained in an array, a structure or a union.Fortran, C and C++ Data Types 216 Table 9-1: Representation of Fortran Data Types Fortran Data Type Format Range INTEGER 2's complement integer -231 to 231-1 INTEGER*2 2's complement integer -32768 to 32767 INTEGER*4 same as INTEGER INTEGER*8 same as INTEGER -263 to 263-1 LOGICAL same as INTEGER true or false LOGICAL*1 8 bit value true or false LOGICAL*2 16 bit value true or false LOGICAL*4 same as INTEGER true or false LOGICAL*8 same as INTEGER true or false BYTE 2's complement -128 to 127 REAL Single-precision floating point 10-37 to 1038 (1) REAL*4 Single-precision floating point 10-37 to 1038 (1) REAL*8 Double-precision floating point 10-307 to 10308 (1) DOUBLE PRECISION Double-precision floating point 10-307 to 10308 (1) COMPLEX See REAL See REALFortran Data Types 217 (1) Approximate value The logical constants .TRUE. and .FALSE. are all ones and all zeroes, respectively. Internally, the value of a logical variable is true if the least significant bit is one and false otherwise. When the option – Munixlogical is set, a logical variable with a non-zero value is true and with a zero value is false. DOUBLE COMPLEX See DOUBLE PRECISION See DOUBLE PRECISION COMPLEX*16 Same as above Same as above CHARACTER*n Sequence of n bytes Fortran Data Type Format RangeFortran, C and C++ Data Types 218 Table 9-2: Real Data Type Ranges Table 9-3: Scalar Type Alignment Data Type Binary Range Decimal Range Digits of Precision REAL 2-126 to 2128 10-37 to 1038 7-8 REAL*8 2-1022 to 21024 10-307 to 10308 15-16 Type Is Aligned on a LOGICAL*1 1-byte boundary LOGICAL*2 2-byte boundary LOGICAL*4 4-byte boundary LOGICAL*8 8-byte boundary BYTE 1-byte boundary INTEGER*2 2-byte boundary INTEGER*4 4-byte boundary INTEGER*8 8-byte boundary REAL*4 4-byte boundary REAL*8 8-byte boundary COMPLEX*8 4-byte boundaryFortran Data Types 219 FORTRAN 77 Aggregate Data Type Extensions The PGF77 compiler supports de facto standard extensions to FORTRAN 77 that allow for aggregate data types. An aggregate data type consists of one or more scalar data type objects. You can declare the following aggregate data types: array consists of one or more elements of a single data type placed in contiguous locations from first to last. structure is a structure that can contain different data types. The members are allocated in the order they appear in the definition but may not occupy contiguous locations. union is a single location that can contain any of a specified set of scalar or aggregate data types. A union can have only one value at a time. The data type of the union member to which data is assigned determines the data type of the union after that assignment. The alignment of an array, a structure or union (an aggregate) affects how much space the object occupies and how efficiently the processor can address members. Arrays use the alignment of their members. Array types align according to the alignment of the array elements. For example, an array of INTEGER*2 data aligns on a 2 byte boundary. Structures and Unions align according to the alignment of the most restricted data type of the structure or union. In the next example, the union aligns on a 4-byte boundary since the alignment of c, the most restrictive element, is four. STRUCTURE /astr/ UNION MAP INTEGER*2 a ! 2 bytes END MAP MAP BYTE b ! 1 byte END MAP COMPLEX*16 8-byte boundary Type Is Aligned on a Fortran, C and C++ Data Types 220 MAP INTEGER*4 c ! 4 bytes END MAP END UNION END STRUCTURE Structure alignment can result in unused space called padding. Padding between members of the structure is called internal padding. Padding between the last member and the end of the space is called tail padding. The offset of a structure member from the beginning of the structure is a multiple of the member’s alignment. For example, since an INTEGER*2 aligns on a 2-byte boundary, the offset of an INTEGER*2 member from the beginning of a structure is a multiple of two bytes. Fortran 90 Aggregate Data Types (Derived Types) The Fortran 90 standard added formal support for aggregate data types. The TYPE statement begins a derived type data specification or declares variables of a specified user-defined type. For example, the following would define a derived type ATTENDEE: TYPE ATTENDEE CHARACTER(LEN=30) NAME CHARACTER(LEN=30) ORGANIZATION CHARACTER (LEN=30) EMAIL END TYPE ATTENDEE In order to declare a variable of type ATTENDEE and access the contents of such a variable, code such as the following would be used: TYPE (ATTENDEE) ATTLIST(100) . . . ATTLIST(1)%NAME = ‘JOHN DOE’ C and C++ Data Types C and C++ Scalars The next table lists C and C++ scalar data types, their size and format. The alignment of a scalar data type is equal to its size. Table 9-5 , “Scalar Alignment” shows scalar alignments that apply to individual scalars and to scalars that are elements of an array or members of a structure or union. Wide characters are supported (character constants prefixed with an L). The size of each wide character is 4 bytes.C and C++ Data Types 221 Table 9-4: C/C++ Scalar Data Types Data Type Size (bytes) Format Range unsigned char 1 ordinal 0 to 255 [signed] char 1 two's-complement integer -128 to 127 unsigned short 2 ordinal 0 to 65535 [signed] short 2 two's-complement integer -32768 to 32767 unsigned int 4 ordinal 0 to 2 32 -1 [signed] int 4 two's-complement integer -2 31 to 2 31 -1 [signed] long [int] (32-bit operating systems and win64) 4 two's-complement integer -2 31 to 2 31 -1 [signed] long [int] (linux86-64 and sua64) 8 two's-complement integer -2 63 to 2 63 -1 unsigned long [int] (32- bit operating systems and win64) 4 ordinal 0 to 2 32 -1 unsigned long [int] (linux86-64 and sua64) 8 ordinal 0 to 2 64 -1 [signed] long long [int] 8 two's-complement integer -2 63 to 2 63 -1 unsigned long long [int] 8 ordinal 0 to 2 64 -1 float 4 IEEE single-precision floating-point 10 -37 to 10 38 (1)Fortran, C and C++ Data Types 222 (1) Approximate value (2) Bit fields occupy as many bits as you assign them, up to 4 bytes, and their length need not be a multiple of 8 bits (1 byte) double 8 IEEE double-precision floating-point 10 -307 to 10 308 (1) long double 8 IEEE double-precision floating-point 10 -307 to 10 308 (1) bit field(2) (unsigned value) 1 to 32 bits ordinal 0 to 2 size -1, where size is the number of bits in the bit field bit field(2) (signed value) 1 to 32 bits two's complement integer -2 size-1 to 2 size-1 -1, where size is the number of bits in the bit field pointer 4 address 0 to 2 32 -1 enum 4 two's complement integer -2 31 to 2 31 -1 Data Type Size (bytes) Format RangeC and C++ Data Types 223 Table 9-5: Scalar Alignment (*)signed or unsigned C and C++ Aggregate Data Types An aggregate data type consists of one or more scalar data type objects. You can declare the following aggregate data types: array consists of one or more elements of a single data type placed in contiguous locations from first to last. class (C++ only) is a class that defines an object and its member functions. The object can contain fundamental data types or other aggregates including other classes. The class members are allocated in the order they appear in the definition but may not occupy contiguous locations. struct is a structure that can contain different data types. The members are allocated in the order they appear in the definition but may not occupy contiguous locations. When a struct is defined with member functions, its alignment issues are the same as those for a class. union is a single location that can contain any of a specified set of scalar or aggregate data types. A union can have only one value at a time. The data type of the union member to which data is assigned determines the data type of the union after that assignment. Data Type Alignment char is aligned on a 1-byte boundary.* short is aligned on a 2-byte boundary.* [long] int is aligned on a 4-byte boundary.* enum is aligned on a 4-byte boundary. pointer is aligned on a 4-byte boundary. float is aligned on a 4-byte boundary. double is aligned on an 8-byte boundary. long double is aligned on an 8-byte boundary.Fortran, C and C++ Data Types 224 Class and Object Data Layout Class and structure objects with no virtual entities and with no base classes, that is just direct data field members, are laid out in the same manner as C structures. The following section describes the alignment and size of these C-like structures. C++ classes (and structures as a special case of a class) are more difficult to describe. Their alignment and size is determined by compiler generated fields in addition to user-specified fields. The following paragraphs describe how storage is laid out for more general classes. The user is warned that the alignment and size of a class (or structure) is dependent on the existence and placement of direct and virtual base classes and of virtual function information. The information that follows is for informational purposes only, reflects the current implementation, and is subject to change. Do not make assumptions about the layout of complex classes or structures. All classes are laid out in the same general way, using the following pattern (in the sequence indicated): • First, storage for all of the direct base classes (which implicitly includes storage for non-virtual indirect base classes as well): • When the direct base class is also virtual, only enough space is set aside for a pointer to the actual storage, which appears later. • In the case of a non-virtual direct base class, enough storage is set aside for its own nonvirtual base classes, its virtual base class pointers, its own fields, and its virtual function information, but no space is allocated for its virtual base classes. • Next, storage for the class’s own fields. • Next, storage for virtual function information (typically, a pointer to a virtual function table). • Finally, storage for its virtual base classes, with space enough in each case for its own non-virtual base classes, virtual base class pointers, fields, and virtual function information. Aggregate Alignment The alignment of an array, a structure or union (an aggregate) affects how much space the object occupies and how efficiently the processor can address members. Arrays use the alignment of their members. Arrays align according to the alignment of the array elements. For example, an array of short data type aligns on a 2-byte boundary.C and C++ Data Types 225 Structures and Unions align according to the most restrictive alignment of the enclosing members. For example the union un1 below aligns on a 4-byte boundary since the alignment of c, the most restrictive element, is four: union un1 { short a; /* 2 bytes */ char b; /* 1 byte */ int c; /* 4 bytes */ }; Structure alignment can result in unused space, called padding. Padding between members of a structure is called internal padding. Padding between the last member and the end of the space occupied by the structure is called tail padding. *** 'Internal Padding in a Structure' on page 225 ***, illustrates structure alignment. Consider the following structure: struct strc1 { char a; /* occupies byte 0 */ short b; /* occupies bytes 2 and 3 */ char c; /* occupies byte 4 */ int d; /* occupies bytes 8 through 11 */ }; Figure 9-1: Internal Padding in a Structure *** 'Tail Padding in a Structure' on page 226 ***, shows how tail padding is applied to a structure aligned on a doubleword boundary. struct strc2{ int m1[4]; /* occupies bytes 0 through 15 */ double m2; /* occupies bytes 16 through 23 */ short m3; /* occupies bytes 24 and 25 */ } st;Fortran, C and C++ Data Types 226 Bit-field Alignment Bit-fields have the same size and alignment rules as other aggregates, with several additions to these rules: • Bit-fields are allocated from right to left. • A bit-field must entirely reside in a storage unit appropriate for its type. Bit-fields never cross unit boundaries. • Bit-fields may share a storage unit with other structure/union members, including members that are not bit-fields. • Unnamed bit-field's types do not affect the alignment of a structure or union. • Items of [signed/unsigned] long long type may not appear in field declarations. Figure 9-2: Tail Padding in a Structure Other Type Keywords in C and C++ The void data type is neither a scalar nor an aggregate. You can use void or void* as the return type of a function to indicate the function does not return a value, or as a pointer to an unspecified data type, respectively.C and C++ Data Types 227 The const and volatile type qualifiers do not in themselves define data types, but associate attributes with other types. Use const to specify that an identifier is a constant and is not to be changed. Use volatile to prevent optimization problems with data that can be changed from outside the program, such as memory-mapped I/O buffers.Fortran, C and C++ Data Types 228Data Types in the 64-Bit Environment 229 10 Programming Considerations for 64-Bit Environments PGI provides 64-bit compilers for the 64-bit Linux, Windows, SUA, and Apple operating systems running on the AMD64 architecture. These compilers can be used to create programs that use 64-bit memory addresses. However, there are limitations in how this capability can be applied. With the exception of Linux86-64, the object file formats on all of the operating systems limit the total cumulative size of code plus static data to 2GB. This includes the code and statically declared data in the program and in system and user object libraries. Linux86-64 implements a mechanism that overcomes this limitation (see “Large Static Data in Linux”). This chapter describes the specifics of how to use the PGI compilers to make use of 64-bit memory addressing. The 64-bit Windows, Linux, SUA, and Apple environments maintain 32-bit compatibility, which means that 32-bit applications can be developed and executed on the corresponding 64-bit operating system. Note The 64-bit PGI compilers are 64-bit applications themselves, and cannot run on anything but 64-bit CPUs running 64-bit Operating Systems. Data Types in the 64-Bit Environment The size of some data types can be different in a 64-bit environment. This section describes the major differences. Refer to the chapter “Fortran, C, and C++ Data Types” for detailed information. C/C++ Data Types On 32-bit Windows, int is 4 bytes, long is 4 bytes, and pointers are 4 bytes. On 64-bit windows, the size of an int is 4 bytes, a long is 4 bytes, and a pointer is 8 bytes. On the 32-bit Linux, SUA, and Apple operating systems, the size of an int is 4 bytes, a long is 4 bytes, and a pointer is 4 bytes. On the 64-bit Linux, SUA, and Apple operating systems, the size of an int is 4 bytes, a long is 8 bytes, and a pointer is 8 bytes.Programming Considerations for 64-Bit Environments 230 Fortran Data Types In Fortran, the default size of the INTEGER type is 4 bytes. The -i8 compiler option may be used to make the default size of all INTEGER data in the program 8 bytes. When using the -Mlarge_arrays option (see “64-Bit Array Indexing”), any 4-byte INTEGER variables that are used to index arrays are silently promoted by the compiler to 8 bytes. This can lead to unexpected consequences, so 8 byte INTEGER variables are recommended for array indexing when using -Mlarge_arrays. Large Static Data in Linux Linux86-64 operating systems support two different memory models. The default model used by PGI compilers is the small memory model, which can be specified using -mcmodel=small. This is the 32-bit model, which limits the size of code plus statically allocated data, including system and user libraries, to 2GB. The medium memory model, specified by -mcmodel=medium, allows combined code and static data areas (.text and .bss sections) larger than 2GB. The -mcmodel=medium option must be used on both the compile command and the link command in order to take effect. The Win64, SUA64, and 64-bit Apple operating systems do not have any support for large static data declarations. There are two drawbacks to using –mcmodel=medium. First, there is increased addressing overhead to support the large data range. This can affect performance, though the compilers seek to minimize the added overhead through careful instruction generation. Second, -mcmodel=medium cannot be used for objects in shared libraries, because there is no OS support for 64-bit dynamic linkage. Large Dynamically Allocated Data Dynamically allocated data objects in programs compiled by the 64-bit PGI compilers can be larger than 2GB. No special compiler options are required to enable this functionality. The size of the allocation is only limited by the system. However, to correctly access dynamically allocated arrays with more than 2G elements you should use the -Mlarge_arrays option (see “64-Bit Array Indexing“).64-Bit Array Indexing 231 64-Bit Array Indexing The 64-bit PGI compilers provide an option, -Mlarge_arrays, that enables 64-bit indexing of arrays This means that, as necessary, 64-bit INTEGER constants and variables are used to index arrays. Note that in the presence of -Mlarge_arrays, the compiler may silently promote 32-bit integers to 64 bits, which can have unexpected side effects. On Linux86-64, the -Mlarge_arrays option also enables single static data objects larger than 2 GB. This option is the default in the presence of -mcmodel=medium. Compiler Options for 64-bit Programming The usual switches that apply to 64-bit programmers seeking to increase the data range of their applications are in the table below.Programming Considerations for 64-Bit Environments 232 Table 10-1: 64-bit Compiler Options The following table summarizes the limits of these programming models: Option Purpose Considerations -mcmodel=medium Enlarge object size; Allow for declared data the size of larger than 2GB Linux86-64 only. Slower execution. Cannot be used with –fPIC. Objects cannot be put into shared libraries -Mlarge_arrays Perform all array-location-toaddress calculations using 64-bit integer arithmetic. Slightly slower execution. Implicit with –mcmodel=medium. Can be used with Win64 and – mcmodel=small. -fPIC Position independent code. Necessary for shared libraries. Dynamic linking restricted to a 32- bit offset. External symbol references should refer to other shared lib routines, rather than the program calling them. -i8 All INTEGER functions, data, and constants not explicitly declared INTEGER*4 are assumed to be INTEGER*8. Users should take care to explicitly declare INTEGER functions as INTEGER*4.Compiler Options for 64-bit Programming 233 Table 10-2: Effects of Options on Memory and Array Sizes Combined Compiler Options Addr. Math Max Size Gbytes A I AS DS TS Comments –tp k8-32 or –tp p7 32 32 2 2 2 32-bit linux86 programs –tp k8-64 –tp p7-64 64 32 2 2 2 64-bit addrlimited by – mcmodel=smal l –tp k8-64 –fpic or –tp p7-64 –fpic 64 32 2 2 2 –fpic incompatible with – mcmodel=medi um –tp k8-64 or –tp p7-64 –mcmodel=medium 64 64 >2 >2 >2 Enable full support for 64-bit data addressing Column Legend A Address Type (A) -size in bits of data used for address calculations, 32-bit or 64-bit. I Index Arithmetic (I)- bit-size of data used to index into arrays and other aggregate data structures. If 32-bit, total range of any single data object is limited to 2GB. AS Maximum Array Size (AS)- the maximum size in gigabytes of any single data object. DS Maximum Data Size (DS)- max size in gigabytes combined of all data objects in .bss TS Maximum Total Size (TS) max size in bytes, in aggregate, of all executable code and data objects in a running program.Programming Considerations for 64-Bit Environments 234 Practical Limitations of Large Array Programming The 64-bit addressing capability of the linux86-64 and Win64 environments can cause unexpected issues when data sizes are enlarged significantly. For example: Table 10-3: 64-Bit Limitations Example: Medium Memory Model and Large Array in C Consider the following example, where the aggregate size of the arrays exceeds 2GB. % cat bigadd.c#include #define SIZE 600000000 /* > 2GB/4 */ static float a[SIZE],b[SIZE]; void main() { long long i,n,m; array initialization Initializing a large array with a data statement may result in very large assembly and object files, where a line of assembler source is required for each element in the initalized array. Compilation and linking can be very time consuming as well. To avoid this issue, consider initializing large arrays in a loop at runtime rather than in a data statement. stack space Stack space can be a problem for data that is stack-based. In Win64, stack space is increased in the link stage to the new_size by adding the link switch –Wl,-stack:new_sizeIn linux86-64, stack size is increased in the environment (note: setting stacksize to unlimited often is not large enough)limit stacksize new_size ! in cshulimit –s new_size ! in bash page swapping If your executable is much larger than the physical size of memory, page swapping can cause it to run dramatically slower and it may even fail. This is not a compiler problem. Try smaller data sets to determine if a problem is due to page thrashing, or not. configured space Be sure your linux86-64 system is configured with swap space sufficiently large to support the data sets used in your application(s). If your memory+swap space is not sufficiently large, your application will likely encounter a segmentation fault at runtime.Example: Medium Memory Model and Large Array in C 235 float c[SIZE]; /* goes on stack */ n=SIZE;m=0; for(i=0;i 2GB in size. Note that if you execute with the above settings in your environment, you may see the following: % bigadd Segmentation fault Execution fails because the stack size is not large enough. Try resetting the stack size in your environment: % limit stacksize 3000M Note that ‘limit stacksize unlimited’ will probably not provide as large a stack as we are using above. % bigadd a[0]=1 b[0]=2 c[0]=3 n=599990000 a[599990000]=5.9999e+08 b[599990000]=1.19998e+09 c[599990000]=1.79997e+09 The size of the bss section of the bigadd executable is now larger than 2GB:Programming Considerations for 64-Bit Environments 236 % size –-format=sysv bigadd | grep bss .bss 4800000008 5245696 % size -–format=sysv bigadd | grep Total Total 4800005080 Example: Medium Memory Model and Large Array in Fortran The following example works with both the PGF95 and PGF77 compilers included in Release 7.0. Both compilers use 64-bit addresses and index arithmetic when the –mcmodel=medium option is used. Consider the following example: % cat mat.f program mat integer i, j, k, size, l, m, n parameter (size=16000) ! >2GB parameter (m=size,n=size) real*8 a(m,n),b(m,n),c(m,n),d do i = 1, m do j = 1, n a(i,j)=10000.0D0*dble(i)+dble(j) b(i,j)=20000.0D0*dble(i)+dble(j) enddo enddo !$omp parallel !$omp do do i = 1, m do j = 1, n c(i,j) = a(i,j) + b(i,j) enddo enddo !$omp do do i=1,m do j = 1, n d = 30000.0D0*dble(i)+dble(j)+dble(j) if(d .ne. c(i,j)) then print *,”err i=”,i,”j=”,j print *,”c(i,j)=”,c(i,j) print *,”d=”,d stop endifExample: Large Array and Small Memory Model in Fortran 237 enddo enddo !$omp end parallel print *, “M =”,M,”, N =”,N print *, “c(M,N) = “, c(m,n) end When compiled with the PGF95 compiler using –mcmodel=medium: % pgf95 –mp –o mat mat.f –i8 –mcmodel=medium % setenv OMP_NUM_THREADS 2 % mat M = 16000 , N = 16000 c(M,N) = 480032000.0000000 Example: Large Array and Small Memory Model in Fortran The next example uses large arrays, but not statically declared. It is broken into a main and subroutine so you can put the subroutine into a shared library. Dynamic allocation of large arrays saves space in the size of executable, saves time initializing data, and the routines can also be compiled with 32-bit compilers, by just decreasing the parameter size below. This example works on Win64 as well. % cat mat_allo.f program mat_allo integer i, j integer size, m, n parameter (size=16000) parameter (m=size,n=size) double precision,allocatable::a(:,:),b(:,:),c(:,:) allocate(a(m,n),b(m,n), c(m,n)) do i = 100, m, 1 do j = 100, n, 1 a(i,j) = 10000.0D0 * dble(i) + dble(j) b(i,j) = 20000.0D0 * dble(i) + dble(j) enddo enddo call mat_add(a,b,c,m,n) print *, “M =”,m,”, N =”,n print *, “c(M,N) = “, c(m,n) end subroutine mat_add(a,b,c,m,n) integer m, n, i, j double precision a(m,n),b(m,n),c(m,n) !$omp do do i = 1, m do j = 1, n c(i,j) = a(i,j) + b(i,j) enddo enddo return end% pgf95 –o mat_allo mat_allo.f –i8 –Mlarge_arrays -mp -fastProgramming Considerations for 64-Bit Environments 238Overview of Calling Conventions 239 11 Inter-language Calling This chapter describes inter-language calling conventions for C, C++, and Fortran programs using the PGI compilers. The following sections describe how to call a Fortran function or subroutine from a C or C++ program and how to call a C or C++ function from a Fortran program. For information on calling assembly language programs, refer to Appendix A, . Overview of Calling Conventions This chapter includes information on the following topics: • Functions and subroutines in Fortran, C, and C++ • Naming and case conversion conventions • Compatible data types • Argument passing and special return values • Arrays and Indexes • Win32 calling conventions Default Fortran calling conventions under Win32 differ from those used under Linux, Win64, and SUA operating systems. Win32 programs compiled using the -Munix Fortran command-line option use the Linux/Win64 convention rather than the default Win32 convention. The sections Inter-language Calling Considerations through Example - C++ Calling Fortran describe how to perform inter-language calling using the Linux/Win64 convention. All information in those sections pertaining to compatibility of arguments applies to Win32 as well. See “Win32 Calling Conventions” on page 253 for details on the symbol name and argument passing conventions used on Win32 platforms. Inter-language Calling Considerations In general, when argument data types and function return values agree you can call a C or C++ function from Fortran and likewise, you can call a Fortran function from C or C++. You may need to develop special procedures in cases where data types for arguments do not agree. For example, the Inter-language Calling 240 Fortran COMPLEX type does not have a matching type in C, it is still possible to provide inter-language calls but there are no general calling conventions for such cases. In this instance, you need to develop a special procedure. Follow these guidelines: • Note that if a C++ function contains objects with constructors and destructors, calling such a function from either C or Fortran will not be possible unless the initialization in the main program is performed from a C++ program where constructors and destructors are properly initialized. • In general, you can call a C function from C++ without problems as long as you use the extern "C" keyword to declare the C function in the C++ program. This prevents name mangling for the C function name. If you want to call a C++ function from C, likewise you have to use the extern "C" keyword to declare the C++ function. This keeps the C++ compiler from mangling the name of the function. • You can use the __cplusplus macro to allow a program or header file to work for both C and C++. For example, the following defines in the header file stdio.h allow this file to work for both C and C++. #ifndef _STDIO_H #define _STDIO_H #ifdef __cplusplus extern "C" { #endif /* __cplusplus */ . . /* Functions and data types defined... */ . #ifdef __cplusplus } #endif /* __cplusplus */ #endif • C++ member functions cannot be declared extern, since their names will always be mangled. Therefore, C++ member functions cannot be called from C or Fortran. Functions and Subroutines Fortran, C, and C++ define functions and subroutines differently. For a Fortran program calling a C or C++ function, observe the following return value convention: Upper and Lower Case Conventions, Underscores 241 • When the C or C++ function returns a value, call it from Fortran as a function, and otherwise call it as a subroutine. For a C/C++ program calling a Fortran function, the call should return a similar type. Table 11-1 , “Fortran and C/C++ Data Type Compatibility” lists compatible types. If the call is to a Fortran subroutine, a Fortran CHARACTER function, or a Fortran COMPLEX function, call it from C/C++ as a function that returns void. The exception to this convention is when a Fortran subroutine has alternate returns; call such a subroutine from C/C++ as a function returning int whose value is the value of the integer expression specified in the alternate RETURN statement. Upper and Lower Case Conventions, Underscores By default on Linux, Win64, and SUA systems, all Fortran symbol names are converted to lower case. C and C++ are case sensitive, so upper-case function names stay upper-case. When you use inter-language calling, you can either name your C/C++ functions with lower-case names, or invoke the Fortran compiler command with the option –Mupcase, in which case it will not convert symbol names to lowercase. When programs are compiled using one of the PGI Fortran compilers on Linux, Win64, and SUA systems, an underscore is appended to Fortran global names (names of functions, subroutines and common blocks). This mechanism distinguishes Fortran name space from C/C++ name space. Use these naming conventions: • If you call a C/C++ function from Fortran, you should rename the C/C++ function by appending an underscore (or use C$PRAGMA C in the Fortran program, refer to 7, “Directives and Pragmas”, for details on C$PRAGMA C). • If you call a Fortran function from C/C++, you should append an underscore to the Fortran function name in the calling program. Compatible Data Types The next table shows compatible data types between Fortran and C/C++. Table 11-2 , “Fortran and C/ C++ Representation of the COMPLEX Type” shows how the Fortran COMPLEX type may be represented in C/C++. If you can make your function/subroutine parameters and return values match types, you should be able to use inter-language calling.Inter-language Calling 242 Table 11-1: Fortran and C/C++ Data Type Compatibility Table 11-2: Fortran and C/C++ Representation of the COMPLEX Type Fortran Type (lower case) C/C++ Type Size (bytes) character x char x 1 character*n x char x[n] n real x float x 4 real*4 x float x 4 real*8 x double x 8 double precision double x 8 integer x int x 4 integer*1 x signed char x 1 integer*2 x short x 2 integer*4 x int x 4 integer*8 x long long x 8 logical x int x 4 logical*1 x char x 1 logical*2 x short x 2 logical*4 int x 4 logical*8 long long x 8 Fortran Type (lower case) C/C++ Type Size (bytes) complex x struct {float r,i;} x; 8Compatible Data Types 243 Fortran Named Common Blocks A named Fortran common block can be represented in C/C++ by a structure whose members correspond to the members of the common block. The name of the structure in C/C++ must have the added underscore. For example, the Fortran common block: INTEGER I COMPLEX C DOUBLE COMPLEX CD DOUBLE PRECISION D COMMON /COM/ i, c, cd, d is represented in C with the following equivalent: extern struct { int i; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; and in C++ with the following equivalent: extern "C" struct { int i; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; complex*8 x struct {float r,i;} x; 8 double complex x struct {double dr,di;} x; 16 Fortran Type (lower case) C/C++ Type Size (bytes)Inter-language Calling 244 Argument Passing and Return Values In Fortran, arguments are passed by reference (i.e. the address of the argument is passed, rather than the argument itself). In C/C++, arguments are passed by value, except for strings and arrays, which are passed by reference. Due to the flexibility provided in C/C++, you can work around these differences. Solving the parameter passing differences generally involves intelligent use of the & and * operators in argument passing when C/C++ calls Fortran and in argument declarations when Fortran calls C/C++. For strings declared in Fortran as type CHARACTER, an argument representing the length of the string is passed to a calling function. On Linux systems, or when using the UNIX calling convention on Windows (-Munix), the compiler places the length argument(s) at the end of the parameter list, following the other formal arguments. The length argument is passed by value, not by reference. Passing by Value (%VAL) When passing parameters from a Fortran subprogram to a C/C++ function, it is possible to pass by value using the %VAL function. If you enclose a Fortran parameter with %VAL(), the parameter is passed by value. For example, the following call passes the integer i and the logical bvar by value. integer*1i logical*1bvar call cvalue (%VAL(i), %VAL(bvar)) Character Return Values “Functions and Subroutines” on page 240 describes the general rules for return values for C/C++ and Fortran inter-language calling. There is a special return value to consider. When a Fortran function returns a character, two arguments need to be added at the beginning of the C/C++ calling function’s argument list: • the address of the return character or characters • the length of the return character Example 11-1, “Character Return Parameters” illustrates the extra parameters, tmp and 10, supplied by the caller:Argument Passing and Return Values 245 Example 11-1: Character Return Parameters CHARACTER*(*) FUNCTION CHF( C1, I) CHARACTER*(*) C1 INTEGER I END extern void chf_(); char tmp[10]; char c1[9]; int i; chf_(tmp, 10, c1, &i, 9); If the Fortran function is declared to return a character value of constant length, for example CHARACTER*4 FUNCTION CHF(), the second extra parameter representing the length must still be supplied, but is not used. NOTE The value of the character function is not automatically NULL-terminated. Complex Return Values When a Fortran function returns a complex value, an argument needs to be added at the beginning of the C/C++ calling function’s argument list; this argument is the address of the complex return value. Example 11-2, “COMPLEX Return Values” illustrates the extra parameter, cplx, supplied by the caller: Example 11-2: COMPLEX Return Values COMPLEX FUNCTION CF(C, I) INTEGER I . . . END extern void cf_(); typedef struct {float real, imag;} cplx; cplx c1; int i; cf_(&c1, &i);Inter-language Calling 246 Array Indices C/C++ arrays and Fortran arrays use different default initial array index values. By default, C/C++ arrays start at 0 and Fortran arrays start at 1. If you adjust your array comparisons so that a Fortran second element is compared to a C/C++ first element, and adjust similarly for other elements, you should not have problems working with this difference. If this is not satisfactory, you can declare your Fortran arrays to start at zero. Another difference between Fortran and C/C++ arrays is the storage method used. Fortran uses column-major order and C/C++ use row-major order. For one-dimensional arrays, this poses no problems. For two-dimensional arrays, where there are an equal number of rows and columns, row and column indexes can simply be reversed. For arrays other than single dimensional arrays, and square two-dimensional arrays, inter-language function mixing is not recommended. Example - Fortran Calling C Example 11-4 , “C function cfunc_” shows a C function that is called by the Fortran main program shown in Example 11-3 , “Fortran Main Program fmain.f”. Notice that each argument is defined as a pointer, since Fortran passes by reference. Also notice that the C function name uses all lower-case and a trailing "_". Example 11-3: Fortran Main Program fmain.f logical*1 bool1 character letter1 integer*4 numint1, numint2 real numfloat1 double precision numdoub1 integer*2 numshor1 external cfunc call cfunc (bool1, letter1, numint1, numint2, & numfloat1, numdoub1, numshor1) write( *, "(L2, A2, I5, I5, F6.1, F6.1, I5)") & bool1, letter1, numint1, numint2, numfloat1, & numdoub1, numshor1 endExample - C Calling Fortran 247 Example 11-4: C function cfunc_ #define TRUE 0xff #define FALSE 0 void cfunc_( bool1, letter1, numint1, numint2, numfloat1,\ numdoub1, numshor1, len_letter1) char *bool1, *letter1; int *numint1, *numint2; float *numfloat1; double *numdoub1; short *numshor1; int len_letter1; { *bool1 = TRUE; *letter1 = 'v'; *numint1 = 11; *numint2 = -44; *numfloat1 = 39.6 ; *numdoub1 = 39.2; *numshor1 = 981; } Compile and execute the program fmain.f with the call to cfunc_ using the following command lines: $ pgcc -c cfunc.c $ pgf95 cfunc.o fmain.f Executing the a.out file should produce the following output: T v 11 -44 39.6 39.2 981 Example - C Calling Fortran Example 11-6 , “C Main Program cmain.c” shows a C main program that calls the Fortran subroutine shown in Example 11-5 , “Fortran Subroutine forts.f”. Notice that each call uses the & operator to pass by reference. Also notice that the call to the Fortran subroutine uses all lower-case and a trailing "_". Example 11-5: Fortran Subroutine forts.f subroutine forts ( bool1, letter1, numint1, & numint2, numfloat1, numdoub1, numshor1) logical*1 bool1 character letter1 integer numint1, numint2 double precision numdoub1 real numfloat1 integer*2 numshor1Inter-language Calling 248 bool1 = .true. letter1 = "v" numint1 = 11 numint2 = -44 numdoub1 = 902 numfloat1 = 39.6 numshor1 = 299 return end Example 11-6: C Main Program cmain.c main () { char bool1, letter1; int numint1, numint2; float numfloat1; double numdoub1; short numshor1; extern void forts_ (); forts_(&bool1,&letter1,&numint1,&numint2,&numfloat1, &numdoub1,&numshor1, 1); printf(" %s %c %d %d %3.1f %.0f %d\n", bool1?"TRUE":"FALSE",letter1,numint1, numint2, numfloat1, numdoub1, numshor1); } To compile this Fortran subroutine and C program, use the following commands: $ pgcc -c cmain.f $ pgf95 -Mnomain cmain.o forts.f Executing the resulting a.out file should produce the following output: TRUE v 11 -44 39.6 902 299 Example - C ++ Calling C Example 11-7: Simple C Function cfunc.c void cfunc(num1, num2, res) int num1, num2, *res; { printf("func: a = %d b = %dExample - C Calling C++ 249 ptr c = %x\n",num1,num2,res); *res=num1/num2; printf("func: res = %d\n",*res); } Example 11-8: C++ Main Program cpmain.C Calling a C Function xtern "C" void cfunc(int n, int m, int *p); #include main() { int a,b,c; a=8; b=2; cout << "main: a = "< extern "C" { extern void forts_(char *,char *,int *,int *, float *,double *,short *); } main () { char bool1, letter1; int numint1, numint2; float numfloat1; double numdoub1; short numshor1; forts_(&bool1,&letter1,&numint1,&numint2,&numfloat1, &numdoub1,&numshor1); cout << " bool1 = "; bool1?cout << "TRUE ":cout << "FALSE "; cout << endl; cout << " letter1 = " << letter1 << endl; cout << " numint1 = " << numint1 << endl; cout << " numint2 = " << numint2 << endl; cout << " numfloat1 = " << numfloat1 << endl; cout << " numdoub1 = " << numdoub1 < int main(){ __m128 __A, __B, result; __A = _mm_set_ps(23.3, 43.7, 234.234, 98.746); __B = _mm_set_ps(15.4, 34.3, 4.1, 8.6); result = _mm_add_ps(__A,__B); return 0; } Instructions Header File MMX mmintrin.h SSE xmmintrin.h SSE2 emmintrin.h SSE3 pmmintrin.h SSSE3 tmmintrin.h SSE4a ammintrin.h ABM intrin.h283 13 C++ Name Mangling Name mangling transforms the names of entities so that the names include information on aspects of the entity’s type and fully qualified name. This is necessary since the intermediate language into which a program is translated contains fewer and simpler name spaces than there are in the C++ language. Specifically: • Overloaded function names are not allowed in the intermediate language. • Classes have their own scopes in C++, but not in the generated intermediate language. For example, an entity x from inside a class must not conflict with an entity x from the file scope. • External names in the object code form a completely flat name space. The names of entities with external linkage must be projected onto that name space so that they do not conflict with one another. A function f from a class A, for example, must not have the same external name as a function f from class B. • Some names are not names in the conventional sense of the word, they're not strings of alphanumeric characters, for example operator=. We can see that there are two problems here: 1. Generating external names that will not clash. 2. Generating alphanumeric names for entities with strange names in C++. Name mangling solves these problems by generating external names that will not clash, and alphanumeric names for entities with strange names in C++. It also solves the problem of generating hidden names for some behind-the-scenes language support in such a way that they will match up across separate compilations. You will see mangled names if you view files that are translated by PGC++, and you do not use tools that demangle the C++ names. Intermediate files that use mangled names include the assembly and object files created by the pgcpp command and the C-like file that can be viewed as output from pgcpp using the +i command-line option The name mangling algorithm for the PGC++ compiler is the same as that for cfront, and also matches the description in Section 7.2, Function Name Encoding, of The Annotated C++ Reference Manual (except for some minor details). Refer to the ARM for a complete description of name mangling.C++ Name Mangling 284 Types of Mangling The following entity names are mangled: • Function names including non-member function names are mangled, to deal with overloading. Names of functions with extern "C" linkage are not mangled. • Mangled function names have the function name followed by __ followed by F followed by the mangled description of the types of the parameters of the function. If the function is a member function, the mangled form of the class name precedes the F. If the member function is static, an S also precedes the F. int f(float); // f__Ff class A int f(float); // f__1AFf static int g(float); // g__1ASFf ; • Special and operator function names, like constructors and operator=(). The encoding is similar to that for normal functions, but a coded name is used instead of the routine name: class A int operator+(float); // __pl__1Aff A(float); // __ct__1Aff ; int operator+(A, float); // __pl__F1Af • Static data member names. The mangled form is the member name followed by __ followed by the mangled form of the class name: class A static int i; // i__1A ; • Names of variables generated for virtual function tables. These have names like vtblmangledclass-name or vtblmangled-base-class-namemangled-class-name. • Names of variables generated to contain runtime type information. These have names like Ttypeencoding and TIDtype-encoding.Mangling Summary 285 Mangling Summary This section lists some of the C++ entities that are mangled and provides some details on the mangling algorithm. For more details, refer to The Annotated C++ Reference Manual. Type Name Mangling Using PGC++, each type has a corresponding mangled encoding. For example, a class type is represented as the class name preceded by the number of characters in the class name, as in 5abcde for abcde. Simple types are encoded as lower-case letters, as in i for int or f for float. Type modifiers and declarators are encoded as upper-case letters preceding the types they modify, as in U for unsigned or P for pointer. Nested Class Name Mangling Nested class types are encoded as a Q followed by a digit indicating the depth of nesting, followed by a _, followed by the mangled-form names of the class types in the fully-qualified name of the class, from outermost to innermost: class A class B // Q2_1A1B ; ; Local Class Name Mangling The name of the nested class itself is mangled to the form described above with a prefix __, which serves to make the class name distinct from all user names. Local class names are encoded as L followed by a number (which has no special meaning; it’s just an identifying number assigned to the class) followed by __ followed by the mangled name of the class (this is not in the ARM, and cfront encodes local class names slightly differently): void f() class A // L1__1A} ; ; This form is used when encoding the local class name as a type. It’s not necessary to mangle the name of the local class itself unless it's also a nested class.C++ Name Mangling 286 Template Class Name Mangling Template classes have mangled names that encode the arguments of the template: template class abc ; abc x; abc__pt__3_ii This describes two template arguments of type int with the total length of template argument list string, including the underscore, and a fixed string, indicates parameterized type as well, the name of the class template.Linux86 and Win32 Programming Model 287 Appendix A. Run-time Environment This appendix describes the programming model supported for compiler code generation, including register conventions and calling conventions for x86 and x64 processor-based systems. Section A1 addresses these conventions for processors running linux86 or Win32 operating systems, section A2 for processors running linux86-64 operating systems, and section A3 for processors running Win64 operating systems. Linux86 and Win32 Programming Model This section defines compiler and assembly language conventions for the use of certain aspects of an x86 processor running a linux86 or Win32 operating system. These standards must be followed to guarantee that compilers, application programs, and operating systems written by different people and organizations will work together. The conventions supported by the PGCC ANSI C compiler implement the application binary interface (ABI) as defined in the System V Application Binary Interface: Intel Processor Supplement and the System V Application Binary Interface, listed in the “Related Publications” section in the Preface. Function Calling Sequence This section describes the standard function calling sequence, including the stack frame, register usage, and parameter passing. Register Usage Conventions The following table defines the standard for register allocation. The 32-bit x86 Architecture provides a number of registers. All the integer registers and all the floating-point registers are global to all procedures in a running program. 288 Table A-1: Register Allocation In addition to the registers, each function has a frame on the run-time stack. This stack grows downward from high addresses. The next table shows the stack frame organization. Type Name Purpose General %eax integer return value %edx dividend register (for divide operations) %ecx count register (shift and string operations) %ebx local register variable %ebp optional stack frame pointer %esi local register variable %edi local register variable %esp stack pointer Floating-point %st(0) floating-point stack top, return value %st(1) floating-point next to stack top %st(...) %st(7) floating-point stack bottomLinux86 and Win32 Programming Model 289 Table A-2: Standard Stack Frame Several key points concerning the stack frame: • The stack is kept double word aligned • Argument words are pushed onto the stack in reverse order (i.e., the rightmost argument in C call syntax has the highest address) A dummy word may be pushed ahead of the rightmost argument in order to preserve doubleword alignment. All incoming arguments appear on the stack, residing in the stack frame of the caller. • An argument’s size is increased, if necessary, to make it a multiple of words. This may require tail padding, depending on the size of the argument. All registers on an x86 system are global and thus visible to both a calling and a called function. Registers %ebp, %ebx, %edi, %esi, and %esp are non-volatile across function calls. Therefore, a function must preserve these registers’ values for its caller. Remaining registers are volatile (scratch). If a calling function wants to preserve such a register value across a function call, it must save its value explicitly. Some registers have assigned roles in the standard calling sequence: %esp The stack pointer holds the limit of the current stack frame, which is the address of the stack’s bottom-most, valid word. At all times, the stack pointer should point to a word-aligned area. Position Contents Frame 4n+8 (%ebp) argument word n previous 8 (%ebp) argument word 0 4 (%ebp) return address 0 (%ebp) caller's %ebp current -4 (%ebp) n bytes of local -n (%ebp) variables and temps290 %ebp The frame pointer holds a base address for the current stack frame. Consequently, a function has registers pointing to both ends of its frame. Incoming arguments reside in the previous frame, referenced as positive offsets from %ebp, while local variables reside in the current frame, referenced as negative offsets from %ebp. A function must preserve this register value for its caller. %eax Integral and pointer return values appear in %eax. A function that returns a structure or union value places the address of the result in %eax. Otherwise, this is a scratch register. %esi, %edi These local registers have no specified role in the standard calling sequence. Functions must preserve their values for the caller. %ecx, %edx Scratch registers have no specified role in the standard calling sequence. Functions do not have to preserve their values for the caller. %st(0) Floating-point return values appear on the top of the floating point register stack; there is no difference in the representation of single or double-precision values in floating point registers. If the function does not return a floating point value, then the stack must be empty. %st(1) - %st(7) Floating point scratch registers have no specified role in the standard calling sequence. These registers must be empty before entry and upon exit from a function. EFLAGS The flags register contains the system flags, such as the direction flag and the carry flag. The direction flag must be set to the “forward” (i.e., zero) direction before entry and upon exit from a function. Other user flags have no specified role in the standard calling sequence and are not reserved. Floating Point Control Word The control word contains the floating-point flags, such as the rounding mode and exception masking. This register is initialized at process initialization time and its value must be preserved. Signals can interrupt processes. Functions called during signal handling have no unusual restriction on their use of registers. Moreover, if a signal handling function returns, the process resumes its original execution path with registers restored to their original values. Thus, programs and compilers may freely use all registers without danger of signal handlers changing their values.Linux86 and Win32 Programming Model 291 Function Return Values Functions Returning Scalars or No Value • A function that returns an integral or pointer value places its result in register %eax. • A function that returns a long long integer value places its result in the registers %edx and %eax. The most significant word is placed in %edx and the least significant word is placed in %eax. • A floating-point return value appears on the top of the floating point stack. The caller must then remove the value from the floating point stack, even if it does not use the value. Failure of either side to meet its obligations leads to undefined program behavior. The standard calling sequence does not include any method to detect such failures nor to detect return value type mismatches. Therefore, the user must declare all functions properly. There is no difference in the representation of single-, double- or extended-precision values in floating-point registers. • Functions that return no value (also called procedures or void functions) put no particular value in any register. • A call instruction pushes the address of the next instruction (the return address) onto the stack. The return instruction pops the address off the stack and effectively continues execution at the next instruction after the call instruction. A function that returns a scalar or no value must preserve the caller's registers as described above. Additionally, the called function must remove the return address from the stack, leaving the stack pointer (%esp) with the value it had before the call instruction was executed. Functions Returning Structures or Unions If a function returns a structure or union, then the caller provides space for the return value and places its address on the stack as argument word zero. In effect, this address becomes a hidden first argument. A function that returns a structure or union also sets %eax to the value of the original address of the caller's area before it returns. Thus, when the caller receives control again, the address of the returned object resides in register %eax and can be used to access the object. Both the calling and the called functions must cooperate to pass the return value successfully: • The calling function must supply space for the return value and pass its address in the stack frame;292 • The called function must use the address from the frame and copy the return value to the object so supplied; • The called function must remove this address from the stack before returning. Failure of either side to meet its obligation leads to undefined program behavior. The standard function calling sequence does not include any method to detect such failures nor to detect structure and union type mismatches. Therefore, you must declare the function properly. The following table illustrates the stack contents when the function receives control, after the call instruction, and when the calling function again receives control, after the ret instruction. Table A-3: Stack Contents for Functions Returning struct/union The following sections of this appendix describe where arguments appear on the stack. The examples are written as if the function prologue described above had been used. Position After Call After Return Position 4n+8 (%esp) argument word n argument word n 4n-4 (%esp) 8 (%esp) argument word 1 argument word 1 0 (%esp) 4 (%esp) value address undefined 0 (%esp) return addressLinux86 and Win32 Programming Model 293 Argument Passing Integral and Pointer Arguments As mentioned, a function receives all its arguments through the stack; the last argument is pushed first. In the standard calling sequence, the first argument is at offset 8(%ebp), the second argument is at offset 12(%ebp), etc., as previously shown in Table A-3 , “Stack Contents for Functions Returning struct/union”. Functions pass all integer-valued arguments as words, expanding or padding signed or unsigned bytes and halfwords as needed. Table A-4: Integral and Pointer Arguments Floating-Point Arguments The stack also holds floating-point arguments: single-precision values use one word and doubleprecision use two. The example below uses only double-precision arguments. Call Argument Stack Address g(1, 2, 3, (void *)0); 1 8 (%ebp) 2 12 (%ebp) 3 16 (%ebp) (void *) 0 20 (%ebp)294 Table A-5: Floating-point Arguments Structure and Union Arguments Structures and unions can have byte, halfword, or word alignment, depending on the constituents. An argument’s size is increased, if necessary, to make it a multiple of words. This may require tail padding, depending on the size of the argument. Structure and union arguments are pushed onto the stack in the same manner as integral arguments, described above. This provides call-by-value semantics, letting the called function modify its arguments without affecting the calling function’s object. In the example below, the argument, s, is a structure consisting of more than 2 words. Call Argument Stack Address h(1.414, 1, 2.998e10); word 0, 1.414 8 (%ebp) word 1, 1.414 12 (%ebp) 1 16 (%ebp) word 0 2.998e10 20 (%ebp) word 1, 2.998e10 24 (%ebp)Linux86 and Win32 Programming Model 295 Table A-6: Structure and Union Arguments Implementing a Stack In general, compilers and programmers must maintain a software stack. Register %esp is the stack pointer. Register %esp is set by the operating system for the application when the program is started. The stack must be a grow-down stack. A separate frame pointer enables calls to routines that change the stack pointer to allocate space on the stack at run-time (e.g. alloca). Some languages can also return values from a routine allocated on stack space below the original top-of-stack pointer. Such a routine prevents the calling function from using %esp-relative addressing to get at values on the stack. If the compiler does not call routines that leave %esp in an altered state when they return, a frame pointer is not needed and is not used if the compiler option –Mnoframe is specified. Although not required, the stack should be kept aligned on 8-byte boundaries so that 8-byte locals are favorably aligned with respect to performance. PGI's compilers allocate stack space for each routine in multiples of 8 bytes. Variable Length Parameter Lists. Parameter passing in registers can handle a variable number of parameters. The C language uses a special method to access variable-count parameters. The stdarg.h and varargs.h files define several functions to access these parameters. A C routine with variable parameters must use the va_start macro to set up a data structure before the parameters can be used. The va_arg macro must be used to access the successive parameters. Call Argument Stack Address i(1,s); 1 8 (%ebp) word 0, s 12 (%ebp) word 1, s 16 (%ebp) ... ...296 C Parameter Conversion. In C, for a called prototyped function, the parameter type in the called function must match the argument type in the calling function. If the called function is not prototyped, the calling convention uses the types of the arguments but promotes char or short to int, and unsigned char or unsigned short to unsigned int and promotes float to double, unless you use the --Msingle option. For more information on the –Msingle option, refer to Chapter 3. If the called function is prototyped, the unused bits of a register containing a char or short parameter are undefined and the called function must extend the sign of the unused bits when needed. Calling Assembly Language Programs Example A-1: C Program Calling an Assembly-language Routine /* File: testmain.c */ main(){ long l_para1 = 0x3f800000; float f_para2 = 1.0; double d_para3 = 0.5; float f_return; extern float sum_3 (long para1, float para2, double para3); f_return = sum_3(l_para1, f_para2, d_para3); printf("Parameter one, type long = %08x\n", l_para1); printf("Parameter two, type float = %f\n", f_para2); printf("Parameter three, type double = %g\n", d_para3); printf("The sum after conversion = %f\n", f_return); } # File: sum_3.s # Computes ( para1 + para2 ) + para3 .text .align 4 .long .EN1-sum_3+0xc8000000 .align 16 .globl sum_3 sum_3: pushl %ebp Linux86-64 Programming Model 297 movl %esp,%ebp subl $8,%esp ..EN1: fildl 8(%ebp) fadds 12(%ebp) faddl 16(%ebp) fstps -4(%ebp) flds -4(%ebp) leave ret .type sum_3,@function .size sum_3,.-sum_3 Linux86-64 Programming Model This section defines compiler and assembly language conventions for the use of certain aspects of an x64 processor running a linux86-64 operating system. These standards must be followed to guarantee that compilers, application programs, and operating systems written by different people and organizations will work together. The conventions supported by the PGCC ANSI C compiler implement the application binary interface (ABI) as defined in the System V Application Binary Interface: AMD64 Architecture Processor Supplement and the System V Application Binary Interface, listed in the “Related Publications” section in the Preface. Note The SUA64 Programming Model is the same as the Win64 Programming Model. Function Calling Sequence This section describes the standard function calling sequence, including the stack frame, register usage, and parameter passing. Register Usage Conventions. The following table defines the standard for register allocation. The x64 Architecture provides a variety of registers. All the general purpose registers, XMM registers, and x87 registers are global to all procedures in a running program. 298 Table A-7: Register Allocation In addition to the registers, each function has a frame on the run-time stack. This stack grows Type Name Purpose General %rax 1st return register %rbx callee-saved; optional base pointer %rcx pass 4th argument to functions %rdx pass 3rd argument to functions; 2nd return register %rsp stack pointer %rbp callee-saved; optional stack frame pointer %rsi pass 2nd argument to functions %rdi pass 1st argument to functions %r8 pass 5th argument to functions %r9 pass 6th argument to functions %r10 temporary register; pass a function’s static chain pointer %r11 temporary register %r12-r15 callee-saved registers XMM %xmm0-%xmm1 pass and return floating point arguments %xmm2-%xmm7 pass floating point arguments %xmm8- %xmm15 temporary registers x87 %st(0) temporary register; return long double arguments %st(1) temporary register; return long double arguments %st(2) - %st(7) temporary registersLinux86-64 Programming Model 299 downward from high addresses. The next table shows the stack frame organization. Table A-8: Standard Stack Frame Key points concerning the stack frame: • The end of the input argument area is aligned on a 16-byte boundary. • The 128-byte area beyond the location of %rsp is called the red zone and can be used for temporary local data storage. This area is not modified by signal or interrupt handlers. • A call instruction pushes the address of the next instruction (the return address) onto the stack. The return instruction pops the address off the stack and effectively continues execution at the next instruction after the call instruction. A function must preserve non-volatile registers (described below). Additionally, the called function must remove the return address from the stack, leaving the stack pointer (%rsp) with the value it had before the call instruction was executed. Position Contents Frame 8n+16 (%rbp) argument eightbyte n previous . . . 16 (%rbp) argument eightbyte 0 8 (%rbp) return address current 0 (%rbp) caller's %rbp current -8 (%rbp) unspecified . . . 0 (%rsp) variable size -128 (%rsp) red zone300 All registers on an x64 system are global and thus visible to both a calling and a called function. Registers %rbx, %rsp, %rbp, %r12, %r13, %r14, and %r15 are non-volatile across function calls. Therefore, a function must preserve these registers’ values for its caller. Remaining registers are volatile (scratch). If a calling function wants to preserve such a register value across a function call, it must save its value explicitly. Registers are used extensively in the standard calling sequence. The first six integer and pointer arguments are passed in these registers (listed in order): %rdi, %rsi, %rdx, %rcx, %r8, %r9. The first eight floating point arguments are passed in the first eight XMM registers: %xmm0, %xmm1, …, %xmm7. The registers %rax and %rdx are used to return integer and pointer values. The registers %xmm0 and %xmm1 are used to return floating point values. Additional registers with assigned roles in the standard calling sequence: %rsp The stack pointer holds the limit of the current stack frame, which is the address of the stack’s bottom-most, valid word. The stack must be 16-byte aligned. %rbp The frame pointer holds a base address for the current stack frame. Consequently, a function has registers pointing to both ends of its frame. Incoming arguments reside in the previous frame, referenced as positive offsets from %rbp, while local variables reside in the current frame, referenced as negative offsets from %rbp. A function must preserve this register value for its caller. RFLAGS The flags register contains the system flags, such as the direction flag and the carry flag. The direction flag must be set to the “forward” (i.e., zero) direction before entry and upon exit from a function. Other user flags have no specified role in the standard calling sequence and are not preserved. Floating Point Control Word The control word contains the floating-point flags, such as the rounding mode and exception masking. This register is initialized at process initialization time and its value must be preserved. Signals can interrupt processes. Functions called during signal handling have no unusual restriction on their use of registers. Moreover, if a signal handling function returns, the process resumes its original execution path with registers restored to their original values. Thus, programs and compilers may freely use all registers without danger of signal handlers changing their values.Linux86-64 Programming Model 301 Function Return Values Functions Returning Scalars or No Value • A function that returns an integral or pointer value places its result in the next available register of the sequence %rax, %rdx. • A function that returns a floating point value that fits in the XMM registers returns this value in the next available XMM register of the sequence %xmm0, %xmm1. • An X87 floating-point return value appears on the top of the floating point stack in %st(0) as an 80-bit X87 number. If this X87 return value is a complex number, the real part of the value is returned in %st(0) and the imaginary part in %st(1). • A function that returns a value in memory also returns the address of this memory in %rax. • Functions that return no value (also called procedures or void functions) put no particular value in any register. Functions Returning Structures or Unions A function can use either registers or memory to return a structure or union. The size and type of the structure or union determine how it is returned. If a structure or union is larger than 16 bytes, it is returned in memory allocated by the caller. To determine whether a 16-byte or smaller structure or union can be returned in one or more return registers, examine the first eight bytes of the structure or union. The type or types of the structure or union’s fields making up these eight bytes determine how these eight bytes will be returned. If the eight bytes contain at least one integral type, the eight bytes will be returned in %rax even if nonintegral types are also present in the eight bytes. If the eight bytes only contain floating point types, these eight bytes will be returned in %xmm0. If the structure or union is larger than eight bytes but smaller than 17 bytes, examine the type or types of the fields making up the second eight bytes of the structure or union. If these eight bytes contain at least one integral type, these eight bytes will be returned in %rdx even if non-integral types are also present in the eight bytes. If the eight bytes only contain floating point types, these eight bytes will be returned in %xmm1. If a structure or union is returned in memory, the caller provides the space for the return value and passes its address to the function as a “hidden” first argument in %rdi. This address will also be returned in %rax.302 Argument Passing Integral and Pointer Arguments Integral and pointer arguments are passed to a function using the next available register of the sequence %rdi, %rsi, %rdx, %rcx, %r8, %r9. After this list of registers has been exhausted, all remaining integral and pointer arguments are passed to the function via the stack. Floating-Point Arguments Float and double arguments are passed to a function using the next available XMM register taken in the order from %xmm0 to %xmm7. After this list of registers has been exhausted, all remaining float and double arguments are passed to the function via the stack. Structure and Union Arguments Structure and union arguments can be passed to a function in either registers or on the stack. The size and type of the structure or union determine how it is passed. If a structure or union is larger than 16 bytes, it is passed to the function in memory. To determine whether a 16-byte or smaller structure or union can be passed to a function in one or two registers, examine the first eight bytes of the structure or union. The type or types of the structure or union’s fields making up these eight bytes determine how these eight bytes will be passed. If the eight bytes contain at least one integral type, the eight bytes will be passed in the first available general purpose register of the sequence %rdi, %rsi, %rdx, %rcx, %r8, %r9 even if non-integral types are also present in the eight bytes. If the eight bytes only contain floating point types, these eight bytes will be passed in the first available XMM register of the sequence from %xmm0 to %xmm7. If the structure or union is larger than eight bytes but smaller than 17 bytes, examine the type or types of the fields making up the second eight bytes of the structure or union. If the eight bytes contain at least one integral type, the eight bytes will be passed in the next available general purpose register of the sequence %rdi, %rsi, %rdx, %rcx, %r8, %r9 even if non-integral types are also present in the eight bytes. If these eight bytes only contain floating point types, these eight bytes will be passed in the next available XMM register of the sequence from %xmm0 to %xmm7. If the first or second eight bytes of the structure or union cannot be passed in a register for some reason, the entire structure or union must be passed in memory.Linux86-64 Programming Model 303 Passing Arguments on the Stack If there are arguments left after every argument register has been allocated, the remaining arguments are passed to the function on the stack. The unassigned arguments are pushed on the stack in reverse order, with the last argument pushed first. Table A-9 , “Register Allocation for Example A-2” shows the register allocation and stack frame offsets for the function declaration and call shown in the following example. Both table and example are adapted from System V Application Binary Interface: AMD64 Architecture Processor Supplement. Example A-2: Parameter Passing typedef struct { int a, b; double d; } structparm; structparm s; int e,f,g,h,i,j,k; float flt; double m,n; extern void func (int e, int f, structparm s, int g, int h, float flt, double m, double n, int i, int j, int k); func (e, f, s, g, h, flt, m, n, i, j, k);304 Table A-9: Register Allocation for Example A-2 Implementing a Stack In general, compilers and programmers must maintain a software stack. The stack pointer, register %rsp, is set by the operating system for the application when the program is started. The stack must grow downwards from high addresses. A separate frame pointer enables calls to routines that change the stack pointer to allocate space on the stack at run-time (e.g. alloca). Some languages can also return values from a routine allocated on stack space below the original top-of-stack pointer. Such a routine prevents the calling function from using %rsp-relative addressing for values on the stack. If the compiler does not call routines that leave %rsp in an altered state when they return, a frame pointer is not needed and may not be used if the compiler option –Mnoframe is specified. The stack must be kept aligned on 16-byte boundaries. Variable Length Parameter Lists. Parameter passing in registers can handle a variable number of parameters. The C language uses a special method to access variable-count parameters. The stdarg.h and varargs.h files define several functions to access these parameters. A C routine with variable parameters must use the va_start macro to set up a data structure before the parameters can be used. The va_arg macro must be used to access the successive parameters. General Purpose Registers Floating Point Registers Stack Frame Offset %rdi: e %xmm0: s.d 0: j %rsi: f %xmm1: flt 8: k %rdx: s.a,s.b %xmm2: m %rcx: g %xmm3: n %r8: h %r9: iLinux86-64 Programming Model 305 For calls that use varargs or stdargs, the register %rax acts as a hidden argument whose value is the number of XMM registers used in the call. C Parameter Conversion. In C, for a called prototyped function, the parameter type in the called function must match the argument type in the calling function. If the called function is not prototyped, the calling convention uses the types of the arguments but promotes char or short to int, and unsigned char or unsigned short to unsigned int and promotes float to double, unless you use the --Msingle option. For more information on the –Msingle option, refer to Chapter 3. Calling Assembly Language Programs Example A-3: C Program Calling an Assembly-language Routine /* File: testmain.c */ main() { long l_para1 = 0x3f800000; float f_para2 = 1.0; double d_para3 = 0.5; float f_return; extern float sum_3 (long para1, float para2, double para3); f_return = sum_3(l_para1, f_para2, d_para3); printf("Parameter one, type long = %08x\n", l_para1); printf("Parameter two, type float = %f\n", f_para2); printf("Parameter three, type double = %g\n", d_para3); printf("The sum after conversion = %f\n", f_return); } # File: sum_3.s # Computes ( para1 + para2 ) + para3 .text .align 16 .globl sum_3 sum_3: pushq %rbp movq %rsp, %rbp306 cvtsi2ssq %rdi, %xmm2 addss %xmm0, %xmm2 cvtss2sd %xmm2, %xmm2 addsd %xmm1, %xmm2 cvtsd2ss %xmm2, %xmm2 movaps %xmm2, %xmm0 popq %rbp ret .type sum_3,@function .size sum_3,.-sum_3 Linux86-64 Fortran Supplement Sections A2.4.1 through A2.4.4 define the Fortran supplement to the ABI for x64 Linux. The register usage conventions set forth in that document remain the same for Fortran.Linux86-64 Programming Model 307 Fortran Fundamental Types Table A-10: Linux86-64 Fortran Fundamental Types Fortran Type Size (bytes) Alignment (bytes) INTEGER 4 4 INTEGER*1 1 1 INTEGER*2 2 2 INTEGER*4 4 4 INTEGER*8 8 8 LOGICAL 4 4 LOGICAL*1 1 1 LOGICAL*2 2 2 LOGICAL*4 4 4 LOGICAL*8 8 8 BYTE 1 1 CHARACTER*n n 1 REAL 4 4 REAL*4 4 4 REAL*8 8 8 DOUBLE PRECISION 8 8 COMPLEX 8 4 COMPLEX*8 8 4 COMPLEX*16 16 8308 A logical constant is one of: • .TRUE. • .FALSE. The logical constants .TRUE. and .FALSE. are defined to be the four-byte values -1 and 0 respectively. A logical expression is defined to be .TRUE. if its least significant bit is 1 and .FALSE. otherwise. Note that the value of a character is not automatically NULL-terminated. Naming Conventions By default, all globally visible Fortran symbol names (subroutines, functions, common blocks) are converted to lower-case. In addition, an underscore is appended to Fortran global names to distinguish the Fortran name space from the C/C++ name space.. Argument Passing and Return Conventions Arguments are passed by reference (i.e. the address of the argument is passed, rather than the argument itself). In contrast, C/C++ arguments are passed by value. When passing an argument declared as Fortran type CHARACTER, an argument representing the length of the CHARACTER argument is also passed to the function. This length argument is a four-byte integer passed by value, and is passed at the end of the parameter list following the other formal arguments. A length argument is passed for each CHARACTER argument; the length arguments are passed in the same order as their respective CHARACTER arguments. A Fortran function, returning a value of type CHARACTER, adds two arguments to the beginning of its argument list. The first additional argument is the address of the area created by the caller for the return value; the second additional argument is the length of the return value. If a Fortran function is declared to return a character value of constant length, for example CHARACTER*4 FUNCTION CHF(), the second extra parameter representing the length of the return value must still be supplied. DOUBLE COMPLEX 16 8 Fortran Type Size (bytes) Alignment (bytes)Linux86-64 Programming Model 309 A Fortran complex function returns its value in memory. The caller provides space for the return value and passes the address of this storage as if it were the first argument to the function. Alternate return specifiers of a Fortran function are not passed as arguments by the caller. The alternate return function passes the appropriate return value back to the caller in %rax. The handling of the following Fortran 90 features is implementation-defined: internal procedures, pointer arguments, assumed-shape arguments, functions returning arrays, and functions returning derived types. Inter-language Calling Inter-language calling between Fortran and C/C++ is possible if function/subroutine parameters and return values match types. If a C/C++ function returns a value, call it from Fortran as a function, otherwise, call it as a subroutine. If a Fortran function has type CHARACTER or COMPLEX, call it from C/ C++ as a void function. If a Fortran subroutine has alternate returns, call it from C/C++ as a function returning int; the value of such a subroutine is the value of the integer expression specified in the alternate RETURN statement. If a Fortran subroutine does not contain alternate returns, call it from C/ C++ as a void function. The following table provides the C/C++ data type corresponding to each Fortran data type.310 Table A-11: Fortran and C/C++ Data Type Compatibility Table A-12: Fortran and C/C++ Representation of the COMPLEX Type Fortran Type C/C++ Type Size (bytes) CHARACTER*n x char x[n] n REAL x float x 4 REAL*4 x float x 4 REAL*8 x double x 8 DOUBLE PRECISION x double x 8 INTEGER x int x 4 INTEGER*1 x signed char x 1 INTEGER*2 x short x 2 INTEGER*4 x int x 4 INTEGER*8 x long x, orlong long x 88 LOGICAL x int x 4 LOGICAL*1 x char x 1 LOGICAL*2 x short x 2 LOGICAL*4 x int x 4 LOGICAL*8 x long x, orlong long x 88 Fortran Type C/C++ Type Size (bytes) COMPLEX x struct {float r, I;} x; 8 COMPLEX*8 x struct {float r, I;} x; 8Linux86-64 Programming Model 311 Arrays C/C++ arrays and Fortran arrays use different default initial array index values. By default, C/C++ arrays start at 0 and Fortran arrays start at 1. A Fortran array can be declared to start at zero. Another difference between Fortran and C/C++ arrays is the storage method used. Fortran uses columnmajor order and C/C++ use row-major order. For one-dimensional arrays, this poses no problems. For two-dimensional arrays, where there are an equal number of rows and columns, row and column indexes can simply be reversed. Inter-language function mixing is not recommended for arrays other than single dimensional arrays and square two-dimensional arrays. Structures, Unions, Maps, and Derived Types. Fields within Fortran structures and derived types, and multiple map declarations within a Fortran union, conform to the same alignment requirements used by C structures. Common Blocks. A named Fortran common block can be represented in C/C++ by a structure whose members correspond to the members of the common block. The name of the structure in C/C++ must have the added underscore. For example, the Fortran common block: INTEGER I, J COMPLEX C DOUBLE COMPLEX CD DOUBLE PRECISION D COMMON /COM/ i, j, c, cd, d is represented in C with the following equivalent: COMPLEX*16 x struct {double dr,di;} x; 16 DOUBLE COMPLEX x struct {double dr,di;} x; 16 Fortran Type C/C++ Type Size (bytes)312 extern struct { int i; int j; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; and in C++ with the following equivalent: extern "C" struct { int i; int j; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; Note that the compiler-provided name of the BLANK COMMON block is implementation specific. Calling Fortran COMPLEX and CHARACTER functions from C/C++ is not as straightforward as calling other types of Fortran functions. Additional arguments must be passed to the Fortran function by the C/ C++ caller. A Fortran COMPLEX function returns its value in memory; the first argument passed to the function must contain the address of the storage for this value. A Fortran CHARACTER function adds two arguments to the beginning of its argument list. The following example of calling a Fortran CHARACTER function from C/C++ illustrates these caller-provided extra parameters: CHARACTER*(*) FUNCTION CHF(C1, I) CHARACTER*(*) C1 INTEGER I END extern void chf_(); char tmp[10]; char c1[9]; int i; chf_(tmp, 10, c1, &i, 9); The extra parameters tmp and 10 are supplied for the return value, while 9 is supplied as the length of c1. Refer to Section 2.8, Argument Passing and Return Conventions, for additional information.Win64/SUA64 Programming Model 313 Win64/SUA64 Programming Model This section defines compiler and assembly language conventions for the use of certain aspects of an x64 processor running a Win64 operating system, including SUA64. These standards must be followed to guarantee that compilers, application programs, and operating systems written by different people and organizations will work together. The conventions supported by the PGCC ANSI C compiler implement the application binary interface (ABI) as defined in the AMD64 Software Conventions document. Function Calling Sequence This section describes the standard function calling sequence, including the stack frame, register usage, and parameter passing. Register Usage Conventions. The following table defines the standard for register allocation. The 64-bit AMD64 and EM64T Architectures provide a number of registers. All the general purpose registers, XMM registers, and x87 registers are global to all procedures in a running program. 314 Table A-13: Register Allocation In addition to the registers, each function has a frame on the run-time stack. This stack grows downward from high addresses. The next table shows the stack frame organization. Type Name Purpose General %rax return value register %rbx callee-saved %rcx pass 1st argument to functions %rdx pass 2nd argument to functions %rsp stack pointer %rbp callee-saved; optional stack frame pointer %rsi callee-saved %rdi callee-saved %r8 pass 3rd argument to functions %r9 pass 4th argument to functions %r10-%r11 temporary registers; used in syscall/sysret instructions %r12-r15 callee-saved registers XMM %xmm0 pass 1st floating point argument; return value register %xmm1 pass 2nd floating point argument %xmm2 pass 3rd floating point argument %xmm3 pass 4th floating point argument %xmm4-%xmm5 temporary registers %xmm6-%xmm15 callee-saved registersWin64/SUA64 Programming Model 315 Table A-14: Standard Stack Frame Key points concerning the stack frame: • The parameter area at the bottom of the stack must contain enough space to hold all the parameters needed by any function call. Space must be set aside for the four register parameters to be “homed” to the stack even if there are less than four register parameters used in a given call. • Sixteen-byte alignment of the stack is required except within a function’s prolog and within leaf functions. All registers on an x64 system are global and thus visible to both a calling and a called function. Registers %rbx, %rsp, %rbp, %rsi, %rdi, %r12, %r13, %r14, and %r15 are non-volatile. Therefore, a called function must preserve these registers’ values for its caller. Remaining registers are scratch. If a calling function wants to preserve such a register value across a function call, it must save a value in its local stack frame. Position Contents Frame 8n-120 (%rbp) argument eightbyte n previous . . . -80 (%rbp) argument eightbyte 5 -88 (%rbp) %r9 home -96 (%rbp) %r8 home -104 (%rbp) %rdx home -112 (%rbp) %rcx home -120 (%rbp) return address current -128 (%rbp) caller's %rbp . . . 0 (%rsp) variable size316 Registers are used in the standard calling sequence. The first four arguments are passed in registers. Integral and pointer arguments are passed in these general purpose registers (listed in order): %rcx, %rdx, %r8, %r9. Floating point arguments are passed in the first four XMM registers: %xmm0, %xmm1, %xmm2, %xmm3. Registers are assigned using the argument’s ordinal position in the argument list. For example, if a function’s first argument is an integral type and its second argument is a floating-point type, the first argument will be passed in the first general purpose register (%rcx) and the second argument will be passed in the second XMM register (%xmm1); the first XMM register and second general purpose register are ignored. Arguments after the first four are passed on the stack. Integral and pointer type return values are returned in %rax. Floating point return values are returned in %xmm0. Additional registers with assigned roles in the standard calling sequence: %rsp The stack pointer holds the limit of the current stack frame, which is the address of the stack’s bottom-most, valid word. The stack pointer should point to a 16-byte aligned area unless in the prolog or a leaf function. %rbp The frame pointer, if used, can provide a way to reference the previous frame on the stack. Details are implementation dependent. A function must preserve this register value for its caller. MXCSR The flags register MXCSR contains the system flags, such as the direction flag and the carry flag. The six status flags (MXCSR[0:5]) are volatile; the remainder of the register is nonvolatile. x87 Floating Point Control Word (FPCSR)The control word contains the floating-point flags, such as the rounding mode and exception masking. This register is initialized at process initialization time and its value must be preserved. Signals can interrupt processes. Functions called during signal handling have no unusual restriction on their use of registers. Moreover, if a signal handling function returns, the process resumes its original execution path with registers restored to their original values. Thus, programs and compilers may freely use all registers without danger of signal handlers changing their values. Win64/SUA64 Programming Model 317 Function Return Values Functions Returning Scalars or No Value • A function that returns an integral or pointer value that fits in 64 bits places its result in %rax. • A function that returns a floating point value that fits in the XMM registers returns this value in %xmm0. • A function that returns a value in memory via the stack places the address of this memory (passed to the function as a “hidden” first argument in %rcx) in %rax. • Functions that return no value (also called procedures or void functions) put no particular value in any register. • A call instruction pushes the address of the next instruction (the return address) onto the stack. The return instruction pops the address off the stack and effectively continues execution at the next instruction after the call instruction. A function that returns a scalar or no value must preserve the caller's registers as described above. Additionally, the called function must remove the return address from the stack, leaving the stack pointer (%rsp) with the value it had before the call instruction was executed. Functions Returning Structures or Unions A function can use either registers or the stack to return a structure or union. The size and type of the structure or union determine how it is returned. A structure or union is returned in memory if it is larger than 8 bytes or if its size is 3, 5, 6, or 7 bytes. A structure or union is returned in %rax if its size is 1, 2, 4, or 8 bytes. If a structure or union is to be returned in memory, the caller provides space for the return value and passes its address to the function as a “hidden” first argument in %rcx. This address will also be returned in %rax. Argument Passing Integral and Pointer Arguments Integral and pointer arguments are passed to a function using the next available register of the sequence %rcx, %rdx, %r8, %r9. After this list of registers has been exhausted, all remaining integral and pointer arguments are passed to the function via the stack.318 Floating-Point Arguments Float and double arguments are passed to a function using the next available XMM register of the sequence %xmm0, %xmm1, %xmm2, %xmm3. After this list of registers has been exhausted, all remaining XMM floating-point arguments are passed to the function via the stack. Array, Structure, and Union Arguments Arrays and strings are passed to functions using a pointer to caller-allocated memory. Structure and union arguments of size 1, 2, 4, or 8 bytes will be passed as if they were integers of the same size. Structures and unions of other sizes will be passed as a pointer to a temporary, allocated by the caller, and whose value contains the value of the argument. The caller-allocated temporary memory used for arguments of aggregate type must be 16-byte aligned. Passing Arguments on the Stack Registers are assigned using the argument’s ordinal position in the argument list. For example, if a function’s first argument is an integral type and its second argument is a floating-point type, the first argument will be passed in the first general purpose register (%rcx) and the second argument will be passed in the second XMM register (%xmm1); the first XMM register and second general purpose register are ignored. Arguments after the first four are passed on the stack; they are pushed on the stack in reverse order, with the last argument pushed first. Table A-15 , “Register Allocation for Example A-4” shows the register allocation and stack frame offsets for the function declaration and call shown in the following example. Example A-4: Parameter Passing typedef struct { int i; float f; } struct1; int i; float f; double d; long l; long long ll; struct1 s1; extern void func (int i, float f, struct1 s1, double d, long long ll, long l); func (i, f, s1, d, ll, l);Win64/SUA64 Programming Model 319 Table A-15: Register Allocation for Example A-4 Implementing a Stack In general, compilers and programmers must maintain a software stack. The stack pointer, register %rsp, is set by the operating system for the application when the program is started. The stack must grow downwards from high addresses. A separate frame pointer enables calls to routines that change the stack pointer to allocate space on the stack at run-time (e.g. alloca). Some languages can also return values from a routine allocated on stack space below the original top-of-stack pointer. Such a routine prevents the calling function from using %rsp-relative addressing to get at values on the stack. If the compiler does not call routines that leave %rsp in an altered state when they return, a frame pointer is not needed and is not used if the compiler option –Mnoframe is specified. The stack must always be 16-byte aligned except within the prolog and within leaf functions. Variable Length Parameter Lists. Parameter passing in registers can handle a variable number of parameters. The C language uses a special method to access variable-count parameters. The stdarg.h and varargs.h files define several functions to access these parameters. A C routine with variable parameters must use the va_start macro to set up a data structure before the parameters can be used. The va_arg macro must be used to access the successive parameters. For unprototyped functions or functions that use varargs, floating-point arguments passed in registers must be passed in both an XMM register and its corresponding general purpose register. General Purpose Registers Floating Point Registers Stack Frame Offset %rcx: i %xmm0: 32: ll %rdx: %xmm1: f 40: l %r8: s1.i, s1.f %xmm2: %r9: %xmm3: d320 C Parameter Conversion. In C, for a called prototyped function, the parameter type in the called function must match the argument type in the calling function. If the called function is not prototyped, the calling convention uses the types of the arguments but promotes char or short to int, and unsigned char or unsigned short to unsigned int and promotes float to double, unless you use the –Msingle option. For more information on the –Msingle option, refer to Chapter 3. If the called function is prototyped, the unused bits of a register containing a char or short parameter are undefined and the called function must extend the sign of the unused bits when needed. Calling Assembly Language Programs Example A-5: C Program Calling an Assembly-language Routine /* File: testmain.c */ main() { long l_para1 = 0x3f800000; float f_para2 = 1.0; double d_para3 = 0.5; float f_return; extern float sum_3 (long para1, float para2, double para3); f_return = sum_3(l_para1, f_para2, d_para3); printf("Parameter one, type long = %08x\n", l_para1); printf("Parameter two, type float = %f\n", f_para2); printf("Parameter three, type double = %g\n", d_para3); printf("The sum after conversion = %f\n", f_return); } # File: sum_3.s # Computes ( para1 + para2 ) + para3 .text .align 16 .globl sum_3 sum_3: pushq %rbp leaq 128(%rsp), %rbp cvtsi2ss %ecx, %xmm0 addss %xmm1, %xmm0 Win64/SUA64 Programming Model 321 cvtss2sd %xmm0, %xmm0 addsd %xmm2, %xmm0 cvtsd2ss %xmm0, %xmm0 popq %rbp ret .type sum_3,@function .size sum_3,.-sum_3 Win64/SUA64 Fortran Supplement Sections A3.4.1 through A3.4.4 define the Fortran supplement to the AMD64 Software Conventions for Win64. The register usage conventions set forth in that document remain the same for Fortran.322 Fortran Fundamental Types Table A-16: Win64/SUA64 Fortran Fundamental Types Fortran Type Size (bytes) Alignment (bytes) INTEGER 4 4 INTEGER*1 1 1 INTEGER*2 2 2 INTEGER*4 4 4 INTEGER*8 8 8 LOGICAL 4 4 LOGICAL*1 1 1 LOGICAL*2 2 2 LOGICAL*4 4 4 LOGICAL*8 8 8 BYTE 1 1 CHARACTER*n n 1 REAL 4 4 REAL*4 4 4 REAL*8 8 8 DOUBLE PRECISION 8 8 COMPLEX 8 4 COMPLEX*8 8 4 COMPLEX*16 16 8Win64/SUA64 Programming Model 323 A logical constant is one of: • .TRUE. • .FALSE. The logical constants .TRUE. and .FALSE. are defined to be the four-byte values -1 and 0 respectively. A logical expression is defined to be .TRUE. if its least significant bit is 1 and .FALSE. otherwise. Note that the value of a character is not automatically NULL-terminated. Fortran Naming Conventions By default, all globally visible Fortran symbol names (subroutines, functions, common blocks) are converted to lower-case. In addition, an underscore is appended to Fortran global names to distinguish the Fortran name space from the C/C++ name space. Fortran Argument Passing and Return Conventions Arguments are passed by reference (i.e. the address of the argument is passed, rather than the argument itself). In contrast, C/C++ arguments are passed by value. When passing an argument declared as Fortran type CHARACTER, an argument representing the length of the CHARACTER argument is also passed to the function. This length argument is a four-byte integer passed by value, and is passed at the end of the parameter list following the other formal arguments. A length argument is passed for each CHARACTER argument; the length arguments are passed in the same order as their respective CHARACTER arguments. A Fortran function, returning a value of type CHARACTER, adds two arguments to the beginning of its argument list. The first additional argument is the address of the area created by the caller for the return value; the second additional argument is the length of the return value. If a Fortran function is declared to return a character value of constant length, for example CHARACTER*4 FUNCTION CHF(), the second extra parameter representing the length of the return value must still be supplied. DOUBLE COMPLEX 16 8 Fortran Type Size (bytes) Alignment (bytes)324 A Fortran complex function returns its value in memory. The caller provides space for the return value and passes the address of this storage as if it were the first argument to the function. Alternate return specifiers of a Fortran function are not passed as arguments by the caller. The alternate return function passes the appropriate return value back to the caller in %rax. The handling of the following Fortran 90 features is implementation-defined: internal procedures, pointer arguments, assumed-shape arguments, functions returning arrays, and functions returning derived types. Interlanguage Calling Inter-language calling between Fortran and C/C++ is possible if function/subroutine parameters and return values match types. If a C/C++ function returns a value, call it from Fortran as a function, otherwise, call it as a subroutine. If a Fortran function has type CHARACTER or COMPLEX, call it from C/ C++ as a void function. If a Fortran subroutine has alternate returns, call it from C/C++ as a function returning int; the value of such a subroutine is the value of the integer expression specified in the alternate RETURN statement. If a Fortran subroutine does not contain alternate returns, call it from C/ C++ as a void function. The following table provides the C/C++ data type corresponding to each Fortran data type.Win64/SUA64 Programming Model 325 Table A-17: Fortran and C/C++ Data Type Compatibility Table A-18: Fortran and C/C++ Representation of the COMPLEX Type Fortran Type C/C++ Type Size (bytes) CHARACTER*n x char x[n] n REAL x float x 4 REAL*4 x float x 4 REAL*8 x double x 8 DOUBLE PRECISION x double x 8 INTEGER x int x 4 INTEGER*1 x signed char x 1 INTEGER*2 x short x 2 INTEGER*4 x int x 4 INTEGER*8 x long long x 8 LOGICAL x int x 4 LOGICAL*1 x char x 1 LOGICAL*2 x short x 2 LOGICAL*4 x int x 4 LOGICAL*8 x long long x 8 Fortran Type C/C++ Type Size (bytes) COMPLEX x struct {float r, I;} x; 8 COMPLEX*8 x struct {float r, I;} x; 8326 Arrays C/C++ arrays and Fortran arrays use different default initial array index values. By default, C/C++ arrays start at 0 and Fortran arrays start at 1. A Fortran array can be declared to start at zero. Another difference between Fortran and C/C++ arrays is the storage method used. Fortran uses column-major order and C/C++ use row-major order. For one-dimensional arrays, this poses no problems. For two-dimensional arrays, where there are an equal number of rows and columns, row and column indexes can simply be reversed. Inter-language function mixing is not recommended for arrays other than single dimensional arrays and square two-dimensional arrays. Structures, Unions, Maps, and Derived Types. Fields within Fortran structures and derived types, and multiple map declarations within a Fortran union, conform to the same alignment requirements used by C structures. Common Blocks. A named Fortran common block can be represented in C/C++ by a structure whose members correspond to the members of the common block. The name of the structure in C/C++ must have the added underscore. For example, the Fortran common block: INTEGER I, J COMPLEX C DOUBLE COMPLEX CD DOUBLE PRECISION D COMMON /COM/ i, j, c, cd, d is represented in C with the following equivalent: COMPLEX*16 x struct {double dr,di;} x; 16 DOUBLE COMPLEX x struct {double dr,di;} x; 16 Fortran Type C/C++ Type Size (bytes)Win64/SUA64 Programming Model 327 extern struct { int i; int j; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; and in C++ with the following equivalent: extern "C" struct { int i; int j; struct {float real, imag;} c; struct {double real, imag;} cd; double d; } com_; Note that the compiler-provided name of the BLANK COMMON block is implementation specific. Calling Fortran COMPLEX and CHARACTER functions from C/C++ is not as straightforward as calling other types of Fortran functions. Additional arguments must be passed to the Fortran function by the C/C++ caller. A Fortran COMPLEX function returns its value in memory; the first argument passed to the function must contain the address of the storage for this value. A Fortran CHARACTER function adds two arguments to the beginning of its argument list. The following example of calling a Fortran CHARACTER function from C/C++ illustrates these caller-provided extra parameters: CHARACTER*(*) FUNCTION CHF(C1, I) CHARACTER*(*) C1 INTEGER I END extern void chf_(); char tmp[10]; char c1[9]; int i; chf_(tmp, 10, c1, &i, 9); The extra parameters tmp and 10 are supplied for the return value, while 9 is supplied as the length of c1. Refer to “Argument Passing and Return Values” on page 244, for additional information.328Diagnostic Messages 329 Appendix B. Messages This appendix describes the various messages that the compiler produces. These messages include the sign-on message and diagnostic messages for remarks, warnings, and errors. The compiler always displays any error messages, along with the erroneous source line, on the screen. If you specify the – Mlist option, the compiler places any error messages in the listing file. You can also use the –v option to display more information about the compiler, assembler, and linker invocations and about the host system. For more information on the –Mlist and –v options, refer to 3, “Command Line Options”. Diagnostic Messages Diagnostic messages provide syntactic and semantic information about your source text. Syntactic information includes information such as syntax errors. Semantic includes information includes such as unreachable code. You can specify that the compiler displays error messages at a certain level with the -Minform option. The compiler messages refer to a severity level, a message number, and the line number where the error occurs. The compiler can also display internal error messages on standard errors. If your compilation produces any internal errors, contact you’re The Portland Group’s technical reporting service by sending e-mail to trs@pgroup.com. If you use the listing file option –Mlist, the compiler places diagnostic messages after the source lines in the listing file, in the following format: PGFTN-etype-enum-message (filename: line) Where: etype is a character signifying the severity level enum is the error number message is the error message filename is the source filename line is the line number where the compiler detected an error.330 Phase Invocation Messages You can display compiler, assembler, and linker phase invocations by using the –v command line option. For further information about this option, see 3, “Command Line Options”. Fortran Compiler Error Messages This section presents the error messages generated by the PGF77 and PGF95 compilers. The compilers display error messages in the program listing and on standard output; and can also display internal error messages on standard error. Message Format Each message is numbered. Each message also lists the line and column number where the error occurs. A dollar sign ($) in a message represents information that is specific to each occurrence of the message. Message List Error message severities: I informative W warning S severe error F fatal error V variable V000 Internal compiler error. $ $ This message indicates an error in the compiler, rather than a user error – although it may be possi- ble for a user error to cause an internal error. The severity may vary; if it is informative or warning, correct object code was probably generated, but it is not safe to rely on this. Regardless of the severity or cause, internal errors should be reported to trs@pgroup.com. F001 Source input file name not specified On the command line, source file name should be specified either before all the switches, or after them. F002 Unable to open source input file: $Fortran Compiler Error Messages 331 Source file name misspelled, file not in current working directory, or file is read protected. F003 Unable to open listing file Probably, user does not have write permission for the current working directory. F004 $ $ Generic message for file errors. F005 Unable to open temporary file Compiler uses directory "/usr/tmp" or "/tmp" in which to create temporary files. If neither of these directories is available on the node on which the compiler is being used, this error will occur. S006 Input file empty Source input file does not contain any Fortran statements other than comments or compiler directives. F007 Subprogram too large to compile at this optimization level $ Internal compiler data structure overflow, working storage exhausted, or some other non-recoverable problem related to the size of the subprogram. If this error occurs at opt 2, reducing the opt level to 1 may work around the problem. Moving the subprogram being compiled to its own source file may eliminate the problem. If this error occurs while compiling a subprogram of fewer than 2000 statements it should be reported to the compiler maintenance group as a possible compiler problem. F008 Error limit exceeded The compiler gives up because too many severe errors were issued; the error limit can be reset on the command line. F009 Unable to open assembly file Probably, user does not have write permission for the current working directory. F010 File write error occurred $ Probably, file system is full. S011 Unrecognized command line switch: $332 Refer to PDS reference document for list of allowed compiler switches. S012 Value required for command line switch: $ Certain switches require an immediately following value, such as "-opt 2". V000 Internal compiler error. $ $ This message indicates an error in the compiler, rather than a user error – although it may be possi- ble for a user error to cause an internal error. The severity may vary; if it is informative or warning, correct object code was probably generated, but it is not safe to rely on this. Regardless of the severity or cause, internal errors should be reported to trs@pgroup.com. F001 Source input file name not specified On the command line, source file name should be specified either before all the switches, or after them. F002 Unable to open source input file: $ Source file name misspelled, file not in current working directory, or file is read protected. F003 Unable to open listing file Probably, user does not have write permission for the current working directory. F004 $ $ Generic message for file errors. F005 Unable to open temporary file Compiler uses directory "/usr/tmp" or "/tmp" in which to create temporary files. If neither of these directories is available on the node on which the compiler is being used, this error will occur. S006 Input file empty Source input file does not contain any Fortran statements other than comments or compiler directives. F007 Subprogram too large to compile at this optimization level $Fortran Compiler Error Messages 333 Internal compiler data structure overflow, working storage exhausted, or some other non-recoverable problem related to the size of the subprogram. If this error occurs at opt 2, reducing the opt level to 1 may work around the problem. Moving the subprogram being compiled to its own source file may eliminate the problem. If this error occurs while compiling a subprogram of fewer than 2000 statements it should be reported to the compiler maintenance group as a possible compiler problem. F008 Error limit exceeded The compiler gives up because too many severe errors were issued; the error limit can be reset on the command line. F009 Unable to open assembly file Probably, user does not have write permission for the current working directory. F010 File write error occurred $ Probably, file system is full. S011 Unrecognized command line switch: $ Refer to PDS reference document for list of allowed compiler switches. S012 Value required for command line switch: $ Certain switches require an immediately following value, such as "-opt 2". S013 Unrecognized value specified for command line switch: $ S014 Ambiguous command line switch: $ Too short an abbreviation was used for one of the switches. W015 Hexadecimal or octal constant truncated to fit data type I016 Identifier, $, truncated to 31 chars334 An identifier may be at most 31 characters in length; characters after the 31st are ignored. S017 Unable to open include file: $ File is missing, read protected, or maximum include depth (10) exceeded. Remember that the file name should be enclosed in quotes. S018 Illegal label $ $ Used for label ’field’ errors or illegal values. E.g., in fixed source form, the label field (first five characters) of the indicated line contains a non-numeric character. S019 Illegally placed continuation line A continuation line does not follow an initial line, or more than 99 continuation lines were specified. S020 Unrecognized compiler directive Refer to user’s manual for list of allowed compiler directives. S021 Label field of continuation line is not blank The first five characters of a continuation line must be blank. S022 Unexpected end of file - missing END statement S023 Syntax error - unbalanced $ Unbalanced parentheses or brackets. W024 CHARACTER or Hollerith constant truncated to fit data type A character or hollerith constant was converted to a data type that was not large enough to contain all of the characters in the constant. This type conversion occurs when the constant is used in an arithmetic expression or is assigned to a non-character variable. The character or hollerith constant is truncated on the right, that is, if 4 characters are needed then the first 4 are used and the remaining characters are discarded. W025 Illegal character ($) - ignoredFortran Compiler Error Messages 335 The current line contains a character, possibly non-printing, which is not a legal Fortran character (characters inside of character or Hollerith constants cannot cause this error). As a general rule, all non-printing characters are treated as white space characters (blanks and tabs); no error message is generated when this occurs. If for some reason, a non-printing character is not treated as a white space character, its hex representation is printed in the form dd where each d is a hex digit. S026 Unmatched quote S027 Illegal integer constant: $ Integer constant is too large for 32 bit word. S028 Illegal real or double precision constant: $ S029 Illegal $ constant: $ Illegal hexadecimal, octal, or binary constant. A hexadecimal constant consists of digits 0..9 and letters A..F or a..f; any other character in a hexadecimal constant is illegal. An octal constant consists of digits 0..7; any other digit or character in an octal constant is illegal. A binary constant consists of digits 0 or 7; any other digit or character in a binary constant is illegal. S030 Explicit shape must be specified for $ S031 Illegal data type length specifier for $ The data type length specifier (e.g. 4 in INTEGER*4) is not a constant expression that is a member of the set of allowed values for this particular data type. W032 Data type length specifier not allowed for $ The data type length specifier (e.g. 4 in INTEGER*4) is not allowed in the given syntax (e.g. DIMENSION A(10)*4). S033 Illegal use of constant $ A constant was used in an illegal context, such as on the left side of an assignment statement or as the target of a data initialization statement.336 S034 Syntax error at or near $ I035 Predefined intrinsic $ loses intrinsic property An intrinsic name was used in a manner inconsistent with the language definition for that intrinsic. The compiler, based on the context, will treat the name as a variable or an external function. S036 Illegal implicit character range First character must alphabetically precede second. S037 Contradictory data type specified for $ The indicated identifier appears in more than one type specification statement and different data types are specified for it. S038 Symbol, $, has not been explicitly declared The indicated identifier must be declared in a type statement; this is required when the IMPLICIT NONE statement occurs in the subprogram. W039 Symbol, $, appears illegally in a SAVE statement $ An identifier appearing in a SAVE statement must be a local variable or array. S040 Illegal common variable $ Indicated identifier is a dummy variable, is already in a common block, or has previously been defined to be something other than a variable or array. W041 Illegal use of dummy argument $ This error can occur in several situations. It can occur if dummy arguments were specified on a PROGRAM statement. It can also occur if a dummy argument name occurs in a DATA, COMMON, SAVE, or EQUIVALENCE statement. A program statement must have an empty argument list. S042 $ is a duplicate dummy argument S043 Illegal attempt to redefine $ $Fortran Compiler Error Messages 337 An attempt was made to define a symbol in a manner inconsistent with an earlier definition of the same symbol. This can happen for a number of reasons. The message attempts to indicate the situation that occurred. intrinsic - An attempt was made to redefine an intrinsic function. A symbol that represents an intrinsic function may be redefined if that symbol has not been previously verified to be an intrinsic function. For example, the intrinsic sin can be defined to be an integer array. If a symbol is verified to be an intrinsic function via the INTRINSIC statement or via an intrinsic function reference then it must be referred to as an intrinsic function for the remainder of the program unit. symbol - An attempt was made to redefine a symbol that was previously defined. An example of this is to declare a symbol to be a PARAMETER which was previously declared to be a subprogram argument. S044 Multiple declaration for symbol $ A redundant declaration of a symbol has occurred. For example, an attempt was made to declare a symbol as an ENTRY when that symbol was previously declared as an ENTRY. S045 Data type of entry point $ disagrees with function $ The current function has entry points with data types inconsistent with the data type of the current function. For example, the function returns type character and an entry point returns type complex. S046 Data type length specifier in wrong position The CHARACTER data type specifier has a different position for the length specifier from the other data types. Suppose, we want to declare arrays ARRAYA and ARRAYB to have 8 elements each having an element length of 4 bytes. The difference is that ARRAYA is character and ARRAYB is integer. The declarations would be CHARACTER ARRAYA(8)*4 and INTEGER ARRAYB*4(8). S047 More than seven dimensions specified for array S048 Illegal use of ’*’ in declaration of array $ An asterisk may be used only as the upper bound of the last dimension. S049 Illegal use of ’*’ in non-subroutine subprogram The alternate return specifier ’*’ is legal only in the subroutine statement. Programs, functions, and block data are not allowed to have alternate return specifiers.338 S050 Assumed size array, $, is not a dummy argument S051 Unrecognized built-in % function The allowable built-in functions are %VAL, %REF, %LOC, and %FILL. One was encountered that did not match one of these allowed forms. S052 Illegal argument to %VAL or %LOC S053 %REF or %VAL not legal in this context The built-in functions %REF and %VAL can only be used as actual parameters in procedure calls. W054 Implicit character $ used in a previous implicit statement An implicit character has been given an implied data type more than once. The implied data type for the implicit character is changed anyway. W055 Multiple implicit none statements The IMPLICIT NONE statement can occur only once in a subprogram. W056 Implicit type declaration The -dclchk switch and an implicit declaration following an IMPLICIT NONE statement will produce a warning message for IMPLICIT statements. S057 Illegal equivalence of dummy variable, $ Dummy arguments may not appear in EQUIVALENCE statements. S058 Equivalenced variables $ and $ not in same common block A common block variable must not be equivalenced with a variable in another common block. S059 Conflicting equivalence between $ and $ The indicated equivalence implies a storage layout inconsistent with other equivalences.Fortran Compiler Error Messages 339 S060 Illegal equivalence of structure variable, $ STRUCTURE and UNION variables may not appear in EQUIVALENCE statements. S061 Equivalence of $ and $ extends common block backwards W062 Equivalence forces $ to be unaligned EQUIVALENCE statements have defined an address for the variable which has an alignment not optimal for variables of its data type. This can occur when INTEGER and CHARACTER data are equivalenced, for instance. I063 Gap in common block $ before $ S064 Illegal use of $ in DATA statement implied DO loop The indicated variable is referenced where it is not an active implied DO index variable. S065 Repeat factor less than zero S066 Too few data constants in initialization statement S067 Too many data constants in initialization statement S068 Numeric initializer for CHARACTER $ out of range 0 through 255 A CHARACTER*1 variable or character array element can be initialized to an integer, octal, or hexadecimal constant if that constant is in the range 0 through 255. S069 Illegal implied DO expression The only operations allowed within an implied DO expression are integer +, -, *, and /.340 S070 Incorrect sequence of statements $ The statement order is incorrect. For instance, an IMPLICIT NONE statement must precede a specification statement which in turn must precede an executable statement. S071 Executable statements not allowed in block data S072 Assignment operation illegal to $ $ The destination of an assignment operation must be a variable, array reference, or vector reference. The assignment operation may be by way of an assignment statement, a data statement, or the index variable of an implied DO-loop. The compiler has determined that the identifier used as the destination, is not a storage location. The error message attempts to indicate the type of entity used. entry point - An assignment to an entry point that was not a function procedure was attempted. external procedure - An assignment to an external procedure or a Fortran intrinsic name was attempted. if the identifier is the name of an entry point that is not a function, an external procedure... S073 Intrinsic or predeclared, $, cannot be passed as an argument S074 Illegal number or type of arguments to $ $ The indicated symbol is an intrinsic or generic function, or a predeclared subroutine or function, requiring a certain number of arguments of a fixed data type. S075 Subscript, substring, or argument illegal in this context for $ This can happen if you try to doubly index an array such as ra(2)(3). This also applies to substring and function references. S076 Subscripts specified for non-array variable $ S077 Subscripts omitted from array $Fortran Compiler Error Messages 341 S078 Wrong number of subscripts specified for $ S079 Keyword form of argument illegal in this context for $$ S080 Subscript for array $ is out of bounds S081 Illegal selector $ $ S082 Illegal substring expression for variable $ Substring expressions must be of type integer and if constant must be greater than zero. S083 Vector expression used where scalar expression required A vector expression was used in an illegal context. For example, iscalar = iarray, where a scalar is assigned the value of an array. Also, character and record references are not vectorizable. S084 Illegal use of symbol $ $ This message is used for many different errors. S085 Incorrect number of arguments to statement function $ S086 Dummy argument to statement function must be a variable S087 Non-constant expression where constant expression required342 S088 Recursive subroutine or function call of $ A function may not call itself. S089 Illegal use of symbol, $, with character length = * Symbols of type CHARACTER*(*) must be dummy variables and must not be used as statement function dummy parameters and statement function names. Also, a dummy variable of type CHARACTER*(*) cannot be used as a function. S090 Hollerith constant more than 4 characters In certain contexts, Hollerith constants may not be more than 4 characters long. S091 Constant expression of wrong data type S092 Illegal use of variable length character expression A character expression used as an actual argument, or in certain contexts within I/O statements, must not consist of a concatenation involving a passed length character variable. W093 Type conversion of expression performed An expression of some data type appears in a context which requires an expression of some other data type. The compiler generates code to convert the expression into the required type. S094 Variable $ is of wrong data type $ The indicated variable is used in a context which requires a variable of some other data type. S095 Expression has wrong data type An expression of some data type appears in a context which requires an expression of some other data type. S096 Illegal complex comparison The relations .LT., .GT., .GE., and .LE. are not allowed for complex values. S097 Statement label $ has been defined more than once More than one statement with the indicated statement number occurs in the subprogram.Fortran Compiler Error Messages 343 S098 Divide by zero S099 Illegal use of $ Aggregate record references may only appear in aggregate assignment statements, unformatted I/O statements, and as parameters to subprograms. They may not appear, for example, in expressions. Also, records with differing structure types may not be assigned to one another. S100 Expression cannot be promoted to a vector An expression was used that required a scalar quantity to be promoted to a vector illegally. For example, the assignment of a character constant string to a character array. Records, too, cannot be promoted to vectors. S101 Vector operation not allowed on $ Record and character typed entities may only be referenced as scalar quantities. S102 Arithmetic IF expression has wrong data type The parenthetical expression of an arithmetic if statement must be an integer, real, or double precision scalar expression. S103 Type conversion of subscript expression for $ The data type of a subscript expression must be integer. If it is not, it is converted. S104 Illegal control structure $ This message is issued for a number of errors involving IF-THEN statements and DO loops. If the line number specified is the last line (END statement) of the subprogram, the error is probably an unterminated DO loop or IF-THEN statement. S105 Unmatched ELSEIF, ELSE or ENDIF statement An ELSEIF, ELSE, or ENDIF statement cannot be matched with a preceding IF-THEN statement. S106 DO index variable must be a scalar variable The DO index variable cannot be an array name, a subscripted variable, a PARAMETER name, a function name, a structure name, etc.344 S107 Illegal assigned goto variable $ S108 Illegal variable, $, in NAMELIST group $ A NAMELIST group can only consist of arrays and scalars which are not dummy arguments and pointerbased variables. I109 Overflow in $ constant $, constant truncated at left A non-decimal (hexadecimal, octal, or binary) constant requiring more than 64-bits produces an overflow. The constant is truncated at left (e.g. ’1234567890abcdef1’x will be ’234567890abcdef1’x). I110 I111 Underflow of real or double precision constant I112 Overflow of real or double precision constant S113 Label $ is referenced but never defined S114 Cannot initialize $ W115 Assignment to DO variable $ in loop S116 Illegal use of pointer-based variable $ $ S117 Statement not allowed within a $ definition The statement may not appear in a STRUCTURE or derived type definition.Fortran Compiler Error Messages 345 S118 Statement not allowed in DO, IF, or WHERE block I119 Redundant specification for $ Data type of indicated symbol specified more than once. I120 Label $ is defined but never referenced I121 Operation requires logical or integer data types An operation in an expression was attempted on data having a data type incompatible with the operation. For example, a logical expression can consist of only logical elements of type integer or logical. Real data would be invalid. I122 Character string truncated Character string or Hollerith constant appearing in a DATA statement or PARAMETER statement has been truncated to fit the declared size of the corresponding identifier. W123 Hollerith length specification too big, reduced The length specifier field of a hollerith constant specified more characters than were present in the character field of the hollerith constant. The length specifier was reduced to agree with the number of characters present. S124 Relational expression mixes character with numeric data A relational expression is used to compare two arithmetic expressions or two character expressions. A character expression cannot be compared to an arithmetic expression. I125 Dummy procedure $ not declared EXTERNAL A dummy argument which is not declared in an EXTERNAL statement is used as the subprogram name in a CALL statement, or is called as a function, and is therefore assumed to be a dummy procedure. This message can result from a failure to declare a dummy array. I126 Name $ is not an intrinsic function346 I127 Optimization level for $ changed to opt 1 $ W128 Integer constant truncated to fit data type: $ An integer constant will be truncated when assigned to data types smaller than 32-bits, such as a BYTE. I129 Floating point overflow. Check constants and constant expressions I130 Floating point underflow. Check constants and constant expressions I131 Integer overflow. Check floating point expressions cast to integer I132 Floating pt. invalid oprnd. Check constants and constant expressions I133 Divide by 0.0. Check constants and constant expressions S134 Illegal attribute $ $ W135 Missing STRUCTURE name field A STRUCTURE name field is required on the outermost structure.Fortran Compiler Error Messages 347 W136 Field-namelist not allowed The field-namelist field of the STRUCTURE statement is disallowed on the outermost structure. W137 Field-namelist is required in nested structures W138 Multiply defined STRUCTURE member name $ A member name was used more than once within a structure. W139 Structure $ in RECORD statement not defined A RECORD statement contains a reference to a STRUCTURE that has not yet been defined. S140 Variable $ is not a RECORD S141 RECORD required on left of $ S142 $ is not a member of this RECORD S143 $ requires initializer W144 NEED ERROR MESSAGE $ $ This is used as a temporary message for compiler development. W145 %FILL only valid within STRUCTURE block The %FILL special name was used outside of a STRUCTURE multiline statement. It is only valid when used within a STRUCTURE multiline statement even though it is ignored. S146 Expression must be character type348 S147 Character expression not allowed in this context S148 Reference to $ required An aggregate reference to a record was expected during statement compilation but another data type was found instead. S149 Record where arithmetic value required An aggregate record reference was encountered when an arithmetic expression was expected. S150 Structure, Record, derived type, or member $ not allowed in this context A structure, record, or member reference was found in a context which is not supported. For example, the use of structures, records, or members within a data statement is disallowed. S151 Empty TYPE, STRUCTURE, UNION, or MAP TYPE - ENDTYPE, STRUCTURE - ENDSTRUCTURE, UNION - ENDUNION MAP - ENDMAP declaration contains no members. S152 All dimension specifiers must be ’:’ S153 Array objects are not conformable $ S154 DISTRIBUTE target, $, must be a processor S155 $ $ S156 Number of colons and triplets must be equal in ALIGN $ with $Fortran Compiler Error Messages 349 S157 Illegal subscript use of ALIGN dummy $ - $ S158 Alternate return not specified in SUBROUTINE or ENTRY An alternate return can only be used if alternate return specifiers appeared in the SUBROUTINE or ENTRY statements. S159 Alternate return illegal in FUNCTION subprogram An alternate return cannot be used in a FUNCTION. S160 ENDSTRUCTURE, ENDUNION, or ENDMAP does not match top S161 Vector subscript must be rank-one array W162 Not equal test of loop control variable $ replaced with < or > test. S163 S164 Overlapping data initializations of $ An attempt was made to data initialize a variable or array element already initialized. S165 $ appeared more than once as a subprogram A subprogram name appeared more than once in the source file. The message is applicable only when an assembly file is the output of the compiler. S166 $ cannot be a common block and a subprogram A name appeared as a common block name and a subprogram name. The message is applicable only when an assembly file is the output of the compiler.350 I167 Inconsistent size of common block $ A common block occurs in more than one subprogram of a source file and its size is not identical. The maximum size is chosen. The message is applicable only when an assembly file is the output of the compiler. S168 Incompatible size of common block $ A common block occurs in more than one subprogram of a source file and is initialized in one subprogram. Its initialized size was found to be less than its size in the other subprogram(s). The message is applicable only when an assembly file is the output of the compiler. W169 Multiple data initializations of common block $ A common block is initialized in more than one subprogram of a source file. Only the first set of initializations apply. The message is applicable only when an assembly file is the output of the compiler. W170 F90 extension: $ $ Use of a nonstandard feature. A description of the feature is provided. W171 F90 extension: nonstandard statement type $ W172 F90 extension: numeric initialization of CHARACTER $ A CHARACTER*1 variable or array element was initialized with a numeric value. W173 F90 extension: nonstandard use of data type length specifier W174 F90 extension: type declaration contains data initialization W175 F90 extension: IMPLICIT range contains nonalpha charactersFortran Compiler Error Messages 351 W176 F90 extension: nonstandard operator $ W177 F90 extension: nonstandard use of keyword argument $ W178 W179 F90 extension: use of structure field reference $ W180 F90 extension: nonstandard form of constant W181 F90 extension: & alternate return W182 F90 extension: mixed non-character and character elements in COMMON $ W183 F90 extension: mixed non-character and character EQUIVALENCE ($,$) W184 Mixed type elements (numeric and/or character types) in COMMON $ W185 Mixed numeric and/or character type EQUIVALENCE ($,$) S186 Argument missing for formal argument $352 S187 Too many arguments specified for $ S188 Argument number $ to $: type mismatch S189 Argument number $ to $: association of scalar actual argument to array dummy argument S190 Argument number $ to $: non-conformable arrays S191 Argument number $ to $ cannot be an assumed-size array S192 Argument number $ to $ must be a label W193 Argument number $ to $ does not match INTENT (OUT) W194 INTENT(IN) argument cannot be defined - $ S195 Statement may not appear in an INTERFACE block $ S196 Deferred-shape specifiers are required for $Fortran Compiler Error Messages 353 S197 Invalid qualifier or qualifier value (/$) in OPTIONS statement An illegal qualifier was found or a value was specified for a qualifier which does not expect a value. In either case, the qualifier for which the error occurred is indicated in the error message. S198 $ $ in ALLOCATE/DEALLOCATE W199 Unaligned memory reference A memory reference occurred whose address does not meet its data alignment requirement. S200 Missing UNIT/FILE specifier S201 Illegal I/O specifier - $ S202 Repeated I/O specifier - $ S203 FORMAT statement has no label S204 $ $ Miscellaneous I/O error. S205 Illegal specification of scale factor The integer following + or - has been omitted, or P does not follow the integer value. S206 Repeat count is zero S207 Integer constant expected in edit descriptor354 S208 Period expected in edit descriptor S209 Illegal edit descriptor S210 Exponent width not used in the Ew.dEe or Gw.dEe edit descriptors S211 Internal I/O not allowed in this I/O statement S212 Illegal NAMELIST I/O Namelist I/O cannot be performed with internal, unformatted, formatted, and list-directed I/O. Also, I/O lists must not be present. S213 $ is not a NAMELIST group name S214 Input item is not a variable reference S215 Assumed sized array name cannot be used as an I/O item or specifier An assumed sized array was used as an item to be read or written or as an I/O specifier (i.e., FMT = array-name). In these contexts the size of the array must be known. S216 STRUCTURE/UNION cannot be used as an I/O item S217 ENCODE/DECODE buffer must be a variable, array, or array elementFortran Compiler Error Messages 355 S218 Statement labeled $ $ S219 S220 Redefining predefined macro $ S221 #elif after #else A preprocessor #elif directive was found after a #else directive; only #endif is allowed in this context. S222 #else after #else A preprocessor #else directive was found after a #else directive; only #endif is allowed in this context. S223 #if-directives too deeply nested Preprocessor #if directive nesting exceeded the maximum allowed (currently 10). S224 Actual parameters too long for $ The total length of the parameters in a macro call to the indicated macro exceeded the maximum allowed (currently 2048). W225 Argument mismatch for $ The number of arguments supplied in the call to the indicated macro did not agree with the number of parameters in the macro’s definition. F226 Can’t find include file $ The indicated include file could not be opened. S227 Definition too long for $ The length of the macro definition of the indicated macro exceeded the maximum allowed (currently 2048). S228 EOF in comment356 The end of a file was encountered while processing a comment. S229 EOF in macro call to $ The end of a file was encountered while processing a call to the indicated macro. S230 EOF in string The end of a file was encountered while processing a quoted string. S231 Formal parameters too long for $ The total length of the parameters in the definition of the indicated macro exceeded the maximum allowed (currently 2048). S232 Identifier too long The length of an identifier exceeded the maximum allowed (currently 2048). S233 W234 Illegal directive name The sequence of characters following a # sign was not an identifier. W235 Illegal macro name A macro name was not an identifier. S236 Illegal number $ The indicated number contained a syntax error. F237 Line too long The input source line length exceeded the maximum allowed (currently 2048). W238 Missing #endif End of file was encountered before a required #endif directive was found. W239 Missing argument list for $Fortran Compiler Error Messages 357 A call of the indicated macro had no argument list. S240 Number too long The length of a number exceeded the maximum allowed (currently 2048). W241 Redefinition of symbol $ The indicated macro name was redefined. I242 Redundant definition for symbol $ A definition for the indicated macro name was found that was the same as a previous definition. F243 String too long The length of a quoted string exceeded the maximum allowed (currently 2048). S244 Syntax error in #define, formal $ not identifier A formal parameter that was not an identifier was used in a macro definition. W245 Syntax error in #define, missing blank after name or arglist There was no space or tab between a macro name or argument list and the macro’s definition. S246 Syntax error in #if A syntax error was found while parsing the expression following a #if or #elif directive. S247 Syntax error in #include The #include directive was not correctly formed. W248 Syntax error in #line A #line directive was not correctly formed. W249 Syntax error in #module A #module directive was not correctly formed. W250 Syntax error in #undef358 A #undef directive was not correctly formed. W251 Token after #ifdef must be identifier The #ifdef directive was not followed by an identifier. W252 Token after #ifndef must be identifier The #ifndef directive was not followed by an identifier. S253 Too many actual parameters to $ The number of actual arguments to the indicated macro exceeded the maximum allowed (currently 31). S254 Too many formal parameters to $ The number of formal arguments to the indicated macro exceeded the maximum allowed (currently 31). F255 Too much pushback The preprocessor ran out of space while processing a macro expansion. The macro may be recursive. W256 Undefined directive $ The identifier following a # was not a directive name. S257 EOF in #include directive End of file was encountered while processing a #include directive. S258 Unmatched #elif A #elif directive was encountered with no preceding #if or #elif directive. S259 Unmatched #else A #else directive was encountered with no preceding #if or #elif directive. S260 Unmatched #endif A #endif directive was encountered with no preceding #if, #ifdef, or #ifndef directive. S261 Include files nested too deeplyFortran Compiler Error Messages 359 The nesting depth of #include directives exceeded the maximum (currently 20). S262 Unterminated macro definition for $ A newline was encountered in the formal parameter list for the indicated macro. S263 Unterminated string or character constant A newline with no preceding backslash was found in a quoted string. I264 Possible nested comment The characters /* were found within a comment. S265 S266 S267 W268 Cannot inline subprogram; common block mismatch W269 Cannot inline subprogram; argument type mismatch This message may be Severe if have gone too far to undo inlining process. F270 Missing -exlib option W271 Can’t inline $ - wrong number of arguments I272 Argument of inlined function not used360 S273 Inline library not specified on command line (-inlib switch) F274 Unable to access file $/TOC S275 Unable to open file $ while extracting or inlining F276 Assignment to constant actual parameter in inlined subprogram I277 Inlining of function $ may result in recursion S278 W279 Possible use of $ before definition in $ The optimizer has detected the possibility that a variable is used before it has been assigned a value. The names of the variable and the function in which the use occurred are listed. The line number, if specified, is the line number of the basic block containing the use of the variable. W280 Syntax error in directive $ messages 280-300 rsvd for directive handling W281 Directive ignored - $ $ S300 Too few data constants in initialization of derived type $Fortran Compiler Error Messages 361 S301 $ must be TEMPLATE or PROCESSOR S302 Unmatched END$ statement S303 END statement for $ required in an interface block S304 EXIT/CYCLE statement must appear in a DO/DOWHILE loop$$ S305 $ cannot be named, $ S306 $ names more than one construct S307 $ must have the construct name $ S308 DO may not terminate at an EXIT, CYCLE, RETURN, STOP, GOTO, or arithmetic IF S309 Incorrect name, $, specified in END statement S310 $ $ Generic message for MODULE errors. W311 Non-replicated mapping for $ array, $, ignored362 W312 Array $ should be declared SEQUENCE W313 Subprogram $ called within INDEPENDENT loop not PURE E314 IPA: actual argument $ is a label, but dummy argument $ is not an asterisk The call passes a label to the subprogram; the corresponding dummy argument in the subprogram should be an asterisk to declare this as the alternate return. I315 IPA: routine $, $ constant dummy arguments This many dummy arguments are being replaced by constants due to interprocedural analysis. I316 IPA: routine $, $ INTENT(IN) dummy arguments This many dummy arguments are being marked as INTENT(IN) due to interprocedural analysis. I317 IPA: routine $, $ array alignments propagated This many array alignments were propagated by interprocedural analysis. I318 IPA: routine $, $ distribution formats propagated This many array distribution formats were propagated by interprocedural analysis. I319 IPA: routine $, $ distribution targets propagated This many array distribution targets were propagated by interprocedural analysis. I320 IPA: routine $, $ common blocks optimized This many mapped common blocks were optimized by interprocedural analysis. I321 IPA: routine $, $ common blocks not optimized This many mapped common blocks were not optimized by interprocedural analysis, either because they were declared differently in different routines, or they did not appear in the main program. I322 IPA: analyzing main program $Fortran Compiler Error Messages 363 Interprocedural analysis is building the call graph and propagating information with the named main program. I323 IPA: collecting information for $ Interprocedural analysis is saving information for the current subprogram for subsequent analysis and propagation. W324 IPA file $ appears to be out of date W325 IPA file $ is for wrong subprogram: $ W326 Unable to open file $ to propagate IPA information to $ I327 IPA: $ subprograms analyzed I328 IPA: $ dummy arguments replaced by constants I329 IPA: $ INTENT(IN) dummy arguments should be INTENT(INOUT) I330 IPA: $ dummy arguments changed to INTENT(IN) I331 IPA: $ inherited array alignments replaced I332 IPA: $ transcriptive distribution formats replaced364 I333 IPA: $ transcriptive distribution targets replaced I334 IPA: $ descriptive/prescriptive array alignments verified I335 IPA: $ descriptive/prescriptive distribution formats verified I336 IPA: $ descriptive/prescriptive distribution targets verified I337 IPA: $ common blocks optimized I338 IPA: $ common blocks not optimized S339 Bad IPA contents file: $ S340 Bad IPA file format: $ S341 Unable to create file $ while analyzing IPA information S342 Unable to open file $ while analyzing IPA informationFortran Compiler Error Messages 365 S343 Unable to open IPA contents file $ S344 Unable to create file $ while collecting IPA information F345 Internal error in $: table overflow Analysis failed due to a table overflowing its maximum size. W346 Subprogram $ appears twice The subprogram appears twice in the same source file; IPA will ignore the first appearance. F347 Missing -ipalib option Interprocedural analysis, enabled with the -ipacollect, -ipaanalyze, or -ipapropagate options, requires the -ipalib option to specify the library directory. W348 Common /$/ $ has different distribution target The array was declared in a common block with a different distribution target in another subprogram. W349 Common /$/ $ has different distribution format The array was declared in a common block with a different distribution format in another subprogram. W350 Common /$/ $ has different alignment The array was declared in a common block with a different alignment in another subprogram. W351 Wrong number of arguments passed to $ The subroutine or function statement for the given subprogram has a different number of dummy arguments than appear in the call. W352 Wrong number of arguments passed to $ when bound to $366 The subroutine or function statement for the given subprogram has a different number of dummy arguments than appear in the call to the EXTERNAL name given. W353 Subprogram $ is missing A call to a subroutine or function with this name appears, but it could not be found or analyzed. I354 Subprogram $ is not called No calls to the given subroutine or function appear anywhere in the program. W355 Missing argument in call to $ A nonoptional argument is missing in a call to the given subprogram. I356 Array section analysis incomplete Interprocedural analysis for array section arguments is incomplete; some information may not be available for optimization. I357 Expression analysis incomplete Interprocedural analysis for expression arguments is incomplete; some information may not be available for optimization. W358 Dummy argument $ is EXTERNAL, but actual is not subprogram The call statement passes a scalar or array to a dummy argument that is declared EXTERNAL. W359 SUBROUTINE $ passed to FUNCTION dummy argument $ The call statement passes a subroutine name to a dummy argument that is used as a function. W360 FUNCTION $ passed to FUNCTION dummy argument $ with different result type The call statement passes a function argument to a function dummy argument, but the dummy has a different result type. W361 FUNCTION $ passed to SUBROUTINE dummy argument $ The call statement passes a function name to a dummy argument that is used as a subroutine.Fortran Compiler Error Messages 367 W362 Argument $ has a different type than dummy argument $ The type of the actual argument is different than the type of the corresponding dummy argument. W363 Dummy argument $ is a POINTER but actual argument $ is not The dummy argument is a pointer, so the actual argument must be also. W364 Array or array expression passed to scalar dummy argument $ The actual argument is an array, but the dummy argument is a scalar variable. W365 Scalar or scalar expression passed to array dummy argument $ The actual argument is a scalar variable, but the dummy argument is an array. F366 Internal error: interprocedural analysis fails An internal error occurred during interprocedural analysis; please report this to the compiler maintenance group. If user errors were reported when collecting IPA information or during IPA analysis, correcting them may avoid this error. I367 Array $ bounds cannot be matched to formal argument Passing a nonsequential array to a sequential dummy argument may require copying the array to sequential storage. The most common cause is passing an ALLOCATABLE array or array expression to a dummy argument that is declared with explicit bounds. Declaring the dummy argument as assumed shape, with bounds (:,:,:), will remove this warning. W368 Array-valued expression passed to scalar dummy argument $ The actual argument is an array-valued expression, but the dummy argument is a scalar variable. W369 Dummy argument $ has different rank than actual argument The actual argument is an array or array-valued expression with a different rank than the dummy argument.368 W370 Dummy argument $ has different shape than actual argument The actual argument is an array or array-valued expression with a different shape than the dummy argument; this may require copying the actual argument into sequential storage. W371 Dummy argument $ is INTENT(IN) but may be modified The dummy argument was declared as INTENT(IN), but analysis has found that the argument may be modified; the INTENT(IN) declaration should be changed. W372 Cannot propagate alignment from $ to $ The most common cause is when passing an array with an inherited alignment to a dummy argument with non- inherited alignment. I373 Cannot propagate distribution format from $ to $ The most common cause is when passing an array with a transcriptive distribution format to a dummy argument with prescriptive or descriptive distribution format. I374 Cannot propagate distribution target from $ to $ The most common cause is when passing an array with a transcriptive distribution target to a dummy argument with prescriptive or descriptive distribution target. I375 Distribution format mismatch between $ and $ Usually this arises when the actual and dummy arguments are distributed in different dimensions. I376 Alignment stride mismatch between $ and $ This may arise when the actual argument has a different stride in its alignment to its template than does the dummy argument. I377 Alignment offset mismatch between $ and $ This may arise when the actual argument has a different offset in its alignment to its template than does the dummy argument. I378 Distribution target mismatch between $ and $ This may arise when the actual and dummy arguments have different distribution target sizes.Fortran Compiler Error Messages 369 I379 Alignment of $ is too complex The alignment specification of the array is too complex for interprocedural analysis to verify or propagate; the program will work correctly, but without the benefit of IPA. I380 Distribution format of $ is too complex The distribution format specification of the array is too complex for interprocedural analysis to verify or propagate; the program will work correctly, but without the benefit of IPA. I381 Distribution target of $ is too complex The distribution target specification of the array is too complex for interprocedural analysis to verify or propagate; the program will work correctly, but without the benefit of IPA. I382 IPA: $ subprograms analyzed Interprocedural analysis succeeded in finding and analyzing this many subprograms in the whole program. I383 IPA: $ dummy arguments replaced by constants Interprocedural analysis has found this many dummy arguments in the whole program that can be replaced by constants. I384 IPA: $ dummy arguments changed to INTENT(IN) Interprocedural analysis has found this many dummy arguments in the whole program that are not modified and can be declared as INTENT(IN). W385 IPA: $ INTENT(IN) dummy arguments should be INTENT(INOUT) Interprocedural analysis has found this many dummy arguments in the whole program that were declared as INTENT(IN) but should be INTENT(INOUT). I386 IPA: $ array alignments propagated Interprocedural analysis has found this many array dummy arguments that could have the inherited array alignment replaced by a descriptive alignment. I387 IPA: $ array alignments verified370 Interprocedural analysis has verified that the prescriptive or descriptive alignments of this many array dummy arguments match the alignments of the actual argument. I388 IPA: $ array distribution formats propagated Interprocedural analysis has found this many array dummy arguments that could have the transcriptive distribution format replaced by a descriptive format. I389 IPA: $ array distribution formats verified Interprocedural analysis has verified that the prescriptive or descriptive distribution formats of this many array dummy arguments match the formats of the actual argument. I390 IPA: $ array distribution targets propagated Interprocedural analysis has found this many array dummy arguments that could have the transcriptive distribution target replaced by a descriptive target. I391 IPA: $ array distribution targets verified Interprocedural analysis has verified that the prescriptive or descriptive distribution targets of this many array dummy arguments match the targets of the actual argument. I392 IPA: $ common blocks optimized Interprocedural analysis has found this many common blocks that could be optimized. I393 IPA: $ common blocks not optimized Interprocedural analysis has found this many common blocks that could not be optimized, either because the common block was not declared in the main program, or because it was declared differently in different subprograms. I394 IPA: $ replaced by constant value The dummy argument was replaced by a constant as per interprocedural analysis. I395 IPA: $ changed to INTENT(IN) The dummy argument was changed to INTENT(IN) as per interprocedural analysis. I396 IPA: array alignment propagated to $Fortran Compiler Error Messages 371 The template alignment for the dummy argument was changed as per interprocedural analysis. I397 IPA: distribution format propagated to $ The distribution format for the dummy argument was changed as per interprocedural analysis. I398 IPA: distribution target propagated to $ The distribution target for the dummy argument was changed as per interprocedural analysis. I399 IPA: common block $ not optimized The given common block was not optimized by interprocedural analysis either because it was not declared in the main program, or because it was declared differently in different subprograms. E400 IPA: dummy argument $ is an asterisk, but actual argument is not a label The subprogram expects an alternate return label for this argument. E401 Actual argument $ is a subprogram, but Dummy argument $ is not declared EXTERNAL The call statement passes a function or subroutine name to a dummy argument that is a scalar variable or array. E402 Actual argument $ is illegal E403 Actual argument $ and formal argument $ have different ranks The actual and formal array arguments differ in rank, which is allowed only if both arrays are declared with the HPF SEQUENCE attribute. E404 Sequential array section of $ in argument $ is not contiguous When passing an array section to a formal argument that has the HPF SEQUENCE attribute, the actual argument must be a whole array with the HPF SEQUENCE attribute, or an array section of such an array where the section is a contiguous sequence of elements.372 E405 Array expression argument $ may not be passed to sequential dummy argument $ When the dummy argument has the HPF SEQUENCE attribute, the actual argument must be a whole array with the HPF SEQUENCE attribute or a contiguous array section of such an array, unless an INTERFACE block is used. E406 Actual argument $ and formal argument $ have different character lengths The actual and formal array character arguments have different character lengths, which is allowed only if both character arrays are declared with the HPF SEQUENCE attribute, unless an INTERFACE block is used. W407 Argument $ has a different character length than dummy argument $ The character length of the actual argument is different than the length specified for the corresponding dummy argument. W408 Specified main program $ is not a PROGRAM The main program specified on the command line is a subroutine, function, or block data subprogram. W409 More than one main program in IPA directory: $ and $ There is more than one main program analyzed in the IPA directory shown. The first one found is used. W410 No main program found; IPA analysis fails. The main program must appear in the IPA directory for analysis to proceed. W411 Formal argument $ is DYNAMIC but actual argument is an expression W412 Formal argument $ is DYNAMIC but actual argument $ is notFortran Compiler Error Messages 373 I413 Formal argument $ has two reaching distributions and may be a candidate for cloning I414 $ and $ may be aliased and one of them is assigned Interprocedural analysis has determined that two formal arguments because the same variable is passed in both argument positions, or one formal argument and a global or COMMON variable may be aliased, because the global or COMMON variable is passed as an actual argument. If either alias is assigned in the subroutine, unexpected results may occur; this message alerts the user that this situation is disallowed by the Fortran standard. F415 IPA fails: incorrect IPA file Interprocedural analysis saves its information in special IPA files in the specified IPA directory. One of these files has been renamed or corrupted. This can arise when there are two files with the same prefix, such as ’a.hpf’ and ’a.f90’. E416 Argument $ has the SEQUENCE attribute, but the dummy parameter $ does not When an actual argument is an array with the SEQUENCE attribute, the dummy parameter must have the SEQUENCE attribute or an INTERFACE block must be used. E417 Interface block for $ is a SUBROUTINE but should be a FUNCTION E418 Interface block for $ is a FUNCTION but should be a SUBROUTINE E419 Interface block for $ is a FUNCTION has wrong result type W420 Earlier $ directive overrides $ directive374 W421 $ directive can only appear in a function or subroutine E422 Nonconstant DIM= argument is not supported E423 Constant DIM= argument is out of range E424 Equivalence using substring or vector triplets is not allowed E425 A record is not allowed in this context E426 WORD type cannot be converted E427 Interface block for $ has wrong number of arguments E428 Interface block for $ should have $ E429 Interface block for $ should not have $ E430 Interface block for $ has wrong $ W431 Program is too large for Interprocedural Analysis to completeFortran Compiler Error Messages 375 W432 Illegal type conversion $ E433 Subprogram $ called within INDEPENDENT loop not LOCAL W434 Incorrect home array specification ignored S435 Array declared with zero size An array was declared with a zero or negative dimension bound, as ’real a(-1)’, or an upper bound less than the lower bound, as ’real a(4:2)’. W436 Independent loop not parallelized$ W437 Type $ will be mapped to $ Where DOUBLE PRECISION is not supported, it is mapped to REAL, and similarly for COMPLEX(16) or COMPLEX*32. E438 $ $ not supported on this platform This construct is not supported by the compiler for this target. S439 An internal subprogram cannot be passed as argument - $ S440 Defined assignment statements may not appear in WHERE statement or WHERE block S441 $ may not appear in a FORALL block376 E442 Adjustable-length character type not supported on this host - $ $ S443 EQUIVALENCE of derived types not supported on this host - $ S444 Derived type in EQUIVALENCE statement must have SEQUENCE attribute - $ A variable or array with derived type appears in an EQUIVALENCE statement. The derived type must have the SEQUENCE attribute, but does not. E445 Array bounds must be integer $ $ The expressions in the array bounds must be integer. S446 Argument number $ to $: rank mismatch The number of dimensions in the array or array expression does not match the number of dimensions in the dummy argument. S447 Argument number $ to $ must be a subroutine or function name S448 Argument number $ to $ must be a subroutine name S449 Argument number $ to $ must be a function name S450 Argument number $ to $: kind mismatchFortran Compiler Error Messages 377 S451 Arrays of derived type with a distributed member are not supported S452 Assumed length character, $, is not a dummy argument S453 Derived type variable with pointer member not allowed in IO - $ $ S454 Subprogram $ is not a module procedure Only names of module procedures declared in this module or accessed through USE association can appear in a MODULE PROCEDURE statement. S455 A derived type array section cannot appear with a member array section - $ A reference like A(:)%B(:), where ’A’ is a derived type array and ’B’ is a member array, is not allowed; a section subscript may appear after ’A’ or after ’B’, but not both. S456 Unimplemented for data type for MATMUL S457 Illegal expression in initialization S458 Argument to NULL() must be a pointer S459 Target of NULL() assignment must be a pointer S460 ELEMENTAL procedures cannot be RECURSIVE378 S461 Dummy arguements of ELEMENATAL procedures must be scalar S462 Arguments and return values of ELEMENATAL procedures cannot have the POINTER attribute S463 Arguments of ELEMENATAL procedures cannot be procedures S464 An ELEMENTAL procedure cannot be passed as argument - $ Fortran Runtime Error Messages This section presents the error messages generated by the runtime system. The runtime system displays error messages on standard output. Message Format The messages are numbered but have no severity indicators because they all terminate program execution. Message List Here are the runtime error messages: 201 illegal value for specifier An improper specifier value has been passed to an I/O runtime routine. Example: within an OPEN statement, form='unknown'.Fortran Runtime Error Messages 379 202 conflicting specifiers Conflicting specifiers have been passed to an I/O runtime routine. Example: within an OPEN statement, form='unformatted', blank='null'. 203 record length must be specified A recl specifier required for an I/O runtime routine has not been passed. Example: within an OPEN statement, access='direct' has been passed, but the record length has not been specified (recl=specifier). 204 illegal use of a readonly file Self explanatory. Check file and directory modes for readonly status. 205 'SCRATCH' and 'SAVE'/'KEEP' both specified In an OPEN statement, a file disposition conflict has occurred. Example: within an OPEN statement, status='scratch' and dispose='keep' have been passed. 206 attempt to open a named file as 'SCRATCH' 207 file is already connected to another unit 208 'NEW' specified for file that already exists 209 'OLD' specified for file that does not exist 210 dynamic memory allocation failed Memory allocation operations occur only in conjunction with namelist I/O. The most probable cause of fixed buffer overflow is exceeding the maximum number of simultaneously open file units. 211 invalid file name380 212 invalid unit number A file unit number less than or equal to zero has been specified. 215 formatted/unformatted file conflict Formatted/unformatted file operation conflict. 217 attempt to read past end of file 219 attempt to read/write past end of record For direct access, the record to be read/written exceeds the specified record length. 220 write after last internal record 221 syntax error in format string A runtime encoded format contains a lexical or syntax error. 222 unbalanced parentheses in format string 223 illegal P or T edit descriptor - value missing 224 illegal Hollerith or character string in format An unknown token type has been found in a format encoded at run-time. 225 lexical error -- unknown token type 226 unrecognized edit descriptor letter in format An unexpected Fortran edit descriptor (FED) was found in a runtime format item. 228 end of file reached without finding groupFortran Runtime Error Messages 381 229 end of file reached while processing group 230 scale factor out of range -128 to 127 Fortran P edit descriptor scale factor not within range of -128 to 127. 231 error on data conversion 233 too many constants to initialize group item 234 invalid edit descriptor An invalid edit descriptor has been found in a format statement. 235 edit descriptor does not match item type Data types specified by I/O list item and corresponding edit descriptor conflict. 236 formatted record longer than 2000 characters 237 quad precision type unsupported 238 tab value out of range A tab value of less than one has been specified. 239 entity name is not member of group 242 illegal operation on direct access file382 243 format parentheses nesting depth too great 244 syntax error - entity name expected 245 syntax error within group definition 246 infinite format scan for edit descriptor 248 illegal subscript or substring specification 249 error in format - illegal E, F, G or D descriptor 250 error in format - number missing after '.', '-', or '+' 251 illegal character in format string 252 operation attempted after end of file 253 attempt to read non-existent record (direct access) 254 illegal repeat count in formatAnachronisms Accepted 383 Appendix C. C++ Dialect Supported The PGC++ compiler accepts the C++ language as defined by The Annotated C++ Reference Manual (ARM) by Ellis and Stroustrup, Addison-Wesley, 1990, including templates, exceptions, and support for the anachronisms described in section 18 of the ARM. This is the same language defined by the language reference for ATT’s cfront version 3.0.1, with the addition of exceptions. PGC++ optionally accepts a number of features erroneously accepted by cfront version 2.1. Using the –b option, PGC++ accepts these features, which may never have been legal C++ but have found their way into some user’s code. Command-line options provide full support of many C++ variants, including strict standard conformance. PGC++ provides command line options that enable the user to specify whether anachronisms and/or cfront 2.1 compatibility features should be accepted. Refer to Section C.4 for details on features that are not part of the ARM but are part of the ANSI C++ working draft X3J16/WG21. Anachronisms Accepted The following anachronisms are accepted when anachronisms are enabled (when the +p option is not used): • overload is allowed in function declarations. It is accepted and ignored. • Definitions are not required for static data members that can be initialized using default initialization. This anachronism does not apply to static data members of template classes; they must always be defined. • The number of elements in an array may be specified in an array delete operation. The value is ignored. • A single operator++() and operator--() function can be used to overload both prefix and postfix operations. • The base class name may be omitted in a base class initializer if there is only one immediate base class.384 • Assignment to this in constructors and destructors is allowed. This is allowed only if anachronisms are enabled and the assignment to this configuration parameter is enabled. • A bound function pointer (a pointer to a member function for a given object) can be cast to a pointer to a function. • A nested class name may be used as a non-nested class name provided no other class of that name has been declared. This anachronism is not applied to template classes. • A reference to a non-const type may be initialized from a value of a different type. A temporary is created, it is initialized from the (converted) initial value, and the reference is set to the temporary. • A reference to a non-const class type may be initialized from an rvalue of the class type or a derived class thereof. No (additional) temporary is used. • A function with old-style parameter declarations is allowed and may participate in function overloading as though it was prototyped. Default argument promotion is not applied to parameter types of such functions when the check for compatibility is done, so that the following declares the overloading of two functions named f: int f(int); int f(x) char x; return x; • It will be noted that in C, this code is legal but has a different meaning: a tentative declaration of f is followed by its definition. • When --nonconst_ref_anachronism is enabled, a reference to a nonconst class can be bound to a class rvalue of the same type or a derived type thereof. struct A { A(int); A operator=(A&); A operator+(const A&); };main () { A b(1); b = A(1) + A(2); // Allowed as anachronism } New Language Features Accepted The following features not in the ARM but in the X3J16/WG21 Working paper are accepted:New Language Features Accepted 385 • The dependent statement of an if, while, do-while, or for is considered to be a scope, and the restriction on having such a dependent statement be a declaration is removed. • The expression tested in an if, while, do-while, or for, as the first operand of a ''?'' operator, or as an operand of the "&&", "::", or "!" operators may have a pointer-to-member type or a class type that can be converted to a pointer-to-member type in addition to the scalar cases permitted by the ARM. • Qualified names are allowed in elaborated type specifiers. • Use of a global-scope qualifier in member references of the form x.::A::B and p->::A::B. • The precedence of the third operand of the ``?'' operator is changed. • If control reaches the end of the main() routine, and main() has an integral return type, it is treated as if a return 0; statement were executed. • Pointers to arrays with unknown bounds as parameter types are diagnosed as errors. • A functional-notation cast of the form A() can be used even if A is a class without a (nontrivial) constructor. The temporary created gets the same default initialization to zero as a static object of the class type. • A cast can be used to select one out of a set of overloaded functions when taking the address of a function. • Template friend declarations and definitions are permitted in class definitions and class template definitions. • Type template parameters are permitted to have default arguments. • Function templates may have nontype template parameters. • A reference to const volatile cannot be bound to an rvalue. • Qualification conversions, such as conversion from T** to T const * const * are allowed. • Digraphs are recognized. • Operator keywords (e.g., and, bitand, etc.) are recognized. • Static data member declarations can be used to declare member constants.386 • wchar_t is recognized as a keyword and a distinct type. • bool is recognized. • RTTI (runtime type identification), including dynamic_cast and the typeid operator, are implemented. • Declarations in tested conditions (in if, switch, for, and while statements) are supported. • Array new and delete are implemented. • New-style casts (static_cast, reinterpret_cast, and const_cast) are implemented. • Definition of a nested class outside its enclosing class is allowed. • mutable is accepted on nonstatic data member declarations. • Namespaces are implemented, including using declarations and directives. Access declarations are broadened to match the corresponding using declarations. • Explicit instantiation of templates is implemented. • typename keyword is implemented. • explicit is accepted to declare Non-converting constructors . • The scope of a variable declared in a for-init-statement of a loop is the scope of the loop, not the surrounding scope. • Member templates are implemented. • The new specialization syntax (using "template<>") is implemented. • Cv-qualifiers are retained on rvalues (in particular, on function return values). • The distinction between trivial and nontrivial constructors has been implemented, as has the distinction between PODs and non-PODs with trivial constructors. • The linkage specification is treated as part of the function type (affecting function overloading and implicit conversions). • extern inline functions are supported, and the default linkage for inline functions is external. • A typedef name may be used in an explicit destructor call.The following language features are not accepted 387 • Placement delete is implemented. • An array allocated via a placement new can be deallocated via delete. • Covariant return types on overriding virtual functions are supported. • enum types are considered to be non-integral types. • Partial specialization of class templates is implemented. • Partial ordering of function templates is implemented. • Function declarations that match a function template are regarded as independent functions, not as “guiding declarations” that are instances of the template. • It is possible to overload operators using functions that take enum types and no class types. • Explicit specification of function template arguments is supported. • Unnamed template parameters are supported. • The new lookup rules for member references of the form x.A::PB and p->A::B are supported. • The notation :: template (and ->template, etc.) is supported. The following language features are not accepted The following feature of the ISO/IEC 14882:1998 C++ standard is not supported: • Exported templates are not implemented Extensions Accepted in Normal C++ Mode The following extensions are accepted in all modes (except when strict ANSI violations are diagnosed as errors, see the –A option): • A friend declaration for a class may omit the class keyword: class A { friend B; // Should be "friend class B" };388 • Constants of scalar type may be defined within classes: class A { const int size = 10; int a[size]; }; • In the declaration of a class member, a qualified name may be used: struct A{ int A::f(); // Should be int f(); } • The preprocessing symbol c_plusplus is defined in addition to the standard __cplusplus. • An assignment operator declared in a derived class with a parameter type matching one of its base classes is treated as a "default'' assignment operator --- that is, such a declaration blocks the implicit generation of a copy assignment operator. (This is cfront behavior that is known to be relied upon in at least one widely used library.) Here's an example: struct A { } ; struct B : public A { B& operator=(A&); }; • By default, as well as in cfront-compatibility mode, there will be no implicit declaration of B::operator=(const B&), whereas in strict-ANSI mode B::operator=(A&) is not a copy assignment operator and B::operator=(const B&) is implicitly declared. • Implicit type conversion between a pointer to an extern “C” function and a pointer to an extern “C++” function is permitted. Here’s an example: extern "C" void f();// f’s type has extern "C" linkage void (*pf) ()// pf points to an extern "C++" function = &f;// error unless implicit conv is allowed This extension is allowed in environments where C and C++ functions share the same calling conventions (though it is pointless unless DEFAULT_C_AND_CPP_FUNTION_TYPES_ARE_DISTINCT is TRUE). When cfront 2.1 Compatibility Mode 389 DEFAULT_IMPL_CONV_BETWEEN_C_AND_CPP_FUNCTION_PTRS_ALLOWED is set, it is enabled by default; it can also be enabled in cfront-compatibility mode or with command-line option – implicit_extern_c_type_conversion. It is disabled in strict-ANSI mode. cfront 2.1 Compatibility Mode The following extensions are accepted in cfront 2.1 compatibility mode in addition to the extensions listed in the following 2.1/3.0 section (i.e., these are things that were corrected in the 3.0 release of cfront): • The dependent statement of an if, while, do-while, or for is not considered to define a scope. The dependent statement may not be a declaration. Any objects constructed within the dependent statement are destroyed at exit from the dependent statement. • Implicit conversion from integral types to enumeration types is allowed. • A non-const member function may be called for a const object. A warning is issued. • A const void * value may be implicitly converted to a void * value, e.g., when passed as an argument. • When, in determining the level of argument match for overloading, a reference parameter is initialized from an argument that requires a non-class standard conversion, the conversion counts as a user-defined conversion. (This is an outright bug, which unfortunately happens to be exploited in the NIH class libraries.) • When a builtin operator is considered alongside overloaded operators in overload resolution, the match of an operand of a builtin type against the builtin type required by the builtin operator is considered a standard conversion in all cases (e.g., even when the type is exactly right without conversion). • A reference to a non-const type may be initialized from a value that is a const-qualified version of the same type, but only if the value is the result of selecting a member from a const class object or a pointer to such an object. • A cast to an array type is allowed; it is treated like a cast to a pointer to the array element type. A warning is issued.390 • When an array is selected from a class, the type qualifiers on the class object (if any) are not preserved in the selected array. (In the normal mode, any type qualifiers on the object are preserved in the element type of the resultant array.) • An identifier in a function is allowed to have the same name as a parameter of the function. A warning is issued. • An expression of type void may be supplied on the return statement in a function with a void return type. A warning is issued. • cfront has a bug that causes a global identifier to be found when a member of a class or one of its base classes should actually be found. This bug is not emulated in cfront compatibility mode. • A parameter of type "const void *'' is allowed on operator delete; it is treated as equivalent to "void *". • A period (".") may be used for qualification where "::" should be used. Only "::'' may be used as a global qualifier. Except for the global qualifier, the two kinds of qualifier operators may not be mixed in a given name (i.e., you may say A::B::C or A.B.C but not A::B.C or A.B::C). A period may not be used in a vacuous destructor reference nor in a qualifier that follows a template reference such as A::B. • cfront 2.1 does not correctly look up names in friend functions that are inside class definitions. In this example function f should refer to the functions and variables (e.g., f1 and a1) from the class declaration. Instead, the global definitions are used. int a1; int e1; void f1(); class A { int a1; void f1(); friend void f() { int i1 = a1; // cfront uses global a1 f1(); // cfront uses global f1 } }; • Only the innermost class scope is (incorrectly) skipped by cfront as illustrated in the following example.cfront 2.1/3.0 Compatibility Mode 391 int a1; int b1; struct A { static int a1; class B { static int b1; friend void f() { int i1 = a1; // cfront uses A::a1 int j1 = b1; // cfront uses global b1 } }; }; • operator= may be declared as a nonmember function. (This is flagged as an anachronism by cfront 2.1) • A type qualifier is allowed (but ignored) on the declaration of a constructor or destructor. For example: class A { A() const; // No error in cfront 2.1 mode }; cfront 2.1/3.0 Compatibility Mode The following extensions are accepted in both cfront 2.1 and cfront 3.0 compatibility mode (i.e., these are features or problems that exist in both cfront 2.1 and 3.0): • Type qualifiers on the this parameter may to be dropped in contexts such as this example: struct A { void f() const; }; void (A::*fp)() = &A::f; This is actually a safe operation. A pointer to a const function may be put into a pointer to nonconst, because a call using the pointer is permitted to modify the object and the function pointed to will actually not modify the object. The opposite assignment would not be safe.392 • Conversion operators specifying conversion to void are allowed. • A nonstandard friend declaration may introduce a new type. A friend declaration that omits the elaborated type specifier is allowed in default mode, but in cfront mode the declaration is also allowed to introduce a new type name. struct A { friend B; }; • The third operator of the ? operator is a conditional expression instead of an assignment expression as it is in the current X3J16/WG21 Working Paper. • A reference to a pointer type may be initialized from a pointer value without use of a temporary even when the reference pointer type has additional type qualifiers above those present in the pointer value. For example, int *p; const int *&r = p; // No temporary used • A reference may be initialized with a null.393 Appendix D. C/C++ MMX/SSE Inline Intrinsics Table D-1: MMX Intrinsics (mmintrin.h) _mm_empty _m_paddd _m_psllw _m_pand _m_empty _mm_add_si64 _mm_slli_pi16 _mm_andnot_si64 _mm_cvtsi32_si64 _mm_adds_pi8 _m_psllwi _m_pandn _m_from_int _m_paddsb _mm_sll_pi32 _mm_or_si64 _mm_cvtsi64x_si64 _mm_adds_pi16 _m_pslld _m_por _mm_set_pi64x _m_paddsw _mm_slli_pi32 _mm_xor_si64 _mm_cvtsi64_si32 _mm_adds_pu8 _m_pslldi _m_pxor _m_to_int _m_paddusb _mm_sll_si64 _mm_cmpeq_pi8 _mm_cvtsi64_si64x _mm_adds_pu16 _m_psllq _m_pcmpeqb _mm_packs_pi16 _m_paddusw _mm_slli_si64 _mm_cmpgt_pi8 _m_packsswb _mm_sub_pi8 _m_psllqi _m_pcmpgtb _mm_packs_pi32 _m_psubb _mm_sra_pi16 _mm_cmpeq_pi16 _m_packssdw _mm_sub_pi16 _m_psraw _m_pcmpeqw _mm_packs_pu16 _m_psubw _mm_srai_pi16 _mm_cmpgt_pi16 _m_packuswb _mm_sub_pi32 _m_psrawi _m_pcmpgtw _mm_unpackhi_pi8 _m_psubd _mm_sra_pi32 _mm_cmpeq_pi32 _m_punpckhbw _mm_sub_si64 _m_psrad _m_pcmpeqd _mm_unpackhi_pi16 _mm_subs_pi8 _mm_srai_pi32 _mm_cmpgt_pi32394 _m_punpckhwd _m_psubsb _m_psradi _m_pcmpgtd _mm_unpackhi_pi32 _mm_subs_pi16 _mm_srl_pi16 _mm_setzero_si64 _m_punpckhdq _m_psubsw _m_psrlw _mm_set_pi32 _mm_unpacklo_pi8 _mm_subs_pu8 _mm_srli_pi16 _mm_set_pi16 _m_punpcklbw _m_psubusb _m_psrlwi _mm_set_pi8 _mm_unpacklo_pi16 _mm_subs_pu16 _mm_srl_pi32 _mm_setr_pi32 _m_punpcklwd _m_psubusw _m_psrld _mm_setr_pi16 _mm_unpacklo_pi32 _mm_madd_pi16 _mm_srli_pi32 _mm_setr_pi8 _m_punpckldq _m_pmaddwd _m_psrldi _mm_set1_pi32 _mm_add_pi8 _mm_mulhi_pi16 _mm_srl_si64 _mm_set1_pi16 _m_paddb _m_pmulhw _m_psrlq _mm_set1_pi8 _mm_add_pi16 _mm_mullo_pi16 _mm_srli_si64 _m_pand _m_paddw _m_pmullw _m_psrlqi _mm_add_pi32 _mm_sll_pi16 _mm_and_si64395 Table D-2: SSE Intrinsics (xmmintrin.h) _mm_add_ss _mm_comige_ss _MM_SET_FLUSH_ZERO_MODE _mm_sub_ss _mm_comineq_ss _mm_load_ss _mm_mul_ss _mm_ucomieq_ss _mm_load1_ps _mm_div_ss _mm_ucomilt_ss _mm_load_ps1 _mm_sqrt_ss _mm_ucomile_ss _mm_load_ps _mm_rcp_ss _mm_ucomigt_ss _mm_loadu_ps _mm_rsqrt_ss _mm_ucomige_ss _mm_loadr_ps _mm_min_ss _mm_ucomineq_ss _mm_set_ss _mm_max_ss _mm_cvtss_si32 _mm_set1_ps _mm_add_ps _mm_cvt_ss2si _mm_set_ps1 _mm_sub_ps _mm_cvtss_si64x _mm_set_ps _mm_mul_ps _mm_cvtps_pi32 _mm_setr_ps _mm_div_ps _mm_cvt_ps2pi _mm_store_ss _mm_sqrt_ps _mm_cvttss_si32 _mm_store_ps _mm_rcp_ps _mm_cvtt_ss2si _mm_store1_ps _mm_rsqrt_ps _mm_cvttss_si64x _mm_store_ps1 _mm_min_ps _mm_cvttps_pi32 _mm_storeu_ps _mm_max_ps _mm_cvtt_ps2pi _mm_storer_ps _mm_and_ps _mm_cvtsi32_ss _mm_move_ss _mm_andnot_ps _mm_cvt_si2ss _mm_extract_pi16 _mm_or_ps _mm_cvtsi64x_ss _m_pextrw _mm_xor_ps _mm_cvtpi32_ps _mm_insert_pi16396 _mm_cmpeq_ss _mm_cvt_pi2ps _m_pinsrw _mm_cmplt_ss _mm_movelh_ps _mm_max_pi16 _mm_cmple_ss _mm_setzero_ps _m_pmaxsw _mm_cmpgt_ss _mm_cvtpi16_ps _mm_max_pu8 _mm_cmpge_ss _mm_cvtpu16_ps _m_pmaxub _mm_cmpneq_ss _mm_cvtpi8_ps _mm_min_pi16 _mm_cmpnlt_ss _mm_cvtpu8_ps _m_pminsw _mm_cmpnle_ss _mm_cvtpi32x2_ps _mm_min_pu8 _mm_cmpngt_ss _mm_movehl_ps _m_pminub _mm_cmpnge_ss _mm_cvtps_pi16 _mm_movemask_pi8 _mm_cmpord_ss _mm_cvtps_pi8 _m_pmovmskb _mm_cmpunord_ss _mm_shuffle_ps _mm_mulhi_pu16 _mm_cmpeq_ps _mm_unpackhi_ps _m_pmulhuw _mm_cmplt_ps _mm_unpacklo_ps _mm_shuffle_pi16 _mm_cmple_ps _mm_loadh_pi _m_pshufw _mm_cmpgt_ps _mm_storeh_pi _mm_maskmove_si64 _mm_cmpge_ps _mm_loadl_pi _m_maskmovq _mm_cmpneq_ps _mm_storel_pi _mm_avg_pu8 _mm_cmpnlt_ps _mm_movemask_ps _m_pavgb _mm_cmpnle_ps _mm_getcsr _mm_avg_pu16 _mm_cmpngt_ps _MM_GET_EXCEPTION_STATE _m_pavgw _mm_cmpnge_ps _MM_GET_EXCEPTION_MASK _mm_sad_pu8 _mm_cmpord_ps _MM_GET_ROUNDING_MODE _m_psadbw397 _mm_cmpunord_ps _MM_GET_FLUSH_ZERO_MODE _mm_prefetch _mm_comieq_ss _mm_setcsr _mm_stream_pi _mm_comilt_ss _MM_SET_EXCEPTION_STATE _mm_stream_ps _mm_comile_ss _MM_SET_EXCEPTION_MASK _mm_sfence _mm_comigt_ss _MM_SET_ROUNDING_MODE _mm_pause _MM_TRANSPOSE4_PS398 Table D-3: SSE2 Intrinsics (emmintrin.h) _mm_load_sd _mm_cmpge_sd _mm_cvtps_pd _mm_srl_epi32 _mm_load1_pd _mm_cmpneq_sd _mm_cvtsd_si32 _mm_srl_epi64 _mm_load_pd1 _mm_cmpnlt_sd _mm_cvtsd_si64x _mm_slli_epi16 _mm_load_pd _mm_cmpnle_sd _mm_cvttsd_si32 _mm_slli_epi32 _mm_loadu_pd _mm_cmpngt_sd _mm_cvttsd_si64x _mm_slli_epi64 _mm_loadr_pd _mm_cmpnge_sd _mm_cvtsd_ss _mm_srai_epi16 _mm_set_sd _mm_cmpord_sd _mm_cvtsi32_sd _mm_srai_epi32 _mm_set1_pd _mm_cmpunord_sd _mm_cvtsi64x_sd _mm_srli_epi16 _mm_set_pd1 _mm_comieq_sd _mm_cvtss_sd _mm_srli_epi32 _mm_set_pd _mm_comilt_sd _mm_unpackhi_pd _mm_srli_epi64 _mm_setr_pd _mm_comile_sd _mm_unpacklo_pd _mm_and_si128 _mm_setzero_pd _mm_comigt_sd _mm_loadh_pd _mm_andnot_si128 _mm_store_sd _mm_comige_sd _mm_storeh_pd _mm_or_si128 _mm_store_pd _mm_comineq_sd _mm_loadl_pd _mm_xor_si128 _mm_store1_pd _mm_ucomieq_sd _mm_storel_pd _mm_cmpeq_epi8 _mm_store_pd1 _mm_ucomilt_sd _mm_movemask_pd _mm_cmpeq_epi16 _mm_storeu_pd _mm_ucomile_sd _mm_packs_epi16 _mm_cmpeq_epi32 _mm_storer_pd _mm_ucomigt_sd _mm_packs_epi32 _mm_cmplt_epi8 _mm_move_sd _mm_ucomige_sd _mm_packus_epi16 _mm_cmplt_epi16 _mm_add_pd _mm_ucomineq_sd _mm_unpackhi_epi8 _mm_cmplt_epi32 _mm_add_sd _mm_load_si128 _mm_unpackhi_epi16 _mm_cmpgt_epi8 _mm_sub_pd _mm_loadu_si128 _mm_unpackhi_epi32 _mm_cmpgt_epi16399 _mm_sub_sd _mm_loadl_epi64 _mm_unpackhi_epi64 _mm_srl_epi16 _mm_mul_pd _mm_store_si128 _mm_unpacklo_epi8 _mm_cmpgt_epi32 _mm_mul_sd _mm_storeu_si128 _mm_unpacklo_epi16 _mm_max_epi16 _mm_div_pd _mm_storel_epi64 _mm_unpacklo_epi32 _mm_max_epu8 _mm_div_sd _mm_movepi64_pi64 _mm_unpacklo_epi64 _mm_min_epi16 _mm_sqrt_pd _mm_move_epi64 _mm_add_epi8 _mm_min_epu8 _mm_sqrt_sd _mm_setzero_si128 _mm_add_epi16 _mm_movemask_epi8 _mm_min_pd _mm_set_epi64 _mm_add_epi32 _mm_mulhi_epu16 _mm_min_sd _mm_set_epi32 _mm_add_epi64 _mm_maskmoveu_si128 _mm_max_pd _mm_set_epi64x _mm_adds_epi8 _mm_avg_epu8 _mm_max_sd _mm_set_epi16 _mm_adds_epi16 _mm_avg_epu16 _mm_and_pd _mm_set_epi8 _mm_adds_epu8 _mm_sad_epu8 _mm_andnot_pd _mm_set1_epi64 _mm_adds_epu16 _mm_stream_si32 _mm_or_pd _mm_set1_epi32 _mm_sub_epi8 _mm_stream_si128 _mm_xor_pd _mm_set1_epi64x _mm_sub_epi16 _mm_stream_pd _mm_cmpeq_pd _mm_set1_epi16 _mm_sub_epi32 _mm_movpi64_epi64 _mm_cmplt_pd _mm_set1_epi8 _mm_sub_epi64 _mm_lfence _mm_cmple_pd _mm_setr_epi64 _mm_subs_epi8 _mm_mfence _mm_cmpgt_pd _mm_setr_epi32 _mm_subs_epi16 _mm_cvtsi32_si128 _mm_cmpge_pd _mm_setr_epi16 _mm_subs_epu8 _mm_cvtsi64x_si128 _mm_cmpneq_pd _mm_setr_epi8 _mm_subs_epu16 _mm_cvtsi128_si32 _mm_cmpnlt_pd _mm_cvtepi32_pd _mm_madd_epi16 _mm_cvtsi128_si64x _mm_cmpnle_pd _mm_cvtepi32_ps _mm_mulhi_epi16 _mm_srli_si128400 Table D-4: SSE3 Intrinsics (pmmintrin.h) _mm_cmpngt_pd _mm_cvtpd_epi32 _mm_mullo_epi16 _mm_slli_si128 _mm_cmpnge_pd _mm_cvtpd_pi32 _mm_mul_su32 _mm_shuffle_pd _mm_cmpord_pd _mm_cvtpd_ps _mm_mul_epu32 _mm_shufflehi_epi16 _mm_cmpunord_pd _mm_cvttpd_epi32 _mm_sll_epi16 _mm_shufflelo_epi16 _mm_cmpeq_sd _mm_cvttpd_pi32 _mm_sll_epi32 _mm_shuffle_epi32 _mm_cmplt_sd _mm_cvtpi32_pd _mm_sll_epi64 _mm_extract_epi16 _mm_cmple_sd _mm_cvtps_epi32 _mm_sra_epi16 _mm_insert_epi16 _mm_cmpgt_sd _mm_cvttps_epi32 _mm_sra_epi32 _mm_addsub_ps _mm_moveldup_ps _mm_loaddup_pd _mm_mwait _mm_hadd_ps _mm_addsub_pd _mm_movedup_pd _mm_hsub_ps _mm_hadd_pd _mm_lddqu_si128 _mm_movehdup_ps _mm_hsub_pd _mm_monitor401 Index Numerics 64-Bit Programming 229 A Auto-parallelization 30 B Basic block 15 Bounds checking 96 C C++ Name Mangling 283 C++ Standard Template Library 207 C/C++ Builtin Functions 197 C/C++ Math Header File 197 Cache tiling failed cache tiling 99 with -Mvect 93 Command-line Options 3, 47, 65 -# 55 -### 55 -A 115 -b 116 -b3 116 -byteswapio 55 -C 56 -c 56 --cfront_2.1 117 --cfront_3.0 117 --create_pch 118 -D 57 -d 56 --diag_error 118 --diag_remark 118 --diag_suppress 118 --diag_warning 118 --display_error_number 118 -dryrun 58 -E 58 -F 58 -fast 59 -fastsse 59 -flagcheck 59 -flags 59 -fPIC 60 -fpic 59 -G 60 -g 60 -g77libs 61 -gopt 60 -help 61 -I 62 -i2, -i4 and -i8 63 --keeplnk 64 -Kflag 63 -L 64 -l 65 -M 119 -Manno 96 -Masmkeyword 83 -Mbackslash 80 -Mbounds 96 -Mbyteswapio 96 -Mcache_align 85 -Mchkfpstk 96 -Mchkptr 97 -Mchkstk 97 -mcmodel=medium 101, 230 -Mconcur 85 -Mcray 86 -MD 119 -Mdaz 74 -Mdclchk 81 -Mdefaultunit 81 -Mdepchk 87 -Mdlines 81 -Mdll 98 -Mdollar 81, 83 -Mdse 87 -Mdwarf1 74 -Mdwarf2 74 -Mdwarf3 74 -Mextend 81 -Mextract 78 -Mfcon 83 -Mfixed 81 -Mflushz 75 -Mfprelaxed 87 -Mfree 81 -Mfunc32 75 -Mgccbugs 98 -Mi4 87 -Minform 99 -Minline 79 -Miomutex 81 -Mipa 87 -Mkeepasm 99 -Mlarge_arrays 75, 231 -Mlfs 78 -Mlist 100 -Mlre 90 -Mmakedll 100 -Mneginfo 99 -Mnoasmkeyword 83 -Mnobackslash 80402 Index -Mnobounds 96 -Mnodaz 74 -Mnodclchk 81 -Mnodefaultunit 81 -Mnodepchk 87 -Mnodlines 81 -Mnodse 87 -Mnoflushz 75 -Mnofprelaxed 87 -Mnoframe 90 -Mnoi4 91 -Mnoiomutex 81 -Mnolarge_arrays 75 -Mnolist 100 -Mnolre 90 -Mnomain 75 -Mnontemporal 75 -Mnoonetrip 81 -Mnoopenmp 100 -Mnopgdllmain 100 -Mnoprefetch 91 -Mnor8 92 -Mnor8intrinsics 92 -Mnorecursive 76 -Mnoreentrant 76 -Mnoref_externals 76 -Mnosave 82 -Mnoscalarsse 93 -Mnosecond_underscore 76 -Mnosgimp 100 -Mnosignextend 77 -Mnosingle 83 -Mnosmart 93 -Mnostartup 78 -Mnostddef 78 -Mnostdlib 78 -Mnostride0 77 -Mnounixlogical 82 -Mnounroll 93 -Mnoupcase 82 -Mnovect 95 -Mnovintr 95 -module 102 -Monetrip 81 -mp 102 -Mpfi 91 -Mpfo 91 -Mprefetch 91 -Mpreprocess 100 -Mprof 75 -Mr8 92 -Mr8intrinsics 92 -Mrecursive 76 -Mreentrant 76 -Mref_externals 76 -Msafe_lastval 77 -Msafeptr 92 -Msave 81 -Mscalarsse 93 -Mschar 83 -Msecond_underscore 76 -Msignextend 77 -Msingle 83 -Msmart 93 -Mstandard 82 -Mstride0 77 -Muchar 84 -Munix 77 -Munixlogical 82 -Mupcase 82 -Mvarargs 77 -Mvect 93 -nfast 103 -O 103 -o 105, 108 --optk_allow_dollar_in_id_chars 119 -P 119 -pc 105 --pch 119 --pch_dir 119 --preinclude 120 -Q 108 -R 109 -r4 and -r8 109 -rc 109 -S 110 -shared 110 -show 110 -silent 110 -soname 111 -t 120 -time 111 -tp 111 -U 113 --use_pch 120 -V 114 -v 114 -W 114 -w 115 Commandline Options syntax 2 Compilation driver 1 Compilers Invoke at command level 1 PGC++ xx PGF77 xx PGF95 xx PGHPF xx cpp 5 D Data Types 215 bitfields 226 C++ class and object layout 224 C++ classes 223 C/C++ aggregate alignment 224 C/C++ scalar data types 220 C/C++ struct 223 C/C++ void 226 DEC structures 219 DEC Unions 219 F90 derived types 220 Fortran 215 internal padding 225 tail padding 225 Directives Fortran 4 optimization 175Index 403 Parallelization 131 prefetch 194 scope 183 E Environment variables 207 MP_BIND 208 MP_BLIST 208 MP_SPIN 208 MP_WARN 208 MPSTKZ 10, 13 NCPUS 209 NCPUS_MAX 209 NO_STOP_MESSAGE 209 OMP_STACK_SIZE 153, 173, 209 OMP_WAIT_POLICY 152, 173, 209 PGI 210 PGI_CONTINUE 210, 211 STATIC_RANDOM_SEED 210 TMPDIR 211 TZ 211 F Filename Conventions 4 extensions 4 Input Files 4 Output Files 6 Floating-point stack 105 Fortran directive summary 176 named common blocks 243 Fortran Parallelization Directives ATOMIC 146 DOACROSS 142 Function Inlining inlining and makefiles 126 inlining examples 127 inlining restrictions 128 I Inter-language Calling 239 %VAL 244 arguments and return values 244 array indices 246 C calling C++ 249 C++ calling C 248 C++ calling Fortran 251 character case conventions 241 character return values 244 compatible data types 241 Fortran calling C 246 Fortran calling C++ 250 underscores 241 L Language options 83 Libraries BLAS 207 FFTs 207 LAPACK 207 LIB3F 206 shared object files 198 Linux 10, 13 Header Files 10, 13 Parallelization 10, 13 Listing Files 96, 99, 100 Loop unrolling 22 Loops failed auto-parallelization 32 innermost 32 scalars 33 timing 32 N Command-line Options --llalign 118 --alternative_tokens 116 --bool 117 --exceptions 118 --pch_messages 120 --using_std 120 O OpenMP C/C++ Pragmas 155 atomic 167 barrier 164 critical 159 flush 168 for 161 master 160 ordered 167 parallel 156 parallel for 164 parallel sections 166 sections 165 single 161 threadprivate 168 OpenMP C/C++ Support Routines omp_destroy_lock() 171 omp_get _thread_num() 169 omp_get_dynamic() 170 omp_get_max_threads() 169 omp_get_nested() 171 omp_get_num_procs() 170 omp_get_num_threads() 169 omp_get_stack_size() 170 omp_get_wtick() 171 omp_get_wtime() 171 omp_in_parallel() 170 omp_init_lock() 171 omp_set_dynamic() 170 omp_set_lock() 171 omp_set_nested() 170 omp_set_num_threads() 169 omp_set_stack_size() 170 omp_test_lock() 172 omp_unset_lock() 172 OpenMP environment variables MPSTKZ 152, 173, 208 OMP_DYNAMIC 152, 172, 173 OMP_NESTED 152, 173 OMP_NUM_THREADS 151, 172 OMP_SCHEDULE 152 OpenMP Fortran Directives 131 ATOMIC 147 BARRIER 142 CRITICAL 136404 Index DO 139 FLUSH 147 MASTER 137 ORDERED 146 PARALLEL 132 PARALLEL DO 143 PARALLEL SECTIONS 145 PARALLEL WORKSHARE 144 SECTIONS 145 SINGLE 138 THREADPRIVATE 148 WORKSHARE 141 OpenMP Fortran Support Routines omp_destroy_lock() 151 omp_get_dynamic() 150 omp_get_max_threads() 149 omp_get_nested() 150 omp_get_num_procs() 149 omp_get_num_threads() 148 omp_get_stack_size() 149 omp_get_thread_num() 149 omp_get_wtick() 151 omp_get_wtime() 150 omp_in_parallel() 150 omp_init_lock() 151 omp_set_dynamic() 150 omp_set_lock() 151 omp_set_nested() 150 omp_set_num_threads() 149 omp_set_stack_size() 150 omp_test_lock() 151 omp_unset_lock() 151 Optimization 175 C/C++ pragmas 43, 187 C/C++ pragmas scope 191 cache tiling 93 Fortran directives 43, 175 Fortran directives scope 183 function inlining 16, 123 global optimization 16, 20 inline libraries 124 Inter-Procedural Analysis 16 IPA 16 local optimization 15 loop optimization 16 loop unrolling 16, 22 loops 90 -O 103 -O0 19 -O1 19 -O2 19 -O3 19 -Olevel 19 parallelization 16, 30 PFO 17 pointers 92 prefetching 91 profile-feedback (PFO) 42 Profile-Feedback Optimization 17 vectorization 16, 24 P Parallelization 30 auto-parallelization 30 failed auto-parallelization 32, 99 -Mconcur auto-parallelization 85 NCPUS environment variable 31 safe_lastval 34 user-directed 102 Parallelization Directives 131 Parallelization Pragmas 155 Pragmas C/C++ 4 optimization 187 scope 191 Prefetch directives 194 Preprocessor cpp 5 Fortran 5 R Run-time Environment 287 S Shared object files 198 T Timing CPU_CLOCK 44 execution 44 SYSTEM_CLOCK 44 Tools PGDBG xx PGPROF xx V Vectorization 24, 93 SSE instructions 95 W Win32 Calling Conventions C 253, 256 Default 253, 255 STDCALL 253, 255 symbol name construction 255 UNIX-style 253, 256 PAPI User’s Guide Version 3.5.0 PAPI USER’S GUIDE TABLE OF CONTENTS PAPI USER’S GUIDE.......................................................1 TABLE OF CONTENTS ........................ ............................1 PREFACE .................................................... ...................4 INTENDED AUDIENCE........... .................................................4 ORGANIZATION OF THIS DOCUMENT... ..................................4 INTRODUCTION TO PAPI................................................... ........................4 INSTALLING PAPI...................................................................... ...............4 C AND FORTRAN CALLING INTERFACES.................................................. .....4 EVENTS..................................................................... .............................4 PAPI COUNTER INTERFACES................................................. .....................4 PAPI TIMERS............................................................. ..............................5 PAPI SYSTEM INFORMATION............................................................. .........5 ADVANCED PAPI FEATURES............................................... ........................5 PAPI ERROR HANDLING...................................... ......................................5 PAPI MAILING LISTS............................................................................ .....5 APPENDICES.................................................... .......................................5 DOCUMENT CONVENTION...................................................... .5 INTRODUCTION TO PAPI .................................... ..........6 WHAT IS PAPI?.................................................... ..................6 BACKGROUND ....................................... ................................6 ARCHITECTURE............................................ ..........................7 INSTALLING PAPI ... .....................................................8 C AND FORTRAN CALLING INTERFACES . ......................9 EVENTS ........................................................... ............10 WHAT ARE EVENTS?........................................... ..................10 NATIVE EVENTS............................. ......................................10 WHAT ARE NATIVE EVENTS?.................................................. ..................10 PRESET EVENTS.......................... .........................................11 WHAT ARE PRESET EVENTS?........................................................... .........11 EVENT QUERY......................... .............................................12 EVENT TRANSLATION................................................... ........15 PAPI’S COUNTER INTERFACES ...................................17 HIGH-LEVEL API.................... ..............................................17 WHAT IS THE HIGH-LEVEL API?............................................................. ...17 - 1 -PAPI User’s Guide Version 3.5.0 INITIALIZING THE HIGH-LEVEL API................. .........................................17 EXECUTION RATE CALLS .................................... ....................................19 READING, ACCUMULATING, AND STOPPING COUNTERS.......................... .....20 LOW-LEVEL API................................................................. ...21 WHAT IS THE LOW-LEVEL API?.............................................................. ...21 INITIALIZATION OF THE LOW-LEVEL API...................... .............................22 EVENT SETS.............................. ...........................................24 WHAT ARE EVENT SETS?.................................................... .....................24 CREATING AN EVENT SET.......................................................... ..............24 ADDING EVENTS TO AN EVENT SET............................. .............................24 STARTING, READING, ADDING, AND STOPPING EVENTS IN AN EVENT SET....26 RESETTING EVENTS IN AN EVENT SET..................... .................................27 REMOVING EVENTS IN AN EVENT SET..................................... ..................28 EMPTYING AND DESTROYING AN EVENT SET.............................................. 29 THE STATE OF AN EVENT SET................................ ..................................30 GETTING AND SETTING OPTIONS.......................................... ...................33 SIMPLE CODE EXAMPLES............................................... .......40 HIGH-LEVEL API.......................................................................... ...........40 LOW-LEVEL API........................................................................... ...........41 PAPI TIMERS ................................... ...........................43 REAL TIME................................... ........................................43 VIRTUAL TIME.................... .................................................44 PAPI SYSTEM INFORMATION .....................................46 EXECUTABLE INFORMATION.. ..............................................46 HARDWARE INFORMATION.................... ..............................47 SUBSTRATE INFORMATION..................... .............................49 ADVANCED PAPI FEATURES ....................................... .50 MULTIPLEXING.............................. ......................................50 WHAT IS MULTIPLEXING?................................... .................50 USING PAPI WITH MULTIPLEXING....................................................... ......50 ISSUES OF MULTIPLEXING............................ ..........................................52 USING PAPI WITH PARALLEL PROGRAMS............................53 THREADS................................................................. .............................53 MPI..................................................................... .................................56 OVERFLOW........................................................................ ...58 WHAT IS AN OVERFLOW?............................................ ............................58 BEGINNING OVERFLOWS IN EVENT SETS............. .....................................58 STATISTICAL PROFILING.............................. .......................60 WHAT IS STATISTICAL PROFILING?............................... ...........................60 GENERATING A PC HISTOGRAM............................................................. ...61 DATA AND INSTRUCTION ADDRESS RESTRICTION...............63 Introduction.......................................................................................... .63 The PAPI Interface.......................................................................... ........63 - 2 -PAPI User’s Guide Version 3.5.0 Itanium Idiosyncrasies............................................... .............................64 Supporting Software...................................................... .........................65 The data_range.c Test Case................................................... ..................65 The papi_native_avail Utility................................... .................................67 PAPI ERROR HANDLING ......... ....................................69 ERROR CODES............................................... .......................69 CONVERTING ERROR CODES TO ERROR MESSAGES..................... ...............69 FURTHER INFORMATION ............................................72 PAPI HOME PAGE................................... ..............................72 PAPI MAILING LISTS............. ..............................................72 REPORTING BUGS................................. ...............................72 PAPI PROGRAMMER’S REFERENCE. ......................................72 TABLE OF PRESET EVENTS............................................... .....73 SUPPORTED PLATFORMS......................................... .............73 SUPPORTED TOOLS................................ ..............................73 HARDWARE REFERENCES....................... ..............................73 BIBLIOGRAPHY ........... ...............................................74 - 3 -PAPI User’s Guide Version 3.5.0 PREFACE............ .................................................... INTENDED AUDIENCE This document is intended to provide the PAPI user with a discussion of how to use the different components and functions of PAPI. The intended users are application developers and performance tool writers who need to access performance data to tune and model application performance. The user is expected to have some level of familiarity with either the C or Fortran programming language. ORGANIZATION OF THIS DOCUMENT INTRODUCTION TO PAPI This section provides an introduction to PAPI by describing the project, its motivation, and its architecture. INSTALLING PAPI This section provides an installation guide for PAPI. It states the necessary steps in order to install PAPI on the various supported operating systems. C AND FORTRAN CALLING INTERFACES This section states the header files in which function calls are defined and the form of the function calls for both the C and Fortran calling interfaces. Also, it provides a table that shows the relation between certain pseudo-types and Fortran variable types. EVENTS This section provides an explanation of events as well as an explanation of native and preset events. The preset query and translation functions are also discussed in this section. There are code examples using native events, preset query, and preset translation with the corresponding output. PAPI COUNTER INTERFACES This section discusses the high-level and low-level interfaces in detail. The initialization and functions of these interfaces are also discussed. Code examples along with the corresponding output are included as well. - 4 -PAPI User’s Guide Version 3.5.0 PAPI TIMERS This section explains the PAPI functions associated with obtaining real and virtual time from the platform’s timers. Code examples along with the corresponding output are included as well. PAPI SYSTEM INFORMATION This section explains the PAPI functions associated with obtaining hardware and executable information. Code examples along with the corresponding output are included as well. ADVANCED PAPI FEATURES This section discusses the advanced features of PAPI, which includes multiplexing, threads, MPI, overflows, and statistical profiling. The functions that are use to implement these features are also discussed. Code examples along with the corresponding output are included as well. PAPI ERROR HANDLING This section discusses the various negative error codes that are returned by the PAPI functions. A table with the names, values, and descriptions of the return codes are given as well as a discussion of the PAPI function that can be used to convert error codes to error messages along with a code example with the corresponding output. PAPI MAILING LISTS This section provides information on two PAPI mailing lists for the users to ask various questions about the project. APPENDICES These appendices provide various listings and tables, such as: a table of preset events and the platforms on which they are supported, a table of PAPI supported tools, more information on native events, multiplexing, overflow, and etc. DOCUMENT CONVENTION handle_error(1) A function that passes the argument of 1. The user should provide this function to handle errors. - 5 -PAPI User’s Guide Version 3.5.0 INTRODUCTION TO PAPI WHAT IS PAPI? PAPI is an acronym for Performance Application Programming Interface. The PAPI Project is being developed at the University of Tennessee’s Innovative Computing Laboratory in the Computer Science Department. This project was created to design, standardize, and implement a portable and efficient API (Application Programming Interface) to access the hardware performance counters found on most modern microprocessors. BACKGROUND Hardware counters exist on every major processor today, such as Intel Pentium, Core, IA-64, AMD Opteron,and IBM POWER series. These counters can provide performance tool developers with a basis for tool development and application developers with valuable information about sections of their code that can be improved. However, there are only a few APIs that allow access to these counters, and many of them are poorly documented, unstable, or unavailable. In addition, performance metrics may have different definitions and different programming interfaces on different platforms. These considerations motivated the development of the PAPI Project. Some goals of the PAPI Project are as follows: • To provide a solid foundation for cross platform performance analysis tools • To present a set of standard definitions for performance metrics on all platforms • To provide a standardize API among users, vendors, and academics • To be easy to use, well documented, and freely available - 6 -PAPI User’s Guide Version 3.5.0 ARCHITECTURE The Figure below shows the internal design of the PAPI architecture. In this figure, we can see the two layers of the architecture: The Portable Layer consists of the API (low level and high level) and machine independent support functions. The Machine Specific Layer defines and exports a machine independent interface to machine dependent functions and data structures. These functions are defined in the substrate layer, which uses kernel extensions, operating system calls, or assembly language to access the hardware performance counters. PAPI uses the most efficient and flexible of the three, depending on what is available. PAPI strives to provide a uniform environment across platforms. However, this is not always possible. Where hardware support for features, such as overflows and multiplexing is not supported, PAPI implements the features in software where possible. Also, processors do not support the same metrics, thus you can monitor different events depending on the processor in use. Therefore, the interface remains constant, but how it is implemented can vary. Throughout this guide, implementation decisions will be documented where it can make a difference to the user, such as overhead costs, sampling, and etc. - 7 - Portable Layer Machine Specific Layer PAPI Machine Dependent Substrate Kernel Extension Operating System Hardware Performance Counters PAPI Low Level PAPI High Level ToolsPAPI User’s Guide Version 3.5.0 INSTALLING PAPI On some of the systems that PAPI supports, you can install PAPI right out of the box without any additional setup. Others require drivers or patches to be installed first. Because installation instructions vary from platform to platform, please find your particular Operating System and hardware section in the /papi/INSTALL.txt file for current information on exactly how to install PAPI for your configuration. - 8 -PAPI User’s Guide Version 3.5.0 C AND FORTRAN CALLING INTERFACES PAPI is written in C. The function calls in the C interface are defined in the header file, papi.h and consist of the following form: PAPI_function_name(arg1, arg2,…) The function calls in the Fortran interface are defined in the header file, fpapi.h and consist of the following form: PAPIF_function_name(arg1, arg2, …, check) As you can see, the C function calls have equivalent Fortran function calls (PAPI_ becomes PAPIF_). This is generally true for most function calls, except for the functions that return C pointers to structures, such as PAPI_get_opt and PAPI_get_executable_info, which are either not implemented in the Fortran interface, or implemented with different calling semantics. In the function calls of the Fortran interface, the return code of the corresponding C routine is returned in the argument, check. For most architectures, the following relation holds between the pseudo-types listed and Fortran variable types: Pseudo-type Fortran type Description C_INT INTEGER Default Integer type C_FLOAT REAL Default Real type C_LONG_LONG INTEGER*8 Extended size integer C_STRING CHARACTER*(PAPI_MAX_STR_LEN) Fortran string C_INT FUNCTION EXTERNAL INTEGER FUNCTION Fortran function returning integer result Array arguments must be of sufficient size to hold the input/output from/to the subroutine for predictable behavior. The array length is indicated either by the accompanying argument or by internal PAPI definitions. Subroutines accepting C_STRING as an argument are on most implementations capable of reading the character string length as provided by Fortran. In these implementations, the string is truncated or space padded as necessary. For other implementations, the length of the character array is assumed to be of sufficient size. No character string longer than PAPI_MAX_STR_LEN is returned by the PAPIF interface. - 9 -PAPI User’s Guide Version 3.5.0 EVENTS WHAT ARE EVENTS? Events are occurrences of specific signals related to a processor’s function. Hardware performance counters exist as a small set of registers that count events, such as cache misses and floating point operations while the program executes on the processor. Monitoring these events facilitates correlation between the structure of source/object code and the efficiency of the mapping of that code to the underlying architecture. Each processor has a number of events that are native to that architecture. PAPI provides a software abstraction of these architecturedependent native events into a collection of preset events that are accessible through the PAPI interface. NATIVE EVENTS WHAT ARE NATIVE EVENTS? Native events comprise the set of all events that are countable by the CPU. There are generally far more native events available than can be mapped onto PAPI preset events. Even if no preset event is available that exposes a given native event, native events can still be accessed directly. To use native events effectively you should be very familiar with the particular platform in use. PAPI provides access to native events on all supported platforms through the low-level interface. Native events use the same interface as used when setting up a preset event, but since a PAPI preset event definition is not available for native events, a native event name must often be translated into an event code before it can be used. Native event codes and names are platform dependent, so native codes for one platform are not likely to work for any other platform. To determine the native events for your platform, see the native event lists for the various platforms in the processor architecture manual. Every attempt is made to keep native event names used by PAPI as similar as possible to those used in the vendor documentation. This is not always possible. The utility code util/papi_native_avail provides insight into the names of the native events for a specific platform. Native events are specified as arguments to the low-level function, PAPI_add_event in a manner similar to adding PAPI preset events. In the following code example, a native event name is converted to an event code and added to an eventset by using PAPI_add_event: - 10 -PAPI User’s Guide Version 3.5.0 #include #include main() { int retval, EventSet = PAPI_NULL; unsigned int native = 0x0; PAPI_event_info_t info; /* Initialize the library */ retval = PAPI_library_init(PAPI_VER_CURRENT); if (retval != PAPI_VER_CURRENT) { printf(“PAPI library init error!\n”); exit(1); } if (PAPI_create_eventset(&EventSet) != PAPI_OK) handle_error(1); /* Find the first available native event */ native = NATIVE_MASK | 0; if (PAPI_get_event_info(native, &info) != PAPI_OK) { if (PAPI_enum_event(&native, 0) != PAPI_OK) handle_error(1); } /* Add it to the eventset */ if (PAPI_add_event(EventSet, native) != PAPI_OK) handle_error(1); } For more code examples using native events, see ctests/native.c and util/native_avail.c in the papi source distribution. PRESET EVENTS WHAT ARE PRESET EVENTS? Preset events, also known as predefined events, are a common set of events deemed relevant and useful for application performance tuning. These events are typically found in many CPUs that provide performance counters and give access to the memory hierarchy, cache coherence protocol events, cycle and instruction counts, functional unit, and pipeline status. Furthermore, preset events are mappings from symbolic names (PAPI preset name) to machine specific definitions (native countable events) for a particular hardware resource. For example, Total Cycles (in user mode) is PAPI_TOT_CYC. Also, PAPI supports presets that may be derived from the underlying hardware metrics. For example, Total L1 Cache Misses (PAPI_L1_TCM) might be the sum of L1 Data Misses and L1 Instruction Misses on a given platform. A preset can be either directly available as a single counter, derived using a combination of counters, or unavailable on any particular platform. - 11 -PAPI User’s Guide Version 3.5.0 The PAPI library names approximately 100 preset events, which are defined in the header file, papiStdEventDefs.h. For a given platform, a subset of these preset events can be counted though either a simple high-level programming interface or a more complete C or Fortran low-level interface. For a representative list of all the preset events on some supported platforms, visit the PAPI web page: http://icl.cs.utk.edu/projects/papi/presets.html. Note that processors and software are revised over time, and this list may not be up to date. To determine exactly which preset events are available on a specific platform, run util/papi_avail.c in the papi source distribution. The exact semantics of an event counter are platform dependent. PAPI preset names are mapped onto available events so as to map as many countable events as possible on different platforms. Due to hardware implementation differences, it is not necessarily feasible to directly compare the counts of a particular PAPI preset event obtained on different hardware platforms. EVENT QUERY The following low-level functions can be called to query about the existence of a preset or native event (in other words, if the hardware supports that certain event), and to get details about that event: C: PAPI_query_event(EventCode) PAPI_get_event_info(EventCode, &info) PAPI_enum_event(&EventCode, modifier) Fortran: PAPIF_query_event(EventCode, check) PAPIF_get_event_info(EventCode, symbol, longDescr, shortDescr, count, note, flags, check) PAPIF_enum_event(&EventCode, modifier, check) ARGUMENTS EventCode -- a defined event, such as PAPI_TOT_INS. symbol -- the event symbol, or name, such as the preset name, PAPI_BR_CN. longDescr -- a descriptive string for the event of length less than PAPI_MAX_STR_LEN. - 12 -PAPI User’s Guide Version 3.5.0 shortDescr -- a short descriptive string for the event of length less than 18 characters. count -- zero if the event CANNOT be counted. note -- additional text information about an event (if available). flags -- provides additional information about an event, e.g., PAPI_DERIVED for an event derived from 2 or more other events. modifier -- modifies the search criteria; for preset events, returns all events or only available events; for native events, the definition is platform dependent. PAPI_query_event asks the PAPI library if the preset or native event can be counted on this architecture. If the event CAN be counted, the function returns PAPI_OK. If the event CANNOT be counted, the function returns an error code. PAPI_get_event_info asks the PAPI library for a copy of an event descriptor. This descriptor can then be used to investigate the details about the event. In Fortran, the individual fields in the descriptor are returned as parameters. PAPI_enum_event asks the PAPI library to return an event code for the next sequential event based on the current event code and the modifier. This function can be used to enumerate all preset or native events on any platform. See util/papi_avail.c or util/papi_native_avail.c for details. EXAMPLE: #include #include main() { int EventSet = PAPI_NULL; unsigned int native = 0x0; int retval, i; PAPI_preset_info_t info; PAPI_preset_info_t *infostructs; /* Initialize the library */ retval = PAPI_library_init(PAPI_VER_CURRENT); if (retval != PAPI_VER_CURRENT) { fprintf(stderr,"PAPI library init error!\n"); exit(1); } /* Check to see if the preset, PAPI_TOT_INS, exists */ if (PAPI_query_event (PAPI_TOT_INS) != PAPI_OK) { fprintf (stderr,"No instruction counter? How lame.\n"); exit(1); - 13 -PAPI User’s Guide Version 3.5.0 } /* Get details about the preset, PAPI_TOT_INS */ if (PAPI_get_event_info(PAPI_TOT_INS,&info) != PAPI_OK) { fprintf (stderr,"No instruction counter? How lame.\n"); exit(1); } if (info.count>0) printf ("This event is available on this hardware.\n"); if (info.flags & PAPI_DERIVED) printf ("This event is a derived event on this hardware.\n"); /* Count the number of available preset events between PAPI_TOT_INS and the end of the preset list */ retval = 0; i = PAPI_TOT_INS; while (PAPI_enum_event(&i, TRUE) == PAPI_OK) { retval++; } OUTPUT (if PAPI_TOT_INS is available on your system): This event is available on this hardware. In the above code example, PAPI_query_event is used to see if a preset (PAPI_TOT_INS) exists, PAPI_get_event_info is used to query details about the event, and PAPI_enum_event is used to count the number of events in the preset list after this preset. On success, all three of these functions return PAPI_OK, and on error, a non-zero error code is returned. - 14 -PAPI User’s Guide Version 3.5.0 EVENT TRANSLATION A preset or native event can be referenced by name or by event code. Most PAPI functions require an event code, while most user input and output is in terms of names. Two low-level functions are provided to translate between these formats: C: PAPI_event_name_to_code(EventName, EventCode) PAPI_event_code_to_name(EventCode, EventName) Fortran: PAPIF_event_name_to_code(EventName, EventCode, check) PAPIF_event_code_to_name(EventCode, EventName, check) ARGUMENTS EventCode -- a preset or native event of integer type, such as PAPI_TOT_INS. EventName -- the event name string, such as the preset name, “PAPI_BR_CN”. Note that the preset does not actually have to be available on a given platform to call these functions. Native event names are platform specific and where feasible match those given in the vendor documentation. PAPI_event_name_to_code is used to translate an ASCII PAPI preset or native event name into an integer PAPI event code. PAPI_event_code_to_name is used to translate an integer PAPI event code into an ASCII PAPI preset or native event name. Using PAPI_event_code_to_name in conjunction with PAPI_enum_event is a good way to explore the names of native events on a specific platform, as shown in the following code example: #include #include main() { int EventCode, retval; char EventCodeStr[PAPI_MAX_STR_LEN]; /* Initialize the library */ retval = PAPI_library_init(PAPI_VER_CURRENT); - 15 -PAPI User’s Guide Version 3.5.0 if (retval != PAPI_VER_CURRENT) { fprintf(stderr, “PAPI library init error!\n”); exit(1); } EventCode = 0 | NATIVE_MASK; do { /* Translate the integer code to a string */ if (PAPI_event_code_to_name(EventCode, EventCodeStr) == PAPI_OK) /* Print all the native events for this platform */ printf("Name: %s\nCode: %x\n", EventCodeStr, EventCode); } while (PAPI_enum_event(&EventCode, 0) == PAPI_OK); } OUTPUT: Name: DATA_MEM_REFS Code: 40000000 Name: DCU_LINES_IN Code: 40000001 Name: DCU_M_LINES_IN Code: 40000002 . . Name: SEG_REG_RENAMES_TOT Code: 40000078 Name: RET_SEG_RENAMES Code: 40000079 The output will vary depending on the platform. This was generated on an Intel Pentium III processor. On success, all the functions return PAPI_OK and on error, a non-zero error code is returned. - 16 -PAPI User’s Guide Version 3.5.0 PAPI’S COUNTER INTERFACES HIGH-LEVEL API WHAT IS THE HIGH-LEVEL API? The high-level API (Application Programming Interface) provides the ability to start, stop, and read the counters for a specified list of events. It is meant for programmers wanting simple event measurements using only PAPI preset events. Some of the benefits of using the high-level API rather than the low-level API are that it is easier to use and requires less setup (additional calls). This ease of use comes with somewhat higher overhead and loss of flexibility. It should also be noted that the high-level API can be used in conjunction with the low-level API and in fact does call the low-level API. However, the high-level API by itself is only able to access those events countable simultaneously by the underlying hardware. There are eight functions that represent the high-level API that allow the user to access and count specific hardware events. Note that these functions can be accessed from both C and Fortran. For a code example of using the high-level interface, see Simple Code Examples: High Level API or ctests/high-level.c in the PAPI source distribution. For full details on the calling semantics of these functions, please refer to the PAPI Programmer’s Reference. INITIALIZING THE HIGH-LEVEL API The PAPI library is initialized implicitly by several high-level API calls. In addition to the three rate calls discussed later, either of the following two functions also implicitly initializes the library: C: PAPI_num_counters() PAPI_start_counters(*events, array_length) Fortran: PAPIF_num_counters(check) PAPIF_start_counters(*events, array_length, check) - 17 -PAPI User’s Guide Version 3.5.0 ARGUMENTS *events -- an array of codes for events such as PAPI_INT_INS or a native event code. array_length -- the number of items in the events array. PAPI_num_counters returns the optimal length of the values array for high-level functions. This value corresponds to the number of hardware counters supported by the current substrate. PAPI_num_counters initializes the PAPI library using PAPI_library_init if necessary. PAPI_start_counters initializes the PAPI library (if necessary) and starts counting the events named in the events array. This function implicitly stops and initializes any counters running as a result of a previous call to PAPI_start_counters. It is the user’s responsibility to choose events that can be counted simultaneously by reading the vendor’s documentation. The size of array_length should be no larger than the value returned by PAPI_num_counters. In the following code example, PAPI_num_counters is used to initialize the library and to get the number of hardware counters available on the system. Also, PAPI_start_counters is used to start counting events: #include main() { int Events[2] = { PAPI_TOT_CYC, PAPI_TOT_INS }; int num_hwcntrs = 0; /* Initialize the PAPI library and get the number of counters available */ if ((num_hwcntrs = PAPI_num_counters()) <= PAPI_OK) handle_error(1); printf("This system has %d available counters.", num_hwcntrs); if (num_hwcntrs > 2) num_hwcntrs = 2; /* Start counting events */ if (PAPI_start_counters(Events, num_hwcntrs) != PAPI_OK) handle_error(1); } POSSIBLE OUTPUT (varies on different systems): This system has 4 available counters. - 18 -PAPI User’s Guide Version 3.5.0 On success, PAPI_num_counters returns the number of hardware counters available on the system and on error, a non-zero error code is returned. Optionally, the PAPI library can be initialized explicitly by using PAPI_library_init. This can be useful if you wish to call PAPI low-level API functions before using the high-level functions. EXECUTION RATE CALLS Three PAPI high-level functions are available to measure floating point or total instruction rates. These three calls are shown below: C: PAPI_flips(*real_time, *proc_time, *flpins, *mflips) PAPI_flops(*real_time, *proc_time, *flpins, *mflops) PAPI_ipc(*real_time, *proc_time, *ins, *ipc) Fortran: PAPIF_flips(real_time, proc_time, flpins, mflips, check) PAPIF_flops(real_time, proc_time, flpins, mflops, check) PAPIF_ipc(real_time, proc_time, ins, ipc, check) ARGUMENTS *real_time -- the total real (wallclock) time since the first rate call. *proc_time -- the total process time since the first rate call. *flpins -- the total floating point instructions since the first rate call. *mflips, *mflops – Millions of floating point operations or instructions per second achieved since the latest rate call. *ins -- the total instructions executed since the first PAPI_ipc call. *ipc – instructions per cycle achieved since the latest PAPI_ipc call. The first execution rate call initializes the PAPI library if needed, sets up the counters to monitor either PAPI_FP_INS, PAPI_FP_OPS or PAPI_TOT_INS (depending on the call), and PAPI_TOT_CYC events, and starts the counters. Subsequent calls to the same rate function will read the counters and return total real time, total process time, total instructions or operations, and the appropriate rate of execution since the last call. A call to PAPI_stop_counters will reinitialize all values to 0. Sequential calls to different execution rate functions will return an error. Note that on many platforms there may be subtle differences between floating point instructions and operations. Instructions are typically those execution elements most directly measured by the hardware counters. They may include floating point load and store instructions, and may count instructions such as FMA as one, even - 19 -PAPI User’s Guide Version 3.5.0 though two floating point operations have occurred. Consult the hardware documentation for your system for more details. Operations represent a derived value where an attempt is made, when possible, to more closely map to the theoretical definition of a floating point event. On success, the rate calls return PAPI_OK and on error, a non-zero error code is returned. For a code example, see ctest/flops.c or ctest/ipc.c in the papi source distribution. READING, ACCUMULATING, AND STOPPING COUNTERS Counters can be read, accumulated, and stopped by calling the following high-level functions, respectively: C: PAPI_read_counters(*values, array_length) PAPI_accum_counters(*values, array_length) PAPI_stop_counters(*values, array_length) Fortran: PAPIF_read_counters(*values, array_length, check) PAPIF_accum_counters(*values, array_length, check) PAPIF_stop_counters(*values, array_length, check) ARGUMENTS *values -- an array where to put the counter values. array_length -- the number of items in the *values array. PAPI_read_counters, PAPI_accum_counters and PAPI_stop_counters all capture the values of the currently running counters into the array, values. Each of these functions behaves somewhat differently. PAPI_read_counters copies the current counts into the elements of the values array, resets the counters to zero, and leaves the counters running. PAPI_accum_counters adds the current counts into the elements of the values array and resets the counters to zero, leaving the counters running. Care should be exercised not to mix calls to PAPI_accum_counters with calls to the execution rate functions. Such intermixing is likely to produce unexpected results. - 20 -PAPI User’s Guide Version 3.5.0 PAPI_stop_counters stops the counters and copies the current counts into the elements of the values array. This call can also be used to reset the rate functions if used with a NULL pointer to the values array. In the following code example, PAPI_read_counters and PAPI_stop_counters are used to copy and stop event counters in an array, respectively: #include #define NUM_EVENTS 2 main() { int Events[NUM_EVENTS] = {PAPI_TOT_INS, PAPI_TOT_CYC}; long_long values[NUM_EVENTS]; /* Start counting events */ if (PAPI_start_counters(Events, NUM_EVENTS) != PAPI_OK) handle_error(1); /* Do some computation here*/ /* Read the counters */ if (PAPI_read_counters(values, NUM_EVENTS) != PAPI_OK) handle_error(1); /* Do some computation here */ /* Stop counting events */ if (PAPI_stop_counters(values, NUM_EVENTS) != PAPI_OK) handle_error(1); } On success, all of these functions return PAPI_OK and on error, a non-zero error code is returned. LOW-LEVEL API WHAT IS THE LOW-LEVEL API? The low-level API (Application Programming Interface) manages hardware events in user-defined groups called Event Sets. It is meant for experienced application programmers and tool developers wanting fine-grained measurement and control of the PAPI interface. Unlike the high-level interface, it allows both PAPI preset and native events. Other features of the low-level API are the ability to obtain information about the executable and the hardware as well as to set options for multiplexing and overflow handling. Some of the benefits of using the low-level API rather than the high-level API are that it increases efficiency and functionality. - 21 -PAPI User’s Guide Version 3.5.0 It should also be noted that the low-level interface could be used in conjunction with the high-level interface, as long as attention is paid to insure that the PAPI library is initialized prior to the first low-level PAPI call. The low-level API is only as powerful as the substrate upon which it is built. Thus, some features may not be available on every platform. The converse may also be true, that more advanced features may be available on every platform and defined in the header file. Therefore, the user is encouraged to read the documentation for each platform carefully. There are approximately 50 functions that represent the low-level API. For a code example of using the low-level interface, see Simple Code Examples: Low-Level API or ctests/low_level.c in the PAPI source distribution. Note that most functions are implemented in both C and Fortran, but some are implemented in only one of these two languages. For full details on the calling semantics of these functions, please refer to the PAPI Programmer’s Reference. INITIALIZATION OF THE LOW-LEVEL API The PAPI library must be initialized before it can be used. It can be initialized explicitly by calling the following low-level function: C: PAPI_library_init(version) Fortran: PAPIF_library_init(check) ARGUMENT version -- upon initialization, PAPI checks the argument against the internal value of PAPI_VER_CURRENT when the library was compiled. This guards against portability problems when updating the PAPI shared libraries on your system. Note that this function must be called before calling any other low-level PAPI function. On success, this function returns PAPI_VER_CURRENT. On error, a positive return code other than PAPI_VER_CURRENT indicates a library version mismatch and a negative return code indicates an initialization error. Beginning with PAPI 3.0, there are a number of options for examining the current version number of PAPI: - 22 -PAPI User’s Guide Version 3.5.0 • PAPI_VERSION produces an integer containing the complete current version including MAJOR, MINOR, and REVISION components. Typically the REVISION component changes with bug fixes or minor enhancements, the MINOR component changes with feature additions or API changes, and the MAJOR component changes with significant API structural changes. • PAPI_VER_CURRENT contains the MAJOR and MINOR components and is useful for determining library compatibility changes. • PAPI_VERSION_MAJOR, • PAPI_VERSION_MINOR, • PAPI_VERSION_REVISION are macros that extract specified component from the version number. The following is a code example of using PAPI_library_init to initialize the PAPI library: #include #include int retval; main() { /* Initialize the PAPI library */ retval = PAPI_library_init(PAPI_VER_CURRENT); if (retval != PAPI_VER_CURRENT && retval > 0) { fprintf(stderr,"PAPI library version mismatch!\n"); exit(1); } if (retval < 0) { fprintf(stderr, “Initialization error!\n”); exit(1); } fprintf(stdout, “PAPI Version Number\n”); fprintf(stdout, “MAJOR: %d\n”, PAPI_MAJOR(retval)); fprintf(stdout, “MINOR: %d\n”, PAPI_MINOR(retval)); fprintf(stdout, “REVISION: %d\n”, PAPI_REVISION(retval)); } OUTPUT FOR PAPI VERSION 3.5.0 PAPI Version Number MAJOR: 3 MINOR: 5 REVISION: 0 - 23 -PAPI User’s Guide Version 3.5.0 EVENT SETS WHAT ARE EVENT SETS? Event Sets are user-defined groups of hardware events (preset or native), which are used in conjunction with one another to provide meaningful information. The user specifies the events to be added to an Event Set, and other attributes, such as: the counting domain (user or kernel), whether or not the events in the Event Set are to be multiplexed, and whether the Event Set is to be used for overflow or profiling. Other settings for the Event Set are maintained by PAPI, such as: what low-level hardware registers to use, the most recently read counter values, and the state of the Event Set (running/not running). Event Sets provide an effective abstraction for the organization of information associated with counting hardware events. The PAPI library manages the memory for Event Sets with a user interface through integer handles to simplify calling conventions. The user is free to allocate and use any number of them provided the substrate can provide the required resources. Only one Event Set can be in active use at any time in a given thread or process. CREATING AN EVENT SET An event set can be created by calling the following the low-level function: C: PAPI_create_eventset (*EventSet) Fortran: PAPIF_create_eventset(EventSet, check) ARGUMENT EventSet -- Address of an integer location to store the new EventSet handle. Once it has been created, the user may add hardware events to the EventSet by calling PAPI_add_event or PAPI_add_events. On success, this function returns PAPI_OK. On error, a non-zero error code is returned. For a code example using this function, see the next section. ADDING EVENTS TO AN EVENT SET Hardware events can be added to an event set by calling the following the low-level functions: - 24 -PAPI User’s Guide Version 3.5.0 C: PAPI_add_event(EventSet, EventCode) PAPI_add_events(EventSet, *EventCode, number) Fortran: PAPIF_add_event(EventSet, EventCode, check) PAPIF_add_events(EventSet, EventCode, number, check) ARGUMENTS EventSet -- an integer handle for a PAPI Event Set as created by PAPI_create_eventset. EventCode -- a defined event such as PAPI_TOT_INS. *EventCode – address of an array of defined events. number -- an integer indicating the number of events in the array *EventCode. PAPI_add_event adds a single hardware event to a PAPI event set. PAPI_add_events does the same as PAPI_add_event, but for an array of hardware event codes. In the following code example, the preset event, PAPI_TOT_INS is added to an event set: #include #include main() { int EventSet = PAPI_NULL; int retval; /* Initialize the PAPI library */ retval = PAPI_library_init(PAPI_VER_CURRENT); if (retval != PAPI_VER_CURRENT) { fprintf(stderr, "PAPI library init error!\n"); exit(1); } /* Create an EventSet */ if (PAPI_create_eventset(&EventSet) != PAPI_OK) handle_error(1); /* Add Total Instructions Executed to our EventSet */ if (PAPI_add_event(EventSet, PAPI_TOT_INS) != PAPI_OK) handle_error(1); } - 25 -PAPI User’s Guide Version 3.5.0 On success, both of these functions return PAPI_OK and on error, a non-zero error code is returned. STARTING, READING, ADDING, AND STOPPING EVENTS IN AN EVENT SET Hardware events in an event set can be started, read, added, and stopped by calling the following low-level functions, respectively: C: PAPI_start(EventSet) PAPI_read(EventSet, *values) PAPI_accum(EventSet, *values) PAPI_stop(EventSet, *values) Fortran: PAPIF_start(EventSet, check) PAPIF_read(EventSet, values, check) PAPIF_accum(EventSet, values, check) PAPIF_stop(EventSet, values, check) ARGUMENTS EventSet -- an integer handle for a PAPI Event Set as created by PAPI_create_eventset. *values -- an array to hold the counter values of the counting events. PAPI_start starts the counting events in a previously defined event set. PAPI_read reads (copies) the counters of the indicated event set into the array, values. The counters are left counting after the read without resetting. PAPI_accum adds the counters of the indicated event set into the array, values. The counters are reset and left counting after the call of this function. PAPI_stop stops the counting events in a previously defined event set and returns the current events. The following is a code example of using PAPI_start to start the counting of events in an event set, PAPI_read to read the counters of the same event set into the array values, and PAPI_stop to stop the counting of events in the event set: #include - 26 -PAPI User’s Guide Version 3.5.0 #include main() { int retval, EventSet = PAPI_NULL; long_long values[1]; /* Initialize the PAPI library */ retval = PAPI_library_init(PAPI_VER_CURRENT); if (retval != PAPI_VER_CURRENT) { fprintf(stderr, "PAPI library init error!\n"); exit(1); } /* Create the Event Set */ if (PAPI_create_eventset(&EventSet) != PAPI_OK) handle_error(1); /* Add Total Instructions Executed to our EventSet */ if (PAPI_add_event(EventSet, PAPI_TOT_INS) != PAPI_OK) handle_error(1); /* Start counting */ if (PAPI_start(EventSet) != PAPI_OK) handle_error(1); /* Do some computation here */ if (PAPI_read(EventSet, values) != PAPI_OK) handle_error(1); /* Do some computation here */ if (PAPI_stop(EventSet, values) != PAPI_OK) handle_error(1); } On success, these functions return PAPI_OK and on error, a non-zero error code is returned. RESETTING EVENTS IN AN EVENT SET The hardware event counts in an event set can be reset to zero by calling the following low-level function: C: PAPI_reset(EventSet) Fortran: - 27 -PAPI User’s Guide Version 3.5.0 PAPI_reset(EventSet, check) ARGUMENT EventSet -- an integer handle for a PAPI event set as created by PAPI_create_eventset. For example, the EventSet in the code example of the previous section could have been reset to zero by adding the following lines: if (PAPI_reset(EventSet) != PAPI_OK) handle_error(1); On success, this function returns PAPI_OK and on error, a non-zero error code is returned. REMOVING EVENTS IN AN EVENT SET A hardware event and an array of hardware events can be removed from an event set by calling the following low-level functions, respectively: C: PAPI_remove_event(EventSet, EventCode) PAPI_remove_events(EventSet, EventCode, number) Fortran: PAPIF_remove_event(EventSet, EventCode, check) PAPIF_remove_events(EventSet, EventCode, number, check) ARGUMENTS EventSet -- an integer handle for a PAPI event set as created by PAPI_create_eventset. EventCode -- a defined event such as PAPI_TOT_INS or a native event. *EventCode -- an array of defined events. number -- an integer indicating the number of events in the array *EventCode. PAPI_remove_event removes a single hardware event from a PAPI event set. PAPI_remove_events, does the same as PAPI_remove_event, but for an array of hardware event codes. In the following code example, PAPI_remove_event is used to remove the event, PAPI_TOT_INS, from an event set: - 28 -PAPI User’s Guide Version 3.5.0 #include #include main() { int retval, EventSet = PAPI_NULL; /* Initialize the PAPI library */ retval = PAPI_library_init(PAPI_VER_CURRENT); if (retval != PAPI_VER_CURRENT) { fprintf(stderr, "PAPI library init error!\n"); exit(1); } /* Create an EventSet */ if (PAPI_create_eventset(&EventSet) != PAPI_OK) handle_error(1); /* Add Total Instructions Executed to our EventSet */ if (PAPI_add_event(EventSet, PAPI_TOT_INS) != PAPI_OK) handle_error(1); /* Remove event */ if (PAPI_remove_event(EventSet, PAPI_TOT_INS) != PAPI_OK) handle_error(1); } On success, these functions return PAPI_OK and on error, a non-zero error code is returned. EMPTYING AND DESTROYING AN EVENT SET All the events in an event set can be emptied and destroyed by calling the following low-level functions, respectively: C: PAPI_cleanup_eventset(EventSet) PAPI_destroy_eventset(EventSet) Fortran: PAPIF_cleanup_eventset(EventSet, check) PAPIF_destroy_eventset(EventSet, check) ARGUMENT EventSet -- an integer handle for a PAPI event set as created by PAPI_create_eventset. - 29 -PAPI User’s Guide Version 3.5.0 Note that the event set must be empty in order to use PAPI_destroy_eventset. In the following code example, PAPI_cleanup_eventset is used to empty all the events from an event set and PAPI_remove_eventset is used to deallocate the memory associated with the empty event set: #include #include main() { int retval, EventSet = PAPI_NULL; /* Initialize the PAPI library */ retval = PAPI_library_init(PAPI_VER_CURRENT); if (retval != PAPI_VER_CURRENT) { fprintf(stderr, "PAPI library init error!\n"); exit(1); } /* Create the EventSet */ if (PAPI_create_eventset(&EventSet) != PAPI_OK) handle_error(1); /* Add Total Instructions Executed to our EventSet */ if (PAPI_add_event(&EventSet, PAPI_TOT_INS) != PAPI_OK) handle_error(1); /* Remove all events in the eventset */ if (PAPI_cleanup_eventset(&EventSet) != PAPI_OK) handle_error(1); /* Free all memory and data structures, EventSet must be empty. */ if (PAPI_destroy_eventset(&EventSet) != PAPI_OK) handle_error(1); } On success, these functions return PAPI_OK and on error, a non-zero error code is returned. THE STATE OF AN EVENT SET The counting state of an Event Set can be obtained by calling the following low-level function: C: PAPI_state(EventSet, *status) Fortran: - 30 -PAPI User’s Guide Version 3.5.0 PAPIF_state(EventSet, status, check) ARGUMENTS EventSet -- an integer handle for a PAPI event set as created by PAPI_create_eventset. status -- an integer containing a Boolean combination of one or more of the following nonzero constants as defined in the PAPI header file, papi.h: PAPI_STOPPED EventSet is stopped PAPI_RUNNING EventSet is running PAPI_PAUSED EventSet temporarily disabled by the library PAPI_NOT_INIT EventSet defined, but not initialized PAPI_OVERFLOWING EventSet has overflow enabled PAPI_PROFILING EventSet has profiling enabled PAPI_MULTIPLEXING EventSet has multiplexing enabled PAPI_ATTACHED EventSet is attached to another thread/process In the following code example, PAPI_state is used to return the counting state of an EventSet: #include #include main () { int retval, status = 0, EventSet = PAPI_NULL; /* Initialize the PAPI library */ retval = PAPI_library_init(PAPI_VER_CURRENT); if (retval != PAPI_VER_CURRENT) { fprintf(stderr, "PAPI library init error!\n"); exit(1); } /* Create the EventSet */ if (PAPI_create_eventset(&EventSet) != PAPI_OK) handle_error(1); /* Add Total Instructions Executed to our EventSet */ if (PAPI_add_event(&EventSet, PAPI_TOT_INS) != PAPI_OK) handle_error(1); /* Start counting */ if (PAPI_state(EventSet, &status) != PAPI_OK) handle_error(1); printf("State is now %d\n", status); if (PAPI_start(EventSet) != PAPI_OK) - 31 -PAPI User’s Guide Version 3.5.0 handle_error(1); if (PAPI_state(EventSet, &status) != PAPI_OK) handle_error(1); printf("State is now %d\n", status); } OUTPUT: State is now 1 State is now 2 On success, this function returns PAPI_OK and on error, a non-zero error code is returned. - 32 -PAPI User’s Guide Version 3.5.0 GETTING AND SETTING OPTIONS The options of the PAPI library or a specific event set can be obtained and set by calling the following low-level functions, respectively: C: PAPI_get_opt(option, ptr) PAPI_set_opt(option, ptr) Fortran: PAPIF_get_clockrate(clockrate) PAPIF_get_domain(EventSet, domain, mode, check) PAPIF_get_granularity(EventSet, granularity, mode, check) PAPIF_get_preload(preload, check) ARGUMENTS option -- is an input parameter describing the course of action. The Fortran calls are implementations of specific options. Possible values are defined in papi.h and briefly described below: Option name Explanation General information requests PAPI_CLOCKRATE Get clockrate in MHz. PAPI_MAX_CPUS Get number of CPUs. PAPI_MAX_HWCTRS Get number of counters. PAPI_EXEINFO Get Executable addresses for text/data/bss. PAPI_HWINFO Get information about the hardware. PAPI_SHLIBINFO Get shared library information used by the program. PAPI_SUBSTRATEINFO Get the PAPI features the substrate supports. PAPI_LIB_VERSION Get the full PAPI version of the library. PAPI_PRELOAD Get ‘‘LD_PRELOAD’’ environment equivalent. Defaults for the global library PAPI_DEFDOM Get/Set the default counting domain for newly created event sets. PAPI_DEFGRN Get/Set the default counting granularity. PAPI_DEBUG Get/Set the PAPI debug state and the debug handler. The - 33 -PAPI User’s Guide Version 3.5.0 available debug states are defined in papi.h. The debug state is available in ptr->debug.level. The debug handler is available in ptr->debug.handler. For information regarding the behavior of the handler, please see the man page for PAPI_set_debug. Multiplexing control PAPI_ MULTIPLEX Get/Set options for multiplexing. PAPI_MAX_MPX_CTRS Get maximum number of multiplexing counters. PAPI_DEF_MPX_USEC Get/Set the sampling time slice in microseconds for multiplexing. Manipulating individual event sets PAPI_ATTACH Get thread or process id to which event set is attached. Returns TRUE if currently attached. Set event set specified in ptr->ptr->attach.eventset to be attached to thread or process id specified in in ptr->attach.tid. PAPI_DETACH Get thread or process id to which event set is attached. Returns TRUE if currently detached. Set event set specified in ptr->ptr->attach.eventset to be detached from any thread or process id. PAPI_DOMAIN Get/Set domain for a single event set. The event set is specified in ptr->domain.eventset PAPI_GRANUL Get/Set granularity for a single event set. The event set is specified in ptr->granularity.eventset. Currently unimplemented. Platform Specific Options PAPI_DATA_ADDRESS Set data address range to restrict event counting for event set specified in ptr->addr.eventset. Starting and ending addresses are specified in ptr->addr.start and ptr- >addr.end, respectively. If exact addresses cannot be instantiated, offsets are returned in ptr->addr.start_off and ptr->addr.end_off. Currently implemented on Itanium only. PAPI_INSTR_ADDRESS Set instruction address range as described above. Itanium only. ptr -- is a pointer to a structure that acts as both an input and output parameter. It is defined in papi.h and below. EventSet -- input; a reference to an EventSetInfo structure clockrate -- output; cycle time of this CPU in MHz; *may* be an estimate generated at init time with a quick timing routine domain -- output; execution domain for which events are counted - 34 -PAPI User’s Guide Version 3.5.0 granularity -- output; execution granularity for which events are counted mode -- input; determines if domain or granularity are default or for the current event set preload -- output; environment variable string for preloading libraries PAPI_get_opt and PAPI_set_opt query or change the options of the PAPI library or a specific event set created by PAPI_create_eventset. In the C interface, these functions pass a pointer to the PAPI_option_t structure. Not all options require or return information in this structure. The Fortran interface is a series of calls implementing various subsets of the C interface. Not all options in C are available in Fortran. Note that a number of options are available as separate entry points in both C and Fortran. This can make calling sequences simpler. Calls that are simply wrappers to PAPI_get_opt and PAPI_set_opt are listed below: PAPI_get_executable_info Get the executable’s address space information. PAPI_get_hardware_info Get information about the system hardware. PAPI_get_multiplex Get the multiplexing status of specified event set. PAPI_get_shared_lib_info Get information about the shared libraries used by the process. PAPI_get_substrate_info Get information about the substrate features. PAPI_set_debug Set the current debug level for PAPI. PAPI_set_domain Set the default execution domain for new event sets. PAPI_set_granularity Get/Set the default granularity for new event sets. PAPI_set_multiplex Convert a standard event set to a multiplexed event set. The PAPI_option_t structure is actually a union of structures that provide specific information for each of the options defined in the table above. This union is defined as shown below: - 35 -PAPI User’s Guide Version 3.5.0 typedef union { PAPI_preload_info_t preload; PAPI_debug_option_t debug; PAPI_granularity_option_t granularity; PAPI_granularity_option_t defgranularity; PAPI_domain_option_t domain; PAPI_domain_option_t defdomain; PAPI_attach_option_t attach; PAPI_multiplex_option_t multiplex; PAPI_hw_info_t *hw_info; PAPI_shlib_info_t *shlib_info; PAPI_exe_info_t *exe_info; PAPI_substrate_info_t *sub_info; PAPI_addr_range_option_t addr; } PAPI_option_t; Each of these individual structures, as defined in papi.h, is shown below: For PAPI_PRELOAD: typedef struct _papi_preload_option { char lib_preload_env[PAPI_MAX_STR_LEN]; char lib_preload_sep; char lib_dir_env[PAPI_MAX_STR_LEN]; char lib_dir_sep; } PAPI_preload_info_t; For PAPI_DEBUG: typedef int (*PAPI_debug_handler_t) (int code); typedef struct _papi_debug_option { int level; PAPI_debug_handler_t handler; } PAPI_debug_option_t; For PAPI_DEFGRN and PAPI_GRANUL: typedef struct _papi_granularity_option { int eventset; int granularity; } PAPI_granularity_option_t; For PAPI_DEFDOM and PAPI_DOMAIN: typedef struct _papi_domain_option { int eventset; int domain; } PAPI_domain_option_t; For PAPI_ATTACH and PAPI_DETACH: typedef struct _papi_attach_option { int eventset; unsigned long tid; } PAPI_attach_option_t; - 36 -PAPI User’s Guide Version 3.5.0 For PAPI_MULTIPLEX and PAPI_DEF_MPX_USEC: typedef struct _papi_multiplex_option { int eventset; int us; int flags; } PAPI_multiplex_option_t; For PAPI_HWINFO: typedef struct _papi_hw_info { int ncpu; /* Number of CPU's in an SMP Node */ int nnodes; /* Number of Nodes in the entire system */ int totalcpus; /* Total number of CPU's in the entire system */ int vendor; /* Vendor number of CPU */ char vendor_string[PAPI_MAX_STR_LEN]; /* Vendor string of CPU */ int model; /* Model number of CPU */ char model_string[PAPI_MAX_STR_LEN]; /* Model string of CPU */ float revision; /* Revision of CPU */ float mhz; /* Cycle time of this CPU */ PAPI_mh_info_t mem_hierarchy; /* PAPI memory heirarchy description */ } PAPI_hw_info_t; For PAPI_SHLIBINFO and PAPI_EXEINFO: typedef struct _papi_address_map { char name[PAPI_HUGE_STR_LEN]; caddr_t text_start; /* Start address of program text segment */ caddr_t text_end; /* End address of program text segment */ caddr_t data_start; /* Start address of program data segment */ caddr_t data_end; /* End address of program data segment */ caddr_t bss_start; /* Start address of program bss segment */ caddr_t bss_end; /* End address of program bss segment */ } PAPI_address_map_t; typedef struct _papi_shared_lib_info { PAPI_address_map_t *map; int count; } PAPI_shlib_info_t; typedef struct _papi_program_info { char fullname[PAPI_HUGE_STR_LEN]; /* path+name */ PAPI_address_map_t address_info; } PAPI_exe_info_t; - 37 -PAPI User’s Guide Version 3.5.0 For PAPI_SUBSTRATEINFO: typedef struct _papi_substrate_option { char name[PAPI_MAX_STR_LEN]; /* Name of the substrate we're using, usually CVS RCS Id */ char version[PAPI_MIN_STR_LEN]; /* Version of this substrate, usually CVS Revision */ char support_version[PAPI_MIN_STR_LEN]; /* Version of the support library */ char kernel_version[PAPI_MIN_STR_LEN]; /* Version of the kernel PMC support driver */ int num_cntrs; /* Number of hardware counters substrate supports */ int num_mpx_cntrs; /* Number of multiplexed counters the substrate or PAPI supports */ int num_preset_events; /* Number of preset events the substrate supports */ int num_native_events; /* Number of native events the substrate supports */ int default_domain; /* The default domain when this substrate is used */ int available_domains; /* Available domains */ int default_granularity; /* Default granularity when this substrate is used */ int available_granularities; /* Available granularities */ int multiplex_timer_sig; /* Signal number used by the multiplex timer, 0 if not */ int multiplex_timer_num; /* Number of the itimer or POSIX 1 timer used by the multiplex timer */ int multiplex_timer_us; /* uS between switching of sets */ int hardware_intr_sig; /* Signal used by hardware to deliver PMC events */ int opcode_match_width; /* Width of opcode matcher if exists, 0 if not */ int reserved_ints[4]; unsigned int hardware_intr:1; /* hw overflow intr, does not need to be emulated in software*/ unsigned int precise_intr:1; /* Performance interrupts happen precisely */ unsigned int posix1b_timers:1; /* Using POSIX 1b interval timers (timer_create) instead of setitimer */ unsigned int kernel_profile:1; /* Has kernel profiling support (buffered interrupts or sprofil-like) */ unsigned int kernel_multiplex:1; /* In kernel multiplexing */ unsigned int data_address_range:1; /* Supports data address range limiting */ unsigned int instr_address_range:1; /* Supports instruction address range limiting */ unsigned int fast_counter_read:1; /* Supports user level PMC read instruction */ unsigned int fast_real_timer:1; /* Supports a fast real timer */ unsigned int fast_virtual_timer:1; /* Supports a fast virtual timer */ unsigned int attach:1; /* Supports attach */ unsigned int attach_must_ptrace:1; /* Attach must first ptrace and stop the thread/process*/ unsigned int edge_detect:1; /* Supports edge detection on events */ unsigned int invert:1; /* Supports invert detection on events */ unsigned int profile_ear:1; /* Supports data/instr/tlb miss address sampling */ unsigned int grouped_cntrs:1; /* Underlying hardware uses counter groups */ unsigned int reserved_bits:16; } PAPI_substrate_info_t; - 38 -PAPI User’s Guide Version 3.5.0 For PAPI_DATA_ADDRESS and PAPI_INSTR_ADDRESS: /* address range specification for range restricted counting */ typedef struct _papi_addr_range_option { /* if both are zero, range disabled */ int eventset; /* eventset to restrict */ caddr_t start; /* user requested start address of address range */ caddr_t end; /* user requested end address of an address range */ int start_off; /* hardware specified offset from start address */ int end_off; /* hardware specified offset from end address */ } PAPI_addr_range_option_t; The file, papi.h, contains current definitions for the structures unioned in the PAPI_option_t structure. Users should refer to papi.h for specifics on the use of fields in these structures. In the following code example, PAPI_get_opt is used to acquire the option, PAPI_MAX_HWCTRS, of an event set and PAPI_set_opt is used to set the option, PAPI_DOMAIN, to the same event set: #include #include main() { int num, retval, EventSet = PAPI_NULL; PAPI_option_t options; /* Initialize the PAPI library */ retval = PAPI_library_init(PAPI_VER_CURRENT); if (retval != PAPI_VER_CURRENT) { fprintf(stderr, "PAPI library init error!\n"); exit(1); } if ((num = PAPI_get_opt(PAPI_MAX_HWCTRS,NULL)) <= 0) handle_error(); printf("This machine has %d counters.0,num); if (PAPI_create_eventset(&EventSet) != PAPI_OK) handle_error(); /* Set the domain of this EventSet to counter user and kernel modes for this process */ memset(&options,0x0,sizeof(options)); options.domain.eventset = EventSet; options.domain.domain = PAPI_DOM_ALL; if (PAPI_set_opt(PAPI_DOMAIN, &options) != PAPI_OK) handle_error(); } - 39 -PAPI User’s Guide Version 3.5.0 POSSIBLE OUTPUT (VARIES ON DIFFERENT PLATFORMS): This machine has 4 counters. On success, these functions return PAPI_OK and on error, a non-zero error code is returned. For more code examples, see ctests/second.c or ctests/third.c in the PAPI source distribution. SIMPLE CODE EXAMPLES HIGH-LEVEL API The following is a simple code example of using the high-level API: #include #define NUM_FLOPS 10000 #define NUM_EVENTS 1 main() { int Events[NUM_EVENTS] = {PAPI_TOT_INS}; long_long values[NUM_EVENTS]; /* Start counting events */ if (PAPI_start_counters(Events, NUM_EVENTS) != PAPI_OK) handle_error(1); /* Defined in tests/do_loops.c in the PAPI source distribution */ do_flops(NUM_FLOPS); /* Read the counters */ if (PAPI_read_counters(values, NUM_EVENTS) != PAPI_OK) handle_error(1); printf("After reading the counters: %lld\n",values[0]); do_flops(NUM_FLOPS); /* Add the counters */ if (PAPI_accum_counters(values, NUM_EVENTS) != PAPI_OK) handle_error(1); printf("After adding the counters: %lld\n", values[0]); do_flops(NUM_FLOPS); /* Stop counting events */ if (PAPI_stop_counters(values, NUM_EVENTS) != PAPI_OK) handle_error(1); printf("After stopping the counters: %lld\n", values[0]); - 40 -PAPI User’s Guide Version 3.5.0 } POSSIBLE OUTPUT: After reading the counters: 441027 After adding the counters: 891959 After stopping the counters: 443994 Notice that on the second line (after adding the counters) the value is approximately twice as large as the first line (after reading the counters) because PAPI_read_counters resets and leaves the counters running, then PAPI_accum_counters adds the value of the current counter into the values array. LOW-LEVEL API The following is a simple code example that applies the same technique as the above example, except it uses the Low-Level API: #include #include #define NUM_FLOPS 10000 main() { int retval, EventSet=PAPI_NULL; long_long values[1]; /* Initialize the PAPI library */ retval = PAPI_library_init(PAPI_VER_CURRENT); if (retval != PAPI_VER_CURRENT) { fprintf(stderr, "PAPI library init error!\n"); exit(1); } /* Create the Event Set */ if (PAPI_create_eventset(&EventSet) != PAPI_OK) handle_error(1); /* Add Total Instructions Executed to our Event Set */ if (PAPI_add_event(EventSet, PAPI_TOT_INS) != PAPI_OK) handle_error(1); /* Start counting events in the Event Set */ if (PAPI_start(EventSet) != PAPI_OK) handle_error(1); /* Defined in tests/do_loops.c in the PAPI source distribution */ do_flops(NUM_FLOPS); /* Read the counting events in the Event Set */ if (PAPI_read(EventSet, values) != PAPI_OK) handle_error(1); - 41 -PAPI User’s Guide Version 3.5.0 printf("After reading the counters: %lld\n",values[0]); /* Reset the counting events in the Event Set */ if (PAPI_reset(EventSet) != PAPI_OK) handle_error(1); do_flops(NUM_FLOPS); /* Add the counters in the Event Set */ if (PAPI_accum(EventSet, values) != PAPI_OK) handle_error(1); printf("After adding the counters: %lld\n",values[0]); do_flops(NUM_FLOPS); /* Stop the counting of events in the Event Set */ if (PAPI_stop(EventSet, values) != PAPI_OK) handle_error(1); printf("After stopping the counters: %lld\n",values[0]); } POSSIBLE OUTPUT: After reading the counters: 440973 After adding the counters: 882256 After stopping the counters: 443913 Notice that in order to get the desired results (the second line approximately twice as large as the first line), PAPI_reset was called to reset the counters, since PAPI_read did not reset the counters. - 42 -PAPI User’s Guide Version 3.5.0 PAPI TIMERS PAPI timers use the most accurate timers available on the platform in use. These timers can be used to obtain both real and virtual time on each supported platform. The real time clock runs all the time (e.g. a wall clock) and the virtual time clock runs only when the processor is running in user mode. REAL TIME Real time can be acquired in clock cycles and microseconds by calling the following low-level functions, respectively: C: PAPI_get_real_cyc() PAPI_get_real_usec() Fortran: PAPIF_get_real_cyc(check) PAPIF_get_real_usec(check) Both of these functions return the total real time passed since some arbitrary starting point and are equivalent to wall clock time. Also, these functions always succeed (error-free) since they are guaranteed to exist on every PAPI supported platform. In the following code example, PAPI_get_real_cyc and PAPI_get_real_usec are used to obtain the real time it takes to create an event set in clock cycles and microseconds, respectively: #include main() { long_long start_cycles, end_cycles, start_usec, end_usec; int EventSet = PAPI_NULL; if (PAPI_library_init(PAPI_VER_CURRENT) != PAPI_VER_CURRENT) exit(1); /* Gets the starting time in clock cycles */ start_cycles = PAPI_get_real_cyc(); /* Gets the starting time in microseconds */ start_usec = PAPI_get_real_usec(); /*Create an EventSet */ - 43 -PAPI User’s Guide Version 3.5.0 if (PAPI_create_eventset(&EventSet) != PAPI_OK) exit(1); /* Gets the ending time in clock cycles */ end_cycles = PAPI_get_real_cyc(); /* Gets the ending time in microseconds */ end_usec = PAPI_get_real_usec(); printf("Wall clock cycles: %lld\n", end_cycles - start_cycles); prinf(“Wall clock time in microseconds: %lld\n”, end_usec - start_usec); } POSSIBLE OUTPUT: Wall clock cycles: 100173 Wall clock time in microseconds: 136 VIRTUAL TIME Virtual time can be acquired in clock cycles and microseconds by calling the following low-level functions, respectively: C: PAPI_get_virt_cyc() PAPI_get_virt_usec() Fortran: PAPIF_get_virt_cyc(check) PAPIF_get_virt_usec(check) Both of these functions return the total number of virtual units from some arbitrary starting point. Virtual units accrue every time a process is running in user-mode. Like the real time counters, these functions always succeed (error-free) since they are guaranteed to exist on every PAPI supported platform. However, the resolution can be as bad as 1/Hz as defined by the operating system on some platforms. In the following code example, PAPI_get_virt_cyc and PAPI_get_virt_usec are used to obtain the virtual time it takes to create an event set in clock cycles and microseconds, respectively: #include main() { long_long start_cycles, end_cycles, start_usec, end_usec; int EventSet = PAPI_NULL; - 44 -PAPI User’s Guide Version 3.5.0 if (PAPI_library_init(PAPI_VER_CURRENT) != PAPI_VER_CURRENT) exit(1); /* Gets the starting time in clock cycles */ start_cycles = PAPI_get_virt_cyc(); /* Gets the starting time in microseconds */ start_usec = PAPI_get_virt_usec(); /*Create an EventSet */ if (PAPI_create_eventset(&EventSet) != PAPI_OK) exit(1); /* Gets the ending time in clock cycles */ end_cycles = PAPI_get_virt_cyc(); /* Gets the ending time in microseconds */ end_usec = PAPI_get_virt_usec(); printf("Virtual clock cycles: %lld\n", end_cycles - start_cycles); prinf(“Virtual clock time in microseconds: %lld\n”, end_usec - start_usec); } POSSIBLE OUTPUT: Virtual clock cycles: 715408 Virtual clock time in microseconds: 976 - 45 -PAPI User’s Guide Version 3.5.0 PAPI SYSTEM INFORMATION EXECUTABLE INFORMATION Information about the executable’s address space can be obtained by using the following low-level function: C: PAPI_get_executable_info() Fortran: PAPIF_get_exe_info(fullname, name, text_start, text_end, data_start, data_end, bss_start, bss_end, lib_preload_env, check) ARGUMENTS The following arguments are implicit in the structure returned by the C function, or explicitly returned by Fortran: fullname -- fully qualified path + filename of the executable name -- filename of the executable with no path information text_start, text_end -- Start and End addresses of program text segment data_start, data_end -- Start and End addresses of program data segment bss_start, bss_end -- Start and End addresses of program bss segment lib_preload_env -- environment variable for preloading libraries Note that the arguments, text_start and text_end, are the only fields that are filled on every architecture. In C, this function returns a pointer to a structure containing information about the current program, such as the start and end addresses of the text, data, and bss segments. In Fortran, the fields of the structure are returned explicitly. In the following code example, PAPI_get_executable_info is used to acquire information about the start and end addresses of the program’s text segment: #include #include main() { - 46 -PAPI User’s Guide Version 3.5.0 const PAPI_exe_info_t *prginfo = NULL; if (PAPI_library_init(PAPI_VER_CURRENT) != PAPI_VER_CURRENT) exit(1); if ((prginfo = PAPI_get_executable_info()) == NULL) exit(1); printf("Start of user program is at %p\n",prginfo->text_start); printf("End of user program is at %p\n",prginfo->text_end); } POSSIBLE OUTPUT: Start of user program is at 0x4000000000000f20 End of user program is at 0x4000000000034e00 In C, on success, the function returns a non-NULL pointer and on error, NULL is returned. In Fortran, on success, the function returns PAPI_OK and on error, a non-zero error code is returned. HARDWARE INFORMATION Information about the system hardware can be obtained by using the following lowlevel function: C: PAPI_get_hardware_info() Fortran: PAPIF_get_hardware_info (ncpu, nnodes, totalcpus, vendor, vendor_string, model, model_string, revision, mhz) ARGUMENTS The following arguments are implicit in the structure returned by the C function, or explicitly returned by Fortran. ncpu -- number of CPUs in an SMP Node nnodes -- number of Nodes in the entire system totalcpus -- total number of CPUs in the entire system vendor -- vendor id number of CPU vendor_string -- vendor id string of CPU model -- model number of CPU model_string -- model string of CPU revision -- Revision number of CPU - 47 -PAPI User’s Guide Version 3.5.0 mhz -- Cycle time of this CPU; *may* be an estimate generated at initial time with a quick timing routine In C, this function returns a pointer to a structure containing information about the hardware on which the program runs, such as: the number of CPUs, CPU model information, and the cycle time of the CPU. In Fortran, the fields of the structure are returned explicitly. Note that if this function were called before PAPI_library_init, it would be undefined. In the following code example, PAPI_get_hardware_info is used to acquire hardware information about the total number of CPUs and the cycle time of the CPU: #include #include main() { const PAPI_hw_info_t *hwinfo = NULL; if (PAPI_library_init(PAPI_VER_CURRENT) != PAPI_VER_CURRENT) exit(1); if ((hwinfo = PAPI_get_hardware_info()) == NULL) exit(1); printf("%d CPU’s at %f Mhz.\n",hwinfo->totalcpus,hwinfo->mhz); } POSSIBLE OUTPUT: 1 CPUs at 733.000000 Mhz. In C, on success, this function returns a non-NULL pointer and on error, NULL is returned. In Fortran, on success, this function returns PAPI_OK and on error, a non-zero error code is returned. - 48 -PAPI User’s Guide Version 3.5.0 SUBSTRATE INFORMATION Implementation details about the current hardware dependent substrate can be obtained by using the following low-level function: C: PAPI_get_substrate_info() Fortran: This call is not implemented in the Fortran interface. ARGUMENTS In C, this function returns a pointer to a structure containing implementation details about the substrate currently in use, such as: the number of counters and multiplexed counters supported, the number of preset and native events available, and whether (and how) certain advanced features are supported. For more details, refer to the definition of the PAPI_substrate_info_t structure found in papi.h, or see the discussion under getting and setting options. Note: if this function is called before PAPI_library_init, its output is undefined. In the following code example, PAPI_get_substrate_info is used to determine how many preset and native events can be counted for a given substrate: #include #include main() { const PAPI_substrate_info_t *subinfo = NULL; if (PAPI_library_init(PAPI_VER_CURRENT) != PAPI_VER_CURRENT) exit(1); if ((subinfo = PAPI_get_substrate_info()) == NULL) exit(1); printf("num_preset_events: %d\n",subinfo->num_preset_events); printf("num_native_events: %d\n",subinfo->num_native_events); } POSSIBLE OUTPUT: num_preset_events: 47 num_native_events: 193 On success, this function returns a non-NULL pointer and on error, NULL is returned. - 49 -PAPI User’s Guide Version 3.5.0 ADVANCED PAPI FEATURES MULTIPLEXING WHAT IS MULTIPLEXING? Multiplexing allows more events to be counted than can be supported by the hardware. When a microprocessor has a limited number of hardware counters, a large application with many hours of run time may require days or weeks of profiling in order to gather enough information on which to base a performance analysis. Multiplexing overcomes this limitation by subdividing the usage of the counter hardware over time (timesharing) among a large number of performance events. USING PAPI WITH MULTIPLEXING INITIALIZATION OF MULTIPLEX SUPPORT Multiplex support in the PAPI library can be enabled and initialized by calling the following low-level function: C: PAPI_muliplex_init() Fortran: PAPIF_multiplex_init(check) The above function sets up the internal structures to allow more events to be counted than there are physical counters. It does this by timesharing the existing counters at some loss in precision. This function should be used after calling PAPI_library_init. After this function is called, the user can proceed to use the normal PAPI routines. It should be also noted that applications that make no use of multiplexing should not call this function. On success, this function returns PAPI_OK and on error, a non-zero error code is returned. For a code example, see the next section. - 50 -PAPI User’s Guide Version 3.5.0 CONVERTING AN EVENT SET INTO A MULTIPLEXED EVENT SET In addition, a standard event set can be converted to a multiplexed event set by the calling the following low-level function: C: PAPI_set_multiplex(EventSet) Fortran: PAPIF_set_multiplex(EventSet) ARGUMENT EventSet -- an integer handle for a PAPI event set as created by PAPI_create_eventset. The above function converts a standard PAPI event set created by a call to PAPI_create_eventset into an event set capable of handling multiplexed events. This function must be used after calling PAPI_multiplex_init and PAPI_create_eventset, but prior to calling PAPI_start. Events can be added to an event set either before or after converting it into a multiplexed set, but the conversion must be done prior to using it as a multiplexed set. In the following code example, PAPI_set_multiplex is used to convert a standard event set into a multiplexed event set: #include int retval, i, EventSet = PAPI_NULL, max_to_add = 6, j = 0; long_long *values; const PAPI_preset_info_t *pset; main() { /* Initialize the PAPI library */ retval = PAPI_library_init(PAPI_VER_CURRENT); if (retval != PAPI_VER_CURRENT) handle_error(1); /* Enable and initialize multiplex support */ if (PAPI_multiplex_init() != PAPI_OK) handle_error(1); /* Create an EventSet */ if (PAPI_create_eventset(&EventSet) != PAPI_OK) handle_error(1); /* Convert the EventSet to a multiplexed event set */ if (PAPI_set_multiplex(EventSet) != PAPI_OK) handle_error(1); - 51 -PAPI User’s Guide Version 3.5.0 for (i=0;ievent_code != PAPI_OK) handle_error(1); if (++j >= max_to_add) break; } } values = (long_long *)malloc(max_to_add*sizeof(long_long)); if (values == NULL) handle_error(1); /* Start counting events */ if (PAPI_start(EventSet) != PAPI_OK) handle_error(1); } On success, both functions return PAPI_OK and on error, a non-zero error code is returned. For more code examples, see ctests/multiplex1.c in the papi source distribution. ISSUES OF MULTIPLEXING The following are some issues concerning multiplexing that the PAPI user should be aware of: • Hardware multiplexing is not supported by all platforms. On those platforms where it is supported, PAPI takes advantage of it. Otherwise, PAPI implements software multiplexing through the use of a high-resolution interval timer. For more information on which platforms support hardware or software multiplexing, see Appendix H. • Multiplexing unavoidably incurs a small amount of overhead when switching events. In addition, no single event is measured for the full analysis time. These factors can adversely affect the precision of reported counter values. In other words, the more events that are multiplexed, the more likely that the results will be statistically skewed. The amount of time spent in the measured regions should be greater than the multiplexing time slice times the number of events measured in order to get acceptable results. • The default time slice for multiplexing is currently set at 100000 microseconds. Occasionally this setting can cause a resonant situation in the code in which a given pattern repeats at the same frequency that timers are switched out. This can by addressed by changing the time slice setting by calling PAPI_set_opt with the PAPI_DEF_MPX_USEC option. - 52 -PAPI User’s Guide Version 3.5.0 • To prevent naïve use of multiplexing by the novice user, the high level API can only access those events countable simultaneously by the underlying hardware, unless a low level function has been called to explicitly enable multiplexing. USING PAPI WITH PARALLEL PROGRAMS THREADS WHAT ARE THREADS? A thread is an independent flow of instructions that can be scheduled to run by the operating system. Multi-threaded programming is a form of parallel programming where several controlled threads are executing concurrently in the program. All threads execute in the same memory space, and can therefore work concurrently on shared data. Threads can run in parallel on several processors, allowing a single program to divide its work between several processors, thus running faster than a single-threaded program, which runs on only one processor at a time. PAPI only supports thread level measurements with kernel or bound threads, which are threads that have a scheduling entity known and handled by the operating system’s kernel. In most cases, such as with SMP or OpenMP complier directives, bound threads will be the default. Each thread is responsible for the creation, start, stop, and read of its own counters. When a thread is created, it inherits no PAPI information from the calling thread. There are some threading packages or APIs that can be used to manipulate threads with PAPI, particularly Pthreads and OpenMP. For those using Pthreads, the user should take care to set the scope of each thread to PTHREAD_SCOPE_SYSTEM attribute, unless the system is known to have a nonhybrid thread library implementation. In addition, PAPI does support unbound or non-kernel threads, but the counts will reflect the total events for the process. Measurements that are done in other threads will get all the same values, namely the counts for the total process. For unbound threads, it is not necessary to call PAPI_thread_init, which will be discussed in the next section. When threads are in use, PAPI allows the user to provide a routine to its library that returns the thread ID of the currently running thread (for example, pthreads_self for Pthreads) and this thread ID is used as a lookup function for the internal data structures. INITIALIZATION OF THREAD SUPPORT Thread support in the PAPI library can be initialized by calling the following low-level function: C: - 53 -PAPI User’s Guide Version 3.5.0 PAPI_thread_init(handle) Fortran: PAPIF_thread_init(handle, check) ARGUMENTS handle -- Pointer to a routine that returns the current thread ID. This function should be called only once, just after PAPI_library_init, and before any other PAPI calls. If the function is called more than once, the application will exit. Also, applications that make no use of threads do not need to call this function. The following example shows the correct syntax for using PAPI_thread_init with OpenMP: C: #include #include if (PAPI_thread_init(omp_get_thread_num) != PAPI_OK) handle_error(1); Fortran: #include “fpapi.h” #include “omp.h” EXTERNAL omp_get_thread_num C Fortran dictates that in order to a pass a subroutine C as an argument, the subroutine must be C declared external! call PAPIF_thread_init(omp_get_thread_num, error) On success, the function, PAPI_thread_init, returns PAPI_OK and on error, a nonzero error code is returned. For a code example of using PAPI_thread_init with Pthreads, see the next section. THREAD ID The identifier of the current thread can be obtained by calling the following low-level function: C: PAPI_thread_id() Fortran: - 54 -PAPI User’s Guide Version 3.5.0 PAPIF_thread_id(check) This function calls the thread id function registered by PAPI_thread_init and returns an unsigned long integer containing the thread identifier. In the following code example, PAPI_thread_init and PAPI_thread_id are used to initialize thread support in the PAPI library and to acquire the identifier of the current thread, respectively, with Pthreads: #include #include main() { unsigned long int tid; if (PAPI_library_init(PAPI_VER_CURRENT) != PAPI_VER_CURRENT) exit(1); if (PAPI_thread_init(pthread_self) != PAPI_OK) exit(1); if ((tid = PAPI_thread_id()) == (unsigned long int)-1) exit(1); printf("Initial thread id is: %lu\n",tid); } OUTPUT: Initial thread id is: 0 On success, this function returns a valid thread identifier and on error, (unsigned long int) –1 is returned. Four more utility functions related to threads are available in PAPI. These functions allow you to register a newly created thread to make it available for reference by PAPI, to remove a registered thread in cases where thread ids may be reused by the system, and to create and access thread-specific storage in a platform independent fashion for use with PAPI. These functions are shown below: C: PAPI_register_thread() PAPI_unregister_thread() PAPI_get_thr_specific(tag, ptr) PAPI_set_thr_specific(tag, ptr) - 55 -PAPI User’s Guide Version 3.5.0 ARGUMENTS tag -- Integer value specifying one of 4 storage locations. ptr -- Pointer to the address of a data structure. For more code examples of using Pthreads and OpenMP with PAPI, see ctests/zero_pthreads.c and ctests/zero_omp.c in the papi source distribution, respectively. Also, for a code example of using SMP with PAPI, see ctests/zero_smp.c in the papi source distribution. MPI MPI is an acronym for Message Passing Interface. MPI is a library specification for message-passing, proposed as a standard by a broadly based committee of vendors, implementers, and users. MPI was designed for high performance on both massively parallel machines and on workstation clusters. More information on MPI can be found at http://www-unix.mcs.anl.gov/mpi. PAPI supports MPI. When using timers in applications that contain multiplexing, profiling, and overflow, MPI uses a default virtual timer and must be converted to a real timer in order to for the application to work properly. Otherwise, the application will exit. Optionally, the supported tools, TAU and SvPablo, can be used to implement PAPI with MPI. The following is a code example of using MPI’s PI program with PAPI: #include #include #include #include int main(argc,argv) int argc; char *argv[]; { int done = 0, n, myid, numprocs, i, rc, retval, EventSet = PAPI_NULL; double PI25DT = 3.141592653589793238462643; double mypi, pi, h, sum, x, a; long_long values[1] = {(long_long) 0}; MPI_Init(&argc,&argv); MPI_Comm_size(MPI_COMM_WORLD,&numprocs); MPI_Comm_rank(MPI_COMM_WORLD,&myid); /*Initialize the PAPI library */ retval = PAPI_library_init(PAPI_VER_CURRENT); if (retval != PAPI_VER_CURRENT) { fprintf(stderr, "PAPI library init error!\n"); exit(1); } - 56 -PAPI User’s Guide Version 3.5.0 /* Create an EventSet */ if (PAPI_create_eventset(&EventSet) != PAPI_OK) handle_error(1); /* Add Total Instructions Executed to our EventSet */ if (PAPI_add_event(EventSet, PAPI_TOT_INS) != PAPI_OK) handle_error(1); /* Start counting */ if (PAPI_start(EventSet) != PAPI_OK) handle_error(1); while (!done) { if (myid == 0) { printf("Enter the number of intervals: (0 quits) "); scanf("%d",&n); } MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD); if (n == 0) break; h = 1.0 / (double) n; sum = 0.0; for (i = myid + 1; i <= n; i += numprocs) { x = h * ((double)i - 0.5); sum += 4.0 / (1.0 + x*x); } mypi = h * sum; MPI_Reduce(&mypi, &pi, 1, MPI_DOUBLE, MPI_SUM, 0,MPI_COMM_WORLD); if (myid == 0) printf("pi is approximately %.16f, Error is %.16f\n", pi, fabs(pi - PI25DT)); } /* Read the counters */ if (PAPI_read(EventSet, values) != PAPI_OK) handle_error(1); printf("After reading counters: %lld\n",values[0]); /* Start the counters */ if (PAPI_stop(EventSet, values) != PAPI_OK) handle_error(1); printf("After stopping counters: %lld\n",values[0]); MPI_Finalize(); } POSSIBLE OUTPUT (AFTER ENTERING 50, 75, AND 100 AS INPUT): Enter the number of intervals: (0 quits) 50 pi is approximately 3.1416259869230028, Error is 0.0000333333332097 Enter the number of intervals: (0 quits) 75 pi is approximately 3.1416074684045965, Error is 0.0000148148148034 - 57 -PAPI User’s Guide Version 3.5.0 Enter the number of intervals: (0 quits) 100 pi is approximately 3.1416009869231254, Error is 0.0000083333333323 Enter the number of intervals: (0 quits) 0 After reading counters: 117393 After stopping counters: 122921 OVERFLOW WHAT IS AN OVERFLOW? An overflow happens when the number of occurrences of a particular hardware event exceeds a specified threshold. PAPI provides the ability to call user-defined handlers when an overflow occurs. This can be done in hardware, if the processor generates an interrupt signal when the counter reaches a specified value, or in software, by setting up a high-resolution interval timer and installing a timer interrupt handler. For software based overflow, PAPI compares the current counter value against the threshold every time the timer interrupt occurs. If the current value exceeds the threshold, then the user’s handler is called from within the signal context with some additional arguments. These arguments allow the user to determine which event overflowed, by how much it overflowed, and at what location in the source code the overflow occurred. Using the same mechanism as for user programmable overflow, PAPI also guards against register precision overflow of counter values. Each counter can potentially be incremented multiple times in a single clock cycle. This fact combined with increasing clock speeds and the small dynamic range of some of the physical counters means that an overflow is likely to occur on platforms where 64-bit counters are not supported in hardware or by the operating system. In those cases, the PAPI implements 64-bit counters in software using the same mechanism that handles overflow dispatch. BEGINNING OVERFLOWS IN EVENT SETS An event set can begin registering overflows by calling the following low-level function: C: PAPI_overflow(EventSet, EventCode, threshold, flags, handler) ARGUMENTS EventSet -- a reference to the event set to use EventCode -- the event to be used for overflow detection threshold -- the overflow threshold value to use - 58 -PAPI User’s Guide Version 3.5.0 flags -- bit map that controls the overflow mode of operation. The only currently valid setting is PAPI_OVERFLOW_FORCE_SW, which overrides the default hardware overflow setting on a platform that supports hardware overflow. handler -- the handler function to call upon overflow This function marks a specific EventCode in an EventSet to generate an overflow signal after every threshold events are counted. Mutiple events within an event set can be programmed to overflow by making successive calls to this function, but only a single overflow handler can be registered. To turn off overflow for a specific event, call PAPI_overflow with EventCode set to the desired event and threshold set to zero. The handler function is a user-supplied callback routine that performs whatever special processing needed to handle the overflow interrupt, including sorting multiple overflowing events from each other. It must conform to the following prototype: C: PAPI_overflow_handler(EventSet, address, overflow_vector, void *context) ARGUMENTS EventSet -- a reference to the event set in use address – the address of the program counter when the overflow occurred overflow_vector – a 64-bit vector that specifies which counter(s) generated the overflow. Bit 0 corresponds to counter 0. The handler should be able to deal with multiple overflow bits per call if more than one event may be set to overflow. context -- a platform dependent structure containing information about the state of the machine when the overflow occurred. This structure is provided for completeness, but can generally be ignored by most users. In the following code example, PAPI_overflow is used to mark PAPI_TOT_INS in order to generate an overflow signal after every 100,000 counted events: #include #include #define THRESHOLD 100000 int total = 0; /* total overflows */ void handler(int EventSet, void *address, long_long overflow_vector, void *context) { fprintf(stderr, "handler(%d) Overflow at %p! vector=0x%llx\n", EventSet, address, overflow_vector); total++; - 59 -PAPI User’s Guide Version 3.5.0 } main() { int retval, EventSet = PAPI_NULL; /* Initialize the PAPI library */ retval = PAPI_library_init(PAPI_VER_CURRENT); if (retval != PAPI_VER_CURRENT) handle_error(1); /* Create the EventSet */ if (PAPI_create_eventset(&EventSet) != PAPI_OK) handle_error(1); /* Add Total Instructions Executed to our EventSet */ if (PAPI_add_event(EventSet, PAPI_TOT_INS) != PAPI_OK) handle_error(1); /* Call handler every 100000 instructions */ retval = PAPI_overflow(EventSet, PAPI_TOT_INS, THRESHOLD, 0, handler); if (retval != PAPI_OK) handle_error(1); /* Start counting */ if (PAPI_start(EventSet) != PAPI_OK) handle_error(1); } On success, this function returns PAPI_OK and on error, a non-zero error code is returned. For more code examples, see ctests/overflow.c, ctests/overflow_twoevents.c or ctests/overflow_pthreads.c in the papi source distribution. STATISTICAL PROFILING WHAT IS STATISTICAL PROFILING? Statistical Profiling involves periodically interrupting a running program and examining the program counter at the time of the interrupt. If this is done for a reasonable number of interrupting intervals, the resulting program counter distribution will be statistically representative of the execution profile of the program with respect to the interrupting event. Performance tools like UNIX prof sample the program address with respect to time and hash the value into a histogram. At program completion, the histogram is analyzed and associated with symbolic information contained in the executable. GNU prof in conjunction with the –p option of the GCC compiler performs exactly this analysis using the process time as the interrupting trigger. PAPI aims to generalize this functionality so that a histogram - 60 -PAPI User’s Guide Version 3.5.0 can be generated using any countable hardware event as the basis for the interrupt signal. GENERATING A PC HISTOGRAM A PC histogram can be generated on any countable event by calling either of the following low-level functions: C: PAPI_profil(buf, bufsiz, offset, scale, EventSet, EventCode, threshold, flags) PAPI_sprofil(prof, profcnt, EventSet, EventCode, threshold, flags) Fortran: PAPI_profil(buf, bufsiz, offset, scale, EventSet, EventCode, threshold, flags, check) AGRUMENTS *buf -- pointer to profile buffer array. bufsiz -- number of entries in *buf. offset -- starting value of lowest memory address to profile. scale -- scaling factor for bin values. EventSet -- The PAPI EventSet to profile when it is started. EventCode -- code of the Event in the EventSet to profile. threshold -- threshold value for the Event triggers the handler. flags -- bit pattern to control profiling behavior. The defined bit values for the flags variable are shown in the table below: Defined bit Description PAPI_PROFIL_POSIX Default type of profiling. PAPI_PROFIL_RANDOM Drop a random 25% of the samples. PAPI_PROFIL_WEIGHTED Weight the samples by their value. PAPI_PROFIL_COMPRESS Ignore samples if hash buckets get big. PAPI_PROFIL_BUCKET_16 Save samples in 16-bit hash buckets. PAPI_PROFIL_BUCKET_32 Save samples in 32-bit hash buckets. PAPI_PROFIL_BUCKET_64 Save samples in 64-bit hash buckets. PAPI_PROFIL_FORCE_SW Force software overflow in profiling. *prof -- pointer to PAPI_sprofil_t structure. profcnt -- number of buffers for hardware profiling (reserved). PAPI_profil creates a histogram of overflow counts for a specified region of the application code by using its first four parameters to create the data structures - 61 -PAPI User’s Guide Version 3.5.0 needed by PAPI_sprofil and then calls PAPI_sprofil to do the work. PAPI_sprofil assumes a pre-initialized PAPI_sprofil_t structure and enables profiling for the EventSet based on its value. Note that the EventSet must be in the stopped state in order for either call to succeed. More than one hardware event can be profiled at the same time by making multiple independent calls to these functions for the same EventSet before calling PAPI_start. This can be useful for the simultaneous generation of profiles of two or more related events, for example L1 cache misses and L2 cache misses. Profiling can be turned off for specific events by calling the function for that event with a threshold of zero. On success, these functions return PAPI_OK and on error, a non-zero error code is returned. For more code examples, see profile.c, profile_twoevents.c or sprofile.c in the ctests directory of the PAPI source distribution. For a more extensive description of the parameters in the PAPI_profil call, see the PAPI_profil man page or its html counterpart at: http://icl.cs.utk.edu/projects/papi/files/html_man3/papi_profil.html In the following code example, PAPI_profil is used to generate a PC histogram: #include #include main() { int retval; int EventSet = PAPI_NULL; unsigned long start, end, length; PAPI_exe_info_t *prginfo; unsigned short *profbuf; /* Initialize the PAPI library */ retval = PAPI_library_init(PAPI_VER_CURRENT); if (retval != PAPI_VER_CURRENT & retval > 0) { fprintf(stderr,"PAPI library version mismatch!0); exit(1); } if (retval < 0) handle_error(retval); if ((prginfo = PAPI_get_executable_info()) == NULL) handle_error(1); start = (unsigned long)prginfo->text_start; end = (unsigned long)prginfo->text_end; length = end - start; - 62 -PAPI User’s Guide Version 3.5.0 profbuf = (unsigned short *)malloc(length*sizeof(unsigned short)); if (profbuf == NULL) handle_error(1); memset(profbuf,0x00,length*sizeof(unsigned short)); if (PAPI_create_eventset(&EventSet) != PAPI_OK) handle_error(retval); /* Add Total FP Instructions Executed to our EventSet */ if (PAPI_add_event(EventSet, PAPI_FP_INS) != PAPI_OK) handle_error(retval); if (PAPI_profil(profbuf, length, start, 65536, EventSet, PAPI_FP_INS, 1000000, PAPI_PROFIL_POSIX | PAPI_PROFIL_BUCKET_16) != PAPI_OK) handle_error(1); /* Start counting */ if (PAPI_start(EventSet) != PAPI_OK) handle_error(1); } DATA AND INSTRUCTION ADDRESS RESTRICTION Introduction Performance instrumentation of data structures, as opposed to code segments, is a feature not widely supported across a range of platforms. One platform on which this feature is supported is the Itanium2. In fact, event counting on Itanium2 can be qualified by a number of conditioners, including instruction address, opcode matching, and data address. We have implemented a generalized PAPI interface for data structure and instruction range performance instrumentation, also referred to as data and instruction range specification, and applied that interface to the specific instance of the Itanium2 platform to demonstrate its viability. This feature is being introduced for the first time in the PAPI 3.5 release. The PAPI Interface Since PAPI is a platform-independent library, care must be taken when extending its feature set so as not to disrupt the existing interface or to clutter the API with calls to functionality that is not available on a large subset of the supported platforms. To that end, we elected to extend an existing PAPI call, PAPI_set_opt(), with the capability of specifying starting and ending addresses of data structures or instructions to be instrumented. The PAPI_set_opt() call previously supported functionality to set a variety of optional capability in the PAPI interface, including debug levels, multiplexing of eventsets, and the scope of counting domains. This call was extended with two new cases to support instruction and data address range specification: PAPI_INSTR_ADDRESS and PAPI_DATA_ADDRESS. To access these - 63 -PAPI User’s Guide Version 3.5.0 options, a user initializes a simple option specific data structure and calls PAPI_set_opt() as illustrated in the code fragment below: ... option.addr.eventset = EventSet; option.addr.start = (caddr_t)array; option.addr.end = (caddr_t)(array + size_array); retval = PAPI_set_opt(PAPI_DATA_ADDRESS, &option); ... The user creates a PAPI eventset and determines the starting and ending addresses of the data to be monitored. The call to PAPI_set_opt then prepares the interface to count events that occur on accesses to data in that range. The specific events to be monitored can be added to the eventset either before or after the data range is specified. In a similar fashion, an instruction range can be set using the PAPI_INSTR_ADDRESS option. If this option is supported on the platform in use, the data is transferred to the platform-specific implementation and handled appropriately. If not supported, the call returns an error message. It may not always be possible to exactly specify the address range of interest. If this is the case, it is important that the user have some way to know what approximations have been made, so that appropriate corrective action can be taken. For instance, to isolate a specific data structure completely, it may be necessary to pad memory before and after the structure with dummy structures that are never accessed. To facilitate this, PAPI_set_opt() returns the offsets from the requested starting and ending addresses as they were actually programmed into the hardware. If the addresses were mapped exactly, these values are zero. An example of this is shown below: ... retval = PAPI_set_opt(PAPI_DATA_ADDRESS, &option); actual.start = (caddr_t)array - option.addr.start_off; actual.end = (caddr_t)(array + size_array) + option.addr.end_off; ... Itanium Idiosyncrasies There are roughly 475 native events available on Itanium 2.160 of them are memory related and can be counted with data address specification in place. 283 can be counted using instruction address specification. All events in an eventset with data or instruction range specification in place must be one of these supported events. Further restrictions also apply to the use of data and instruction range specification, as described below. Data addresses can only be specified in coarse mode. Although four independent pairs of data address registers exist in the hardware and would suggest that four disjoint address regions can be monitored simultaneously, the Intel documentation strongly suggests that this is not a good idea. Further, the underlying software takes advantage of these register pairs to - 64 -PAPI User’s Guide Version 3.5.0 tune the range of addresses that is actually monitored. See the discussion under Data Address Ranges for further detail. Instruction Address Ranges Instruction ranges can be specified in one of two ways: coarse and fine. In fine mode, addresses can be specified exactly, but both the start and end addresses must exist on the same 4K byte page. In other words, the address range must be less than 4K bytes, and the addresses can only differ in the bottom 12 bits. If fine mode cannot be used, the underlying perfmon library automatically switches to coarse address specification. Four pairs of registers are available to specify coarse instruction address ranges. The restrictions to coarse address specification are discussed below. Data Address Ranges Data addresses can only be specified in coarse mode. As with instruction ranges, four pairs of registers are available to specify the data address ranges. Use of coarse mode addressing for either instruction or data address specification can cause some anomalous results. The Intel documentation points out that starting and ending addresses cannot be specified exactly, since the hardware representation relies on powers-of-two bitmasks. The perfmon library tries to optimize the alignment of these power-of-two regions to cover the addresses requested as effectively as possible with the four sets of registers available. Perfmon first finds the largest power-of-two address region completely contained within the requested addresses. Then it finds successively smaller power-of-two regions to cover the errors on the high and low end of the requested address range. The effective result is that the actual range specified is always equal to or larger than, and completely contains the requested range, and can occupy from one to four pairs of address registers. In some cases this can result in significant overcounts of the events of interest, especially if two active data structures are located in close proximity to each other. This may require that the developer insert some padding structures before and/or after a particular structure of interest to guarantee accurate counts. Supporting Software To make this new PAPI feature more accessible and easier to use, a test case was developed to both provide a coding example and to exercise and test the fuctionality of the data ranging features of the Itanium 2. In addition, the papi_native_event utility was modified to make it easier to identify events that support these features. The data_range.c Test Case A test case, called data_range was developed that measures memory load and store events on three different types of data structures. Three static arrays of 16,384 ints were declared in the program, and three dynamic arrays of 16,384 ints - 65 -PAPI User’s Guide Version 3.5.0 were malloc'd. The data range was specified sequentially to be the starting and ending addresses of each of: • the pointers to the malloc'd arrays; • the malloc'd arrays themselves; • the statically declared arrays. The work done in each case consisted of loading an initialization value into each element of each array, and then summing the values of each element. This should produce 16,384 loads and 16,384 stores on each element. For the pointers, the size was 8 bytes and the starting and ending addresses could be specified exactly. Outputis shown below: Measure loads and stores on the pointers to the allocated arrays Expected loads: 32768; Expected stores: 0 These loads result from accessing the pointers to compute array addresses. They will likely disappear with higher levels of optimization. Requested Start Address: 0x6000000000011640; Start Offset: 0x 0; Actual Start Address: 0x6000000000011640 Requested End Address: 0x6000000000011648; End Offset: 0x 0; Actual End Address: 0x6000000000011648 loads_retired: 32768 stores_retired: 0 Requested Start Address: 0x6000000000011628; Start Offset: 0x 0; Actual Start Address: 0x6000000000011628 Requested End Address: 0x6000000000011630; End Offset: 0x 0; Actual End Address: 0x6000000000011630 loads_retired: 32768 stores_retired: 0 Requested Start Address: 0x6000000000011638; Start Offset: 0x 0; Actual Start Address: 0x6000000000011638 Requested End Address: 0x6000000000011640; End Offset: 0x 0; Actual End Address: 0x6000000000011640 loads_retired: 32768 stores_retired: 0 For the allocated arrays, small offsets were introduced in each case, and the resulting error in the loads and stores is exactly what would be predicted by the activity in the adjacent memory locations: Measure loads and stores on the allocated arrays themselves Expected loads: 16384; Expected stores: 16384 Requested Start Address: 0x6000000004044010; Start Offset: 0x 10; Actual Start Address: 0x6000000004044000 Requested End Address: 0x6000000004054010; End Offset: 0x 0; Actual End Address: 0x6000000004054010 loads_retired: 16384 - 66 -PAPI User’s Guide Version 3.5.0 stores_retired: 16384 Requested Start Address: 0x6000000004054020; Start Offset: 0x 20; Actual Start Address: 0x6000000004054000 Requested End Address: 0x6000000004064020; End Offset: 0x 0; Actual End Address: 0x6000000004064020 loads_retired: 16388 stores_retired: 16388 Requested Start Address: 0x6000000004064030; Start Offset: 0x 30; Actual Start Address: 0x6000000004064000 Requested End Address: 0x6000000004074030; End Offset: 0x 10; Actual End Address: 0x6000000004074040 loads_retired: 16392 stores_retired: 16392 For the static arrays, the locations of the arrays resulted in significant offsets, and hence significant errors. The most interesting case is the second one, in which the starting offset can be seen to force the inclusion of all three pointers to the malloc'd arrays. Because of this the loads retired count is too high by 98310, almost exactly 3 * 32768 = 98204: Measure loads and stores on the static arrays These values will differ from the expected values by the size of the offsets. Expected loads: 16384; Expected stores: 16384 Requested Start Address: 0x60000000000218cc; Start Offset: 0x 18cc; Actual Start Address: 0x6000000000020000 Requested End Address: 0x60000000000318cc; End Offset: 0x 734; Actual End Address: 0x6000000000032000 loads_retired: 18432 stores_retired: 18432 Requested Start Address: 0x60000000000118cc; Start Offset: 0x 18cc; Actual Start Address: 0x6000000000010000 Requested End Address: 0x60000000000218cc; End Offset: 0x 734; Actual End Address: 0x6000000000022000 loads_retired: 115155 stores_retired: 16845 Requested Start Address: 0x60000000000318cc; Start Offset: 0x 18cc; Actual Start Address: 0x6000000000030000 Requested End Address: 0x60000000000418cc; End Offset: 0x 734; Actual End Address: 0x6000000000042000 loads_retired: 17971 stores_retired: 17971 The papi_native_avail Utility To effectively use the instruction and address range specification feature for Itanium 2, one must know which of the roughly 475 available native events support these features. In addition, there are other qualifiers to Itanium 2 native events that are valuable to inspect. For these reasons, the papi_native_avail utility was - 67 -PAPI User’s Guide Version 3.5.0 enhanced to make it possible to filter the list of native events by these qualifiers. A help feature was added to this utility to make it easier to remember the Itanium specific options: > papi_native_avail --help This is the PAPI native avail program. It provides availability and detail information for PAPI native events. Usage: papi_native_avail [options] Options: -h, --help print this help message --darr display Itanium events that support Data Address Range Restriction --dear display Itanium Data Event Address Register events only --iarr display Itanium events that support Instruction Address Range Restriction --iear display Itanium Instruction Event Address Register events only --opcm display Itanium events that support OpCode Matching NOTE: The last five options are mutually exclusive. If any of these options are specified on the command line, only those events that support that option are displayed. Even so, the list can be extensive, with roughly 160 events supporting data address ranging, and even more supporting instruction address ranging. - 68 -PAPI User’s Guide Version 3.5.0 PAPI ERROR HANDLING ERROR CODES All of the functions contained in the PAPI library return standardized error codes in which the values that are greater than or equal to zero indicate success and those that are less than zero indicate failure, as shown in the table below: VALUE SYMBOL DEFINITION 0 PAPI_OK No error -1 PAPI_EINVAL Invalid argument -2 PAPI_ENOMEM Insufficient memory -3 PAPI_ESYS A system or C library call failed, please check errno -4 PAPI_ESBSTR Substrate returned an error, usually the result of an unimplemented feature -5 PAPI_ECLOST Access to the counters was lost or interrupted -6 PAPI_EBUG Internal error, please send mail to the developers -7 PAPI_ENOEVNT Hardware event does not exist -8 PAPI_ECNFLCT Hardware event exists, but cannot be counted due to counter resource limitations -9 PAPI_ENOTRUN No events or event sets are currently not counting -10 PAPI_EISRUN Event Set is currently running -11 PAPI_ENOEVST No such event set available -12 PAPI_ENOTPRESET Event is not a valid preset -13 PAPI_ENOCNTR Hardware does not support performance counters -14 PAPI_EMISC ‘Unknown error’ code -15 PAPI_EPERM You lack the necessary permissions CONVERTING ERROR CODES TO ERROR MESSAGES Error codes can be converted to error messages by calling the following low-level functions: C: PAPI_perror(code, destination, length) PAPI_strerror(code) Fortan: PAPIF_perror(code, destination, check) - 69 -PAPI User’s Guide Version 3.5.0 ARGUMENTS code -- the error code to interpret *destination -- "the error message in quotes" length -- either 0 or strlen(destination) PAPI_perror fills the string, destination, with the error message corresponding to the error code (code). The function copies length worth of the error description string corresponding to code into destination. The resulting string is always null terminated. If length is 0, then the string is printed to stderr. PAPI_strerror returns a pointer to the error message corresponding to the error code (code). If the call fails, the function returns a NULL pointer. Otherwise, a nonNULL pointer is returned. Note that this function is not implemented in Fortran. In the following code example, PAPI_perror is used to convert error codes to error messages: #include #include main() { int EventSet = PAPI_NULL; int native = 0x0; char error_str[PAPI_MAX_STR_LEN]; /* Initialize the PAPI library */ retval = PAPI_library_init(PAPI_VER_CURRENT); if (retval != PAPI_VER_CURRENT && retval > 0) { fprintf(stderr,"PAPI library version mismatch!\n"); exit(1); } if ((retval = PAPI_create_eventset(&EventSet)) != PAPI_OK) { fprintf(stderr, "PAPI error %d: %s\n",retval, PAPI_strerror(retval)); exit(1); } /* Add Total Instructions Executed to our EventSet */ if ((retval = PAPI_add_event(&EventSet, PAPI_TOT_INS)) != PAPI_OK) { PAPI_perror(retval,error_str,PAPI_MAX_STR_LEN); fprintf(stderr,"PAPI_error %d: %s\n",retval,error_str); exit(1); } /* Add POWER4 native event */ retval = PAPI_event_name_to_code(“PM_LD_MISS_L1”, &native); if ((retval = PAPI_add_event(&EventSet, native)) != PAPI_OK) { /* Dump error string directly to stderr. */ PAPI_perror(retval,NULL,NULL); - 70 -PAPI User’s Guide Version 3.5.0 exit(1); } /* Start counting */ if ((retval = PAPI_start(EventSet)) != PAPI_OK) handle_error(retval); } OUTPUT: Invalid argument Notice that the above output was generated from the last call to PAPI_perror. On success, PAPI_perror returns PAPI_OK and on error, a non-zero error code is returned. - 71 -PAPI User’s Guide Version 3.5.0 FURTHER INFORMATION In addition to the information in this Users Guide on Programming with PAPI, a number of other information sources are available. Most of these are available on the PAPI website. Below we list some of these resources and indicate where they can be found. PAPI HOME PAGE The PAPI Project home page can be found at: http://icl.cs.utk.edu/papi/ PAPI MAILING LISTS PAPI has two mailing lists for users and developers. For more information about the mailing lists and to subscribe or modify your mailing list settings, click the “Mailing & User Lists” tab under “Contacts’ on the PAPI web page, or go directly to: http://icl.cs.utk.edu/papi/custom/index.html?lid=50&slid=67 REPORTING BUGS If you find a bug in PAPI, you can send mail to one of the mailing lists above, or peruse the mail archives to see if it has been reported by anyone else. You can also submit the bug directly to PAPIzilla, the PAPI bug tracking website. To access PAPIzilla, click the “Bug Reporting” tab under “Contacts’ on the PAPI web page, or go directly to: http://icl.cs.utk.edu/projects/papi/bugz/ PAPI PROGRAMMER’S REFERENCE Function by function documentation for PAPI can be found in a number of formats. PAPI man pages are part of the standard installation package. For a properly installed PAPI, you should be able to display a man page for any PAPI function by typing: > man PAPI_xxx where “PAPI_xxx” is a PAPI function name. In addition, this information is also available in HTML format on the PAPI website at: http://icl.cs.utk.edu/projects/papi/files/html_man3/papi.html - 72 -PAPI User’s Guide Version 3.5.0 If you want a printable version of the Programmer’s Reference, you can find it in Word and PDF formats under the ‘Documentation’ tab on the PAPI website. TABLE OF PRESET EVENTS A table of present events, their standard definitions, and the platforms on which they are defined can be found at: http://icl.cs.utk.edu/projects/papi/presets.html Note that tables inevitably become outdated. Always use the util/papi_avail utility for the most current definitions of preset events on your platform. SUPPORTED PLATFORMS A table of currently supported hardware platforms and operating systems can be found on the PAPI website under the “Supported Platforms” tab. SUPPORTED TOOLS A list of tools that support PAPI, along with a brief description of each tool and a link to further information, can be found under the “Tools” tab on the PAPI website. A second list of tools that support PAPI, along with a list of links to related projects and links to a variety of vendor documentation can be found on the PAPI website under the “Links” tab. HARDWARE REFERENCES A series of links to vendor and third party hardware documentation on performance counter resources can also be found on the PAPI website under the “Links” tab. - 73 -PAPI User’s Guide Version 3.5.0 BIBLIOGRAPHY Browne, S., J. Dongarra J., Garner N., London K., and Mucci, P., "A Portable Programming Interface for Performance Evaluation on Modern Processors,” University of Tennessee Technical Report, Knoxville, Tennessee, July 2000. http://icl.cs.utk.edu/papi/documents Browne, S., Dongarra J., Garner N., London K., and Mucci, P., "A Scalable CrossPlatform Infrastructure for Application Performance Tuning Using Hardware Counters,” Proc. SC'2000, November 2000. http://icl.cs.utk.edu/papi/documents Dongarra, J., London, K., Moore, S., Mucci, P., and Terpstra, D. "Using PAPI for Hardware Performance Monitoring on Linux Systems,” Conference on Linux Clusters: The HPC Revolution, Urbana, Illinois, June 25-27, 2001. http://icl.cs.utk.edu/papi/documents London, K., Moore, S., Mucci, P., Seymour, K., and Luczak, R. "The PAPI CrossPlatform Interface to Hardware Performance Counters,” Department of Defense Users' Group Conference Proceedings, Biloxi, Mississippi, June 18-21, 2001. http://icl.cs.utk.edu/papi/documens London, K., Dongarra, J., Moore, S., Mucci, P., Seymour, K., and Spencer, T. "Enduser Tools for Application Performance Analysis Using Hardware Counters,” International Conference on Parallel and Distributed Computing Systems, Dallas, TX, August 8-10, 2001. http://icl.cs.utk.edu/papi/documents Mucci, P., Moore, S., and Smeds, Nils. “Performance Tuning Using Hardware Counter Data,” Proc. SC’2001, November 2001. http://icl.cs.utk.edu/papi/documents Mucci, P. “The IA64 Hardware Performance Monitor and PAPI”, The 2001 International Conference on Parallel and Distributed Processing Techniques and Applications, June 2001. http://icl.cs.utk.edu/papi/documents Mucci, P. “PAPI -- The Performance Application Programming Interface,” April 2000. http://icl.cs.utk.edu/papi/documents “POSIX Threads Programming.” http://www.llnl.gov/computing/tutorials/workshops/workshop/pthreads/MAIN.html - 74 - SuperLU Users' Guide James W. Demmel 1 John R. Gilbert 2 Xiaoye S. Li 3 Septemb er, 1999 Last update: October, 2003 1 Computer Science Division, University of California, Berkeley, CA 94720. (demmel@cs.berkeley.edu). The research of Demmel and Li was supported in part by NSF grant ASC{9313958, DOE grant DE{FG03{ 94ER25219, UT Subcontract No. ORA4466 from ARPA Contract No. DAAL03{91{C0047, DOE grant DE{ FG03{94ER25206, and NSF Infrastructure grants CDA{8722788 and CDA{9401156. 2Department of Computer Science, University of California, Santa Barbara, CA 93106. (gilbert@cs.ucsb.edu). The research of this author was supported in part by the Institute for Mathematics and Its Applications at the University of Minnesota and in part by DARPA Contract No. DABT63-95-C0087. Copyright c 1994-1997 by Xerox Corporation. All rights reserved. 3 Lawrence Berkeley National Lab, MS 50F1650, 1 Cyclotron Rd, Berkeley, CA 94720. (xiaoye@nersc.gov). This work was supported in part by the Director, OÆce of Advanced Scientic Computing Research, Division of Mathematical, Information, and Computational Sciences of the U.S. Department of Energy under contract numb er DE-AC03-76SF00098.Contents 1 Introduction 3 1.1 Purpose of SuperLU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Overall Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 What the three libraries have in common . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3.1 Input and Output Data Formats . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3.2 Tuning Parameters for BLAS . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3.3 Performance Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3.4 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3.5 Ordering the Columns of A for Sparse Factors . . . . . . . . . . . . . . . . . 7 1.3.6 Iterative Renement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.3.7 Error Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3.8 Solving a Sequence of Related Linear Systems . . . . . . . . . . . . . . . . . . 9 1.3.9 Interfacing to other languages . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.4 How the three libraries dier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.4.1 Input and Output Data Formats . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.4.2 Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.4.3 Pivoting Strategies for Stability . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.4.4 Memory Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.4.5 Interfacing to other languages . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.5 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.6 Software Status and Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.7 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2 Sequential SuperLU (Version 3.0) 14 2.1 About SuperLU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2 How to call a SuperLU routine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.3 Matrix data structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.4 Options argument . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.5 Permutations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.5.1 Ordering for sparsity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.5.2 Partial pivoting with threshold . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.6 Symmetric Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.7 Memory management for L and U . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.8 User-callable routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.8.1 Driver routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.8.2 Computational routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.8.3 Utility routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 12.9 Matlab interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.10 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.10.1 File structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.10.2 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.10.3 Performance-tuning parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.11 Example programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.12 Calling from Fortran . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3 Multithreaded SuperLU (Version 1.1) 38 3.1 About SuperLU MT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.2 Storage types for L and U . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.3 User-callable routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.3.1 Driver routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.3.2 Computational routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.4 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.4.1 File structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.4.2 Performance issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.5 Example programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.6 Porting to other platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.6.1 Creating multiple threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.6.2 Use of mutexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4 Distributed SuperLU with MPI (Version 2.0) 46 4.1 About SuperLU DIST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.2 Formats of the input matrices A and B . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.2.1 Global input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.2.2 Distributed input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.3 Distributed data structures for L and U . . . . . . . . . . . . . . . . . . . . . . . . . 47 4.4 Process grid and MPI communicator . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4.4.1 2D process grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4.4.2 Arbitrary grouping of processes . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.5 Basic steps to solve a linear system . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 4.6 Algorithmic background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.7 User-callable routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 4.7.1 Driver routine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 4.7.2 Computational routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.7.3 Utility routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.8 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.8.1 File structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.8.2 Performance-tuning parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.9 Example programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.10 Fortran 90 Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.10.1 Callable functions in the Fortran 90 module le spuerlu mod.f90 . . . . . . . . 64 4.10.2 C wrapper functions callable by Fortran in le spuerlu c2f wrap.c . . . . . . . 65 2Chapter 1 Introduction 1.1 Purpose of SuperLU This document describes a collection of three related ANSI C subroutine libraries for solving sparse linear systems of equations AX = B. Here A is a square, nonsingular, n  n sparse matrix, and X and B are dense n  nrhs matrices, where nrhs is the numb er of right-hand sides and solution vectors. Matrix A need not b e symmetric or denite; indeed, SuperLU is particularly appropriate for matrices with very unsymmetric structure. All three libraries use variations of Gaussian elimination optimized to take advantage both of sparsity and the computer architecture, in particular memory hierarchies (caches) and parallelism. In this introduction we refer to all three libraries collectively as SuperLU. The three libraries within SuperLU are as follows. Detailed references are also given (see also [21]).  Sequential SuperLU is designed for sequential processors with one or more layers of memory hierarchy (caches) [5].  Multithreaded SuperLU (SuperLU MT) is designed for shared memory multiprocessors (SMPs), and can eectively use up to 16 or 32 parallel processors on suÆciently large matrices in order to speed up the computation [6].  Distributed SuperLU (SuperLU DIST) is designed for distributed memory parallel processors, using MPI [26] for interprocess communication. It can eectively use hundreds of parallel processors on suÆciently large matrices [23, 24]. Table 1.1 summarizes the current status of the software. All the routines are implemented in C, with parallel extensions using Pthreads (POSIX threads for shared-memory programming) or MPI (for distributed-memory programming). We provide Fortran interface for all three libraries. Sequential SuperLU SuperLU MT SuperLU DIST Platform serial shared-memory distributed-memory Language C C + Pthreads C + MPI (with Fortran interface) (or pragmas) Data type real/complex real real/complex single/double double double Table 1.1: SuperLU software status. 3The rest of the Introduction is organized as follows. Section 1.2 describes the high-level algorithm used by all three libraries, pointing out some common features and dierences. Section 1.3 describes the detailed algorithms, data structures, and interface issues common to all three routines. Section 1.4 describes how the three routines dier, emphasizing the dierences that most aect the user. Section 1.6 describes the software status, including planned developments, bug reporting, and licensing. 1.2 Overall Algorithm A simple description of sparse Gaussian elimination is as follows: 1. Compute a triangular factorization PrDrADcPc = LU. Here Dr and Dc are diagonal matrices to equilibrate the system, Pr and Pc are permutation matrices. Premultiplying A by Pr reorders the rows of A, and p ostmultiplying A by Pc reorders the columns of A. Pr and Pc are chosen to enhance sparsity, numerical stability, and parallelism. L is a unit lower triangular matrix (Lii = 1) and U is an upper triangular matrix. The factorization can also be applied to non-square matrices. 2. Solve AX = B by evaluating X = A1B = (D1 r P 1 r LUP 1 c D1 c ) 1B = Dc(Pc(U 1 (L 1 (Pr(DrB))))). This is done eÆciently by multiplying from right to left in the last expression: Scale the rows of B by Dr . Multiplying PrB means permuting the rows of DrB. Multiplying L 1 (PrDrB) means solving nrhs triangular systems of equations with matrix L by substitution. Similarly, multiplying U 1 (L 1 (PrDrB)) means solving triangular systems with U. The simplest implementation, used by the \simple driver routines" within SuperLU and Sup erLU MT, is as follows: Simple Driver Algorithm 1. Choose Pc to order the columns of A to increase the sparsity of the computed L and U factors, and hopefully increase parallelism (for SuperLU MT). 2. Compute the LU factorization of APc. SuperLU and SuperLU MT can perform dynamic pivoting of the rows during factorization for numerical stability, computing Pr, L and U at the same time. 3. Solve the system using Pr , Pc, L and U as described above. (Dr = Dc = I) The simple driver subroutines for double precision real data are called dgssv and pdgssv for SuperLU and SuperLU MT, respectively. The letter d in the subroutine names means double precision real; other options are s for single precision real, c for single precision complex, and z for double precision complex. The subroutine naming scheme is analogous to the one used in LAPACK [1]. SuperLU DIST does not include this simple driver. There is also an \expert driver subroutine" that can provide more accurate solutions, compute error bounds, and solve a sequence of related linear systems more economically. It is available in all three libraries. Expert Driver Algorithm 4 SuperLU Users’ Guide James W. Demmel 1 John R. Gilbert 2 Xiaoye S. Li 3 September 1999 Last update: June 2009 1 Computer Science Division, University of California, Berkeley, CA 94720. (demmel@cs.berkeley.edu). The research of Demmel and Li was supported in part by NSF grant ASC–9313958, DOE grant DE–FG03– 94ER25219, UT Subcontract No. ORA4466 from ARPA Contract No. DAAL03–91–C0047, DOE grant DE– FG03–94ER25206, and NSF Infrastructure grants CDA–8722788 and CDA–9401156. 2Department of Computer Science, University of California, Santa Barbara, CA 93106. (gilbert@cs.ucsb.edu). The research of this author was supported in part by the Institute for Mathematics and Its Applications at the University of Minnesota and in part by DARPA Contract No. DABT63-95-C0087. Copyright c 1994-1997 by Xerox Corporation. All rights reserved. 3 Lawrence Berkeley National Lab, MS 50F-1650, 1 Cyclotron Rd, Berkeley, CA 94720. (xsli@lbl.gov). This work was supported in part by the Director, O?ce of Advanced Scienti?c Computing Research, Division of Mathematical, Information, and Computational Sciences of the U.S. Department of Energy under contract number DE-AC03-05CH11231.Contents 1 Introduction 4 1.1 Purpose of SuperLU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 Overall Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 What the three libraries have in common . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3.1 Input and Output Data Formats . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3.2 Tuning Parameters for BLAS . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.3.3 Performance Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.3.4 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3.5 Ordering the Columns of A for Sparse Factors . . . . . . . . . . . . . . . . . 8 1.3.6 Iterative Re?nement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.3.7 Error Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.3.8 Solving a Sequence of Related Linear Systems . . . . . . . . . . . . . . . . . . 10 1.3.9 Interfacing to other languages . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.4 How the three libraries di?er . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.4.1 Input and Output Data Formats . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.4.2 Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.4.3 Pivoting Strategies for Stability . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.4.4 Memory Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.4.5 Interfacing to other languages . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.5 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.6 Software Status and Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.7 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2 Sequential SuperLU (Version 4.0) 16 2.1 About SuperLU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.2 How to call a SuperLU routine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.3 Matrix data structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.4 Options argument . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.5 Permutations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.5.1 Ordering for sparsity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.5.2 Partial pivoting with threshold . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.6 Symmetric Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.7 Incomplete LU factorization (ILU) preconditioner . . . . . . . . . . . . . . . . . . . . 26 2.8 Memory management for L and U . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.9 User-callable routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 12.9.1 Driver routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.9.2 Computational routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.9.3 Utility routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.10 Matlab interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.11 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.11.1 File structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.11.2 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.11.3 Performance-tuning parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.12 Example programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.13 Calling from Fortran . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3 Multithreaded SuperLU (Version 2.0) 42 3.1 About SuperLU MT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.2 Storage types for L and U . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.3 User-callable routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.3.1 Driver routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.3.2 Computational routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.4 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.4.1 File structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.4.2 Performance issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.5 Example programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.6 Porting to other platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.6.1 Creating multiple threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.6.2 Use of mutexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4 Distributed SuperLU with MPI (Version 2.3) 50 4.1 About SuperLU DIST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 4.2 Formats of the input matrices A and B . . . . . . . . . . . . . . . . . . . . . . . . . 50 4.2.1 Global input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 4.2.2 Distributed input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 4.3 Distributed data structures for L and U . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.4 Process grid and MPI communicator . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.4.1 2D process grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.4.2 Arbitrary grouping of processes . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.5 Basic steps to solve a linear system . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.6 Algorithmic background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.7 User-callable routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.7.1 Driver routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.7.2 Computational routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 4.7.3 Utility routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 4.8 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.8.1 File structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.8.2 Performance-tuning parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 65 4.9 Example programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 4.10 Fortran 90 Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.10.1 Callable functions in the Fortran 90 module ?le spuerlu mod.f90 . . . . . . . . 71 24.10.2 C wrapper functions callable by Fortran in ?le spuerlu c2f wrap.c . . . . . . . 72 3Chapter 1 Introduction 1.1 Purpose of SuperLU This document describes a collection of three related ANSI C subroutine libraries for solving sparse linear systems of equations AX = B. Here A is a square, nonsingular, n × n sparse matrix, and X and B are dense n × nrhs matrices, where nrhs is the number of right-hand sides and solution vectors. Matrix A need not be symmetric or de?nite; indeed, SuperLU is particularly appropriate for matrices with very unsymmetric structure. All three libraries use variations of Gaussian elimination optimized to take advantage both of sparsity and the computer architecture, in particular memory hierarchies (caches) and parallelism. In this introduction we refer to all three libraries collectively as SuperLU. The three libraries within SuperLU are as follows. Detailed references are also given (see also [21]). • Sequential SuperLU is designed for sequential processors with one or more layers of memory hierarchy (caches) [5]. • Multithreaded SuperLU (SuperLU MT) is designed for shared memory multiprocessors (SMPs), and can e?ectively use up to 16 or 32 parallel processors on su?ciently large matrices in order to speed up the computation [6]. • Distributed SuperLU (SuperLU DIST) is designed for distributed memory parallel processors, using MPI [27] for interprocess communication. It can e?ectively use hundreds of parallel processors on su?ciently large matrices [23, 24]. Table 1.1 summarizes the current status of the software. All the routines are implemented in C, with parallel extensions using Pthreads (POSIX threads for shared-memory programming) or MPI (for distributed-memory programming). We provide Fortran interface for all three libraries. The rest of the Introduction is organized as follows. Section 1.2 describes the high-level algorithm used by all three libraries, pointing out some common features and di?erences. Section 1.3 describes the detailed algorithms, data structures, and interface issues common to all three routines. Section 1.4 describes how the three routines di?er, emphasizing the di?erences that most a?ect the user. Section 1.6 describes the software status, including planned developments, bug reporting, and licensing. 4Sequential SuperLU SuperLU MT SuperLU DIST Platform serial shared-memory distributed-memory Language C C + Pthreads C + MPI (with Fortran interface) (or OpenMP) Data type real/complex real/complex real/complex single/double single/double double Table 1.1: SuperLU software status. 1.2 Overall Algorithm A simple description of the algorithm for solving linear equations by sparse Gaussian elimination is as follows: 1. Compute a triangular factorization PrDrADcPc = LU. Here Dr and Dc are diagonal matrices to equilibrate the system, Pr and Pc are permutation matrices. Premultiplying A by Pr reorders the rows of A, and postmultiplying A by Pc reorders the columns of A. Pr and Pc are chosen to enhance sparsity, numerical stability, and parallelism. L is a unit lower triangular matrix (Lii = 1) and U is an upper triangular matrix. The factorization can also be applied to non-square matrices. 2. Solve AX = B by evaluating X = A-1B = (D-1 r P -1 r LUP -1 c D-1 c ) -1B = Dc(Pc(U -1 (L -1 (Pr(DrB))))). This is done e?ciently by multiplying from right to left in the last expression: Scale the rows of B by Dr. Multiplying PrB means permuting the rows of DrB. Multiplying L -1 (PrDrB) means solving nrhs triangular systems of equations with matrix L by substitution. Similarly, multiplying U -1 (L -1 (PrDrB)) means solving triangular systems with U. In addition to complete factorization, we also have limited support for incomplete factorization (ILU) preconditioner. The simplest implementation, used by the “simple driver routines” within SuperLU and SuperLU MT, is as follows: Simple Driver Algorithm 1. Choose Pc to order the columns of A to increase the sparsity of the computed L and U factors, and hopefully increase parallelism (for SuperLU MT). 2. Compute the LU factorization of APc. SuperLU and SuperLU MT can perform dynamic pivoting of the rows during factorization for numerical stability, computing Pr, L and U at the same time. 3. Solve the system using Pr, Pc, L and U as described above. (Dr = Dc = I) The simple driver subroutines for double precision real data are called dgssv and pdgssv for SuperLU and SuperLU MT, respectively. The letter d in the subroutine names means double precision real; other options are s for single precision real, c for single precision complex, and z for double precision complex. The subroutine naming scheme is analogous to the one used in LAPACK [1]. SuperLU DIST does not include this simple driver. 5There is also an “expert driver subroutine” that can provide more accurate solutions, compute error bounds, and solve a sequence of related linear systems more economically. It is available in all three libraries. Expert Driver Algorithm 1. Equilibrate the matrix A, i.e. compute diagonal matrices Dr and Dc so that Aˆ = DrADc is “better conditioned” than A, i.e. Aˆ-1 is less sensitive to perturbations in Aˆ than A-1 is to perturbations in A. 2. Preorder the rows of Aˆ (SuperLU DIST only), i.e. replace Aˆ by PrAˆ where Pr is a permutation matrix. We call this step “static pivoting”, and it is only done in the distributed memory algorithm. 3. Order the columns of Aˆ to increase the sparsity of the computed L and U factors, and hopefully increase parallelism (for SuperLU MT and SuperLU DIST). In other words, replace Aˆ by APˆ T c in SuperLU and SuperLU MT, or replace Aˆ by PcAPˆ T c in SuperLU DIST, where Pc is a permutation matrix. 4. Compute the LU factorization of Aˆ . SuperLU and SuperLU MT can perform dynamic pivoting of the rows during factorization for numerical stability. In contrast, SuperLU DIST uses the order computed by the preordering step but replaces tiny pivots by larger numbers for stability. 5. Solve the system using the computed triangular factors. 6. Iteratively re?ne the solution, again using the computed triangular factors. This is equivalent to Newton’s method. 7. Compute error bounds. Both forward and backward error bounds are computed, as described below. The expert driver subroutines for double precision real data are called dgssvx, pdgssvx and pdgssvx for SuperLU, SuperLU MT and SuperLU DIST, respectively. Sequential SuperLU also provides single precision real (s), single precision complex (c), and double precision complex (z) versions. SuperLU MT only provides double precision real (d). SuperLU DIST provides both double precision real (d) and complex (z). The driver routines are composed of several lower level computational routines for computing permutations, computing LU factorization, solving triangular systems, and so on. The LU factorization routine for all three libraries also handles nonsquare matrices. For large matrices, the LU factorization steps takes most of the time, although choosing Pc to order the columns can also be time-consuming. 1.3 What the three libraries have in common 1.3.1 Input and Output Data Formats Sequential SuperLU and SuperLU MT accept A and B as single precision real, double precision real, and both single and double precision complex. SuperLU DIST accepts double precision real or complex. 6A is stored in a sparse data structure according to the struct SuperMatrix, which is described in section 3.2. In particular, A may be supplied in either column-compressed format (“HarwellBoeing format”), or row-compressed format (i.e. AT stored in column-compressed format). B, which is overwritten by the solution X, is stored as a dense matrix in column-major order. In SuperLU DIST, A and B can be either replicated or distributed across all processes. (The storage of L and U di?ers among the three libraries, as discussed in section 1.4.) 1.3.2 Tuning Parameters for BLAS All three libraries depend on having high performance BLAS (Basic Linear Algebra Subroutine) libraries [20, 8, 7] in order to get high performance. In particular, they depend on matrix-vector multiplication or matrix-matrix multiplication of relatively small dense matrices. The sizes of these small dense matrices can be tuned to match the “sweet spot” of the BLAS by setting certain tuning parameters described in section 2.11.3 for SuperLU, in section 3.4.2 for SuperLU MT, and in section 4.8.2 for SuperLU DIST. (In addition, SuperLU MT and SuperLU DIST let one control the number of parallel processes to be used, as described in section 1.4.) 1.3.3 Performance Statistics Most of the computational routines use a struct to record certain kinds of performance data, namely the time and number of ?oating point operations in each phase of the computation, and data about the sizes of the matrices L and U. These statistics are collected during the computation. A statistic variable is declared with the following type: typedef struct { int *panel_histo; /* histogram of panel size distribution */ double *utime; /* time spent in various phases */ float *ops; /* floating-point operations at various phases */ int TinyPivots; /* number of tiny pivots */ int RefineSteps; /* number of iterative refinement steps */ } SuperLUStat_t; For both SuperLU and SuperLU MT, there is only one copy of these statistics variable. But for SuperLU DIST, each process keeps a local copy of this variable, and records its local statistics. We need to use MPI reduction routines to ?nd any global information, such as the sum of the ?oating-point operation count on all processes. Before the computation, routine StatInit() should be called to malloc storage and perform initialization for the ?elds panel histo, utime, and ops. The algorithmic phases are de?ned by the enumeration type PhaseType in SRC/util.h. In the end, routine StatFree() should be called to free storage of the above statistics ?elds. After deallocation, the statistics are no longer accessible. Therefore, users should extract the information they need before calling StatFree(), which can be accomplished by calling (P)StatPrint(). An inquiry function dQuerySpace() is provided to compute memory usage statistics. This routine should be called after the LU factorization. It calculates the storage requirement based on the size of the L and U data structures and working arrays. 71.3.4 Error Handling Invalid arguments and (P)XERBLA Similar to LAPACK, for all the SuperLU routines, we check the validity of the input arguments to each routine. If an illegal value is supplied to one of the input arguments, the error handler XERBLA is called, and a message is written to the standard output, indicating which argument has an illegal value. The program returns immediately from the routine, with a negative value of INFO. Computational failures with INFO > 0 A positive value of INFO on return from a routine indicates a failure in the course of the computation, such as a matrix being singular, or the amount of memory (in bytes) already allocated when malloc fails. ABORT on unrecoverable errors A macro ABORT is de?ned in SRC/util.h to handle unrecoverable errors that occur in the middle of the computation, such as malloc failure. The default action of ABORT is to call superlu abort and exit(char *msg) which prints an error message, the line number and the ?le name at which the error occurs, and calls the exit function to terminate the program. If this type of termination is not appropriate in some environment, users can alter the behavior of the abort function. When compiling the SuperLU library, users may choose the C preprocessor de?nition -DUSER ABORT = my abort At the same time, users would supply the following my abort function my abort(char *msg) which overrides the behavior of superlu abort and exit. 1.3.5 Ordering the Columns of A for Sparse Factors There is a choice of orderings for the columns of A both in the simple or expert driver, in section 1.2: • Natural ordering, • Multiple Minimum Degree (MMD) [26] applied to the structure of AT A, • Multiple Minimum Degree (MMD) [26] applied to the structure of AT + A, • Column Approximate Minimum Degree (COLAMD) [4], and • Use a Pc supplied by the user as input. COLAMD is designed particularly for unsymmetric matrices when partial pivoting is needed, and does not require explicit formation of AT A. It usually gives comparable orderings as MMD on AT A, and is faster. The orderings based on graph partitioning heuristics are also popular, as exempli?ed in the MeTiS package [18]. The user can simply input this ordering in the permutation vector for Pc. Note 8that many graph partitioning algorithms are designed for symmetric matrices. The user may still apply them to the structures of AT A or AT + A. Our routines getata() and at plus a in the ?le get perm c.c can be used to form AT A or AT + A. 1.3.6 Iterative Re?nement Step 6 of the expert driver algorithm, iterative re?nement, serves to increase accuracy of the computed solution. Given the initial approximate solution x from step 5, the algorithm for step 6 is as follows (where x and b are single columns of X and B, respectively): Compute residual r = Ax - b While residual too large Solve Ad = r for correction d Update solution x = x - d Update residual r = Ax - b end while If r and then d were computed exactly, the updated solution x - d would be the exact solution. Roundo? prevents immediate convergence. The criterion “residual too large” in the iterative re?nement algorithm above is essentially that BERR = max i |ri |/si (1.1) exceeds the machine roundo? level, or is continuing to decrease quickly enough. Here si is the scale factor si = (|A|
|x| + |b|)i = X j |Aij |
|xj | + |bi | In this expression |A| is the n-by-n matrix with entries |A| ij = |Aij |, |b| and |x| are similarly column vectors of absolute entries of b and x, respectively, and |A|
|x| is conventional matrixvector multiplication. The purpose of this stopping criterion is explained in the next section. 1.3.7 Error Bounds Step 7 of the expert driver algorithm computes error bounds. It is shown in [2, 28] that BERR de?ned in Equation 1.1 measures the componentwise relative backward error of the computed solution. This means that the computed x satis?es a slightly perturbed linear system of equations (A + E)x = b + f, where |Eij | = BERR
|Aij | and |fi | = BERR
|bi | for all i and j. It is shown in [2, 32] that one step of iterative re?nement usually reduces BERR to near machine epsilon. For example, if BERR is 4 times machine epsilon, then the computed solution x is identical to the solution one would get by changing each nonzero entry of A and b by at most 4 units in their last places, and then solving this perturbed system exactly. If the nonzero entries of A and b are uncertain in their bottom 2 bits, then one should generally not expect a more accurate solution. Thus BERR is a measure of backward error speci?cally suited to solving sparse linear systems of equations. Despite roundo?, BERR itself is always computed to within about ±n times machine epsilon (and usually much more accurately) and so BERR is quite accurate. 9In addition to backward error, the expert driver computes a forward error bound F ERR = kxtrue - xk8/kxk8 Here kxk8 = maxi |xi |. Thus, if F ERR = 10 -6 then each component of x has an error bounded by about 10 -6 times the largest component of x. The algorithm used to compute F ERR is an approximation; see [2, 17] for a discussion. Generally F ERR is accurate to within a factor of 10 or better, which is adequate to say how many digits of the large entries of x are correct. (SuperLU DIST’s algorithm for F ERR is slightly less reliable [24].) 1.3.8 Solving a Sequence of Related Linear Systems It is very common to solve a sequence of related linear systems A(1)X(1) = B(1) , A(2)X(2) = B(2) , ... rather than just one. When A(1) and A(2) are similar enough in sparsity pattern and/or numerical entries, it is possible to save some of the work done when solving with A(1) to solve with A(2) . This can result in signi?cant savings. Here are the options, in increasing order of “reuse of prior information”: 1. Factor from scratch. No previous information is used. If one were solving just one linear system, or a sequence of unrelated linear systems, this is the option to use. 2. Reuse Pc, the column permutation. The user may save the column permutation and reuse it. This is most useful when A(2) has the same sparsity structure as A(1) , but not necessarily the same (or similar) numerical entries. Reusing Pc saves the sometimes quite expensive operation of computing it. 3. Reuse Pc, Pr and data structures allocated for L and U. If Pr and Pc do not change, then the work of building the data structures associated with L and U (including the elimination tree [14]) can be avoided. This is most useful when A(2) has the same sparsity structure and similar numerical entries as A(1) . When the numerical entries are not similar, one can still use this option, but at a higher risk of numerical instability (BERR will always report whether or not the solution was computed stably, so one cannot get an unstable answer without warning). 4. Reuse Pc, Pr, L and U. In other words, we reuse essentially everything. This is most commonly used when A(2) = A(1) , but B(2) =6 B(1) , i.e. when only the right-hand sides di?er. It could also be used when A(2) and A(1) di?ered just slightly in numerical values, in the hopes that iterative re?nement converges (using A(2) to compute residuals but the triangular factorization of A(1) to solve). Because of the di?erent ways L and U are computed and stored in the three libraries, these 4 options are speci?ed slightly di?erently; see Chapters 2 through 4 for details. 1.3.9 Interfacing to other languages It is possible to call all the drivers and the computational routines from Fortran. However, currently the Fortran wrapper functions are not complete. The users are expected to look at the Fortran example programs in the FORTRAN/ directory, together with the C “bridge” routine, and learn how to call SuperLU from a Fortran program. The users can modify the C bridge routine to ?t their needs. 101.4 How the three libraries di?er 1.4.1 Input and Output Data Formats All Sequential SuperLU and SuperLU MT routines are available in single and double precision (real or complex), but SuperLU DIST routines are available only in double precision (real or complex). L and U are stored in di?erent formats in the three libraries: • L and U in Sequential SuperLU. L is a “column-supernodal” matrix, in storage type SCformat. This means it is stored sparsely, with supernodes (consecutive columns with identical structures) stored as dense blocks. U is stored in column-compressed format NCformat. See section 2.3 for details. • L and U in SuperLU MT. Because of parallelism, the columns of L and U may not be computed in consecutive order, so they may be allocated and stored out of order. This means we use the “column-supernodal-permuted” format SCPformat for L and “column-permuted” format NCPformat for U. See section 3.2 for details. • L and U in SuperLU DIST. Now L and U are distributed across multiple processors. As described in detail in Sections 4.3 and 4.4, we use a 2D block-cyclic format, which has been used for dense matrices in libraries like ScaLAPACK [3]. But for sparse matrices, the blocks are no longer identical in size, and vary depending on the sparsity structure of L and U. The detailed storage format is discussed in section 4.3 and illustrated in Figure 4.1. 1.4.2 Parallelism Sequential SuperLU has no explicit parallelism. Some parallelism may still be exploited on an SMP by using a multithreaded BLAS library if available. But it is likely to be more e?ective to use SuperLU MT on an SMP, described next. SuperLU MT lets the user choose the number of parallel threads to use. The mechanism varies from platform to platform and is described in section 3.6. SuperLU DIST not only lets the user specify the number of processors, but how they are arranged into a 2D grid. Furthermore, MPI permits any subset of the processors allocated to the user may be used for SuperLU DIST, not just consecutively numbered processors (say 0 through P-1). See section 4.4 for details. 1.4.3 Pivoting Strategies for Stability Sequential SuperLU and SuperLU MT use the same pivoting strategy, called threshold pivoting, to determine the row permutation Pr. Suppose we have factored the ?rst i - 1 columns of A, and are seeking the pivot for column i. Let ami be a largest entry in magnitude on or below the diagonal of the partially factored A: |ami | = maxj=i |aji |. Depending on a threshold 0 < u = 1 input by the user, the code will use the diagonal entry aii as the pivot in column i as long as |aii | = u
|ami |, and otherwise use ami . So if the user sets u = 1, ami (or an equally large entry) will be selected as the pivot; this corresponds to the classical partial pivoting strategy. If the user has ordered the matrix so that choosing diagonal pivots is particularly good for sparsity or parallelism, then smaller values of u will tend to choose those diagonal pivots, at the risk of less numerical stability. Using u = 0 11guarantees that the pivots on the diagonal will be chosen, unless they are zero. The error bound BERR measure how much stability is actually lost. Threshold pivoting turns out to be hard to parallelize on distributed memory machines, because of the ?ne-grain communication and dynamic data structures required. So SuperLU DIST uses a new scheme called static pivoting instead. In static pivoting the pivot order (Pr) is chosen before numerical factorization, using a weighted perfect matching algorithm [9], and kept ?xed during factorization. Since both row and column orders (Pr and Pc) are ?xed before numerical factorization, we can extensively optimize the data layout, load balance, and communication schedule. The price is a higher risk of numeric instability, which is mitigated by diagonal scaling, setting very tiny pivots to larger values, and iterative re?nement [24]. Again, error bound BERR measure how much stability is actually lost. 1.4.4 Memory Management Because of ?ll-in of entries during Gaussian elimination, L and U typically have many more nonzero entries than A. If Pr and Pc are not already known, we cannot determine the number and locations of these nonzeros before performing the numerical factorization. This means that some kind of dynamic memory allocation is needed. Sequential SuperLU lets the user either supply a preallocated space work[] of length lwork, or depend on malloc/free. The variable FILL can be used to help the code predict the amount of ?ll, which can reduce both fragmentation and the number of calls to malloc/free. If the initial estimate of the size of L and U from FILL is too small, the routine allocates more space and copies the current L and U factors to the new space and frees the old space. If the routine cannot allocate enough space, it calls a user-speci?able routine ABORT. See sections 1.3.4 for details. SuperLU MT is similar, except that the current alpha version cannot reallocate more space for L and U if the initial size estimate from FILL is too small. Instead, the program calls ABORT and the user must start over with a larger value of FILL. See section 3.4.2. SuperLU DIST actually has a simpler memory management chore, because once Pr and Pc are determined, the structures of L and U can be determined e?ciently and just the right amount of memory allocated using malloc and later free. So it will call ABORT only if there is really not enough memory available to solve the problem. 1.4.5 Interfacing to other languages Sequential SuperLU has a Matlab interface to the driver via a MEX ?le. See section 2.10 for details. 1.5 Performance SuperLU library incorporates a number of novel algorithmic ideas developed recently. These algorithms also exploit the features of modern computer architectures, in particular, the multi-level cache organization and parallelism. We have conducted extensive experiments on various platforms, with a large collection of test matrices. The Sequential SuperLU achieved up to 40% of the theoretical ?oating-point rate on a number of processors, see [5, 21]. The mega?op rate usually increases with increasing ratio of ?oating-point operations count over the number of nonzeros in the L and U factors. The parallel LU factorization in SuperLU MT demonstrated 5–10 fold speedups on a range of commercially popular SMPs, and up to 2.5 Giga?ops factorization rate, see [6, 21]. 12The parallel LU factorization in SuperLU DIST achieved up to 100 fold speedup on a 512-processor Cray T3E, and 10.2 Giga?ops factorization rate, see [23]. 1.6 Software Status and Availability All three libraries are freely available for all uses, commercial or noncommercial, subject to the following caveats. No warranty is expressed or implied by the authors, although we will gladly answer questions and try to ?x all reported bugs. We ask that proper credit be given to the authors and that a notice be included if any modi?cations are made. The following Copyright applies to the whole SuperLU software. Copyright (c) 2003, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from U.S. Dept. of Energy) All rights reserved. Redistribution and use in source and binary forms, with or without modi?cation, are permitted provided that the following conditions are met: (1) Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. (2) Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. (3) Neither the name of Lawrence Berkeley National Laboratory, U.S. Dept. of Energy nor the names of its contributors may be used to endorse or promote products derived from this software without speci?c prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ”AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Some routines carry the additional notices as follows. 1. Some subroutines carry the following notice: Copyright (c) 1994 by Xerox Corporation. All rights reserved. THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY EXPRESSED OR IMPLIED. ANY USE IS AT YOUR OWN RISK. 13Permission is hereby granted to use or copy this program for any purpose, provided the above notices are retained on all copies. Permission to modify the code and to distribute modi?ed code is granted, provided the above notices are retained, and a notice that the code was modi?ed is included with the above copyright notice. 2. The MC64 routine (only used in SuperLU DIST) carries the following notice: COPYRIGHT (c) 1999 Council for the Central Laboratory of the Research Councils. All rights reserved. PACKAGE MC64A/AD AUTHORS Iain Du? (i.du?@rl.ac.uk) and Jacko Koster (jak@ii.uib.no) LAST UPDATE 20/09/99 *** Conditions on external use *** The user shall acknowledge the contribution of this package in any publication of material dependent upon the use of the package. The user shall use reasonable endeavours to notify the authors of the package of this publication. The user can modify this code but, at no time shall the right or title to all or any part of this package pass to the user. The user shall make available free of charge to the authors for any purpose all information relating to any alteration or addition made to this package for the purposes of extending the capabilities or enhancing the performance of this package. The user shall not pass this code directly to a third party without the express prior consent of the authors. Users wanting to licence their own copy of these routines should send email to hsl@aeat.co.uk None of the comments from the Copyright notice up to and including this one shall be removed or altered in any way. All three libraries can be obtained from the following URLs: http://crd.lbl.gov/~xiaoye/SuperLU/ http://www.netlib.org/scalapack/prototype/ In the future, we will add more functionality in the software, such as sequential and parallel incomplete LU factorizations, as well as parallel symbolic and ordering algorithms for SuperLU DIST; these latter routines would replace MC64 and have no restrictions on external use. All bugs reports and queries can be e-mailed to xsli@lbl.gov and demmel@cs.berkeley.edu. 1.7 Acknowledgement With great gratitude, we acknowledge Stan Eisenstat and Joesph Liu for their signi?cant contributions to the development of Sequential SuperLU. Meiyue Shao helped the development of the incomplete factorization ILU routines in sequential SuperLU. We would like to thank Jinqchong Teo for helping generate the code in Sequential SuperLU to work with four ?oating-point data types, and Daniel Schreiber for doing this with SuperLU MT. Yu Wang and William F. Mitchell developed the Fortran 90 interface for SuperLU DIST. Laura Grigori developed the parallel symbolic factorization code for SuperLU DIST. We thank Tim Davis for his contribution of some subroutines related to column ordering and suggestions on improving the routines’ interfaces. We thank Ed Rothberg of Silicon Graphics for 14discussions and providing us access to the SGI Power Challenge during the SuperLU MT development. We acknowledge the following organizations that provided the computer resources during our code development: NERSC at Lawrence Berkeley National Laboratory, Livermore Computing at Lawrence Livermore National Laboratory, NCSA at University of Illinois at Urbana-Champaign, Silicon Graphics, and Xerox Palo Alto Research Center. We thank UC Berkeley and NSF Infrastructure grant CDA-9401156 for providing Berkeley NOW. 15Chapter 2 Sequential SuperLU (Version 4.0) 2.1 About SuperLU In this chapter, SuperLU will always mean Sequential SuperLU. SuperLU package contains a set of subroutines to solve sparse linear systems AX = B. Here A is a square, nonsingular, n × n sparse matrix, and X and B are dense n × nrhs matrices, where nrhs is the number of right-hand sides and solution vectors. Matrix A need not be symmetric or de?nite; indeed, SuperLU is particularly appropriate for matrices with very unsymmetric structure. The package uses LU decomposition with partial (or threshold) pivoting, and forward/back substitutions. The columns of A may be preordered before factorization (either by the user or by SuperLU); this preordering for sparsity is completely separate from the factorization. To improve backward stability, we provide working precision iterative re?nement subroutines [2]. Routines are also available to equilibrate the system, estimate the condition number, calculate the relative backward error, and estimate error bounds for the re?ned solutions. We also include a Matlab MEX-?le interface, so that our factor and solve routines can be called as alternatives to those built into Matlab. The LU factorization routines can handle non-square matrices, but the triangular solves are performed only for square matrices. Starting from Version 4.0, we provide the incomplete factorization ILU routines which can be used as preconditioners for iterative solvers. The factorization algorithm uses a graph reduction technique to reduce graph traversal time in the symbolic analysis. We exploit dense submatrices in the numerical kernel, and organize computational loops in a way that reduces data movement between levels of the memory hierarchy. The resulting algorithm is highly e?cient on modern architectures. The performance gains are particularly evident for large problems. There are “tuning parameters” to optimize the peak performance as a function of cache size. For a detailed description of the algorithm, see reference [5]. SuperLU is implemented in ANSI C, and must be compiled with a standard ANSI C compiler. It includes versions for both real and complex matrices, in both single and double precision. The ?le names for the single-precision real version start with letter “s” (such as sgstrf.c); the ?le names for the double-precision real version start with letter “d” (such as dgstrf.c); the ?le names for the single-precision complex version start with letter “c” (such as cgstrf.c); the ?le names for the double-precision complex version start with letter “z” (such as zgstrf.c). 16? ? s u u l u l p e u l l r ? ? ? ? 19.00 21.00 21.00 0.63 21.00 -13.26 -13.26 0.57 23.58 7.58 5.00 21.00 0.63 0.57 -0.24 -0.77 34.20 ? ? Original matrix A Factors F = L + U - I s = 19, u = 21, p = 16, e = 5, r = 18, l = 12 Figure 2.1: A 5 × 5 matrix and its L and U factors. 2.2 How to call a SuperLU routine As a simple example, let us consider how to solve a 5×5 sparse linear system AX = B, by calling a driver routine dgssv(). Figure 2.1 shows matrix A, and its L and U factors. This sample program is located in SuperLU/EXAMPLE/superlu.c. The program ?rst initializes the three arrays, a[], asub[] and xa[], which store the nonzero coe?cients of matrix A, their row indices, and the indices indicating the beginning of each column in the coe?cient and row index arrays. This storage format is called compressed column format, also known as Harwell-Boeing format [10]. Next, the two utility routines dCreate CompCol Matrix() and dCreate Dense Matrix() are called to set up the matrix structures for A and B, respectively. The routine set default options() sets the default values to the input options argument. This controls how the matrix will be factorized and how the system will be solved. After calling the SuperLU routine dgssv(), the B matrix is overwritten by the solution matrix X. In the end, all the dynamically allocated data structures are de-allocated by calling various utility routines. SuperLU can perform more general tasks, which will be explained later. #include "dsp_defs.h" main(int argc, char *argv[]) { /* * Purpose * ======= * * This is the small 5x5 example used in the Sections 1 and 2 of the * User’s Guide to illustrate how to call a SuperLU routine, and the * matrix data structures used by SuperLU. * */ SuperMatrix A, L, U, B; double *a, *rhs; double s, u, p, e, r, l; int *asub, *xa; int *perm_r; /* row permutations from partial pivoting */ 17int *perm_c; /* column permutation vector */ int nrhs, info, i, m, n, nnz, permc_spec; superlu_options_t options; SuperLUStat_t stat; /* Initialize matrix A. */ m = n = 5; nnz = 12; if ( !(a = doubleMalloc(nnz)) ) ABORT("Malloc fails for a[]."); if ( !(asub = intMalloc(nnz)) ) ABORT("Malloc fails for asub[]."); if ( !(xa = intMalloc(n+1)) ) ABORT("Malloc fails for xa[]."); s = 19.0; u = 21.0; p = 16.0; e = 5.0; r = 18.0; l = 12.0; a[0] = s; a[1] = l; a[2] = l; a[3] = u; a[4] = l; a[5] = l; a[6] = u; a[7] = p; a[8] = u; a[9] = e; a[10]= u; a[11]= r; asub[0] = 0; asub[1] = 1; asub[2] = 4; asub[3] = 1; asub[4] = 2; asub[5] = 4; asub[6] = 0; asub[7] = 2; asub[8] = 0; asub[9] = 3; asub[10]= 3; asub[11]= 4; xa[0] = 0; xa[1] = 3; xa[2] = 6; xa[3] = 8; xa[4] = 10; xa[5] = 12; /* Create matrix A in the format expected by SuperLU. */ dCreate_CompCol_Matrix(&A, m, n, nnz, a, asub, xa, SLU_NC, SLU_D, SLU_GE); /* Create right-hand side matrix B. */ nrhs = 1; if ( !(rhs = doubleMalloc(m * nrhs)) ) ABORT("Malloc fails for rhs[]."); for (i = 0; i < m; ++i) rhs[i] = 1.0; dCreate_Dense_Matrix(&B, m, nrhs, rhs, m, SLU_DN, SLU_D, SLU_GE); if ( !(perm_r = intMalloc(m)) ) ABORT("Malloc fails for perm_r[]."); if ( !(perm_c = intMalloc(n)) ) ABORT("Malloc fails for perm_c[]."); /* Set the default input options. */ set_default_options(&options); options.ColPerm = NATURAL; /* Initialize the statistics variables. */ StatInit(&stat); dgssv(&options, &A, perm_c, perm_r, &L, &U, &B, &stat, &info); dPrint_CompCol_Matrix("A", &A); dPrint_CompCol_Matrix("U", &U); dPrint_SuperNode_Matrix("L", &L); print_int_vec("\nperm_r", m, perm_r); 18/* De-allocate storage */ SUPERLU_FREE (rhs); SUPERLU_FREE (perm_r); SUPERLU_FREE (perm_c); Destroy_CompCol_Matrix(&A); Destroy_SuperMatrix_Store(&B); Destroy_SuperNode_Matrix(&L); Destroy_CompCol_Matrix(&U); StatFree(&stat); } 2.3 Matrix data structures SuperLU uses a principal data structure SuperMatrix (de?ned in SRC/supermatrix.h) to represent a general matrix, sparse or dense. Figure 2.2 gives the speci?cation of the SuperMatrix structure. The SuperMatrix structure contains two levels of ?elds. The ?rst level de?nes all the properties of a matrix which are independent of how it is stored in memory. In particular, it speci?es the following three orthogonal properties: storage type (Stype) indicates the type of the storage scheme in *Store; data type (Dtype) encodes the four precisions; mathematical type (Mtype) speci?es some mathematical properties. The second level (*Store) points to the actual storage used to store the matrix. We associate with each Stype XX a storage format called XXformat, such as NCformat, SCformat, etc. The SuperMatrix type so de?ned can accommodate various types of matrix structures and appropriate operations to be applied on them, although currently SuperLU implements only a subset of this collection. Speci?cally, matrices A, L, U, B, and X can have the following types: A L U B X Stype SLU NC or SLU NR SLU SC SLU NC SLU DN SLU DN Dtype 1 any any any any any Mtype SLU GE SLU TRLU SLU TRU SLU GE SLU GE In what follows, we illustrate the storage schemes de?ned by Stype. Following C’s convention, all array indices and locations below are zero-based. • A may have storage type SLU NC or SLU NR. The SLU NC format is the same as the HarwellBoeing sparse matrix format [10], that is, the compressed column storage. typedef struct { int nnz; /* number of nonzeros in the matrix */ void *nzval; /* array of nonzero values packed by column */ int *rowind; /* array of row indices of the nonzeros */ int *colptr; /* colptr[j] stores the location in nzval[] and rowind[] which starts column j. It has ncol+1 entries, and colptr[ncol] = nnz. */ } NCformat; 1 Dtype can be one of SLU S, SLU D, SLU C or SLU Z. 19typedef struct { Stype_t Stype; /* Storage type: indicates the storage format of *Store. */ Dtype_t Dtype; /* Data type. */ Mtype_t Mtype; /* Mathematical type */ int nrow; /* number of rows */ int ncol; /* number of columns */ void *Store; /* pointer to the actual storage of the matrix */ } SuperMatrix; typedef enum { SLU_NC, /* column-wise, not supernodal */ SLU_NR, /* row-wise, not supernodal */ SLU_SC, /* column-wise, supernodal */ SLU_SR, /* row-wise, supernodal */ SLU_NCP, /* column-wise, not supernodal, permuted by columns (After column permutation, the consecutive columns of nonzeros may not be stored contiguously. */ SLU_DN, /* Fortran style column-wise storage for dense matrix */ SLU_NR_loc /* distributed compressed row format */ } Stype_t; typedef enum { SLU_S, /* single */ SLU_D, /* double */ SLU_C, /* single-complex */ SLU_Z /* double-complex */ } Dtype_t; typedef enum { SLU_GE, /* general */ SLU_TRLU, /* lower triangular, unit diagonal */ SLU_TRUU, /* upper triangular, unit diagonal */ SLU_TRL, /* lower triangular */ SLU_TRU, /* upper triangular */ SLU_SYL, /* symmetric, store lower half */ SLU_SYU, /* symmetric, store upper half */ SLU_HEL, /* Hermitian, store lower half */ SLU_HEU /* Hermitian, store upper half */ } Mtype_t; Figure 2.2: SuperMatrix data structure. 20The SLU NR format is the compressed row storage de?ned below. typedef struct { int nnz; /* number of nonzeros in the matrix */ void *nzval; /* array of nonzero values packed by row */ int *colind; /* array of column indices of the nonzeros */ int *rowptr; /* rowptr[j] stores the location in nzval[] and colind[] which starts row j. It has nrow+1 entries, and rowptr[nrow] = nnz. */ } NRformat; The factorization and solve routines in SuperLU are designed to handle column-wise storage only. If the input matrix A is in row-oriented storage, i.e., in SLU NR format, then the driver routines (dgssv() and dgssvx()) actually perform the LU decomposition on AT , which is column-wise, and solve the system using the L T and U T factors. The data structures holding L and U on output are di?erent (swapped) from the data structures you get from column-wise input. For more detailed descriptions about this process, please refer to the leading comments of the routines dgssv() and dgssvx(). Alternatively, the users may call a utility routine dCompRow to CompCol() to convert the input matrix in SLU NR format to another matrix in SLU NC format, before calling SuperLU. The de?nition of this routine is void dCompRow_to_CompCol(int m, int n, int nnz, double *a, int *colind, int *rowptr, double **at, int **rowind, int **colptr); This conversion takes time proportional to the number of nonzeros in A. However, it requires storage for a separate copy of matrix A. • L is a supernodal matrix with the storage type SLU SC. Due to the supernodal structure, L is in fact stored as a sparse block lower triangular matrix [5]. typedef struct { int nnz; /* number of nonzeros in the matrix */ int nsuper; /* index of the last supernode */ void *nzval; /* array of nonzero values packed by column */ int *nzval_colptr; /* nzval_colptr[j] stores the location in nzval[] which starts column j */ int *rowind; /* array of compressed row indices of rectangular supernodes */ int *rowind_colptr;/* rowind_colptr[j] stores the location in rowind[] which starts column j */ int *col_to_sup; /* col_to_sup[j] is the supernode number to which column j belongs */ int *sup_to_col; /* sup_to_col[s] points to the starting column of the s-th supernode */ } SCformat; 21• Both B and X are stored as conventional two-dimensional arrays in column-major order, with the storage type SLU DN. typedef struct { int lda; /* leading dimension */ void *nzval; /* array of size lda-by-ncol to represent a dense matrix */ } DNformat; Figure 2.3 shows the data structures for the example matrices in Figure 2.1. For a description of NCPformat, see section 2.5.1. 2.4 Options argument Options Argument The options argument is the input argument to control the behaviour of the libraries. The user can tell the solvers how the linear systems should be solved based on some known characteristics of the system. For example, for diagonally dominant matrices, choosing the diagonal pivots ensures stability; there is no need for numerical pivoting (i.e., Pr can be an Identity matrix). In another situation where a sequence of matrices with the same sparsity pattern need be factorized, the column permutation Pc (and also the row permutation Pr, if the numerical values are similar) need be computed only once, and reused thereafter. In these cases, the solvers’ performance can be much improved over using the default settings. Options is implemented as a C structure containing the following ?elds: • Fact Speci?es whether or not the factored form of the matrix A is supplied on entry, and if not, how the matrix A will be factorized base on the previous history, such as factor from scratch, reuse Pc and/or Pr, or reuse the data structures of L and U. • Trans Speci?es whether to solve the transposed system. • Equil Speci?es whether to equilibrate the system (scale A’s rows and columns to have unit norm). • ColPerm Speci?es how to permute the columns of the matrix for sparsity preservation. • IterRefine Speci?es whether to perform iterative re?nement, and in what precision to compute the residual. • SymmetricMode Speci?es whether to use the symmetric mode. • DiagPivotThresh Speci?es the threshold used for a diagonal entry to be an acceptable pivot. 22• A = { Stype = SLU_NC; Dtype = SLU_D; Mtype = SLU_GE; nrow = 5; ncol = 5; *Store = { nnz = 12; nzval = [ 19.00, 12.00, 12.00, 21.00, 12.00, 12.00, 21.00, 16.00, 21.00, 5.00, 21.00, 18.00 ]; rowind = [ 0, 1, 4, 1, 2, 4, 0, 2, 0, 3, 3, 4 ]; colptr = [ 0, 3, 6, 8, 10, 12 ]; } } • U = { Stype = SLU_NC; Dtype = SLU_D; Mtype = SLU_TRU; nrow = 5; ncol = 5; *Store = { nnz = 11; nzval = [ 21.00, -13.26, 7.58, 21.00 ]; rowind = [ 0, 1, 2, 0 ]; colptr = [ 0, 0, 0, 1, 4, 4 ]; } } • L = { Stype = SLU_SC; Dtype = SLU_D; Mtype = SLU_TRLU; nrow = 5; ncol = 5; *Store = { nnz = 11; nsuper = 2; nzval = [ 19.00, 0.63, 0.63, 21.00, 0.57, 0.57, -13.26, 23.58, -0.24, 5.00, -0.77, 21.00, 34.20 ]; nzval_colptr = [ 0 3, 6, 9, 11, 13 ]; rowind = [ 0, 1, 4, 1, 2, 4, 3, 4 ]; rowind_colptr = [ 0, 3, 6, 6, 8, 8 ]; col_to_sup = [ 0, 1, 1, 2, 2 ]; sup_to_col = [ 0, 1, 3, 5 ]; } } Figure 2.3: The data structures for a 5 × 5 matrix and its LU factors, as represented in the SuperMatrix data structure. Zero-based indexing is used. 23• ILU DropRule Speci?es the dropping rules for ILU. • ILU DropTol Speci?es the numerical dropping threshold for ILU. • ILU FillFactor Speci?es the expected ?ll ratio upper bound for ILU. • ILU MILU Speci?es which version of modi?ed ILU to use. • PrintStat Speci?es whether to print the solver’s statistics. The routine set default options() sets the following default values: Fact = DOFACT /* factor from scratch */ Trans = NOTRANS Equil = YES ColPerm = COLAMD SymmetricMode = NO DiagPivotThresh = 1.0 /* partial pivoting */ IterRefine = NOREFINE PrintStat = YES To use the ILU routines, such as dgsitrf(), the user should call ilu set default options() to set the default values: ILU_DropRule = DROP_BASIC | DROP_AREA; ILU_DropTol = 1e-4; ILU_FillFactor = 10.0; ILU_MILU = SMILU_2; DiagPivotThresh = 0.1; The other possible values for each ?eld are documented in the source code SRC/slu util.h. The users can reset each default value according to their needs. 2.5 Permutations Two permutation matrices are involved in the solution process. In fact, the actual factorization we perform is PrAP T c = LU, where Pr is determined from partial pivoting (with a threshold pivoting option), and Pc is a column permutation chosen either by the user or SuperLU, usually to make the L and U factors as sparse as possible. Pr and Pc are represented by two integer vectors perm r[] and perm c[], which are the permutations of the integers (0 : m - 1) and (0 : n - 1), respectively. 242.5.1 Ordering for sparsity Column reordering for sparsity is completely separate from the LU factorization. The column permutation Pc should be applied before calling the factorization routine dgstrf(). In principle, any ordering heuristic used for symmetric matrices can be applied to AT A (or A+AT if the matrix is nearly structurally symmetric) to obtain Pc. Currently, we provide the following ordering options through options argument. The options.ColPerm ?eld can take the following values: • NATURAL: use natural ordefring (i.e., Pc = I). • MMD AT PLUS A: use minimum degree ordering on the structure of AT + A. • MMD ATA: use minimum degree ordering on the structure of AT A. • COLAMD: use approximate minimum degree column ordering. • MY PERMC: use the ordering given in the permutation vector perm c[], which is input by the user. If options.ColPerm is set to the last value, the library will use the permutation vector perm c[] as an input, which may be obtained from any other ordering algorithm. For example, the nesteddissection type of ordering codes include Metis [18], Chaco [16] and Scotch [29]. Alternatively, the users can provide their own column permutation vector. For example, it may be an ordering suitable for the underlying physical problem. Both driver routines dgssv and dgssvx take perm c[] as an input argument. After permutation Pc is applied to A, we use SLU NCP format to represent the permuted matrix AP T c , in which the consecutive columns of nonzeros may not be stored contiguously in memory. Therefore, we need two separate arrays of pointers, colbeg[] and colend[], to indicate the beginning and end of each column in nzval[] and rowind[]. typedef struct { int nnz; /* number of nonzeros in the matrix */ void *nzval; /* array of nonzero values, packed by column */ int *rowind; /* array of row indices of the nonzeros */ int *colbeg; /* colbeg[j] points to the location in nzval[] and rowind[] which starts column j */ int *colend; /* colend[j] points to one past the location in nzval[] and rowind[] which ends column j */ } NCPformat; 2.5.2 Partial pivoting with threshold We have included a threshold pivoting parameter u ? [0, 1] to control numerical stability. The user can choose to use a row permutation obtained from a previous factorization. (The argument options.Fact = SamePattern SameRowPerm should be passed to the factorization routine dgstrf().) The pivoting subroutine dpivotL() checks whether this choice of pivot satis?es the threshold; if not, it will try the diagonal element. If neither of the above satis?es the threshold, the maximum magnitude element in the column will be used as the pivot. The pseudo-code of the pivoting policy for column j is given below. 25(1) compute thresh = u |amj |, where |amj | = maxi=j |aij |; (2) if user speci?es pivot row k and |akj | = thresh and akj = 0 6 then pivot row = k; else if |ajj | = thresh and ajj = 0 6 then pivot row = j; else pivot row = m; endif; Two special values of u result in the following two strategies: • u = 0.0: either use user-speci?ed pivot order if available, or else use diagonal pivot; • u = 1.0: classical partial pivoting. 2.6 Symmetric Mode In many applications, matrix A may be diagonally dominant or nearly so. In this case, pivoting on the diagonal is su?cient for stability and is preferable for sparsity to o?-diagonal pivoting. To do this, the user can set a small (less-than-one) diagonal pivot threshold (e.g., 0.0, 0.01) and choose an (AT + A)–based column permutation algorithm. We call this setting symmetric mode. In this case, the options.SymmetricMode = YES must be set. Note that, when a diagonal entry is smaller than the threshold, the code will still choose an o?-diagonal pivot. That is, the row permutation Pr may not be Identity. Please refer to [22] for more discussion on the symmetric mode. 2.7 Incomplete LU factorization (ILU) preconditioner Starting from SuperLU version 4.0, we provide the ILU routines to be used as preconditioners for iterative solvers. Our ILU method can be considered to be a variant of the ILUTP method originally proposed by Saad [31], which combines a dual dropping strategy with numerical pivoting (“T” stands for threshold, and “P” stands for pivoting). We adapted the classic dropping strategies of ILUTP in order to incorporate supernode structures and to accommodate dynamic supernodes due to partial pivoting. For the secondary dropping strategy, we proposed an area-based ?ll control method, which is more ?exible and numerically robust than the traditional column-based scheme. Furthermore, we incorporated several heuristics for adaptively modifying various threshold parameters as the factorization proceeds, which improves the robustness of the algorithm. The details can be found in [25]. 2.8 Memory management for L and U In the sparse LU algorithm, the amount of space needed to hold the data structures of L and U cannot be accurately predicted prior to the factorization. The dynamically growing arrays include those for the nonzero values (nzval[]) and the compressed row indices (rowind[]) of L, and for the nonzero values (nzval[]) and the row indices (rowind[]) of U. 26Two alternative memory models are presented to the user: • system-level – based on C’s dynamic allocation capability (malloc/free); • user-level – based on a user-supplied work[] array of size lwork (in bytes). This is similar to Fortran-style handling of work space. Work[] is organized as a two-ended stack, one end holding the L and U data structures, the other end holding the auxiliary arrays of known size. Except for the di?erent ways to allocate/deallocate space, the logical view of the memory organization is the same for both schemes. Now we describe the policies in the memory module. At the outset of the factorization, we guess there will be FILL*nnz(A) ?lls in the factors and allocate corresponding storage for the above four arrays, where nnz(A) is the number of nonzeros in original matrix A, and FILL is an integer, say 20. (The value of FILL can be set in an inquiry function sp ienv(), see section 2.11.3.) If this initial request exceeds the physical memory constraint, the FILL factor is repeatedly reduced, and attempts are made to allocate smaller arrays, until the initial allocation succeeds. During the factorization, if any array size exceeds the allocated bound, we expand it as follows. We ?rst allocate a chunk of new memory of size EXPAND times the old size, then copy the existing data into the new memory, and then free the old storage. The extra copying is necessary, because the factorization algorithm requires that each of the aforementioned four data structures be contiguous in memory. The values of FILL and EXPAND are normally set to 20 and 1.5, respectively. See xmemory.c for details. After factorization, we do not garbage-collect the extra space that may have been allocated. Thus, there will be external fragmentation in the L and U data structures. The settings of FILL and EXPAND should take into account the trade-o? between the number of expansions and the amount of fragmentation. Arrays of known size, such as various column pointers and working arrays, are allocated just once. All dynamically-allocated working arrays are freed after factorization. 2.9 User-callable routines The naming conventions, calling sequences and functionality of these routines mimic the corresponding LAPACK software [1]. In the routine names, such as dgstrf, we use the two letters GS to denote general sparse matrices. The leading letterx stands for S, D, C, or Z, specifying the data type. 2.9.1 Driver routines We provide two types of driver routines for solving systems of linear equations. The driver routines can handle both column- and row-oriented storage schemes. • A simple driver dgssv(), which solves the system AX = B by factorizing A and overwriting B with the solution X. • An expert driver dgssvx(), which, in addition to the above, also performs the following functions (some of them optionally): 27– solve ATX = B; – equilibrate the system (scale A’s rows and columns to have unit norm) if A is poorly scaled; – estimate the condition number of A, check for near-singularity, and check for pivot growth; – re?ne the solution and compute forward and backward error bounds. • An expert driver dgsisx(), which gives the approximate solutions of linear equations AX = B or ATX = B, using the ILU factorization from dgsitrf(). An estimation of the condition number is provide, and the pivot growth is computed. These driver routines cover all the functionality of the computational routines. We expect that most users can simply use these driver routines to ful?ll their tasks with no need to bother with the computational routines. 2.9.2 Computational routines The users can invoke the following computational routines, instead of the driver routines, to directly control the behavior of SuperLU. The computational routines can only handle column-oriented storage. • dgstrf(): Factorize. This implements the ?rst-time factorization, or later re-factorization with the same nonzero pattern. In re-factorizations, the code has the ability to use the same column permutation Pc and row permutation Pr obtained from a previous factorization. The input argument options contains several scalar arguments to control how the LU decomposition and the numerical pivoting should be performed. dgstrf() can handle non-square matrices. • dgsitrf(): ILU. This implements the incomplete LU factorization The input argument options contains several scalar arguments to control how the incomplete facotirzation and the numerical pivoting should be performed. • dgstrs(): Triangular solve. This takes the L and U triangular factors, the row and column permutation vectors, and the right-hand side to compute a solution matrix X of AX = B or ATX = B. • dgscon(): Estimate condition number. Given the matrix A and its factors L and U, this estimates the condition number in the one-norm or in?nity-norm. The algorithm is due to Hager and Higham [17], and is the same as CONDEST in sparse Matlab. • dgsequ()/dlaqgs(): Equilibrate. dgsequ ?rst computes the row and column scalings Dr and Dc which would make each row and each column of the scaled matrix DrADc have equal norm. dlaqgs then applies them to the original matrix A if it is indeed badly scaled. The equilibrated A overwrites the original A. 28• dgsrfs(): Re?ne solution. Given A, its factors L and U, and an initial solution X, this does iterative re?nement, using the same precision as the input data. It also computes forward and backward error bounds for the re?ned solution. 2.9.3 Utility routines The utility routines can help users create and destroy the SuperLU matrices easily. These routines reside in two places: SRC/util.c contains the routines that are precision-independent; SRC/{s,d,c,z}util.c contains the routines dependent on precision. Here, we list the prototypes of these routines. /* Create a supermatrix in compressed column format. A is the output. */ dCreate_CompCol_Matrix(SuperMatrix *A, int m, int n, int nnz, double *nzval, int *rowind, int *colptr, Stype_t stype, Dtype_t dtype, Mtype_t mtype); /* Create a supermatrix in compressed row format. A is the output. */ dCreate_CompRow_Matrix(SuperMatrix *A, int m, int n, int nnz, double *nzval, int *colind, int *rowptr, Stype_t stype, Dtype_t dtype, Mtype_t mtype); /* Copy matrix A into matrix B, both in compressed column format. */ dCopy_CompCol_Matrix(SuperMatrix *A, SuperMatrix *B); /* Create a supermatrix in dense format. X is the output.*/ dCreate_Dense_Matrix(SuperMatrix *X, int m, int n, double *x, int ldx, Stype_t stype, Dtype_t dtype, Mtype_t mtype); /* Create a supermatrix in supernodal format. L is the output. */ dCreate_SuperNode_Matrix(SuperMatrix *L, int m, int n, int nnz, double *nzval, int *nzval_colptr, int *rowind, int *rowind_colptr, int *col_to_sup, int *sup_to_col, Stype_t stype, Dtype_t dtype, Mtype_t mtype); /* Convert the compressed row fromat to the compressed column format. */ dCompRow_to_CompCol(int m, int n, int nnz, double *a, int *colind, int *rowptr, double **at, int **rowind, int **colptr); /* Print a supermatrix in compressed column format. */ dPrint_CompCol_Matrix(char *what, SuperMatrix *A); /* Print a supermatrix in supernodal format. */ dPrint_SuperNode_Matrix(char *what, SuperMatrix *A); 29/* Print a supermatrix in dense format. */ dPrint_Dense_Matrix(char *what, SuperMatrix *A); /* Deallocate the storage structure *Store. */ Destroy_SuperMatrix_Store(SuperMatrix *A); /* Deallocate the supermatrix structure in compressed column format. */ Destroy_CompCol_Matrix(SuperMatrix *A) /* Deallocate the supermatrix structure in supernodal format. */ Destroy_SuperNode_Matrix(SuperMatrix *A) /* Deallocate the supermatrix structure in permuted compressed column format. */ Destroy_CompCol_Permuted(SuperMatrix *A) /* Deallocate the supermatrix structure in dense format. */ Destroy_Dense_Matrix(SuperMatrix *A) 2.10 Matlab interface In the SuperLU/MATLAB subdirectory, we have developed a set of MEX-?les interface to Matlab. Typing make in this directory produces executables to be invoked in Matlab. The current Makefile is set up so that the MEX-?les are compatible with Matlab Version 5. The user should edit Makefile for Matlab Version 4 compatibility. Right now, only the factor routine dgstrf() and the simple driver routine dgssv() are callable by invoking superlu and lusolve in Matlab, respectively. Superlu and lusolve correspond to the two Matlab built-in functions lu and \ . In Matlab, when you type help superlu you will ?nd the following description about superlu’s functionality and how to use it. SUPERLU : Supernodal LU factorization Executive summary: [L,U,p] = superlu(A) is like [L,U,P] = lu(A), but faster. [L,U,prow,pcol] = superlu(A) preorders the columns of A by min degree, yielding A(prow,pcol) = L*U. Details and options: With one input and two or three outputs, SUPERLU has the same effect as LU, except that the pivoting permutation is returned as a vector, not a matrix: [L,U,p] = superlu(A) returns unit lower triangular L, upper triangular U, and permutation vector p with A(p,:) = L*U. [L,U] = superlu(A) returns permuted triangular L and upper triangular U 30with A = L*U. With a second input, the columns of A are permuted before factoring: [L,U,prow] = superlu(A,psparse) returns triangular L and U and permutation prow with A(prow,psparse) = L*U. [L,U] = superlu(A,psparse) returns permuted triangular L and triangular U with A(:,psparse) = L*U. Here psparse will normally be a user-supplied permutation matrix or vector to be applied to the columns of A for sparsity. COLMMD is one way to get such a permutation; see below to make SUPERLU compute it automatically. (If psparse is a permutation matrix, the matrix factored is A*psparse’.) With a fourth output, a column permutation is computed and applied: [L,U,prow,pcol] = superlu(A,psparse) returns triangular L and U and permutations prow and pcol with A(prow,pcol) = L*U. Here psparse is a user-supplied column permutation for sparsity, and the matrix factored is A(:,psparse) (or A*psparse’ if the input is a permutation matrix). Output pcol is a permutation that first performs psparse, then postorders the etree of the column intersection graph of A. The postorder does not affect sparsity, but makes supernodes in L consecutive. [L,U,prow,pcol] = superlu(A,0) is the same as ... = superlu(A,I); it does not permute for sparsity but it does postorder the etree. [L,U,prow,pcol] = superlu(A) is the same as ... = superlu(A,colmmd(A)); it uses column minimum degree to permute columns for sparsity, then postorders the etree and factors. For a description about lusolve’s functionality and how to use it, you can type help lusolve LUSOLVE : Solve linear systems by supernodal LU factorization. x = lusolve(A, b) returns the solution to the linear system A*x = b, using a supernodal LU factorization that is faster than Matlab’s builtin LU. This m-file just calls a mex routine to do the work. By default, A is preordered by column minimum degree before factorization. Optionally, the user can supply a desired column ordering: x = lusolve(A, b, pcol) uses pcol as a column permutation. It still returns x = A\b, but it factors A(:,pcol) (if pcol is a permutation vector) or A*Pcol (if Pcol is a permutation matrix). x = lusolve(A, b, 0) suppresses the default minimum degree ordering; 31that is, it forces the identity permutation on columns. Two M-?les trysuperlu.m and trylusolve.m are written to test the correctness of superlu and lusolve. In addition to testing the residual norms, they also test the function invocations with various number of input/output arguments. 2.11 Installation 2.11.1 File structure The top level SuperLU/ directory is structured as follows: SuperLU/README instructions on installation SuperLU/CBLAS/ needed BLAS routines in C, not necessarily fast SuperLU/EXAMPLE/ example programs SuperLU/INSTALL/ test machine dependent parameters; this Users’ Guide SuperLU/MAKE_INC/ sample machine-specific make.inc files SuperLU/MATLAB/ Matlab mex-file interface SuperLU/SRC/ C source code, to be compiled into the superlu.a library SuperLU/TESTING/ driver routines to test correctness SuperLU/Makefile top level Makefile that does installation and testing SuperLU/make.inc compiler, compile flags, library definitions and C preprocessor definitions, included in all Makefiles Before installing the package, you may need to edit SuperLU/make.inc for your system. This make include ?le is referenced inside each of the Makefiles in the various subdirectories. As a result, there is no need to edit the Makefiles in the subdirectories. All information that is machine speci?c has been de?ned in make.inc. Sample machine-speci?c make.inc are provided in the MAKE INC/ subdirectory for several systems, including IBM RS/6000, DEC Alpha, SunOS 4.x, SunOS 5.x (Solaris), HP-PA and SGI Iris 4.x. When you have selected the machine on which you wish to install SuperLU, you may copy the appropriate sample include ?le (if one is present) into make.inc. For example, if you wish to run SuperLU on an IBM RS/6000, you can do: cp MAKE INC/make.rs6k make.inc For systems other than those listed above, slight modi?cations to the make.inc ?le will need to be made. In particular, the following three items should be examined: 1. The BLAS library. If there is a BLAS library available on your machine, you may de?ne the following in make.inc: BLASDEF = -DUSE VENDOR BLAS BLASLIB = The CBLAS/ subdirectory contains the part of the C BLAS needed by the SuperLU package. However, these codes are intended for use only if there is no faster implementation of the BLAS already available on your machine. In this case, you should do the following: 1) In make.inc, unde?ne (comment out) BLASDEF, de?ne: BLASLIB = ../blas$(PLAT).a 322) In the SuperLU/ directory, type: make blaslib to make the BLAS library from the routines in the CBLAS/ subdirectory. 2. C preprocessor de?nition CDEFS. In the header ?le SRC/Cnames.h, we use macros to determine how C routines should be named so that they are callable by Fortran. 2 The possible options for CDEFS are: • -DAdd : Fortran expects a C routine to have an underscore post?xed to the name; • -DNoChange: Fortran expects a C routine name to be identical to that compiled by C; • -DUpCase: Fortran expects a C routine name to be all uppercase. 3. The Matlab MEX-?le interface. The MATLAB/ subdirectory includes Matlab C MEX-?les, so that our factor and solve routines can be called as alternatives to those built into Matlab. In the ?le SuperLU/make.inc, de?ne MATLAB to be the directory in which Matlab is installed on your system, for example: MATLAB = /usr/local/matlab At the SuperLU/ directory, type: make matlabmex to build the MEX-?le interface. After you have built the interface, you may go to the MATLAB/ subdirectory to test the correctness by typing (in Matlab): trysuperlu trylusolve A Makefile is provided in each subdirectory. The installation can be done completely automatically by simply typing make at the top level. 2.11.2 Testing The test programs in SuperLU/INSTALL subdirectory test two routines: • slamch()/dlamch() determines properties of the ?oating-point arithmetic at run-time (both single and double precision), such as the machine epsilon, under?ow threshold, over?ow threshold, and related parameters; • SuperLU timer () returns the time in seconds used by the process. This function may need to be modi?ed to run on your machine. The test programs in the SuperLU/TESTING subdirectory are designed to test all the functions of the driver routines, especially the expert drivers. The Unix shell script ?les xtest.csh are used to invoke tests with varying parameter settings. The input matrices include an actual sparse matrix SuperLU/EXAMPLE/g10 of dimension 100 × 100, 3 and numerous matrices with special properties from the LAPACK test suite. Table 2.1 describes the properties of the test matrices. 2 Some vendor-supplied BLAS libraries do not have C interfaces. So the re-naming is needed in order for the SuperLU BLAS calls (in C) to interface with the Fortran-style BLAS. 3Matrix g10 is ?rst generated with the structure of the 10-by-10 ?ve-point grid, and random numerical values. The columns are then permuted by COLMMD ordering from Matlab. 33Matrix type Description 0 sparse matrix g10 1 diagonal 2 upper triangular 3 lower triangular 4 random, ? = 2 5 ?rst column zero 6 last column zero 7 last n/2 columns zero 8 random, ? = p 0.1/e 9 random, ? = 0.1/e 10 scaled near under?ow 11 scaled near over?ow Table 2.1: Properties of the test matrices. e is the machine epsilon and ? is the condition number of matrix A. Matrix types with one or more columns set to zero are used to test the error return codes. Test Type Test ratio Routines 0 ||LU - A||/(n||A||e) dgstrf 1 ||b - Ax||/(||A|| ||x||e) dgssv, dgssvx 2 ||x - x * ||/(||x * ||?e) dgssvx 3 ||x - x * ||/(||x * || F ERR) dgssvx 4 BERR/e dgssvx Table 2.2: Types of tests. x * is the true solution, F ERR is the error bound, and BERR is the backward error. For each command line option speci?ed in dtest.csh, the test program ddrive reads in or generates an appropriate matrix, calls the driver routines, and computes a number of test ratios to verify that each operation has performed correctly. If the test ratio is smaller than a preset threshold, the operation is considered to be correct. Each test matrix is subject to the tests listed in Table 2.2. Let r be the residual r = b - Ax, and let mi be the number of nonzeros in row i of A. Then the componentwise backward error BERR and forward error F ERR [1] are calculated by: BERR = max i |r| i (|A| |x| + |b|)i . F ERR = || |A-1 | f ||8 ||x||8 . Here, f is a nonnegative vector whose components are computed as fi = |r| i + mi e (|A| |x| + |b|)i , and the norm in the numerator is estimated using the same subroutine used for estimating the condition number. BERR measures the smallest relative perturbation one can make to each entry of A and of b so that the computed solution is an exact solution of the perturbed problem. F ERR is an estimated bound on the error kx * - xk8/kxk8, where x * is the true solution. For further details on error analysis and error bounds estimation, see [1, Chapter 4] and [2]. 2.11.3 Performance-tuning parameters SuperLU chooses such machine-dependent parameters as block size by calling an inquiry function sp ienv(), which may be set to return di?erent values on di?erent machines. The declaration of this function is 34On-chip External Machine Cache Cache w maxsup rowblk colblk RS/6000-590 256 KB – 8 100 200 40 MIPS R8000 16 KB 4 MB 20 100 800 100 Alpha 21064 8 KB 512 KB 8 100 400 40 Alpha 21164 8 KB-L1 4 MB 16 50 100 40 96 KB-L2 Sparc 20 16 KB 1 MB 8 100 400 50 UltraSparc-I 16 KB 512 KB 8 100 400 40 Cray J90 – – 1 100 1000 100 Table 2.3: Typical blocking parameter values for several machines. int sp ienv(int ispec); Ispec speci?es the parameter to be returned, (See reference [5] for their de?nitions.) ispec = 1: the panel size (w) = 2: the relaxation parameter to control supernode amalgamation (relax) = 3: the maximum allowable size for a supernode (maxsup) = 4: the minimum row dimension for 2D blocking to be used (rowblk) = 5: the minimum column dimension for 2D blocking to be used (colblk) = 6: the estimated ?lls factor for L and U, compared with A Users are encouraged to modify this subroutine to set the tuning parameters for their own local environment. The optimal values depend mainly on the cache size and the BLAS speed. If your system has a very small cache, or if you want to e?ciently utilize the closest cache in a multilevel cache organization, you should pay special attention to these parameter settings. In our technical paper [5], we described a detailed methodology for setting these parameters for high performance. The relax parameter is usually set between 4 and 8. The other parameter values which give good performance on several machines are listed in Table 2.3. In a supernode-panel update, if the updating supernode is too large to ?t in cache, then a 2D block partitioning of the supernode is used, in which rowblk and colblk determine that a block of size rowblk × colblk is used to update current panel. If colblk is set greater than maxsup, then the program will never use 2D blocking. For example, for the Cray J90 (which does not have cache), w = 1 and 1D blocking give good performance; more levels of blocking only increase overhead. 2.12 Example programs In the SuperLU/EXAMPLE/ subdirectory, we present a few sample programs to illustrate how to use various functions provded in SuperLU. The users can modify these examples to suit their applications. Here are the brief descriptions of the double precision version of the examples: • dlinsol: use simple driver dgssv() to solve a linear system one time. • dlinsol1: use simple driver dgssv() in the symmetric mode. • dlinsolx: use dgssvx() with the full (default) set of options to solve a linear system. 35• dlinsolx1: use dgssvx() to factorize A ?rst, then solve the system later. • dlinsolx2: use dgssvx() to solve systems repeatedly with the same sparsity pattern of matrix A. • superlu: the small 5x5 sample program in Section 2.2. In this directory, a Makefile is provided to generate the executables, and a README ?le describes how to run these examples. 2.13 Calling from Fortran The SuperLU/FORTRAN/ subdirectory contains an example of using SuperLU from a Fortran program. The General rules for mixing Fortran and C programs are as follows. • Arguments in C are passed by value, while in Fortran are passed by reference. So we always pass the address (as a pointer) in the C calling routine. (You cannot make a call with numbers directly in the parameters.) • Fortran uses 1-based array addressing, while C uses 0-based. Therefore, the row indices (rowind[]) and the integer pointers to arrays (colptr[]) should be adjusted before they are passed into a C routine. Because of the above language di?erences, in order to embed SuperLU in a Fortran environment, users are required to use “wrapper” routines (in C) for all the SuperLU routines that will be called from Fortran programs. The example c fortran dgssv.c in the FORTRAN/ directory shows how a wrapper program should be written. This program is listed below. #include "dsp_defs.h" #define HANDLE_SIZE 8 typedef struct { SuperMatrix *L; SuperMatrix *U; int *perm_c; int *perm_r; } factors_t; int c_fortran_dgssv_(int *iopt, int *n, int *nnz, int *nrhs, double *values, int *rowind, int *colptr, double *b, int *ldb, int factors[HANDLE_SIZE], /* a handle containing the pointer to the factored matrices */ int *info) { 36/* * This routine can be called from Fortran. * * iopt (input) int * Specifies the operation: * = 1, performs LU decomposition for the first time * = 2, performs triangular solve * = 3, free all the storage in the end * * factors (input/output) integer array of size 8 * If iopt == 1, it is an output and contains the pointer pointing to * the structure of the factored matrices. * Otherwise, it it an input. * */ SuperMatrix A, AC, B; SuperMatrix *L, *U; int *perm_r; /* row permutations from partial pivoting */ int *perm_c; /* column permutation vector */ int *etree; /* column elimination tree */ SCformat *Lstore; NCformat *Ustore; int i, panel_size, permc_spec, relax; trans_t trans; double drop_tol = 0.0; mem_usage_t mem_usage; superlu_options_t options; SuperLUStat_t stat; factors_t *LUfactors; trans = NOTRANS; if ( *iopt == 1 ) { /* LU decomposition */ /* Set the default input options. */ set_default_options(&options); /* Initialize the statistics variables. */ StatInit(&stat); /* Adjust to 0-based indexing */ for (i = 0; i < *nnz; ++i) --rowind[i]; for (i = 0; i <= *n; ++i) --colptr[i]; dCreate_CompCol_Matrix(&A, *n, *n, *nnz, values, rowind, colptr, 37SLU_NC, SLU_D, SLU_GE); L = (SuperMatrix *) SUPERLU_MALLOC( sizeof(SuperMatrix) ); U = (SuperMatrix *) SUPERLU_MALLOC( sizeof(SuperMatrix) ); if ( !(perm_r = intMalloc(*n)) ) ABORT("Malloc fails for perm_r[]."); if ( !(perm_c = intMalloc(*n)) ) ABORT("Malloc fails for perm_c[]."); if ( !(etree = intMalloc(*n)) ) ABORT("Malloc fails for etree[]."); /* * Get column permutation vector perm_c[], according to permc_spec: * permc_spec = 0: natural ordering * permc_spec = 1: minimum degree on structure of A’*A * permc_spec = 2: minimum degree on structure of A’+A * permc_spec = 3: approximate minimum degree for unsymmetric matrices */ permc_spec = 3; get_perm_c(permc_spec, &A, perm_c); sp_preorder(&options, &A, perm_c, etree, &AC); panel_size = sp_ienv(1); relax = sp_ienv(2); dgstrf(&options, &AC, drop_tol, relax, panel_size, etree, NULL, 0, perm_c, perm_r, L, U, &stat, info); if ( *info == 0 ) { Lstore = (SCformat *) L->Store; Ustore = (NCformat *) U->Store; printf("No of nonzeros in factor L = %d\n", Lstore->nnz); printf("No of nonzeros in factor U = %d\n", Ustore->nnz); printf("No of nonzeros in L+U = %d\n", Lstore->nnz + Ustore->nnz); dQuerySpace(L, U, &mem_usage); printf("L\\U MB %.3f\ttotal MB needed %.3f\texpansions %d\n", mem_usage.for_lu/1e6, mem_usage.total_needed/1e6, mem_usage.expansions); } else { printf("dgstrf() error returns INFO= %d\n", *info); if ( *info <= *n ) { /* factorization completes */ dQuerySpace(L, U, &mem_usage); printf("L\\U MB %.3f\ttotal MB needed %.3f\texpansions %d\n", mem_usage.for_lu/1e6, mem_usage.total_needed/1e6, mem_usage.expansions); } } 38/* Restore to 1-based indexing */ for (i = 0; i < *nnz; ++i) ++rowind[i]; for (i = 0; i <= *n; ++i) ++colptr[i]; /* Save the LU factors in the factors handle */ LUfactors = (factors_t*) SUPERLU_MALLOC(sizeof(factors_t)); LUfactors->L = L; LUfactors->U = U; LUfactors->perm_c = perm_c; LUfactors->perm_r = perm_r; factors[0] = (int) LUfactors; /* Free un-wanted storage */ SUPERLU_FREE(etree); Destroy_SuperMatrix_Store(&A); Destroy_CompCol_Permuted(&AC); StatFree(&stat); } else if ( *iopt == 2 ) { /* Triangular solve */ /* Initialize the statistics variables. */ StatInit(&stat); /* Extract the LU factors in the factors handle */ LUfactors = (factors_t*) factors[0]; L = LUfactors->L; U = LUfactors->U; perm_c = LUfactors->perm_c; perm_r = LUfactors->perm_r; dCreate_Dense_Matrix(&B, *n, *nrhs, b, *ldb, SLU_DN, SLU_D, SLU_GE); /* Solve the system A*X=B, overwriting B with X. */ dgstrs (trans, L, U, perm_c, perm_r, &B, &stat, info); Destroy_SuperMatrix_Store(&B); StatFree(&stat); } else if ( *iopt == 3 ) { /* Free storage */ /* Free the LU factors in the factors handle */ LUfactors = (factors_t*) factors[0]; SUPERLU_FREE (LUfactors->perm_r); SUPERLU_FREE (LUfactors->perm_c); Destroy_SuperNode_Matrix(LUfactors->L); Destroy_CompCol_Matrix(LUfactors->U); SUPERLU_FREE (LUfactors->L); 39SUPERLU_FREE (LUfactors->U); SUPERLU_FREE (LUfactors); } else { fprintf(stderr, "Invalid iopt=%d passed to c_fortran_dgssv()\n"); exit(-1); } } Since the matrix structures in C cannot be directly returned to Fortran, we use a handle named factors to access those structures. The handle is essentially an integer pointer pointing to the factored matrices obtained from SuperLU. So the factored matrices are opaque objects to the Fortran program, but can only be manipulated from the C wrapper program. The Fortran program FORTRAN/f77 main.f shows how a Fortran program may call c fortran dgssv(), and is listed below. A README ?le in this directory describes how to compile and run this program. program f77_main integer maxn, maxnz parameter ( maxn = 10000, maxnz = 100000 ) integer rowind(maxnz), colptr(maxn) real*8 values(maxnz), b(maxn) integer n, nnz, nrhs, ldb, info integer factors(8), iopt * * Read the matrix file in Harwell-Boeing format call hbcode1(n, n, nnz, values, rowind, colptr) * nrhs = 1 ldb = n do i = 1, n b(i) = 1 enddo * * First, factorize the matrix. The factors are stored in factor() handle. iopt = 1 call c_fortran_dgssv( iopt, n, nnz, nrhs, values, rowind, colptr, $ b, ldb, factors, info ) * if (info .eq. 0) then write (*,*) ’Factorization succeeded’ else write(*,*) ’INFO from factorization = ’, info endif * * Second, solve the system using the existing factors. iopt = 2 40call c_fortran_dgssv( iopt, n, nnz, nrhs, values, rowind, colptr, $ b, ldb, factors, info ) * if (info .eq. 0) then write (*,*) ’Solve succeeded’ write (*,*) (b(i), i=1, 10) else write(*,*) ’INFO from triangular solve = ’, info endif * Last, free the storage allocated inside SuperLU iopt = 3 call c_fortran_dgssv( iopt, n, nnz, nrhs, values, rowind, colptr, $ b, ldb, factors, info ) * stop end 41Chapter 3 Multithreaded SuperLU (Version 2.0) 3.1 About SuperLU MT Among the various steps of the solution process in the sequential SuperLU, the LU factorization dominates the computation; it usually takes more than 95% of the sequential runtime for large sparse linear systems. We have designed and implemented an algorithm to perform the factorization in parallel on machines with a shared address space and multithreading. The parallel algorithm is based on the e?cient sequential algorithm implemented in SuperLU. Although we attempted to minimize the amount of changes to the sequential code, there are still a number of non-trivial modi?cations to the serial SuperLU, mostly related to the matrix data structures and memory organization. All these changes are summarized in Table 3.1 and their impacts on performance are studied thoroughly in [6, 21]. In this part of the Users’ Guide, we describe only the changes that the user should be aware of. Other than these di?erences, most of the material in chapter 2 is still applicable. Construct Parallel algorithm panel restricted so it does not contain branchings in the elimination tree supernode restricted to be a fundamental supernode in the elimination tree supernode storage use either static or dynamic upper bound (section 3.4.2) pruning & DFS use both G(L T ) and pruned G(L T ) to avoid locking Table 3.1: The di?erences between the parallel and the sequential algorithms. 3.2 Storage types for L and U As in the sequential code, the type for the factored matrices L and U is SuperMatrix (Figure 2.2), however, their storage formats (stored in *Store) are changed. In the parallel algorithm, the adjacent panels of the columns may be assigned to di?erent processes, and they may be ?nished and put in global memory out of order. That is, the consecutive columns or supernodes may not be stored contiguously in memory. Thus, in addition to the pointers to the beginning of each column or supernode, we need pointers to the end of the column or supernode. In particular, the storage type for L is SCP (Supernode, Column-wise and Permuted), de?ned as: 42typedef struct { int nnz; /* number of nonzeros in the matrix */ int nsuper; /* number of supernodes */ void *nzval; /* pointer to array of nonzero values, packed by column */ int *nzval_colbeg; /* nzval_colbeg[j] points to beginning of column j in nzval[] */ int *nzval_colend; /* nzval_colend[j] points to one past the last element of column j in nzval[] */ int *rowind; /* pointer to array of compressed row indices of the supernodes */ int *rowind_colbeg;/* rowind_colbeg[j] points to beginning of column j in rowind[] */ int *rowind_colend;/* rowind_colend[j] points to one past the last element of column j in rowind[] */ int *col_to_sup; /* col_to_sup[j] is the supernode number to which column j belongs */ int *sup_to_colbeg;/* sup_to_colbeg[s] points to the first column of the s-th supernode / int *sup_to_colend;/* sup_to_colend[s] points to one past the last column of the s-th supernode */ } SCPformat; The storage type for U is NCP, de?ned as: typedef struct { int nnz; /* number of nonzeros in the matrix */ void *nzval; /* pointer to array of nonzero values, packed by column */ int *rowind; /* pointer to array of row indices of the nonzeros */ int *colbeg; /* colbeg[j] points to the location in nzval[] and rowind[] which starts column j */ int *colend; /* colend[j] points to one past the location in nzval[] and rowind[] which ends column j */ } NCPformat; The table below summarizes the data and storage types of all the matrices involved in the parallel routines: A L U B X Stype SLU NC or SLU NR SLU SCP SLU NCP SLU DN SLU DN Dtype any any any any any Mtype SLU GE SLU TRLU SLU TRU SLU GE SLU GE 3.3 User-callable routines As in the sequential SuperLU, we provide both computational routines and driver routines. To name those routines that involve parallelization in the call-graph, we prepend a letter p to the 43names of their sequential counterparts, for example pdgstrf. For the purely sequential routines, we use the same names as before. Here, we only list the routines that are di?erent from the sequential ones. 3.3.1 Driver routines We provide two types of driver routines for solving systems of linear equations. The driver routines can handle both column- and row-oriented storage schemes. • A simple driver pdgssv, which solves the system AX = B by factorizing A and overwriting B with the solution X. • An expert driver pdgssvx, which, in addition to the above, also performs the following functions (some of them optionally): – solve ATX = B; – equilibrate the system (scale A’s rows and columns to have unit norm) if A is poorly scaled; – estimate the condition number of A, check for near-singularity, and check for pivot growth; – re?ne the solution and compute forward and backward error bounds. 3.3.2 Computational routines The user can invoke the following computational routines to directly control the behavior of SuperLU. The computational routines can only handle column-oriented storage. Except for the parallel factorization routine pdgstrf, all the other routines are identical to those appeared in the sequential superlu. • pdgstrf: Factorize (in parallel). This implements the ?rst-time factorization, or later re-factorization with the same nonzero pattern. In re-factorizations, the code has the ability to use the same column permutation Pc and row permutation Pr obtained from a previous factorization. Several scalar arguments control how the LU decomposition and the numerical pivoting should be performed. pdgstrf can handle non-square matrices. • dgstrs: Triangular solve. This takes the L and U triangular factors, the row and column permutation vectors, and the right-hand side to compute a solution matrix X of AX = B or ATX = B. • dgscon: Estimate condition number. Given the matrix A and its factors L and U, this estimates the condition number in the one-norm or in?nity-norm. The algorithm is due to Hager and Higham [17], and is the same as condest in sparse Matlab. 44• dgsequ/dlaqgs: Equilibrate. dgsequ ?rst computes the row and column scalings Dr and Dc which would make each row and each column of the scaled matrix DrADc have equal norm. dlaqgs then applies them to the original matrix A if it is indeed badly scaled. The equilibrated A overwrites the original A. • dgsrfs: Re?ne solution. Given A, its factors L and U, and an initial solution X, this does iterative re?nement, using the same precision as the input data. It also computes forward and backward error bounds for the re?ned solution. 3.4 Installation 3.4.1 File structure The top level SuperLU MT/ directory is structured as follows: SuperLU_MT_2.0/README instructions on installation SuperLU_MT_2.0/CBLAS/ BLAS routines in C, functional but not fast SuperLU_MT_2.0/DOC/ Users’ Guide SuperLU_MT_2.0/EXAMPLE/ example programs SuperLU_MT_2.0/INSTALL/ test machine dependent parameters SuperLU_MT_2.0/SRC/ C source code, to be compiled into libsuperlu_mt.a SuperLU_MT_2.0/TESTING/ driver routines to test correctness SuperLU_MT_2.0/lib/ SuperLU_MT library archive libsuperlu_mt.a SuperLU_MT_2.0/Makefile top level Makefile that does installation and testing SuperLU_MT_2.0/MAKE_INC sample machine-specific make.inc files SuperLU_MT_2.0/make.inc compiler, compiler flags, library definitions and C preprocessor definitions, included in all Makefiles. (You may need to edit it to suit for your system before compiling the whole package.) We have ported the parallel programs to a number of platforms, which are re?ected in the make include ?les provided in the top level directory, for example, make.pthreads, make.openmp, make.ibm, make.sun, make.sgi, make.cray. If you are using one of these machines, such as an IBM, you can simply copy make.sun into make.inc before compiling. If you are not using any of the machines to which we have ported, you will need to read section 3.6 about the porting instructions. The rest of the installation and testing procedure is similar to that described in section 2.11 for the serial SuperLU. Then, you can type make at the top level directory to ?nish installation. In the SuperLU MT/TESTING subdirectory, you can type pdtest.csh to perform testings. 3.4.2 Performance issues Memory management for L and U In the sequential SuperLU, four data arrays associated with the L and U factors can be expanded dynamically, as described in section 2.8. In the parallel code, the expansion is hard and costly to 45implement, because when a process detects that an array bound is exceeded, it has to send a signal to and suspend the execution of the other processes. Then the detecting process can proceed with the array expansion. After the expansion, this process must wake up all the suspended processes. In this release of the parallel code, we have not yet implemented the above expansion mechanism. For now, the user must pre-determine an estimated size for each of the four arrays through the inquiry function sp ienv(). There are two interpretations for each integer value FILL returned by calling this function with ispec = 6, 7, or 8. A negative number is interpreted as the ?lls growth factor, that is, the program will allocate (-FILL)*nnz(A) elements for the corresponding array. A positive number is interpreted as the true amount the user wants to allocate, that is, the program will allocate FILL elements for the corresponding array. In both cases, if the initial request exceeds the physical memory constraint, the sizes of the arrays are repeatedly reduced until the initial allocation succeeds. int sp ienv(int ispec); Ispec speci?es the parameter to be returned: ispec = . . . = 6: size of the array to store the values of the L supernodes (nzval) = 7: size of the array to store the columns in U (nzval/rowind) = 8: size of the array to store the subscripts of the L supernodes (rowind); If the actual ?ll exceeds any array size, the program will abort with a message showing the current column when failure occurs, and indicating how many elements are needed up to the current column. The user may reset a larger ?ll parameter for this array and then restart the program. To make the storage allocation more e?cient for the supernodes in L, we devised a special storage scheme. The need for this special treatment and how we implement it are fully explained and studied in [6, 21]. Here, we only sketch the main idea. Recall that the parallel algorithm assigns one panel of columns to one process. Two consecutive panels may be assigned to two di?erent processes, even though they may belong to the same supernode discovered later. Moreover, a third panel may be ?nished by a third process and put in memory between these two panels, resulting in the columns of a supernode being noncontiguous in memory. This is undesirable, because then we cannot directly call BLAS routines using this supernode unless we pay the cost of copying the columns into contiguous memory ?rst. To overcome this problem, we exploited the observation that the nonzero structure for L is contained in that of the Householder matrix H from the Householder sparse QR transformation [11, 12]. Furthermore, it can be shown that a fundamental supernode of L is always contained in a fundamental supernode of H. This containment property is true for for any row permutation Pr in PrA = LU. Therefore, we can pre-allocate storage for the L supernodes based on the size of H supernodes. Fortunately, there exists a fast algorithm (almost linear in the number of nonzeros of A) to compute the size of H and the supernodes partition in H [13]. In practice, the above static prediction is fairly tight for most problems. However, for some others, the number of nonzeros in H greatly exceeds the number of nonzeros in L. To handle this situation, we implemented an algorithm that still uses the supernodes partition in H, but dynamically searches the supernodal graph of L to obtain a much tighter bound for the storage. Table 6 in [6] demonstrates the storage e?ciency achieved by both static and dynamic approach. In summary, our program tries to use the static prediction ?rst for the L supernodes. In this case, we ignore the integer value given in the function sp ienv(6), and simply use the nonzero 46count of H. If the user ?nds that the size of H is too large, he can invoke the dynamic algorithm at runtime by setting the following UNIX shell environment variable: setenv SuperLU DYNAMIC SNODE STORE 1 The dynamic algorithm incurs runtime overhead. For example, this overhead is usually between 2% and 15% on a single processor RS/6000-590 for a range of test matrices. Symmetric structure pruning In both serial and parallel algorithms, we have implemented Eisenstat and Liu’s symmetric pruning idea of representing the graph G(L T ) by a reduced graph G' , and thereby reducing the DFS traversal time. A subtle di?culty arises in the parallel implementation. When the owning process of a panel starts DFS (depth-?rst search) on G' built so far, it only sees the partial graph, because the part of G' corresponding to the busy panels down the elimination tree is not yet complete. So the structural prediction at this stage can miss some nonzeros. After performing the updates from the ?nished supernodes, the process will wait for all the busy descendant panels to ?nish and perform more updates from them. Now, we make a conservative assumption that all these busy panels will update the current panel so that their nonzero structures are included in the current panel. This approximate scheme works ?ne for most problems. However, we found that this conservatism may sometimes cause a large number of structural zeros (they are related to the supernode amalgamation performed at the bottom of the elimination tree) to be included and they in turn are propagated through the rest of the factorization. We have implemented an exact structural prediction scheme to overcome this problem. In this scheme, when each numerical nonzero is scattered into the sparse accumulator array, we set the occupied ?ag as well. Later when we accumulate the updates from the busy descendant panels, we check the occupied ?ags to determine the exact nonzero structure. This scheme avoids unnecessary zero propagation at the expense of runtime overhead, because setting the occupied ?ags must be done in the inner loop of the numeric updates. We recommend that the user use the approximate scheme (by default) ?rst. If the user ?nds that the amount of ?ll from the parallel factorization is substantially greater than that from the sequential factorization, he can then use the accurate scheme. To invoke the second scheme, the user should recompile the code by de?ning the macro: -D SCATTER FOUND for the C preprocessor. The inquiry function sp ienv() For some user controllable constants, such as the blocking parameters and the size of the global storage for L and U, SuperLU MT calls the inquiry function sp ienv() to retrieve their values. The declaration of this function is int sp ienv(int ispec). The full meanings of the returned values are as follows: 47ispec = 1: the panel size w = 2: the relaxation parameter to control supernode amalgamation (relax) = 3: the maximum allowable size for a supernode (maxsup) = 4: the minimum row dimension for 2D blocking to be used (rowblk) = 5: the minimum column dimension for 2D blocking to be used (colblk) = 6: size of the array to store the values of the L supernodes (nzval) = 7: size of the array to store the columns in U (nzval/rowind) = 8: size of the array to store the subscripts of the L supernodes (rowind) We should take into account the trade-o? between cache reuse and amount of parallelism in order to set the appropriate w and maxsup. Since the parallel algorithm assigns one panel factorization to one process, large values may constrain concurrency, even though they may be good for uniprocessor performance. We recommend that w and maxsup be set a bit smaller than the best values used in the sequential code. The settings for parameters 2, 4 and 5 are the same as those described in section 2.11.3. The settings for parameters 6, 7 and 8 are discussed in section 3.4.2. In the ?le SRC/sp ienv.c, we provide sample settings of these parameters for several machines. 3.5 Example programs In the SuperLU MT/EXAMPLE/ subdirectory, we present a few sample programs to illustrate the complete calling sequences to use the simple and expert drivers to solve systems of equations. Examples are also given to illustrate how to perform a sequence of factorizations for the matrices with the same sparsity pattern, and how SuperLU MT can be integrated into the other multithreaded application such that threads are created only once. A Makefile is provided to generate the executables. A README ?le in this directory shows how to run these examples. The leading comment in each routine describes the functionality of the example. 3.6 Porting to other platforms We have provided the parallel interfaces for a number of shared-memory machines. Table 3.2 lists the platforms on which we have tested the library, and the respective make.inc ?les. The most portable interface for shared memory programming is POSIX threads [30], since nowadays many commercial UNIX operating systems have support for it. We call our POSIX threads interface the Pthreads interface. To use this interface, you can copy make.pthreads into make.inc and then compile the library. In the last column of Table 3.2, we list the runtime environment variable to be set in order to use multiple CPUs. For example, to use 4 CPUs on the Origin2000, you need to set the following before running the program: setenv MP SET NUMTHREADS 4 In the source code, all the platform speci?c constructs are enclosed in the C #ifdef preprocessor statement. If your platform is di?erent from any one listed in Table 3.2, you need to go to these places and create the parallel constructs suitable for your machine. The two constructs, concurrency and synchronization, are explained in the following two subsections, respectively. 48Programming Environment make.inc Platforms Model Variable make.pthreads Machines with POSIX threads pthreads make.openmp Machines with OpenMP OpenMP OMP NUM THREADS make.alpha DEC Alpha Servers DECthreads make.cray Cray C90/J90 microtasking NCPUS make.ibm IBM Power series pthreads make.origin SGI/Cray Origin2000 parallel C MP SET NUMTHREADS make.sgi SGI Power Challenge parallel C MPC NUM THREADS make.sun Sun Ultra Enterprise Solaris threads Table 3.2: Platforms on which SuperLU MT was tested. Mutex Critical region ULOCK allocate storage for a column of matrix U LLOCK allocate storage for row subscripts of matrix L LULOCK allocate storage for the values of the supernodes NSUPER LOCK increment supernode number nsuper SCHED LOCK invoke Scheduler() which may update global task queue Table 3.3: Five mutex variables. 3.6.1 Creating multiple threads Right now, only the factorization routine pdgstrf is parallelized, since this is the most timeconsuming part in the whole solution process. There is one single thread of control on entering and exiting pdgstrf. Inside this routine, more than one thread may be created. All the newly created threads begin by calling the thread function pdgstrf thread and they are concurrently executed on multiple processors. The thread function pdgstrf thread expects a single argument of type void*, which is a pointer to the structure containing all the shared data objects. 3.6.2 Use of mutexes Although the threads pdgstrf thread execute independently of each other, they share the same address space and can communicate e?ciently through shared variables. Problems may arise if two threads try to access (at least one is to modify) the shared data at the same time. Therefore, we must ensure that all memory accesses to the same data are mutually exclusive. There are ?ve critical regions in the program that must be protected by mutual exclusion. Since we want to allow di?erent processors to enter di?erent critical regions simultaneously, we use ?ve mutex variables as listed in Table 3.3. The user should properly initialize them in routine ParallelInit, and destroy them in routine ParallelFinalize. Both these routines are in ?le pxgstrf synch.c. 49Chapter 4 Distributed SuperLU with MPI (Version 2.3) 4.1 About SuperLU DIST In this part, we describe the SuperLU DIST library designed for distributed-memory parallel computers using SPMD parallel programming model. The library is implemented in ANSI C, using MPI [27] for communication, and is highly portable. We have tested the code on a number of platforms, including IBM SP, Cray XT, SGI Altix, and numerous Linux clusters. The library includes routines to handle both real and complex matrices in double precision. The parallel routine names for the double-precision real version start with letters “pd” (such as pdgstrf); the parallel routine names for double-precision complex version start with letters “pz” (such as pzgstrf). 4.2 Formats of the input matrices A and B We provide two input interfaces for matrices A and B—one is global, and the other is entirely distributed. 4.2.1 Global input The input matrices A and B are globally available (replicated) on all the processes. The storage type for A is SLU NC (compressed column), as in sequential case (see Section 2.3). The user-callable routines with this interface all have the names “xxxxxxx ABglobal”. If there is su?cient memory, this interface is faster than the distributed input interface described in the next section, because the latter requires more data re-distribution at di?erent stages of the algorithm. 4.2.2 Distributed input Both input matrices A and B are distributed among all the processes. They use the same distribution based on block rows. That is, each process owns a block of consecutive rows of A and B. Each local part of sparse matrix A is stored in a compressed row format, called SLU NR loc storage type, which is de?ned below. typedef struct { 50int nnz_loc; /* number of nonzeros in the local submatrix */ int m_loc; /* number of rows local to this process */ int fst_row; /* row number of the first row in the local submatrix */ void *nzval; /* pointer to array of nonzero values, packed by row */ int *rowptr; /* pointer to array of beginning of rows in nzval[] and colind[] */ int *colind; /* pointer to array of column indices of the nonzeros */ } NRformat_loc; Let mi be the number of rows owned by the ith process. Then the global row dimension for A is nrow = PP-1 i=0 mi . The global column dimension is ncol. Both nrow and ncol are recorded in the higher level SuperMatrix data structure, see Figure 2.2. The utility routine dCreate CompRowLoc Matrix dist can help the user to create the structure for A. The de?nition of this routine is void dCreate_CompRowLoc_Matrix_dist(SuperMatrix *A, int m, int n, int nnz_loc, int m_loc, int fst_row, double *nzval, int *colind, int *rowptr, Stype_t stype, Dtype_t dtype, Mtype_t mtype); where, the ?rst argument is output and the rest are inputs. The local full matrix B is stored in the standard Fortran-style column major format, with dimension m loc × nrhs, and ldb refers to the local leading dimension in the local storage. 4.3 Distributed data structures for L and U We distribute both L and U matrices in a two-dimensional block-cyclic fashion. We ?rst identify the supernode boundary based on the nonzero structure of L. This supernode partition is then used as the block partition in both row and column dimensions for both L and U. The size of each block is matrix dependent. It should be clear that all the diagonal blocks are square and full (we store zeros from U in the upper triangle of the diagonal block), whereas the o?-diagonal blocks may be rectangular and may not be full. The matrix in Figure 4.1 illustrates such a partition. By block-cyclic mapping we mean block (I, J) (0 = I, J = N - 1) is mapped into the process at coordinate {I mod nprow, J mod npcol} of the nprow×npcol 2D process grid. Using this mapping, a block L(I, J) in the factorization is only needed by the row of processes that own blocks in row I. Similarly, a block U(I, J) is only needed by the column of processes that own blocks in column J. In this 2D mapping, each block column of L resides on more than one process, namely, a column of processes. For example in Figure 4.1, the second block column of L resides on the column processes {1, 4}. Process 4 only owns two nonzero blocks, which are not contiguous in the global matrix. The schema on the right of Figure 4.1 depicts the data structure to store the nonzero blocks on a process. Besides the numerical values stored in a Fortran-style array nzval[] in column major order, we need the information to interpret the location and row subscript of each nonzero. This is stored in an integer array index[], which includes the information for the whole block column and for each individual block in it. Note that many o?-diagonal blocks are zero and hence not stored. Neither do we store the zeros in a nonzero block. Both lower and upper triangles 510 1 3 4 2 ... 1 2 ... 4 0 2 3 0 2 0 1 2 0 3 4 3 4 3 4 0 1 2 0 1 0 2 3 3 0 1 2 0 4 0 3 3 4 3 0 1 2 0 1 1 Process Mesh Global Matrix L 5 5 5 5 U 5 5 5 index Storage of block column of L # of blocks nzval block # row subscripts i1 i2 # of full rows block # row subscripts i1 i2 # of full rows LDA of nzval Figure 4.1: The 2 block-cyclic layout and the data structure to store a local block column of L. of the diagonal block are stored in the L data structure. A process owns ?N/npcol? block columns of L, so it needs ?N/nprow? pairs of index/nzval arrays. For U, we use a row oriented storage for the block rows owned by a process, although for the numerical values within each block we still use column major order. Similar to L, we also use a pair of index/nzval arrays to store a block row of U. Due to asymmetry, each nonzero block in U has the skyline structure as shown in Figure 4.1 (see [5] for details on the skyline structure). Therefore, the organization of the index[] array is di?erent from that for L, which we omit showing in the ?gure. 4.4 Process grid and MPI communicator All MPI applications begin with a default communication domain that includes all processes, say Np, of this parallel job. The default communicator MPI COMM WORLD represents this communication domain. The Np processes are identi?ed as a linear array of process IDs in the range 0 . . . Np - 1. 4.4.1 2D process grid For SuperLU DIST library, we create a new process group derived from an existing group using Ng processes. There is a good reason to use a new group rather than MPI COMM WORLD, that is, the message passing calls of the SuperLU library will be isolated from those in other libraries or in the user’s code. For better scalability of the LU factorization, we map the 1D array of Ng processes into a logical 2D process grid. This grid will have nprow process rows and npcol process columns, such that nprow * npcol = Ng. A process can be referenced either by its rank in the new group or by its coordinates within the grid. The routine superlu gridinit maps already-existing processes to a 2D process grid. superlu_gridinit(MPI_Comm Bcomm, int nprow, int npcol, gridinfo_t *grid); 52This process grid will use the ?rst nprow * npcol processes from the base MPI communicator Bcomm, and assign them to the grid in a row-major ordering. The input argument Bcomm is an MPI communicator representing the existing base group upon which the new group will be formed. For example, it can be MPI COMM WORLD. The output argument grid represents the derived group to be used in SuperLU DIST. Grid is a structure containing the following ?elds: struct { MPI_Comm comm; /* MPI communicator for this group */ int iam; /* my process rank in this group */ int nprow; /* number of process rows */ int npcol; /* number of process columns */ superlu_scope_t rscp; /* process row scope */ superlu_scope_t cscp; /* process column scope */ } grid; In the LU factorization, some communications occur only among the processes in a row (column), not among all processes. For this purpose, we introduce two process subgroups, namely rscp (row scope) and cscp (column scope). For rscp (cscp) subgroup, all processes in a row (column) participate in the communication. The macros MYROW(iam, grid) and MYCOL(iam, grid) give the row and column coordinates in the 2D grid of the process who has rank iam. NOTE: All processes in the base group, including those not in the new group, must call this grid creation routine. This is required by the MPI routine MPI Comm create to create a new communicator. 4.4.2 Arbitrary grouping of processes It is sometimes desirable to divide up the processes into several subgroups, each of which performs independent work of a single application. In this situation, we cannot simply use the ?rst nprow * npcol processes to de?ne the grid. A more sophisticated process-to-grid mapping routine superlu gridmap is designed to create a grid with processes of arbitrary ranks. superlu_gridmap(MPI_Comm Bcomm, int nprow, int npcol, int usermap[], int ldumap, gridinfo_t *grid); The array usermap[] contains the processes to be used in the newly created grid. usermap[] is indexed like a Fortran-style 2D array with ldumap as the leading dimension. So usermap[i+j*ldumap] (i.e., usermap(i,j) in Fortran notation) holds the process rank to be placed in {i, j} position of the 2D process grid. After grid creation, this subset of processes is logically numbered in a consistent manner with the initial set of processes; that is, they have the ranks in the range 0 . . . nprow * npcol - 1 in the new grid. For example, if we want to map 6 processes with ranks 11 . . . 16 into a 2 × 3 grid, we de?ne usermap = {11, 14, 12, 15, 13, 16} and ldumap = 2. Such a mapping is shown below 0 1 2 0 11 12 13 1 14 15 16 53NOTE: All processes in the base group, including those not in the new group, must call this routine. Superlu gridinit simply calls superlu gridmap with usermap[] holding the ?rst nprow * npcol process ranks. 4.5 Basic steps to solve a linear system In this section, we use a complete sample program to illustrate the basic steps required to use SuperLU DIST. This program is listed below, and is also available as EXAMPLE/pddrive.c in the source code distribution. All the routines must include the header ?le superlu ddefs.h (or superlu zdefs.h, the complex counterpart) which contains the de?nitions of the data types, the macros and the function prototypes. #include #include "superlu_ddefs.h" main(int argc, char *argv[]) /* * Purpose * ======= * * The driver program PDDRIVE. * * This example illustrates how to use PDGSSVX with the full * (default) options to solve a linear system. * * Five basic steps are required: * 1. Initialize the MPI environment and the SuperLU process grid * 2. Set up the input matrix and the right-hand side * 3. Set the options argument * 4. Call pdgssvx * 5. Release the process grid and terminate the MPI environment * * On the Cray T3E, the program may be run by typing * mpprun -n pddrive -r -c * */ { superlu_options_t options; SuperLUStat_t stat; SuperMatrix A; ScalePermstruct_t ScalePermstruct; LUstruct_t LUstruct; SOLVEstruct_t SOLVEstruct; gridinfo_t grid; 54double *berr; double *b, *xtrue; int_t m, n, nnz; int_t nprow, npcol; int iam, info, ldb, ldx, nrhs; char trans[1]; char **cpp, c; FILE *fp, *fopen(); nprow = 1; /* Default process rows. */ npcol = 1; /* Default process columns. */ nrhs = 1; /* Number of right-hand side. */ /* ------------------------------------------------------------ INITIALIZE MPI ENVIRONMENT. ------------------------------------------------------------*/ MPI_Init( &argc, &argv ); /* Parse command line argv[]. */ for (cpp = argv+1; *cpp; ++cpp) { if ( **cpp == ’-’ ) { c = *(*cpp+1); ++cpp; switch (c) { case ’h’: printf("Options:\n"); printf("\t-r : process rows (default %d)\n", nprow); printf("\t-c : process columns (default %d)\n", npcol); exit(0); break; case ’r’: nprow = atoi(*cpp); break; case ’c’: npcol = atoi(*cpp); break; } } else { /* Last arg is considered a filename */ if ( !(fp = fopen(*cpp, "r")) ) { ABORT("File does not exist"); } break; } } /* ------------------------------------------------------------ INITIALIZE THE SUPERLU PROCESS GRID. 55------------------------------------------------------------*/ superlu_gridinit(MPI_COMM_WORLD, nprow, npcol, &grid); /* Bail out if I do not belong in the grid. */ iam = grid.iam; if ( iam >= nprow * npcol ) goto out; /* ------------------------------------------------------------ GET THE MATRIX FROM FILE AND SETUP THE RIGHT HAND SIDE. ------------------------------------------------------------*/ dcreate_matrix(&A, nrhs, &b, &ldb, &xtrue, &ldx, fp, &grid); if ( !(berr = doubleMalloc_dist(nrhs)) ) ABORT("Malloc fails for berr[]."); /* ------------------------------------------------------------ NOW WE SOLVE THE LINEAR SYSTEM. ------------------------------------------------------------*/ /* Set the default input options. */ set_default_options_dist(&options); m = A.nrow; n = A.ncol; /* Initialize ScalePermstruct and LUstruct. */ ScalePermstructInit(m, n, &ScalePermstruct); LUstructInit(m, n, &LUstruct); /* Initialize the statistics variables. */ PStatInit(&stat); /* Call the linear equation solver. */ pdgssvx(&options, &A, &ScalePermstruct, b, ldb, nrhs, &grid, &LUstruct, &SOLVEstruct, berr, &stat, &info); /* Check the accuracy of the solution. */ pdinf_norm_error(iam, ((NRformat_loc *)A.Store)->m_loc, nrhs, b, ldb, xtrue, ldx, &grid); PStatPrint(&options, &stat, &grid); /* Print the statistics. */ /* ------------------------------------------------------------ DEALLOCATE STORAGE. 56------------------------------------------------------------*/ PStatFree(&stat); Destroy_CompRowLoc_Matrix_dist(&A); ScalePermstructFree(&ScalePermstruct); Destroy_LU(n, &grid, &LUstruct); LUstructFree(&LUstruct); if ( options.SolveInitialized ) { dSolveFinalize(&options, &SOLVEstruct); } SUPERLU_FREE(b); SUPERLU_FREE(xtrue); SUPERLU_FREE(berr); /* ------------------------------------------------------------ RELEASE THE SUPERLU PROCESS GRID. ------------------------------------------------------------*/ out: superlu_gridexit(&grid); /* ------------------------------------------------------------ TERMINATES THE MPI EXECUTION ENVIRONMENT. ------------------------------------------------------------*/ MPI_Finalize(); } Five basic steps are required to call a SuperLU routine: 1. Initialize the MPI environment and the SuperLU process grid. This is achieved by the calls to the MPI routine MPI Init() and the SuperLU routine superlu gridinit(). In this example, the communication domain for SuperLU is built upon the MPI default communicator MPI COMM WORLD. In general, it can be built upon any MPI communicator. Section 4.4 contains the details about this step. 2. Set up the input matrix and the right-hand side. This example uses the interface with the distributed input matrices, see Section 4.2.2. In most practical applications, the matrices can be generated on each process without the need to have a centralized place to hold them. But for this example, we let process 0 read the input matrix stored on disk in Harwell-Boeing format [10] (a.k.a. compressed column storage), and distribute it to all the other processes, so that each process only owns a block of rows of matrix. The right-hand side matrix is generated so that the exact solution matrix consists of all ones. The subroutine dcreate matrix() accomplishes this task. 3. Initialize the input arguments: options, ScalePermstruct, LUstruct, stat. The input argument options controls how the linear system would be solved—use equilibration or not, how to order the rows and the columns of the matrix, use iterative re?nement or 57not. The subroutine set default options dist() sets the options argument so that the solver performs all the functionality. You can also set it up according to your own needs, see section 4.7.1 for the ?elds of this structure. ScalePermstruct is the data structure that stores the several vectors describing the transformations done to A. LUstruct is the data structure in which the distributed L and U factors are stored. Stat is a structure collecting the statistics about runtime and ?op count, etc. 4. Call the SuperLU routine pdgssvx(). 5. Release the process grid and terminate the MPI environment. After the computation on a process grid has been completed, the process grid should be released by a call to the SuperLU routine superlu gridexit(). When all computations have been completed, the MPI routine MPI Finalize() should be called. 4.6 Algorithmic background Although partial pivoting is used in both sequential and shared-memory parallel factorization algorithms, it is not used in the distributed-memory parallel algorithm, because it requires dynamic adaptation of data structures and load balancing, and so is hard to make it scalable. We use alternative techniques to stabilize the algorithm, which include statically pivot large elements to the diagonal, single-precision diagonal adjustment to avoid small pivots, and iterative re?nement. Figure 4.2 sketches our GESP algorithm (Gaussian elimination with Static Pivoting). Numerical experiments show that for a wide range of problems, GESP is as stable as GEPP [24]. Step (1) is accomplished by a weighted bipartite matching algorithm due to Du? and Koster [9]. Currently, process 0 computes Pr and then broadcasts it to all the other processes. If the distributed input interface is used (Section 4.2.2), we ?rst gather the distributed matrix A onto processor 0. Work is underway to remove this sequential bottleneck. In Step (2), we provide several ordering options, such as multiple minimum degree ordering [26] on the graphs of A + AT , or the MeTiS [18] ordering on the graphs of A + AT . The user can use any other ordering in place of the ones provided in the library. (Note, since we will pivot on the diagonal in step (4), an ordering based on the structure of A + AT almost always yields sparser factors than that based on the structure of AT A. This is di?erent from SuperLU and SuperLU MT, where we allow to pivot o?-diagonal.) In this step, when a sequential ordering algorithm is used, every process runs the same algorithm independently. Step (3) can be done either sequentially or in parallel depending on how the options argument is set (see Section 4.7.1 for details.) The parallel symbolic factorization is a newly added feature in the latest v2.1 release. It is designed tightly around the separator tree returned from a graph partitioning type of ordering (presently we use ParMeTiS [19]), and works only on power-of-two processors. We ?rst re-distribute the graph of A onto the largest 2 q number of processors which is smaller than the total Np processors, then perform parallel symbolic factorization, and ?nally re-populate the {L\U} structure to all Np processors. The algorithm and performance was studied in [15]. To invoke parallel symbolic factorization, the user needs to set the two ?elds of the options argument as follows: options.ParSymbFact = YES options.ColPerm = PARMETIS; 58(1) Perform row/column equilibration and row permutation: A ? Pr
Dr
A
Dc, where Dr and Dc are diagonal matrices and Pr is a row permutation chosen to make the diagonal large compared to the o?-diagonal. (2) Find a column permutation Pc to preserve sparsity: A ? Pc
A
P T c (3) Perform symbolic analysis to determine the nonzero structures of L and U. (4) Factorize A = L
U with control of diagonal magnitude: if ( |aii | < v e
kAk1 ) then set aii to v e
kAk1 endif (5) Perform triangular solutions using L and U. (6) If needed, use an iterative solver like GMRES or iterative re?nement (shown below) iterate: r = b - A
x . . . sparse matrix-vector multiply Solve A
dx = r . . . triangular solution berr = maxi |r|i (|A|
|x|+|b|)i . . . componentwise backward error if ( berr > e and berr = 1 2
lastberr ) then x = x + dx lastberr = berr goto iterate endif (7) If desired, estimate the condition number of A Figure 4.2: The outline of the GESP algorithm. 59Note that, even if the user sets options.ColPerm to use an ordering algorithm other than ParMeTiS, the driver routine overrides it with ParMeTiS when it sees options.ParSymbFact = YES. Steps (4) to (7) are the most time-consuming steps and were parallelized a while ago, see the papers [24, 22]. 4.7 User-callable routines 4.7.1 Driver routines There are two driver routines to solve systems of linear equations, which are named pdgssvx ABglobal for the global input interface, and pdgssvx for the distributed interface. We recommend that the general users, especially the beginners, use a driver routine rather than the computational routines, because correctly using the driver routine does not require thorough understanding of the underlying data structures. Although the interface of these routines are simple, we expect their rich functionality can meet the requirements of most applications. Pdgssvx ABglobal/pdgssvx perform the following functions: • Equilibrate the system (scale A’s rows and columns to have unit norm) if A is poorly scaled; • Find a row permutation that makes diagonal of A large relative to the o?-diagonal; • Find a column permutation that preserves the sparsity of the L and U factors; • Solve the system AX = B for X by factoring A followed by forward and back substitutions; • Re?ne the solution X. Options argument One important input argument to pdgssvx ABglobal/pdgssvx is options, which controls how the linear system will be solved. Although the algorithm presented in Figure 4.2 consists of seven steps, for some matrices not all steps are needed to get accurate solution. For example, for diagonally dominant matrices, choosing the diagonal pivots ensures the stability; there is no need for row pivoting in step (1). In another situation where a sequence of matrices with the same sparsity pattern need be factorized, the column permutation Pc (and also the row permutation Pr, if the numerical values are similar) need be computed only once, and reused thereafter. (Pr and Pc are implemented as permutation vectors perm r and perm c.) For the above examples, performing all seven steps does more work than necessary. Options is used to accommodate the various requirements of applications; it contains the following ?elds: • Fact This option speci?es whether or not the factored form of the matrix A is supplied on entry, and if not, how the matrix A will be factored base on some assumptions of the previous history. fact can be one of: – DOFACT: the matrix A will be factorized from scratch. – SamePattern: the matrix A will be factorized assuming that a factorization of a matrix with the same sparsity pattern was performed prior to this one. Therefore, this factorization will reuse column permutation vector perm c. 60– SampPattern SameRowPerm: the matrix A will be factorized assuming that a factorization of a matrix with the same sparsity pattern and similar numerical values was performed prior to this one. Therefore, this factorization will reuse both row and column permutation vectors perm r and perm c, both row and column scaling factors Dr and Dc, and the distributed data structure set up from the previous symbolic factorization. – FACTORED: the factored form of A is input. • Equil This option speci?es whether to equilibrate the system. • ParSymbFact This option speci?es whether to perform parallel symbolic factorization. If it is set to YES, the ColPerm ?eld should be set to PARMETIS. Otherwise, the driver routine pdgssvx will use ParMeTiS anyway, ignoring the other setting in ColPerm. • ColPerm This option speci?es the column ordering method for ?ll reduction. – NATURAL: natural ordering. – MMD AT PLUS A: minimum degree ordering on the structure of AT + A. – MMD ATA: minimum degree ordering on the structure of AT A. – METIS AT PLUS A: MeTiS ordering on the structure of AT + A. – PARMETIS: ParMeTiS ordering on the structure of AT + A. – MY PERMC: use the ordering given in perm c input by the user. • RowPerm This option speci?es how to permute rows of the original matrix. – NATURAL: use the natural ordering. – LargeDiag: use a weighted bipartite matching algorithm to permute the rows to make the diagonal large relative to the o?-diagonal. – MY PERMR: use the ordering given in perm r input by the user. • ReplaceTinyPivot This option speci?es whether to replace the tiny diagonals by v e
||A|| during LU factorization. • IterRefine This option speci?es how to perform iterative re?nement. – NO: no iterative re?nement. – DOUBLE: accumulate residual in double precision. – EXTRA: accumulate residual in extra precision. (not yet implemented.) • SolveInitialized (Used only by the distributed input interface) This option speci?es whether the initialization has been performed to the triangular solve. 61• RefineInitialized (Used only by the distributed input interface) This option speci?es whether the initialization has been performed to the sparse matrix-vector multiplication routine needed in the iterative re?nement. • PrintStat Speci?es whether to print the solver’s statistics. There is a routine named set default options dist() that sets the default values of these options, which are: fact = DOFACT /* factor from scratch */ equil = YES ParSymbFact = NO colperm = MMD_AT_PLUS_A rowperm = LargeDiag /* use MC64 */ ReplaceTinyPivot = YES IterRefine = DOUBLE SolveInitialized = NO RefineInitialized = NO PrintStat = YES 4.7.2 Computational routines The experienced users can invoke the following computational routines to directly control the behavior of SuperLU DIST in order to meet their requirements. • pdgstrf(): Factorize in parallel. This routine factorizes the input matrix A (or the scaled and permuted A). It assumes that the distributed data structures for L and U factors are already set up, and the initial values of A are loaded into the data structures. If not, the routine symbfact() should be called to determine the nonzero patterns of the factors, and the routine pddistribute() should be called to distribute the matrix. Pdgstrf() can factor non-square matrices. • pdgstrs()/pdgstrs Bglobal(): Triangular solve in parallel. This routine solves the system by forward and back substitutions using the the L and U factors computed by pdgstrf(). Pdgstrs() takes distributed B. For pdgstrs Bglobal(), B must be globally available on all processes. • pdgsrfs()/pdgsrfs ABXglobal(): Re?ne solution in parallel. Given A, its factors L and U, and an initial solution X, this routine performs iterative re?nement. Pdgsrfs() takes distributed A, B and X. For pdgsrfs ABXglobal(), A, B and X must be globally available on all processes. 4.7.3 Utility routines The following utility routines can help users create and destroy the SuperLU DIST matrices. These routines reside in three places: SRC/util.c, SRC/{d,z}util.c, and SRC/p{d,z}util.c. Most of the utility routines in sequential SuperLU can also be used in SuperLU DIST for the local data, see 62Section 2.9.3. Here, we only list those new routines speci?c to SuperLU DIST. Note that in order to avoid name clash between SuperLU and SuperLU DIST, we append “ dist” to each routine name in SuperLU DIST. /* Create a supermatrix in distributed compressed row format. A is output. */ dCreate_CompRowLoc_Matrix_dist(SuperMatrix *A, int_t m, int_t n, int_t nnz_loc, int_t m_loc, int_t fst_row, double *nzval, int_t *colind, int_t *rowptr, Stype_t stype, Dtype_t dtype, Mtype_t mtype); /* Deallocate the supermatrix in distributed compressed row format. */ Destroy_CompRowLoc_Matrix_dist(SuperMatrix *A); /* Allocate storage in ScalePermstruct. */ ScalePermstructInit(const int_t m, const int_t n, ScalePermstruct_t *ScalePermstruct); /* Deallocate ScalePermstruct */ ScalePermstructFree(ScalePermstruct_t *ScalePermstruct); /* Allocate storage in LUstruct. */ LUstructInit(const int_t m, const int_t n, LUstruct_t *LUstruct); /* Deallocate the distributed L & U factors in LUstruct. */ Destroy_LU(int_t n, gridinfo_t *grid, LUstruct_t *LUstruct); /* Deallocate LUstruct. */ LUstructFree(LUstruct_t *LUstruct); /* Initialize the statistics variable. */ PStatInit(SuperLUStat_t *stat); /* Print the statistics. */ PStatPrint(superlu_options_t *options, SuperLUStat_t *stat, gridinfo_t *grid); /* Deallocate the statistics variable. */ PStatFree(SuperLUStat_t *stat); 4.8 Installation 4.8.1 File structure The top level SuperLU DIST/ directory is structured as follows: SuperLU_DIST/README instructions on installation 63SuperLU_DIST/CBLAS/ BLAS routines in C, functional but not fast SuperLU_DIST/DOC/ Users’ Guide SuperLU_DIST/EXAMPLE/ example programs SuperLU_DIST/INSTALL/ test machine dependent parameters SuperLU_DIST/SRC/ C source code, to be compiled into libsuperlu_dist.a SuperLU_DIST/lib/ contains library archive libsuperlu_dist.a SuperLU_DIST/Makefile top level Makefile that does installation and testing SuperLU_DIST/make.inc compiler, compiler flags, library definitions and C preprocessor definitions, included in all Makefiles. (You may need to edit it to suit for your system before compiling the whole package.) SuperLU_DIST/MAKE_INC/ sample machine-specific make.inc files Before installing the package, you may need to edit SuperLU DIST/make.inc for your system. This make include ?le is referenced inside all the Make?les in the various subdirectories. As a result, there is no need to edit the Make?les in the subdirectories. All information that is machine speci?c has been de?ned in this include ?le. Sample machine-speci?c make.inc are provided in the MAKE INC/ directory for several platforms, such as Cray T3E and IBM SP. When you have selected the machine to which you wish to install SuperLU DIST, you may copy the appropriate sample include ?le (if one is present) into make.inc. For example, if you wish to run on a Cray T3E, you can do: cp MAKE INC/make.t3e make.inc For the systems other than those listed above, slight modi?cations to the make.inc ?le will need to be made. In particular, the following items should be examined: 1. The BLAS library. If there is a BLAS library available on your machine, you may de?ne the following in make.inc: BLASDEF = -DUSE VENDOR BLAS BLASLIB = The CBLAS/ subdirectory contains the part of the BLAS (in C) needed by SuperLU DIST package. However, these routines are intended for use only if there is no faster implementation of the BLAS already available on your machine. In this case, you should do the following: 1) In make.inc, unde?ne (comment out) BLASDEF, de?ne: BLASLIB = ../lib/libblas$(PLAT).a 2) At the top level SuperLU DIST directory, type: make blaslib to create the BLAS library from the routines in CBLAS/ subdirectory. 2. External libraries: MeTiS and ParMeTiS. If you will use MeTiS or ParMeTiS ordering, or parallel symbolic factorization (which depends on ParMeTiS), you will need to install them yourself. Since ParMeTiS package already contains the source code for the MeTiS library, you can just download ParMeTiS at: http://glaros.dtc.umn.edu/gkhome/metis/parmetis/download After you have installed it, you should de?ne the following in make.inc: 64METISLIB = -L -lmetis PARMETISLIB = -L -lparmetis 3. C preprocessor de?nition CDEFS. In the header ?le SRC/Cnames.h, we use macros to determine how C routines should be named so that they are callable by Fortran. 1 The possible options for CDEFS are: • -DAdd : Fortran expects a C routine to have an underscore post?xed to the name; • -DNoChange: Fortran expects a C routine name to be identical to that compiled by C; • -DUpCase: Fortran expects a C routine name to be all uppercase. A Makefile is provided in each subdirectory. The installation can be done automatically by simply typing make at the top level. 4.8.2 Performance-tuning parameters Similar to sequential SuperLU, several performance related parameters are set in the inquiry function sp ienv(). The declaration of this function is int sp ienv(int ispec); Ispec speci?es the parameter to be returned 2 : ispec = 2: the relaxation parameter to control supernode amalgamation = 3: the maximum allowable size for a block = 6: the estimated ?lls factor for the adjacency structures of L and U The values to be returned may be set di?erently on di?erent machines. The setting of maximum block size (parameter 3) should take into account the local Level 3 BLAS speed, the load balance and the degree of parallelism. Small block size may result in better load balance and more parallelism, but poor individual node performance, and vice versa for large block size. 4.9 Example programs In the SuperLU DIST/EXAMPLE/ directory, we present a few sample programs to illustrate the complete calling sequences to use the expert driver to solve systems of equations. These include how to set up the process grid and the input matrix, how to obtain a ?ll-reducing ordering. A Makefile is provided to generate the executables. A README ?le in this directory shows how to run these examples. The leading comment in each routine describes the functionality of the example. The two basic examples are pddrive ABglobal() and pddrive(). The ?rst shows how to use the global input interface, and the second shows how to use the distributed input interface. 1 Some vendor-supplied BLAS libraries do not have C interfaces. So the re-naming is needed in order for the SuperLU BLAS calls (in C) to interface with the Fortran-style BLAS. 2 The numbering of 2, 3 and 6 is consistent with that used in SuperLU and SuperLU MT. 654.10 Fortran 90 Interface We developed a complete Fortran 90 interface for SuperLU DIST. All the interface ?les and an example driver program are located in the SuperLU DIST/FORTRAN/ directory. Table 4.1 lists all the ?les. f pddrive.f90 An example Fortran driver routine. superlu mod.f90 Fortran 90 module that de?nes the interface functions to access SuperLU DIST’s data structures. superlupara.f90 It contains parameters that correspond to SuperLU DIST’s enums. hbcode1.f90 Fortran function for reading a sparse Harwell-Boeing matrix from the ?le. superlu c2f wrap.c C wrapper functions, callable from Fortran. The functions fall into three classes: 1) Those that allocate a structure and return a handle, or deallocate the memory of a structure. 2) Those that get or set the value of a component of a struct. 3) Those that are wrappers for SuperLU DIST functions. dcreate dist matrix.c C function for distributing the matrix in a distributed compressed row format. Table 4.1: The Fortran 90 interface ?les and an example driver routine. Note that in this interface, all objects (such as grid, options, etc.) in SuperLU DIST are opaque, meaning their size and structure are not visible to the Fortran user. These opaque objects are allocated, deallocated and operated in the C side and not directly accessible from Fortran side. They can only be accessed via handles that exist in Fortran’s user space. In Fortran, all handles have type INTEGER. Speci?cally, in our interface, the size of Fortran handle is de?ned by superlu ptr in superlupara.f90. For di?erent systems, the size might need to be changed. Then using these handles, Fortran user can call C wrapper routines to manipulate the opaque objects. For example, you can call f create gridinfo(grid handle) to allocate memory for structure grid, and return a handle grid handle. The sample program illustrates the basic steps required to use SuperLU DIST in Fortran to solve systems of equations. These include how to set up the processor grid and the input matrix, how to call the linear equation solver. This program is listed below, and is also available as f pddrive.f90 in the subdirectory. Note that the routine must include the moudle superlu mod which contains the de?nitions of all parameters and the Fortran wrapper functions. A Make?le is provided to generate the executable. A README ?le in this directory shows how to run the example. program f_pddrive ! ! Purpose ! ======= ! ! The driver program F_PDDRIVE. 66! ! This example illustrates how to use F_PDGSSVX with the full ! (default) options to solve a linear system. ! ! Seven basic steps are required: ! 1. Create C structures used in SuperLU ! 2. Initialize the MPI environment and the SuperLU process grid ! 3. Set up the input matrix and the right-hand side ! 4. Set the options argument ! 5. Call f_pdgssvx ! 6. Release the process grid and terminate the MPI environment ! 7. Release all structures ! use superlu_mod include ’mpif.h’ implicit none integer maxn, maxnz, maxnrhs parameter ( maxn = 10000, maxnz = 100000, maxnrhs = 10 ) integer rowind(maxnz), colptr(maxn) real*8 values(maxnz), b(maxn), berr(maxnrhs) integer n, m, nnz, nrhs, ldb, i, ierr, info, iam integer nprow, npcol integer init integer(superlu_ptr) :: grid integer(superlu_ptr) :: options integer(superlu_ptr) :: ScalePermstruct integer(superlu_ptr) :: LUstruct integer(superlu_ptr) :: SOLVEstruct integer(superlu_ptr) :: A integer(superlu_ptr) :: stat ! Create Fortran handles for the C structures used in SuperLU_DIST call f_create_gridinfo(grid) call f_create_options(options) call f_create_ScalePermstruct(ScalePermstruct) call f_create_LUstruct(LUstruct) call f_create_SOLVEstruct(SOLVEstruct) call f_create_SuperMatrix(A) call f_create_SuperLUStat(stat) ! Initialize MPI environment call mpi_init(ierr) 67! Initialize the SuperLU_DIST process grid nprow = 2 npcol = 2 call f_superlu_gridinit(MPI_COMM_WORLD, nprow, npcol, grid) ! Bail out if I do not belong in the grid. call get_GridInfo(grid, iam=iam) if ( iam >= nprow * npcol ) then go to 100 endif if ( iam == 0 ) then write(*,*) ’ Process grid ’, nprow, ’ X ’, npcol endif ! Read Harwell-Boeing matrix, and adjust the pointers and indices ! to 0-based indexing, as required by C routines. if ( iam == 0 ) then open(file = "g20.rua", status = "old", unit = 5) call hbcode1(m, n, nnz, values, rowind, colptr) close(unit = 5) ! do i = 1, n+1 colptr(i) = colptr(i) - 1 enddo do i = 1, nnz rowind(i) = rowind(i) - 1 enddo endif ! Distribute the matrix to the gird call f_dcreate_matrix_dist(A, m, n, nnz, values, rowind, colptr, grid) ! Setup the right hand side nrhs = 1 call get_CompRowLoc_Matrix(A, nrow_loc=ldb) do i = 1, ldb b(i) = 1.0 enddo ! Set the default input options call f_set_default_options(options) ! Change one or more options ! call set_superlu_options(options,Fact=FACTORED) 68! Initialize ScalePermstruct and LUstruct call get_SuperMatrix(A,nrow=m,ncol=n) call f_ScalePermstructInit(m, n, ScalePermstruct) call f_LUstructInit(m, n, LUstruct) ! Initialize the statistics variables call f_PStatInit(stat) ! Call the linear equation solver call f_pdgssvx(options, A, ScalePermstruct, b, ldb, nrhs, & grid, LUstruct, SOLVEstruct, berr, stat, info) if (info == 0) then write (*,*) ’Backward error: ’, (berr(i), i = 1, nrhs) else write(*,*) ’INFO from f_pdgssvx = ’, info endif ! Deallocate SuperLU allocated storage call f_PStatFree(stat) call f_Destroy_CompRowLoc_Matrix_dist(A) call f_ScalePermstructFree(ScalePermstruct) call f_Destroy_LU(n, grid, LUstruct) call f_LUstructFree(LUstruct) call get_superlu_options(options, SolveInitialized=init) if (init == YES) then call f_dSolveFinalize(options, SOLVEstruct) endif ! Release the SuperLU process grid 100 call f_superlu_gridexit(grid) ! Terminate the MPI execution environment call mpi_finalize(ierr) ! Destroy all C structures call f_destroy_gridinfo(grid) call f_destroy_options(options) call f_destroy_ScalePermstruct(ScalePermstruct) call f_destroy_LUstruct(LUstruct) call f_destroy_SOLVEstruct(SOLVEstruct) call f_destroy_SuperMatrix(A) call f_destroy_SuperLUStat(stat) stop 69end Similar to the driver routine pddrive.c in C, seven basic steps are required to call a SuperLU DIST routine in Fortran: 1. Create C structures used in SuperLU: grid, options, ScalePermstruct, LUstruct, SOLVEstruct, A and stat. This is achieved by the calls to the C wrapper “create” routines f create XXX(), where XXX is the name of the corresponding structure. 2. Initialize the MPI environment and the SuperLU process grid. This is achieved by the calls to mpi init() and the C wrapper routine f superlu gridinit(). Note that f superlu gridinit() requires the numbers of row and column of the process grid. In this example, we set them to be 2, respectively. 3. Set up the input matrix and the right-hand side. This example uses the distributed input interface, so we need to convert the input matrix to the distributed compressed row format. Process 0 ?rst reads the input matrix stored on disk in Harwell-Boeing format by calling Fortran routine hbcode1(). The ?le name in this example is g20.rua. Then all processes call a C wrapper routine f dcreate dist matrix() to distribute the matrix to all the processes distributed by block rows. The right-hand side matrix in this example is a column vector of all ones. Note that, before setting the right-hand side, we use get CompRowLoc Matrix() to get the number of local rows in the distributed matrix A. One important note is that all the C routines use 0-based indexing scheme. Therefore, after process 0 reads the matrix in compressed column format, we decrement its column pointers (colptr) and row indices (rowind) by 1 so they become 0-based indexing. 4. Set the input arguments: options, ScalePermstruct, LUstruct, and stat. The input argument options controls how the linear system would be sloved. The routine f set default options dist() sets the options argument so that the slover performs all the functionalities. You can also set it according to your own needs, using a call to the Fortran routine set superlu options(). LUstruct is the data struture in which the distributed L and U factors are stored. ScalePermstruct is the data struture in which several vectors describing the transformations done to matrix A are stored. stat is a structure collecting the statistcs about runtime and ?op count. These three structures can be set by calling the C wrapper “init” routines f XXXInit. 5. Call the C wrapper routine f pdgssvx() to solve the equation. 6. Release the process grid and terminate the MPI environment. After the computation on a process grid has been completed, the process grid should be released by a call to f spuerlu gridexit(). When all computations have been completed, the C wrapper routine mpi ?nalize() should be called. 7. Deallocate all the structures. First we need to deallocate the storage allocated by SuperLU DIST by a set of “free” calls. Note that this should be called before f spuerlu gridexit(), since some of the “free” calls use the grid. Then we call the C wrapper “destroy” routines f destroy XXX() to destroy all the Fortran handles. Note that f destroy gridinfo() should be called after f spuerlu gridexit(). 704.10.1 Callable functions in the Fortran 90 module ?le spuerlu mod.f90 The Fortran 90 module superlu mod contains the interface routines that can manipulate a SuperLU DIST object from Fortran. The object is pointed to by the corresponding handle input to these routines. The routines are divided into two sets. One set is to get the properties of an object, with the routine names “get XXX()”. Another set is to set some properties for an object, with the routine names “set XXX()”. These functions have optional arguments, so the users do not have to provide the full set of parameters. Superlu mod module uses superluparam mod module that de?nes all the integer constants corresponding to the enumeration constants in SuperLU DIST. Below are the calling sequences of all the routines. subroutine get_GridInfo(grid, iam, nprow, npcol) integer(superlu_ptr) :: grid integer, optional :: iam, nprow, npcol subroutine get_SuperMatrix(A, nrow, ncol) integer(superlu_ptr) :: A integer, optional :: nrow, ncol subroutine set_SuperMatrix(A, nrow, ncol) integer(superlu_ptr) :: A integer, optional :: nrow, ncol subroutine get_CompRowLoc_Matrix(A, nrow, ncol, nnz_loc, nrow_loc, fst_row) integer(superlu_ptr) :: A integer, optional :: nrow, ncol, nnz_loc, nrow_loc, fst_row subroutine set_CompRowLoc_Matrix(A, nrow, ncol, nnz_loc, nrow_loc, fst_row) integer(superlu_ptr) :: A integer, optional :: nrow, ncol, nnz_loc, nrow_loc, fst_row subroutine get_superlu_options(opt, Fact, Trans, Equil, RowPerm, & ColPerm, ReplaceTinyPivot, IterRefine, & SolveInitialized, RefineInitialized) integer(superlu_ptr) :: opt integer, optional :: Fact, Trans, Equil, RowPerm, ColPerm, & ReplaceTinyPivot, IterRefine, SolveInitialized, & RefineInitialized subroutine set_superlu_options(opt, Fact, Trans, Equil, RowPerm, & ColPerm, ReplaceTinyPivot, IterRefine, & SolveInitialized, RefineInitialized) integer(superlu_ptr) :: opt integer, optional :: Fact, Trans, Equil, RowPerm, ColPerm, & ReplaceTinyPivot, IterRefine, SolveInitialized, & RefineInitialized 714.10.2 C wrapper functions callable by Fortran in ?le spuerlu c2f wrap.c This ?le contains the Fortran-callable C functions which wraps around the user-callable C routines in SuperLU DIST. The functions are divided into three classes: 1) allocate a C structure and return a handle to Fortran, or deallocate the memory of of a C structure given its Fortran handle; 2) get or set the value of certain ?elds of a C structure given its Fortran handle; 3) wrapper functions for the SuperLU DIST C functions. Below are the calling sequences of these routines. /* functions that allocate memory for a structure and return a handle */ void f_create_gridinfo(fptr *handle) void f_create_options(fptr *handle) void f_create_ScalePermstruct(fptr *handle) void f_create_LUstruct(fptr *handle) void f_create_SOLVEstruct(fptr *handle) void f_create_SuperMatrix(fptr *handle) void f_create_SuperLUStat(fptr *handle) /* functions that free the memory allocated by the above functions */ void f_destroy_gridinfo(fptr *handle) void f_destroy_options(fptr *handle) void f_destroy_ScalePermstruct(fptr *handle) void f_destroy_LUstruct(fptr *handle) void f_destroy_SOLVEstruct(fptr *handle) void f_destroy_SuperMatrix(fptr *handle) void f_destroy_SuperLUStat(fptr *handle) /* functions that get or set certain fields in a C structure. */ void f_get_gridinfo(fptr *grid, int *iam, int *nprow, int *npcol) void f_get_SuperMatrix(fptr *A, int *nrow, int *ncol) void f_set_SuperMatrix(fptr *A, int *nrow, int *ncol) void f_get_CompRowLoc_Matrix(fptr *A, int *m, int *n, int *nnz_loc, int *m_loc, int *fst_row) void f_set_CompRowLoc_Matrix(fptr *A, int *m, int *n, int *nnz_loc, int *m_loc, int *fst_row) void f_get_superlu_options(fptr *opt, int *Fact, int *Trans, int *Equil, int *RowPerm, int *ColPerm, int *ReplaceTinyPivot, int *IterRefine, int *SolveInitialized, int *RefineInitialized) void f_set_superlu_options(fptr *opt, int *Fact, int *Trans, int *Equil, int *RowPerm, int *ColPerm, int *ReplaceTinyPivot, int *IterRefine, int *SolveInitialized, int *RefineInitialized) /* wrappers for SuperLU_DIST routines */ void f_dCreate_CompRowLoc_Matrix_dist(fptr *A, int *m, int *n, int *nnz_loc, int *m_loc, int *fst_row, double *nzval, 72int *colind, int *rowptr, int *stype, int *dtype, int *mtype) void f_set_default_options(fptr *options) void f_superlu_gridinit(int *Bcomm, int *nprow, int *npcol, fptr *grid) void f_superlu_gridexit(fptr *grid) void f_ScalePermstructInit(int *m, int *n, fptr *ScalePermstruct) void f_ScalePermstructFree(fptr *ScalePermstruct) void f_PStatInit(fptr *stat) void f_PStatFree(fptr *stat) void f_LUstructInit(int *m, int *n, fptr *LUstruct) void f_LUstructFree(fptr *LUstruct) void f_Destroy_LU(int *n, fptr *grid, fptr *LUstruct) void f_Destroy_CompRowLoc_Matrix_dist(fptr *A) void f_dSolveFinalize(fptr *options, fptr *SOLVEstruct) void f_pdgssvx(fptr *options, fptr *A, fptr *ScalePermstruct, double *B, int *ldb, int *nrhs, fptr *grid, fptr *LUstruct, fptr *SOLVEstruct, double *berr, fptr *stat, int *info) void f_check_malloc(int *iam) 73Bibliography [1] E. Anderson, Z. Bai, C. Bischof, J. Demmel, J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, A. McKenney, S. Ostrouchov, and D. Sorensen. LAPACK Users’ Guide, Release 2.0. SIAM, Philadelphia, 1995. 324 pages. [2] M. Arioli, J. W. Demmel, and I. S. Du?. Solving sparse linear systems with sparse backward error. SIAM J. Matrix Anal. Appl., 10(2):165–190, April 1989. [3] L. S. Blackford, J. Choi, E. D’Azevedo, J. Demmel, I. Dhillon, J. Dongarra, S. Hammarling, G. Henry, A. Petitet, K. Stanley, D. Walker, and R. C. Whaley. ScaLAPACK Users’ Guide. SIAM, Philadelphia, 1997. 325 pages. [4] Timothy A. Davis, John R. Gilbert, Stefan I. Larimore, and Esmond Ng. A column approximate minimum degree ordering algorithm. Technical Report TR-00-005, Computer and Information Sciences Department, University of Florida, 2000. submitted to ACM Trans. Math. Software. [5] James W. Demmel, Stanley C. Eisenstat, John R. Gilbert, Xiaoye S. Li, and Joseph W. H. Liu. A supernodal approach to sparse partial pivoting. SIAM J. Matrix Analysis and Applications, 20(3):720–755, 1999. [6] James W. Demmel, John R. Gilbert, and Xiaoye S. Li. An asynchronous parallel supernodal algorithm for sparse gaussian elimination. SIAM J. Matrix Analysis and Applications, 20(4):915–952, 1999. [7] J. Dongarra, J. Du Croz, I. S. Du?, and S. Hammarling. A Set of Level 3 Basic Linear Algebra Subprograms. ACM Trans. Math. Soft., 16:1–17, 1990. [8] J. Dongarra, J. Du Croz, S. Hammarling, and Richard J. Hanson. An Extended Set of FORTRAN Basic Linear Algebra Subprograms. ACM Trans. Math. Soft., 14(1):1–17, March 1988. [9] Iain S. Du? and Jacko Koster. The design and use of algorithms for permuting large entries to the diagonal of sparse matrices. SIAM J. Matrix Analysis and Applications, 20(4):889–901, 1999. [10] I.S. Du?, R.G. Grimes, and J.G. Lewis. Users’ guide for the Harwell-Boeing sparse matrix collection (release 1). Technical Report RAL-92-086, Rutherford Appleton Laboratory, December 1992. [11] Alan George, Joseph Liu, and Esmond Ng. A data structure for sparse QR and LU factorizations. SIAM J. Sci. Stat. Comput., 9:100–121, 1988. 74[12] Alan George and Esmond Ng. Symbolic factorization for sparse Gaussian elimination with partial pivoting. SIAM J. Sci. Stat. Comput., 8(6):877–898, 1987. [13] J. R. Gilbert, X. S. Li, E. G. Ng, and B. W. Peyton. Computing row and column counts for sparse QR and LU factorization. BIT, 41(4):693–710, 2001. [14] John R. Gilbert and Esmond G. Ng. Predicting structure in nonsymmetric sparse matrix factorizations. In Alan George, John R. Gilbert, and Joseph W.H. Liu, editors, Graph theory and sparse matrix computation, pages 107–139. Springer-Verlag, New York, 1993. [15] Laura Grigori, James W. Demmel, and Xiaoye S. Li. Parallel symbolic factorization for sparse LU with static pivoting. SIAM J. Scienti?c Computing, 29(3):1289–1314, 2007. [16] B. Hendrickson and R. Leland. The CHACO’s User’s Guide. Technical Report SAND93- 2339•UC-405, Sandia National Laboratories, Albuquerque, 1993. http://www.cs.sandia. gov/~bahendr/chaco.html. [17] N. J. Higham. Accuracy and Stability of Numerical Algorithms. SIAM, Philadelphia, PA, 1996. [18] G. Karypis and V. Kumar. MeTiS – a software package for partitioning unstructured graphs, partitioning meshes, and computing ?ll-reducing orderings of sparse matrices – version 4.0. University of Minnesota, September 1998. http://www-users.cs.umn.edu/~karypis/metis/. [19] G. Karypis, K. Schloegel, and V. Kumar. ParMeTiS: Parallel graph partitioning and sparse matrix ordering library – version 3.1. University of Minnesota, 2003. http://www-users.cs. umn.edu/~karypis/metis/parmetis/. [20] C. Lawson, R. Hanson, D. Kincaid, and F. Krogh. Basic Linear Algebra Subprograms for Fortran usage. ACM Trans. Math. Soft., 5:308–323, 1979. [21] Xiaoye S. Li. Sparse Gaussian elimination on high performance computers. Technical Report UCB//CSD-96-919, Computer Science Division, U.C. Berkeley, September 1996. Ph.D dissertation. [22] Xiaoye S. Li. An overview of SuperLU: Algorithms, implementation, and user interface. ACM Trans. Mathematical Software, 31(3):302–325, September 2005. [23] Xiaoye S. Li and James W. Demmel. Making sparse Gaussian elimination scalable by static pivoting. In Proceedings of SC98: High Performance Networking and Computing Conference, Orlando, Florida, November 7–13 1998. [24] Xiaoye S. Li and James W. Demmel. SuperLU DIST: A scalable distributed-memory sparse direct solver for unsymmetric linear systems. ACM Trans. Mathematical Software, 29(2):110– 140, June 2003. [25] Xiaoye S. Li and Meiyue Shao. A supernodal approach to imcomplete LU factorization with partial pivoting. Technical report, Lawrence Berkeley National Laboratory, June 2009. [26] Joseph W.H. Liu. Modi?cation of the minimum degree algorithm by multiple elimination. ACM Trans. Math. Software, 11:141–153, 1985. 75[27] Message Passing Interface (MPI) forum. http://www.mpi-forum.org/. [28] W. Oettli and W. Prager. Compatibility of approximate solution of linear equations with given error bounds for coe?cients and right hand sides. Num. Math., 6:405–409, 1964. [29] Francois Pellegrini. Scotch and libScotch 5.1 User’s Guide (version 5.1.1). INRIA Bordeaux Sud-Ouest, Universit´e Bordeaux I. October 14, 2008. http://www.labri.fr/perso/ pelegrin/scotch/. [30] POSIX System Application Arogram Interface: Threads extension [C Language], POSIX 1003.1c draft 4. IEEE Standards Department. [31] Y. Saad. ILUT: A dual threshold incomplete LU factorization. Numerical Linear Algebra with Applications, 1(4):387–402, 1994. [32] R.D. Skeel. Iterative re?nement implies numerical stability for Gaussian elimination. Mathematics of Computation, 35(151):817–832, 1980. 76 February 2011 Programming Environments Release Announcement The following product/versions are released: Product/Version =============== Cray Compiling Environment 7.3.2 CCE 7.3.2 Cray Message Passing Toolkit 5.2.0 MPT 5.2.0 PMI 1.0-1.0000.8256.50 PMI-devel 1.0-1.0000.8256.50 Environment Setup and Compiling support 5.08 xt-asyncpe 4.8 Third Party Products 5.06 Global Arrays 4.3.3 Intel-sup 12.0.2.137 Cray Debugger Support tools 1.2.1 STAT 1.1.3 ATP 1.1.1 PGI compiler 11.1.0 PGI 11.1.0 TotalView 8.9.0 TotalView 8.9.0 Cray Application Developer's Environment 5.08 CADE 5.08 Cray Application Developer's Environment Supplement 2.08 CADES 2.08 These products are available for download from CrayPort at: http://crayport.cray.com/Pages/default.aspx Note: At the end of this announcement is a list of the latest PE product versions. If you find this useful, please let me know. Operating System Dependencies: ============================== These products were tested on Cray XT systems running the CLE 2.2 and CLE 3.1 operating systems and on Cray XE systems running the CLE 3.1 operating system. Detailed descriptions ===================== Cray Compiling Environment 7.3.2 ================================ CCE 7.3.2 ========= Purpose: -------- The CCE 7.3.2 update provides Bug Fixes to the CCE 7.3.1 release for Cray XT and Cray XE systems. Bugs Fixed in 7.3.2 are: 767964 - Cray compiler bug report - Fortran 2003 standard -- parent child Known Limitations: ------------------ Version 7.3 of the CCE release updates C and C++ to provide GCC version 4.4.4 compatibility, therefore: - C++ codes that use the pre-standard headers, must be updated to the ISO C++ standard headers. - C++ codes, including C++ libraries, must be recompiled. An ABI change occurred in GCC 4.4.4. See http://gcc.gnu.org/gcc-4.4/changes.html. If code passes or returns structures with a flexible array member, a complex float member, or a long double member, it is affected by this ABI change and must be recompiled. The CCE 7.3 C++ compiler does not allow use of Pre ISO C++ Standard Library header files that have the .h extension. The format and size of the pointer-to-shared data type has changed in order to support PGAS applications with more than 64K PEs. As a result, applications written in coarray Fortran or UPC cannot combine object files compiled with CCE 7.3 and object files compiled with previously released CCE compilers. This restriction also applies to the PGAS runtime library; it is not possible to link CCE 7.2 (or earlier) object files with the CCE 7.3 PGAS runtime library. This release includes partial support for the proposed OpenMP 3.1 standard specification. As a result, applications using OpenMP API constructs cannot combine object files compiled with CCE 7.3 and object files compiled with previously released CCE compilers. This restriction also applies to the OpenMP runtime library released with CCE 7.3; it is not possible to link CCE 7.2 (or earlier) object files with the CCE 7.3 OpenMP runtime library. Dependencies: ------------- The CCE 7.3 release is supported on Cray XE systems that run on the Cray Linux Environment (CLE) operating system, version 3.1 and on Cray XT systems that run CLE 2.1 or later. Note: CLE 3.1 UP02 or CLE 3.1 UP01G plus PS20 is recommended for PGAS performance on Cray XE systems. CCE 7.3 requires that gcc/4.4.4 be installed. GCC 4.4.4 does not need to be a default GNU environment. The following product versions are the minimum versions required to be used with CCE 7.3.2 compilers on Cray XT and Cray XE systems: acml 4.4.0 MPT 5.1.0 or later. hdf5-netcdf 1.8 (HDF5 1.8.5 and netcdf 4.1.1) CrayPat 5.1.2 libsci 10.4.9 PETSc 3.1.04 xt-asyncpe 4.4 or later. cray-prgenv 1.0.2 (Cray XT systems running CLE 2.1 or CLE 2.2 OS) Installation instructions: -------------------------- To install the PrgEnv-cray, CCE, programming environment, copy the rpms and execute these commands: export CRAY_INSTALL_DEFAULT=1 rpm -ivh --oldpackage cce-7.3.2-106.x86_64.rpm unset CRAY_INSTALL_DEFAULT XT systems running CLE 2.1 or 2.2 ONLY export CRAY_INSTALL_DEFAULT=1 rpm -ivh --oldpackage cray-prgenv-1.0.2-1.x86_64.rpm unset CRAY_INSTALL_DEFAULT CCE 7.3.0 requires a new license key. Cray Message Passing Toolkit ============================ MPT 5.2.0 ========= Purpose: -------- The following features were added to MPT 5.2.0 over MPT 5.1.4: - This new version is based on MPICH2 1.3a2 and includes numerous fixes. It also includes some additional error checking. One class of these errors involves the use of MPI_IN_PLACE. Some users may find that using the MPICH_NO_BUFFER_ALIAS_CHECK environment variable is a useful workaround until their application can be corrected. See the intro_mpi man page for more information concerning this environment variable. In addition the following features were added in MPT updates since the MPT 5.1.0 release and are also included: - Improvements to the default MPI_Bcast implementation to use a binomial tree algorithm for all message sizes, within the context of the smp-aware bcast for Cray XE systems. Preliminary usage shows significant improvement when using more than 512 cores. Above 10K cores a 16K byte message was over 200X faster. Users can set the MPICH_BCAST_ONLY_TREE environment variable to disable. For more information about this feature see the intro_mpi man page. - If a consistent ordering of the reduce operation is needed for bitwise reproducibility, regardless of system configuration, two new environment variables have been added to allow that. They are MPICH_ALLREDUCE_NO_SMP and MPICH_REDUCE_NO_SMP and are desribed further in the intro_mpi man page. - Improvements to the default MPI_Alltoall collective algorithm based on the number of ranks and message sizes for Cray XE systems. Initial results show a 30% to over 2X improvement for greater than 4K byte messages. See the intro_mpi man page for the MPICH_ALLTOALL_SHORT_MSG environment variable for more information. - Improvements to the default MPI_Allgather and MPI_Allgatherv collective algorithms based on the number of ranks and messages sizes for Cray XE systems. Initial results indicate over 50X performance improvement for messages 1024 bytes or less and at very high process counts. See the intro_mpi man page to learn more about MPICH_ALLGATHER_VSHORT_MSG and MPICH_ALLGATHERV_VSHORT_MSG environment variables which were added. - The MPI memory footprint has been significantly reduced for Cray XE systems when running at large core counts. - Add the MPICH_COLL_OPT_OFF environment variable for Cray XE systems. For more information about this feature see the intro_mpi man page. - Improve MPI_Finalize performance on Cray XE systems by using an edge_color algorithm to stagger the rank access. For a 150K rank job MPI_Finalize time was about 8 times faster. - Improve small word injection rate by placement of certain internal data structures in both MPI and uGNI near the gemini NIC for Cray XE systems. Initial measurements have seen an improvement of almost 4 times using 24 ranks/node with nearest neighbor nodes. The MPICH_GNI_MBOX_PLACEMENT environment variable will need to be set to enable this performance improvement as well as the CLE 3.1 UP02 release which includes the uGNI changes. See the intro_mpi man page for information on the MPICH_GNI_MBOX_PLACEMENT environment variable. - Improvements to MPI-IO collective buffering. - A check is now made during SHMEM initialization to detect potential oversubscription of memory, and if detected, the layout of the memory regions is printed. In addition, a new environment variable, SHMEM_MEMINFO_DISPLAY, causes SHMEM to display the layout even if oversubscription is not detected. See the SHMEM_MEMINFO_DISPLAY environment variable in the intro_shmem man page for more information. In addition numerous bugs were fixed in the MPT updates since MPT 5.1.0 and are listed below: 764987 Improve performance of MPI_Finalize at high process counts 742474,744363,754252,763505,764480 - Prevent attempted oversubscription of memory during shmem_init() on XT systems 765571, 760251, 761685 - compiler warning about gethostbyname for all MPI programs 766580 apps intermittently hang at startup 757369 Low small word injection rate on G34 Gemini nodes Requires CLE 3.1 UP02 or later to see full performance improvement. 767103 Optimize the gni netmod dynamic connection code 767821 Enhance MPICH2 to utilize fix for bug 767706 to allow lazy dereg of 4KB pages 767917 MPI_Comm_split has poor O(np^2) scaling performance 767999 need to improve MPICH2 gni netmod workqueue management 768026 Use /proc/buddyinfo data when checking and displaying SHMEM memory usage Product and OS Dependencies for Gemini systems: ----------------------------------------------- The MPT 5.2.0 release is supported on Cray XE systems running the CLE 3.1UP00 CNL or later operating systems and these products: xt-asyncpe 4.0 or later pmi 1.0-1.0000.8256.50.6.gem alps cce 7.2 or later (optional) cray-libugni 2.1 or later cray-libugni-devel 2.1 or later cray-libudreg 1.3 or later cray-libudreg-devel 1.3 or later cray-libxpmem 0.1 or later cray-libxpmem-devel 0.1 or later cray-libdmapp 1.0 or later cray-libdmapp-devel 1.0 or later Product and OS Dependencies for Seastar systems: ------------------------------------------------ The MPT 5.2.0 release is supported on Cray XT systems running the CLE 2.1 Update 3 CNL or later operating systems. xt-asyncpe 4.2 or later pmi 1.0-1.0000.8256.50.1.ss alps cce 7.2 or later Installation: ------------ export CRAY_INSTALL_DEFAULT=1 For Gemini systems: rpm -ivh cray-mpt-5.2.0-gem0_8274.x86_64.rpm For Seastar systems: rpm -ivh cray-mpt-5.2.0-ss0_8274.x86_64.rpm unset CRAY_INSTALL_DEFAULT PMI 1.0-1.0000.8256.50 and PMI-devel 1.0-1.0000.8256.50 ============================ Purpose: -------- Improved package versioning and RPM naming convention for the PMI product. The package version number is now correctly reported in the RPM name. The shared library current/age/rev values now accurately reflect the package version as well." Installation: ------------- install for Seastar: export CRAY_INSTALL_DEFAULT=1 rpm -ivh cray-libpmi0-1.0-1.0000.8256.50.1.ss.x86_64.rpm rpm -ivh cray-libpmi-devel-1.0-1.0000.8256.50.1.ss.x86_64.rpm unset CRAY_INSTALL_DEFAULT install for gemini: export CRAY_INSTALL_DEFAULT=1 rpm -ivh cray-libpmi0-1.0-1.0000.8256.50.6.gem.x86_64.rpm rpm -ivh cray-libpmi-devel-1.0-1.0000.8256.50.6.gem.x86_64.rpm unset CRAY_INSTALL_DEFAULT Environment Setup and Compiling support 5.08 ============================================ xt-asyncpe 4.8 ============== Purpose: -------- Bugfix release: 768957 syntax error in /opt/cray/xt-asyncpe/4.7/bin/CC 768968 craype_hugepages2m and craype_hugepages8m are missing Features: --------- Support for redesign of petsc and trilinos, and addition of Third Party Solvers Library product, (tpsl), which will support both petsc and trilinos. To see more information and alternatives to using the hugepages feature, look at the man page "man intro_hugepages". Product and OS Dependencies: ---------------------------- xt-asyncpe 4.5 and beyond supports the cray GPU-PE, onesided and numatoolkit. xt-asyncpe 4.5 and beyond supports the pathscale 4.0 library builds xt-asyncpe 4.4 and beyond supports the cce 7.3 PE. xt-asyncpe 4.0 and beyond support the gemini network products. Documentation: -------------- See man pages for cc, CC, and ftn. Installation: ------------- export CRAY_INSTALL_DEFAULT=1 rpm -ivh --oldpackage xt-asyncpe-4.8-23.i386.rpm unset CRAY_INSTALL_DEFAULT Third Party Products 5.06 ========================= Global Arrays 4.3.3 =================== Purpose ------- Global Arrays is a portable Non-Uniform Memory Access (NUMA) shared- memory programming environment for distributed and shared memory computers. It augments the message-passing model by providing a shared-memory like access to distributed dense arrays. Operating System Dependencies: ------------------------------ The Global Arrays 4.3.3 release is supported on Cray XT and Cray XE systems running the CLE 3.1 or later operating systems. Installation Instructions: -------------------------- export CRAY_INSTALL_DEFAULT=1 rpm -ivh ga-4.3.3-0.x86_64.rpm unset CRAY_INSTALL_DEFAULT Intel-sup 12.0.2.137 ==================== Purpose: -------- Provide the module file to support compiling with Intel Composer XE for Linux version 12.0.2.137. The following bugs are fixed in the intel 12.0.2.137 release. 751664 Intel C/C++ fails to note that OpenMP loop construct must be perfectly nested [Issue 565468] 752265 Intel C++ OpenMP parallel for rejects viter=viter-1 for-incr with [Issue 565487] 758724 Intel C OpenMP msg 'An OpenMP "for" must have a loop-invariant increment...' issued in error [Issue 589866] Product and OS Dependencies: ---------------------------- Intel 12.0.2.137 is supported on Cray XT and XE systems running the CLE 2.1 CNL or later operating systems. The Intel Compiler Suite version 12.0.2.137 (C/C++ and FTN) must be installed in the default location. The Intel C/C++ only installation is not supported because the Intel Fortran run time libraries are required by Cray XT libraries (libsci, for example) when using the Intel compiler. These products are also required: intel-prgenv 1.0.0 or later xt-asyncpe 4.6 or later libsci 10.3.6 or later MPT 3.2.0 or later hdf5-netcdf 1.3 or later Documentation: -------------- /opt/intel/composerxe-2011.2.137/Documentation/en_US/Release_NotesC.pdf /opt/intel/composerxe-2011.2.137/Documentation/en_US/Release_NotesF.pdf Installation: ------------- export CRAY_INSTALL_DEFAULT=1 rpm -ihv --oldpackage xt-intelsup-12.0.2.137-2.x86_64.rpm unset CRAY_INSTALL_DEFAULT PGI compiler =========== PGI 11.1.0 ---------- Features of PGI 11.1.0 are documented at: http://www.pgroup.com/doc/pgirn111.pdf The following bugs are fixed in the PGI 11.1.0 release. 768230 PGI 11 compilers produce dynamic executables, even with -Bstatic 768436 GAMESS Compile Error using PGI 11 [17602] Product and OS Dependencies: ---------------------------- pgi 11.1.0 is supported on Cray XT systems running the UNICOS/lc 2.0 CNL and CLE 2.1 CNL or later operating systems. NOTE: Licensing change: PGI issued licensing policy has changed. The license will work with existing PGI compilers and all new compilers released prior to a specified date. A new license will be required to use any compilers released after the expiration date. Documentation: -------------- Documentation for PGI 11.1.0 is in /opt/pgi/11.1.0/linux86-64/11.1/doc/ PGI User's Guide, pgi11ug.pdf PGI Tools Guide, pgi11tools.pdf PGI Workstation 11.1 Release Notes, pgirn111.pdf PGI Fortran Reference, pgifortref.pdf PGI Workstation 11.1 Installation Guide, pgiwsinstall111.pdf Installation: ------------- export CRAY_INSTALL_DEFAULT=1 rpm -ihv --oldpackage pgi-11.1.0-4.x86_64.rpm unset CRAY_INSTALL_DEFAULT Cray Debugger Support tools 1.2.1 ================================= STAT 1.1.3 ---------- Purpose: -------- Bugs fixed in this release: 768708 - STAT returned error type STAT_WARNING: host parse error on line 2 Known Issues: ------------- Due to a python-gtk issue, STATview will is not supported on systems running CLE 3.0 or CLE 3.1 less than UP02. dotty can be used as an alternate viewer. Product and OS Dependencies: ---------------------------- The STAT 1.1.3 release is supported on Cray XT systems running the CLE 2.2 CNL or later operating systems. The STAT 1.1.3 release is supported on Cray XE systems running the CLE 3.1UP00 CNL or later operating systems. Installation: ------------- export CRAY_INSTALL_DEFAULT=1 rpm -ivh stat-1.1.3-4.x86_64.rpm unset CRAY_INSTALL_DEFAULT ATP 1.1.1 ---------- Purpose: -------- Eliminated side effect interactions between the ATP and STAT modules. These bugs were fixed in this release: 768479 The reported rank to backtrace routine association is often incorrect. 768554 Short running apps that exit normally, elicit error messages from aptFrontend Installation: ------------- export CRAY_INSTALL_DEFAULT=1 rpm -ivh atp-1.1.1-0.x86_64.rpm unset CRAY_INSTALL_DEFAULT TotalView Debugger ================== TotalView 8.9.0 --------------- Purpose: -------- This package includes the 8.9.0 release of the Totalview Technologies debugger. Known Limitations ----------------- The Replay Engine fails on multiple node applications. Installation: ------------- tar -xvf cray-xt-totalview-8.9.0-0-totalview-support-1.1.1-3.tar ./Install Cray Application Developer's Environment ======================================== CADE 5.08 ========= Purpose ------- CADE is a collection of the latest basic programming environment for Cray XT and Cray XE systems. CADE does not include licensed products (CCE, PGI, Performance tools and etc). This package plus the licensed products (compilers, performance tools and debuggers) will update the system to the latest Programming Environment. The contents of this CADE release package are: CADE-5.08-02.iso ---------------- acml-4.4.0-1.x86_64.rpm atp-1.1.1-0.x86_64.rpm chapel-1.2.1-gem3.x86_64.rpm chapel-1.2.1-ss3.x86_64.rpm cray-gcc-gmp-4.3.2-2.x86_64.rpm cray-gcc-mpc-0.8.1-2.x86_64.rpm cray-gcc-mpfr-2.3.1-20.x86_64.rpm cray-gcc-mpfr-2.4.2-2.x86_64.rpm cray-gdb-7.2-21.x86_64.rpm cray-libpmi0-1.0-1.0000.8256.50.1.ss.x86_64.rpm cray-libpmi0-1.0-1.0000.8256.50.6.gem.x86_64.rpm cray-libpmi-devel-1.0-1.0000.8256.50.1.ss.x86_64.rpm cray-libpmi-devel-1.0-1.0000.8256.50.6.gem.x86_64.rpm cray-modules-3.1.6.6-10.x86_64.rpm cray-modules-3.2.6.6-1.0000.3887.0.0.gem.x86_64.rpm cray-modules-3.2.6.6-1.0000.3887.0.0.ss.x86_64.rpm cray-mpt-5.2.0-gem0_8274.x86_64.rpm cray-mpt-5.2.0-ss0_8274.x86_64.rpm fftw-2.1.5.2-1.x86_64.rpm fftw-3.2.2.1-3.x86_64.rpm ga-4.3.3-0.x86_64.rpm hdf5_netcdf-1.8-3.x86_64.rpm intel-prgenv-1.0.0-1.x86_64.rpm iobuf-2.0.1-1.x86_64.rpm java-jdk1.6.0_20-1.x86_64.rpm lgdb-1.2-3.x86_64.rpm libfast_mv-1.0.8-999.x86_64.rpm libonesided-ntk-1.0.0-1.gem.x86_64.rpm mrnet-3.0.0-0.x86_64.rpm netcdf-3.6.2-1.x86_64.rpm petsc-3.1.04-1.x86_64.rpm stat-1.1.3-4.x86_64.rpm trilinos-10.6.2.0-2.x86_64.rpm xt-asyncpe-4.8-23.i386.rpm xt-gcc-4.1.2-10.x86_64.rpm xt-gcc-4.2.4-5.x86_64.rpm xt-gcc-4.4.2-6.x86_64.rpm xt-gcc-4.4.4-4.x86_64.rpm xt-gcc-4.5.1-3.x86_64.rpm xt-gcc-4.5.2-4.x86_64.rpm xt-intelsup-11.1.073-2.x86_64.rpm xt-intelsup-12.0.0.084-6.x86_64.rpm xt-intelsup-12.0.1.107-5.x86_64.rpm xt-intelsup-12.0.2.137-2.x86_64.rpm xt-libsci-10.5.0-1.x86_64.rpm xt-pathscalesup-3.2-2.x86_64.rpm xt-pathscalesup-3.2.99-1.x86_64.rpm Installation Instructions: -------------------------- To install this CADE release, please see S-2465-50. S-2465-50 is available at http://docs.cray.com/. Cray Application Developer's Environment Supplement =================================================== CADES 2.08 ========== Purpose ------- The CADES package is a supplement to the CADE. CADES is a licensed product, please contact your Cray sales representative for information. The contents of this CADES release package are: CADES-2.08-02iso ----------------- cade-prgenv-3.1.49A-4.x86_64.rpm cray-gni-headers-2.1-1.0301.2792.5.1.gem.noarch.rpm cray-libdmapp1-2.2-1.0301.2791.5.1.gem.x86_64.rpm cray-libdmapp-devel-2.2-1.0301.2791.5.1.gem.x86_64.rpm cray-libportals1-2.2.0-1.0301.24560.5.2.ss.x86_64.rpm cray-libportals-devel-2.2.0-1.0301.24560.5.2.ss.x86_64.rpm cray-librca0-1.0.0-2.0301.24564.5.18.gem.x86_64.rpm cray-librca0-1.0.0-2.0301.24564.5.18.ss.x86_64.rpm cray-librca-devel-1.0.0-2.0301.24564.5.18.gem.x86_64.rpm cray-librca-devel-1.0.0-2.0301.24564.5.18.ss.x86_64.rpm cray-libudreg0-2.1-1.0301.2797.5.2.gem.x86_64.rpm cray-libudreg-devel-2.1-1.0301.2797.5.2.gem.x86_64.rpm cray-libugni0-2.1-1.0301.2798.5.2.gem.x86_64.rpm cray-libugni-devel-2.1-1.0301.2798.5.2.gem.x86_64.rpm cray-libxpmem0-0.1-2.0301.24575.5.2.gem.x86_64.rpm cray-libxpmem-devel-0.1-2.0301.24575.5.2.gem.x86_64.rpm lsb-cray-hss-devel-5.1.0-1.0501.25011.337.x86_64.rpm pgroupd-lin64-v1161.tar.gz xe-sysroot-3.1.49A.securitypatch.20110131-1.x86_64.rpm xt-pgisup-10.9.0-3.x86_64.rpm xt-pgisup-11.0.0-5.x86_64.rpm xt-pgisup-11.1.0-4.x86_64.rpm xt-pgisup-9.0.4-7.x86_64.rpm xt-prgenv-xc-2.1.41HD-2.x86_64.rpm xt-prgenv-xc-2.1.50HD-2.x86_64.rpm xt-prgenv-xc-2.1.56HD-16778.x86_64.rpm xt-prgenv-xc-2.1.65HD-19576.x86_64.rpm xt-prgenv-xc-2.2.31-17782.x86_64.rpm xt-prgenv-xc-2.2.41-18649.x86_64.rpm xt-prgenv-xc-2.2.48B-19851.x86_64.rpm xt-prgenv-xc-2.2.67-22329.x86_64.rpm xt-prgenv-xc-2.2.73-24355.x86_64.rpm xt-prgenv-xc-2.2.74-24791.x86_64.rpm xt-sysroot-2.1.20110104-3.x86_64.rpm xt-sysroot-2.2.20110104-3.x86_64.rpm xt-sysroot-3.1.49A.securitypatch.20110131-1.x86_64.rpm xt-xcpe-2.1.41HD-14317.x86_64.rpm xt-xcpe-2.1.50HD-15784.x86_64.rpm xt-xcpe-2.1.56HD-16778.x86_64.rpm xt-xcpe-2.1.65HD-19576.x86_64.rpm xt-xcpe-2.2.31-17782.x86_64.rpm xt-xcpe-2.2.41-18649.x86_64.rpm xt-xcpe-2.2.48B-19851.x86_64.rpm xt-xcpe-2.2.67-22329.x86_64.rpm xt-xcpe-2.2.73-24355.x86_64.rpm xt-xcpe-2.2.74-24791.x86_64.rpm Installation Instructions: -------------------------- See Cray Application Developer's Environment Supplement Installation Guide and Release Notes S-2485-208 for additional details. S-2485-208 is available at http://docs.cray.com/. Latest PE Product Versions: =========================== This list contains the latest version of all PE products. Cray Compiling Environment CCE 7.3.2 cce 7.3.2 cray-prgenv 1.0.2 Flexnet license manager software Cray Debugger Supporting Tools CDST 1.2.1 atp 1.1.1 lgdb 1.2 mrnet 3.0.0 stat 1.1.3 Cray Environment Setup and Compiling support CEVN 5.08 xt-asyncpe 4.8 cray-modules 3.1.6.6 (for SLES 10/CLE 2.2) cray-modules 3.2.6.6-1.0000.3887.0.0.gem cray-modules 3.2.6.6-1.0000.3887.0.0.ss Chapel Chapel 1.2.1 chapel 1.2.1-gem chapel 1.2.1-ss Cray Message Passing Toolkit CMPT 5.2.0 cray-mpt 5.2.0-gem cray-mpt 5.2.0-ss cray-libpmi0 1.0-1.0000.8256.50 cray-libpmi-devel 1.0-1.0000.8256.50 Cray Performance Measurement & Analysis Tools CPMAT 5.1.3 apprentice2-desktop-5.1.3-999.tar perftools-5.1.3-6.x86_64.rpm Flexnet license manager software Cray Scientific and Math Libraries CSML 5.1.01 acml 4.4.0 fftw 2.1.5.2 fftw 3.2.2.1 libfast_mv 1.0.8 petsc 3.1.04 trilinos 10.6.2.0 xt-libsci 10.5.0 Allinea's Distributed Debugging Tool DDT Debugger 2.6.1 ddt-2.6_xt2.2-1.x86_64.rpm (CLE 2.2 and earlier) ddt-2.6_xt3.0-0.x86_64.rpm (CLE 3.0 and later) TotalView Debugger TotalView 8.9.0 cray-xt-totalview-8.9.0-0-totalview-support-1.1.1-3.tar PGI Compiler PGI 11.1.0 Third party products for the Programming environment Third Party Products 5.06 cray-gdb 7.2 hdf5_netcdf 1.8 netcdf 3.6.2 iobuf 2.0.1 java jdk1.6.0_20 ga 4.3.3 libonesided-ntk 1.0.0 cray-gcc-gmp 4.3.2 cray-gcc-mpc 0.8.1 cray-gcc-mpfr 2.4.2 xt-gcc 4.4.4 xt-gcc 4.5.2 intel-prgenv 1.0.0 xt-intelsup 12.0.2.137 xt-pathscalesup 3.2 xt-pathscalesup 3.2.99 Cray Application Developer's Environment CADE 5.08 CADE-5.08-02.iso Cray Application Developer's Environment Supplement CADES 2.08 CADES-2.08-02.iso Guide to Parallel Vector Applications 004–2182–003St. Peter’s Basilica image courtesy of ENEL SpA and InfoByte SpA. Disk Thrower image courtesy of Xavier Berenguer, Animatica. Copyright © 1994, 1995, 1999 Silicon Graphics, Inc. All Rights Reserved. This manual or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Silicon Graphics, Inc. LIMITED AND RESTRICTED RIGHTS LEGEND Use, duplication, or disclosure by the Government is subject to restrictions as set forth in the Rights in Data clause at FAR 52.227-14 and/or in similar or successor clauses in the FAR, or in the DOD, DOE or NASA FAR Supplements. Unpublished rights reserved under the Copyright Laws of the United States. Contractor/manufacturer is Silicon Graphics, Inc., 1600 Amphitheatre Pkwy., Mountain View, CA 94043-1351. Autotasking, CF77, Cray, Cray Ada, CraySoft, Cray Y-MP, Cray-1, CRInform, CRI/TurboKiva, HSX, LibSci, MPP Apprentice, SSD, SUPERCLUSTER, UNICOS, X-MP EA, and UNICOS/mk are federally registered trademarks and Because no workstation is an island, CCI, CCMT, CF90, CFT, CFT2, CFT77, ConCurrent Maintenance Tools, COS, Cray Animation Theater, Cray APP, Cray C90, Cray C90D, Cray C++ Compiling System, CrayDoc, Cray EL, Cray J90, Cray J90se, CrayLink, Cray NQS, Cray/REELlibrarian, Cray S-MP, Cray SSD-T90, Cray SV1, Cray T90, Cray T3D, Cray T3E, CrayTutor, Cray X-MP, Cray XMS, Cray-2, CSIM, CVT, Delivering the power . . ., DGauss, Docview, EMDS, GigaRing, HEXAR, IOS, ND Series Network Disk Array, Network Queuing Environment, Network Queuing Tools, OLNET, RQS, SEGLDR, SMARTE, SUPERLINK, System Maintenance and Remote Testing Environment, Trusted UNICOS, and UNICOS MAX are trademarks of Cray Research, L.L.C., a wholly owned subsidiary of Silicon Graphics, Inc. Silicon Graphics is a registered trademark and the Silicon Graphics logo is a trademark of Silicon Graphics, Inc. MS-DOS is a trademark of Microsoft Corporation. UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company Limited. VMS is a trademark of Digital Equipment Corporation. X/Open is a trademark of X/Open Company Ltd. X Window System and the X device are trademarks of the Open Group. Portions of this manual, speci?cally reference to the X Window System and X Toolkit, are based on reference materials that are copyrighted © 1988 by the Massachusetts Institute of Technology. Permissions to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, provided that the above copyright notice appear in all copies and that the both the copyright notice and this permission notice appear in supporting documentation, and that the name of M.I.T. not be used in advertising or publicity pertaining to distribution of the software without speci?c, written prior permission. M.I.T. makes no representation about the suitability of this software for any purpose. It is provided “as is” without express or implied warranty. The UNICOS operating system is derived from UNIX® System V. The UNICOS operating system is also based in part on the Fourth Berkeley Software Distribution (BSD) under license from The Regents of the University of California.New Features Guide to Parallel Vector Applications 004–2182–003 This revision changes the descriptions of some tools regarding support for the multistreaming option on Cray SV1 systems.Record of Revision Version Description 1.3 October 1994 Original Printing. Information contained in this document was previously found in the UNICOS Performance Utilities Reference Manual. 2.0 November 1995 This publication has been updated to support the CrayTools 2.0 release. 3.1 August 1998 This publication has been updated to support the CrayTools 3.1 release. 3.2 January 1999 This publication has been updated to support the CrayTools 3.2 release. 3.3 July 1999 This publication has been updated to support the CrayTools 3.3 release. 004–2182–003 iContents Page About This Guide xvii Related Publications . . . . . . . . . . . . . . . . . . . . . . . xvii Obtaining Publications . . . . . . . . . . . . . . . . . . . . . . . xvii Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . xviii Reader Comments . . . . . . . . . . . . . . . . . . . . . . . . xix Introduction [1] 1 Performance Measurement and Accuracy . . . . . . . . . . . . . . . . . 2 Tools Described in This Manual . . . . . . . . . . . . . . . . . . . . 3 Tools Described in Other Manuals . . . . . . . . . . . . . . . . . . . 4 Tool Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . 5 Impact of Tools . . . . . . . . . . . . . . . . . . . . . . . . . 6 Selection of Tools . . . . . . . . . . . . . . . . . . . . . . . . 7 Using the X Window System Interface . . . . . . . . . . . . . . . . . . 9 Getting Started . . . . . . . . . . . . . . . . . . . . . . . . 10 Common Options . . . . . . . . . . . . . . . . . . . . . . . 12 Resources . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . 14 Color . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Pointing Device . . . . . . . . . . . . . . . . . . . . . . . . 15 Component Naming Conventions . . . . . . . . . . . . . . . . . . . 15 Buttons . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Menus . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Keyboard Accelerators . . . . . . . . . . . . . . . . . . . . . . 16 Cray Research Logo . . . . . . . . . . . . . . . . . . . . . . . 17 Help Menu . . . . . . . . . . . . . . . . . . . . . . . . . 17 004–2182–003 iiiGuide to Parallel Vector Applications Page Help Selection . . . . . . . . . . . . . . . . . . . . . . . 17 Version Selection . . . . . . . . . . . . . . . . . . . . . . 18 Release Information Selection . . . . . . . . . . . . . . . . . 18 ATExpert [2] 19 Autotasking Expert System . . . . . . . . . . . . . . . . . . . . . 19 Getting Started with ATExpert . . . . . . . . . . . . . . . . . . . . 20 atexpert Command Line . . . . . . . . . . . . . . . . . . . . 23 Instrumenting Your Code . . . . . . . . . . . . . . . . . . . . . 24 Using the X Window System Interface . . . . . . . . . . . . . . . . . . 25 Main Window . . . . . . . . . . . . . . . . . . . . . . . . 25 Parallel Region Monitor . . . . . . . . . . . . . . . . . . . . . 25 Speedup Display . . . . . . . . . . . . . . . . . . . . . . . 26 Information Panel . . . . . . . . . . . . . . . . . . . . . . 27 File Menu . . . . . . . . . . . . . . . . . . . . . . . . . 27 Open. . . Selection . . . . . . . . . . . . . . . . . . . . . . 28 Open demo file Selection . . . . . . . . . . . . . . . . . . . 29 Quit Selection . . . . . . . . . . . . . . . . . . . . . . . 29 Display Menu . . . . . . . . . . . . . . . . . . . . . . . . 30 Subroutines. . . Selection . . . . . . . . . . . . . . . . . . . 31 Parallel Regions. . . Selection . . . . . . . . . . . . . . . . . 32 Loops. . . Selection . . . . . . . . . . . . . . . . . . . . . . 33 Case statements. . . Selection . . . . . . . . . . . . . . . . . . 34 Observations. . . Selection . . . . . . . . . . . . . . . . . . . 35 Report. . . Selection . . . . . . . . . . . . . . . . . . . . . 39 Source. . . Selection . . . . . . . . . . . . . . . . . . . . . 41 Options Menu . . . . . . . . . . . . . . . . . . . . . . . . 42 Using the ASCII Interface . . . . . . . . . . . . . . . . . . . . . . 44 Displaying Data . . . . . . . . . . . . . . . . . . . . . . . . 44 Generating Reports . . . . . . . . . . . . . . . . . . . . . . . 45 iv 004–2182–003Contents Page Interpreting ATExpert Graphs . . . . . . . . . . . . . . . . . . . . 46 Determining Serial Time . . . . . . . . . . . . . . . . . . . . . 47 Finding Large Serial Time . . . . . . . . . . . . . . . . . . . . . 47 Determining and Evaluating Key Performance Areas . . . . . . . . . . . . 47 Determining Parallel Overhead . . . . . . . . . . . . . . . . . . . 48 Interpreting Speedup Curves That Level Off or Plateau . . . . . . . . . . . . 48 Determining Dominant Autotasked Subroutines . . . . . . . . . . . . . . 49 Interpreting Bars and Messages . . . . . . . . . . . . . . . . . . . 49 Understanding ATExpert Concepts . . . . . . . . . . . . . . . . . . . 50 Instrumentation . . . . . . . . . . . . . . . . . . . . . . . . 50 Overhead Cost . . . . . . . . . . . . . . . . . . . . . . . . 51 atx.raw Output File . . . . . . . . . . . . . . . . . . . . . . 58 Variable Work Loops . . . . . . . . . . . . . . . . . . . . . . 62 Dedicated and Maximum Speedup Curves . . . . . . . . . . . . . . . . 63 Software Anomalies . . . . . . . . . . . . . . . . . . . . . . . 64 flowview [3] 67 The Flowtrace Feature . . . . . . . . . . . . . . . . . . . . . . . 67 Getting Started with the flowview Command . . . . . . . . . . . . . . . 68 flowview Command-line Options . . . . . . . . . . . . . . . . . . 68 Instrumenting Your Code . . . . . . . . . . . . . . . . . . . . . 72 Fortran Compiler . . . . . . . . . . . . . . . . . . . . . . . 73 Cray C++ and Cray C Compilers . . . . . . . . . . . . . . . . . . 73 Using the X Window System Interface . . . . . . . . . . . . . . . . . . 74 File Menu . . . . . . . . . . . . . . . . . . . . . . . . . 76 Open. . . Selection . . . . . . . . . . . . . . . . . . . . . . 77 Open Demo File Selection . . . . . . . . . . . . . . . . . . . 77 Capture Running Process Selection . . . . . . . . . . . . . . . . 77 004–2182–003 vGuide to Parallel Vector Applications Page Export Data... Selection . . . . . . . . . . . . . . . . . . . 77 Quit flowview Selection . . . . . . . . . . . . . . . . . . . . 78 Select Report Desired: Menu . . . . . . . . . . . . . . . . . . 78 Summary Selection . . . . . . . . . . . . . . . . . . . . . . 79 Routines Sorted by: Selection . . . . . . . . . . . . . . . . . 79 Caller/Callees Selection . . . . . . . . . . . . . . . . . . . 80 Environment Selection . . . . . . . . . . . . . . . . . . . . . 80 Text Calling Tree Selection . . . . . . . . . . . . . . . . . . 81 Report Preferences. . . Selection . . . . . . . . . . . . . . . . 81 Select Graph Desired: Menu . . . . . . . . . . . . . . . . . . 81 Pie Chart Selection . . . . . . . . . . . . . . . . . . . . . 82 Bar Chart Selection . . . . . . . . . . . . . . . . . . . . . 82 Calling Tree Selection . . . . . . . . . . . . . . . . . . . . 82 Text Report Selection . . . . . . . . . . . . . . . . . . . . . 83 Using the ASCII Interface . . . . . . . . . . . . . . . . . . . . . . 83 Using vi Windows . . . . . . . . . . . . . . . . . . . . . . . . 84 Additional Flowtrace Features . . . . . . . . . . . . . . . . . . . . 84 Selective Tracing . . . . . . . . . . . . . . . . . . . . . . . . 84 Using the FLOWMARK Subroutine . . . . . . . . . . . . . . . . . . 84 Using Other Methods . . . . . . . . . . . . . . . . . . . . . 87 SETPLIMQ Subroutine - Reporting Each Call . . . . . . . . . . . . . . . 88 flodump Command - for Programs That Terminated Abnormally . . . . . . . . 90 Environment Variables . . . . . . . . . . . . . . . . . . . . . . 90 Accuracy of CPU Times . . . . . . . . . . . . . . . . . . . . . . 91 Caution about Underlying Parallelism . . . . . . . . . . . . . . . . . . 92 Signal Handling in the Flowtrace Library . . . . . . . . . . . . . . . . . 92 flowview Sample Output . . . . . . . . . . . . . . . . . . . . . 93 Hardware Performance Monitor (hpm) [4] 97 vi 004–2182–003Contents Page hpm Command . . . . . . . . . . . . . . . . . . . . . . . . . 98 hpm Output . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Multiple hpm Execution . . . . . . . . . . . . . . . . . . . . . . 101 jumpview [5] 103 The Jumptrace Feature . . . . . . . . . . . . . . . . . . . . . . . 103 Getting Started with the jumpview Command . . . . . . . . . . . . . . . 104 jumpview Command-line Options . . . . . . . . . . . . . . . . . . 105 Gathering Data Using the jt Command . . . . . . . . . . . . . . . . 111 Symbols That the jumpview Command Uses . . . . . . . . . . . . . . 112 Instrumenting Your Code . . . . . . . . . . . . . . . . . . . . . 113 Compiler Debug Symbols . . . . . . . . . . . . . . . . . . . . . 114 Using the X Window System Interface . . . . . . . . . . . . . . . . . . 115 File Menu . . . . . . . . . . . . . . . . . . . . . . . . . 117 Open. . . Selection . . . . . . . . . . . . . . . . . . . . . . 118 Open Demo File Selection . . . . . . . . . . . . . . . . . . . 118 Export Data... Selection . . . . . . . . . . . . . . . . . . . 118 Quit jumpview Selection . . . . . . . . . . . . . . . . . . . . 118 Select Report Desired: Menu . . . . . . . . . . . . . . . . . . 119 Observations Selection . . . . . . . . . . . . . . . . . . . . 120 Summary Selection . . . . . . . . . . . . . . . . . . . . . . 121 Routines: Selection . . . . . . . . . . . . . . . . . . . . . 121 Details Within Routines: Selection . . . . . . . . . . . . . . . 122 Routine Name Search Selection . . . . . . . . . . . . . . . . . 123 Caller/Callees Selection . . . . . . . . . . . . . . . . . . . 123 Disassemble Selection . . . . . . . . . . . . . . . . . . . . . 123 Groups: Selection . . . . . . . . . . . . . . . . . . . . . . 123 Group Name Search Selection . . . . . . . . . . . . . . . . . . 124 Show Groups Selection . . . . . . . . . . . . . . . . . . . . . 124 004–2182–003 viiGuide to Parallel Vector Applications Page Environment Selection . . . . . . . . . . . . . . . . . . . . . 124 Report Preferences. . . Selection . . . . . . . . . . . . . . . . 124 Select Graph Desired: Menu . . . . . . . . . . . . . . . . . . 125 Pie Chart Selection . . . . . . . . . . . . . . . . . . . . . 126 Group Pie Chart Selection . . . . . . . . . . . . . . . . . . . 127 Bar Chart Selection . . . . . . . . . . . . . . . . . . . . . 127 Text Report Selection . . . . . . . . . . . . . . . . . . . . . 127 Text Calling Tree Selection . . . . . . . . . . . . . . . . . . 127 Using the ASCII Interface . . . . . . . . . . . . . . . . . . . . . . 127 Using vi Windows . . . . . . . . . . . . . . . . . . . . . . . . 128 Accuracy of CPU Times . . . . . . . . . . . . . . . . . . . . . . 129 jumpview Sample Output . . . . . . . . . . . . . . . . . . . . . 129 perfview [6] 135 The Perftrace Feature . . . . . . . . . . . . . . . . . . . . . . . 135 Getting Started with the perfview Command . . . . . . . . . . . . . . . 136 perfview Command-line Options . . . . . . . . . . . . . . . . . . 136 Instrumenting Your Code . . . . . . . . . . . . . . . . . . . . . 143 CF90 Compiler . . . . . . . . . . . . . . . . . . . . . . . 144 Cray C++ and Cray C Compilers . . . . . . . . . . . . . . . . . . 144 Using the X Window System Interface . . . . . . . . . . . . . . . . . . 145 File Menu . . . . . . . . . . . . . . . . . . . . . . . . . 147 Open. . . Selection . . . . . . . . . . . . . . . . . . . . . . 148 Open Demo File Selection . . . . . . . . . . . . . . . . . . . 148 Capture Running Process Selection . . . . . . . . . . . . . . . . 148 Export Data... Selection . . . . . . . . . . . . . . . . . . . 148 Quit perfview Selection . . . . . . . . . . . . . . . . . . . . 149 Select Report Desired: Menu . . . . . . . . . . . . . . . . . . 149 Observations Selection . . . . . . . . . . . . . . . . . . . . 150 viii 004–2182–003Contents Page Summary Selection . . . . . . . . . . . . . . . . . . . . . . 151 Show Routines Sorted by: Selection . . . . . . . . . . . . . . . 151 Program Summary Selection . . . . . . . . . . . . . . . . . . . 152 Full Program Report Selection . . . . . . . . . . . . . . . . . 152 Machine Workload Summary Selection . . . . . . . . . . . . . . . 152 Full Machine Workload Report Selection . . . . . . . . . . . . . 152 User-Defined Reports: Selection . . . . . . . . . . . . . . . . 152 Environment Selection . . . . . . . . . . . . . . . . . . . . . 153 Report Preferences. . . Selection . . . . . . . . . . . . . . . . 153 Select Graph Desired: Menu . . . . . . . . . . . . . . . . . . 154 Pie Chart Selection . . . . . . . . . . . . . . . . . . . . . 155 Bar Chart Selection . . . . . . . . . . . . . . . . . . . . . 155 Text Report Selection . . . . . . . . . . . . . . . . . . . . . 155 Using the ASCII Interface . . . . . . . . . . . . . . . . . . . . . . 155 Using vi Windows . . . . . . . . . . . . . . . . . . . . . . . . 156 Additional Perftrace Features . . . . . . . . . . . . . . . . . . . . . 157 Selective Tracing . . . . . . . . . . . . . . . . . . . . . . . . 157 Using the FLOWMARK Subroutine . . . . . . . . . . . . . . . . . . 157 Using Other Methods . . . . . . . . . . . . . . . . . . . . . 159 SETPLIMQ Subroutine - Reporting Each Call . . . . . . . . . . . . . . . 161 perfdmp Command - Programs That Terminated Abnormally . . . . . . . . . . 161 Environment Variables . . . . . . . . . . . . . . . . . . . . . . 162 Throttle Feature . . . . . . . . . . . . . . . . . . . . . . . 163 Perftrace Statistics and Overhead . . . . . . . . . . . . . . . . . . . 164 Caution about Underlying Parallelism . . . . . . . . . . . . . . . . . . 164 Signal Handling in the Perftrace Library . . . . . . . . . . . . . . . . . 164 perfview Report Output . . . . . . . . . . . . . . . . . . . . . . 165 Command-line User-de?ned Reports . . . . . . . . . . . . . . . . . 165 Command-line Observation . . . . . . . . . . . . . . . . . . . . 166 004–2182–003 ixGuide to Parallel Vector Applications Page User-de?ned Reports . . . . . . . . . . . . . . . . . . . . . . . 166 Report File . . . . . . . . . . . . . . . . . . . . . . . . . 167 Report Records . . . . . . . . . . . . . . . . . . . . . . . . 167 Comment Records . . . . . . . . . . . . . . . . . . . . . . 167 Marker Records to Start User-variable De?nitions . . . . . . . . . . . . . 168 User-variable De?nition Records . . . . . . . . . . . . . . . . . . 168 Marker Records to Identify a Report . . . . . . . . . . . . . . . . . 169 Heading De?nition Records . . . . . . . . . . . . . . . . . . . 170 Report Field De?nition Records . . . . . . . . . . . . . . . . . . 170 Prede?ned Variables . . . . . . . . . . . . . . . . . . . . . . 173 Processing of Variables . . . . . . . . . . . . . . . . . . . . . . 178 Units Designators . . . . . . . . . . . . . . . . . . . . . . . 179 Report Generation . . . . . . . . . . . . . . . . . . . . . . . 181 Full Report De?nition File Examples . . . . . . . . . . . . . . . . . 181 Example 1 . . . . . . . . . . . . . . . . . . . . . . . . . 181 Example 2 . . . . . . . . . . . . . . . . . . . . . . . . . 184 Example 3 . . . . . . . . . . . . . . . . . . . . . . . . . 185 Example 4 . . . . . . . . . . . . . . . . . . . . . . . . . 187 perfview Reports . . . . . . . . . . . . . . . . . . . . . . . . 190 procview [7] 197 Process Monitoring . . . . . . . . . . . . . . . . . . . . . . . . 197 Getting Started with the procview Command . . . . . . . . . . . . . . . 198 procview Command-line Options . . . . . . . . . . . . . . . . . . 198 Gathering Data with the procstat Command . . . . . . . . . . . . . . 203 Using the X Window System Interface . . . . . . . . . . . . . . . . . . 204 File Menu . . . . . . . . . . . . . . . . . . . . . . . . . 206 Open. . . Selection . . . . . . . . . . . . . . . . . . . . . . 207 x 004–2182–003Contents Page Open Demo File Selection . . . . . . . . . . . . . . . . . . . 207 Export Data... Selection . . . . . . . . . . . . . . . . . . . 207 Quit procview Selection . . . . . . . . . . . . . . . . . . . . 207 Reports Menu . . . . . . . . . . . . . . . . . . . . . . . . 208 Files Selection . . . . . . . . . . . . . . . . . . . . . . . 208 Processes Selection . . . . . . . . . . . . . . . . . . . . . 209 Miscellaneous Selection . . . . . . . . . . . . . . . . . . . . 209 Environment Selection . . . . . . . . . . . . . . . . . . . . . 210 Report Preferences. . . Selection . . . . . . . . . . . . . . . . 210 Graphs Menu . . . . . . . . . . . . . . . . . . . . . . . . 211 Pie Chart Selection . . . . . . . . . . . . . . . . . . . . . 211 Bar Chart Selection . . . . . . . . . . . . . . . . . . . . . 211 Text Report Selection . . . . . . . . . . . . . . . . . . . . . 212 Generate X/Y Plots Selection . . . . . . . . . . . . . . . . . . 212 Using the ASCII Interface . . . . . . . . . . . . . . . . . . . . . . 212 Using vi Windows . . . . . . . . . . . . . . . . . . . . . . . . 213 procview Sample Output . . . . . . . . . . . . . . . . . . . . . 214 profview [8] 219 The Pro?ling Feature . . . . . . . . . . . . . . . . . . . . . . . 219 Getting Started with the profview Command . . . . . . . . . . . . . . . 221 profview Command-line Options . . . . . . . . . . . . . . . . . . 222 Gathering Data with the prof Command . . . . . . . . . . . . . . . . 226 prof Command-line Options . . . . . . . . . . . . . . . . . . . 227 prof Final Reports . . . . . . . . . . . . . . . . . . . . . . 228 prof Command Reports . . . . . . . . . . . . . . . . . . . . 229 Symbols that the prof Command Uses . . . . . . . . . . . . . . . . 230 Using prof Environment Variables . . . . . . . . . . . . . . . . . 231 Instrumenting Your Code . . . . . . . . . . . . . . . . . . . . . 233 004–2182–003 xiGuide to Parallel Vector Applications Page Compiler Debug Symbols . . . . . . . . . . . . . . . . . . . . . 233 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 234 Using the X Window System Interface . . . . . . . . . . . . . . . . . . 235 File Menu . . . . . . . . . . . . . . . . . . . . . . . . . 237 Open. . . Selection . . . . . . . . . . . . . . . . . . . . . . 238 Open Demo File Selection . . . . . . . . . . . . . . . . . . . 238 Capture Running Process Selection . . . . . . . . . . . . . . . . 238 Export Data... Selection . . . . . . . . . . . . . . . . . . . 238 Quit profview Selection . . . . . . . . . . . . . . . . . . . . 239 Select Report Desired: Menu . . . . . . . . . . . . . . . . . . 239 Summary Selection . . . . . . . . . . . . . . . . . . . . . . 240 Modules: Selection . . . . . . . . . . . . . . . . . . . . . . 240 Details Within Modules: Selection . . . . . . . . . . . . . . . 241 Symbols: Selection . . . . . . . . . . . . . . . . . . . . . . 242 Buckets: Selection . . . . . . . . . . . . . . . . . . . . . . 242 Raw Bucket List Selection . . . . . . . . . . . . . . . . . . . 243 Raw Bucket Histogram Selection . . . . . . . . . . . . . . . . . 243 Environment Selection . . . . . . . . . . . . . . . . . . . . . 243 Report Preferences. . . Selection . . . . . . . . . . . . . . . . 243 Select Graph Desired: Menu . . . . . . . . . . . . . . . . . . 244 Pie Chart Selection . . . . . . . . . . . . . . . . . . . . . 245 Bar Chart Selection . . . . . . . . . . . . . . . . . . . . . 245 Memory Graph Selection . . . . . . . . . . . . . . . . . . . . 245 Text Report Selection . . . . . . . . . . . . . . . . . . . . . 245 Using the ASCII Interface . . . . . . . . . . . . . . . . . . . . . . 245 Using vi Windows . . . . . . . . . . . . . . . . . . . . . . . . 246 Usage Considerations . . . . . . . . . . . . . . . . . . . . . . . 247 Strategies for Best Results . . . . . . . . . . . . . . . . . . . . . 247 Reporting Parts of Programs . . . . . . . . . . . . . . . . . . . . 247 xii 004–2182–003Contents Page profview Sample Output . . . . . . . . . . . . . . . . . . . . . 248 Appendix A Hardware Performance Monitor Counters 251 Appendix B Interpreting Performance Statistics for Cray C90 Series Systems 259 HPM Statistics . . . . . . . . . . . . . . . . . . . . . . . . . 260 Overview of General Statistics . . . . . . . . . . . . . . . . . . . 261 Balance of Operations . . . . . . . . . . . . . . . . . . . . . 262 Mega?op Ratings . . . . . . . . . . . . . . . . . . . . . . . 263 CPU Memory References . . . . . . . . . . . . . . . . . . . . 263 I/O Memory References . . . . . . . . . . . . . . . . . . . . 264 B/T Memory References . . . . . . . . . . . . . . . . . . . . 265 Hold-issue Statistics . . . . . . . . . . . . . . . . . . . . . . . 265 Instruction Summary . . . . . . . . . . . . . . . . . . . . . . 267 Vector Operations . . . . . . . . . . . . . . . . . . . . . . . 269 Average Vector Length . . . . . . . . . . . . . . . . . . . . . . 271 Examples of Hardware Performance Monitor Statistics . . . . . . . . . . . . . 272 Highly Scalar Code . . . . . . . . . . . . . . . . . . . . . . . 272 Highly Vectorized Code . . . . . . . . . . . . . . . . . . . . . 274 Code That Performs in One Vector Functional Unit . . . . . . . . . . . . . 276 Code That Performs in Two Vector Functional Units . . . . . . . . . . . . . 278 Extensive Memory Bank Con?icts . . . . . . . . . . . . . . . . . . 280 Memory Con?ict Removed . . . . . . . . . . . . . . . . . . . . 282 Calling Overhead . . . . . . . . . . . . . . . . . . . . . . . 284 Spaghetti Code . . . . . . . . . . . . . . . . . . . . . . . . 286 Autotasked Example . . . . . . . . . . . . . . . . . . . . . . 288 Typical User Code . . . . . . . . . . . . . . . . . . . . . . . 290 Nearly Peak Speed . . . . . . . . . . . . . . . . . . . . . . . 291 Vectorized Logical Operations (with AVL) . . . . . . . . . . . . . . . . 293 004–2182–003 xiiiGuide to Parallel Vector Applications Page Vectorized Logical Operations (AVL Disabled) . . . . . . . . . . . . . . . 295 Index 297 Figures Figure 1. Help Menu . . . . . . . . . . . . . . . . . . . . . . 17 Figure 2. Main ATExpert Window . . . . . . . . . . . . . . . . . . . 22 Figure 3. File Menu . . . . . . . . . . . . . . . . . . . . . . 27 Figure 4. ATExpert Load File Input Panel . . . . . . . . . . . . . . . . 28 Figure 5. Display Menu . . . . . . . . . . . . . . . . . . . . . 30 Figure 6. Autotasked Subroutines Display . . . . . . . . . . . . . . . . 31 Figure 7. Parallel Regions Display . . . . . . . . . . . . . . . . . . . 33 Figure 8. Loops Display . . . . . . . . . . . . . . . . . . . . . . 34 Figure 9. Case Statements Display . . . . . . . . . . . . . . . . . . 35 Figure 10. Observations Display . . . . . . . . . . . . . . . . . . . 36 Figure 11. Observation Con?guration Options Display . . . . . . . . . . . . 37 Figure 12. Report Display . . . . . . . . . . . . . . . . . . . . . 40 Figure 13. Options Menu . . . . . . . . . . . . . . . . . . . . . 42 Figure 14. Con?guration Options Display . . . . . . . . . . . . . . . . 42 Figure 15. Overhead Breakdown . . . . . . . . . . . . . . . . . . . 53 Figure 16. Main flowview Window . . . . . . . . . . . . . . . . . . 76 Figure 17. File Menu . . . . . . . . . . . . . . . . . . . . . . 77 Figure 18. Select Report Desired: Menu . . . . . . . . . . . . . . . 79 Figure 19. flowview Report Preferences Window . . . . . . . . . . . . . . 81 Figure 20. Select Graph Desired: Menu . . . . . . . . . . . . . . . 82 Figure 21. Main jumpview Window . . . . . . . . . . . . . . . . . . 117 Figure 22. File Menu . . . . . . . . . . . . . . . . . . . . . . 118 Figure 23. Select Report Desired: Menu . . . . . . . . . . . . . . . 120 xiv 004–2182–003Contents Page Figure 24. jumpview Report Preferences Window . . . . . . . . . . . . . . 125 Figure 25. Select Graph Desired: Menu . . . . . . . . . . . . . . . 126 Figure 26. Main perfview Window . . . . . . . . . . . . . . . . . . 147 Figure 27. File Menu . . . . . . . . . . . . . . . . . . . . . . 148 Figure 28. Select Report Desired: Menu . . . . . . . . . . . . . . . 150 Figure 29. perfview Report Preferences Window . . . . . . . . . . . . . . 154 Figure 30. Select Graph Desired: Menu . . . . . . . . . . . . . . . 154 Figure 31. Main procview Window . . . . . . . . . . . . . . . . . . 206 Figure 32. File Menu . . . . . . . . . . . . . . . . . . . . . . 207 Figure 33. Reports Menu . . . . . . . . . . . . . . . . . . . . . 208 Figure 34. procview Report Preferences Window . . . . . . . . . . . . . . 210 Figure 35. Graphs Menu . . . . . . . . . . . . . . . . . . . . . 211 Figure 36. procview Plotting Control Panel . . . . . . . . . . . . . . . 212 Figure 37. Main profview Window . . . . . . . . . . . . . . . . . . 237 Figure 38. File Menu . . . . . . . . . . . . . . . . . . . . . . 238 Figure 39. Select Report Desired: Menu . . . . . . . . . . . . . . . 240 Figure 40. profview Report Preferences Window . . . . . . . . . . . . . . 244 Figure 41. Select Graph Desired: Menu . . . . . . . . . . . . . . . 244 Tables Table 1. Tools Described in This Manual . . . . . . . . . . . . . . . . . 3 Table 2. Other Performance Analysis Tools . . . . . . . . . . . . . . . . 5 Table 3. User Interfaces Available for the Commands . . . . . . . . . . . . . 6 Table 4. Performance Impact . . . . . . . . . . . . . . . . . . . . 7 Table 5. Information Provided by the Tools . . . . . . . . . . . . . . . . 8 Table 6. X Toolkit Command-line Arguments . . . . . . . . . . . . . . . 12 Table 7. Compiler Debugging Information . . . . . . . . . . . . . . . . 114 Table 8. Perftrace Environment Variables . . . . . . . . . . . . . . . . 162 Table 9. Compiler Debugging Information . . . . . . . . . . . . . . . . 233 004–2182–003 xvGuide to Parallel Vector Applications Page Table 10. Cray T90 Series without IEEE Floating-point Arithmetic System Counters . . . 251 Table 11. Cray T90 Series with IEEE Floating-point Arithmetic System Counters . . . . 253 Table 12. Cray C90 System Counters . . . . . . . . . . . . . . . . . . 254 Table 13. Cray SV1 and Cray J90 System Counters . . . . . . . . . . . . . 256 xvi 004–2182–003About This Guide This publication documents the performance utilities being supported by CrayTools, which is a part of the Cray Programming Environment release. These performance utilities run on UNICOS operating systems. Related Publications The following documents contain additional information that may be helpful: • Compiler Information File (CIF) Reference Manual • UNICOS User Commands Reference Manual • UNICOS File Formats and Special Files Reference Manual • UNICOS System Libraries Reference Manual • Cray C/C++ Reference Manual • CF90 Commands and Directives Reference Manual • Fortran Language Reference Manual, Volume 1 • Fortran Language Reference Manual, Volume 2 • Fortran Language Reference Manual, Volume 3 • OpenMP Fortran Application Program Interface Obtaining Publications The User Publications Catalog describes the availability and content of all Cray Research hardware and software documents that are available to customers. Customers who subscribe to the Cray Inform (CRInform) program can access this information on the CRInform system. To order a document, call +1 651 683 5907. Silicon Graphics employees may send electronic mail to orderdsk@sgi.com (UNIX system users). Customers who subscribe to the CRInform program can order software release packages electronically by using the Order Cray Software option. 004–2182–003 xviiGuide to Parallel Vector Applications Customers outside of the United States and Canada should contact their local service organization for ordering and documentation information. Conventions The following conventions are used throughout this document: Convention Meaning command This ?xed-space font denotes literal items such as commands, ?les, routines, path names, signals, messages, and programming language structures. manpage(x) Man page section identi?ers appear in parentheses after man page names. The following list describes the identi?ers: 1 User commands 1B User commands ported from BSD 2 System calls 3 Library routines, macros, and opdefs 4 Devices (special ?les) 4P Protocols 5 File formats 7 Miscellaneous topics 7D DWB-related information 8 Administrator commands Some internal routines (for example, the _assign_asgcmd_info() routine) do not have man pages associated with them. variable Italic typeface denotes variable entries and words or concepts being de?ned. user input This bold, ?xed-space font denotes literal items that the user enters in interactive sessions. Output is shown in nonbold, ?xed-space font. abbreviation Underlining indicates the shortest possible abbreviation for a command. xviii 004–2182–003About This Guide [ ] Brackets enclose optional portions of a command or directive line. ... Ellipses indicate that a preceding element can be repeated. The default shell in the UNICOS and UNICOS/mk operating systems, referred to as the standard shell, is a version of the Korn shell that conforms to the following standards: • Institute of Electrical and Electronics Engineers (IEEE) Portable Operating System Interface (POSIX) Standard 1003.2–1992 • X/Open Portability Guide, Issue 4 (XPG4) The UNICOS and UNICOS/mk operating systems also support the optional use of the C shell. Cray UNICOS Version 10.0 is an X/Open Base 95 branded product. Reader Comments If you have comments about the technical accuracy, content, or organization of this document, please tell us. Be sure to include the title and part number of the document with your comments. You can contact us in any of the following ways: • Send e-mail to the following address: techpubs@sgi.com • Send a fax to the attention of “Technical Publications” at: +1 650 932 0801. • Use the Feedback option on the Technical Publications Library World Wide Web page: http://techpubs.sgi.com • Call the Technical Publications Group, through the Technical Assistance Center, using one of the following numbers: For SGI IRIX based operating systems: 1 800 800 4SGI For UNICOS or UNICOS/mk based operating systems or Cray Origin 2000 systems: 1 800 950 2729 (toll free from the United States and Canada) or +1 651 683 5600 004–2182–003 xixGuide to Parallel Vector Applications • Send mail to the following address: Technical Publications SGI 1600 Amphitheatre Pkwy. Mountain View, California 94043–1351 We value your comments and will respond to them promptly. xx 004–2182–003Introduction [1] This manual describes tools for analyzing and tuning the performance of parallel vector application programs. To help clarify the purposes, features, and requirements of the tools, this chapter discusses the following material: • Performance measurement and accuracy • Tools described in this manual • Tools described in other manuals • Tool interfaces • Impact of tools • Selection of tools • Using the X Window System interface • Component naming conventions The following tools are described in this manual in alphabetical order: • atexpert(1) - Gauges the effectiveness of Autotasking, by predicting the speedup in a dedicated environment • flowview(1) - Displays times of subroutines and functions instrumented by the Flowtrace feature • hpm(1) - Monitors machine performance during program execution • jumpview(1) - Displays times of subroutines and functions generated by the Jumptrace feature • perfview(1) - Displays times of subroutines and functions instrumented by the Perftrace feature and the Hardware Performance Monitor device • procview(1) - Displays program execution statistics on I/O, process, and memory generated by the procstat(1) command • profview(1) - Displays the relative execution time used by individual parts of your program and generated by the Pro?ling feature Table 1, page 3, through Table 5, page 8, in the following sections help de?ne and differentiate the commands, subroutines, and features that are used in 004–2182–003 1Guide to Parallel Vector Applications performance tuning. Within the tables, the tools are listed alphabetically, irrespective of capitalization. 1.1 Performance Measurement and Accuracy Tuning your code for better performance provides the following bene?ts: • Reduces job turn-around time • Makes ef?cient use of the computing resources available Most of the tools in this manual measure or report information on the run-time performance of an application running under the UNICOS operating system. The performance data may re?ect CPU execution time, multitasking ef?ciency, I/O use, or other types of data. When you use the results of these tools, it is important to remember that the numbers presented are not exact but are affected by the following factors: • Instrumentation technique • Granularity of information collection (for example, collecting from the program versus a routine or loop), especially compared to instrumentation overhead • Other processes running on the UNICOS system • Statistical variation These factors produce the following inaccuracies: • Numerical results from two runs of the same program with the same instrumentation technique will likely differ. • Results from two runs of the same program with different instrumentation techniques, even if they purport to measure the same factors, will likely differ. In most practical situations, these variations are not important. For example, although one tool indicates that a certain loop takes 25% of a program’s CPU time and a second tool indicates 27%, both results identify a loop that should be examined. Further, the fact that differences exist may provide you with additional information. For example, the Pro?ling feature can show a heavily used routine, but the Flowtrace feature can help identify a single call in the source code that is accounting for most of the use of that routine. The fact that the numbers 2 004–2182–003Introduction [1] displayed are different re?ects different views of the code, not that one is more correct than the other. 1.2 Tools Described in This Manual For every tool that gathers data from the execution of programs, a corresponding data reporting tool exists. Table 1, page 3 lists the tools that are described in this manual. Table 1. Tools Described in This Manual Tool Type Description Section atexpert(1) Command Predicts speedups of parallel processing in an autotasked program. The prediction is for a dedicated system and is made from data collected from one run on a system that is not dedicated. Chapter 2, page 19 flowview(1) Command Displays information from the raw data generated by the Flowtrace library. Chapter 3, page 67 Flowtrace Feature Times subroutines and functions during program execution. The flowview(1) command displays Flowtrace data. Chapter 3, page 67 FLOWMARK Subroutine Times speci?ed areas of programs (not necessarily subroutines or functions). FLOWMARK is a special feature of Flowtrace and Perftrace. Chapter 3, page 67 hpm(1) Command Gathers Hardware Performance Monitor (HPM) statistics for an entire program. The perfview(1) command displays data from the hpm(1) command. Chapter 4, page 97 jumpview(1) Command Displays information from the raw data generated by the jt(1) command. Chapter 5, page 103 jt(1) Command Gathers timing statistics on code during execution. The jumpview(1) command displays data generated by the jt(1) command. Chapter 5, page 103 Jumptrace Feature Provides exact timings of subroutines and functions during program execution. The jumpview(1) command displays Jumptrace data gathered by the jt(1) command. Chapter 5, page 103 004–2182–003 3Guide to Parallel Vector Applications Tool Type Description Section perfview(1) Command Displays information from raw data generated by the Perftrace library, the hpm(1) command, or the hpmall(8) administrator command. Chapter 6, page 135 Perftrace Feature Times subroutines and functions based on statistics gathered from the Hardware Performance Monitor (HPM) device. The perfview(1) command displays Perftrace data. Chapter 6, page 135 procview(1) Command Displays information from the raw data generated by the procstat(1) command. Chapter 7, page 197 procstat(1) Command Gathers statistics about I/O, process, and memory as a program executes. The procview(1) command displays data generated by the procstat(1) command. Chapter 7, page 197 profview(1) Command Displays information from the raw data generated by the prof(1) command. Chapter 8, page 219 Pro?ling Feature Statistically gathers the relative execution time used by individual parts of a program. The profview(1) command displays Pro?ling data. Chapter 8, page 219 1.3 Tools Described in Other Manuals Table 2, page 5, lists tools that are related to performance analysis but are not described in this manual. See the associated man pages for information on these features. 4 004–2182–003Introduction [1] Table 2. Other Performance Analysis Tools Tool Type Description ftnlist(1) Command Generates a Fortran listing. hpmall(8) Command Gathers Hardware Performance Monitor (HPM) statistics for the entire system workload. The perfview(1) command displays data generated by the hpmall(8) command. hpmflop(1) Command Reports hardware performance statistics for individual user jobs. ja(1) Command Provides accounting data about many aspects of job processing. mtdump(1) Command Produces various noninteractive reports from data written by the multitasking history trace feature. SECOND(3) Subroutine Reports elapsed CPU time accumulated by all processes in a program. time(1) Command Reports elapsed and execution times for a command. xbrowse(1) Command Provides an interactive, graphic interface in which to view and edit Fortran. 1.4 Tool Interfaces Table 3, page 6, identi?es the user interfaces available for the performance tools that display output. Other tools, which gather data to send to a ?le, are not listed here. The headings in Table 3, page 6, have the following meanings: • In the Command line column, "Yes" means that command-line options can be used to produce reports in a ?le. • In the Line-mode interactive column, "Yes" means that interactive commands can be used to control output to a terminal. • In the X Window System column, "Yes" means that an X Window System interface can be used to control and display tool output. 004–2182–003 5Guide to Parallel Vector Applications Table 3. User Interfaces Available for the Commands Tool Command line Line-mode interactive X Window System atexpert Yes Yes Yes flowview Yes Yes Yes hpm Yes - 1 - 1 hpmall Yes - 1 - 1 ja Yes - - jumpview Yes Yes Yes mtdump Yes - - perfview Yes Yes Yes procstat Yes - procview Yes Yes Yes profview Yes Yes Yes time Yes - - 1.5 Impact of Tools Table 4, page 7, itemizes some considerations and requirements for using the performance tools that run with programs. The table does not include reporting tools. In Table 4, page 7, the headings have the following meanings: • In the CPU overhead column, "Low" indicates the tool incurs no signi?cant overhead; "High" indicates the tool does incur signi?cant overhead. • In the Recompile, Reload, and Change source columns, "Yes" indicates the tool requires program recompiling or reloading, or source code changes; a dash (-) indicates it does not require program recompiling or reloading, or source code changes. • In the Multitask column, "Yes" means the tool operates correctly with multitasked programs. The abbreviation "Cond" (conditional) indicates that 1 To report hpm and hpmall command output, use the perfview command. 6 004–2182–003Introduction [1] the tools (Perftrace and procstat) can be run successfully with a multitasked program if the NCPUS environment variable is set to 1 before the user program begins execution. Table 4. Performance Impact Tool CPU overhead Recompile Reload Change source Multitask atexpert Low 2 Yes Yes - Yes Flowtrace High Yes Yes - Yes 3 , Cond ja Low - - - Yes Jumptrace High Recommended Yes - Yes Perftrace High Yes Yes - Cond FLOWMARK High Yes Yes Yes Cond procstat Low - - - Cond Pro?ling Low Recommended Yes - Yes 1.6 Selection of Tools Table 5, page 8, categorizes the tools according to the information they provide. The categories appear from general to speci?c; the tools named for each category are in the suggested order of use. 2 Overhead is inversely proportional to the granularity and number of invocations of parallel regions. 3 On UNICOS systems, the Flowtrace feature works with all types of Autotasking (automatically detected tasking and directive-based tasking), and with macrotasking that involves calls to the tskstart(3) routine, but it does not work with multitasking using the t_fork(3) routine. 004–2182–003 7Guide to Parallel Vector Applications Table 5. Information Provided by the Tools To identify . . . Use these tools CPU time used by an entire program /bin/time (not C shell time) ja STOP Fortran statement 4 Routine using the most CPU time Pro?ling (prof and profview) Flowtrace (flowview) Perftrace (perfview) atexpert (used with Autotasking) Jumptrace (jt and jumpview) CPU time used by one routine Flowtrace (flowview) Perftrace (perfview) FLOWMARK (Flowtrace, Perftrace) Pro?ling (prof and profview) Jumptrace (jt and jumpview) atexpert (used with Autotasking) CPU time used in one loop or code section Pro?ling (prof and profview) FLOWMARK (Flowtrace, Perftrace) Jumptrace (jt and jumpview) atexpert (used with Autotasking) CPU time used in library routines Pro?ling (prof and profview) Jumptrace (jt and jumpview) Routines that you should consider inlining Flowtrace (flowview) Perftrace (perfview) Jumptrace (jt and jumpview) Program’s call tree (dynamic) Flowtrace (flowview) Ef?ciency of system hardware use hpm (perfview) Perftrace (perfview) Jumptrace (jt and jumpview) Ef?ciency of parallel processing atexpert (used with Autotasking) ja hpm mtdump (used with macrotasking) 4 See your CF90 language reference manual for more information on the STOP statement. 8 004–2182–003Introduction [1] To identify . . . Use these tools Amount and speed of program I/O procstat and procview ja Amount of central memory used procstat and procview STOP Fortran statement 4 Process creation/termination history procstat and procview ja Scope of variables for Autotasking xbrowse Use of Fortran common variables throughout program xbrowse 1.7 Using the X Window System Interface This section assumes that you have the X Window System running on your workstation and are familiar with its operation. The utilities referred to in this chapter (and documented elsewhere in this manual) run on UNICOS systems and communicate with the X Window System server software on your workstation. These performance utilities have been run with workstations manufactured by several vendors, but differences in implementation of the X Window System may affect the behavior, appearance, or in extreme cases, the usability of the utilities on a particular system. If you experience dif?culty running the utilities on your workstation, report the problem(s) to your system support staff. The X Window System was developed at the Massachusetts Institute of Technology in 1984. It has been ported to most UNIX operating systems, and it also runs under the VMS operating system and the MS-DOS operating system. The UNICOS systems are based on the UNIX operating system, which also supports the X Window System. For more information on the X Window System, see any of the following O’Reilly and Associates, Inc., publications: • XLIB Programming Manual • X Window System User’s Guide • X Toolkit Intrinsic Programming Manual The following components of the X Window System interface will be discussed: • Getting started • Common options 004–2182–003 9Guide to Parallel Vector Applications • Resources • Troubleshooting • Color • Pointing device 1.7.1 Getting Started To use the X Window System interface with the performance utilities, you must be using a workstation that is running the X Window System, version 11, release 3 or after. If your workstation does not support the X Window System, you cannot use this interface to the performance utilities, and you must use the line-mode version (see individual descriptions of the performance utilities). Contact your system administrator for information about the availability of the X Window System. When your workstation is running the X Window System, use TCP/IP to establish a session on the UNICOS system. Be sure that the DISPLAY environment variable speci?es the screen on which your performance utilities window should be presented. For example, if your workstation name is ursa, set the environment variable as follows: Standard shell (sh) and Korn ( ksh) shells: $ DISPLAY=ursa:0.0;export DISPLAY C shell (csh): % setenv DISPLAY ursa:0.0 You must ensure that the system can make connections to the X Window System ?leserver by executing the xhost client command on your workstation. Many users include such a command in their .xinitrc ?le. The following example adds host sn2022 to the access control list for the workstation: % xhost + sn2022 The xhost(1) command is invoked on your workstation, not on the system. When you enter any of the performance utility commands noted in this manual, the command detects whether to invoke the X Window System afterchecking the following options in the order listed: 10 004–2182–003Introduction [1] 1. If the -L option is speci?ed on a performance utility command line, the utility assumes that you want the line-mode interface and does not attempt to run the X Window System mode. 2. If the -L option is not speci?ed on the utility command line, the utility ?rst checks for the presence of the DISPLAY environment variable and/or the -display command-line option. If either of these is speci?ed, the tool will try to open contact with the speci?ed workstation. If this is successful, the tool will run in the X Window System mode. If the contact is unsuccessful, the tool will run in interactive line-mode. 3. If you specify any of the following options on the command line, the utility assumes that you want the noninteractive command-line mode (flowview, jumpview, procview, perfview, or profview) or interactive line-mode (ATExpert): flowview: -A, -a, -c, -H, -h, -j, -k, -l, -m, -T, -u, -v jumpview: -A, -c, -H perfview: -A, -a, -B, -C, -c, -H, -h, -l, -M, -m, -U, -u, -v procview: -a, -c, -E, -F, -H, -O, -P, -S, -z profview: -A, -a, -B, -b, -c, -D, -d, -H, -h, -i, -m, -y, -z atexpert: -r (can be used alone or with -f) 4. If you specify none of the options listed in the previous step, the utility is usually invoked with the line-mode interface (if available). 5. If no options are speci?ed on the command line, it defaults to an X Window System interface. A few seconds after the performance utility is invoked with the X Window System interface, a cursor appears in the upper left corner of your screen. Using the mouse, move the cursor to the desired screen position and click the left mouse button. The performance utility window appears on your screen at the indicated position. The behavior discussed in this section is for the twm window manager. Other window managers behave differently when new windows appear on the screen. For example, some place all windows in the upper left-hand corner of the screen, making you responsible for moving the window to the desired location. In some cases, you can specify the desired location by using the -geometry option on the command line. In this case, you should specify only the location. 004–2182–003 11Guide to Parallel Vector Applications Do not override the window size. The following example causes the window to appear at location X=100, Y=100 on the screen. $ profview -geometry +100+100 raw.file 1.7.2 Common Options Many X Window System client programs use a common set of names for their command-line arguments. The X Toolkit automatically handles the following arguments, which you can enter directly on the command line. These values override values set in your .Xdefaults ?le (if any). In many cases, you can provide abbreviations for these options (for example, -g or -geo rather than -geometry) but you should use the names and abbreviations as documented in the following list. Table 6, page 12, provides the common command-line arguments that the X Toolkit handles. Table 6. X Toolkit Command-line Arguments Argument Description -bg color or -background color Either option speci?es the window background color, color. -bd color or -bordercolor color Either option speci?es the window border color, color. -bw number or -borderwidth number Either option speci?es the window border width, number, in pixels. -display display Speci?es the name of the X Window System ?leserver to use. -fg color or -foreground color Either option speci?es the text or graphics color, color. -fn font or -font font Either option speci?es the display text font, font. -geometry geometry Speci?es the main window’s initial size and location. Use this option to change only the window location. Changing the window size with -geometry can interfere with the display of graphics. -iconic Speci?es that your application should start out in an iconic state. The window manager that your system uses controls how this state is represented. 12 004–2182–003Introduction [1] Argument Description -name name Selects the name, name, under which resources for the application should be found. This option is useful when using shell aliases to distinguish between invocations of an application, and it eliminates the need to resort to creating links to alter the executable ?le name. -rv or -reverse Either option instructs the program to simulate reverse video, if possible, sometimes by swapping foreground and background colors. Usually, it is used only on monochromatic displays, and it may not be implemented correctly on all systems. +rv Instructs the program not to simulate reverse video. If the reverse video does not work correctly on your system, you can use this to override any defaults. -title string Speci?es the window title as string. Depending on the window manager used, you may display the title information as a window header. -xrm resourcestring Speci?es a name and value for the resource to override any defaults. You can also use it for setting resources that do not have explicit command-line arguments. 1.7.3 Resources The man page for each tool describes the resources that may be changed to alter the visual appearance of an X Window system client. This is speci?c to each tool. The man page for each tool describes the resources that may be changed, speci?c to each tool. You also can change the resources for common screen items used by a tool (for example, a text display window). These resources are speci?ed in your .Xdefaults ?le. This ?le can reside in your home directory on the UNICOS system, or it can reside on your X Window System server (workstation). To make the ?le global, execute the xrdb(1) command. The contents of the .Xdefaults ?le is a series of ASCII records, usually generated using a text editor. A typical record contains an indicator of the application being changed, linked with the resource that is recognized by the application. This name is followed by a colon (:). The speci?ed value for the resource follows the colon. For example, the following line changes the width of the Flowview main text window to 900 pixels. The application’s generic name is Flowview. The name of the resource checked by the tool is mainTextWindowWidth. These are 004–2182–003 13Guide to Parallel Vector Applications linked by an asterisk (*), which acts similarly to a wildcard used in shell commands. Flowview*mainTextWindowWidth: 900 For example, the following line changes the pie chart radius in the profview command to 200 pixels. The application’s generic name is Profview. The name of the resource checked by the tool is drawPieChartRedius. These are linked by an asterisk ( *), which acts similarly to a wildcard used in shell commands. Profview*drawPieChartRadius: 200 X Window System resource designators can be considerably more complex than the previous example indicates. See the list of standard X Window System references described for each tool in the appropriate chapter. 1.7.4 Troubleshooting If you have never worked with UNICOS systems with X Window System applications, you may have dif?culty when ?rst trying to bring up these performance tools. You ?rst should try to run simple applications, such as /usr/bin/X11/xcalc. Be certain that the DISPLAY environment variable is set to your workstation, and that you have run the xhost command on your workstation ( do not run xhost on UNICOS systems). It is also important that your UNICOS host knows the correct network pathway to your workstation. If you can use telnet to connect your workstation to the host, you could try using the numeric address of your workstation (followed by :0) as the DISPLAY environment variable, rather than its name. For example, if your workstation has address 127.162.83.10, you could try specifying a DISPLAY of 127.162.83.10:0. Try this method only if the UNICOS system does not recognize your workstation by name. When you can run a simple application like xcalc, other X Window System tools should execute without dif?culties. 1.7.5 Color The X Window System tools described in this manual generally do not specify particular colors for their displays. However, if you are running the tools on a workstation that supports color, you can alter some of the colors in certain windows by specifying them as resources. 14 004–2182–003Introduction [1] 1.7.6 Pointing Device The X Window System requires that your workstation have a pointing device, such as a mouse. Most of the X Window System tools described in this manual expect primary selections to be made by pressing the leftmost mouse button when the cursor is located on the desired menu item. 1.8 Component Naming Conventions The terms buttons, menus, and keyboard accelerators are used to describe various components of the performance utilities. To avoid confusion, the following sections describe naming conventions. 1.8.1 Buttons Buttons let you open menus and perform other functions. To activate a button, position the arrow cursor on the button and press the left mouse button. This is referred to as clicking on a button. The following types of buttons are used by the utilities: • The menu bar at the top of the main window displays buttons that open menus. When you click on one of the buttons, the menu opens. (For other ways of opening menus, see Section 1.8.3, page 16.) • The Cray Research logo also functions as a button. When activated, this button displays the product name, current version number, and applicable copyright information. • Additional buttons appear on various pop-up displays. The Close button that appears on every pop-up display is an example of this type of button. 1.8.2 Menus Menus are lists of available options. You can open a menu and make a selection in the following ways: • Click on a menu button and a pop-up menu opens. Then click on the name of the menu option you want to use. The menu remains open until you move the cursor out of the menu boundaries and press the left mouse button. If you click on a second menu button, the ?rst menu closes. • Use the press and drag method to make a selection by positioning the cursor on a menu button. Then, press and hold the left mouse button while 004–2182–003 15Guide to Parallel Vector Applications dragging the mouse down. A pull-down menu opens. When the option you want to use is highlighted, release the mouse button. The option is selected and the menu closes automatically. • After you have selected one menu, you can use the left and right arrow keys to move from menu to menu. Using the arrow key to move to a new menu button automatically closes the previous menu. Menus display the following peel-off icon in the upper right-hand corner: This icon indicates that you can convert menus into separate X Window System windows. After you have created a peel-off window, you can move, resize, iconify, or otherwise manipulate the window in the same manner as any X Window System window. To create a peel-off window from a menu, position the cursor on the peel-off icon and press any mouse button. To close a peel-off window, position the cursor on the Close button and press any mouse button. 1.8.3 Keyboard Accelerators Keyboard accelerators help speed experienced users through frequently used menu options. To use a keyboard accelerator, make certain your cursor is inside the tool’s main window. Accelerators work in the following ways: • You can use the CONTROL key with another character key. For example, the keyboard accelerator for closing a tool is CONTROL-Q, which performs the same function as selecting Quit from the File menu. Keyboard accelerators are not case-sensitive; therefore, CONTROL-Q and CONTROL-q perform the same function. Menu notation uses the caret (^) symbol to signify the CONTROL key. Where an accelerator is available for an option, it is included in the option description. • You can type the underlined letter on menu buttons. Typing the underlined letter while pressing the CONTROL key when no menu is open, opens the menu associated with the letter. Typing the underlined letter while pressing the CONTROL key when a menu is open, opens the menu option associated with the letter. 16 004–2182–003Introduction [1] 1.8.4 Cray Research Logo Clicking on the Cray Research logo opens a display that shows current tool version and copyright information. 1.8.5 Help Menu The Help menu options provide online information about how the tool works and displays version and copyright information. Figure 1, page 17 shows the Help menu. Figure 1. Help Menu 1.8.5.1 Help Selection Clicking on the Help option opens a separate window that contains help information. After the window is open, you can ?nd information in several ways: • Use the left mouse button to click on a topic in the Sections column of the window. You will move directly to that topic in the help text. • Search for a string within the help text by entering it in the Search: ?eld and then pressing the RETURN key. • Click on one of the bold, underlined words in the text display on the right side of the window by positioning the cursor over the word and pressing the left mouse button. This links you automatically to more detailed information about that subject. • Scroll through the text by pressing the SPACE key while the cursor is on the right side of the display. • Use the cursor and mouse to scroll through the help text by using the scroll bar on the right side of the window. 004–2182–003 17Guide to Parallel Vector Applications To close the window, select Quit from the File menu. 1.8.5.2 Version Selection Clicking on the Version selection opens a window that displays the current version of the tool and copyright information. 1.8.5.3 Release Information Selection Clicking on the Release Information menu selection gives you information on what is new in the current release. 18 004–2182–003ATExpert [2] Autotasking can generally be described as the automatic distribution of loop iterations to multiple processors. You can allow Autotasking to be fully automatic, that is, not requiring your intervention, or you can interact directly with the Autotasking system to enhance performance. For more information on Autotasking, see the CF90 Commands and Directives Reference Manual or Optimizing Application Code on UNICOS Systems. 2.1 Autotasking Expert System The Autotasking expert system, ATExpert, is a tool developed to measure and graphically display the Autotasking performance of a job run on an arbitrarily loaded UNICOS system. This tool can predict speedups on a dedicated system by using data collected from one run on a nondedicated system. It shows where a program is spending its time and whether those areas are executed serially or with multiple processors. It also serves as a guide for improving Autotasking performance. ATExpert has the following characteristics and capabilities: • Works with multitasked codes. • Instrumentation incurs CPU overhead. • Requires recompilation, reloading, and execution. • Instrumentation focuses on timing each parallel region identi?ed by the Autotasking system. Times for starting, stopping, and performing parallel work in a parallel region are measured. ATExpert also measures the time to perform work between parallel regions (called preceding serial time) and following the last parallel region (called ending serial time). Instrumentation is required by the Cray C compiler, the Cray C++ compiler, and the CF90 compiler. After compiling and loading the instrumented ?le, the autotasked job is executed. At program termination, the ATExpert library routine produces a ?le (called atx.raw) that contains collected performance data. You can view this data by using the ATExpert postprocessing tool. • Overhead caused by timing is kept low (10% to 20%) by carefully locating calls to the real-time clock function and avoiding large numbers of subroutine calls. This permits fairly accurate computation of various kinds of Autotasking overhead and allows ATExpert to make accurate projections of 004–2182–003 19Guide to Parallel Vector Applications speedups that use N processors; N can range from 4 through 164. However, overhead increases for programs that have small-granularity parallel regions. • Information can be accessed by using an X Window System graphical interface, a simple ASCII interface, or a report. The easiest way to view data collected during the run of an autotasked job is to use the X Window System graphical interface of the ATExpert postprocessing tool. (See Section 2.3, page 25, for information about using the X Window System interface. For information about the ASCII interface, see Section 2.4, page 44. For information about the report interface, see Section 2.4.2, page 45.) • ATExpert gives accurate performance measurements of a program, subroutines, parallel regions, and loops in a given application. From these measurements, a program can be examined to see where further performance gains can be made. Serial sections of the run can be scrutinized to ?nd more potential parallelism. Autotasking overhead can also be detailed to determine where bottlenecks are occurring. ATExpert will, at each level of analysis, graphically show you how the program has performed. • You also can use ATExpert with other tools, such as profview and flowview, to get an even more accurate feel for where you can gain additional performance. For complete information on how ATExpert run-time data is collected and analyzed, see Section 2.6, page 50. 2.2 Getting Started with ATExpert Note: You should run ATExpert as a background process, thereby freeing the command line for other uses by adding an ampersand (&) at the end of the command line, as shown. If you already have invoked ATExpert, or any other application, as an active (foreground) process, you can move it to the background by pressing CONTROL-Z and entering bg at the shell prompt (not within the ATExpert window). This command sequence stops the process and then runs it in the background. To start ATExpert using the X Window System interface, enter the following command: atexpert & You also can include a ?le name on the command line to invoke ATExpert and open a speci?ed ?le, as follows: 20 004–2182–003ATExpert [2] atexpert -f ?le & The ?le name you include on the command line speci?es the raw-format output ?le that ATExpert will process. If you do not specify a ?le name, the default is atx.raw. See Section 2.6.3, page 58, for more information about the atx.raw ?le. The preceding command line starts ATExpert in its X Window System mode; that is, if the DISPLAY environment variable is set, ATExpert automatically tries to execute in X Window System mode. If the DISPLAY environment variable is not set, the default is ASCII interactive mode, in which you are prompted for additional options and data is displayed in a more tabular format. (See Section 2.4, page 44, for information about using the ASCII interface.) You also can use ATExpert with a command-line option that generates reports for a ?le (-r). (See Section 2.2.1, page 23, for complete command-line options, and Section 2.4.2, page 45, for information about report generation.) Figure 2, page 22, shows the main ATExpert window generated by the X Window System interface with the atx.demo ?le opened. 004–2182–003 21Guide to Parallel Vector Applications Parallel region monitor Speedup display Information panel Figure 2. Main ATExpert Window 22 004–2182–003ATExpert [2] For descriptions of screen areas, see Section 2.3.1, page 25, through Section 2.3.1.3, page 27. Note: ATExpert displays shown on your screen may differ slightly from those in this document due to variances among window managers. 2.2.1 atexpert Command Line Command options are as follows for the atexpert command: atexpert [-b] [-f ?le] [-h] [-L] [-m] [-N cpus] [-o] [-p] [-r type] [-v] -b Shows the preceding serial time versus the parallel times for the given speedup chart. -f ?le Speci?es the raw-format ?le with which ATExpert will work. Default is atx.raw. -h Writes help to stdout. Specify this option only when using the ASCII or report interface. -L Speci?es that ATExpert ignore the DISPLAY environment variable (if set), and forces execution of an ASCII interactive session. You are prompted for additional options after the interactive session is started. Specify this option only when using the ASCII interface. -m Speci?es that ATExpert should plot maximum speedups rather than plotting dedicated speedups. Default is to plot dedicated speedups. -N cpus Speci?es the number of CPUs (maximum of 16) for which ATExpert will extrapolate data from the autotasked program. -o Directs ATExpert to give the overhead breakdown (in clock periods) for the number of CPUs set. -p Speci?es page mode to set a page break at each screenful of information. Specify this option only when using the ASCII interface. -r type Generates reports for the ?le based on type speci?ed, as follows. The -f and -r option 004–2182–003 23Guide to Parallel Vector Applications combination is required for noninteractive environment use. You may specify any combination of the p, s, r, and l types or the R type. Specify this option only when using the report interface. p Program summary s Subroutine summary r Parallel region summary l Loop summary R Report formatted and sent to stdout -v Speci?es verbose mode, which produces observations about the program, subroutines or function, parallel regions, and loops. Specify this option only when using the ASCII or report interface. 2.2.2 Instrumenting Your Code To use ATExpert with CF90 code, use the following command line: f90 -eX ?le.f90 If you have not included Autotasking directives in your code, you must add -03 to the command line to set the level of optimization. To use ATExpert with Cray C++ or Cray C code, use the following command lines, respectively: CC -h atexpert ?le.C cc -h atexpert ?le.c If you have not included Autotasking directives in your code, you must add -03 or -h task3 to the command line. ! Caution: Multistreaming programs will not execute when instrumented for ATExpert. 24 004–2182–003ATExpert [2] 2.3 Using the X Window System Interface When using ATExpert with the X Window System interface, you see several different windows. (Figure 2, page 22, shows the main window.) Most windows are active; that is, selecting an item within them causes some action to occur. The action may be to update data in existing displays or to pop up a new display with more detailed information. For example, selecting a subroutine name in the Autotasked Subroutines window pops up a window with parallel region data. Section 2.3.1, page 25, through Section 2.3.4, page 42, provide details about the following components of the ATExpert graphical display: • Main window • File menu • Display menu • Options menu Explanations and screens shown are based on the X Window System interface; if you are running an ASCII session, see Section 2.4, page 44. See Section 1.7, page 9, for more information on the X Window System interface. 2.3.1 Main Window The main ATExpert window contains several display areas: the parallel region monitor, the speedup display, and the information panel. 2.3.1.1 Parallel Region Monitor In Figure 2, page 22, the top portion of the main window is called the parallel region monitor. This display shows parallel times, as well as the serial time preceding the parallel regions, for parallel regions. (See Section 2.5.1, page 47, for information on determining serial time.) The vertical axis shows speedup, and the area of work displayed in each box corresponds to the amount of work done. Therefore, the more area a box has, the more impact that region has on the program. Also, boxes with higher speedups show performance gains; boxes (especially boxes with a large area) that show poor speedups potentially lower the entire program performance. The dotted line that runs the width of the screen shows average program speedup. It is signi?cant to note the line in relation to each region as it highlights the relative importance that a particular region has to the program as a whole. 004–2182–003 25Guide to Parallel Vector Applications Parallel regions are shown in original execution order. You can scroll this window horizontally, in case the number of parallel regions exceeds the number of regions that can be meaningfully displayed in a small area. The parallel region monitor is an active display. Therefore, if you click on a particular parallel region, which will then be highlighted to indicate it has been selected, the other displays and windows will be updated with information about the selected region. See Section 2.3.3.2, page 32, for more information about parallel regions. 2.3.1.2 Speedup Display In Figure 2, page 22, the speedup display (located below the parallel region monitor) shows performance speedup curves. This display shows speedup curves for programs, subroutines or functions, parallel regions, or loops. A diagonal line and two curves are displayed and indicate the following information: • The diagonal line represents linear speedup, which is the speedup a selected portion of the program would achieve if the program was 100% parallel and Autotasking incurred no overhead. • The two curves show ideal and expected speedups if the number of CPUs used for the portion of the program currently selected is increased. The top curve shows ideal speedup predicted by Amdahl’s Law. (See Section 2.6.2, for a discussion of Amdahl’s Law). It represents the speedups you would expect given the serial time but with no overhead associated with Autotasking. It shows the effect that serial time has on overall performance. The second curve shows expected dedicated speedup, which is the performance improvement in wall-clock time you would expect to achieve on a dedicated system with the use of Autotasking. Dedicated speedups are simply the values of tu / tn; tu is the unitasking time, and tn is the parallel time for n CPUs for the portion of the program displayed. When possible, measured values are used for tu and tn. If a measured value for tn does not exist or seems to be unusually large or small, an extrapolation algorithm that simulates Autotasking is used to generate a value of tn. When loop speedups are displayed, you can identify the use of measured times by the appearance of heavy dots on the speedup curve at those numbers of CPUs. Extrapolated times do not have dots displayed on the curve. The diagonal line and two curves partition the display into three areas, as shown in Figure 2, page 22. The ?rst area, up to the dedicated speedup curve, shows the effect of parallel execution or parallel work. You want this area as 26 004–2182–003ATExpert [2] large as possible. The second area between the two curves represents overhead. Finally, the top area above the Amdahl’s Law curve shows the effect of serial code on overall performance. The last two areas should be as small as possible for peak parallel performance. Within this display, you can click on any CPU point and a window will pop up showing a breakdown of the various sources of overhead for the selected number of CPUs used for the portion of the program currently displayed in the window. See Section 2.6.2, for a complete description of overhead costs and an example of the Overheads pop-up window that shows the breakdown of overhead. 2.3.1.3 Information Panel The information panel is located at the bottom of the main window and provides you with information about the status of the tool. If an error or warning message is generated, it will be displayed here and you will hear a beep. Note: As you work within ATExpert, you will see other displays in addition to the main window. These displays are created in pop-up windows as a result of selecting a menu option or clicking in an active window. 2.3.2 File Menu The File menu selections let you perform operations such as opening ?les, opening a demo ?le, and exiting ATExpert. Figure 3, page 27, shows the File menu. Figure 3. File Menu 004–2182–003 27Guide to Parallel Vector Applications File menu selections are described in Section 2.3.2.1, page 28, through Section 2.3.2.3, page 29. 2.3.2.1 Open. . . Selection The Open... selection lets you open (load) ?les to be used by ATExpert. You can also activate this selection by using the keyboard accelerator sequence CONTROL-O. The Open... selection creates an input panel on which you enter the name of the ?le to be opened. The input panel accepts shell wildcard conventions. Figure 4, page 28, shows the input panel for opening ?les. Figure 4. ATExpert Load File Input Panel The input line, near the top of the panel, shows the path to the current directory (that is, the directory from which you invoked ATExpert). To remove this path name, click on the Clear button or position the cursor on the input line and use the BACKSPACE or DELETE keys to remove the current path name. You also can click on the directory name from which you want to load a ?le 28 004–2182–003ATExpert [2] from the list displayed under the Directories column of the ATExpert Load File input panel. The ?le name entered on the input line speci?es a raw-format (atx.raw) output ?le for the session and loads it into ATExpert. This option starts ATExpert analysis, ?lls the main window, and pops up the Autotasked Subroutines window, which lists subroutines in the program and shows their parallel and serial times. In addition to entering a ?le name on the input line and pressing the RETURN key or the Load button, you can double click on a ?le name in the Files column of the input panel to load that ?le. The panel has the following buttons: Button Description Load Loads the ?les entered on the input line. Pressing RETURN is the same as clicking on the Open button. Separate multiple ?le names with a space. Filter Updates the directory or ?le list based on current ?le path and ?lter options. Clear Clears the input line. Close Closes the input panel. 2.3.2.2 Open demo file Selection The Open demo file option opens a sample atx.raw ?le for you to view. 2.3.2.3 Quit Selection The Quit selection displays a con?rmation window on which you can con?rm your decision to exit. You can also choose this selection by using the keyboard accelerator sequence CONTROL-Q. The con?rmation panel has the following buttons: Button Description Confirm Exits the tool and returns you to the operating system. Pressing RETURN is the same as clicking on the Confirm button. 004–2182–003 29Guide to Parallel Vector Applications Cancel Cancels the request to quit. Pressing ESCAPE is the same as clicking on the Cancel button. To suppress display of the con?rmation panel, add the following command line to your .Xdefaults ?le: atexpert*quitConfirm: never 2.3.3 Display Menu The Display menu selections let you show information about various aspects of your code. Figure 5, page 30, shows the Display menu. Menu selections listed above the line are used to retrieve windows you have closed; those listed below the line are used to open new windows. Figure 5. Display Menu Display menu selections are described in Section 2.3.3.1, page 31, through Section 2.3.3.7, page 41. 30 004–2182–003ATExpert [2] Note: When viewing information from the Subroutines..., Parallel Regions..., Loops..., and Case statements... selections, you should view output in terms of wall-clock time. That is, if you have two subroutines that show similar amounts of work (see the Parallel Region Monitor for amount of work) and one subroutine’s parallel regions performed better than the other, the subroutine that has the short parallel line is the one that performed the most effectively. 2.3.3.1 Subroutines. . . Selection The Subroutines... selection retrieves the window that lists autotasked subroutines or functions, and provides information about parallel and serial time spent within them. This window is created by default when an atx.raw ?le is loaded. You should use this selection only if you have previously closed the window. You can also activate this selection by using the keyboard accelerator sequence CONTROL-S. Figure 6, page 31, shows sample output from the Subroutines... option. Figure 6. Autotasked Subroutines Display Subroutine or function names are shown on the left of the display. To the right of each subroutine/function name the following information is displayed in bar graph format: 004–2182–003 31Guide to Parallel Vector Applications Information Description Preceding Serial Indicates the time from when the previous parallel region ended to the time that the next parallel region begins. This means that part of the preceding serial time may come from the end of the last subroutine or function. It also may very likely include time from calls made to other unitasking subroutines or functions before coming to this subroutine or function’s ?rst parallel region. Parallel Indicates the sum of parallel times of all parallel regions in this subroutine or function. Ending Serial Indicates the time from the end of the last parallel region until the completion of the program. The Autotasked Subroutines display is an active window; selecting a subroutine or function by clicking on it pops up another display that shows the selected subroutine’s parallel regions. (See Section 2.3.3.2, page 32, for information about parallel regions.) To close the display, click on the Close button. 2.3.3.2 Parallel Regions. . . Selection The Parallel Regions... selection retrieves a window that displays timing information about parallel regions of code. This window is displayed if you select a subroutine or function name from the Autotasked Subroutines display; or you can use this selection to reopen the display if you have previously closed it. Figure 7, page 33, shows sample output from the Parallel Regions... selection. 32 004–2182–003ATExpert [2] Figure 7. Parallel Regions Display The Parallel Regions display is an active window. Selecting a parallel region pops up the Loops window that displays loops in the selected parallel region. As in the Subroutines... selection, serial and parallel timing information is shown to the right of each subroutine or function name. To close the display, click on the Close button. 2.3.3.3 Loops. . . Selection The Loops... selection retrieves the window that displays timing information about loops in a selected region of code. Unlike the output from the Subroutines... and Parallel Regions... selections, however, output from the Loops... selection shows unitasked time versus autotasked time. This window is displayed if you select a parallel region from the Parallel Regions display; or you can use this selection to reopen the display if you have previously closed it. You can also activate this selection by using the keyboard accelerator sequence CONTROL-L. Figure 8, page 34, shows sample output from the Loops... selection. 004–2182–003 33Guide to Parallel Vector Applications Figure 8. Loops Display Loop names appear on the left side of the Loops display. Generally, loop names are the names from the original source ?le. With the C and C++ compiler, loops do not have names, but are identi?ed by line numbers. (See the Cray C/C++ Reference Manual, for more information.) To close the display, click on the Close button. 2.3.3.4 Case statements. . . Selection The Case statements... selection provides an additional level of detail about autotasked code. (CASE structure is indicated whenever a CASE label appears in the Loops display.) Selecting one of the CASE labels opens a window that details individual unitasked times for each CASE. This information lets you see how work is distributed across the cases. Figure 9, page 35, shows sample output from the Case statements... selection. 34 004–2182–003ATExpert [2] Figure 9. Case Statements Display To close the display, click on the Close button. 2.3.3.5 Observations. . . Selection The Observations... selection offers interpretations of ATExpert data and makes suggestions for improving performance. These observations also are available through the -v option of the atexpert command line. Figure 10, page 36, shows sample output from the Observations... selection. 004–2182–003 35Guide to Parallel Vector Applications Figure 10. Observations Display The Observations display has the following buttons: Button Description Graph Updates the speedup graph to re?ect data for the currently selected program, subprogram, parallel region, or loop. You will use this button only if you have con?gured the Observations... selection to not update graphs and observations at the same time. (See Section 2.3.3.5.1, page 37.) 36 004–2182–003ATExpert [2] Config Pops up a window that lets you select the format and content of displayed data. Selections you make by using Config override any defaults. (See Section 2.3.3.5.1, page 37.) Show Pops up a menu that lets you select observations at the program, subroutine or function, parallel region, or loop level. To view observations at the various levels, you must ?rst select a speci?c object before clicking on this button. Save Pops up a window on which you can specify the ?le name to which currently displayed observations text will be saved. Close Closes the display without doing anything. 2.3.3.5.1 Con?guring Observation Options When you click on the Config button of the Observations display, the following pop-up input panel is displayed, which lets you tailor observations made about your code (refer to Figure 11, page 37): Figure 11. Observation Con?guration Options Display The Observation Con?guration Options display contains the following options: 004–2182–003 37Guide to Parallel Vector Applications Option Description Fraction Parallel Speci?es the desired parallel fraction. Default is 0.80. Threshold Speci?es the value used as a measure of minimum granularity. Default is 2000. Fraction Overhead Speci?es the minimum acceptable Autotasking overhead fraction. Default is 0.10. Number of CPUs Speci?es the number of CPUs for which Autotasking rules are being applied. Default is the number of CPUs on the system where the program executed. Update graphs and observations together When selected (highlighted), the speedup graph is updated automatically with code constants currently selected in the Observations window. For additional information about the meaning of options available through the Observation Con?guration Options display, see Section 2.3.3.5.2, page 38. 2.3.3.5.2 How Observations Are Determined Observations are derived from the application of Autotasking rules, as follows: • Observations are made with respect to the following formula: Tp = ST + Tu/N + O This formula states that the parallel execution time of a program or any portion of it (Tp) is a function of the following variables: Variable Description ST Serial time preceding a parallel region or following the last parallel region at the end of the program; that is, the amount of work outside parallel regions. T p Wall-clock time at the given level when using N CPUs in parallel regions. T u Total unitasked time of work inside parallel regions and loops. 38 004–2182–003ATExpert [2] N Number of CPUs for which the Autotasking rules are being applied. O Total overhead from all sources associated with parallel execution. • Observations try to bring about maximum parallel performance by maximizing the time of work inside parallel regions (Tu) and minimizing serial time (ST) and overhead (O). ATExpert selects rules and makes observations based on the following formula: Tu/(Tu + ST) > FP O/(Tu/N) < FO FP is the desired parallel fraction; default is 0.8. FO is the minimum acceptable Autotasking overhead fraction; default is 0.1. FP and FO are con?guration parameters. Increasing FP has the effect of identifying areas in a program in which ST is dominant. Decreasing FO has the effect of focusing more attention on overhead. • The rule base is organized to focus primary attention on ST and secondary attention on O because you are most likely to improve program performance by making serial portions of the program parallel. Except for load imbalance, overhead costs are relatively ?xed, and you can do little to change them other than to increase the granularity of parallel work inside parallel regions. • Autotasking becomes effective only when some minimum amount of parallel work exists. This is called the granularity of parallel work. The rule base uses a threshold value as a measure of minimum granularity. • All times and overheads are weighted by the appropriate sample counts or trip counts. 2.3.3.6 Report. . . Selection The Report... selection displays, in text format, the data from which the graphical displays are derived. You can also activate this selection by using the keyboard accelerator sequence CONTROL-R. Figure 12, page 40, shows a sample report. 004–2182–003 39Guide to Parallel Vector Applications Figure 12. Report Display The report contains header information that describes the environment in which your program executed. Then, for each subroutine or function, the report lists all parallel regions and the loops within individual regions. Overhead types listed in the report are explained in Section 2.6.2. Overhead types and other notations are indicated on the report, as follows: 40 004–2182–003ATExpert [2] Type/notation Description min Indicates minimum times and counts of overhead and nonoverhead items. ave Indicates average times and counts of overhead and nonoverhead items. max Indicates maximum times and counts of overhead and nonoverhead items. Nu Indicates the number of times the unitasked version of code was executed. Tu Indicates the time required to execute the unitasked version of code. N n / S n pairs Indicates the numbers of samples taken and the speedup gained at a speci?ed number of CPUs, as shown in the following example: N8 S8 1 7.5 This pair indicates that one sample was taken at 8 CPUs and that the speedup at that number of CPUs was 7.5. ~ (tilde) The tilde (~) symbol following any number indicates that an extrapolated number was used instead of a measured one. The Report display has the following buttons: Button Description Save Saves the report to a user-speci?ed ?le Close Closes the display without doing anything 2.3.3.7 Source. . . Selection The Source... selection lets you view a source ?le. When selected, this selection displays the Load File input panel on which you can specify the ?le you want to view. When the ?le is loaded, the ?rst occurrence of any currently selected subroutine, parallel region, or loop is highlighted in the source ?le. You cannot edit the Source display; however, you can use CONTROL-S to search for a string within the source ?le. 004–2182–003 41Guide to Parallel Vector Applications 2.3.4 Options Menu The Options menu lets you specify the format and content of the ATExpert display. Figure 13, page 42, shows the Options menu. Figure 13. Options Menu Con?guration speci?cations selected using the Configure data... selection override ATExpert defaults. You can also activate this selection by using the keyboard accelerator sequence CONTROL-C. Figure 14, page 42, shows con?guration options. Figure 14. Con?guration Options Display 42 004–2182–003ATExpert [2] The Con?guration Options display contains the following options: Option Description Number of CPUs Speci?es the number of CPUs with which ATExpert analyzes your program. If you change this selection, you must reload the ?le. Default is the number of CPUs on the system in which the program executed. Plot Numbers Speci?es whether to display numbers on the graph. Default is on. Threshold Test Instructs ATExpert to treat all loops that failed threshold tests as if they had enough work to pass the test. If you change this selection, you must reload the ?le. Default is on. Filter Serial Time Speci?es that ATExpert ignore the effects of serial time and displays only the effects of parallel times. If you change this selection, you must reload the ?le. Default is off. The following ?lters are available: Program Filters by program serial time Subroutine Filters by subroutine or function serial time Parallel Regions Filters by parallel region serial time Plot Speedup Speci?es type of lines to plot in the speedup display. The following lines are available: Dedicated Speci?es plotting of dedicated line Maximum Speci?es plotting of maximum line Overhead Speci?es plotting of overhead line To close the display, click on the Close button. 004–2182–003 43Guide to Parallel Vector Applications 2.4 Using the ASCII Interface If you do not use the X Window System, ATExpert provides a simpli?ed ASCII interface. Although more limited than the X Window System version, this interface does present atx.raw ?le information. (See Section 2.2.1, page 23, for command-line options to specify the ASCII interface.) 2.4.1 Displaying Data To use the ASCII interface to ATExpert, you ?rst must instrument your program to generate timing data and produce an atx.raw ?le. (See Section 2.2.2, page 24, for information about compiler commands for instrumenting your code.) Then, you must choose the options you would like from the command line. In the following simple case, you could specify the following interactive command line: atexpert -L -b -f atx.raw This command line causes ATExpert to load the instrumentation ?le and to display a program performance curve, as shown in Section 2.4.1, page 44. (This is the same graph, that is shown in Figure 2, page 22, the X Window System interface.) Welcome to the Autotasking(tm) Expert System. Program summary: atx.raw Speed up 8 | * 3.48 7 | * 3.31 6 | * 3.17 5 | * 2.93 CPU 4 | * 2.71 3 | * 2.32 2 | * 1.75 1 | * 0.96 |----|-----|----|----|----|----|----|----| 1 2 3 4 5 6 7 8 Speed Ups Enter parallel section to analyze: After ATExpert is executing, entering a question mark (?) at the prompt prints available interactive options. Available selections are as follows: 44 004–2182–003ATExpert [2] Option Description -b Toggles the histogram breakdown. This gives more information about preceding serial and parallel times for subroutines/functions, parallel regions, and loops. all Plots all inner loops. help Prints the help ?le information. list Lists all parallel subroutines/functions, regions, and loops. l[oops] Plots all inner loops. You also can specify this option by using the letter l. p[rog[ram]] Plots the program summary. You also can specify this option by using the abbreviations p or prog. quit Quits the ATExpert session. r [eg[ions]] Plots the summary of all parallel regions. You also can specify this option by using the abbreviations r or reg. s[ub[routines]] Plots the summary of all parallel subroutines/functions. You also can specify this option by using the abbreviations s or sub. [subroutine name][region name][loop number] Specifying subroutine name, region name, and loop number plots the speci?ed subroutine or function, region, and/or loop. -o Toggles the overhead breakdown for the number of CPUs requested. -m Plots the maximum curve rather than the dedicated curve. ? Prints a help screen. 2.4.2 Generating Reports ATExpert provides several noninteractive reports. You can use the -r option to generate reports, which are then directed to stdout. This option is especially useful if you are con?ned to a noninteractive interface. 004–2182–003 45Guide to Parallel Vector Applications To get a simple report of program speedup, use the following ATExpert command: atexpert -L -rp -f atx.raw To get a report on program and subroutine speedups, overhead information for each, and have the report redirected to a ?le, use the following command: atexpert -L -rps -o -f atx.raw > atx.report If a full report of speedups for all loops, parallel regions, subroutines, and the program is needed, use the following command: atexpert -L -rpsrl -f atx.raw If you need a formatted version of the atx.raw ?le, use the following command: atexpert -L -rR -f atx.raw See Section 2.2.1, page 23, for a list of all report types available. 2.5 Interpreting ATExpert Graphs You can make many interpretations from information that ATExpert displays. Section 2.5.1, page 47, through Section 2.5.7, page 49, discuss some initial interpretations you can make to help you get started using ATExpert. These are the ?rst things most people look for when ATExpert displays program speedup curves. After you become familiar with these, you can begin to make your own, more detailed, interpretations and can interpret ATExpert’s wealth of performance data more comfortably. The two key questions ATExpert is trying to answer for you are, "What improvement did using the Autotasking option give me?" and "What can I do to improve performance?" The ?rst speedup chart you see (shown in Figure 2, page 22, as the Speedup Display) shows the expected speedup for your entire program. If speedup is signi?cantly less than you expected, Section 2.5.1, page 47, through Section 2.5.7, page 49, offer some suggestions you can follow to determine why your program is not running as well in parallel as you might have expected. 46 004–2182–003ATExpert [2] 2.5.1 Determining Serial Time The area between the top (Amdahl’s Law) curve and the diagonal line in the speedup window represents serial time in your program. If this area is large, you will want to look at large areas of preceding serial time before parallel regions and see whether they can be autotasked by the use of directives. 2.5.2 Finding Large Serial Time ATExpert provides the following three ways to ?nd large serial time portions in your program: 1. The parallel region monitor at the top of the display area shows serial time as small, dark boxes with a height of 1. (The 1.00 is not displayed.) Click the mouse over a long box and ATExpert immediately changes the display to the corresponding parallel region that follows this serial code. Subroutine or function and parallel region names in the corresponding windows are highlighted. Click on the parallel region name and the associated loop identi?ers are shown in a new loop window. 2. Select the Observations selection from the Display menu. Within the observation text is a section that lists the ?ve top serial portions of your program. 3. Look for long bars of preceding or ending serial time in the Subroutine window and click on a subroutine or function name to display parallel regions in the subroutine or function. Click on the parallel region names with long bold bars to ?nd the ?rst loop in the parallel regions. Note: The code preceding the ?rst loop of the highlighted parallel region is the serial code to investigate. Remember that serial code extends all the way back to the previous parallel region, which might be in another subroutine or function. Serial time includes all the code that is executed after the previous parallel region. It may include many subroutine or function calls and returns, plus I/O and other system or library calls. 2.5.3 Determining and Evaluating Key Performance Areas To improve your program, you must ?nd the key performance areas in your program and evaluate how effectively these key performance areas are being autotasked or run in parallel. The dominant performance areas of your program are displayed in the parallel region monitor section at the top of the display area. The large shaded boxes 004–2182–003 47Guide to Parallel Vector Applications represent areas with large amounts of parallel work. The height of a box in the parallel region monitor section indicates parallel speedup. Therefore, you want parallel boxes to be tall. Long, ?at parallel boxes point to key parallel regions that are not achieving a high speedup. In particular, you should investigate those boxes with a height that falls below the dashed line across the parallel region monitor section. Click the mouse on these boxes. Instantly, ATExpert relocates you to the subroutine or function and parallel region in question, with the corresponding names highlighted. Investigate these to see what is happening and what can be done to improve speedups. 2.5.4 Determining Parallel Overhead The area between the top curve (Amdahl’s Law) and the lower curve (dedicated speedup) represents overhead. If this area is large, overhead is hindering parallel performance. Click the mouse on the CPU area of the speedup display. A detailed breakdown of various sources of overhead will appear in a new window. Section 2.6.2, gives descriptions of each type of overhead. 2.5.5 Interpreting Speedup Curves That Level Off or Plateau A plateau on a speedup curve can mean several things. If the Amdahl’s Law curve also plateaus near the dedicated speedup curve, serial time in your program is dominating performance. You should concentrate your optimization efforts on the large serial portions of your program. However, if the Amdahl’s Law curve shows acceptable speedups, this indicates a possible large load imbalance problem or parallel regions with small-granularity work. To determine which of these problems exist, pop up the Overhead display by clicking on one of the curved lines in the speedup display. Compare the size of load imbalance overhead to other sources of parallel overhead. (See Section 2.6.2, for information about overhead cost and Figure 15, page 53, to see a sample Overheads display.) Sometimes you can resolve a load imbalance problem either by reorganizing a nested loop structure or by selecting an intermediate level loop to be autotasked. To improve a CASE structure, put larger cases ?rst or break them into smaller cases. If the problem seems to be small-granularity parallel work, investigate the corresponding key performance areas for ways to bring parallel execution to a higher level; either by bringing it to an outer level loop or to a higher level 48 004–2182–003ATExpert [2] subroutine or function. Inline expansion of a routine also may generate higher-level parallel execution. If it is dif?cult to remove a speedup curve plateau, you may want to consider setting the NCPUS environment variable to control the number of CPUs used in your program. This will make more effective use of the CPU resources on your system. 2.5.6 Determining Dominant Autotasked Subroutines To determine the dominant subroutine or functions that were autotasked, look at the list of subroutine or function names in the Autotasked Subroutines display. These are all of the routines that contain parallel regions. If a dominant routine that you know about is not listed, it either was not executed or it was not autotasked. The length of parallel bars in this window shows the amount of parallel work within each listed routine; therefore, longer light bars represent the dominant routines. If these bars do not correspond to your expectations, check the corresponding serial bars. If they are long, this indicates that some dominant loops in these routines are not being autotasked or could be autotasked better. Click on the routine names to bring up its parallel regions, and then click on parallel region names to locate the code by its loop identi?ers. 2.5.7 Interpreting Bars and Messages A vertical bar next to a subroutine or function name or a variable work loop message in the information panel section of the main window (as shown in Figure 2, page 22) both indicate variable work loops in your program. Brie?y, a variable work loop is a parallel loop in which the time per concurrent iteration varies considerably from one execution of a parallel region to the next. Section 2.6.4, page 62, describes these loops in more detail. ATExpert usually handles variable work loops fairly well. However, situations do arise when variations in time are so large that ATExpert’s algorithms break down and start producing bad speedups. When speedup curves for these loops are displayed, ATExpert prints a warning message that indicates you should be wary of the speedup values. 004–2182–003 49Guide to Parallel Vector Applications 2.6 Understanding ATExpert Concepts Section 2.6.1, page 50, through Section 2.6.6, page 64, explain the following ATExpert concepts: • Instrumentation • Overhead costs • atx.raw output ?le • Variable work loops • Dedicated and maximum speedup curves • Software anomalies 2.6.1 Instrumentation Instrumentation records the number of unitasked, scalar iterations for each loop; tallies the number of concurrent iterations executed in each parallel execution of a loop; and stores other data associated with a parallel region or loop. (For information on how to instrument your code, see Section 2.2.2, page 24.) Instrumentation, including calls to the real-time clock function, is added to your code to record the following times: • Begin parallel region time (BP). This is the time just before the parallel region is entered. Only the master CPU records this time. • Begin control structure time (BCS). This is the time just before the control logic for each parallel loop in a parallel region is entered. Each master and slave CPU records this time. • Top of concurrent iteration loop (Top). This is the time when a CPU is about to begin its ?rst concurrent iteration. Each master and slave CPU records this time. A top time is also recorded by the master before it executes each loop in the serial version of a parallel region’s loops. • Bottom of concurrent iteration loop (Bottom). This is the time when a CPU ?nishes its last concurrent iteration. Each master and slave CPU records this time. • End parallel region time (EP). This is the time just after the parallel region is ended. Only the master CPU records this time. 50 004–2182–003ATExpert [2] When an instrumented program terminates, data that has been gathered and summarized is written to a ?le named atx.raw. See Section 2.6.3, page 58, for a description of the contents of this ?le. You can assign another name to the ?le, rather than atx.raw. Before executing the instrumented program, set the ATXFILE environment variable, as follows: Standard shell (sh) or Korn shell (ksh): $ ATXFILE=?lename;export ATXFILE C shell (csh): % setenv ATXFILE ?lename 2.6.2 Overhead Cost ATExpert de?nes overhead as the difference in speedup between that predicted by Amdahl’s Law for multiprocessing and that measured or projected from an actual Autotasking run. Amdahl’s Law for multiprocessing is shown in the following equation: S(N) = 1 fs + N fp Variable Description S(N) Maximum expected speedup from multitasking N Number of processors available for parallel execution f p Fraction of a program that can execute in parallel f s Fraction of a program that is serial = 1 - f p The speedup from multitasking, S(N), is in terms of wall-clock time, not CPU time. When ATExpert uses this formula, it computes the fraction of time spent in parallel code, f p , from exploited parallelism rather than parallelism that might actually exist in a program. The distinction is important, because a program might contain parallelism at a very high level, but Autotasking cannot exploit it 004–2182–003 51Guide to Parallel Vector Applications because it cannot perform dependence analysis across procedure boundaries. A user, however, can insert directives to exploit parallelism at a high level. ATExpert would re?ect this by computing larger values of S(N) for the program as a whole and for the subroutines and functions that contain the inserted directives. ATExpert treats unexploited parallelism in a program as serial time. Although ATExpert can guide you to areas of your program with signi?cant amounts of serial time, it cannot detect parallelism inside library routines (such as those in libsci) or in routines that were not compiled using ATExpert instrumentation. Thus, calls to library routines outside parallel regions are treated as serial time, even though parallel execution may occur inside the routines. For a given portion of your program, f p is computed as the time to execute the parallel pieces of the program in unitasked mode, divided by the time to execute the entire portion in unitasked mode. Thus, for example, f p for a parallel loop is 1.0 because the entire portion of code is parallel. A subroutine or function, program, or parallel region has serially executed code outside of the loops. f p for these is less than 1.0. For example, a subroutine contains a parallel region that requires 10,000 clock periods in unitasked mode. This subroutine also contains 5,000 clock periods of serial time outside the parallel region. For this example, f p = 0.67 (10000/15000) and S (8) = 2.42. This is the ideal speedup, as predicted by Amdahl’s Law for this subroutine. Amdahl’s Law assumes no overhead is associated with multiprocessing. Serial time is not considered overhead by the preceding de?nition. Instead, serial time merely affects the value of f p used in Amdahl’s Law. For ATExpert, f p re?ects the amount of parallelism detected and exploited by the Autotasking software. It may not be an accurate measure of your program’s potential parallelism. Overhead can occur from many sources within parallel regions. Each source contributes to a lower speedup than that predicted by Amdahl’s Law. The Overheads display in Figure 15, page 53, shows overhead totals for a sample program. Note: To generate an Overheads display, click the mouse on the speedup display. For more information about determining parallel overhead, see Section 2.5.4, page 48. 52 004–2182–003ATExpert [2] Figure 15. Overhead Breakdown The following sources of overhead are measured and displayed by ATExpert (screen abbreviations are shown in parentheses): • Begin Parallel (bp) overhead - This overhead is the time that the master task requires to initialize parallel execution in a parallel region. During this time, 004–2182–003 53Guide to Parallel Vector Applications the master task is performing threshold tests, if necessary, testing whether Autotasking or microtasking is occurring at a higher level, setting up shared registers, and unparking any slave CPUs that may be waiting in the Autotasking library. In terms of measured times, bp is the time from the beginning of the parallel region to the ?rst control structure. • Slave Arrival (sa) overhead - Slave arrival time is the time it takes for the ?rst slave CPU to reach the ?rst control structure in the slave routine. It is measured from the time recorded by the master at the end of its begin parallel code. • Wait Slave (ws) overhead - This overhead is the time that the master task must wait for slave CPUs to return to the Autotasking library before it can resume serial execution outside the parallel region. It varies depending on whether the master or a slave process takes the last concurrent iteration. • Begin Control (bc) structure overhead - This overhead is the time it takes to compute the number of concurrent iterations in the next parallel loop, to establish synchronization parameters for parallel execution, and to detect whether a master or slave process has any work to do in the parallel loop. In this sense, work means concurrent iterations. In terms of measured times, bc is the time from the beginning of the control structure to the top of the concurrent iteration loop. • Convoy Time (cv) overhead- This overhead is the time required to get all remaining slave CPUs (after the ?rst) to the ?rst control structure in the slave routine. Adding sa and cv overhead represents the latency in getting all slave CPUs into the slave routine and ready to take a concurrent iteration. If this overhead is too high, it is possible for all concurrent iterations to have been taken by the time the last slave CPUs arrive. • Iteration Overhead (io) - Iteration overhead is the additional cost incurred during the execution of concurrent iterations. This includes the time to obtain a concurrent iteration and to test whether the last iteration has already been started. Usually, the time to execute this scalar code is covered in the hardware by the completion of the ?nal vector instructions in an inner loop. However, instances (such as reduction loops) occur when this is not true. Iteration overhead also may include hardware delays from the use of multiple CPUs, such as additional memory bank con?icts or shared-register contention. These are dif?cult to measure directly. The change in iteration 54 004–2182–003ATExpert [2] overhead as more CPUs execute in a parallel loop is used as an estimate for these. • Loop Sync (ls) overhead - Loop synchronization overhead is the time required for all participating CPUs to ?nish their last concurrent iteration after the ?rst CPU ?nishes its ?nal concurrent iteration. This synchronization ensures that all of the results from the previous parallel loop are in memory before starting the next parallel loop. ATExpert tries to decompose loop synchronization overhead into components associated with load imbalance, slave arrival latency, and convoy time because these more closely re?ect the cause of loop synchronization delays. Thus, ATExpert seldom displays a nonzero value for ls overhead. The ls overhead is a measure of the quality of the ATExpert algorithm to decompose Autotasking overhead into its separate components. Because this algorithm is based on measured times, ls sometimes re?ects noisy times rather than an incorrect algorithm. In other situations, ls can be nonzero because of the reasons listed for "other" overhead (for example, a negative value of ls can result when the compiler generates higher performing parallel code than the corresponding serial code in the three-code model). • Load Imbalance (li) overhead - Load imbalance occurs when the work (that is, concurrent iterations) cannot be distributed evenly across the participating CPUs. If no other sources of overhead exist, ATExpert computes li overhead as the difference between a perfect loop time partitioning for N CPUs and the discrete distribution of the given number of concurrent iterations (Iters) across N CPUs. The formula that ATExpert uses is as follows: li = tu Iters * ciel(Iters/N) tu N The tu variable is the unitasking time for the loop. (The formula is a little more complex for vector concurrent loops or CASE structures because the time per concurrent iteration or case varies.) The term ceil (Iters / N) represents the maximum number of iterations that at least one CPU must take. Basically, this formula says that whenever the number of iterations is not exactly divisible by the number of CPUs, a load imbalance exists. 004–2182–003 55Guide to Parallel Vector Applications Suppose a loop that contains 16 iterations executes in time tu. The perfect distribution of parallel time for N CPUs is for each CPU to execute for time tu / N. For values of N equal to 1, 2, 4, 8, and 16, the iterations are exactly divisible by N, and each CPU will execute in time tu / N (if no other overhead exists). However, for N equal to any other value, 16 iterations cannot be partitioned evenly. If N equals 5, for example, one CPU must take four iterations and the rest will take three iterations. This creates a load imbalance of the following: li = tu 16 * 4 tu 5 = tu 20 Thus, load imbalance is contributing a 5% overhead when executing this loop with ?ve CPUs. Load imbalance overhead is sometimes hidden by the effects of sa and cv sources of overhead. ATExpert displays the remaining in?uence of load imbalance, after sa and cv have been considered. • Inter Loop (il) overhead - Inter Loop time is the time required to reach the top of the next control structure after all CPUs have ?nished their last iteration of the previous control structure. It applies only when two or more parallel loops exist within a parallel region or one parallel loop is executed multiple times in a parallel region. • Wait Send (wt) overhead - A quantity that re?ects the wall-clock time impact that wait/send has on the region of code being viewed. For example, if a lot of work is surrounding the wait/send, processors may be scattered throughout and the wait/send may have little impact. • Other overhead - Other overhead is a measure of the inaccuracy of the ATExpert instrumentation and algorithms. It results when the ATExpert extrapolation algorithm produces an estimated parallel time that is smaller than the measured time for a given number of CPUs. This algorithm uses measured values of times and overhead when possible. The following are known to be causes of what ATExpert shows as "other overhead." – "Noise" introduced in the timings because of running on a loaded system, with unrelated jobs running in other CPUs. Memory bank con?icts from these other jobs affect both the unitasking and parallel execution times of loops. 56 004–2182–003ATExpert [2] – The ATExpert extrapolation algorithm contains some simplifying assumptions that are sometimes not valid. As a result, the model sometimes produces parallel times that are more ideal than real. – The ATExpert library routine summarizes the instrumentation data gathered during the execution of a parallel region. Only minimum, maximum, and average times are eventually written into the atx.raw ?le; thereafter, ATExpert generally uses only the minimum times. This occasionally introduces variations between unitasking times and parallel times with different numbers of CPUs. ATExpert assumes that all measured times in the atx.raw ?le are valid, especially if many samples are taken. When plugged into the extrapolation algorithm, small variations may show up as "other overhead.” – A related source of inaccuracy occurs when loop iterations are not always exactly identical in terms of work performed. IF-THEN-ELSE logic, gather/scatter memory reference patterns, and varying inner-loop iteration counts can contribute to small variations in measured times from one sample to the next. These may not be suf?cient to warrant being called variable work loops, but they may cause ATExpert to produce an overhead value that is unrealistically too small or too large. When plugged in to the extrapolation algorithm, estimated parallel times for different numbers of CPUs may be inaccurate as a result. – The compilers do not produce identical code in the three parts of the three-code model. Thus, unitasked code may run faster or slower than the master’s parallel code or the slave code. The master’s parallel code also may run faster or slower than the slave’s parallel code. In the worst case, one may be executing in vector mode and the other in scalar (report this situation by ?ling a Software Problem Report (SPR) against the compiler product you are using). ATExpert assumes that the compiler generates identical code with identical performance characteristics for each part of the three-code model. Thus, minimum times are assumed to be valid for any number of CPUs and any part of the three-code model. Plugging these into the extrapolation algorithm may produce times that are inconsistent with the code that the compiler generated. – In loops that fail a threshold test, ATExpert forces the times to be the unitasked times, plus some time to perform the threshold test. However, when overheads are computed for the loop, the difference between the forced time and Amdahl’s Law appears as "other overhead." Amdahl’s Law, in this case, is simply unitasked time divided by the number of CPUs. As the number of CPUs increases, the difference, appearing as other overhead, increases. 004–2182–003 57Guide to Parallel Vector Applications 2.6.3 atx.raw Output File The atx.raw output ?le allows a postprocessing tool to access data and interpret information easily. You should access the atx.raw ?le by using the ATExpert postprocessing tool. This section brie?y describes each of the line types in the header. All times are in clock periods. Header information: Line type Description DP Total number of disconnects in parallel code and the average number of clock periods per disconnect. DT Execution date and time of this instrumentation. Also the elapsed time and CPU time of this execution. DS Total number of disconnects in serial code and the average number of clock periods per disconnect. MX Maximum and average CPUs used in parallel regions and the maximum memory usage for this run. NP Number of parallel regions. The number of times parallel regions were run parallel and the number of times the parallel regions were run unitasked. SY System ID, operating system release, and system type. VS Version of ATExpert. Parallel region information: Line type Description BP Minimum, maximum, and average time to begin parallel execution. It is the time from the entry of a parallel region to the beginning of the ?rst control structure. PR Parallel region information. This line contains the master routine name, the parallel region (slave routine) name, the number of times the parallel region was executed, the number of parallel loops in the parallel region, the number of CPUs that 58 004–2182–003ATExpert [2] entered during parallel execution, and the number of disconnects that occurred for which the ATExpert library could not determine where to subtract the associated wall-clock time (the times are rejected unless they are the only samples taken). PS Minimum, maximum, and average preceding serial time, which is the interval between the end of the previous parallel region and the start of the current parallel region and the name of the preceding parallel region. As such, it can include subroutine calls, returns, nonparallel loops, and any other type of serial code. For the ?rst parallel region in a subroutine, it is not the time from the top of the routine. SA Minimum, maximum, and average time for the ?rst slave CPU to arrive in the slave routine. Slave arrival overhead is described in Section 2.6.2. Parallel work (loop/case) information: Line type Description BC Minimum, maximum, and average begin control structure times. The begin control structure time is the interval from the entry to a control structure to the top of the concurrent iteration loop. For CASE control structures, the BC lines contain the times to reach the corresponding CASE. DO DO loop or CASE identi?er, and the number of unitasked, scalar iterations, and the number of concurrent iterations. The identi?er usually is the Fortran DO loop statement number (label). However, if one does not exist, particularly for a CASE, the subroutine line number is used. The DO line differs somewhat from other lines, because different data can be contained for different types of parallel work. For example, the following tags can appear on the DO line: 004–2182–003 59Guide to Parallel Vector Applications Tag Description VECTOR Indicates a parallel vector loop CASE Indicates a CASE control structure SAVELAST Implies that the master task must take the last concurrent iteration VECRED Indicates a parallel vector reduction loop GUIDED # Indicates guided scheduling with a minimum number of iterations If these tags do not appear on the DO line, the loop iterations are assumed to be distributed one at a time or in chunks to the participating CPUs. Three DO lines with the same identi?er inside the same parallel region designate a variable work loop. See Section 2.6.4, page 62, for more information. IL Minimum, maximum, and average interloop times. IO Minimum, maximum, and average iteration overhead times. The maximum value is not the maximum iteration overhead measured; rather it is the minimum value measured when the maximum number of CPUs was working. This is used to give ATExpert an estimate of memory or shared-register contention during the concurrent execution of a parallel loop. See Section 2.6.2, for a description of iteration overhead. LS Minimum, maximum, and average loop synchronization times. See 60 004–2182–003ATExpert [2] Section 2.6.2 for a discussion of loop synchronization overhead. PD Minimum, maximum, and average processor delta times, which is the interval of time between arriving CPUs at the top of a control structure. Tn Number of parallel samples run with n CPUs and the minimum corresponding parallel time. For variable work loops, the three T n lines represent the average, minimum, and maximum times. If a T n line does not exist for a given value of n, no samples were taken with that number of CPUs working. TU Number of unitasked samples run and the minimum unitasked time. For variable work loops, the three TU lines represent the average, minimum, and maximum times, respectively. WS Minimum, maximum, and average wait slave times. See Section 2.6.2 for a discussion of wait slave overhead. BP, SA, BC, PD, IO, LS, IL, and WS lines are written to the atx.raw ?le only if reasonable, nonzero times were measured for them. Other information: Line type Description EC A warning error condition code and associated messages string. ES Time from the end of the last parallel region until the completion of the program and the name of the last parallel region executed. 004–2182–003 61Guide to Parallel Vector Applications INFO Additional information to be used to interpret the error condition. 2.6.4 Variable Work Loops Variable work loops are de?ned as parallel loops in which ATExpert detects signi?cant variations in the per iteration wall-clock times among different executions of the loops. These loops probably will contain IF-THEN-ELSE statements with varying degrees of truth, inner loops with varying iteration counts, calls to concurrent routines with varying execution paths, and/or vector reduction loops with varying vector lengths. Variable work loops are a dif?cult problem for ATExpert, because it cannot produce one measured time that accurately depicts loop performance. When a variable work loop is identi?ed, ATExpert prints three timing records for the loop, rather than the usual one. The three records correspond to the average, minimum, and maximum times measured for the loop. ATExpert does not generate variable work loop information for parallel case structures. Only minimum times are used for parallel cases. This may be changed in a later release of ATExpert. Variable work loops are identi?ed by a small vertical bar preceding the loop identi?er, parallel region name, or subroutine or function name. A warning message also is printed after ATExpert has read an atx.raw ?le that contains a variable work loop. When the speedup graph is displayed for a variable work loop, error bars also appear along the speedup curve. This represents ATExpert’s best guess at the possible range of speedups that you can expect for the known variations of measured times. The algorithm used to detect variable work loops is not perfect. Noise in the measured times can cause ATExpert to identify a variable work loop when one does not exist. The reverse also is true, but it is less common. Also, because of the possibly noisy times, ATExpert skips the variable work loop algorithm for very small granularity loops (unitasking times of a few thousand clock periods). As a result of having a very limited set of measured times for a variable work loop, the ATExpert extrapolation algorithm may produce very inaccurate times. If the variation of times is such that the extrapolation algorithm may break down, a warning message is printed to caution users about the reliability of the displayed performance data. 62 004–2182–003ATExpert [2] Users are asked to use speedups from variable work loops with caution. If possible, compare the information ATExpert is showing you with other knowledge you may have about your program. Run your instrumented program more than once, and check whether ATExpert produces the same speedups; if not, be very skeptical of the quality of the ATExpert measured values or extrapolation algorithms. Use its results only as a relative guide to your program’s Autotasking performance, not as an absolute measure. 2.6.5 Dedicated and Maximum Speedup Curves ATExpert tries to predict dedicated speedups from data collected on a nondedicated system; this is the default. However, ATExpert provides an option to show maximum speedups, which are de?ned as those computed by using minimum times in all speedup calculations. The use of minimum times, however, may not accurately re?ect actual execution paths taken by the program. Dedicated speedups use a formula in which minimum times are used in all calculations, except when adding in preceding serial time. For this calculation, a fraction of the difference between the average and minimum preceding serial time is added in. The rationale for this dedicated speedup calculation is that preceding serial code often contains alternative paths. Examples of alternative execution paths are as follows: • I/O performed every so many time steps • Initialization occurring on the ?rst invocation of a routine and never thereafter • Data-driven algorithms When alternative paths are taken, serial code performance can vary greatly. Using minimum times to predict expected speedups for a subroutine or function or program can yield wildly inaccurate results. For example, ATExpert predicts a program speedup of 4.3 when displaying the maximum curves for an important application program. The dedicated curves show a speedup of 2.3, which is much closer to the measured speedup of 2.1. If alternative paths are not taken in preceding serial code, average and minimum times usually are close in value and the effect of adding the difference is minimal. Very little difference should exist between maximum and dedicated curves for programs with no alternative paths (or none of signi?cance). The ATExpert hardwired dedicated speedup algorithm seems to work well for most application programs. Feedback from users indicates that predicted 004–2182–003 63Guide to Parallel Vector Applications speedups consistently are close to those actually measured on a dedicated system. Instances will occur, however, when the ATExpert hardwired algorithm does not produce an accurate speedup prediction. For example, if huge amounts of I/O are performed infrequently, ATExpert’s algorithm may break down. In this situation, I/O may dominate the program’s real performance. However, ATExpert’s algorithm may be more biased toward minimum preceding serial times. Therefore, it will predict larger speedups than you actually may obtain on a dedicated system. A quick way to check whether ATExpert may be predicting inaccurate dedicated speedups is to compare them with the maximum speedups. If the difference between speedups is great, the ATExpert dedicated speedup algorithm may not be correctly accounting for the actual alternative paths taken through the program’s serial code. You can use the parallel region monitor section to identify areas of the program in which the dedicated speedup algorithm is having trouble. These areas will be those in which preceding serial time (shown as boxes with a height of 1) changes dramatically when switching between maximum and dedicated curves. You can inspect these areas of the program to see what effect, if any, alternate paths may be having on the program’s performance. A future release of ATExpert may provide more user control over the dedicated speedup algorithm as a con?guration option. 2.6.6 Software Anomalies Known anomalies exist with ATExpert. This section details the known problems and discusses potential areas of misunderstandings. • The atexpert command - ATExpert measures performance of Autotasking, not microtasking or macrotasking. • Dedicated speedups - ATExpert’s calculation of dedicated speedups seems to be accurate for most programs; however, situations do exist in which the predicted dedicated speedups may be inaccurate. (See Section 2.6.5, page 63, for a more complete description of when this problem may occur.) Also, ATExpert’s predictions are based on the environment from which the measurements were made. If the MP_DEDICATED environment variable is set, it makes predictions based on this value, because the value is part of the environment. ATExpert cannot predict speedups for when this variable is set from measurements taken when it is not set. 64 004–2182–003ATExpert [2] • Extrapolation - The extrapolation system is a smart system, but it does need as much data as possible to make correct extrapolations. The more timing data collected for a program, the more accurate the extrapolation will be for areas that did not get all of the CPUs from the system that it could effectively use. • I/O - If you have heavy I/O in your program, the preceding serial times in those areas is minimum time. • Multiple disconnects - The ATExpert report ?le contains the number and length of disconnects. The higher the number of disconnects, the less accurate the numbers may be. ATExpert does handle disconnects, and it subtracts the disconnect time from the timed information, but large numbers and lengths of disconnects still can cause some problems. • Preceding serial time - The preceding serial time is de?ned as the time from the end of the last parallel region to the beginning of the next parallel region. In the case of the ?rst parallel region, this is the time from program startup to the beginning of the ?rst parallel region. In all cases, the serial time can cross subroutine calls and is not to be mistaken for time from the beginning of the subroutine. • Sharply de?ned peaks and valleys in the speedup curve - Sometimes ATExpert’s speedup curves may have signi?cant peaks and valleys. Peaks and plateaus usually are reasonable, however, deep valleys or sharp peaks are not. These occur when measured overhead values either are not available and must be estimated, or they are inaccurate (possibly because of too few samples). In either case, extrapolated parallel times to ?ll in gaps in measured times may be too large (or small) compared with adjacent measured times. The result is a speedup curve with sharply de?ned peaks and valleys. A future release of ATExpert will address this problem. In the interim, rerunning an instrumented program often eliminates the peaks or valleys, because better overhead values are measured or the gaps are ?lled in. • Single-CPU instrumentation - ATExpert can produce valid speedup predictions even when the instrumented program is run on only one CPU. The quality of this prediction, however, is proportional to the granularity of work in parallel regions. The predictions that ATExpert makes should improve as granularity increases. • Threshold tests - Threshold tests that the Autotasking software generates can affect ATExpert’s interpretation of timing measurements. For example, if iteration counts vary from one invocation of a parallel region to the next, a threshold test can force unitasking samples to be taken for low iteration 004–2182–003 65Guide to Parallel Vector Applications counts and Autotasking samples to be taken for large iteration counts. This can lead to larger than expected speedups or the detection of variable loops. ATExpert tries to detect threshold tests and to interpret the results accordingly, but it is not perfect. If a threshold test may be giving misleading results, ATExpert will print a warning message. • Uninstrumented multitasked libraries - Libraries that are microtasked or autotasked without ATExpert instrumentation will be measured as preceding serial time. This includes libsci calls. • Variable work - When variable work occurs at the loop level, it also can affect the projected program speedup. See Section 2.6.4, page 62, for a detailed explanation of variable work loops. • Variable iteration loops - ATExpert can display inaccurate speedups for a program dominated by loops with varying numbers of concurrent iterations. ATExpert generally uses the minimum times from running the maximum number of concurrent iterations. This tends to bias overall speedups, however, toward larger values. Actual program speedups may be less than those predicted by ATExpert. If the average number of concurrent iterations is more than 20% different than the maximum, ATExpert will print a warning message. • Wait Slave overhead - The wait slave overhead can be misleading if all slaves return to the Autotasking library before the master task ?nishes its last iteration, which typically is not the case. See Section 2.6.2 for a discussion of wait slave overhead. 66 004–2182–003flowview [3] The flowview(1) command displays information from Flowtrace raw-format output. This chapter will discuss the following topics: • Using the flowview command • Instrumenting your code • Using the X Window System interface • Using the ASCII interface • Using vi windows • Additional Flowtrace features • flowview sample output 3.1 The Flowtrace Feature The Flowtrace feature gathers information about all traced procedure calls during execution of a program. When the program terminates normally, these statistics are written to a ?le. You can generate reports from this data by using the flowview postprocessing tool. Flowtrace gathers, among others, the following statistics: • Number of times a traced routine was called • Total amount of CPU time spent in a traced routine • Names of traced callers for each traced routine Time spent in a traced routine called by another traced routine is accounted properly to the calling routine. Time spent in an untraced routine (such as a library routine) is included in the statistics for its traced caller. You should consider the following when using Flowtrace: • Flowtrace works with all types of Autotasking and with macrotasking involving calls to the TSKSTART(3) routine. Flowtrace does not work with multitasking that uses the t_fork (see tfork(3)) routine. 004–2182–003 67Guide to Parallel Vector Applications • Flowtrace incurs CPU overhead. • Flowtrace requires recompilation and reloading. • flowmark(3) and SETLIMQ(3) subroutines require source code changes. 3.2 Getting Started with the flowview Command The flowview command can be run interactively, using both the X Window System interface and line-mode, or it can generate a report based on command-line options. flowview presents an interactive menu when no command-line report option is included; otherwise, you can specify an output option for redirection to a ?le. The following sections discuss how to use command-line options to produce flowview reports. For information on producing reports by using the X Window System interface, see Section 3.3, page 74. 3.2.1 flowview Command-line Options The flowview command line is as follows. Speci?cation of any of the following options causes flowview to run in noninteractive mode. Report output is written to stdout, and flowview terminates. flowview [-L] [-u] [-c] [-h] [-A] [-m] [-a] [-v] [-w] [-l] [-j] [-k] [-T] [-H] [-export[_csv]] [-export_tabbed] [-export_sylk] [-o] [-p] [-N] [-R root] [-y treelower] [-z treeupper] [-S nstars] [-O toproutines] [-D ?ag] [-X] [-e] [-V] [raw?le] -L Prevents execution of the X Window System. You should always specify -L when you want to generate a report by using a command-line option. You also can use this option to specify line-mode interactive when no other report-generation options are speci?ed. See Section 3.10, page 93, for more information. -u Shows all traced routines, sorted in descending order by the amount of CPU time used. The ?rst few lines of this report should show you where the most CPU time is being spent. 68 004–2182–003flowview [3] -c Generates a report that shows pertinent information about the environment con?guration in which the original code ran. This report includes machine serial number, machine type, clock speed, and so on, when this information is available from the raw input ?le. -h Shows all traced routines, sorted in descending order by a calculated factor called the inline factor. This statistic is generated by dividing the number of calls to a routine by the average time spent in that routine. Thus, this factor is high when a small routine is called many times. If the factor is 1 or greater, the routine may be a candidate for inlining. If these routines are inlined into their callers, this action may speed up overall execution time. -A Generates reports for all flowview report options, except -H. -m Generates a summary report that contains a listing of routines using the most CPU time, sorted by the amount of time they used. The -O option controls the number of routines shown; default is 5. The same number of inline candidates also are displayed. -a Shows all traced routines, sorted in ascending alphabetic order by routine name. If multitasked details are being listed, routines with the same name will be shown in order by task number. -v Shows all traced routines, sorted in ascending order by average amount of CPU time used during each call. Routines called many times that use very little time per call (suggested number would be 50 microseconds or less) could be candidates for inlining (see the -h option). -w Shows all traced routines, sorted in descending order by average amount of CPU time used during each call. -l Shows all traced routines, sorted in descending order by number of times called. You can use this 004–2182–003 69Guide to Parallel Vector Applications report to identify inlining candidates (see the -h option). -j Shows all traced routines, with details showing the names of the calling routines. This option also discusses how much CPU time was used by the routine when called by the various other routines. -k Invokes a "caller/callee" report. All traced routines are shown, with a listing of which routines they called, and which routines called them. Numeric data is not displayed. -T Generates a text-based dynamic call tree that re?ects all callers and callees traced during execution of user code. Repeated sections of the tree are noted. To limit the scope of this tree, use the -y and -z options. To select the root of the tree, use the -R option. -H Writes all interactive help ?les directly to stdout. -export[_csv] Analyzes the speci?ed data ?le and sends the results to stdout in a comma-separated format. This allows the data to be imported into other analysis tools or into word processors in the form of a table. -export_tabbed Analyzes the speci?ed data ?le and sends the results to stdout in a tab-separated format. This allows the data to be imported into other analysis tools or into word processors in the form of a table. -export_sylk Analyzes the speci?ed data ?le and sends the results to stdout in a SYLK format. This allows the data to be imported into common spreadsheet applications. -X Forces X Window System mode. If the program cannot initialize the X Window System, it will exit with a nonzero status. This option is intended for use with special scripts. The option is not needed for normal use. -e Turns on a ?ag that causes the X Window System mode of flowview to use vi windows to display 70 004–2182–003flowview [3] reports. By default, reports are displayed on the screen by using the X ASCII Widget displays. vi windows allow editing and saving of report output. -V Lists version number and copyright notice on stderr. raw?le Speci?es the name of the raw-format output ?le from an execution of code using the Flowtrace feature. The default name of this ?le is flow.data. If raw?le is speci?ed as one dash (-), flowview reads the standard input. The following options control the format and content of various reports generated by flowview. You may specify the options for either interactive (X Window System or line mode) or noninteractive operation of flowview. If speci?ed for interactive, the options become the default for the interactive session. You can change option values during the interactive session. Option Description -o Causes display of CPU times in clock periods, rather than in seconds. Affects all reports that display CPU times, as well as the dynamic calling tree. -p Causes the inclusion of CPU times and percentages in the dynamic calling tree. These numbers are displayed in both the graphical and text-based calling trees. By default, no CPU times are included in the tree. -N Lists calling tree routines in alphabetical order. By default, routines are listed in the order executed. -R root Speci?es the name of the root routine to use for generation of text-based calling tree output. Argument root (which is case-sensitive) must be speci?ed, and it must be the name of a routine in the executed program. -y treelower Speci?es the lowest level displayed in the text-based calling tree; default is 1. When used with the -z option, this option limits the scope of the calling tree displayed. Argument treelower is a decimal integer, and it must be less than or equal 004–2182–003 71Guide to Parallel Vector Applications to the value speci?ed for treeupper if that option is used. -z treeupper Speci?es the highest level displayed in the text-based calling tree; default is 9999. When used with the -y option, this option limits the scope of the calling tree displayed. Argument treeupper is a decimal integer, and it must be greater than or equal to the value speci?ed for treelower if that option is used. -S nstars Speci?es the number of stars (asterisks), expressed as a decimal integer, to be displayed on the right-hand side of report entries to represent 100%; default is 25. Setting this value to 0 effectively suppresses the asterisks. No practical limit exists to nstars, but specifying very large numbers may make the report unreadable. -O toproutines Speci?es the maximum number of routines to display in the summary report (-m) and the observation report (-B), expressed as a decimal integer; default is 5. For example, if this value is 10, the ten highest users of CPU time, along with the ten top candidates for inlining, will be listed in the summary. -D ?ag Determines whether multitasking details are combined when reports are written. Argument ?ag is either y for yes or n for no; default is y (combines multitasking details). If this ?ag is n, or the data in the raw ?le is already combined across tasks, report information is combined. If this ?ag is y, and the raw ?le contains timings detailed for each task, report information shows times for each task in which the routine was executed. 3.2.2 Instrumenting Your Code You can enable the Flowtrace feature by using the CF90, Cray C++, or Cray C compilers. The following sections discuss instrumenting code using these compilers. 72 004–2182–003flowview [3] 3.2.2.1 Fortran Compiler Using the CF90 compiler, you can enable Flowtrace for an entire program or for particular subprograms. This section describes how to process an entire program. See Section 3.6.1, page 84, for information about processing subprograms. To enable Flowtrace for an entire program, include -ef on the f90(1) command line. This option activates the Flowtrace option of the compiler and include the appropriate tracing library routines during the load step. To generate report output after your program has run to completion, you must enter a separate command. See Section 3.2.1, page 68, for a description of the flowview(1) command. The following examples show the use of Flowtrace with the f90 command (the pgm.f ?le is user code; flow.report is the speci?ed output ?le). CF90 example: f90 -ef yourcode.f90 ./a.out flowview -Luch > flow.report 3.2.2.2 Cray C++ and Cray C Compilers To enable Flowtrace using the Cray C or Cray C++ compilers, include the -F option on the cc(1) command line for the C compiler and the CC(1) command line for the C++ compiler; this applies to the entire executable program. In the following example, the pgm.c ?le is user code; flow.report is the speci?ed output ?le: cc -F pgm.c ./a.out flowview -Luch > flow.report Note: If your code uses the setjmp(3) or longjmp(3) functions, Flowtrace may fail or give misleading results. Flowtrace keeps its own stack of called routines. If you jump out of a routine without its knowledge, Flowtrace may apply timings to the wrong routine or abort. If your code executes recursive function calls, Flowtrace may terminate abnormally due to internal stack over?ow. 004–2182–003 73Guide to Parallel Vector Applications 3.3 Using the X Window System Interface The following sections discuss how to use the X Window System interface to produce flowview reports. See Section 1.7, page 9, for more information on the X Window System interface. To enter resource changes, use the class name Flowview. For example, to change the width of the main text window to 900 pixels, enter the following command line: Flowview*mainTextWindowWidth: 900 To change the default values of widget resources used in flowview, refer to the widgets by name or class. The widget classes used in flowview are Form, Text, and List. Names and descriptions of widgets are as follows: Widget Description flowview Highest-level form for all flowview windows text Text widget that contains all ASCII report information File File menu button Reports Report menu button Graphs Graph menu button Help Help menu button status Bottom status line In addition to the standard X Window System resources, flowview uses the following application-speci?c resources for user customization: Resource Description mainTextWindowWidth Width of text and graphics window, expressed as the decimal number of pixels. Default is 570. mainTextWindowHeight Height of text and graphics window, expressed as the decimal number of pixels. Default is 400. 74 004–2182–003flowview [3] drawTreeConnectingLines A Boolean value (either True or False) that indicates whether the calling tree graphic display should draw connecting lines between routines. Default is True. drawTreeVertSeparation A decimal number of pixels used to give vertical separation between routine names in the graphical calling tree. Default is 2. drawTreeHorizSeparation A decimal number of pixels used to give horizontal separation between routine names in the graphical calling tree. This value can never be fewer than 10 pixels. Default is 30. drawTreeConnectingLineXOffset A decimal number of pixels used to separate the connecting line drawn from parent routines to children routines. This value can never be fewer than 6. Default is 10. drawTreeBoxes A Boolean value (either True or False) used to determine whether bounding boxes are drawn around names shown in the calling tree. Default is True. drawTreeFixedBoxSize A decimal number of characters to use for drawing ?xed-size boxes around routine names in the calling tree. A 0 value indicates that variable-length boxes are drawn and a size increment is used. Default is 0. drawTreeBoxIncrement A decimal number of characters to use for each increment of size when drawing variable-length boxes around routine names. Default is 15. Figure 16, page 76, shows the main flowview window. 004–2182–003 75Guide to Parallel Vector Applications Figure 16. Main flowview Window 3.3.1 File Menu File menu selections permit operations such as opening ?les, opening a demo ?le, and exiting flowview. Figure 17, page 77, shows the File menu. 76 004–2182–003flowview [3] Figure 17. File Menu 3.3.1.1 Open. . . Selection The Open... selection lets you open (load) ?les to be used by flowview. This selection creates an input panel on which you enter the name of the ?le to be opened. The input panel accepts shell wildcard conventions. 3.3.1.2 Open Demo File Selection The Open Demo File selection loads a sample raw ?le that is processed by flowview. 3.3.1.3 Capture Running Process Selection The Capture Running Process selection allows flowview to interactively capture Flowtrace statistics from a running program that has been enabled for the Flowtrace feature. 3.3.1.4 Export Data... Selection The Export Data... selection lets you choose the format and the ?le name for data saved from the display. You can use that ?le to format the data according to your own needs. The formats available are as follows: • CSV, or comma-separated • Tab-separated 004–2182–003 77Guide to Parallel Vector Applications • SYLK, a common spreadsheet format 3.3.1.5 Quit flowview Selection The Quit flowview selection displays a con?rmation panel on which you can con?rm your decision to exit. The con?rmation panel has the following buttons: Button Description Confirm Exits the tool and returns you to the operating system. Pressing RETURN has the same effect. Cancel Cancels the request to quit. To suppress display of the con?rmation panel, add the following command line to your .Xdefaults ?le: If you add the con?rmation panel control command to your .Xdefaults ?le, remember to issue the xrdb(1) command to make your changes known to flowview. flowview*quitConfirm: never Note: If you add the con?rmation panel control command to your .Xdefaults ?le, remember to issue the xrdb(1) command to make your changes known to flowview. 3.3.2 Select Report Desired: Menu The Select Report Desired: menu selections permits you to select the way reports will be viewed. Figure 18, page 79, shows the Select Report Desired: menu. 78 004–2182–003flowview [3] Figure 18. Select Report Desired: Menu 3.3.2.1 Summary Selection The Summary selection displays a summary of signi?cant routines from your Flowtrace execution. The summary contains the following: • Selected routines that used the greatest amount of CPU time • Selected routines that would be the best candidates for inlining 3.3.2.2 Routines Sorted by: Selection The Routines Sorted by: selection lets you choose the routine report you want to see and lets you select the way in which the routines are sorted. When you choose the Routines Sorted by: selection, flowview displays a pop-up menu from which you can generate routine reports, sorted by the following factors: Factor Description Time Displays traced routines sorted by CPU time used. The routines shown at the start of this report are those using the most CPU time. 004–2182–003 79Guide to Parallel Vector Applications Inline Factor Displays traced routines sorted by a statistic indicating whether the routines might be candidates for inlining. Routines shown with factors of 1 or greater probably will bene?t from inlining. Name Displays traced routines sorted alphabetically by routine name. Number of calls Displays traced routines sorted by the number of calls made to the routines. Routines shown at the start of this report are those with the most calls, and, therefore, probably will bene?t from inlining. Average Time per call Displays traced routines sorted by average CPU time used per call. You will see an additional pop-up menu that requests you to enter the desired order of sorting. If you select the ascending sort of average time per call, the routines shown at the start of this report are those with the least CPU time per call, and, therefore, probably will bene?t from inlining. Showing Called-By Timings Displays traced routines, with the number of calls and CPU time, that were gathered for each traced caller of the routine. The routines shown at the start of this report are those with the most CPU time. 3.3.2.3 Caller/Callees Selection The Caller/Callees selection displays a listing of all routines that shows their traced callers and callees (if any). Routines are listed in alphabetical order, as are their callers and callees. 3.3.2.4 Environment Selection The Environment selection displays a listing of information about the working environment for your original Flowtrace execution and current flowview execution. The information displayed includes the following: • Date and time of program execution • Name of executable ?le used 80 004–2182–003flowview [3] • Serial number and clock speed of machine on which Flowtrace was executed • Flowtrace overhead statistics 3.3.2.5 Text Calling Tree Selection The Text Calling Tree selection displays a text version of the calling tree, much like the tree generated by the line-oriented mode. 3.3.2.6 Report Preferences. . . Selection When you select Report Preferences..., you will see an additional window appear on the screen. This window lets you alter selections that control the appearance of reports generated by flowview. Figure 19, page 81, shows the Report Preferences window. Figure 19. flowview Report Preferences Window 3.3.3 Select Graph Desired: Menu The Select Graph Desired: menu selections let you view program information graphically. Figure 20, page 82, shows the Select Graph Desired: menu. 004–2182–003 81Guide to Parallel Vector Applications Figure 20. Select Graph Desired: Menu 3.3.3.1 Pie Chart Selection The Pie Chart selection causes flowview to display a pie chart that depicts the relative amount of CPU time spent in the ?ve most active routines in your program. You can also access this display by clicking on the pie chart icon near the top of the main window. The pie chart itself is also a menu. If you press the left mouse button while the cursor is inside of a pie slice for an active routine, a report will appear showing additional detail. The pie chart is the ?rst display that you will see when flowview starts up. 3.3.3.2 Bar Chart Selection The Bar Chart selection displays the percentage of CPU time spent in all of the routines in your program in bar chart format. You can also access this display by clicking on the horizontal bar chart icon near the top of the main window. The bar chart itself is also a menu. If you press the left mouse button while the cursor is inside of a pie slice for an active routine, a report will appear showing additional detail. 3.3.3.3 Calling Tree Selection The Calling Tree selection displays a report that shows all traced routines, organized into a text-formatted calling tree. You can also access this display by clicking on the tree icon near the top of the main window. 82 004–2182–003flowview [3] 3.3.3.4 Text Report Selection The Text Report selection displays a table that gives a detailed report of the activity within a single module. This is the report you receive when you click on a module in the pie chart or bar chart display. You can also access this display by clicking on the text page icon near the top of the main window. 3.4 Using the ASCII Interface When running flowview in line-mode interactive, specify the generation of reports by entering a number, followed by RETURN (or ENTER). Available reports are shown on a menu. To change the option’s value, enter the number of the desired option, followed by RETURN (or ENTER). Reports and options previously listed are available in interactive mode. Online help is available in line-mode interactive. Reports are displayed using a pager program; the default pager is more(1). If you want to use another pager program, change the FLOWVIEW_PAGE environment variable. The name of the temporary ?le that contains report output will be appended to the end of the command, and the command will be executed. The FLOWVIEW_PAGE environment variable can have up to 100 command-line options. The environment string can contain up to 500 characters. flowview does not perform special parsing of command options. It is assumed that this pager command can be executed from a normal shell (that is, this command can be found on the default $PATH of the user). You can change the default pager command interactively from the Options menu. If you prefer to view the reports by using the vi(1) editor, enter the following commands: Standard (sh) and Korn (ksh) shells: $ FLOWVIEW_PAGE=vi ; export FLOWVIEW_PAGE $ flowview rawfile C (csh) shell: % setenv FLOWVIEW_PAGE vi % flowview rawfile You can specify these environment variable values permanently, if desired, in your .profile or .login ?le. 004–2182–003 83Guide to Parallel Vector Applications 3.5 Using vi Windows Use of vi windows assumes that the vi command can be executed from a normal shell (that is, vi can be found on your default $PATH). The vi editor windows may be slow to appear. Do not press the same mouse buttons repeatedly. Before logging out, close all vi windows. 3.6 Additional Flowtrace Features In addition to tracing procedure calls in entire programs, Flowtrace provides the following features: • Processes selected subroutines • Provides the SETPLIMQ entry point • Recovers information from abnormally terminated programs • Can be controlled by environment variables ! Caution: Multistreaming programs will not execute when instrumented for Flowtrace. 3.6.1 Selective Tracing Because the use of Flowtrace incurs additional CPU overhead, you may want to limit the program areas that are measured. Tracing only selected routines, or sections of routines, can help lower the CPU overhead expended by Flowtrace. This section describes techniques for using selective tracing. 3.6.1.1 Using the FLOWMARK Subroutine The FLOWMARK subroutine lets you measure the performance of selected sections of code within a program unit, such as loops. To use FLOWMARK, call it from the source program immediately preceding and following the code you want to measure. The ?rst invocation includes, as an argument, a name for the code block (to be used in Flowtrace output). The second invocation uses 0 as an argument. You do not have to invoke Flowtrace on the compiler command line to use FLOWMARK calls. 84 004–2182–003flowview [3] The name used as an argument in the ?rst invocation of FLOWMARK is represented as a null-terminated character string. In Fortran, this is speci?ed as a left-justi?ed Hollerith string of no more than 7 characters (such as ’STRING’L). In C, it is speci?ed as a string within double quotation marks, such as "STRING". The name should be different from any routine name in your program. Although you may embed nonprinting characters or blanks in a name passed to FLOWMARK, their use is not encouraged. Embedding special characters (such as tabs or new lines) can make the reports generated by flowview unreadable. When you use FLOWMARK, Flowtrace treats the marked code block as a distinct subroutine. The marked code block is given the name speci?ed in the ?rst call to FLOWMARK. The two calls to FLOWMARK are analogous to calling and returning from a subroutine consisting of your code block. Flowtrace statistics are accurate only if the granularity of the routines or marked areas is suf?ciently high. To use FLOWMARK to time a code region, ensure that the code region uses at least 50 microseconds per call. The examples that follow show how to use the FLOWMARK subroutine. Fortran example: ... CALL FLOWMARK (’PHANTOM’L) DO 10, I=1,10 ... 10 CONTINUE CALL FLOWMARK (0) C example: main() { int zero = 0; void FLOWMARK(); ... ... FLOWMARK("phantom"); for (i = 0 ; i<0 ; i++) { ... /* loop performs desired work */ ... } FLOWMARK(&zero); 004–2182–003 85Guide to Parallel Vector Applications } If you want to time a very small section of code or a routine that uses very little time, you can isolate the code area as a subroutine if it is not a subroutine already. Call the routine many times (for example, 10,000). Place FLOWMARK calls around the entire loop, and divide the resulting statistics by the loop count (10,000 in this example). This technique, shown in the following example, also works well for Perftrace, described in Section 6.1, page 135. Assume that the following source code is from the mysource.f ?le: program timer common /test/ data1,data2.... c c initialize the common data here c call flowmark(’loop1’l) do 100 i=1,10000 call sub1 100 continue call flowmark(0) end subroutine sub1 common /test/ data1,data2.... c c c perform calculations on the common data c items c end Compile and execute mysource.f: f90 -ef mysource.f ./a.out flowview -L The data used in the routine to be timed is placed in common blocks. This action prevents the compiler from eliminating all calculations during its optimization phase. Note that Flowtrace is not speci?ed on the f90 command line because FLOWMARK calls are suf?cient to load the correct libraries. To get the actual ?gures for the code fragment timed, you must divide the flowview statistics by the number of times the routine was called (10,000 in 86 004–2182–003flowview [3] this case). You can obtain similar results by using calls to SECOND before and after the calling loop. If the code in sub1 is autotasked, FLOWMARK will not work correctly. 3.6.1.2 Using Other Methods To trace speci?c subprograms, insert directives in your source code. The CF90 compiler handles Flowtrace directives in the following way: • If Flowtrace is enabled on the command line (-ef on the f90 command), you can selectively disable routines for Flowtrace by placing the following directive just after the ?rst statement of the routine (PROGRAM, FUNCTION, or SUBROUTINE). The disabling stays in effect until the next program unit (PROGRAM, FUNCTION, or SUBROUTINE) is encountered. !DIR$ NOFLOW • If Flowtrace is not enabled on the compiler command line, you can selectively enable routines for Flowtrace by placing the following directive just after the ?rst statement of the routine (PROGRAM, FUNCTION, or SUBROUTINE). The enabling stays in effect until the next program unit (PROGRAM, FUNCTION, or SUBROUTINE) is encountered. !DIR$ FLOW To trace only subroutine one, but nothing else, use the source code. Do not enable the Flowtrace feature on the command line. In example 1, !dir$ flow is in effect only for subroutine one. Subroutines that follow do not have Flowtrace enabled. Example 1: program main call one call two stop end subroutine one !dir$ flow do 100 i=1,10000 ... ... 100 continue return 004–2182–003 87Guide to Parallel Vector Applications end subroutine two do 200 i=1,30000 ... ... 200 continue return end To trace all routines except subroutine two, use the source code in example 2. Enable the Flowtrace feature on the command line. The disabling is in effect only for subroutine two. Subroutines that follow have Flowtrace enabled. Example 2: program main call one call two stop end subroutine one do 100 i=1,10000 ... ... 100 continue return end subroutine two !dir$ noflow do 200 i=1,30000 ... ... 200 continue return end 3.6.2 SETPLIMQ Subroutine - Reporting Each Call Subroutine SETPLIMQ enables Flowtrace to write a line to stderr for every CALL or RETURN statement executed, listing timings for each call. Internal timing information included in the output can be used in debugging. 88 004–2182–003flowview [3] SETPLIMQ is requested by a subroutine call in your source code. In Fortran, this call would appear as follows: CALL SETPLIMQ (dublcalls) The value of dublcalls speci?es the number of trace lines printed. Because one line is produced for each CALL and each RETURN, you should set dublcalls to twice the number of CALL statements for which tracing is desired. These lines are interspersed with other output on stderr. You may set the dublcalls value without changing your source code. Set the FLOW_PLIMQ environment variable to the dublcalls value desired. Existing internal calls to the SETPLIMQ routine will override any external setting made using the environment variable. ! Caution: Never set the dublcalls value to a negative number. Setting the value to a negative number causes unpredictable behavior. Because this option can generate extensive output, you should trace only areas of special interest. Following are ways to select the range that SETPLIMQ reports: • Place CALL SETPLIMQ (dublcalls) immediately before the range to be reported and follow with CALL SETPLIMQ(0). This approach allows conventional Flowtrace to operate on the remainder of the program. • If you are using CF90, you can place the SETPLIMQ call at the beginning of the program and use !DIR$ FLOW and !DIR$ NOFLOW to delimit the ranges to be reported. Flowtrace reports only the speci?ed ranges. Output from SETPLIMQ is sent to stderr. See Section 4.2, page 99, for information on redirecting stderr to another ?le. Example of trace lines: FLOWTRACE -- Call 1 3 ADDCOM by MAIN 0000:00:00.001289478 FLOWTRACE -- Call 1 4 ADDFILE by ADDCOM 0000:00:00.001800522 FLOWTRACE -- Call 1 5 ADDMOD by ADDFILE 0000:00:00.002324922 FLOWTRACE -- Call 1 6 ADDTCOM by ADDMOD 0000:00:00.002847966 FLOWTRACE -- Return 1 5 ADDTCOM to ADDMOD 0000:00:00.003371472 FLOWTRACE -- Return 1 4 ADDMOD to ADDFILE 0000:00:00.003885522 FLOWTRACE -- Return 1 3 ADDFILE to ADDCOM 0000:00:00.004400766 FLOWTRACE -- Return 1 2 ADDCOM to MAIN 0000:00:00.004917492 The ?rst number given is the number of the task in which this event occurred; the following number gives the stack depth at the time of the event. 004–2182–003 89Guide to Parallel Vector Applications The number given on the right is the accumulated CPU time (in seconds) at the time of the event. 3.6.3 flodump Command - for Programs That Terminated Abnormally The flodump(1) command recovers and writes Flowtrace tables to stdout after abnormal traced program termination under the following conditions: • The program was compiled with the Flowtrace option enabled (see Section 3.2.2, page 72) or the program executed calls to the FLOWMARK subroutine. • The version of Flowtrace loaded with your program was the same as that generated with the flodump command. For more information on flodump, see the flodump(1) man page. 3.6.4 Environment Variables You can control the operation of the Flowtrace library by the setting of the following environment variables at execution time: Variable Action FLOW_DATA Causes Flowtrace to write the raw statistical data to the ?le name indicated in the string, rather than to flow.data (the default ?le name). FLOW_MDETAIL Controls the level of detail collected by Flowtrace for multitasked programs. If this variable is set to nonzero, Flowtrace will maintain separate tables and statistics for each task executed. If many tasks are generated, it potentially can cause large overhead and use all available memory at run time. For autotasked programs, the cost in memory is multiplied only by the number of CPUs used. Therefore, you should enable this option when tracing autotasked programs. Default is to combine all CPU ?gures across all tasks. FLOW_PLIMQ Setting this variable to nonzero is the equivalent of executing the SETPLIMQ subroutine. Operation of this routine is described in Section 3.6.2, page 88. This variable allows a program that does not have any references to the SETPLIMQ subroutine 90 004–2182–003flowview [3] to perform this type of detail tracing without modifying the user code. If this variable is 0 (default), the only way to use SETPLIMQ is to code direct calls in the user program. The user’s program can override the externally set value even 3.7 Accuracy of CPU Times Because the Flowtrace libraries execute at the same time as your program, they must try to adjust the CPU times they measure for your routines by the amount of CPU time they require to do their work. This methodology is not perfectly accurate because the libraries cannot completely remove evidence of their execution time from the total statistics. You can observe this by running a "do nothing" routine. Such a routine might look like the following two examples. Fortran example: SUBROUTINE NOTHING END C example: nothing() { } If you execute these routines with Flowtrace enabled, CPU times will accumulate for these routines even though they do not appear to do any work (the routines do perform some housekeeping functions as part of the entry and exit code). The majority of the CPU time you see results from execution of the Flowtrace routines themselves. Note: Routines that use less than 50 microseconds of CPU time per call will show inaccurate timings in Flowtrace and flowview. Accuracy of reported times decreases as the average time per call decreases below 50 microseconds. Be aware that the overhead required to call a subroutine or function is not included by Flowtrace in statistics for that routine. Rather, the CPU time used for this overhead is summed into the counters for the CALLER routine. Thus, the performance impact of small, frequently called routines may not be fully revealed by Flowtrace. 004–2182–003 91Guide to Parallel Vector Applications Additional timing considerations exist for multitasked programs. In particular, on systems other than Cray T90 and Cray C90 series systems, the CPU times shown by flowview for multitasked routines do not include wait-semaphore time. Wait-semaphore time is included in the CPU times of Cray T90 and Cray C90 series systems. Note: If you are using Flowtrace to time routines, your program should not refer to the cpused(3) or TSECND(3) functions. Use of these functions with Flowtrace can cause incorrect timings to be reported for both systems. 3.8 Caution about Underlying Parallelism Even though your program may not be compiled to run multitasked, some underlying libraries (for example, libsci.a) are set up to take advantage of extra available CPUs on your machine. Their execution in other CPUs may affect the overall statistics generated by Flowtrace. If in doubt about this phenomenon, run your program with the NCPUS environment variable set to 1 when using Flowtrace. 3.9 Signal Handling in the Flowtrace Library The Flowtrace library gathers statistics for programs that terminate due to signals (without a core dump) by setting up certain signals during initialization. Usually, the signal catches are set before the ?rst traced routine is entered. When the signal occurs, Flowtrace is entered. Flowtrace then performs normal termination activities, such as writing the raw data ?le. This signal handling may interfere with signal handling of the user program in the following ways: • If the user program catches signals, its signal code may overwrite the catches set by Flowtrace. Therefore, if the signals are caught by the user program, and then cause termination, the flow.data ?le may be empty. • If the user program sets signals and then enters traced code for the ?rst time, Flowtrace may overwrite the catches set by the user. In such a case, the program may not behave as expected if the signals occur. • If a signal is already set to be ignored, Flowtrace will not overwrite it. This behavior is necessary to allow correct execution of user programs with such special programs as nohup(1). Flowtrace catches the following signals: 92 004–2182–003flowview [3] Signal Description SIGHUP Hangup signal (signal 1) SIGINT Interrupt signal (CONTROL-c on the terminal) (signal 2) SIGCPULIM CPU time-limit signal (signal 26) SIGTERM External kill (signal 15) Signals that cause a core dump are not caught because you can recover the Flowtrace information from the core ?le by using the flodump command. 3.10 flowview Sample Output Statistics generated by Flowtrace and displayed by flowview are not exact and should not be interpreted to be accounting information. The flowview command is very sensitive to the format of input data. Raw input that does not conform to the correct format may cause flowview to terminate abnormally. Certain flowview command-line options interfere with options processed by the X Window System. If you have the DISPLAY environment variable set but want to run in line-mode interactive or use command-line options to generate reports, you must specify -L on the flowview command line. The following examples show these reports: • Summary report • Routines sorted by time used • Routines sorted by inline factor • Text calling tree Example 1 shows a summary report. 004–2182–003 93Guide to Parallel Vector Applications Example 1: TOP 5 SIGNIFICANT ROUTINES ------------------------------ (CPU Times are Shown in Seconds) Routine Name Tot Time # Calls Avg Time Percentage Accum% ---------------- -------- -------- -------- ---------- ------- FTREE 5.84E+01 12 4.87E+00 39.88 39.88 ********* TREEF 5.63E+01 60000 9.38E-04 38.45 78.33 ********* DTREE 1.35E+01 12 1.12E+00 9.22 87.55 ** GFORSN 9.16E+00 60000 1.53E-04 6.26 93.80 * NAYBOR 2.85E+00 63 4.52E-02 1.94 95.75 TOP 5 POSSIBLE CANDIDATES FOR IN-LINING ------------------------------------------ GFORSA GFORSN Example 2 shows routines sorted by time used. Example 2: FLOWTRACE RESULTS OF ROUTINES SORTED BY TIME USED (DESCENDING) (CPU Times are Shown in Seconds) Routine Name Tot Time # Calls Avg Time Percentage Accum% ---------------- -------- -------- -------- ---------- ------- FTREE 5.84E+01 12 4.87E+00 39.88 39.88 ********* TREEF 5.63E+01 60000 9.38E-04 38.45 78.33 ********* DTREE 1.35E+01 12 1.12E+00 9.22 87.55 ** GFORSN 9.16E+00 60000 1.53E-04 6.26 93.80 * NAYBOR 2.85E+00 63 4.52E-02 1.94 95.75 PLOTPT 2.00E+00 3 6.66E-01 1.36 97.11 INDEXX 1.81E+00 189 9.60E-03 1.24 98.35 GFORSA 1.34E+00 60000 2.23E-05 0.91 99.26 EOST 5.02E-01 12 4.18E-02 0.34 99.61 REVTREE 2.93E-01 11 2.66E-02 0.20 99.81...... Example 3 shows routines sorted by inline factor. 94 004–2182–003flowview [3] Example 3: FLOWTRACE RESULTS OF ROUTINES SORTED BY ’IN-LINE’ FACTOR (DESCENDING) (CPU Times are Shown in Seconds) (Factors Greater Than 1 Could Indicate Candidates for In-Lining) Routine Name Tot Time # Calls Avg Time Percentage Accum% "In-Line" Factor ------------ -------- -------- -------- ---------- ------- -------------- GFORSA 1.34E+00 60000 2.23E-05 0.91 0.91 22.88 GFORSN 9.16E+00 60000 1.53E-04 6.26 7.17 3.34 * ============================================================================= TREEF 5.63E+01 60000 9.38E-04 38.45 45.62 0.54 ********* GETUSED 9.44E-05 19 4.97E-06 0.00 45.62 0.03 FIXH 6.19E-05 12 5.16E-06 0.00 45.62 0.02 FTREE 5.84E+01 12 4.87E+00 39.88 85.50 0.00 ********* DTREE 1.35E+01 12 1.12E+00 9.22 94.72 0.00 ** NAYBOR 2.85E+00 63 4.52E-02 1.94 96.66 0.00 PLOTPT 2.00E+00 3 6.66E-01 1.36 98.02 0.00 INDEXX 1.81E+00 189 9.60E-03 1.24 99.26 0.00 EOST 5.02E-01 12 4.18E-02 0.34 99.61 0.00 REVTREE 2.93E-01 11 2.66E-02 0.20 99.81 0.00 MTREE 1.25E-01 2 6.24E-02 0.09 99.89 0.00 TOTEN 9.43E-02 6 1.57E-02 0.06 99.96 0.00...... Example 4 shows the text calling tree. Example 4: Ordinal Level Routine Name ------- ----- -------------------------- 1 1 *SYSTEM 2 2 SPH 3 3 EVOL 4 4 HEADER 5 4 INTEGS 6 5 GETUSED 7 5 STEP 8 6 DERIV 9 7 FTREE 10 8 ARTVIS 11 8 DTREE 12 9 FIXH 13 9 TREEF 004–2182–003 95Guide to Parallel Vector Applications 14 10 GFORSA 15 10 GFORSN 16 8 ENERG 17 8 EOST 18 8 GRADPOT...... 96 004–2182–003Hardware Performance Monitor (hpm) [4] Using the Hardware Performance Monitor (HPM) device available on UNICOS systems, the hpm(1) command reports machine performance during execution of programs written in any language available under the UNICOS operating system. On all UNICOS systems, except Cray T90 and Cray C90 systems, hpm can issue any one of four kinds of reports: program summary, hold-issue conditions, memory use, or vectorization. On Cray T90 and Cray C90 systems, hpm issues one report that contains hpm statistics. Appendix B, page 259, discusses the interpretation of statistics from hpm. Note: HPM does not work with Parallel Virtual Machine (PVM) code on UNICOS systems. The hpm command also can gather hardware performance statistics and write them to a raw-format ?le for postprocessing by the perfview(1) command. perfview produces text-like observations about the hardware performance statistics. Producing a report with perfview is discussed in Chapter 6, page 135. The hpm command reports only on whole programs. Because it does not require a separate compilation with the Flowtrace option and does not introduce overhead, hpm is better suited to some problems than others. The Perftrace feature also uses the HPM device. See Chapter 6, page 135, for more information. (Using the Perftrace library in combination with hpm, however, can produce unpredictable results from both utilities.) If your program creates additional processes (normal or multitasking siblings), data for those processes is added to the overall totals. The mega?op value given by hpm can be derived from the elapsed clock time and total CPU time. Calculated in this way, the mega?op rating is correct even when more than one CPU is used. All other statistics are calculated on the basis of total CPU time, regardless of how many CPUs were used. hpm does not break down these statistics for individual tasks. See the note in Section 4.1, page 98 concerning charges for multitasked programs. You should consider the following information when using the hpm command: • The hpm command works with multitasked codes. • The hpm command incurs no signi?cant CPU overhead. 004–2182–003 97Guide to Parallel Vector Applications 4.1 hpm Command The hpm command gathers statistics from the Hardware Performance Monitor (HPM). The command can either generate a report from these statistics, or write them in raw-data format for postprocessing by the perfview command. On Cray T90 and Cray C90 series systems, one execution of the user’s program under hpm is all that is required to gather available performance statistics. On other systems, a single execution of the user’s program under hpm generates only the statistics from one of the four available HPM counter groups. Appendix B, page 259, discusses the interpretation of the statistics generated by the HPM device on Cray T90 and Cray C90 series systems. To use hpm, compile and load your program normally. The hpm command is used similarly to the time(1) command in that your program name is included as an argument operand to the hpm command. The format for the hpm command follows: hpm [-g grp] [-o ?le] [-r] [-V] [-d] program [args] -g grp Speci?es the number of the hardware monitor group to be used. This option does not apply to Cray T90 and Cray C90 series systems because those systems have only one group. grp is an integer in the range 0 through 3, as follows: 0 Execution summary (default) 1 Hold issue conditions 2 Memory activity 3 Vector events and instruction summary -o ?le Writes output to the speci?ed ?le. If this option is not speci?ed, hpm writes output to stderr. -r Generates raw output for postprocessing by tools such as the perfview and awk(1) commands. -V Displays the current hpm version number and a short copyright notice. -d Displays additional rates as though run in dedicated mode. Some rates are based on 98 004–2182–003Hardware Performance Monitor (hpm) [4] wall-clock time, rather than CPU time. Rates roughly estimate concurrent times when the user’s program is multitasked. The displayed rates should not be considered to be exact. The extra wall-clock time required to execute the code to be monitored, combined with the extra wall-clock time needed to notify the hpm command that the users command is ?nished, can easily exceed the wall-clock time needed to execute the code. If you need more precise measurements of execution time versus wall-clock time, use the atexpert(1) command or the ja(1) command. You can run your program on a dedicated machine, or run your program (and hpm) under the ded(8) command. However, even running under ded or on a dedicated machine cannot guarantee exact dedicated timings. program Speci?es executable ?le to be run. args Speci?es arguments to program. Note: On all UNICOS systems, except the Cray T90 and Cray C90 series systems, when monitor group 1 is selected, the operating systems use counter 1 (wait-semaphore time) as a way to avoid accounting charges to multitasked programs for time not spent in useful work. With groups 0, 2, or 3, this information is unavailable; therefore, the accounting charges may be higher and less consistent. 4.2 hpm Output By default, the hpm command writes report-style output to stderr. You also can use the -o option on the hpm command line to write command output to a speci?c ?le. However, you should postprocess the raw-format output of the hpm command by using the perfview program, as shown in example 3 that follows. Example 1 sends both the hpm execution summary and your program’s default output (?les stdout and stderr) to the screen (assuming interactive usage). The user program being compiled is written in Fortran. 004–2182–003 99Guide to Parallel Vector Applications Example 1: f90 mycode.f hpm ./a.out Example 2 sends the hpm execution summary output to the ?le summary, while the program’s default output (both stdout and stderr) are sent to the screen. The program being compiled is written in C. Example 2: cc mycode.c hpm -o summary ./a.out Example 3 generates raw-format hpm data for HPM counter group 3, while allowing the program’s default output to appear on the screen. This program is effective on all UNICOS systems except the Cray T90 and Cray C90 series systems. On Cray T90 and Cray C90 series systems, all data is generated. The resulting data is displayed using perfview. Its report is written to the report.out ?le. The user program being compiled is written in Fortran. Example 3: f90 mycode.f hpm -r -g 3 -o hpm.data ./a.out perfview -LBuchM hpm.data >report.out If you want to save the output of the hpm command, and you do not want to use the -o option on the hpm command line, you must redirect stderr to a ?le. The notation used for redirecting stderr is different between the standard (sh) or Korn ( ksh) shells and the C ( csh) shell, as shown in the examples that follow. Example 4 sends the hpm report output to the report.hpm ?le, and the default output of the program (stdout) to the screen. Example 4 (standard (sh) and Korn (ksh) shells): f90 mycode.f hpm ./a.out 2>report.hpm Example 5 sends the report output of hpm to the report.hpm ?le, and the default output of the program (stdout) to the program.out ?le. Example 5 (C (csh) shell): 100 004–2182–003Hardware Performance Monitor (hpm) [4] f90 mycode.f (hpm -g3 ./a.out > program.out) >& report.hpm 4.3 Multiple hpm Execution It may be useful to generate program statistics from more than one HPM counter group. This action is especially useful when using the perfview tool to view the statistics. It is a requirement of UNICOS system hardware (except Cray T90 and Cray C90 series systems) that you re-execute your program each time you want to gather data from a different HPM counter group, as shown in the following examples. Note: Multiple execution is not required on Cray T90 and Cray C90 series systems. Example 1 generates reports for all four HPM counter groups onto the report.out ?le. The user program in this example is written in Fortran. Line 6 combines all four ?les into one; line 7 removes the four temporary ?les (optional). Example 1: f90 mycode.f hpm -g0 -o data.0 ./a.out hpm -g1 -o data.1 ./a.out hpm -g2 -o data.2 ./a.out hpm -g3 -o data.3 ./a.out cat data.[0-3] > report.out rm data.[0-3] Example 2 combines the raw data from the hpm command. The perfview command reads the combined data. Output is written to the perfview.report ?le. The sample user program being compiled is written in C. Example 2: cc mycode.c hpm -g0 -r -o perf.0 ./a.out hpm -g1 -r -o perf.1 ./a.out hpm -g2 -r -o perf.2 ./a.out hpm -g3 -r -o perf.3 ./a.out cat perf.[0-3] > hpm.data perfview -LBuchM hpm.data > perfview.report 004–2182–003 101Guide to Parallel Vector Applications The perfview command also may be executed in interactive mode, with the additional capability to use the X Window System interface. You should not mix executions of different programs in the input ?le for the perfview command. As example 2 shows, it is permissible (and desirable) to combine up to four runs of the same program (under different HPM counter groups) in perfview input. 102 004–2182–003jumpview [5] This chapter discusses the following topics: • Using the jumpview command • Gathering data with the jt command • Instrumenting your code • Using the X Window System interface • jumpview sample output • Using the ASCII interface • Using vi windows The timings reported by the jumpview(1) command are provided by a simulator that incorporates the behavior of all CPU hardware available on UNICOS systems. These timings are always reproducible. To perform this simulation, your program must be executed with a special kernel loaded with the Jumptrace feature. This kernel analyzes the executing instructions of the program, and it alters them to generate counts of the number of times any code block was executed. Other information also is stored during code execution. Actual timings cannot be calculated until the jumpview program runs. This design lets the Jumptrace feature show timings as small as one clock period; far smaller than the timings possible with any other performance utility. The information within the simulator allows Jumptrace to show statistics, such as mega?ops, without requiring the use of a Hardware Performance Monitor (HPM). Because all routines are traced, Jumptrace can supply performance information for library routines, a feature not available with Flowtrace. 5.1 The Jumptrace Feature The Jumptrace feature lets you determine the exact timing of every executed code block in your program. This feature can be contrasted with the Flowtrace and Pro?ling features. The Flowtrace feature times routines by using less accurate operating system timer measurements. The Pro?ling feature samples and records program execution addresses. 004–2182–003 103Guide to Parallel Vector Applications The following commands are associated with the Jumptrace feature: • The jt(1) command gathers statistics on code during execution. • The jumpview(1) command calculates timing and creates a report from jt command output. You should consider the following when using Jumptrace: • Jumptrace works with multitasked codes. • Jumptrace incurs CPU overhead. • Reloading is required to use Jumptrace. • Jumptrace does not work on segmented programs or on programs that use shared text. To use Jumptrace, you must reload these programs. 5.2 Getting Started with the jumpview Command The jumpview command can run in three modes, depending on the options speci?ed and your user operating environment. The modes are as follows: • If you do not specify command-line options, jumpview will try to operate in interactive mode. It ?rst determines whether your system is con?gured to run under the X Window System. If not, jumpview displays a text menu of options, and you can interact with it in line mode. Choose the desired menu option by entering the number or letter of the menu choice, followed by RETURN (or ENTER). Reports are displayed by a pager program. The default pager program is more. See the more(1) man page for more information. If you prefer, you can change the pager program to another command. • If jumpview determines that you are con?gured to run under the X Window System, it tries to open windows on your workstation screen. If this is successful, a main window appears that contains a large text window that has control buttons across the top. When the cursor is in one of the top control buttons, you can select the desired operations by pressing the left mouse button. In most cases, this action causes additional menus to pop up. You can then make selections from these submenus. Text is displayed in the text window, while graphical information is displayed in a graphics window. Depending on the type of data being displayed, these windows overlay one another. • If you specify one or more command-line options that cause jumpview to generate reports, jumpview writes these reports to stdout and terminates. 104 004–2182–003jumpview [5] The following sections discuss how to use command-line options to produce jumpview reports. For information on producing reports by using the X Window System interface, see Section 5.3, page 115. 5.2.1 jumpview Command-line Options The jumpview command presents an interactive menu when no command-line report option is included; otherwise, an output option can be speci?ed for redirection to a ?le. The most commonly used report options are -S, -R, and -c. The jumpview command line is as follows: jumpview [-S] [-c] [-r] [-A] [-H] [-export[_csv]] [-export_tabbed] [-export_sylk] [-L] [-V] [-X] [-B routine-name] [-C group-name] [-D detail-reports] [-F de?nition-?le] [-Ggroup-reports] [-O option-values] [-R routine-reports] [-T routine] [raw?le] [Xoptions] -S Generates a report that tries to make observations about the performance behavior of the executed program and its traced routines. -c Generates a report that shows pertinent information about the environment in which the original code ran. This report includes machine serial number, machine type, clock speed, and so on (when this information is available from the raw input ?le). -r Generates output that contains all details of statistical information for all code blocks in the executed routines. Unlike the reports described previously, this output is written in a raw format suitable for postprocessing by such tools as awk. This raw ?le cannot be reread by jumpview. -A Activates the following group of report options and generates their reports: -R aicstvwefr -D bsl -c This includes all routine reports, all block detail reports, and an environment report. -H Writes all online help topics directly to stdout. 004–2182–003 105Guide to Parallel Vector Applications -export[_csv] Analyzes the speci?ed data ?le and sends the results to stdout in a comma-separated format. This allows the data to be imported into other analysis tools or into word processors in the form of a table. -export_tabbed Analyzes the speci?ed data ?le and sends the results to stdout in a tab-separated format. This allows the data to be imported into other analysis tools or into word processors in the form of a table. -export_sylk Analyzes the speci?ed data ?le and sends the results to stdout in a SYLK format. This allows the data to be imported into common spreadsheet applications. -L Prevents running of the X Window System. This option is needed only if you are running the X Window System and want to use the line-mode interactive interface of jumpview. It is not needed for normal use. -V Lists the version number and a copyright notice to stderr. -X Forces X Window System mode. If the program cannot initialize the X Window System, it exits with a nonzero status. This option is intended for use with special scripts. It is not needed for normal use. -B routine-name Generates a report that shows detailed information about all routines that have names that match the routine-name value. This string is treated as a regular expression. Case of characters used in routine-name is signi?cant. -C group-name Generates a report that shows detailed information about all groups that have names that match the group-name value. This string is treated as a regular expression. Case of characters used in group-name is signi?cant. -D detail-reports Generates one or more block detail reports. The detail-reports argument determines which reports will show. The detail-reports argument must be 106 004–2182–003jumpview [5] one or more single letters combined. The letters, which may be uppercase or lowercase, take on the following values: b Describes all code blocks individually s Combines statistics for symbolic location in your routines l Combines statistics and shows only those for code loops -F de?nition-?le By default, jumpview reads from the $H ?le looking for group de?nitions. To override this ?le name, de?ne the JUMPVIEW_REPORT_FILE environment variable to the desired ?le name. To override the de?nition ?le name, specify it as the de?nition-?le argument for this option. To disable de?nition processing, specify -F NONE. -G group-reports Generates one or more group reports. The reports that show depend on the group-reports argument. group-reports must be one or more single letters combined. The letters, which may be uppercase or lowercase, take on the following values: u Sorts groups by CPU time h Sorts groups by inline factor m Sorts groups by ?oating-point operation a Sorts groups by group name v Sorts groups by average CPU time per call (ascending) w Sorts groups by average CPU time per call (descending) l Sorts groups by number of calls g Lists all groups -R routine-reports Generates one or more routine reports. The reports that show depend on the routine-reports argument. routine-reports must be one or more single letters combined. The letters, which may 004–2182–003 107Guide to Parallel Vector Applications be uppercase or lowercase, take on the following values: a Sorts alphabetic by routine name i Sorts by inline factor c Sorts by number of calls to each routine s Sorts summary of signi?cant routines t Sorts by CPU time used by each routine v Sorts by average CPU time/call (ascending) w Sorts by average CPU time/call (descending) e Sorts caller/callee report f Sorts by ?oating-point operations per second r Shows routines in a calling tree -T routine Generates a report that shows the disassembled portion of the routine speci?ed by the routine argument. The name of a routine used as an input value must exactly match this string. Case is signi?cant. raw?le Speci?es the name of the raw-format output ?le generated by the jt command. The program running under its control must have been previously loaded with the Jumptrace library (libtrace.a). Default name is jump.data. If raw?le is speci?ed as one dash (-), jumpview reads the standard input. Xoptions Speci?es X Window System defaults. The X Toolkit automatically handles the arguments described in Table 6, page 12. The arguments may be entered directly on the jumpview 108 004–2182–003jumpview [5] command line. These values override any values set in your .Xdefaults ?le. The following options control the format and content of various reports generated by jumpview. You may specify the options for either interactive (X Window System or line mode) or noninteractive operation of jumpview. If speci?ed for interactive, the options become the default for the interactive session. You can change option values during the interactive session. Option Description -O option-values Allows speci?cation of operating parameters for the jumpview command. String option-values must be of the form: opt,opt,opt=value,opt Separate options from one another by a comma. Embedded blanks are not allowed in the option string. Options that require a value must have an equal sign (=) between the option and the value. Case is not signi?cant. Valid values for the option-values argument are as follows: Value Result clocks Causes all CPU times to appear as clock periods, rather than seconds. noclocks Causes all CPU times to appear as seconds, rather than clock periods (default). longformat Causes most reports to show all details of Jumptrace statistics, rather than a short summary. nolongformat Causes most reports to show only selected Jumptrace statistics, rather than all details (default). treetimes Causes CPU times to be included in the calling tree displays and reports. notreetimes Prevents CPU times from being included in the call tree display and reports. 004–2182–003 109Guide to Parallel Vector Applications timernotation Enables special notation of code blocks by the internal code timer that shows code block type (default). notimernotation Suppresses special timer notation. viwindows When using the X Window System, causes each report output display to appear in a separate window. noviwindows When using the X Window System, shows all report output in the main X Window System display (default). addressing= value Sets addressing scheme used when showing code blocks; default is routine. annotation= value Sets symbol annotation mode; default is library. cutoff= value Sets cutoff percentage for detail or long format reports; default is 95%. minblock= nnn Sets the minimum block starting address that will be disassembled. This value must be in parcel format. This format is made up of zero or more octal digits, followed by a single letter in the range a through d. This option is used with the -T command-line option; default is 0a. maxblock= nnn Sets the maximum block starting address that will be disassembled. This value must be in parcel format. This format is made up of zero or more octal digits, followed by a single letter in the range a through d. This option is used with the -T command-line parameter; default is 3777777777a (octal value). top= value Sets the number of top routines shown in the summary report; default is 5. treelow= value Sets the lowest level shown in the calling tree; default is 1. treehigh= value Sets the highest level shown in the calling tree; default is 9999. 110 004–2182–003jumpview [5] stars= value Sets the number of asterisks (stars) to show for 100% on report data lines; default is 25. 5.2.2 Gathering Data Using the jt Command The jt(1) command gathers performance statistics on the functions or subroutines as well as blocks of executable program code inside the functions in your program. The data that jt collects can be processed by the jumpview(1) command to produce a report. The jt command is the Jumptrace feature shell program that activates the Jumptrace library previously loaded with a program. When the jt command is issued, the library gathers statistics during program execution and sends the data to jt. The jt shell writes the data, as speci?ed, to a raw data ?le. The jt command line is as follows: jt [-o raw?le] [-V] [-S group] [-J crit?le] commands -o raw?le Writes output data to ?le raw?le. By default, data is written to the jump.data ?le in the current directory. The raw?le ?le can be used as input to the jumpview program. -V Writes the current version number of jt and a copyright message to stderr. -S group Enables selective tracing and speci?es the group of selection criteria that will be used. The group of criteria must be present in a ?le. Absence of the -S option indicates that full tracing is desired, and no criteria is checked. The criteria ?le will be checked only if you specify this option. -J crit?le Speci?es an alternative ?le name that contains the selection criteria. Otherwise, the criteria must be in your $HOME/.jtrc ?le. The criteria ?le is an ASCII ?le that usually is generated by an editor. The ?le is organized into groups. Each group represents a selection of routines that a user wants to enable or disable for tracing. The design allows several different ?les to be maintained in 004–2182–003 111Guide to Parallel Vector Applications the criteria ?le. Each group is pre?xed by a record of the format .groupname . commands Speci?es the commands to be monitored by the Jumptrace kernel. You can include arguments to this command on the same line. There is no default. The Jumptrace kernel cannot process a program or script in which more than one program has been loaded with the Jumptrace library. If the program being executed was not loaded with the Jumptrace library, the raw data generated will not be useful. 5.2.2.1 Symbols That the jumpview Command Uses The symbols generated by the jumpview command conform with the latest compilers. In particular, the CF90, Cray C++, and Cray C compilers mark areas that are part of Autotasking code. The following symbols are generated by the CF90, Cray C++, and Cray C compilers, and they are used in jumpview reports. Symbol Description (No symbol) Beginning address for the routine, before any executable code. Also appears when debugging symbols are not enabled during program compilation. Ex.routine The entry point of a routine. This symbol marks all C functions at their entry points. This symbol marks all Fortran subroutines, functions, and ENTRY names. See the text following this list for a discussion of double initial letters. Px.routine The start of the routine’s executable code. All activity before this symbol is the "housekeeping" and linkage code for the routine. See the text following this list for a discussion of double initial letters. Lx.61 Line 61 in your source program. See the text following this list for a discussion of double initial letters. 112 004–2182–003jumpview [5] Sx.10A The entry to outer loop DO 10. See the text following this list for a discussion of double initial letters. Sx.10B The exit from outer loop DO 10. Sx.10C, Sx.10D, and so on 10C and 10D are the entry to and exit from a loop nested within another loop when both loops end on label 10; another nesting level would use 10E and 10F, and so on. Sx.10A2 Indicates loop unrolling. The loop is replaced by a long copy and an optional short copy; the long copy equals n iterations of the loop. If the trip count is an even multiple of n, 10A1 is the entry to the long copy, and no short copy exists. If the trip count divided by n has a remainder, 10A1 is the short copy and 10A2 If your program is multitasked, the beginning of the symbol name may contain two letters rather than one. The second letter has the following meaning: M Master code S Slave code U Unitasked code Thus, the symbol LU.150 indicates line 150 of your program and the code executed is in the unitasked version of the routine. 5.2.3 Instrumenting Your Code To enable the Jumptrace feature, load your program with the libtrace.a library. This is usually done on the loader or compiling system command line by using the -ltrace option. Then, execute your program by using the jt command. When your program has completed execution, the gathered statistics data will be available for jumpview to process. In all of the following examples, debugging symbols have been enabled to allow reporting of detail timings within routines. The examples show jumpview being run in interactive mode. 004–2182–003 113Guide to Parallel Vector Applications Fortran 90 example: f90 -G1 -ltrace yourcode.f90 jt ./a.out jumpview The following example compiles and loads the mycode.c ?le, executes jt to gather Jumptrace statistics for a C program, and invokes jumpview interactively. C example: cc -ltrace -Gp mycode.c jt ./a.out jumpview 5.2.4 Compiler Debug Symbols This section shows which compiler options allow the generation of debug symbols. The jt and jumpview commands use information generated by the compiler debug option. For example, the routine name shown in the listing may include an additional symbol, which helps pinpoint the exact location of the executing code. The amount and type of information generated varies by compiler. Table 7, page 114, shows the following options to the compiler commands. The options allow compiler-generated symbols to be displayed. Table 7. Compiler Debugging Information Language Command Option Description CF90 f90 -G1 Good information; partial optimization enabled. Very little impact on code ef?ciency (10% to 20% slowdown possible). Cray C, Cray C++ cc, CC -Gn All line numbers present; no optimization. -Gp All key line numbers present; partial optimization. (10% to 20% slowdown possible.) 114 004–2182–003jumpview [5] Language Command Option Description -Gf No useful line numbers present; full optimization. -g All line numbers present; no optimization. 5.3 Using the X Window System Interface The following sections discuss how to use the X Window System interface to produce jumpview reports. See Section 1.7, page 9, for more information on the X Window System interface. For information on producing reports by using command-line options, see Section 5.2.1, page 105. To enter resource changes, use the class name Jumpview. For example, to change the width of the main text window to 900 pixels, enter the following command line: Jumpview*mainTextWindowWidth: 900 To change the default values of widget resources used in jumpview, refer to the widgets by name or class. The widget classes used in jumpview are Form, Text, Label, Toggle, and Scroll. Names and descriptions of widgets are as follows: Widget Description jumpview Highest-level form for all jumpview windows text Text widget that contains all ASCII report information File File menu button Reports Report menu button Graphs Graph menu button Help Help menu button status Bottom status line In addition to the standard X resources, jumpview uses the following application-speci?c resources for user customization: 004–2182–003 115Guide to Parallel Vector Applications Resource Description mainTextWindowWidth Width of text and graphics window, expressed as the decimal number of pixels. Default is 570. mainTextWindowHeight Height of text and graphics window, expressed as the decimal number of pixels. Default is 400. Figure 21, page 117, shows the main jumpview window. The icons near the top of the display offer you the same functionality as the menu selections through a graphical user interface. 116 004–2182–003jumpview [5] Figure 21. Main jumpview Window 5.3.1 File Menu File menu selections permit operations such as opening ?les, opening a demo ?le, and exiting jumpview. Figure 22, page 118, shows the File menu. 004–2182–003 117Guide to Parallel Vector Applications Figure 22. File Menu 5.3.1.1 Open. . . Selection The Open... selection lets you open (load) ?les to be used by jumpview. This selection creates an input panel on which you enter the name of the ?le to be opened. The input panel accepts shell wildcard conventions. 5.3.1.2 Open Demo File Selection The Open Demo File selection loads a sample raw ?le that is processed by jumpview. 5.3.1.3 Export Data... Selection The Export Data... selection lets you choose the format and the ?le name for data saved from the display. You can use that ?le to format the data according to your own needs. The formats available are as follows: • CSV, or comma-separated • Tab-separated • SYLK, a common spreadsheet format 5.3.1.4 Quit jumpview Selection The Quit jumpview selection displays a con?rmation panel on which you can con?rm your decision to exit. 118 004–2182–003jumpview [5] The con?rmation panel has the following buttons: Button Description Confirm Exits the tool and returns to the operating system. Pressing RETURN has the same effect. Cancel To suppress display of the con?rmation panel, add the following command line to your .Xdefaults ?le: jumpview*quitConfirm: never Note: If you add the con?rmation panel control command to your.Xdefaults ?le, remember to issue the xrdb(1) command to make your changes known to jumpview. 5.3.2 Select Report Desired: Menu The Select Report Desired: menu selections permit you to select the way reports will be viewed. Figure 23, page 120, shows the Select Report Desired: menu. 004–2182–003 119Guide to Parallel Vector Applications Figure 23. Select Report Desired: Menu 5.3.2.1 Observations Selection The Observations selection displays an observations report. For data generated by Jumptrace, the observations cover both the entire program and the top ?ve routines in the program. Items reported in the observations include the following: • Indication of the degree of vectorization of the program or routine • Discussion of ?oating-point operations that occurred • Identi?cation of possible bottlenecks in the program or routine • Recommendations to the user for possible increased ef?ciency 120 004–2182–003jumpview [5] 5.3.2.2 Summary Selection The Summary selection displays a summary of signi?cant routines from your Jumptrace execution. The summary contains the following: • Selected routines that used the greatest amount of CPU time • Selected routines that would be the best candidates for inlining 5.3.2.3 Routines: Selection The Routines: selection lets you choose the routine report you want to see. When you choose the Routines: selection, jumpview displays a pull-down menu from which you can generate routine reports. The following selections will appear: Selection Description Time Displays traced routines sorted by the amount of CPU time used. Floating-Point Operations/Sec Displays traced routines sorted by the number of ?oating-point operations per second. Inline Factor Displays traced routines sorted by a statistic indicating whether the routine is a candidate for inlining. Name Displays traced routines sorted alphabetically by routine name. Number of Calls Displays traced routines sorted by the number of calls made to the routines. Average Time per Call Displays traced routines sorted by average CPU time used per call. Another pop-up menu appears and you can select whether you want to see the routines sorted by average time per call, ascending, or descending order. If you choose the ascending sort, the routines shown at the start of this report are those with the least CPU time per call and, therefore, the most likely candidates for inlining. 004–2182–003 121Guide to Parallel Vector Applications 5.3.2.4 Details Within Routines: Selection When you choose the Details Within Routines: selection, jumpview displays a pull-down menu from which you can select the desired detail report. The following selections will appear: Selection Description Blocks Combined by Symbol Displays traced routines sorted by CPU time used. The routines shown at the start of this report are those using the most CPU time. All code blocks that reside within the memory area covered by a debugging symbol are combined, and the data is shown for that symbol. If no debugging symbols are enabled for a routine, the symbol name will be shown as (no symbol). All Code Blocks Displays traced routines sorted by CPU time used. The routines shown at the start of this report are those using the most CPU time. All code blocks for the routine are shown, with their detail statistics. If a code block has a symbolic name associated with it, that name will be shown. A symbolic name may have more than one code block. Only Code Blocks in Loops Displays traced routines sorted by CPU time used. The routines shown at the start of this report are those using the most CPU time. All code blocks for the routine are shown, but only blocks that terminate loops will show performance statistics. Because loops can be nested, and CPU time in a routine may not always be used inside of a loop, the total of all loop CPU time percentages shown may not be 100%. Special Raw Block Output Generates a raw-format output that contains all statistical information about all routines and all code blocks in those routines. 122 004–2182–003jumpview [5] 5.3.2.5 Routine Name Search Selection The Routine Name Search selection lets you choose which routines are shown in a special long-format report. When you choose the Routine Name Search selection, jumpview displays a pop-up dialog box that lets you enter a search string. This search string is treated as a regular expression. 5.3.2.6 Caller/Callees Selection The Caller/Callees selection displays a listing of all routines that shows their traced callers and callees (if any). Routines are listed in alphabetical order, as are their callers and callees. 5.3.2.7 Disassemble Selection The Disassemble selection causes a new window to appear, letting you set the operating parameters of the disassembly report. 5.3.2.8 Groups: Selection The Groups: selection lets you choose the group report you want to see. This selection is available only if you have de?ned routine groups. When you choose the Groups: selection, jumpview displays a pull-down menu from which you can generate a group report. The report is sorted by the following: • Time • Floating-point Operations per Sec • Inline Factor • Name • Number of Calls • Average Time per Call 004–2182–003 123Guide to Parallel Vector Applications 5.3.2.9 Group Name Search Selection The Group Name Search selection lets you choose which groups are shown in a special long-format report. This selection is available only if you have de?ned routine groups. 5.3.2.10 Show Groups Selection The Show Groups selection lets you view all routine groups de?ned in your group de?nition ?le. This selection is available only if you have de?ned routine groups. 5.3.2.11 Environment Selection The Environment selection displays a listing of information about the working environment for your original Jumptrace execution, and current jumpview execution. Information displayed includes the following: • Date and time of program execution • Name of executable ?le used • Serial number and clock speed of machine on which Jumptrace was executed • Version of Jumptrace kernel • Version of jumpview 5.3.2.12 Report Preferences. . . Selection When you select Report Preferences..., you will see an additional window appear on the screen. This window lets you alter the options that control the appearance of the various reports generated by jumpview. Figure 24, page 125, shows the report preferences window. 124 004–2182–003jumpview [5] Figure 24. jumpview Report Preferences Window 5.3.3 Select Graph Desired: Menu The Select Graph Desired: menu selections let you view program information graphically. Figure 25, page 126, shows the Select Graph Desired: menu. 004–2182–003 125Guide to Parallel Vector Applications Figure 25. Select Graph Desired: Menu By clicking on buttons for the items in the lower half of the menu, you can change the information that jumpview displays. By clicking on the formats in the upper half of the menu, you can change the format in which the information is displayed. 5.3.3.1 Pie Chart Selection The Pie Chart selection causes jumpview to display a pie chart that depicts the relative amount of CPU time spent in the ?ve most active routines in your program. You can also access this display by clicking on the pie chart icon near the top of the main window. The data shown in the graph can be changed by pressing the left mouse button when the cursor is on one of the buttons for the selections below the line. 126 004–2182–003jumpview [5] The pie chart itself is also a menu. If you press the left mouse button while the cursor is inside of a pie slice for an active routine, a report will appear showing additional detail. The pie chart is the ?rst display that you will see when jumpview starts up. 5.3.3.2 Group Pie Chart Selection The Group Pie Chart selection is available only if you have speci?ed routine groups by using the group de?nition ?le. The pie chart display shows the relative amount of CPU time spent in the ?ve most active routine groups de?ned in your program. 5.3.3.3 Bar Chart Selection The Bar Chart selection displays information in bar chart format. You can also access this display by clicking on the horizontal bar chart icon near the top of the main window. The bar chart itself is also a menu. If you press the left mouse button while the cursor is inside of the bar of the bar chart for an active routine, a report will appear showing additional detail. 5.3.3.4 Text Report Selection The Text Report selection displays a table that gives a detailed report of the activity within a single module. This is the report you receive when you click on a module in the pie chart or bar chart display. You can also access this display by clicking on the text page icon near the top of the main window. 5.3.3.5 Text Calling Tree Selection The Text Calling Tree selection displays a report that shows all traced routines, organized into a text-formatted calling tree. You can also access this display by clicking on the tree icon near the top of the main window. 5.4 Using the ASCII Interface When running jumpview in line-mode interactive, you can specify the generation of reports by entering a number, followed by RETURN (or ENTER). Available reports are shown on a menu. To change the option’s value, enter the number of the desired option, followed by RETURN (or ENTER). 004–2182–003 127Guide to Parallel Vector Applications The reports and options previously listed are available in interactive mode. Online help is available in line-mode interactive. Reports are displayed using a pager program; the default pager is more. If you want to use another pager program, change the JUMPVIEW_PAGE environment variable. When this variable is changed, the name of the temporary ?le that contains report output is appended to the end of the command, and the command is executed. The JUMPVIEW_PAGE environment variable can have up to 100 command-line options. The environment string can contain up to 500 characters. jumpview does not perform special parsing of command options. It is assumed that this pager command can be executed from a normal shell (that is, this command can be found on the default $PATH of the user). You also can change the default pager command interactively from the Options menu. If you prefer to view the reports by using the vi editor, enter the following commands: Standard (sh) and Korn (ksh) shells: $ JUMPVIEW_PAGE=vi ; export JUMPVIEW_PAGE $ jumpview rawfile C (csh) shell: % setenv JUMPVIEW_PAGE vi % jumpview rawfile You can specify these environment variable values permanently, if desired, in your .profile or .login ?le. 5.5 Using vi Windows Use of vi windows assumes that the vi command can be executed from a normal shell (that is, vi can be found on your default $PATH). The vi editor windows may take a few moments to appear. Do not press the same mouse buttons repeatedly. Before logging out, close all vi windows. 128 004–2182–003jumpview [5] 5.6 Accuracy of CPU Times The CPU times given by the jumpview command are calculated by analysis of the actual instructions executed in your program. These times are reproducible. However, they may not exactly duplicate times measured on a dedicated system, for the following reasons: • The jumpview command cannot accurately determine memory contention delays. • The jumpview command does not determine delays due to instruction buffer fetches. • Stride information is not available for vector register loads and stores. • Shared register access delays are unknown. • The jumpview command determines the total number of vector operations and the number of times each vector instruction was executed. However, this information is not suf?cient to time accurately blocks of vector instructions if vector loops are run with varying trip counts. jumpview computes an average trip count for a vector loop and assumes that this applies to all executions of the loop. It also assumes that the modulus of this trip count is executed on the ?rst pass. Although these assumptions are generally valid, instances exist in which they produce inaccurate times. Times computed by jumpview should almost always be within 20% of those measured on a dedicated system. Computed times are frequently within a few percentage points. 5.7 jumpview Sample Output Statistics generated by the Jumptrace feature and displayed by jumpview are not exact and should not be interpreted to be accounting information. The jumpview command is very sensitive to the format of input data. Raw input that does not conform to the correct format may cause jumpview to terminate abnormally. Certain command-line options interfere with options processed by the X Window System. If you have the DISPLAY environment variable set and want to generate noninteractive reports or run in line-mode interactive, you must specify -L on the command line. The following examples show these reports: 004–2182–003 129Guide to Parallel Vector Applications • Summary report (jumpview -Rs) • Report of routines sorted by mega?ops achieved (jumpview -Gu) • Long-format report of routines sorted by mega?ops achieved (jumpview -Rf) • Detail report (jumpview -D) Example 1 shows a summary report. Example 1: JUMPTRACE DATA REPORT Showing the Top 5 Significant Routines (CPU Times are Shown in Seconds) Name Called Time Avg Tim EX % ACM % Mflops -------- ------- -------- -------- ----- ----- ----- TREEF 10000 9.01E+00 9.01E-04 42.7 42.7 9.3 ********** FTREE 2 6.63E+00 3.32E+00 31.5 74.2 19.9 ******* NAYBOR 32 1.58E+00 4.93E-02 7.5 81.7 6.5 * SQRT% 859733 1.26E+00 1.4E-06 6.0 87.6 13.6 * INDEXX 96 9.31E-01 .70E-03 4.4 92.1 1.3 * Example 2 shows routines sorted by mega?ops achieved. Example 2: JUMPTRACE DATA REPORT Showing Routines Sorted by MegaFlops per Second (CPU Times are Shown in Seconds) Name Called Time Avg Tim EX % ACM % Mflops Mmems -------- ------- -------- -------- ----- ----- ----- ----- %SQRT% 56658 2.50E-01 4.42E-06 1.2 1.2 190.2 0.7 GFORSN 10000 6.37E-01 6.37E-05 3.0 4.2 140.6 181.4 %RTOR% 79 7.22E-04 9.14E-06 0.0 4.2 110.3 21.7 DTREE 2 1.89E-01 9.45E-02 0.9 5.1 107.8 99.1 INFORM 1 1.06E-03 1.06E-03 0.0 5.1 75.8 94.6 GFORSA 10000 1.36E-01 1.36E-05 0.6 5.8 64.0 71.6 ENERG 2 1.44E-04 7.18E-05 0.0 5.8 61.7 44.6 ANGMOM 1 2.15E-04 2.15E-04 0.0 5.8 56.9 40.3 PRGRAD 2 1.44E-04 7.20E-05 0.0 5.8 52.7 53.3 STEP 1 4.88E-04 4.88E-04 0.0 5.8 47.7 72.0 130 004–2182–003jumpview [5] ARTVIS 2 1.60E-04 7.98E-05 0.0 5.8 47.6 71.8 FTREE 2 6.63E+00 3.32E+00 31.5 37.2 19.9 13.5 ******** =========================================================== Totals 978763 2.11E+01 100.0 100.0 19.6 17.6 Example 3 shows a long-format report of routines sorted by mega?ops achieved. Example 3: Routine: TREEF was responsible for 42.7% of total CPU time. It was called 10000 times, and used 9.01E+00 seconds of CPU time, (1.41E+09 clock periods). Average CPU time used per call was 9.01E-04 seconds, ( 140510 clock periods). The routine performed 9.3 million floating-point operations per second. The routine performed 4.4 million logical operations per second. The routine performed 12.3 million memory operations per second. Total scalar floating-point operations performed: Scalar Floating-Point Add 53041071 Scalar Floating-Point Multiply 30998118 Total operations in all functional units performed: Scalar Floating-Point Add 53041071 Scalar Floating-Point Multiply 30998118 Scalar Integer Add 21078624 Scalar Logical 13050782 Scalar Shift 20000 Scalar Pop 26248438 Total vector/block memory references: Vector/Block Read (Port A) 290000 Vector/Block Read (Port B) 120000 Vector/Block Write 290000 004–2182–003 131Guide to Parallel Vector Applications Total functional unit hold-issue clock periods: Memory Write/Scalar Port (C) 150000 Total result register hold-issue clock periods: A Register 13360782 S Register 1.69E+08 B Register 20000 T Register 70000 Total operand register hold-issue clock periods: A Register 41873320 S Register 4.18E+08 B Register 350000 T Register 8150924 Total memory words read into registers: A Register 10000 S Register 41088680 B Register 290000 T Register 120000 Total memory words written from registers: A Register 40000 S Register 69342652 B Register 170000 T Register 120000 Example 4 shows a detail report. Example 4: JUMPTRACE DATA REPORT Showing Routines with Details (CPU Times are Shown in Seconds) Routine: TREEF Program Percentage = 42.72% CPU Time Total = 9.01E+00 Symbol Name Trips CPU Time Mflop Rtn% Acc% ---------------- ------- -------- ------ ----- ----- E.TREEF 10000 1.13E-02 0.9 0.1 0.1 S.10 166964 1.50E-02 0.0 0.2 0.3 L.4230 156964 3.32E-02 0.0 0.4 0.7 132 004–2182–003jumpview [5] S.20 6849274 4.39E-01 0.0 4.9 5.5 * L.4233 6692310 4.55E+00 13.2 50.5 56.0 ************ L.4242 1966023 2.77E-01 7.1 3.1 59.1 L.4243 1669460 2.25E-01 0.0 2.5 61.6 L.4244 834730 2.09E-01 0.0 2.3 63.9 L.4248 296563 6.84E-02 0.0 0.8 64.7 L.4253 4726287 2.77E+00 7.9 30.8 95.4 ******* L.4262 1458614 3.37E-01 0.0 3.7 99.2 L.4264 156964 6.14E-02 0.0 0.7 99.8 L.4267 10000 2.12E-03 0.0 0.0 99.9 L.4278 10000 1.19E-02 0.8 0.1 100.0 004–2182–003 133perfview [6] The perfview(1) command generates and displays information from Perftrace raw-format output. This chapter will discuss the following topics: • Using the perfview command • Instrumenting your code • Using the X Window System interface • Using the ASCII interface • Using vi windows • Additional Perftrace features • Signal handling in the Perftrace library • perfview report output 6.1 The Perftrace Feature The Perftrace feature gives the same kind of statistics about computer hardware performance as those generated by the hpm command. Perftrace statistics are detailed for individual program units within your program. Perftrace monitors scalar activity, hold-issue conditions, memory use, and vectorization similar to the hpm command. Appendix B, page 259, detail the Hardware Performance Monitor (HPM) counters used by Perftrace and discuss statistic interpretation. To invoke Perftrace, specify the Flowtrace option on the compilation command line and force the load of the libperf.a library rather than the Flowtrace library routines, which reside in libc.a. By setting an environment variable, you can generate a Perftracing report at the time your instrumented executable ?le terminates. See the perftrace(7) man page for more details. You can generate reports from Perftrace output by using the perfview command. This chapter contains examples of output from both user-de?ned perfview reports (see Section 6.11, page 166) and nonuser-de?ned perfview reports (see Section 6.10, page 165). The following commands and subroutines are associated with Perftrace: 004–2182–003 135Guide to Parallel Vector Applications • The perfview command produces a report from Perftrace raw-format output. • The perfdmp(1) command recovers Perftrace data if an application program terminates abnormally. • The FLOWMARK subroutine allows code blocks within program units to be timed separately. You should consider the following when using Perftrace: • Multistreaming programs will not execute when instrumented for Perftrace. • Perftrace works with multitasked codes only if the NCPUS environment variable is set to 1 before the user program begins execution. If this is not done, the program may terminate abnormally or give misleading results. • Perftrace incurs CPU and operating system overhead, which in some cases is signi?cant. To decrease system overhead use selective tracing. See Section 6.6.1, page 157, for more information. • Use of the FLOWMARK subroutine requires source code changes. • Use of the Perftrace feature requires recompilation and reloading. 6.2 Getting Started with the perfview Command By default, the perfview command runs interactively, either with the X Window System interface or in line mode. When you do not specify a report on the command line, perfview presents an interactive menu. If you specify a report option, the perfview command creates one or more reports from the raw output generated by Perftrace, hpm, or hpmall, and writes the report output to stdout. The following sections discuss how to use the perfview command line to produce perfview reports. For information on producing reports using the X Window System interface, see Section 6.3, page 145. 6.2.1 perfview Command-line Options The perfview command line is as follows. Speci?cation of any of the following options causes perfview to run in noninteractive mode. Report output is written to stdout, and perfview terminates. 136 004–2182–003perfview [6] perfview [-L] [-B] [-u] [-h] [-c] [-M] [-a] [-A] [-C gncm] [-C nn] [-H] [-export[_csv]] [-export_tabbed] [-export_sylk] [-l] [-m] [-v] [-D] [-U reportname] [-F report?le] [-o] [-O toproutines] [-R] [-P cutoff] [-S nstars] [-X] [-e] [-V] [raw?le] -L Prevents execution of the X Window System. Always specify this option whenever you want to generate a report by using a command-line option. You also can use this option to specify line-mode interactive when no other report-generation options are speci?ed. -B Generates a report of observations about the performance of the executed program and its traced routines. On all UNICOS systems, except Cray T90 Cray C90 series systems, if more than one set of HPM counters is available in the raw ?le, this report will show the relationships of interest between those counters. See Section 6.3.2.1, page 150, for more information about the observation report. -u Shows all traced routines sorted in descending order by amount of CPU time used. The ?rst few lines of this report indicate where the most CPU time is being spent. If perfview is reading data generated by the hpm or hpmall command, this report is replaced by a summary report of the overall statistical results. See also the -R option. -h Shows all traced routines, sorted in descending order by a calculated factor called the inline factor. This statistic is generated by dividing the number of calls to a routine by the average time spent in that routine. Thus, this factor is high when a small routine is called many times. If the factor is 1 or greater, the routine may be a candidate for inlining. If these routines are inlined to their callers, overall execution time may decrease. 004–2182–003 137Guide to Parallel Vector Applications If perfview is reading data generated by the hpm or hpmall command, this report is replaced by a summary report of the overall statistical results. See also the -R option. -c Generates a report that shows pertinent information about the environment in which the original code ran. This report includes machine serial number, machine type, clock speed, and so on, when this information is available from the raw input ?le. -M Shows all traced routines sorted in descending order by the number of ?oating-point operations per second that were performed during program execution. This report is always available with Cray T90 and Cray C90 series systems. For systems other than Cray T90 and Cray C90 series systems, this report is useful only if the raw input data contains either the HPM group 0 and/or group 3 data. If neither of these groups is present in the raw data, the report is sorted in descending order by the amount of CPU time spent in each routine. -a Shows all traced routines sorted in alphabetic ascending order by routine name. If perfview is reading data generated by the hpm(1) or hpmall(8) commands, this report is replaced by a summary report of the overall statistical results. See also the -R option. -A Generates reports for all report options listed in this section, except -C and -H. -C gncm Shows all traced routines, sorted in descending order by the HPM counter speci?ed in this option. For input data from Cray SV1 and Cray J90 series systems, the format of the counter speci?cation is gncm; the letters c and g are invariant. n is the HPM group number desired, and it must be a 138 004–2182–003perfview [6] digit from 0 to 3. m is the desired counter number and must be a digit from 0 to 7. For example, -C g3c0 speci?es a report sorted by counter 0 of group 3. If the raw input data does not contain HPM statistics from the speci?ed group, the report will contain only a warning of this fact. If perfview is reading data generated by the hpm or hpmall commands, this report is replaced by a summary report of the overall statistical results. See also the -R option. -C nn For input data from Cray T90 and Cray C90 series systems, the counter speci?cation is nn; nn is the desired counter number, and must be a decimal number from 0 through 31. For example, -C20 speci?es a report sorted by counter 20. -H Writes all online help ?les directly to stdout. -export[_csv] Analyzes the speci?ed data ?le and sends the results to stdout in a comma-separated format. This allows the data to be imported into other analysis tools or into word processors in the form of a table. -export_tabbed Analyzes the speci?ed data ?le and sends the results to stdout in a tab-separated format. This allows the data to be imported into other analysis tools or into word processors in the form of a table. -export_sylk Analyzes the speci?ed data ?le and sends the results to stdout in a SYLK format. This allows the data to be imported into common spreadsheet applications. -l Shows all traced routines sorted in descending order by the number of times called. You can use this report to identify inlining candidates. The description for the -h option contains related information. 004–2182–003 139Guide to Parallel Vector Applications If perfview is reading data generated by the hpm or hpmall command, this report is replaced by a summary report of the overall statistical results. See also the -R option. -m Generates a summary report that contains a listing of traced routines using the most CPU time, sorted by the amount of time used. The -O option controls the number of routines shown; default is 5. -O also controls the number of inline candidates displayed. If perfview is reading data generated by the hpm or hpmall command, this report is replaced by a summary report of the overall statistical results. -v Shows all traced routines, sorted in ascending order by the average amount of CPU time used during each call. You can use this report to identify inlining candidates. The description for the -h option contains related information. If perfview is reading data generated by the hpm or hpmall command, this report is replaced by a summary report of overall statistical results. See also the -R option. -U reportname Generates a user-de?ned report and writes it to stdout. See the -F option description for more information. -X Forces X Window System mode. If the program cannot initialize the X Window System, it exits with a nonzero status. This option is intended for use with special scripts; it is not needed for normal use. -e Causes the X Window System mode of perfview to use vi windows to display reports. By default, reports are displayed on the screen by using the Text widget. vi windows allows editing and saving of report output. -V Lists version number and a copyright notice on stderr. 140 004–2182–003perfview [6] raw?le Speci?es the name of the raw-format output ?le from: the execution of a program using the Perftrace feature, perfdmp command output, the raw-format output of an hpm execution, or the raw-format output of an hpmall execution. You cannot mix these data types. However, when Perftrace or hpm data is used on systems other than Cray T90 and Cray C90 series systems, this ?le may contain the combined raw data from up to four separate executions of the same program, with each run being executed under a different hpm counter group. To combine raw ?les, use the cat(1) command. When using a Cray T90 or Cray C90 series system, all data is written with one execution. The default name of this ?le is perf.data. If raw?le is speci?ed as one dash (-), perfview reads from the standard input. ! Caution: If you mix the Perftrace or hpm raw data from different user source programs, perfview may report misleading results. The following options control the format and content of various reports generated by perfview. You can specify the options for either interactive (X Window System or line mode) or noninteractive operation of perfview. If speci?ed for interactive, the options become the default for the interactive session. You can change option values during the interactive session. Option Description -D Speci?es the use of unadjusted statistics. By default, perfview uses the adjusted statistics generated by Perftrace in all reports. If you want to see unadjusted ?gures, including Perftrace execution, use the -D option. Usually, Perftrace subtracts an estimate of its own execution times and performance ?gures from the statistics gathered for user routines. When perfview reads raw data from the execution of the hpm or hpmall command, the adjusted and unadjusted ?gures are identical. 004–2182–003 141Guide to Parallel Vector Applications -F report?le Speci?es a user-de?ned report and report de?nition ?le. report?le is the name of the user report de?nition ?le to be read by perfview. If you omit the -F option, perfview reads variable and report de?nitions from the default ?le. perfview ?rst looks for a default name in the PERFVIEW_REPORT_FILE environment variable. If that variable is not speci?ed, perfview looks in the $HOME/.perfvrc ?le. If a user-speci?ed report de?nition ?le is missing, perfview terminates. If the default $HOME/.perfvrc ?le is missing, perfview assumes that you do not want to generate user reports. If you do not want any report de?nition ?le to be read, specify -F NONE on the command line. The reportname argument is the name of a user report to be generated by perfview. The name given here must match the name of a report de?ned in the user report de?nition ?le. If you specify the -U option, the indicated report is written to stdout, and perfview terminates. See Section 6.3.2.8, page 152, for more information. -o Displays CPU times in clock periods, rather than in seconds. Affects all reports that display CPU times. -O toproutines Speci?es the maximum number of routines to display in the summary report (-m) and the observation report (-B); default is 5. For example, if this value is 10, the 10 highest users of CPU time will be listed. -R Reports all available details. By default, perfview generates short-format reports designed to display pertinent information for each routine on one line of output. If you want to see reports that contain all available details of the HPM counters for the routines, specify the --R option. This option does not affect the content of user-de?ned reports. 142 004–2182–003perfview [6] -P cutoff Speci?es a percentage cutoff for all long-format reports. Use a decimal number from 0 through 100; default is 95. Specifying 0 requests only the statistics for the routine that used the most time. Specifying 100 requests the statistics for all routines. When a long-format report is being written and the total percentage of CPU time for all routines shown exceeds this cutoff percentage, the report stops with an informational message. This option does not affect the content of short-format or user-de?ned reports. -S nstars Speci?es the number (decimal integer) of stars (asterisks) to be displayed on the right-hand side of report entries to represent 100%; default is 25. Setting this value to 0 effectively suppresses the asterisks. No practical limit exists to nstars, but specifying very large numbers may make the report unreadable. The value of this option also affects the number of stars printed in user-de?ned reports if that ?eld is selected in the report de?nition. 6.2.2 Instrumenting Your Code This section provides information about enabling the Perftrace feature and producing a Perftrace report. On all UNICOS systems, except Cray T90 and Cray C90 series systems, each run of Perftrace can generate data for each of four Hardware Performance Monitor (HPM) statistics groups. You can combine statistical data from different runs of the same program (using the cat command) and have the perfview command read this combined set of data. This feature allows perfview to make more accurate observations about your program’s hardware usage. The four HPM groups reported are as follows: • Execution summary that shows all operations • Hold-issue conditions by hardware unit 004–2182–003 143Guide to Parallel Vector Applications • Memory references by type, including con?icts • Instruction and vector operation by instruction grouping Note: Cray T90 and Cray C90 series systems generate data during one run. You can enable Perftrace with the CF90, Cray C++, and Cray C compilers. 6.2.2.1 CF90 Compiler Using the CF90 Fortran compiler, you can enable Perftrace for an entire program or for particular subprograms. This section describes how to process an entire program. See Section 6.6.1, page 157, for information about processing subprograms. To process an entire program, include -ef -lperf on the f90 command line. These options activate the tracing option of the compiler and include the Perftrace library routines during the load step. The following examples show the use of Perftrace with the f90 command. The ?le yourcode.f90 is Fortran 90 code. CF90 example: f90 -ef yourcode.f90 -lperf env PERF_GROUP=0 ./a.out perfview On all UNICOS systems, except the Cray T90 and Cray C90 series systems, perfview can perform better observations if the hardware statistics from all four HPM groups are gathered and combined. The following example generates four statistics ?les for the same program and then combines the ?les for report generation by perfview: f90 -ef -lperf pgm.f env PERF_GROUP=0 PERF_DATA=perf.0 ./a.out env PERF_GROUP=1 PERF_DATA=perf.1 ./a.out env PERF_GROUP=2 PERF_DATA=perf.2 ./a.out env PERF_GROUP=3 PERF_DATA=perf.3 ./a.out cat perf.[0-3] | perfview -LBuchM - > perf.report 6.2.2.2 Cray C++ and Cray C Compilers To enable Perftrace with the C and C++ compilers, include the -F -lperf options on the cc(1) command line for the C compiler and the CC(1) command 144 004–2182–003perfview [6] line for the C++ compiler; this applies to the whole executable program. In the following example, the Cray C++ compiler is used; pgm.c ?le is user code; and perf.report is the speci?ed output ?le: CC -F -lperf pgm.C ./a.out perfview -LBMuch > perf.report Note: If your code uses the setjmp(3) or longjmp(3) functions, Perftrace may fail or give misleading results. Perftrace keeps its own stack of called routines. If you jump out of a routine without its knowledge, Perftrace may apply timings to the incorrect routine or terminate abnormally. If your code executes recursive function calls, Perftrace may terminate abnormally due to internal stack over?ow. 6.3 Using the X Window System Interface You can produce perfview reports by submitting the Perftrace raw-format output to the perfview command. The following sections discuss how to use the X Window System interface to produce perfview reports. See Section 1.7, page 9, for more information on the X Window System interface. For information on producing reports with the perfview command line, see Section 6.2.1, page 136. To enter resource changes, use the class name of Perfview. The following example changes the width of the main text window to 900 pixels: Perfview*mainTextWindowWidth: 900 To change the default values of widget resources used in perfview, refer to the widgets by name or by class. The widget classes used in perfview are Form, Text, and List. Names and descriptions of widgets are as follows: Widget Description perfview Highest-level form for all perfview windows text Text widget that contains all ASCII report information File File menu button Reports Report menu button Graphs Graph menu button 004–2182–003 145Guide to Parallel Vector Applications Help Help menu button status Status line at the bottom of the window In addition to the standard X Window System resources, perfview uses the following application-speci?c resources for user customization: Resource Description mainTextWindowWidth Width of text and graphics window, expressed as the decimal number of pixels. Default is 570. mainTextWindowHeight Height of text and graphics window, expressed as the decimal number of pixels. Default is 400. Figure 26, page 147, shows the main perfview window. 146 004–2182–003perfview [6] Figure 26. Main perfview Window 6.3.1 File Menu File menu selections permit operations such as opening ?les, opening a demo ?le, and exiting perfview. Figure 27, page 148, shows the File menu. 004–2182–003 147Guide to Parallel Vector Applications Figure 27. File Menu 6.3.1.1 Open. . . Selection The Open... selection lets you open (load) ?les to be used by perfview. This selection creates an input panel on which you enter the name of the ?le to be opened. The input panel accepts shell wildcard conventions. 6.3.1.2 Open Demo File Selection The Open Demo File selection loads a sample raw ?le that is processed by perfview. 6.3.1.3 Capture Running Process Selection The Capture Running Process selection allows perfview to interactively capture Perftrace statistics from a running program that has been enabled for the Perftrace feature. 6.3.1.4 Export Data... Selection The Export Data... selection lets you choose the format and the ?le name for data saved from the display. You can use that ?le to format the data according to your own needs. The formats available are as follows: • CSV, or comma-separated • Tab-separated 148 004–2182–003perfview [6] • SYLK, a common spreadsheet format 6.3.1.5 Quit perfview Selection The Quit perfview selection displays a con?rmation panel on which you can con?rm your decision to exit. The con?rmation panel has the following buttons: Button Description Confirm Exits the tool and returns you to the operating system. Pressing RETURN has the same effect. Cancel Cancels the request to quit. To suppress display of the con?rmation panel, add the following command line to your .Xdefaults ?le: perfview*quitConfirm: never Note: If you add the con?rmation panel control command to your .Xdefaults ?le, remember to issue the xrdb(1) command to make your changes known to Perfview. 6.3.2 Select Report Desired: Menu The Select Report Desired: menu selections permit you to select the way reports will be viewed. Figure 28, page 150, shows the Select Report Desired: menu. 004–2182–003 149Guide to Parallel Vector Applications Figure 28. Select Report Desired: Menu 6.3.2.1 Observations Selection The Observations selection displays a special observations report. For data generated by Perftrace, the observations cover both the entire program and the top ?ve routines within the program. For data generated by the hpm command, the observations cover only the entire program. Items reported in the observations include the following: • Indication of the degree of vectorization of the program or routine • Discussion of ?oating-point operations that occurred • Identi?cation of possible bottlenecks in the program or routine • Recommendations to the user for possible speed improvements • Indication of whether a routine is a candidate for inlining 150 004–2182–003perfview [6] 6.3.2.2 Summary Selection The Summary selection displays a summary of signi?cant routines from Perftrace execution. This selection is not available for hpm or hpmall data. The summary contains the following: • Selected routines that used the most amount of CPU time • Selected routines that would be the best candidates for inlining 6.3.2.3 Show Routines Sorted by: Selection The Show Routines Sorted by: selection lets you choose the prede?ned routine report you want to see. These reports are available only with data generated by the Perftrace library. When you select the Show Routines Sorted by: selection, perfview displays a pull-down menu from which you can generate routine reports. The following selections will appear: Selection Description Time Displays traced routines sorted by the amount of CPU time used. The routines shown at the start of this report are those using the most CPU time. Floating-Point Operations/Sec Displays traced routines sorted by the number of ?oating-point operations per second. Routines that appear at the start of this report are those that achieved the highest mega?op rate. Inline Factor Displays traced routines sorted by a statistic indicating whether the routine is a candidate for inlining. Name Displays traced routines sorted alphabetically by routine name. Number of Calls Displays traced routines sorted by the number of calls made to the routines. Average Time per Call Displays traced routines sorted by average CPU time used per call. Another pop-up menu appears and you can select whether you want to see the routines sorted by average time per call, ascending, or descending order. If you choose the 004–2182–003 151Guide to Parallel Vector Applications ascending sort, the routines shown at the start of this report are those with the least CPU time per call and, therefore, the most likely candidates for inlining. Specific HPM Counter Displays a report showing all traced routines sorted by the value of a single HPM counter statistic. 6.3.2.4 Program Summary Selection The Program Summary selection is available only if the raw input to perfview is the output from the execution of the hpm command. The report summarizes selected information about the overall code performance for the program. 6.3.2.5 Full Program Report Selection The Full Program Report selection is available only if the raw input to perfview is the output from execution of the hpm command. All HPM statistics gathered for the program are shown. 6.3.2.6 Machine Workload Summary Selection The Machine Workload Summary selection is available only if the raw input to perfview is the output from execution of the hpmall command. The report summarizes selected information about the overall code performance for the machine workload during the time of sampling. 6.3.2.7 Full Machine Workload Report Selection The Full Machine Workload Report selection is available only if the perfview raw input is the output from the execution of the hpmall command. All HPM statistics gathered for the machine workload are shown during the sample period. 6.3.2.8 User-Defined Reports: Selection The User-Defined Reports: selection lets you choose and process user-de?ned reports. When you select the following options from this submenu, the appropriate actions will occur: 152 004–2182–003perfview [6] Selection Description Run Report(s) When this option is selected, and you have de?ned only one user report in the report de?nition ?le, the option is executed and the results are displayed in the text window. Change Report Descriptor Filename When this option is selected, another menu appears that lets you enter a new value for the ?le name to be read. This ?le is assumed to contain report descriptor records. Reset Report Descriptor Filename to Default When this option is selected, perfview resets the ?le name for reading report descriptors to the default ?le name. Reread Current Report Descriptor File When this option is selected, perfview tries to reread the currently de?ned report de?nition ?le. 6.3.2.9 Environment Selection The Environment selection displays a listing of information about the working environment for your original Perftrace, hpm, or hpmall execution, and current perfview execution. Information displayed includes the following: • Date and time of program execution • Name of executable ?le used • Serial number and clock speed of machine on which Perftrace was executed • Perftrace overhead statistics 6.3.2.10 Report Preferences. . . Selection When you select Report Preferences..., you will see an additional window appear on the screen. This window lets you alter the options that control the appearance of reports generated by perfview. Figure 29, page 154, shows the report preferences window. 004–2182–003 153Guide to Parallel Vector Applications Figure 29. perfview Report Preferences Window 6.3.3 Select Graph Desired: Menu The Select Graph Desired: menu selections let you view program information graphically. Figure 30, page 154, shows the Select Graph Desired: menu. Figure 30. Select Graph Desired: Menu 154 004–2182–003perfview [6] 6.3.3.1 Pie Chart Selection The Pie Chart selection causes perfview to display a pie chart that depicts the relative amount of CPU time spent in the ?ve most active routines in your program. You can also access this display by clicking on the pie chart icon near the top of the main window. The pie chart itself is also a menu. If you press the left mouse button while the cursor is inside of a pie slice for an active routine, a report will appear showing additional detail. The pie chart is the ?rst display that you will see when perfview starts up. 6.3.3.2 Bar Chart Selection The Bar Chart selection displays the percentage of CPU time spent in all of the routines in your program in bar chart format. You can also access this display by clicking on the horizontal bar chart icon near the top of the main window. The bar chart itself is also a menu. If you press the left mouse button while the cursor is inside of a pie slice for an active routine, a report will appear showing additional detail. 6.3.3.3 Text Report Selection The Text Report selection displays a table that gives a detailed report of the activity within a single module. This is the report you receive when you click on a module in the pie chart or bar chart display. You can also access this display by clicking on the text page icon near the top of the main window. 6.4 Using the ASCII Interface When running perfview in line-mode interactive, you can specify the generation of reports by entering a number, followed by RETURN (or ENTER). Available reports are shown on a menu. To change the option’s value, enter the number of the desired option, followed by RETURN (or ENTER). For example, entering the menu item number for User Reports causes generation of user-de?ned reports. If no active user-de?ned reports exist, you will receive an error message. If only one report is de?ned in the report-de?nition ?le, it will be executed immediately and you will see report output on your screen. If two or more user-de?ned reports exist, a new menu will be displayed showing their names. 004–2182–003 155Guide to Parallel Vector Applications The reports and options previously listed are available in interactive mode. Online help is available in line-mode interactive. Reports are displayed using a pager program. The default pager is more. If you want to use a different pager program, enter its command name in the PERFVIEW_PAGE environment variable. The name of the temporary ?le that contains report output is appended to the end of the command, and the command is executed. The PERFVIEW_PAGE environment variable can have up to 100 command-line options. The environment string can contain up to 500 characters. perfview does not perform special parsing of the command options. It is assumed that this pager command can be executed from a normal shell (that is, this command can be found on the default $PATH of the user). You also can change the default pager command interactively from the Options menu. If you prefer to view the reports by using the vi editor, enter the following commands: Standard (sh) and Korn (ksh) shells: $ PERFVIEW_PAGE=vi ; export PERFVIEW_PAGE $ perfview rawfile C (csh) shell: % setenv PERFVIEW_PAGE vi % perfview rawfile You can specify these environment variable values permanently, if desired, in your .profile or .login ?le. 6.5 Using vi Windows Use of vi windows assumes that the vi command can be executed from a normal shell (that is, vi can be found on your default $PATH). The vi editor windows may take a few moments to appear. Do not press the same mouse buttons repeatedly. Before logging out, close all vi windows. 156 004–2182–003perfview [6] 6.6 Additional Perftrace Features In addition to tracing procedure calls in entire programs, Perftrace provides the following features: • Processes selected subroutines • Provides the SETPLIMQ entry point • Recovers information from abnormally terminated programs • Can be controlled by environment variables 6.6.1 Selective Tracing Because the use of Perftrace incurs additional CPU overhead, you may want to limit the areas of your program that are measured. Tracing only selected routines, or sections of routines, can help lower the CPU overhead expended by Perftrace. The following sections describes some techniques for using selective tracing. 6.6.1.1 Using the FLOWMARK Subroutine The FLOWMARK subroutine lets you measure the performance of selected sections of code, such as loops, within a program unit. To use FLOWMARK, call it from your source program immediately preceding and following the code you want to measure. The ?rst invocation includes, as an argument, a name for the code block (to be used in Perftrace output). The second invocation uses 0 as an argument. You do not have to invoke Perftrace on the compiler command line to use FLOWMARK calls. The name used as an argument in the ?rst invocation of FLOWMARK is represented as a null-terminated character string. This is speci?ed in Fortran as a left-justi?ed Hollerith string of no more than 7 characters (’STRING’L). In C, it is speci?ed as a string within double quotation marks, such as "STRING". The name should be different from any routine names in your program. Although you may embed nonprinting characters in a name passed to FLOWMARK, their use is not encouraged. Embedding special characters (such as tabs or newlines) can make the reports generated by the perfview command unreadable. When you use FLOWMARK, Perftrace treats the marked code block as a distinct subroutine. The subroutine is given the name speci?ed in the ?rst call to FLOWMARK. The two calls to FLOWMARK are analogous to calling and returning from a subroutine consisting of your code block. 004–2182–003 157Guide to Parallel Vector Applications Perftrace statistics are accurate only if the granularity of the routines or marked areas are suf?ciently high. To use FLOWMARK to time a code region, ensure that the code region uses at least 50 microseconds per call. The following examples show how to use the FLOWMARK subroutine. Fortran example: ... CALL FLOWMARK (’PHANTOM’L) DO 10, I=1,10 ... 10 CONTINUE CALL FLOWMARK (0) ... C++/C example: main() { int zero = 0; void FLOWMARK(); ... ... FLOWMARK("phantom"); for (i = 0 ; i<0 ; i++) { ... /* loop performs desired work */ ... } FLOWMARK(&zero); } If you want to time a very small section of code, or a routine that uses very little time, you can isolate the code area as a subroutine if it is not already a subroutine. Then, call the routine many times (for example, 10,000). Place FLOWMARK calls around the entire loop, and divide the resulting statistics by the loop count (10,000 in this example). This technique also works well for Flowtrace, described in Chapter 3, page 67. Assume that the following source code is from the mysource.f ?le. program timer common /test/ data1,data2.... 158 004–2182–003perfview [6] c c initialize the common data here c call flowmark(’loop1’l) do 100 i=1,10000 call sub1 100 continue call flowmark(0) end subroutine sub1 common /test/ data1,data2.... c c c perform calculations on the common data c items c end Compile and execute mysource.f: f90 -ef mysource.f -lperf ./a.out perfview -L The data used in the routine to be timed is placed in common blocks. This action prevents the compiler from eliminating all calculations during the optimization phase. Perftrace is not speci?ed on the f90 command line because the FLOWMARK calls are suf?cient to load the correct libraries. To get the actual ?gures for the code fragment timed, divide the perfview statistics by the number of times the routine was called (10,000 in this case). For statistics averaged over time, such as mega?ops per second, you do not have to divide by the number of times the routine was called. Note: Routines that use less than 50 microseconds of CPU time per call will show inaccurate timings in perfview. Accuracy of reported times decreases as the average time per call decreases below 50 microseconds. 6.6.1.2 Using Other Methods The CF90 compiler handles Perftrace directives in the following ways: • If Perftrace is enabled on the command line (-ef for the f90 command line), you can selectively disable routines for Perftrace by placing the 004–2182–003 159Guide to Parallel Vector Applications following directive just after the ?rst statement of the routine (PROGRAM, FUNCTION, or SUBROUTINE). The disabling stays in effect until the next program unit (PROGRAM, FUNCTION, or SUBROUTINE) is encountered. !DIR$ NOFLOW • If Perftrace is not enabled on the command line, you can selectively enable routines for Perftrace by placing the following directive just after the ?rst statement of the routine (PROGRAM, FUNCTION, or SUBROUTINE). The enabling stays in effect until the next program unit (PROGRAM, FUNCTION, or SUBROUTINE) is encountered. !DIR$ FLOW To trace all routines except subroutine two, use the following source code. Enable Perftrace on the command line. Disabling is only in effect for subroutine two. Subroutines that follow have Perftrace enabled. Example 1: program main call one call two stop end subroutine one do 100 i=1,10000 ... ... 100 continue return end subroutine two !dir$ noflow do 200 i=1,30000 ... ... 200 continue return end To trace only subroutine one, but nothing else, use the following source code. Do not enable Perftrace on the command line. The enabling (indicated by 160 004–2182–003perfview [6] !dir$ flow) is in effect only for subroutine one. Subroutines that follow do not have Perftrace enabled. Example 2: program main call one call two stop end subroutine one !dir$ flow do 100 i=1,10000 ... ... 100 continue return end subroutine two do 200 i=1,30000 ... ... 200 continue return end 6.6.2 SETPLIMQ Subroutine - Reporting Each Call An entry point is present in Perftrace for the SETPLIMQ call that is also available in Flowtrace. Thus, if you are using Flowtrace, you do not have to change source code to run with Perftrace. This is a dummy entry point in Perftrace; calling it has no effect on the behavior of Perftrace. 6.6.3 perfdmp Command - Programs That Terminated Abnormally If your program terminates abnormally with Perftrace active and a core dump occurs (ensure that the ?le core is present in the current directory), you can use the perfdmp(1) command to recover partial statistics from the Perftrace run. Redirect stdout from perfdmp to a ?le, and execute perfview. See the perfdmp(1) man page for more information. 004–2182–003 161Guide to Parallel Vector Applications 6.6.4 Environment Variables Perftrace options are speci?ed by environment variables. An environment variable is speci?ed within the env(1) command, along with the command to be executed, as follows: • Except on Cray T90 and Cray C90 series systems, the following example changes the system hardware statistics group to 3 (instruction and operation counts): env PERF_GROUP=3 ./a.out • Except on Cray T90 and Cray C90 series systems, the following example changes the system hardware statistics group to 1 (Hold Issue conditions) and writes the raw data to a ?le called data.1 in the current directory. Usually, Perftrace would write the raw data to perf.data. env PERF_GROUP=1 PERF_DATA=data.1 ./a.out Table 8, page 162, lists the Perftrace environment variables. Table 8. Perftrace Environment Variables Variable Range Default Description PERF_DATA string perf.data Setting this variable to a string causes Perftrace to write the raw statistical data to the ?le name indicated in string, rather than to perf.data (default ?le name). If string is a dash (-) or stdout, the report is written to standard output. If string is stderr, the report is written to stderr. PERF_GROUP 0-3 0 Hardware monitor group to be used (Cray SV1 and Cray J90 systems only.) 0 = Execution summary 1 = Hold-issue conditions 2 = Memory activity 3 = Vector activity 162 004–2182–003perfview [6] Variable Range Default Description PERF_MAXCALL1 0- n 0 Throttle threshold value of the number of calls to any one routine. Default is that no throttle will be used. PERF_MAXTIME 1 0- n 50 Throttle threshold time in microseconds per call to any one routine. 6.6.4.1 Throttle Feature If the PERF_MAXCALL environment variable is set to a nonzero value, the throttle feature is enabled. Each time a routine is traced, its total call count will be checked against PERF_MAXCALL. When any routine is called more than PERF_MAXCALL times, and its average time per call (accumulated so far) is below the PERF_MAXTIME threshold, that routine will be "throttled." This means that Perftrace replaces all internal tracing code for that routine with NOOPs. The throttled routine ceases to have statistics gathered for it by Perftrace. Throttle action has two major consequences: • Because Perftrace is never entered again for a throttled routine, overhead can be lessened signi?cantly. • Remaining execution statistics for a throttled routine are gathered and accumulated into the statistics for the routines that call the throttled routine. Keep this fact in mind when reviewing statistics for such a caller. The throttle feature has the following limitations: • If a throttled routine resides in a segmented program, the NOOPs inserted may disappear when the segment is reloaded, thus nullifying the effect of the throttle. If you know which routines will be throttled, locate them in the root segment. • Throttling cannot be performed for a shared-text (read-only code) program. Typical values for the throttle variables are 10,000 for PERF_MAXCALL and 100 for PERF_MAXTIME. Using a value lower than 100 for PERF_MAXTIME can be self-defeating, because Perftrace uses about 20 microseconds per call for overhead. 1 These variables control throttling. See Section 6.6.4.1, page 163 for more information. 004–2182–003 163Guide to Parallel Vector Applications 6.7 Perftrace Statistics and Overhead Because the counters include Perftrace execution time, along with program execution time, counts are approximate even for programs that terminate normally. Perftrace subtracts its estimated times and counter values from default information, but some error can be expected (probably less than 1% over an entire code). Some performance counters vary from one program execution to the next. perfview allows the option of reporting unadjusted statistics. Each user call requires several thousand clock periods to be processed by Perftrace. The cost of running the tool can be estimated by multiplying this ?gure by the number of traced calls in your code. Then adjust the number of clock periods by the speed of your machine. This applies only to the cost of using the tool; the reported statistics are already adjusted in this way. 6.8 Caution about Underlying Parallelism Even though your program may not be compiled to run multitasked, some underlying libraries (for example, libsci.a) are set up to take advantage of extra available CPUs on your machine. Their execution in other CPUs may affect the integrity of the statistics generated by Perftrace. If in doubt about this phenomenon, run your program with the NCPUS environment variable set to 1 when using Perftrace. 6.9 Signal Handling in the Perftrace Library The Perftrace library gathers statistics for programs that terminate due to signals (without a core dump) by setting up certain signals during initialization. Usually, the signal catches are set before the ?rst traced routine is entered. When the signal occurs, Perftrace is entered. Perftrace then performs normal termination activities, such as writing the raw data ?le. This signal handling may interfere with the signal handling of the user program in the following ways: • If the user program catches signals, its signal code may overwrite the catches set by Perftrace. Therefore, if the signals are caught by the user program, and then cause termination, the perf.data ?le may be empty. • If the user program sets signals and then enters traced code for the ?rst time, Perftrace may overwrite the catches set by the user. In such a case, the program may not behave as expected if the signals occur. 164 004–2182–003perfview [6] • If a signal is already set to be ignored, Perftrace will not overwrite it. This behavior is necessary to allow correct execution of user programs with special programs such as nohup(1). Perftrace catches the following signals: Signal Description SIGHUP Hangup signal (signal 1) SIGINT Interrupt signal (CONTROL-C on the keyboard) (signal 2) SIGCPULIM CPU time-limit signal (signal 26) SIGTERM External kill (signal 15) Signals that cause a core dump are not caught, because Perftrace information can be recovered from the core ?le by using the flodump command. 6.10 perfview Report Output Statistics generated by Perftrace and the hpm and hpmall commands and displayed by perfview are not exact and should not be interpreted to be accounting information. The perfview command is very sensitive to the format of its input data. Raw input that does not conform to the correct format may cause perfview to terminate abnormally. The same is true for user report de?nitions; any input error is considered fatal. Certain command-line options interfere with options processed by the X Window System. If you have the DISPLAY environment variable set but want to generate reports by using command-line options or to run in line-mode interactive, you must specify -L on the command line. Not specifying -L in these situations can produce unpredictable results. 6.10.1 Command-line User-de?ned Reports Because Hardware Performance Monitor (HPM) counter data can contain over 40 different statistics for each routine, a single report program cannot display all of the information you may want. Therefore, perfview lets you de?ne your own variables and report line formats for display as User Reports. 004–2182–003 165Guide to Parallel Vector Applications When perfview is initiated, it looks for a ?le in $HOME/.perfvrc (or for the ?le speci?ed by the PERFVIEW_REPORT_FILE environment variable). If $HOME/.perfvrc does not exist, perfview assumes that you do not want to run user-de?ned reports. You must generate the report-de?nition ?le in ASCII format, usually by using a text editor. The ?le contains individual records of a ?xed format. 6.10.2 Command-line Observation The report generated by specifying perfview -B, which also is available with run-time menu selections in the X Window System and interactive interfaces, produces an observations report. Unlike the other perfview reports, which display objective, numerical data, this report analyzes the data. See Section 6.12, page 190, for an example of an observations report. To produce the observations report, perfview makes assessments based on the Perftrace or hpm data, and it suggests ways to improve your program. Because these suggestions are based on very limited information about your program, treat them with caution. If you are new to perfview or to UNICOS systems in general, however, these observations may provide some insights that will help you get started optimizing code. 6.11 User-de?ned Reports The perfview command lets you generate reports of your own design, showing both your program and its routines with only the variables you want to see, in the order you want them displayed in the report. You can also use these de?nitions to generate variables of your own choosing, calculated from the existing HPM statistics. The following sections contain speci?cations for user-de?ned reports. The following topics will be discussed: • Report ?le • Report records • Prede?ned variables • Processing of variables • Units designators 166 004–2182–003perfview [6] • Report generation • Full report de?nition ?le examples 6.11.1 Report File By default, user report de?nitions are read from the home directory of the user, from the .perfvrc ?le. You can override this ?le name in several different ways. First, before perfview tries to read $HOME/.perfvrc, it checks the PERFVIEW_REPORT_FILE environment variable. If this variable is de?ned, perfview uses it as the default report de?nition ?le. You can override the default by using the -F option with the perfview command. You also can alter the name of the ?le that contains report de?nitions during interactive processing for perfview. You generate the report de?nition ?le in ASCII format, usually by using a text editor. The ?le contains individual records of a ?xed format. 6.11.2 Report Records The record types are as follows: • Comment records • Marker records to start the user-variable de?nitions • User-variable de?nition records • Marker records to identify the report and start its ?eld de?nitions • Heading de?nition records for lines to be displayed at the top of a report • Report ?eld de?nition records The following sections discuss these record types. 6.11.2.1 Comment Records Comment records start with two (..) symbols in columns 1 and 2. Any other text may follow these symbols. The content of these records is ignored. Example: .. this is a comment, but could have been .. an unused report descriptor line 004–2182–003 167Guide to Parallel Vector Applications 6.11.2.2 Marker Records to Start User-variable De?nitions Marker records to start user variable de?nitions contain the string .variables, starting in column 1. If comments are placed after this string, you must separate them from the string by a space. Only one record of this type may exist. It is not required, but you may want to use it if you are de?ning your own report variables calculated from existing HPM statistics. Example: .variables (starts the variable de?nitions) If you use this marker, it must precede any report de?nitions. 6.11.2.3 User-variable De?nition Records User-variable de?nition records may take on two possible formats: • newvariable = oldvariable • newvariable = oldvariable op oldvariable The newvariable variable is a new user-de?ned variable name. It must not have been previously de?ned. The name of the new variable starts in column 1 of the record. All user variable names can contain any printable characters, except spaces. Case is signi?cant. You must use white space to separate each ?eld in the record from other ?elds. You must place the equals (=) symbol between the newvariable variable and the ?rst oldvariable variable. oldvariable may be any variable known to perfview, including both special prede?ned variables and any previously de?ned user variables. If the record ends with the ?rst oldvariable, the operation is assumed to be a copy of the oldvariable variable to the newvariable. If op is present, it implies an arithmetic operation of two oldvariable variables to yield a newvariable. op must be one of the following: * Multiply. + Add. 168 004–2182–003perfview [6] - Subtract second oldvariable from ?rst oldvariable. / Divide ?rst oldvariable by second oldvariable (divide by 0 yields 0). diff The absolute value of the difference of the two oldvariable variables. pos Subtract second oldvariable from ?rst oldvariable. If the result is negative, replace the result with 0. If the result is positive, leave it as is. max The maximum value of the two oldvariable variables. User-variable de?nition records end either at end-of-?le or at the occurrence of a report de?nition marker. The following example calculates the myvar variable as the total number of ?oating-point operations divided by the total amount of time spent in the routine (in clock periods). The resulting statistic would be ?oating-point operations (FLOPs) per clock period. myvar = flops / totavg The following example calculates the floperinst variable as the total number of ?oating-point operations divided by the number of instructions issued (HPM group 0, counter 0). The resulting statistic is the total FLOPs per instruction. floperinst = flops / g0c0 The following example calculates the number of scalar ?oating-point operations by subtracting the number of vector ?oating-point operations from the total number of ?oating–point operations. If the data was processed on a Cray J90 system, perfview checks to ensure that this counter was generated on a Cray J90 system. The resulting statistic could be a negative number. scalflop = flops - jg3c7 6.11.2.4 Marker Records to Identify a Report Marker records identify a report and start its ?eld de?nitions. You may de?ne up to 20 different reports in your de?nition ?le. Each set of report de?nitions is preceded by a record of this type. This record has the following format: .report name The characters .report must start in column 1. name must be a single ?eld, separated from .report by white space. name cannot contain white space. The name speci?ed must not have been previously de?ned in this speci?cation ?le. 004–2182–003 169Guide to Parallel Vector Applications These marker records, and the report ?eld de?nition records that follow them, are optional. If you include report de?nitions, they must follow the variable de?nitions (if any). All user-de?ned reports are optional, because perfview can generate shortand long-form reports of its own internal design. The contents of the data read from the raw-data ?le drive the formats of the reports. 6.11.2.5 Heading De?nition Records Heading de?nition records describe the headings you want placed at the beginning of each report. No practical limit exists on the number of heading lines that can be described. The report program displays the headings in the order read from the de?nition ?le. A heading de?nition record has the following format: .heading[m] The characters .heading must start in column 1. If you omit the optional m variable, this heading will always be printed and perfview uses the rest of this record as heading contents, starting with column 9. Or, for example, if the value of the m variable is j90, this heading will be printed only if the original Perftrace, hpm, or hpmall data was generated on a Cray SV1 or Cray J90 series system. If the heading is printed, perfview uses the rest of this record as heading contents, starting with column 10. m System on Which Original Data Was Generated c90 Cray C90 t90 Cray T90 j90 Cray J90 or Cray SV1 6.11.2.6 Report Field De?nition Records Each report ?eld de?nition record describes a ?eld of the report. These ?elds are displayed by the report program, left to right, shown in the order read from the de?nition ?le. The ?rst ?eld listed in any user report is always the name of the current routine. You do not have to specify this ?eld. Report ?eld de?nition records have the following format: variable header units format sort 170 004–2182–003perfview [6] The variable name starts in column 1. All following ?elds are separated by white space. The ?elds are as follows: Field Description variable Must be a previously de?ned variable name. Can either be a variable you de?ned in the .variables section, or a prede?ned variable known to perfview. Case is signi?cant. A list of prede?ned variables follows this section. header Text that appears in the ?eld heading record before any routines are listed by perfview. The width of this ?eld also is used to print the dashed lines below the ?eld header line. If this header ?eld contains spaces, it must be surrounded by quotation marks ("). When you use quotation marks, the header itself cannot contain any quotation marks. This ?eld may contain only printable characters, with the following single exception: If the ?eld contains only the characters, the ?eld is taken to be a new line. The printed header contains a new line at this point, as well as a dashed line. You must include this string in the printf format ?eld; otherwise, headers and data may be misaligned. The most common use of this special format is to generate multiline reports. You must ensure that such a report is readable. units A units designator. perfview can perform some calculations of the report variable just before it is printed. If you do not want perfview to alter the value being printed, specify asis in this ?eld. Several other units designators are available for your use. See Section 6.11.5, page 179, for more information on these units designators. format This is the printf(3) format that is used to display the variable in the report. This string is passed directly to the printf function. 004–2182–003 171Guide to Parallel Vector Applications If this ?eld contains blanks, you must enclose it with quotation marks ("). When quotation marks are used, there cannot be any quotation marks in the ?eld. Do not place any special C++/C-language escape sequences (such as a tab) in this ?eld. Most report variables are written as ?oating-point numbers. However, certain report variables are written with a ?xed format and you cannot override such formats. In these cases, you should specify dummy information for the format ?eld. There is one special case for the contents of this ?eld. If the ?eld contains only the characters, the data is taken to be a new line. The report print position will contain a new line at this point. You must include this string in the header contents ?eld; otherwise, headers and data printed may be misaligned. The most common use of this special format is to generate multiline reports. You must ensure that such a report is readable. sort (Optional) Speci?es whether this variable is used to sort the routines in the report. This ?eld can contain an optional decimal number. The number indicates whether this item will be used as a sort key in the report. The highest sort key is 1, with other keys following in numeric order. If the key is preceded by a dash (-), it is assumed that the sort will be by descending, rather than ascending, value. Default is ascending value. In the following example, the variable to be printed is the number of ?oating-point operations performed by the routine. flops "MEGAFLOPS" mpersec %9.2f -1 The heading for this ?eld of the report is MEGAFLOPS, a string that consists of 9 characters. The ?eld width of the printf format also consists of 9 characters. The unit descriptor (mpersec) tells perfview to divide the raw count of 172 004–2182–003perfview [6] FLOPS by the number of seconds this routine executed, and to divide the result further by 1,000,000. The optional sort indicator is set, so this is the primary key for the report listing; the key is descending. This ?eld contains the mega?op rate of the routine, and the routines are sorted by descending mega?op rate when the report is generated. In the following example, a single HPM counter is printed, the total number of machine instructions issued. The units indicator (asis) tells perfview not to alter the statistic before printing. The width of the heading is 18 characters, a value matched by the printf format designator. g0c0 "INSTRUCTIONS ISSUED" asis %18.0f This variable is printed as a ?oating-point number. Print format does not contain the fractional part. In the following example, the percentage of all CPU time for which this routine is responsible is printed. It is printed in a ?eld 6 characters wide. The heading for this report column is also 6 characters wide. routinepercent "RTNPCT" asis %6.2f In the following example, zero or more stars (asterisks) are printed to indicate the magnitude of the percentage of the whole program for which this routine is responsible. The printf format and the units designator are ignored. This variable should be printed as the last ?eld in the report line, because its length is not known until it is printed. stars "MAGNITUDE OF ROUTINE" asis dummy In the following example, one HPM counter is printed for the Cray C90 system. This counter is the total number of CPU memory references performed. The units indicator (asis) tells perfview not to alter the statistic before printing. The width of the heading is 14 characters, a value matched by the printf format designator. c4 "CPU MEMORY REFS" asis %14.0f This variable is printed as a ?oating-point number. The print format does not contain the fractional part of the number. 6.11.3 Prede?ned Variables Some variables are not usable with hpm or hpmall data. This section notes these variables. 004–2182–003 173Guide to Parallel Vector Applications Unless so indicated, all variables are treated as ?oating-point numbers. The following items are printed with numeric values in the report only if the raw-data ?le contains statistics from the appropriate counter group. If you have requested one of these values, and the desired counter group is not present in the raw data, the ?eld will be printed as all dashes (-). The variables tot0 through tot3 are printed with only numeric values in the report. If you have requested one of these values, and the desired counter group is not present in the raw data, the ?eld will be printed as all dashes (-). Variable Description tot Total CPU time accumulated on the Cray T90 or Cray C90 series system. This is the same as counter 0. For convenience, this number is also available as totavg. tot0 Total CPU time used by a routine when run under counter group 0. Cray SV1 and Cray J90 series systems only. tot1 Total CPU time used by a routine when run under counter group 1. Cray SV1 and Cray J90 series systems only. tot2 Total CPU time used by a routine when run under counter group 2. Cray SV1 and Cray J90 series systems only. tot3 Total CPU time used by a routine when run under counter group 3. Cray SV1 and Cray J90 series systems only. totavg Total CPU time used by a routine, averaged over all groups present in the raw data ?le. You also may designate total CPU time on Cray T90 and Cray C90 series systems. This variable is always available, since it is derived from the average of all counter groups present in the input data on Cray SV1 and Cray J90 series systems. flops Total number of ?oating-point operations performed by a routine. This is the total of adds, multiplies, and reciprocals. This variable will be printed with numeric data if data from counter 174 004–2182–003perfview [6] group 0 is present in the raw input data read by perfview and that data came from a Cray SV1 or Cray J90 series system. If the raw data does not contain group 0, this ?eld will be printed as all dashes (-). This value is always available for data from Cray T90 and Cray C90 series systems. vflops Total number of vector ?oating-point operations performed by a routine. This variable is printed with numeric data. If the raw data does not contain group 3, this ?eld will be printed as all dashes (-). This value is always available for data from Cray T90 and Cray C90 series systems. calls Total number of times this routine was called when the program was run with Perftrace active. (Not available for hpm or hpmall data.) callo Total number of times this routine called other traced routines when the program was run with Perftrace active. (Not available for hpm or hpmall data.) noop No variables are calculated. This variable lets you specify nondata items in the format ?eld for printf. The most common use is to allow printing of new lines in the report. This technique also may be used to annotate the report with ?xed text values when desired. You cannot use this variable to calculate other user-de?ned variables. throttle Throttle ?ag for this routine is printed. The value for the ?ag is ?xed. It will either be the string T, to indicate that the routine was throttled during its execution, or a blank. If a blank is used, it must be enclosed in quotation marks ("). You should specify the printf % option for a string of percentage signs. You cannot use this variable to calculate other user-de?ned variables. (Not available for hpm or hpmall data.) 004–2182–003 175Guide to Parallel Vector Applications routinepercent Percentage of total program CPU time for which this routine was responsible. A number from 0.0 through 100.0. (Not available for hpm or hpmall data.) accumpercent Total percentage of all routines displayed so far in the report, including the current routine. A number from 0.0 through 100.0. (Not available for hpm or hpmall data.) name Name of the current routine. For the totals line, this string is Totals. The totals line is the only report line printed for hpm or hpmall data. The perfview command prints this ?eld by using a ?xed format; therefore, the printf format ?eld is ignored. The algorithm used is as follows: perfview prints the name inside a 16-character ?eld. If the name is longer than 16 characters, it will print the whole name, follow it with a new line, and then print 17 blank characters. This new line and the additional blanks allow for proper alignment of ?elds across the page of the report. The ?rst ?eld of all user reports is the routine name, so you do not have to specify the name in your report ?eld de?nitions. This variable cannot be used to calculate user-de?ned variables. 1 The constant 1 (one). This variable is most commonly used in calculations of user-de?ned variables. 100 The constant 100 (hundred). This variable is most commonly used in calculations of user-de?ned variables. 1000000 The constant 1,000,000 (million). This variable is most commonly used in calculations of user-de?ned variables. stars Zero or more asterisks (*) are printed. Their number is calculated from the magnitude of the percentage of the total CPU time for which this routine is responsible. Default is to use 25 stars to 176 004–2182–003perfview [6] represent 100%. Because this ?eld varies in length, you probably should specify it as the last report ?eld. (Not available for hpm or hpmall data.) You cannot use this variable to calculate user-de?ned variables. inline The inline factor. This factor is calculated by perfview, to be equal to the total calls to a routine divided by the average time spent in a routine for each call (expressed in clock periods). Not available for hpm or hpmall data. cn A special variable format used to describe one HPM counter value for Cray T90 or Cray C90 series systems. The de?nition must start with the letter c. n This part of the variable name speci?es the counter number to use. It must be a decimal number from 0 through 31. Example: c1 -- counter 1 (Number of instructions issued.) [l]gncm A special variable format used to describe one HPM counter value for Cray SV1 or Cray J90 series systems. The de?nition contains three parts: l (Optional) If you omit l, it is assumed that this counter is valid for Cray SV1 and Cray J90 series systems. If you specify l with the letter j, this variable can be used only with Cray SV1 and Cray J90 series system data. If you want to use an individual counter, be sure to specify variables for each machine type. When the 004–2182–003 177Guide to Parallel Vector Applications report is generated, only the variable that is valid for the current machine type will be displayed. gn This part of the variable name speci?es the counter group number to use. You must specify the letter g; n is a digit from 0 through 3. cm This part of the variable name speci?es the counter number to use within the counter group. You must specify the letter c; m is a digit from 0 through 7. Examples: g0c0 Group 0, counter 0 (the number of instructions issued). This variable is the same for all machine types. It will always be printed and used in calculations, as long as group 0 data is present. g0c3 Group 0, counter 3 will be calculated and printed, without regard to the machine type that generated the original HPM data. The user must determine that this is the desired statistic. 6.11.4 Processing of Variables Some of the variables described in the preceding section are available only in certain HPM counter groups. You may refer to any variable or counter in your variable de?nitions and report ?eld de?nitions. The following rules apply when the report is generated: • If a speci?ed or derived counter is referenced in a report ?eld, and that value depends on the presence of a particular HPM counter group in the 178 004–2182–003perfview [6] raw input data, and the input data does not contain that counter group, the ?eld is still printed. However, the ?eld value is printed as all dashes (-). • If a speci?ed or derived counter is referenced in a report ?eld, and that value depends on the presence of input data from a speci?c machine type, and the raw data ?le is from a different machine type, the ?eld will not be printed. No space will be allocated in the printed line for the ?eld. If you want to print or derive counters that are different on the different machine types, be sure to include ?eld de?nitions for all types. Data for inappropriate machine type ?elds will be bypassed, and data for correct type ?elds will be printed. 6.11.5 Units Designators You may specify one of the following units designators in your report ?eld de?nitions: Designator Description persec After the speci?ed variable is calculated, perfview will additionally divide the variable by the number of CPU seconds (averaged across counter groups) used by this routine. Thus, if the variable is flops, the printed value shows ?oating point operations per second (FLOPS). mpersec Same as persec, but the value is divided by 1,000,000. This unit lets you display mega?ops. sec After the speci?ed variable is calculated, perfview converts it from clock periods to seconds by multiplying the value by the clock speed of the machine on which the program ran. It is assumed that the units of the variable to be displayed are clock periods. asis The speci?ed variable is printed with its current value. perfview makes no additional calculations. 004–2182–003 179Guide to Parallel Vector Applications pctotal This units designator is available only for certain prede?ned variables, such as flops, or individual HPM counters. perfview divides the variable by the total value of the same variable for all routines and then converts this result to a percentage. The ?nal value printed will be a number from 0.0 through 100.0. You can designate the following variables with the pctotal unit value: Any single counter (cn or [l]gncm) tot0 through tot3 totavg flops vflops calls callo inline You cannot use the following variables with this units designator: noop throttle routinepercent accumpercent name 100 1000000 1 stars Any user-de?ned variable 180 004–2182–003perfview [6] 6.11.6 Report Generation User reports are generated separately from those standard to perfview. You select user reports either from an interactive menu or on the command line, using the -U option. A user-de?ned report is generated in this order: 1. All headings are printed. 2. Header lines are generated that contain each user-de?ned ?eld. 3. A single dashed line is generated that matches your ?elds in width. 4. For every routine being reported, your report ?elds are printed in left-to-right order. The ?elds are written as one record for each routine. 5. After the last routine, a single barrier line (all === symbols) is written to match the width of the dashed line in 3. 6. A single report record is written that contains totals for the entire program. The report is listed with the name Totals. If input data is gathered from an execution of hpm or hpmall, only the Totals line is generated. The ?rst report ?eld is always the routine name. perfview allows 16 characters for this name. If the actual name length exceeds 16 characters, perfview will print the full name followed by a new line and 17 blanks. The routine name ?eld is always included at the beginning of the header, the dashed line, and each report data line. 6.11.7 Full Report De?nition File Examples The following sections contain examples of ?les used to drive the user-de?ned reports for perfview. 6.11.7.1 Example 1 This example generates a somewhat nonsensical user variable that is the sum of all group 0 counters. The persec units designator also appears. Certain counters are displayed only if the data read by perfview is from a Cray J90 system. .variables j = g0c0 + g0c1 004–2182–003 181Guide to Parallel Vector Applications Add the counters one at a time, because the syntax allows only two items to be added. j2 = j + g0c2 j3 = j2 + g0c3 j4 = j3 + g0c4 j5 = j4 + g0c5 j6 = j5 + g0c6 j7 represents all counters in group 0 added together: j7 = j6 + g0c7 Report de?nition record follows. This report is referenced under the name one: .report one The following headings appear at the top of this report: .heading ROUTINES SORTED BY MEGAFLOPS .heading WITH SPECIAL COUNTER CONTAINING ALL COUNTERS .heading ADDED TOGETHER .heading The ?rst ?eld is always the routine name. Print mega?ops next. This is the primary sort key for the routines, and sorting is by FLOPS, shown in descending order. flops MFLOPS mpersec %6.1f -1 Next, print the statistic generated by adding all of the group 0 counters: j7 " all added" asis %14.0f Now, print the number of times this routine is called: calls " CALLED" asis " %14.0f" Print the number of instructions issued in this routine. This counter is the same regardless of the mainframe type: g0c0 "INST ISSUED" asis %11.0f The following ?eld prints the number of ?oating multiplies (millions) that occurred per second. The following ?eld prints only if data is from a Cray J90 system: jg0c4 "FP MULT/SEC" mpersec %11.2f 182 004–2182–003perfview [6] The following shows report output when this command is run with data from counter group 0: % perfview -L -U one -F report ?lename perf.data ROUTINES SORTED BY MEGAFLOPS WITH SPECIAL COUNTER CONTAINING ALL COUNTERS ADDED TOGETHER ROUTINE NAME MFLOPS all added CALLED INST ISSUED FP MULT/SEC ---------------- ------ -------------- ----------- ----------- ----------- TWO 147.0 33096300 300 1384500 110.23 ONEV 97.5 16160804 300 1203300 97.49 ONES 11.0 43514796 300 21302948 11.00 NOTHING 3.1 38688 300 10165 2.32 $MAIN 0.1 181949 1 46064 0.03 ========================================================================== Totals (All Traced Routines) 46.5 92992544 1201 23946978 38.75 The following shows report output when perfview is run with data from counter group 3: ROUTINES SORTED BY MEGAFLOPS WITH SPECIAL COUNTER CONTAINING ALL COUNTERS ADDED TOGETHER ROUTINE NAME MFLOPS all added CALLED INST ISSUED FP MULT/SEC ---------------- ------ -------------- ----------- ----------- ----------- ONES ------ -------------- 300 ----------- ----------- TWO ------ -------------- 300 ----------- ----------- ONEV ------ -------------- 300 ----------- ----------- $MAIN ------ -------------- 1 ----------- ----------- NOTHING ------ -------------- 300 ----------- ----------- ========================================================================== Totals (All Traced Routines) ------ -------------- 1201 ----------- ----------- Because most of the data needed by this report is from counter group 0, but the raw data read is from counter group 3, the ?elds that need this information are ?lled with dashes (-). Because the primary sort is by mega?ops and the group 0 data is not available, the routines are sorted by CPU time, in descending order. 004–2182–003 183Guide to Parallel Vector Applications 6.11.7.2 Example 2 This example processes Cray C90 data. To get an idea of instruction ef?ciency, calculate a ?gure that is the number of instructions per ?oating-point operation (FLOP). The noop variable generates a text marker between the inline factor and the routine percentage. The printed headings show the origin, by system type, of input data. .variables De?ne instructions per FLOP for Cray C90 systems. 90inperflop = c1 / flops .report example Headings, including empty lines, are as follows: .heading .heading .heading ROUTINES SORTED BY VECTOR MEGAFLOPS The following heading line prints only for data from a Cray C90 system: .headingc90 RUN ON A CRAY C90 The following heading line generates a blank heading line: .heading The routine name is printed as the ?rst ?eld. Next, the vector FLOPS are printed as mega?ops. It is the primary sort key. vflops VMFLOPS mpersec %7.1f -1 Then, the instructions per FLOP are printed. Only one of these ?elds will print, depending on the data source. inperflop "INST/FLOP" asis %9.2f 90inperflop "INST/FLOP" asis %9.2f Finally, the inline factor for each routine is printed, as follows: inline "INLINE FACTOR" asis "%13.7E" The following ?eld does not print data; instead it prints a | symbol at this report ?eld position. The header line for this ?eld is blank. 184 004–2182–003perfview [6] noop " " asis "|" Next, the percentage of total CPU time for which this routine is responsible is printed. routinepercent "RTPCT" asis %5.1f Then, the total accumulated CPU time percentage so far in the report is printed. accumpercent "ACCUM PCT" asis %9.1f The following shows the output for this report setup when this command is executed with Cray C90 system data: % perfview -L -U example -F report ?lename perf.data ROUTINES SORTED BY VECTOR MEGAFLOPS RUN ON A CRAY C90 system ROUTINE NAME VMFLOPS INST/FLOP INLINE FACTOR RTPCT ACCUM PCT ---------------- ------- --------- ------------- - ----- --------- SPLIT 558.7 0.06 5.8205276E-09 | 2.6 2.6 TEST 0.1 41.01 1.1039356E-05 | 0.0 2.6 JOINED 0.0 4.20 1.5472419E-10 | 97.4 100.0 ================================================================== Totals (All Traced Routines) 14.5 0.00 1.3564412E-09 | 100.0 100.0 6.11.7.3 Example 3 In this example, there is a ?eld that requires group 3 counters, as well as ?elds that require group 0 counters. The routine name is printed at the end, as well as at the start, of the report lines. The noop variable is used to generate a text marker between the inline factor and the routine percentage. The printed headings show the origin, by system type, of input data. .report example No user-de?ned variables are needed. .heading ROUTINES SORTED BY VECTOR MEGAFLOPS The following heading line prints only for data from Cray T90 systems: .headingt90 RUN ON A CRAY T90 The following generates a blank heading line: 004–2182–003 185Guide to Parallel Vector Applications .heading The routine name is printed as the ?rst ?eld. Next, vector FLOPS are printed as mega?ops. This ?eld prints useful information only if group 3 counters are present in the raw data. This ?eld is used as the primary sort key in descending order. If the group 3 counters are not present in the raw data, the sort is by CPU time used, listed in descending order. vflops VMFLOPS mpersec %7.1f -1 Next, the inline factor for each routine is printed, as follows: inline "INLINE FACTOR" asis "%13.7E" The following ?eld does not print data, but instead prints a | symbol at this report ?eld position (the header line for this ?eld is blank): noop " " asis "|" Next, the percentage of total CPU time for which this routine is responsible is printed: routinepercent "RTPCT" asis %5.1f Then, the total accumulated CPU time percentage used so far in the report is printed. accumpercent "ACCUM PCT" asis %9.1f Last, print the routine name again. This is printed using a special format, so the format ?eld is a dummy value. name "NAME" asis dummy The following shows output for this report setup when this command is run with only group 0 counter data: % perfview -L -U example -F report ?lename perf.data ROUTINES SORTED BY VECTOR MEGAFLOPS RUN ON A CRAY T90 ROUTINE NAME VMFLOPS INLINE FACTOR RTPCT ACCUM PCT NAME ---------------- ------- ------------- - ----- --------- ---- ONES ------- 2.1147921E-03 | 70.5 70.5 ONES TWO ------- 7.0657097E-03 | 21.1 91.6 TWO 186 004–2182–003perfview [6] ONEV ------- 1.8746529E-02 | 7.9 99.5 ONEV $MAIN ------- 4.2393181E-06 | 0.4 99.9 $MAIN NOTHING ------- 1.4854918E+00 | 0.1 100.0 NOTHING ============================================================= Totals (All Traced Routines) ------- 2.3883851E-02 | 100.0 100.0 Totals (All Traced Because group 3 data is not available, sorting is by CPU time, in descending order. The following shows output for this report setup when run with only group 3 counter data: ROUTINES SORTED BY VECTOR MEGAFLOPS RUN ON A CRAY T90 ROUTINE NAME VMFLOPS INLINE FACTOR RTPCT ACCUM PCT NAME ---------------- ------- ------------- - ----- --------- ---- TWO 147.1 7.0741405E-03 | 21.1 21.1 TWO ONEV 97.8 1.8815773E-02 | 7.9 29.0 ONEV $MAIN 0.1 4.2301899E-06 | 0.4 29.4 $MAIN ONES 0.0 2.1164154E-03 | 70.5 99.9 ONES NOTHING 0.0 1.5077398E+00 | 0.1 100.0 NOTHING ============================================================= Totals (All Traced Routines) 38.8 2.3909932E-02 | 100.0 100.0 Totals (All Traced The routine sort is now correct, based on the available information. 6.11.7.4 Example 4 This example derives values from several different counters and groups: .variables The following is ?oating-point operations per instruction issued: fpinstru = flops / g0c0 cpu_mem_y = tg0c6 The following generates memory references per FLOP: memperflopy = cpu_mem_y / flops 004–2182–003 187Guide to Parallel Vector Applications The following generates number of various vector hold-issue counts per FLOP: hivr = g1c4 hivu = g1c5 hivm = g1c7 hivrperflop = hivr / flops hivuperflop = hivu / flops hivmperflop = hivm / flops This report is named two: .report two The heading for this report is one blank line. .heading The ?rst ?eld printed is the name of the routine. The second ?eld is the number of mega?ops achieved by the routine. flops "MFLOPS" mpersec %6.1f The third ?eld is the percentage of total ?oating-point operations for which this routine was responsible. This is the number of operations, not the rate at which FLOPS were executed. flops "FLOP PCT" pctotal %8.1f Now the statistic generated in the .variables section is printed. This is the number of ?oating-point operations performed per instructions issued. fpinstru "FLOPS/INSTR" asis %11.1f This ?eld is the number of memory references that occurred per second (millions). cpu_mem_y "MMEM/SEC" mpersec %8.1f This variable was calculated in the .variables section. The value is the number of memory references that occurred for every ?oating-point operation. memperflopy "MEM/FLOP" asis %8.1f This variable also was shown earlier. It is the number of clock periods spent holding-issue for vector registers that occurred for every ?oating-point operation. 188 004–2182–003perfview [6] hivrperflop "HIVR/FLOP" asis %9.1f This variable is the same as shown earlier, but holding issue for vector functional units. hivuperflop "HIVU/FLOP" asis %9.1f Note: No sort keys are listed with any of the ?elds. Therefore, the routines are sorted by CPU time used, in descending order. The following example shows output from this report setup when run with data from counter group 0 only: ROUTINE NAME MFLOPS FLOP PCT FLOPS/INSTR MMEM/SEC MEM/FLOP HIVR/FLOP HIVU/FLOP HIVM/FLOP ------------- ------ -------- ----------- -------- -------- --------- --------- ONES 11.0 16.7 0.1 33.2 3.0 --------- --------- TWO 147.0 66.7 8.7 110.3 0.8 --------- --------- ONEV 97.5 16.7 2.5 292.5 3.0 --------- --------- $MAIN 0.1 0.0 0.0 35.5 238.6 --------- --------- NOTHING 3.1 0.0 0.1 8.5 2.8 --------- --------- =============================================================================== Totals (All Traced Routines) 46.5 100.0 0.8 70.1 1.5 --------- --------- Because counter group 1 was not available, the hold-issue condition counters did not print numeric values. The following example shows output from this report setup when run with data from counter groups 0 and 1 only: ROUTINE NAME MFLOPS FLOP PCT FLOPS/INSTR MMEM/SEC MEM/FLOP HIVR/FLOP HIVU/FLOP ------------- ------ -------- ----------- -------- -------- --------- --------- ONES 11.0 16.7 0.1 33.2 3.0 0.0 0.0 TWO 147.0 66.7 8.7 110.3 0.8 0.6 0.6 ONEV 97.6 16.7 2.5 292.7 3.0 0.9 0.1 $MAIN 0.1 0.0 0.0 35.5 238.6 31.7 99.4 NOTHING 3.1 0.0 0.1 8.5 2.8 0.0 0.1 =============================================================================== Totals (All Traced Routines) 46.5 100.0 0.8 70.1 1.5 0.5 0.4 Now, the ?op count and the hold-issue counts are available; therefore, all report ?elds can be printed. 004–2182–003 189Guide to Parallel Vector Applications 6.12 perfview Reports The following examples show the report output generated when various options are used on the perfview command line. Example 1 shows a typical observations report (-B option). Example 1: OBSERVATIONS ABOUT YOUR PROGRAM ------------------------------- Whole Program ------------- Observations: This program achieved 15.6 million floating-point operations per second during its execution, of which 5.0 were performed in the vector units. This program appears to be dominated by scalar operations. This observation is based on a low ratio of vector operations to scalar operations. The vector/scalar floating-point operation ratio (0.5) provides confirmation that the program is dominated by scalar operations. Suggestions: Study this program to determine if any parts of its code can be vectorized. Routine FTREE [Rank:1] ------------------------------- Observations: This routine was responsible for 39.3% of the total program 190 004–2182–003perfview [6] CPU time. This is an important percentage of the whole program. This routine achieved 14.5 million floating-point operations per second during its execution, of which 0.0 were performed in the vector units. This routine appears to be dominated by scalar operations. This observation is based on a low ratio of vector operations to scalar operations. The vector/scalar floating-point operation ratio (0.0) provides confirmation that the routine is dominated by scalar operations. Suggestions: Study this routine to determine if any parts of its code can be vectorized. 004–2182–003 191Guide to Parallel Vector Applications Example 2 shows a typical report of all routines sorted by time (-u option). Example 2: PERFTRACE OF ALL ROUTINES SORTED BY TIME USED (DESCENDING) (CPU Times are Shown in Seconds) NOTE: Group 3 summary will follow group 0 summary Group 0 Counter Summary Name Called Time Avg Tim EX % ACM % Mmems Mflops -------- ------- -------- -------- ----- ----- ----- ----- FTREE 12 5.93E+01 4.94E+00 39.1 39.1 8.9 14.5 ********* TREEF 60000 5.89E+01 9.82E-04 38.9 78.0 5.1 8.5 ********* DTREE 12 1.39E+01 1.16E+00 9.2 87.2 3.8 13.7 ** GFORSN 60000 9.52E+00 1.59E-04 6.3 93.5 62.0 70.6 * NAYBOR 63 2.96E+00 4.70E-02 2.0 95.4 3.9 6.3 PLOTPT 3 2.18E+00 7.25E-01 1.4 96.9 8.2 2.1 INDEXX 189 1.96E+00 1.04E-02 1.3 98.1 4.5 1.2 GFORSA 60000 1.63E+00 2.71E-05 1.1 99.2 44.6 54.3 ========================================================== Totals 180460 1.52E+02 100.0 100.0 10.6 15.6 Group 3 Counter Summary Name Called Time Avg Tim Ex % SMips V I/L VMflops -------- ------- -------- -------- ----- ----- ----- ----- FTREE 12 6.02E+01 5.02E+00 39.3 45.9 0.0 0.0 ********* TREEF 60000 5.93E+01 9.88E-04 38.7 30.7 0.0 0.0 ********* DTREE 12 1.39E+01 1.16E+00 9.1 42.8 0.1 0.0 ** GFORSN 60000 9.78E+00 1.63E-04 6.4 4.7 12.7 68.5 * NAYBOR 63 2.97E+00 4.71E-02 1.9 33.7 0.0 0.0 PLOTPT 3 2.21E+00 7.36E-01 1.4 28.3 0.0 0.0 INDEXX 189 1.96E+00 1.04E-02 1.3 18.9 0.1 0.0 GFORSA 60000 1.66E+00 2.77E-05 1.1 11.0 24.3 51.8 192 004–2182–003perfview [6] Example 3 shows a typical report that shows the inline factor (-h option). Example 3: PERFTRACE OF ALL ROUTINES SORTED BY ’IN-LINE’ FACTOR (DESCENDING) (CPU Times are Shown in Seconds) (Factors Greater Than 1 Could Indicate Candidates for In-Lining) NOTE: Group 3 summary will follow group 0 summary Group 0 Counter Summary Name Called Time Avg Tim EX % ACM % Mmems Mflops "In-Line" Factor -------- ------- -------- -------- ----- ----- ----- ----- -------------- GFORSA 60000 1.63E+00 2.71E-05 1.1 1.1 44.6 54.3 18.79 GFORSN 60000 9.52E+00 1.59E-04 6.3 7.4 62.0 70.6 3.21 * ================================================================================ TREEF 60000 5.89E+01 9.82E-04 38.9 46.2 5.1 8.5 0.52 ********* GETUSED 19 1.25E-04 6.55E-06 0.0 46.2 5.9 4.4 0.02 FIXH 12 7.4 8E-05 6.24E-06 0.0 46.2 3.2 2.2 0.02 FTREE 12 5.93E+01 4.94E+00 39.1 85.4 8.9 14.5 0.00 ********* DTREE 12 1.3 9E+01 1.16E+00 9.2 94.5 3.8 13.7 0.00 ** NAYBOR 63 2.96E+00 4.70E-02 2.0 96.5 3.9 6.3 0.00 PLOTPT 3 2.18E+00 7.25E-01 1.4 97.9 8.2 2.1 0.00 INDEXX 189 1.96E+00 1.04E-02 1.3 99.2 4.5 1.2 0.00 Example 4 shows a typical report showing all details of performance statistics for one routine (-R option speci?ed, report generated by -M option). Example 4: PERFTRACE OF ALL ROUTINES SORTED BY MEGAFLOPS (DESCENDING) ROUTINE:ANGMOM Used 0.00% of all time, 0.00% accumulated so far. Called 6 times. Caller 0 times. Average time/call: 7.58E-04 (seconds) Group 0: CPU seconds : 4.82E-03 CP executing : 567139 004–2182–003 193Guide to Parallel Vector Applications Million inst/sec (MIPS) : 7.82 Instructions : 37656 Avg. clock periods/inst : 15.06 % CP holding issue : 91.86 CP holding issue : 520994 Inst.buffer fetches/sec : 0.01M Inst.buf. fetches: 48 Floating adds/sec : 50.08M F.P. adds : 241302 Floating multiplies/sec : 68.51M F.P. multiplies : 330090 Floating reciprocal/sec : 0.01M F.P. reciprocals : 30 I/O mem. references/sec : 0.01M I/O references : 71 CPU mem. references/sec : 112.57M CPU references : 542388 Floating ops/CPU second : 118.59M Group 1: CPU seconds : 4.49E-03 CP executing : 528894 Hold issue condition % of all CPs actual # of CPs Waiting on semaphores : 0.00 0 Waiting on shared registers : 0.00 0 Waiting on A-registers/funct. units: 2.29 12127 Waiting on S-registers/funct. units: 0.28 1482 Waiting on V-registers : 36.53 193220 Waiting on vector functional units : 44.90 237493 Waiting on scalar memory references: 0.01 54 Waiting on block memory references : 34.21 180948 Group 2: CPU seconds : 3.89E-03 CP executing : 457387 Inst. buffer fetches/sec : 0.01M total fetches : 48 Scalar memory refs/sec : 0.02M actual refs : 84 avg conflict/ref 3.60: actual conflicts : 302 I/O memory refs/sec : 0.00M actual refs : 0 avg conflict/ref ....: actual conflicts : 0 Block memory refs/sec : 139.55M actual refs : 542304 avg conflict/ref 0.77: actual conflicts : 416199 Vector memory refs/sec : 139.55M actual refs : 542304 Group 3: CPU seconds : 4.99E-03 CP executing : 587276 (octal) type of instruction inst./CPUsec actual inst. % of all inst. (000-017)jump/special : 0.21M 1062 2.81 (020-137)scalar : 3.79M 18930 50.17 (140-157,175)vector integer/log.: 0.01M 36 0.10 (160-174)vector floating point : 1.82M 9078 24.06 194 004–2182–003perfview [6] (176-177)vector memory : 1.73M 8622 22.85 type of operation ops/CPUsec actual ops avg. VL Vector integer&logical : 0.29M 1440 40.00 Vector floating point : 114.47M 571134 62.91 Vector memory : 108.69M 542304 62.90 Scalar (All) : 3.79M 18930 ----------------------------------------------------------------------------- ... ... ... 004–2182–003 195procview [7] The procview(1) command formats the output generated by the procstat(1) command, and can present the statistics in several formats. (procstat also can create a formatted report by itself, but this is more dif?cult to read than the reports created by procview.) The procstat(1) command monitors execution of any process or group of processes that can be invoked by one command. procstat generates statistics about I/O activity, process activity, and memory activity. A command is any executable program or shell script. This chapter discusses the following topics: • Process monitoring • Getting started with the procview command • Using the X Window System interface • Using the ASCII interface • Using vi windows • procview sample output 7.1 Process Monitoring The information that procstat generates always includes process start and exit and, by default, includes memory activity and I/O usage (including secondary data segment usage). The procstat command maintains I/O statistics for all processes executed at the same time as your program, including any processes initiated by a multitasked program. If procview cannot identify the origin of a certain process, it enters (unknown) in some ?elds. The procstat command is used in the same way as the time(1) command; the name of the program to be monitored (and any arguments) are put on the same command line and executed by procstat. The procview command works similarly to profview(1). You should consider the following when using the procstat command: 004–2182–003 197Guide to Parallel Vector Applications • The procstat command works on all UNICOS systems. • The procstat command works correctly with multitasked codes only if the NCPUS environment variable is set to 1 before the user program begins execution. • The procstat command incurs no signi?cant CPU overhead. ! Caution: The procstat command does not accept multistreaming programs. 7.2 Getting Started with the procview Command The procview command processes the raw-format data generated by the procstat command. procview can be run interactively, both using the X Window System interface and in line mode. procview presents an interactive menu when no command-line report option is included; otherwise, an output option can be speci?ed to redirect the report to a ?le. The following sections discuss how to use command-line options to produce procview reports. See Section 7.3, page 204, for information on producing procview reports using the X Window System interface. 7.2.1 procview Command-line Options Speci?cation of any of the following options causes procview to run in noninteractive mode. Report output is written to stdout, and procview terminates. The default is that procview runs interactively and prompts for report speci?cations. The most commonly used report options are -L, -Fu, and -Sn. procview [-L] [-F sort] [-S sort] [-P sort] [-c] [-z] [-E] [-O searchstring] [-H] [-export[_csv]] [-export_tabbed] [-export_sylk] [-a] [-x] [-l] [-w] [-y] [-n] [-X] [-e] [-V] raw?le -L Prevents execution of X Window System processing. Always specify this option when you want to generate a noninteractive report. Some of the procview command-line options also display options for the X Window System. You can use the -L option to force line-mode interactive when no other report-generation options are speci?ed. See Section 7.6, page 214, for more information. 198 004–2182–003procview [7] -F sort Generates one or more Fortran File Reports, with each report sorted according to the sort speci?cation value. sort must be one or more single letters combined. The letters, which may be uppercase or lowercase, take on the following values: n Sorts by ?le name u Sorts by Fortran unit number s Sorts by maximum ?le size i Sorts by I/O rate (bytes per second) r Sorts by I/O read rate (bytes per second) w Sorts by I/O write rate (bytes per second) t Sorts by thrash percentage (number of bytes moved divided by the ?le size) a Sorts by I/O wait time o Sort by time at ?rst open of ?le m Sorts by bytes moved (total I/O activity) d Sorts by bytes read e Sorts by bytes written c Sorts by asynchronous I/O wait time Example: -F nu generates two reports, one sorted by ?le name, the other sorted by the Fortran unit number. -S sort Generates one or more System File Reports, with each report sorted according to the sort speci?cation value. sort must be one or more single letters combined. The letters, which may be uppercase or lowercase, take on the following values: n Sorts by ?le name 004–2182–003 199Guide to Parallel Vector Applications c Sorts by system ?le descriptor number u Sorts by Fortran unit number s Sorts by maximum ?le size i Sorts by I/O rate (bytes per second) r Sorts by I/O read rate (bytes per second) w Sorts by I/O write rate (bytes per second) t Sorts by thrash percentage (number of bytes moved divided by the ?le size) a Sorts by I/O wait time o Sorts by time at ?rst open of ?le m Sorts by bytes moved (total I/O activity) d Sorts by bytes read e Sorts by bytes written y Sorts by asynchronous I/O wait time Example: -S nc generates two reports, one sorted by ?le name, the other sorted by the system ?le descriptor number. -P sort Generates one or more Process Reports, with each report sorted according to the sort speci?cation value. sort must be one or more single letters combined. The letters, which may be uppercase or lowercase, take on the following values: p Sorts by process ID number m Sorts by maximum memory used s Sorts by wall-clock starting time Example: -P pm generates two reports, one sorted by process ID number, the other sorted by the maximum memory used. 200 004–2182–003procview [7] -c Generates a chronological listing of all packets processed by procstat when the original program executed. -z Generates a listing that shows the numbers and types of all packets processed by procstat when the original program executed. -E Generates a report that shows the working environment for procstat and procview. Information shown includes date and time of run, machine serial number, and so on. -O searchstring Generates a long-form report of ?les with names that match the searchstring string given in this option. The search string is treated as a regular expression. See the ed(1) man page for more information about regular expressions. -H Writes all interactive help topics directly to stdout. -export[_csv] Analyzes the speci?ed data ?le and sends the results to stdout in a comma-separated format. This allows the data to be imported into other analysis tools or into word processors in the form of a table. -export_tabbed Analyzes the speci?ed data ?le and sends the results to stdout in a tab-separated format. This allows the data to be imported into other analysis tools or into word processors in the form of a table. -export_sylk Analyzes the speci?ed data ?le and sends the results to stdout in a SYLK format. This allows the data to be imported into common spreadsheet applications. -a Selects all reports noted in this section and writes their contents to stdout. This option does not include reports generated by options -O or -H. The reports run include -F m, -S m, and -P p. -X Forces the X Window System mode. If the program cannot initialize the X Window System, it will exit with a nonzero status. This option is 004–2182–003 201Guide to Parallel Vector Applications intended for use with special scripts. The option is not needed for normal use. -e Causes the X Window System mode of procview to use vi windows to display all reports. By default, reports are displayed using the X ASCII widget. Use of vi windows lets you edit and save the report output. -V Lists version number of procview, along with a copyright notice, on stderr. raw?le Speci?es the name of the raw-format output ?le from an execution of procstat, as speci?ed on the -R option. If raw?le is speci?ed as a dash (-), procview reads from the standard input. raw?le is a required operand. The following options control the format and content of various reports that procview generates. The options can be speci?ed for either interactive (X Window System or line mode) or noninteractive operation. If speci?ed for interactive operation. The options become the default for the interactive session. You can change option values during the interactive session. Option Description -x In non-Fortran reports, lists all ?les. By default, lists only ?les that had some I/O activity associated with them. -l Enables long-format reports. By default, procview generates short-format reports. In the short-format reports, signi?cant statistics are written on one line of output for each process or ?le. Long-format reports contain all details of procstat information. -w Causes procview to show I/O wait time in machine cycles. Default is hours:minutes:seconds.fraction. -y Causes sorting only by the speci?ed sorting key, such as that indicated by the -Fu option. Default is if the raw data ?le from procstat contains more than one process, the ?les are sorted in reports by the process number, followed by the speci?ed sorting key. -n Causes procview to print all reported numbers as is, even if this causes report output to be out of alignment. By default, procview displays numbers that over?ow their report ?elds by converting them to scienti?c notation. Thus, if a ?eld is 7 202 004–2182–003procview [7] characters wide and the value is 10,000,000, it is printed as "1.000E7." 7.2.2 Gathering Data with the procstat Command The procstat command produces information about process activity, working with standard UNICOS libraries. A report is written to the speci?ed ?le; in normal usage, statistical data is written in the raw format used by the procview command. By default, the procstat report includes all available statistics. Information about process start and exit is always produced. To disable I/O and memory statistics, use the -i or -m option. Following is an example of monitoring the program a.out, using f90 code: f90 yourcode.f90 procstat -R raw.data ./a.out procview raw.data Following is an example of monitoring the program a.out, with its arguments arg1 and arg2. File a.raw is speci?ed for the procstat output, and then as the input to procview. procstat -R a.raw ./a.out arg1 arg2 procview -L -Sn a.raw > a.report Note: If a program appears to run correctly by itself but fails when run under the procstat command, program handling of allocated memory may be faulty. The program may be getting memory by using the malloc(3) call and then improperly assuming that the new memory is cleared to 0. The procstat library support code may not always set heap memory to 0, even for the main section of a program. The procstat command has the following format: procstat [-R raw?le] [-r rpt?le] [-m] [-i] [-d] [-V] command -R raw?le Writes the raw?le ?le from which procview can create various reports. This is the recommended usage. You must specify either -R or -r, or both. -r rpt?le Writes a formatted report to rpt?le. 004–2182–003 203Guide to Parallel Vector Applications -m Reports on memory activity, as reported by the ssbreak(2) system call. This information is included by default; this option overrides the default of -i, unless that also is speci?ed. -i Reports on I/O activity, including activity of secondary data segments (SDS). This information is included by default. This option overrides the default of -m, unless that option also is speci?ed. -d Detailed I/O information is gathered on each process counted by the procstat command. This option can generate very large amounts of information, and should be used with caution. -V Lists the version of procstat on stderr, along with a copyright message. command Speci?es command to be monitored (for example, a.out). If this command is a shell script, procstat also monitors all progeny of the script. 7.3 Using the X Window System Interface The following sections discuss how to use the X Window System interface to produce procview reports. See Section 1.7, page 9, for more information on the X Window System. To enter resource changes, use the class name of Procview. For example, to change the width of the main text window to 900 pixels, enter the following line: Procview*mainTextWindowWidth: 900 To change the default values of widget resources used in procview, refer to the widgets by name or class. The widget classes used in procview are Form, Text, Command, Toggle, Scroll, and Label. Names and descriptions of the widgets are as follows: Widget Description procview Highest-level form for all procview windows text Text widget that contains all ASCII report information File File menu button 204 004–2182–003procview [7] Reports Report menu button Graphs Graph menu button Help Help menu button status Status line at bottom of window In addition to the standard X Window System resources, procview uses the following application-speci?c resources for user customization: Resource Description mainTextWindowWidth Width of text window, expressed as the decimal number of pixels. Default is 570. mainTextWindowHeight Height of text window expressed as the decimal number of pixels. Default is 400. Figure 31, page 206, shows the main procview window. 004–2182–003 205Guide to Parallel Vector Applications Figure 31. Main procview Window 7.3.1 File Menu File menu selections permit operations such as opening ?les, opening a demo ?le, and exiting procview. Figure 32, page 207, shows the File menu. 206 004–2182–003procview [7] Figure 32. File Menu 7.3.1.1 Open. . . Selection The Open... selection lets you open (load) ?les to be used by procview. This selection creates an input panel on which you enter the name of the ?le to be opened. The input panel accepts shell wildcard conventions. 7.3.1.2 Open Demo File Selection The Open Demo File selection loads a sample raw ?le that is processed by procview. 7.3.1.3 Export Data... Selection The Export Data... selection lets you choose the format and the ?le name for data saved from the display. You can use that ?le to format the data according to your own needs. The formats available are as follows: • CSV, or comma-separated • Tab-separated • SYLK, a common spreadsheet format 7.3.1.4 Quit procview Selection The Quit procview selection displays a con?rmation panel on which you can con?rm your decision to exit. The con?rmation panel has the following buttons: 004–2182–003 207Guide to Parallel Vector Applications Button Description Confirm Exits the tool and returns to the operating system. Pressing RETURN has the same effect. Cancel To suppress display of the con?rmation panel, add the following command line to your .Xdefaults ?le: procview*quitConfirm: never Note: If you add the con?rmation panel control command to your .Xdefaults ?le, remember to issue the xrdb(1) command to make your changes known to procview. 7.3.2 Reports Menu The Reports menu selections permit you to select the way reports will be viewed. Figure 33, page 208, shows the Reports menu. Figure 33. Reports Menu 7.3.2.1 Files Selection The Files selection contains additional menus that let you generate reports that show various ?les processed by your program. 208 004–2182–003procview [7] Menu Description Fortran Only (Short/Long Report) Lets you generate a report that displays all Fortran ?les, whether or not they show any I/O activity. Files, such as stdin or stdout, that are not processed by the Fortran I/O library do not appear in this report. You are given the option to generate either a short report or a more detailed long report. All User Files (Short/Long Report) Lets you generate a report that displays all user ?les, whether or not they are processed by Fortran. System ?les (such as stdin or stdout) appear. You are given the option to generate either a short report or a more detailed long report. Select Filename Lets you generate a detailed report that displays one or more user ?les that have names that match a search string you have entered. 7.3.2.2 Processes Selection The Processes selection lets you see either a short report or a detailed report of all processes that initiated during your procstat session. Appropriate information about each process is shown. The reports that are generated are as follows: • Sorted by process ID number • Sorted by maximum memory used • Sorted by start time 7.3.2.3 Miscellaneous Selection When you choose the Miscellaneous selection, procview shows you another menu that lets you select the type of report you want to see. These report types are placed in the "Miscellaneous" category, because they are not necessarily processes or ?les. 004–2182–003 209Guide to Parallel Vector Applications Selection Description Procstat Packet Types Shows a report that contains counts of packet types received by the procstat shell program. Chronological Packets Shows either a short or detailed (long) report that contains the contents of each packet received by the procstat shell program, in the order received, during program execution. 7.3.2.4 Environment Selection This selection shows a report that contains information about the original program execution under procstat, as well as information about this procview execution. 7.3.2.5 Report Preferences. . . Selection When you select Report Preferences..., an additional window appears that lets you alter the options that control the appearance of reports. Figure 34, page 210, shows the report preferences window. Figure 34. procview Report Preferences Window 210 004–2182–003procview [7] 7.3.3 Graphs Menu The Graphs menu lets you view program information graphically. Figure 35, page 211, shows the Graphs menu. Figure 35. Graphs Menu 7.3.3.1 Pie Chart Selection The Pie Chart selection causes procview to display a pie chart that depicts the relative amount of I/O time spent in the ?ve most active routines in your program. You can also access this display by clicking on the pie chart icon near the top of the main window. The pie chart itself is also a menu. If you press the left mouse button while the cursor is inside of a pie slice for an active routine, a report will appear showing additional detail. The pie chart is the ?rst display that you will see when procview starts up. 7.3.3.2 Bar Chart Selection The Bar Chart selection displays the percentage of I/O time spent in all of the routines in your program in bar chart format. You can also access this display by clicking on the horizontal bar chart icon near the top of the main window. The bar chart itself is also a menu. If you press the left mouse button while the cursor is inside of a pie slice for an active routine, a report will appear showing additional detail. 004–2182–003 211Guide to Parallel Vector Applications 7.3.3.3 Text Report Selection The Text Report selection displays a table that gives a detailed report of the activity within a single module. This is the report you receive when you click on a module in the pie chart or bar chart display. You can also access this display by clicking on the text page icon near the top of the main window. 7.3.3.4 Generate X/Y Plots Selection When you select Generate X/Y Plots, a control-panel window appears that lets you select and control the data to be plotted by the xgraph program. Section 7.3.3.2, page 211, shows the control panel window. Figure 36. procview Plotting Control Panel 7.4 Using the ASCII Interface When running procview in line-mode interactive, you can specify the generation of reports by entering a number followed by RETURN (or ENTER). The available reports are shown on a menu. Entering the number for options changes the menu to show alterable options. To change the value of the report, enter the number of the desired option, followed by RETURN (or ENTER). 212 004–2182–003procview [7] The reports and options previously listed are available in interactive mode. Online help is available in line-mode interactive. Reports are displayed using a pager program. The default pager is more. If you want to use another pager program, change the PROCVIEW_PAGE environment variable. The name of the temporary ?le containing the report output will be appended to the end of this command, and the command executed. The PROCVIEW_PAGE environment variable can have up to 100 command-line options, and the environment string can contain up to 500 characters. procview does not perform special parsing of command options. It is assumed that this pager command can be executed from a normal shell (that is, this command can be found on the default $PATH of the user). You also can change the default pager command interactively from the Options menu. If you prefer to view the reports with the vi editor, enter the following for the standard (sh) or Korn (ksh) shell: $ PROCVIEW_PAGE=vi ; export PROCVIEW_PAGE $ procview raw?le Enter the following for the C (csh) shell: % setenv PROCVIEW_PAGE vi % procview raw?le You can specify these environment variable values permanently, if desired, in your .profile or .login ?le. 7.5 Using vi Windows Use of vi windows assumes that you can execute the vi command from a normal shell (that is, vi can be found on your default $PATH). The vi editor windows may take a few moments to appear. Do not press the same mouse buttons repeatedly. Before logging out, close all vi windows. 004–2182–003 213Guide to Parallel Vector Applications 7.6 procview Sample Output The procview command is very sensitive to the format of its input data. Raw input that is not from the execution of procstat could cause procview to terminate abnormally. Certain command-line options interfere with options processed by the X Window System. If you have the DISPLAY environment variable set and want to generate noninteractive reports or to run in line-mode interactive, you must specify -L on the command line. In the X Window System mode, procview performs data graphing using the xgraph command. xgraph cannot process all options or data types. If a graph does not appear on the screen, check the main interactive window for error messages generated by xgraph. The following examples show these reports: • Sort by ?le name (procview -Fn) • Sort by I/O rate by using long format (procview -Fi -l) • Sort by process ID number (procview -Pp) • Sort by chronological order (procview -c) In example 1, sort is by ?le name. Example 1: PROCSTAT FILE REPORT Showing Only Fortran Files (Sorted by File Name) (Process ID is: 31756) Unit I/O Bytes Fortran I/O Avg Byte Avg I/O Rate Filename Num Method Processed Statements per Stmt (Megabytes/Sec) -------- ---- ------ ---------- ----------- -------- -------------- file10 10 Fmt/S 142768 202 706.8 48.657 file11 11 Fmt/S 142768 202 706.8 48.938 file12 12 Fmt/S 142768 202 706.8 47.698 file13 13 Fmt/S 142768 202 706.8 46.848 file14 14 Fmt/S 142768 202 706.8 60.733 file15 15 Fmt/S 0 0 0.0 0.000 file16 16 Fmt/S 142768 202 706.8 60.571 214 004–2182–003procview [7] file17 17 Fmt/S 142768 202 706.8 55.788 file18 18 Fmt/S 142768 202 706.8 54.129 file19 19 Fmt/S 142768 202 706.8 60.757 file20 20 Fmt/S 142768 202 706.8 52.998 =================================================================== Total Files = 11 1427680 2020 706.8 53.192 In example 2, sort is by I/O rate by using the long format. Example 2: PROCSTAT FILE REPORT Showing Only Fortran Files (Sorted by File I/O Rate) Fortran Unit Number 19 File Name file19 Command Executed ./a.out Date/Time at Open 03/12/91 15:33:53 Date/Time at Close 03/12/91 15:33:54 System File Descriptor 5 Type of I/O sequential formatted File Structure text File Size 110000 (bytes) Total Data Transferred 142768 (bytes) Fortran I/O Count of Real Statement Statements Time ------------ ---------- -------------- READ 100 .0028 WRITE 100 .2085 REWIND 1 .0004 ENDFILE 1 .0003 CLOSE 1 .0002 706.8 Bytes transferred per Fortran I/O statement 95.54% Of Fortran I/O statements did not initiate a system request System I/O # of # Bytes # Bytes Wait Time (Clock Periods) 004–2182–003 215Guide to Parallel Vector Applications Function Calls Processed Requested Max Min Total ------------ ------- --------- --------- --------- --------- --------- Read 1 32768 32768 20721 20721 20721 Write 8 110000 110000 207474 10460 370917 Seek 3 n/a n/a 3359 3315 10013 Truncate 2 n/a n/a 7023 5009 12032 System I/O Avg Bytes Percent of Average I/O Rate Function Per Call File Moved (MegaBytes/Second) ------------ ----------- ---------- ------------------- Read 32768.0 29.8 263.565 Write 13750.0 100.0 49.427 Seek n/a n/a n/a Truncate n/a n/a n/a In example 3, sort is by process ID number. Example 3: PROCSTAT FILE REPORT Showing All Processes Run Under Procstat (Sorted by Process ID #) Note: only user files that moved data are counted. Command Process Process Start Max Mem # Usr # Fortran Number Date/Time (octal) Files Files ---------------- ------- ----------------- -------- ------ --------- /a.out 31756 03/12/91 15:33:52 314766 12 11 ========================================================================== Total Processes: 1 12 11 In example 4, sort is by chronological order. Example 4: PROCSTAT SPECIAL REPORT Showing All Packets as Received From Procstat Procstat was executed on 03/12/91, at 15:33:52, on machine serial #1405, a CRAY J90. 15:33:52.0000 Initialization of Process 31756 216 004–2182–003procview [7] Argument Values = ./a.out 15:33:52.0017 File Open for Process 31756 File Desc # 2, Name:stderr 15:33:52.0458 File Open for Process 31756 File Desc # 5, Name:file10 15:33:52.0462 Fortran File Open for Process 31756 Fortran Unit #10, Name:file10 15:33:52.0473 Memory Request for Process 31756 Memory Requested = 11000 (octal) 15:33:52.2524 File Trunc for 31756 File Desc #5, Name:file10 New File Size: 110000 (bytes)...... 004–2182–003 217profview [8] The profview(1) command creates one or more reports from the raw output generated by the prof(1) command (using the -x option). The reports created are based on the Pro?ling feature, and indicate the relative amount of execution time used by individual parts of your program (represented as segments of program memory). Study of Pro?ling output lets you optimize areas in your programs that demand the most execution time. You can use Pro?ling commands (profview and prof) with any program that runs under the UNICOS operating system, regardless of the language. The profview command produces ?ner-grained results than those produced by flowview. This chapter discusses the following topics: • The Pro?ling feature • Getting started with the profview command • Gathering data with the prof command • Instrumenting your code • Using the X Window System interface • Using the ASCII interface • Using vi windows • Usage considerations • profview sample output 8.1 The Pro?ling Feature You should consider the following information when using the Pro?ling feature: • Pro?ling works with multitasked codes. • Pro?ling requires reloading, and recompilation is recommended. • Pro?ling incurs no signi?cant CPU overhead, but its use does require additional memory. 004–2182–003 219Guide to Parallel Vector Applications • Pro?ling gives poor results when used with segmented programs. If a program must be segmented, place the routines that you are interested in Pro?ling in the root segment. The following steps should be followed when using the Pro?ling feature: 1. Load your program with the libprof.a library. 2. Execute your program. During this execution, the operating system samples the program address (P) register of your program at regular intervals. During normal termination, the Pro?ling library writes the raw counts of addresses to a ?le. 3. Execute the prof command, which merges the symbols present in your original executable program with the statistics gathered during execution. Usually, you will instruct prof to generate raw-format output, and you will send this output to a ?le. 4. Execute the profview command, which can report Pro?ling information either in an interactive mode or in a noninteractive mode. A typical report includes a line for each bucket or routine (a bucket re?ects a location in your program at which CPU time was being spent), including a histogram bar and execution statistics, both as an absolute count and as a percentage. To terminate Pro?ling and generate data, your program must be able to catch the following signals: SIGHUP, SIGINT, SIGTERM, and SIGCPULIM. If the user’s code also catches these signals, Pro?ling may not terminate correctly. For most purposes, the best way to read prof output is by using the profview utility, which can be used interactively or noninteractively, with reports directed to a ?le. The display from profview shows the number of hits for each bucket, including a bar graph. Spikes on the bar graph indicate code where the most time is spent. If your program was loaded with the prof library, a system routine records the address, at regular intervals, of the instruction currently being executed. Addresses are grouped in buckets. Bucket size is selectable; each time the monitor reads an address within a particular bucket, a hit is recorded for that bucket. Because of the time interval between samples, prof results involve probability. At worst, an address could be executed many times but never sampled. The 220 004–2182–003profview [8] current load on the system also can affect monitoring. Variables speci?ed in the env command change the prof library parameters to control the number of samples (for example, PROF_CLK_TCK selects the time interval, and PROF_WPB selects the bucket size). The profview command also provides a selection that lets you display percentages in the form of a con?dence interval. The prof and profview utilities give useful information for multitasked programs. The system shows the relative times spent in routines, whether or not your program is multitasked. The information given is based on an aggregate of all execution time and is not detailed by task. If you need detailed information about parallel ef?ciency, use the atexpert command, described in Section 2.2.1, page 23. The microtasking and Autotasking systems alter your code and generate new routine names, as well as new numbers for lines and statements. prof reports these altered symbols, rather than the original symbols in your program. The profview command displays statistics gathered by the prof command. These statistics are not exact. To determine the con?dence levels of the statistics, use the profview command with the -rc option. 8.2 Getting Started with the profview Command The profview command creates one or more reports from the raw output generated by the prof command (using the -x option). profview can be run interactively, using the X Window System interface and in line-mode. profview presents an interactive menu when no command-line report selection is included; otherwise, an output selection can be speci?ed and the report output redirected to a ?le. See the prof(1) man page for more information. You can produce a Pro?ling report by using these methods: • Submit the prof command raw-format output to the profview command. • Generate a ?nal report by using the prof command. This report is not as ?exible as the one produced by profview, but it may be preferred by some users. The following sections describe how to use the profview and prof command-line options to generate reports. See Section 8.3, page 235, for information on producing profview reports using the X Window System interface. 004–2182–003 221Guide to Parallel Vector Applications 8.2.1 profview Command-line Options Speci?cation of any of the following options causes profview to run in noninteractive mode. Report output is written to stdout, and profview terminates. The profview command line is as follows. profview [-L] [-h] [-D] [-c] [-A] [-m] [-d] [-y] [-z] [-a] [-B] [-b] [-i] [-H] [-export[_csv]] [-export_tabbed] [-export_sylk] [-v mode] [-s] [-p threshold] [-t totpct] [-r opt] [-n nstars] [-o conf] [-C] [-X] [-e] [-V] raw?le -L Prevents execution of X Window System processing. Always specify this option when you want to generate a report by using a command-line selection. You also can use -L to specify line-mode interactive when no other report-generation options are speci?ed. -h Lists routines and their hit counts, sorted by percentage of hits for the entire program. The ?rst few lines usually show the routines that used the most time. -D Lists symbols and their hit counts (in routines with nonzero hit counts). Routines are sorted by percentage of hits within the program; within each routine, symbols are listed in address order. Each symbol’s hit count is the sum for all buckets up to the next symbol. If debugging symbols were not enabled when a module was compiled, this report will show only (No symbol) as a designator for each line within these modules. -c Reports environment information for original user code execution, the prof run, and the current profview session. This information includes date and time of the user and prof run, bucket size, start and end address of the areas pro?led, and so on. -A Enables all report options listed in this section, except the -H option; reports are generated to stdout. 222 004–2182–003profview [8] -m Generates a summary of routines and symbols that shows 10% or more of total CPU activity (you can change the threshold). This report provides a quick look at performance information. -d Lists buckets (by address) and their hit counts (in routines with nonzero hit counts). Buckets are not combined for symbols; therefore, one symbol can be listed with many buckets. See -D for combined information. If debugging symbols were not enabled when a module was compiled, this report will contain only octal bucket addresses. To relate a particular bucket to a statement in your source code, you need a listing of the generated code. For Fortran, to generate source and generated code listings use the -egs compiler options. For Cray C++ or Cray C, use the selection -hlisting compiler options. -y Lists all symbols and their addresses, in address order. Both the compiler and the loader generate symbols. Statistical information is not provided. -z Lists all modules and their addresses. The loader generates these names for the program run. The listing is in address order. Statistical information is not provided. -a Lists routines that had nonzero hit counts, with percentages, sorted in alphabetic order. -B Lists buckets and their hit counts, with symbols from compilers and loader. Buckets are sorted by bucket address. -b Lists buckets and their hit counts. This is a raw report, sorted by bucket address. No matching symbols are shown. This is the only report that can be generated if the executable ?le (a.out) of the original user execution contains no compiler-generated or loader-generated symbols. -i Lists a histogram of the frequency of bucket hits, listed by number of hits per bucket. This report helps you determine whether the pro?ling hits 004–2182–003 223Guide to Parallel Vector Applications were spread uniformly over the code, or highly concentrated in only a few buckets (or, more important, modules). Large codes often show a wide spread of activity at the bucket level. -H Writes all online help topics directly to stdout. -export[_csv] Analyzes the speci?ed data ?le and sends the results to stdout in a comma-separated format. This allows the data to be imported into other analysis tools or into word processors in the form of a table. -export_tabbed Analyzes the speci?ed data ?le and sends the results to stdout in a tab-separated format. This allows the data to be imported into other analysis tools or into word processors in the form of a table. -export_sylk Analyzes the speci?ed data ?le and sends the results to stdout in a SYLK format. This allows the data to be imported into common spreadsheet applications. -X Forces X Window System mode. If the program cannot initialize the X Window System, it will exit with a nonzero status. This selection is intended for use with special script. It is not needed for normal use. -e Causes the X Window System mode of profview to use vi windows to display all reports. The default is that the reports are displayed on the screen by using the Xaw Text Widget. Use of vi windows lets you edit and save the report output. -V Lists version number of profview, along with a copyright notice. raw?le Speci?es the name of the raw-format output ?le produced from an execution of prof using the -x option. If raw?le is speci?ed as a dash (-), 224 004–2182–003profview [8] profview reads from the standard input. raw?le is a required selection. The following selections control the format and content of various reports generated by profview. The selections may be speci?ed for either interactive (X Window System or line mode), or noninteractive operation of profview. If speci?ed for interactive, the options become the default for the interactive session. These options can be changed during the interactive session. Option Description -v mode Controls annotation of the symbols that are given more verbose explanations in the reports generated by profview. mode must be one of the following letters: l Annotates library names if their names are present in an internal table of commonly used libraries. (Default.) n No annotation is tried. a Annotates all symbols when possible. -s All-symbols ?ag. Displays all symbols in a routine in the detail reports (-D, -d). By default, only symbols where activity occurred (and certain selected symbols) are shown. -p threshold Speci?es a decimal integer that indicates the percentage of activity to use as the minimum for inclusion of any module or symbol in the summary report (speci?ed by the -m option). -t totpct Speci?es a decimal integer that indicates the percentage of total activity to use as the cutoff for terminating the detail reports (-D, -d). When the total percentage of all modules exceeds this number, the report will be terminated. Specifying 100 for this value allows all modules to be listed. Specifying 0 restricts the report to only the module with the greatest activity. -r opt Indicates the type of information to display on the right-hand side of any report line that 004–2182–003 225Guide to Parallel Vector Applications contains some activity. opt must be one of the following letters: s Asterisks (stars); default. c Con?dence intervals. t Estimated CPU time. n None (nothing shown on the right). -n nstars Speci?es the number of asterisks (stars) to be displayed on the right-hand side of report entries to represent 100%; default is 25. In noninteractive mode, -n nstars speci?es the number of stars only when you also specify stars by using the -rs option or use the default. In interactive mode, -n nstars sets the default. Setting this value to 0 effectively suppresses the asterisks. No practical limit exists, but specifying very large numbers may make the reports unreadable. -o conf Speci?es the degree of con?dence desired. conf must be 90, 95, or 99; default is 95. In noninteractive mode, -o conf speci?es the con?dence level only when you also request con?dence-level information by using the -rc option. In interactive mode, -o conf sets the default. -C Retains the module structure of C++/C programs that could contain many functions embedded in a single module. Default is that profview splits individual functions from C++/C programs so that they appear to be separate modules. 8.2.2 Gathering Data with the prof Command To gather Pro?ling data, use the prof command after you execute a program that was loaded with the prof library. If you want to produce a ?nal report by using prof, see Section 8.2.2.2, page 228. The prof utility takes information from the following resources: • An executable ?le; default is a.out. 226 004–2182–003profview [8] • A ?le written by the prof library during execution of a program; default is prof.data. Note: The program represented in the data ?le is the most recent program that was loaded with the prof library and executed; this is not necessarily the current a.out ?le. You must ensure that these agree. See the caution in Section 8.2.5, page 234. You can specify the form of the output written by prof. For most purposes, the most usable form is speci?ed by the -x option; this output can be read by the profview command. 8.2.2.1 prof Command-line Options The prof command line is as follows: prof [-x] [-f] [-l] [-m work?le] [-V] [-p procid] [-r] [-F core?le] [subjprog] -x Prints output in a raw form that the profview command or tools (such as awk) can use. This option is required for use with profview. -f Speci?es that prof continue execution even if an error occurs during reading of the a.out (executable) ?le. Usually, if an error occurs during reading of the a.out ?le, prof terminates execution with an error message written to stderr. When an error occurs during reading of the prof.data ?le, prof always terminates abnormally. -l Suppresses use of compiler debugging symbols. The only level of detail shown is on the module boundary. -m work?le Speci?es the input ?le for this command. The default is ?le prof.data; this is the ?le in which pro?ling data was written by the prof library during program execution. -V Displays the version number of prof, along with a brief copyright notice. 004–2182–003 227Guide to Parallel Vector Applications -p procid Causes prof not to read prof.data work ?le (see the -m option for information about the work ?le) but, instead, to capture the pro?ling statistics from a running program. The procid value is the decimal value for the process ID of the running program. That program must have been loaded with the libprof.a library before statistics capture can occur. You cannot specify this selection with the -m or -r options. -r Causes prof not to read the prof.data work ?le (see the -m option for information about the work ?le) but, instead, to recover the pro?ling statistics from the core dump of a program that had terminated abnormally. The abnormally terminated program must have been loaded with the libprof.a library before statistics recovery can occur. Default ?le name for the core dump is core. To change this name, use the -F option. You cannot specify the -r option with the -p or -m options. -F core?le Speci?es an alternative name for the core ?le to be recovered when the -r selection is speci?ed. Default is core. subjprog Speci?es the name of your executable program to be pro?led. This program should have been executed before the prof command, after being loaded along with the prof library. Default is a.out. 8.2.2.2 prof Final Reports In addition to generating raw format output for postprocessing, the prof command can generate a ?nal report. The following examples write a report to the pgm.report ?le. Example 1: Summary by routine f90 -l prof pgm.f ./a.out prof -st a.out > pgm.report 228 004–2182–003profview [8] Example 2: Detailed report with ?lter f90 -G1 -l prof pgm.f ./a.out prof -H1 a.out > pgm.report 8.2.2.3 prof Command Reports Output from the prof command shows seven columns, giving the following information: Column Description Label The label used by prof to identify the bucket. Address Address of the label or an associated bucket. Bucket Octal address of the ?rst word of the bucket. Secs Estimated total time. Number of hits multiplied by the time interval. Default is 512 microseconds. Hits Number of hits for this bucket. ptime% The prof time percentage. Percent of hits for this bucket within the total hits for this program. This ?gure is of more interest than rtime% because it omits hits that are not part of the program. The total for comparison is typically 10% to 20% shorter than the total for rtime%. rtime% Run-time percentage. Percent of hits for this bucket within the total number of hits that prof monitored. This ?gure is of interest primarily for pro?les of parts of programs. (far right) Bar graph that shows column 6, ptime%. The prof command lists routine names rather than entry points. Only the routine names and symbols that contain active code as their labels are listed. When prof displays a compiler-generated symbol, as opposed to a library routine, it concatenates that symbol with the routine name. A symbol generated by the user or by the compiler is represented in the following form: routine.S.symbol 004–2182–003 229Guide to Parallel Vector Applications Line number locations are represented as follows: routine.L.number 8.2.2.4 Symbols that the prof Command Uses The symbols generated by the prof command conform with the latest compilers. In particular, the CF90, Cray C++, and Cray C compilers mark areas that are part of Autotasking code. The following entities are obtained from tables generated by the CF90, Cray C++, and Cray C compilers, and they are used in prof and profview reports: Entity Description (No symbol) Beginning address for the routine, before any executable code. Also appears when debugging symbols are not enabled during program compilation. Ex.routine Marks the entry point of a routine. This symbol marks C functions at their entry points. This symbol marks all Fortran subroutines, functions, and ENTRY names. See the text following this list for a discussion of double initial letters. Px.routine Marks the start of the routine’s executable code. All activity before this symbol is housekeeping and linkage code for the routine. See the text following this list for a discussion of double initial letters. Lx.61 Refers to the speci?ed line (61, in this example) in your source program. See the text following this list for a discussion of double initial letters. Sx.10A Entry to outer loop DO 10. See the text following this list for a discussion of double initial letters. Sx.10B Exit from outer loop DO 10. Sx.10C, Sx.10D, and so on Entry to, and exit from, a loop nested within another loop when both loops end on label 10; another nesting level would use 10E and 10F, and so on. Sx.10A1, Sx.10A2 Indicates loop unrolling. The loop is replaced by a long copy and selective short copy in which the 230 004–2182–003profview [8] long copy equals n iterations of the loop. If the trip count is an even multiple of n, 10A1 is the entry to the long copy and no short copy exists. If the trip count divided by n has a remainder, 10A1 is the short copy, and 10A2 is the long. Sx.43uA, Sx.43uB Represents the top and bottom of a DO loop that is not numbered. This type of loop would use the DO...ENDDO type of notation. For this type of DO loop, the number in the symbol (43, in this example) represents the line number within the program at which the loop starts. If your program is multitasked, the beginning of the symbol name may contain two letters rather than one. The second letter has the following meaning: M Master code S Slave code U Unitasked code Thus, the symbol LU.150 indicates line 150 of your program, and the code executed is in the unitasked version of the routine. 8.2.2.5 Using prof Environment Variables You can control the prof library by using environment variables entered on the env command before the name of your program. Example: env PROF_WPB=1 ./a.out The prof library environment variables follow, listed in the most often used order. Values are assumed to be decimal, unless a leading 0 is added. Variable Description PROF_CLK_TCK The shortest sample interval. This is the interval in microseconds at which the program will be interrupted to gather statistics. The value is prevented from falling below a value calculated from the number of available CPUs multiplied by 100 microseconds; the calculated number becomes the default minimum sample interval. Thus, if you are running on a machine with two CPUs, 004–2182–003 231Guide to Parallel Vector Applications you cannot specify a PROF_CLK_TCK of less than 200 microseconds. The normal default interval is 512 microseconds (1953 Hz), or 100 times the number of available CPUs, whichever is higher. Setting the NCPUS environment variable overrides the number of available CPUs, and it affects the preceding calculation. PROF_WPB Words per bucket; default is 4. Lets you control the granularity of your pro?le and compare pro?les based on different bucket sizes, for a check on the statistics generated. Using the smallest bucket size possible is desirable (minimum is 1) but the smaller the bucket size, the greater the amount of space taken up in user memory and by the prof.data ?le. PROF_SADDR Starting address; the lowest address that the prof library will monitor. Default is 0. To specify an octal value, use a leading 0. PROF_EADDR Ending address; the highest address that the prof library will monitor. Default is the program’s highest executable address. To specify an octal value, use a leading 0. Setting this value reduces the amount of memory needed to perform Pro?ling. If necessary, you can determine an appropriate setting for this value from the size(1) command (the ?rst number given is the size of executable text) by studying either the load map for your program or a previous execution of prof. Usually, you do not have to alter either PROF_SADDR or PROF_EADDR. The amount of heap memory needed to hold the buckets depends on the values of PROF_SADDR, PROF_EADDR, and PROF_WPB. PROF_DATA By default, the library writes the pro?le data to the prof.data ?le in the current directory. If you want to have the data written to another ?le, specify the name of that ?le by using PROF_DATA. 232 004–2182–003profview [8] You also should specify this alternate ?le name by using the -m option on the prof command line. 8.2.3 Instrumenting Your Code You can enable the Pro?ling feature from the CF90, Cray C++, Cray C, and loader command lines. The following sections provide examples of the command lines and itemize prof library variables. The profview command is very sensitive to the format of its input data. Raw input that does not contain symbols restricts reporting to only the raw bucket report (as speci?ed by the profview -b option). 8.2.4 Compiler Debug Symbols This section shows which compiler selections allow the generation of debug symbols. The prof and profview commands use information generated by the compiler’s debug selection. For instance, the routine name shown in the listing may include an additional symbol, which helps pinpoint the exact location of the executing code. The amount and type of information generated varies by compiler. Table 9, page 233, lists the options to the compiler commands that allow compiler-generated symbols to be displayed. Table 9. Compiler Debugging Information Language Command Option Description CF90 f90 -G1 Good information; partial optimization enabled. Very little impact on code ef?ciency. (10% to 20% slowdown possible.) Cray C, Cray C++ cc, CC -Gn All line numbers present; no optimization. -Gp All key line numbers present; partial optimization. (10% to 20% slowdown possible.) 004–2182–003 233Guide to Parallel Vector Applications 8.2.5 Examples This section contains examples of command-line sequences to enable Pro?ling. Each example performs the same functions. The integrated f90, CC, or cc command compiles your program using the debugging options, then loads the compiled program by using the prof library. a.out executes the loaded program. The bucket size is set at 1 word. (If the second line were replaced by ./a.out, the default setting of 4 words per bucket would be used.) When you enter profview as shown, it is used interactively, such as with the X Window System interface. ! Caution: Make sure the prof command reads the executable ?le, run to generate Pro?ling statistics, not the data ?le from a previous run (or from a different program). To ensure that the correct ?le is read, the prof command checksums the executable ?le and compares this checksum with the checksum calculated when your program was run. When these checksums do not match, prof issues an error message. Fortran 90 example: f90 -G1 yourcode.f90 -lprof env PROF_WPB=1 ./a.out prof -x a.out >pgm.prof profview pgm.prof Cray C example: cc -Gp -lprof -lsci pgm.c env PROF_WPB=1 ./a.out prof -x a.out >pgm.prof profview pgm.prof With Cray C or Cray C++ examples, ensure that if you specify the C library on your cc or CC command line (with the -lc operand), you put this option after the speci?cation of the prof library (-lprof). Correct example: cc -Gp -lprof -lsci -lc pgm.c Incorrect example: cc -Gp -lc -lprof -lsci pgm.c 234 004–2182–003profview [8] The incorrect example executes to completion without error, but no Pro?ling data is written to the prof.data ?le. 8.3 Using the X Window System Interface The following sections discuss how to use the X Window System interface to produce profview reports. See Section 1.7, page 9, for more information on the X Window System interface. For information on producing reports using command-line options, see Section 8.2.1, page 222. Certain command-line options interfere with options that the X Window System processes. If you have the DISPLAY environment variable set, but want to generate reports by command-line options or run in line-mode interactive, you must specify the -L option on the command line. If you omit -L in these situations, unpredictable results can occur. To enter resource changes, use the class name of profview, which is Profview. For example, to change the width of the main text window to 900 pixels, enter the following command line: Profview*mainTextWindowWidth: 900 To change the default values of widget resources used in profview, refer to the widgets by name or class. The widget classes used in profview are Form, Text, and List. Names and descriptions of the widgets are as follows: Widget Description profview Highest-level form for all profview windows text Text widget that contains all ASCII report information File File menu button Reports Report menu button Graphs Graph menu button Help Help menu button status Status line at bottom of window In addition to the standard X Window System resources, profview uses the following application-speci?c resources for user customization: 004–2182–003 235Guide to Parallel Vector Applications Resource Description mainTextWindowWidth Width of text and graphics window expressed as the decimal number of pixels. Default is 570. mainTextWindowHeight Height of text and graphics window expressed as the decimal number of pixels. Default is 400. scrollbarWidth Width of the scroll bars included with the graphical displays expressed as the decimal number of pixels. Default is 14. Figure 37, page 237, shows the main profview window. 236 004–2182–003profview [8] Figure 37. Main profview Window 8.3.1 File Menu File menu selections permit operations such as opening ?les, opening a demo ?le, and exiting profview. Figure 38, page 238, shows the File menu. 004–2182–003 237Guide to Parallel Vector Applications Figure 38. File Menu 8.3.1.1 Open. . . Selection The Open... selection lets you open (load) ?les to be used by profview. This selection creates an input panel on which you enter the name of the ?le to be opened. The input panel accepts shell wildcard conventions. 8.3.1.2 Open Demo File Selection The Open Demo File selection loads a sample raw ?le that is processed by profview. 8.3.1.3 Capture Running Process Selection The Capture Running Process selection allows flowview to interactively capture Flowtrace statistics from a running program that has been enabled for the Flowtrace feature. 8.3.1.4 Export Data... Selection The Export Data... selection lets you choose the format and the ?le name for data saved from the display. You can use that ?le to format the data according to your own needs. The formats available are as follows: • CSV, or comma-separated • Tab-separated 238 004–2182–003profview [8] • SYLK, a common spreadsheet format 8.3.1.5 Quit profview Selection The Quit profview selection displays a con?rmation panel on which you can con?rm your decision to exit. The con?rmation panel has the following buttons: Button Description Confirm Exits the tool and returns to the operating system. Pressing RETURN has the same effect. Cancel To suppress display of the con?rmation panel, add the following command line to your .Xdefaults ?le: profview*quitConfirm: never Note: If you add the con?rmation panel control command to your .Xdefaults ?le, remember to issue the xrdb(1) command to make your changes known to profview. 8.3.2 Select Report Desired: Menu The Select Report Desired: menu selections permit users to select the way reports are viewed. Figure 39, page 240, shows the Select Report Desired: menu. 004–2182–003 239Guide to Parallel Vector Applications Figure 39. Select Report Desired: Menu 8.3.2.1 Summary Selection The Summary selection displays a summary of signi?cant pro?le information. The summary contains the following items: • A listing of modules that used more than a given percentage of all user CPU time; default is 10%. To change this value, use the Options selection. • A partial listing of symbols from within those modules previously listed. The only locations from within the routines that are listed are those with a CPU proportion that exceeds the threshold percentage. 8.3.2.2 Modules: Selection When you choose the Modules: selection, profview displays a pull-down menu that has the following selections: 240 004–2182–003profview [8] Selection Description Active Modules Sorted by Hits Displays all active modules sorted by the amount of hits in descending order. Hit counts re?ect the amount of execution activity in a particular area of your program. The ?rst few lines of this report show the most active modules. Active Modules Sorted by Name Displays all active modules sorted by module name in alphabetical order. All Modules Sorted by Name Displays all modules that were loaded with your program, sorted by module name in alphabetical order. All Modules Sorted by Address Displays all modules that were loaded with your program, sorted by module parcel address in ascending memory order. 8.3.2.3 Details Within Modules: Selection When you choose the Details Within Modules: selection, profview displays a pull-down menu that has the following selections: Selection Description Grouped by Symbol Displays the total of all buckets and hits for each compiler-generated symbolic location. This method is the most useful because it displays activity in a way that most closely relates to your original source code. Grouped by Bucket Displays all bucket locations within the modules. If you are an experienced profview user, you may want to select this method because the level of detail can be as small as one word of machine instructions. No totaling occurs with this method. 004–2182–003 241Guide to Parallel Vector Applications 8.3.2.4 Symbols: Selection The Symbols: selection lets you choose a listing of all symbolic locations present in original program execution. These locations include both user code and library routines. The symbolic locations are generated by both the loader (module names), and selectively, by the compilers used to generate the user modules (symbolic names). This report does not contain statistical information. When you choose the Symbols: selection, profview displays a pull-down menu that has selections letting you view all symbols, sorted by the following items: Item Description Symbol Name Displays a list of all symbols, sorted alphabetically by module name, and then by symbol name. Symbol Address Displays a list of all symbols, sorted by their location in memory. 8.3.2.5 Buckets: Selection The Buckets: selection lets you choose raw bucket reports. These reports are based solely on the data written by prof, and they re?ect the contents of Pro?ling buckets that contain nonzero values. When you choose the Buckets: selection, profview displays a pull-down menu that has the following selections: Selection Description With Symbols Displays all buckets for which there was activity during program execution. If prof assigned a symbol to that bucket, any symbolic location associated with each bucket is displayed. Buckets are sorted by memory location. No Symbols Displays all buckets for which there was activity, but no symbolic information. This report is similar to a raw prof dump. Buckets are sorted by memory location. 242 004–2182–003profview [8] Histogram Displays a report of the bucket hits, grouped in a histogram. You can use this report to determine the concentration of hits. 8.3.2.6 Raw Bucket List Selection The Raw Bucket List selection displays all of the buckets for which there was activity during program execution. 8.3.2.7 Raw Bucket Histogram Selection The Raw Bucket Histogram selection generates a report of bucket hits, grouped in a histogram. You can use this report to determine the concentration of hits. Hit counts re?ect the amount of execution activity in a particular area of your program. 8.3.2.8 Environment Selection The Environment selection displays a listing of information about the working environment for your original program execution, original prof run, and current profview execution. Information displayed is as follows: • Date and time of program execution • Date and time of prof execution • Name of executable ?le used • Total number of hits for the program 8.3.2.9 Report Preferences. . . Selection When you select Report Preferences 0, you will see an additional window appear on the screen. This window lets you alter the selections that control the appearance of the various reports generated by profview. Figure 40, page 244, shows the report preferences window. 004–2182–003 243Guide to Parallel Vector Applications Figure 40. profview Report Preferences Window 8.3.3 Select Graph Desired: Menu The Select Graph Desired: menu lets you view program information graphically. Figure 41, page 244, shows the Select Graph Desired: menu. Figure 41. Select Graph Desired: Menu 244 004–2182–003profview [8] 8.3.3.1 Pie Chart Selection The Pie Chart selection causes profview to display a pie chart that depicts the relative amount of CPU time spent in the ?ve most active routines in your program. You can also access this display by clicking on the pie chart icon near the top of the main window. The pie chart itself is also a menu. If you press the left mouse button while the cursor is inside of a pie slice for an active routine, a report will appear showing additional detail. The pie chart is the ?rst display that you will see when profview starts up. 8.3.3.2 Bar Chart Selection The Bar Chart selection displays the percentage of CPU time spent in all of the routines in your program in bar chart format. You can also access this display by clicking on the horizontal bar chart icon near the top of the main window. The bar chart itself is also a menu. If you press the left mouse button while the cursor is inside of a pie slice for an active routine, a report will appear showing additional detail. 8.3.3.3 Memory Graph Selection The Memory Graph selection displays CPU use by memory location. A separate window lets you manipulate the display. You can also access this display by clicking on the vertical bar chart icon near the top of the main window. 8.3.3.4 Text Report Selection The Text Report selection displays a table that gives a detailed report of the activity within a single module. This is the report you receive when you click on a module in the pie chart or bar chart display. You can also access this display by clicking on the text page icon near the top of the main window. 8.4 Using the ASCII Interface When running profview in line-mode interactive, you must specify the generation of reports by entering a number followed by RETURN (or ENTER). The available reports are shown on a menu. To change the option’s value, enter the number of the desired option, followed by RETURN (or ENTER). 004–2182–003 245Guide to Parallel Vector Applications The reports and options previously listed are available in interactive mode. Online help is available in line-mode interactive. Reports are displayed using a pager program; the default pager is more. If you want to use another pager program, change the PROFVIEW_PAGE environment variable. The name of the temporary ?le that contains the report output is appended to the end of this command, and the command is executed. The PROFVIEW_PAGE environment variable can have up to 100 command-line options, and the environment string itself can contain up to 500 characters. profview does not perform special parsing of the command options. It is assumed that this pager command can be executed from a normal shell (that is, this command can be found on the default $PATH of the user). You also can change the default pager command interactively from the Options menu. If you prefer to view the reports by using the vi editor, enter the following commands. Standard (sh) or Korn (ksh) shells: $ PROFVIEW_PAGE=vi ; export PROFVIEW_PAGE $ profview rawfile C (csh) shell: % setenv PROFVIEW_PAGE vi % profview rawfile You can specify these environment variable values permanently, if desired, in your .profile or .login ?le. 8.5 Using vi Windows Use of vi windows assumes that you can execute the vi command from a normal shell (that is, vi can be found on your default $PATH). The vi editor windows may take a few moments to appear. Do not press the same mouse buttons repeatedly. Before logging out, close all vi windows. 246 004–2182–003profview [8] 8.6 Usage Considerations This section presents several subjects for more effective or specialized usage of the prof command: • Strategies for best results • Reporting of selected parts of programs 8.6.1 Strategies for Best Results As already shown, the most effective way of using prof typically includes the compiler’s debugging selection and a report generated by profview. You can improve prof results in the following ways: • Use the smallest bucket size possible, so that no routine or section of a routine is measured as part of another. However, additional memory is needed to hold bucket counts; this requirement increases as bucket size decreases. For most purposes, the default of 4 words per bucket is acceptable. • Theoretically, the more samples taken, the more accurate the results. However, after a point, taking more samples increases cost without signi?cantly improving accuracy. You may study the con?dence intervals associated with a desired degree of con?dence (default is 95%) to help determine whether suf?cient samples are present. It may not always be necessary to know the exact percentage of time used by a particular function in your program. For example, little practical difference exists between 25% and 27% of total time. Generally, you want to know only whether to spend optimization effort on that function. • Execution time should be short enough to be practical. If your program runs too long for this kind of testing, consider various ways to shorten execution, such as modifying data. 8.6.2 Reporting Parts of Programs Using prof to monitor only selected parts of programs saves memory and shortens report output. prof can monitor a selected part of your program if you specify the starting and ending addresses of that part. Use the PROF_SADDR and PROF_EADDR environment variables shown in Section 8.2.2.5, page 231. You can get addresses of routines from the profview output of a previous prof run (generated by the profview -Lz option). 004–2182–003 247Guide to Parallel Vector Applications 8.7 profview Sample Output The following examples show the following reports: • Summary report • Modules sorted by time used • Details within a module Example 1 shows a summary report. Example 1: USER PROFILE SUMMARY OF SIGNIFICANT ITEMS List of Modules with More than 10% Activity Module Name Hit Count PCT ACCUM% ----------- --------- ------ ------ TREEF 117880 37.11 37.11 ********* FTREE 114145 35.93 73.04 ******** Listing of Symbols Within These Modules Which Have > 10% Activity Module Name Symbol Name Min Bkt Max Bkt Rel Addr Hit Count Modul% ----------- ------------ -------- -------- -------- --------- ------ TREEF L.59 26757 26767 47 55849 47.38 ********* TREEF L.79 26776 27004 66 30212 25.63 ****** TREEF L.82 27005 27010 75 15109 12.82 *** FTREE S.40A 5245 5460 371 95996 84.10 ********* FTREE S.35 5461 5523 605 15141 13.26 *** Example 2 shows a modules-sorted-by-time-used report. Example 2: USER PROFILE ALL ACTIVE MODULES 248 004–2182–003profview [8] SORTED BY HIT COUNT (DESCENDING) Module Name Hit Count PCT ACCUM % ---------------- --------- ----- ------ TREEF 117880 37.11 37.11 ********* FTREE 114145 35.93 73.04 ******** SQRT% (math--scalar square root) 25383 7.99 81.03 * GFORSN 15685 4.94 85.97 * DTREE 14833 4.67 90.64 * NAYBOR 7359 2.32 92.96 INDEXX 3996 1.26 94.21 Example 3 shows a details-within-a-module report. Example 3: USER PROFILE DETAIL REPORT OF ACTIVITY WITHIN MODULES (USING TOTAL CUTOFF OF 95%) Module (FTREE) Program Percentage = 35.93 Accumulated Percentage = 73.04 Note: Bucket counts are combined for each symbol. Symbol Name Min Bkt Max Bkt Rel Addr Hit Count Modul% ACCUM% ------------ -------- -------- -------- --------- ------ ------ S.10A 5012 5025 136 9 0.01 0.01 S.10B 5113 5113 237 23 0.02 0.03 S.60A 5114 5144 240 165 0.14 0.17 L.106 5162 5162 306 6 0.01 0.18 L.109 5163 5171 307 41 0.04 0.21 S.15A1 5172 5200 316 71 0.06 0.28 S.15A2 5201 5213 325 161 0.14 0.42 L.114 5213 S.15B 5214 5244 340 2288 2.00 2.42 S.40A 5245 5460 371 95996 84.10 86.52 ********** S.35 5461 5523 605 15141 13.26 99.79 *** S.40B 5524 5535 650 79 0.07 99.86 S.50 5536 5557 662 123 0.11 99.96 S.60 5560 5563 704 42 0.04 00.00 004–2182–003 249Hardware Performance Monitor Counters [A] This appendix describes the contents of the hardware performance monitor counters used by the hpm and Perftrace software. The counters are different on the following systems: • Cray T90 series systems without IEEE ?oating-point arithmetic • Cray T90 series systems with IEEE ?oating-point arithmetic • Cray C90 systems • Cray SV1 and Cray J90 systems The Cray T90 and Cray C90 series systems have one group that contains 32 counters; the Cray SV1 and Cray J90 series systems have 4 groups that contain 8 counters each. The tables that follow in this chapter describe the counters on the different systems. Table 10, page 251, describes the system counters for the Cray T90 series without ?oating-point arithmetic. Table 10. Cray T90 Series without IEEE Floating-point Arithmetic System Counters Group Counter Description Number of: 0 Clock periods monitored (total CPU time) 1 Instructions issued 2 Clock periods holding issue 3 Instruction buffer fetches 4 CPU memory references 5 Clock periods for memory references 6 I/O memory references 7 Cache misses Number of clock 8 A registers periods holding issue 9 S registers 004–2182–003 251Guide to Parallel Vector Applications Group Counter Description for: 10 V registers 11 B/T registers 12 Functional units 13 Shared registers 14 Memory ports 15 Number of cache hits Number of 16 000-004 (Special) instructions: 17 Branch instructions 18 A-register instructions 19 B/T memory instructions 20 S-register instructions 21 Scalar integer instructions 22 Scalar ?oating-point instructions 23 S/A register memory instructions Number of operations: 24 Vector logical 25 Vector shift/pop/LZ 26 Vector integer adds 27 Vector ?oating multiplies 28 Vector ?oating adds 29 Vector ?oating reciprocals 30 Vector memory reads 31 Vector memory writes Table 11, page 253, describes the system counters for the Cray T90 series with ?oating-point arithmetic. 252 004–2182–003Hardware Performance Monitor Counters [A] Table 11. Cray T90 Series with IEEE Floating-point Arithmetic System Counters Group Counter Description Number of: 0 Clock periods monitored (total CPU time) 1 Instructions issued 2 Clock periods holding issue 3 Instruction buffer fetches 4 CPU memory references 5 Clock periods for memory references 6 I/O memory references 7 Cache misses Number of clock 8 A registers periods holding issue 9 S registers for: 10 V registers 11 B/T registers 12 Functional units 13 Shared registers 14 Memory ports 15 Number of cache hits Number of 16 000-004 (Special) instructions: 17 Branch instructions 18 A-register instructions 19 B/T memory instructions 20 S-register instructions 21 Scalar integer instructions 22 Scalar ?oating-point multiply and 128-bit multiply 23 S/A register memory instructions 004–2182–003 253Guide to Parallel Vector Applications Group Counter Description Number of operations: 24 Vector logical 25 Vector shift/pop/LZ 26 Vector integer adds 27 Vector ?oating multiplies and 128-bit multiplies 28 Vector ?oating adds, compares, and converts 29 Vector ?oating divides and square roots 30 Vector memory reads 31 Vector memory writes Table 12, page 254, describes the Cray C90 system counters. Table 12. Cray C90 System Counters Group Counter Description Number of: 0 Clock periods monitored (total CPU time) 1 Instructions issued 2 Clock periods holding issue 3 Instruction buffer fetches 4 CPU port memory references 5 CPU port memory con?icts 6 I/O port memory references 7 I/O port memory con?icts Number of clock 8 A registers periods holding issue 9 S registers for: 10 V registers 11 B/T registers 254 004–2182–003Hardware Performance Monitor Counters [A] Group Counter Description 12 Functional units 13 Shared registers 14 Memory ports 15 Miscellaneous Number of 16 000-004 (Special) instructions: 17 Branch instructions 18 A-register instructions 19 B/T memory instructions 20 S-register instructions 21 Scalar integer instructions 22 Scalar ?oating-point instructions 23 S/A register memory instructions Number of operations: 24 Vector logical 25 Vector shift/pop/LZ 26 Vector integer adds 27 Vector ?oating multiplies 28 Vector ?oating adds 29 Vector ?oating reciprocals 30 Vector memory reads 31 Vector memory writes Table 13, page 256, describes the Cray SV1 and Cray J90 series system counters. 004–2182–003 255Guide to Parallel Vector Applications Table 13. Cray SV1 and Cray J90 System Counters Group Counter Description 0 0 Instructions issued 1 Clock periods holding issue 2 Instruction buffer fetches 3 Floating-point add operations 4 Floating-point multiply operations 5 Floating-point reciprocal operations 6 CPU port memory references 7 Cache hits 1 0 Waiting on A registers 1 Waiting on S registers 2 Waiting on V registers 3 Waiting on B, T registers 4 Waiting on Vector Functional Units 5 Shared registers 6 Waiting on memory ports 7 Waiting on miscellaneous 2 0 Instruction fetches 1 Cache hits 2 Scalar memory writes 3 B, T memory references 4 Scalar memory references 5 CPU memory writes 6 CPU memory references 7 CPU memory con?icts 256 004–2182–003Hardware Performance Monitor Counters [A] Group Counter Description 3 0 000-017 instructions 1 020-077 instructions 2 100-137 instructions 3 140-157 and 175 instructions 4 160-174 instructions 5 176 and 177 instructions 6 Vector integer/logical operations 7 Vector ?oating-point operations 004–2182–003 257Interpreting Performance Statistics for Cray C90 Series Systems [B] This appendix discusses some of the conclusions that can be drawn from reports generated by hpm and perfview for data generated on Cray C90 series systems. From either utility, the report generated will indicate certain characteristics of your program. The following sections discuss what to look for in report output. Only counters of particular interest are discussed. On Cray C90 series systems, there is one group of 32 counters. These counters are grouped together in a logical organization. This appendix discusses each logical grouping separately. All of the counters discussed in this appendix appear on the full report created with the perfview or hpm command. For each of these groupings of statistics, there will be a general discussion of important counters, followed by a more detailed discussion of the statistics. The discussions are followed by a series of examples that show hpm command output for several representative or extreme conditions. Most of this appendix and its examples directly mention the Fortran programming language; however, most of the information also applies to other programming languages. To use these utilities most effectively, consider the following points: • Because hpm results apply to your entire program, important information about individual routines can be hidden. Only Perftrace can reveal some of these effects. However, hpm has the advantage of adding no overhead, and requires little extra effort to use. • Each clock period is 4.0 ns on a Cray C90 series system. A FLOP is a ?oating-point (real number) operation, speci?cally add, multiply, or reciprocal. Other operations use combinations of these operations, such as division, and functions, such as square root. A FLOP can take as little as 1 clock period (CP). Mega?ops (MFLOPS) are a million ?oating-point operations per second. The listings used for this appendix were generated using the UNICOS 7.0 version of the hpm command. All listings that appear in this section were generated on a Cray C90 series system. 004–2182–003 259Guide to Parallel Vector Applications B.1 HPM Statistics Following is an example of the HPM statistics for Cray C90 series systems. The following section contains an explanation of the contents. CPU seconds : 0.65 CP executing : 155142980 Million inst/sec (MIPS) : 30.67 Instructions : 19828455 Avg. clock periods/inst : 7.82 % CP holding issue : 80.65 CP holding issue : 125118596 Inst.buffer fetches/sec : 0.00M Inst.buf. fetches: 397 Floating ops/sec : 386.74M F.P. ops : 250000220 Vector Floating ops/sec : 386.74M Vec F.P. ops : 250000044 CPU mem. references/sec : 464.29M actual refs : 300130912 avg conflict/ref : 0.08 actual conflicts : 23472136 VEC mem. references/sec : 464.13M actual refs : 300024239 B/T mem. references/sec : 0.13M actual refs : 83850 I/O mem. references/sec : 0.35M actual refs : 227812 avg conflict/ref : 0.18 actual conflicts : 41822 Hold issue condition % of all CPs actual # of CPs Waiting on A-regs & access : 3.27 5074992 Waiting on S-regs & access : 0.16 249147 Waiting on V-registers : 58.53 90807856 Waiting on B/T-registers : 0.21 319098 Waiting on Functional Units : 34.16 53000273 Waiting on Shared Registers : 0.00 326 Waiting on Memory Ports : 9.28 14402004 Waiting on Miscellaneous : 1.94 3002561 (octal) instruction type inst./CPUsec actual inst. % of all insts. (000-004)Special : 1.56M 1005498 5.07 (005-017)Branch : 1.61M 1037728 5.23 (02x,030-033)A Register : 18.87M 12200356 61.53 (034-037)B/T Memory : 0.03M 20705 0.10 (040-043,071-077)S Register : 0.04M 24899 0.13 (044-061)Scalar Integer : 0.02M 15794 0.08 (062-070)Scalar Floating-Point : 0.00M 176 0.00 (10x-13x)Scalar Memory : 0.04M 22823 0.12 (140-177)All Vector : 8.51M 5500476 27.74 type of vector operation ops/CPUsec actual ops 260 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] Vector Logical : 0.02M 10415 Vector Shift/Pop/LZ : 0.00M 28 Vector Integer Add : 0.00M 110 Vector Floating Multiply : 309.39M 200000022 Vector Floating Add : 0.00M 22 Vector Floating Reciprocal : 77.35M 50000000 Vector Memory Read : 309.39M 200001501 Vector Memory Write : 154.73M 100022738 Average Vector Length for all Operations : 100.00 B.1.1 Overview of General Statistics Following are some basic observations that concern the statistics in the previous section: • The value for Floating ops/CPU second should fall between 30 MFLOPS and 1000 MFLOPS on Cray C90 series systems. Low values indicate mostly scalar code; high values (for example, those within 200 MFLOPS of the maximum values just mentioned) indicate a high degree of vectorization. When interpreting mega?op ?gures, consider which operations are counted as FLOPS. • Because the vector add, multiply, and reciprocal functional units (along with others) can operate simultaneously, the best mega?op rates occur when the numbers for these operations are well-balanced between functional units. • The line Million inst/sec (MIPS) displays the number of instructions issued in millions per second. This number should fall between 20 and 250 on Cray C90 systems. A low value accompanied by a high mega?op rating indicates execution of long vector instructions; few instructions yield many operations. A high value indicates primarily scalar code. • The ?gure for % CP holding issue is the percentage of clock periods during which the issuance of an instruction was held (delayed). This ?gure can indicate inef?cient code, but it is not a ?rm indicator because even the most optimized code can show a high percentage ?gure. The ?gure for actual clocks holding issue is often more signi?cant. Because this is a count rather than a percentage, any new effort at optimization should reduce this number. The goal for hold-issues is that they occur on vector functional units. Counters shown later in the report present hold-issue information in more detail. 004–2182–003 261Guide to Parallel Vector Applications • An instruction buffer contains 32 words fetched from memory. The fastest code execution consists of ?lling a buffer and executing a loop within that buffer. Almost as fast would be to execute a loop within more than one buffer (up to eight). Large numbers for Inst.buf.fetches/sec, such as those greater than 1 million per second, may indicate spaghetti code, which has too many jumps and not enough sequential execution. See the example in Section B.2.8, page 286. • I/O memory references are not usually under your control; these counts apply to any I/O on the system while your program runs. The following sections discuss these topics: • Balance of operations • Mega?op ratings • CPU memory references • I/O memory references • B/T memory references B.1.1.1 Balance of Operations Primitive ?oating-point operations are those that have their own functional units. These operations, which are the ?oating-point operations per second (FLOPS) used in mega?op ratings, use as little as 1 clock period each, in up to three functional units simultaneously. This is faster than composite operations, such as ?oating-point divides (each of which requires three multiplies and one reciprocal), power operations (such as X**3.99), or exponentials (the EXP function). The example in Section B.2.2, page 274, shows the effect of ?oating-point divisions. Many such composite operations are generated by compilers or performed by library routines. Intrinsic functions on UNICOS systems use a wide variety of composite operations. In addition, some functions (such as powers) use logical instructions. The following goals are closely related: • Execution is faster when more than one functional unit is used simultaneously. Simultaneous usage is indicated only indirectly by low hold-issue ?gures and high mega?op ?gures. For example, if you must perform an add and a multiply on an array, this should be done in one loop, rather than in two separate loops (one for adding and one for multiplying). 262 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] The combined loop generates code that makes simultaneous use of functional units. • A balance among the three kinds of primitive operations is desirable. To judge this balance, compare the counters for the three operations. Balanced numbers should be accompanied by low hold-issue ?gures, indicating simultaneous usage. B.1.1.2 Mega?op Ratings The peak speed attainable, with all three types of ?oating-point operations active, using long vectors and no hold-issue con?icts, is about 1.5 billion FLOPs. See the example in Section B.2.11, page 291, for a code that approaches this peak rate. The ratio of vector to scalar ?oating-point operations is an important performance measure and can be determined from the listing by checking the ?gures for Vector Floating ops/sec and (062-070) Scalar Floating-Point. Other ratios of vector to scalar (such as for memory operations and integer/logical operations) also can be derived from the statistics given in the listing. B.1.1.3 CPU Memory References If the number of CPU memory references greatly exceeds the sum of ?oating-point adds and multiplies, your program might be performing unnecessary memory references. However, a typical primitive calculation isolated in a loop is inherently memory-bound; the high ratio of references to operations cannot be optimized away. Example: DO 100 I=1,10000 Z(I)=X(I)*Y(I) 100 CONTINUE This loop generates 10,000 ?oating-point multiplies and 30,000 CPU memory references (two loads, one store). This 3:1 ratio is inherent in the operation. On Cray C90 systems, this code makes use of the two load pathways and one store pathway to memory. In more complex codes that contain many different calculations, the proportions between calculations and memory activity should balance out. If they are far 004–2182–003 263Guide to Parallel Vector Applications out of balance, the code may be memory-bound, preventing it from achieving the best performance. For example, a memory reference to a FLOP ratio of 20:1 indicates that the code is spending most of its time moving data around. The CPU mem. reference/sec counters indicate memory activity and memory con?icts. A vector memory reference carried out with no con?ict takes 1 CP. When a con?ict occurs, it can cause a delay (hold-issue condition) of, potentially, any number of clock periods. The line labeled avg conflict/ref shows the following ratio: clock_periods_spent_on_conflicts / number of memory references In an ideal situation, every memory reference would take only 1 CP. Thus, the number of clock periods holding-issue per memory reference would be 0. However, in a more realistic environment, there are some hold-issues for memory. In fact, one con?ict can cause a delay of many clock periods. In the previous example, the con?ict per CPU memory reference is reasonably low. If this number were to exceed 1.0, there would be cause for concern. A comparison of vector and scalar memory references helps provide another indication of the level of vectorization. Scalar memory references is shown later in the report as (10x-13x) Scalar Memory. A low vector/scalar ratio might indicate that the code could be more fully vectorized. B.1.1.4 I/O Memory References A good ?gure for I/O memory references is lower than 1% of all memory references, but a higher ?gure can result from conditions beyond your control (indicating that other activity is occurring in the system), or from an ef?cient overlap of I/O with computation by such means as asynchronous I/O. The hardware monitor cannot distinguish I/O memory references for your program from those that occur for other purposes while your program executes; the ?gure for I/O memory references is, therefore, a system-wide statistic. You cannot control the activity of other programs, especially on a multiple-CPU machine, so you can get a high ?gure that is independent of your program. If your code properly overlaps I/O with computation, and both activities occur heavily, the I/O memory reference count can be high. A high count indicates ef?cient system usage. Even if the ratio of I/O memory con?icts is high (over 1.0), you should compare the actual counts of I/O memory references and CPU memory references. Typically, the I/O reference count is much smaller than the CPU count, so a 264 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] high ratio of I/O reference con?icts does not indicate a problem. The percentage of con?icts varies signi?cantly from one program execution to the next. B.1.1.5 B/T Memory References The B/T memory reference ?gure is derived from the normal HPM statistics. The rate of these references usually will give you a clue as to whether your program is jumping between routines at a high speed (spaghetti code). The B and T registers are loaded and stored as part of the housekeeping code for high-level languages, such as Fortran or C/C++. Their level of activity indicates how much overhead is being expended to enter and exit routines and functions. A rate of B/T memory references of more than 1 million per second indicates a program that might bene?t from inlining of some routines. Using Jumptrace, Flowtrace, or Perftrace can help pinpoint these routines because they usually show very high call counts coupled with a low average CPU time per call. B.1.2 Hold-issue Statistics The counters under the Hold issue condition heading show the causes for the various hold-issue conditions. For many purposes, these numbers appear to be of limited interest because optimizing compilers remove most bottlenecks in the use of registers, functional units, and memory. However, hold-issue statistics often reveal the true behavior of a code. It is desirable to remove impediments to high execution speed. These statistics show the location of those impediments. Study of hold-issue statistics can be particularly helpful to users working in assembly language to attain higher speed, or in the design of an optimizer for compiler-generated code. Because subtle balances exist between the various hold-issue conditions, trying to lower one counter may increase others. Following is a sample of HPM hold-issue statistics: Hold issue condition % of all CPs actual # of CPs Waiting on A-regs & access : 3.27 5074992 Waiting on S-regs & access : 0.16 249147 Waiting on V-registers : 58.53 90807856 Waiting on B/T-registers : 0.21 319098 Waiting on Functional Units : 34.16 53000273 Waiting on Shared Registers : 0.00 326 Waiting on Memory Ports : 9.28 14402004 Waiting on Miscellaneous : 1.94 3002561 004–2182–003 265Guide to Parallel Vector Applications Following are some speci?c topics concerning hold-issue statistics: • More than one hold-issue condition can occur on the same clock period. Each counter has a maximum percentage value of 100%; therefore, the total percentage of all counters can exceed 100%. However, the ?gure for hold-issue percentage shown near the top of the summary section of the report re?ects the total number of clock periods in which one or more active hold-issue conditions occurred. That percentage has a maximum of 100% and is distinct from the total of counter percentages. • A useful comparison is between the sum of waiting on A registers and S registers, and the statistics for waiting on V registers. This yields a useful vector-to-scalar ratio for hold-issue conditions. The resulting ratio is relevant because higher activity in a particular area (vector, for instance) usually will show a higher percentage of hold-issue conditions. Thus, you can infer the degree of one activity over the other by looking at hold-issue percentages. • Very high values for memory ports hold-issue (for instance, greater than 30%) might indicate memory bank con?icts. Examine the avg conflict/ ref value given in the summary section to determine if there is a high percentage of memory con?icts. The example in Section B.2.5, page 280, shows the effects of a memory bank con?ict on counter values. • For maximum performance, the highest percentage of hold-issues should be on the vector registers and functional units. Because vector units are the fastest devices for calculation, high values may indicate ef?cient code. • The percentage ?gure for Waiting on Shared Registers indicates the percentage of the total execution time spent waiting for semaphores and other shared resources. • The hold-issue counter for Waiting on Functional Units usually contains counts associated primarily with vector functional units. Therefore, it is a good measure of vector activity. The time spent waiting on scalar functional units, on the other hand, usually shows up as time spent waiting on the S registers. This behavior occurs because calculation results are needed sooner in scalar code, and functional unit overlap does not often occur. • An issue for code optimizers is chaining, particularly correct use of registers and overlapping use of functional units. Codes that chain vector operations often show high hold-issue counts for both V registers and functional units. Suboptimal chaining operations may show high counts for only functional units. 266 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] • High mega?op rates (see Section B.1.1.2, page 263) coupled with high hold-issue counts for CPU memory references may indicate that your code is spending too much time waiting for operands from memory (a slower device). If this is the case, try to modify your algorithm to combine operations and to decrease the dependence on operand fetches. Remember, however, that optimizing compilers already perform much of this analysis and modi?cation. • For multitasked codes, the ja command tells how much CPU-time overlap occurred, but not whether the overlap was spent waiting or calculating. The line labeled Waiting on Shared Registers indicates the percentage of the total overlap time spent waiting. Multiply this percentage by the total execution time to get the amount of time spent waiting. You can use the ATExpert program to determine parallel ef?ciency for autotasked programs. B.1.3 Instruction Summary This group of counters shows how many instructions were issued, grouped by type. The octal numbers at the beginning of each line represent the range of instruction opcodes for the category shown. The instruction count for all vector instructions is calculated from the other counters. Scalar instruction counts are shown in some detail. Each scalar instruction generates a single scalar operation, so you can consider these instruction counts to be identical with scalar operation counts. One vector instruction can generate up to 128 operations. The average vector length for all vector operations is given at the end of the report. (octal) instruction type inst./CPUsec actual inst. % of all insts. (000-004)Special : 1.56M 1005498 5.07 (005-017)Branch : 1.61M 1037728 5.23 (02x,030-033)A Register : 18.87M 12200356 61.53 (034-037)B/T Memory : 0.03M 20705 0.10 (040-043,071-077)S Register : 0.04M 24899 0.13 (044-061)Scalar Integer : 0.02M 15794 0.08 (062-070)Scalar Floating-Point : 0.00M 176 0.00 (10x-13x)Scalar Memory : 0.04M 22823 0.12 (140-177)All Vector : 8.51M 5500476 27.74 Following is a description of each instruction type: 004–2182–003 267Guide to Parallel Vector Applications Opcode Comment 000-004 Special. Primarily used for housekeeping. These instructions also include those that set and clear semaphores. Thus, they could be important for multitasked codes. The overall percentage of these types of instructions should never be very high. 005-017 Branch. Primarily used for jumping out of sequential execution. The instruction count and percentage are usually low but can be high for spaghetti code, or very highly scalar code. 02x, 030-033 A register. Primarily used for addressing calculations. 034-037 B/T memory. When entering or leaving a function or routine in a high-level language, such as C or Fortran, these instructions are used as part of the housekeeping. A rate of greater than 1 million of these instructions per second would indicate spaghetti code. This type of code should be studied with the goal of inlining as many key routines as possible. The Jumptrace, Flowtrace, and Perftrace tools can help pinpoint which routines might bene?t most from inlining. 040-043, 071-077 S register. This set of instructions performs register-to-register operations. It does not include scalar arithmetic activities. The rate of these instructions can be expected to be high (over 10 million per second) for a scalar-dominated code, because these instructions play an important role in scalar codes. 044-061 Scalar integer. 268 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] This statistic includes both scalar integer (add and subtract), as well as scalar logical operations. The rate of these instructions will be low (fewer than 1 million per second) for a vector-dominated code. The rate will be high (greater than 1 million per second) for scalar-dominated code. However, a high rate does not guarantee that the code is scalar-dominated. 062-070 Scalar ?oating-point. These instructions include all operations in the scalar ?oating-point functional units: add, multiply, and reciprocal approximation. The rate of these instructions will be low (fewer than 1 million per second) for a vector-dominated code. The rate will be high (greater than 1 million per second) for scalar-dominated code. However, a high rate does not guarantee that the code is scalar-dominated. 10x-13x Scalar memory. Includes all instructions for scalar (both A and S) register loading and storing. The rate of these instructions will be low (fewer than 1 million per second) for a vector-dominated code. The rate will be high (greater than 1 million per second) for scalar-dominated code. However, a high rate does not guarantee that the code is scalar-dominated 140-177 All vector. This value is derived from the HPM statistics. The rate at which vector instructions are issued may not be as important a statistic as the rate of vector operations. See Section B.1.4, page 269, for a discussion of vector statistics. B.1.4 Vector Operations The last grouping of lines in the report shows statistics for the operations performed for vector instructions. These vector operation counts are much more informative than the vector instruction counts because one vector 004–2182–003 269Guide to Parallel Vector Applications instruction can generate up to 128 operations. This contrasts with the scalar instruction counts, in which one scalar instruction causes one scalar operation. type of vector operation ops/CPUsec actual ops Vector Logical : 0.02M 10415 Vector Shift/Pop/LZ : 0.00M 28 Vector Integer Add : 0.00M 110 Vector Floating Multiply : 309.39M 200000022 Vector Floating Add : 0.00M 22 Vector Floating Reciprocal : 77.35M 50000000 Vector Memory Read : 309.39M 200001501 Vector Memory Write : 154.73M 100022738 Comparison between vector and scalar operation counts is useful for judging the level of vectorization. For instance, you can compare scalar ?oating-point operation counts (instructions) with total vector ?oating-point operation counts. The scalar integer/logical counts also can be compared with the total number of vector integer and vector logical operations. Finally, scalar memory references can be compared with vector memory operation counts. When the vector counts are high (10 times or more than scalar), the code can be considered to be vector. When the scalar counts are equal to or higher than vector counts, the code can be considered to be dominated by scalar operations. Studying this section of the report can help you determine the relative balance between the vector functional units. Because each functional unit (and thus the operation rates shown here) is independent of the others, the execution rate of the program can be enhanced if the different units are used simultaneously and independently. If useful work can be combined in the same loop for two different functional units, for instance, the execution speed of the code can be nearly doubled. A typical example might be to combine loops so that ?oating-point multiplies occur at the same time as ?oating-point adds. If the operations are isolated in separate loops, the compiler probably will schedule them to execute at different times, thus losing the advantage of simultaneous functional unit usage. Previous machine types accumulated misleading statistics for vector logical operations when the AVL unit was enabled. For Cray C90 series systems, this is no longer the case. All vector logical operation counts and rates are consistent and correct as measured by the hardware. Vector logical operations have their own counter for this machine type, but scalar logical operations are included in the counter (044-061)Scalar Integer. A ratio of vector versus scalar integer/logical operations can be derived from these statistics. 270 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] Cray C90 series systems have an additional vector logical functional unit. Section B.2.12, page 293, shows a code that generates many vector logical operations, both with and without the AVL unit enabled. B.1.5 Average Vector Length The last line in the report shows Average Vector Length for all Operations. It is derived from the number of vector instructions and the total number of vector operations. The maximum value is 128 in a highly vectorized code. In practical operation, an average vector length of 30 is acceptable. Some codes will not show long vectors; improving these may be beyond the practical capability of the compiler and available expertise, and it may require a change to the algorithm being used. Short vector operations are usually inef?cient; the minimum ef?cient length depends on the system and other factors, but ?gures in the range 3 to 5 are often used as a minimum ef?cient vector length. Even when the average vector length seems to indicate ef?cient code, it can be misleading. A length approaching 128 does not always mean your code is ef?ciently vectorized. Example 1: DO 100 I=1,100 DO 100 J=1,128 Z(J,I)=X(J,I)*Y(J,I) 100 CONTINUE The code shown in example 1 is slower than that shown in example 2. Example 2: DO 100 I=1,100*128 ZZ(I)=XX(I)*YY(I) 100 CONTINUE Although both loops would show an average vector length of 128, the ?rst code is slower because the outer loop generates extra overhead. An average vector length of less than 10 could indicate a problem in your code. However, current compiler optimizers make good use of the hardware, and even very short vector operations perform better than scalar operations. 004–2182–003 271Guide to Parallel Vector Applications B.2 Examples of Hardware Performance Monitor Statistics The following sections give examples of hpm output to show the kind of statistics that can result from various code characteristics. The codes used were designed to show these characteristics in an extreme form, and they may not always re?ect real world codes. Most examples are given in pairs, in which each member of the pair shows a certain effect. For an individual routine or bracketed area of code, Perftrace results would be very similar to the hpm results show. In each example report, lines that are discussed in the accompanying text are indicated by a quotation mark ("). B.2.1 Highly Scalar Code The code for this example contained extensive ?oating-point calculation. Vectorization was disabled during compilation. Section B.2.2, page 274, shows hpm output for the same code when vectorization was enabled. CPU seconds : 0.71 CP executing : 169967133 "Million inst/sec (MIPS): 133.60 Instructions : 94615088 Avg. clock periods/inst : 1.80 % CP holding issue : 23.67 CP holding issue : 40233165 Inst.buffer fetches/sec : 0.01M Inst.buf. fetches: 5836 "Floating ops/sec : 35.32M F.P. ops : 25010126 Vector Floating ops/sec : 0.00M Vec F.P. ops : 48 CPU mem. references/sec : 42.46M actual refs : 30068956 avg conflict/ref : 0.09 actual conflicts : 2746834 VEC mem. references/sec : 0.01M actual refs : 4257 B/T mem. references/sec : 0.06M actual refs : 41852 I/O mem. references/sec : 0.00M actual refs : 0 avg conflict/ref : .... actual conflicts : 0 Hold issue condition % of all CPs actual # of CPs Waiting on A-regs & access : 0.02 41664 "Waiting on S-regs & access : 19.73 33532220 Waiting on V-registers : 0.00 449 Waiting on B/T-registers : 3.60 6126689 Waiting on Functional Units : 0.00 540 Waiting on Shared Registers : 0.00 284 Waiting on Memory Ports : 0.10 178157 272 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] Waiting on Miscellaneous : 0.22 374889 (octal) instruction type inst./CPUsec actual inst. % of all insts. (000-004)Special : 0.00M 698 0.00 (005-017)Branch : 1.39M 983472 1.04 (02x,030 033)A Register : 2.42M 1713418 1.81 (034-037)B/T Memory : 0.01M 4705 0.00 "(040-043,071-077)S Register : 52.03M 36847985 38.95 (044-061)Scalar Integer : 0.04M 31709 0.03 "(062-070)Scalar Floating-Point : 35.32M 25010078 26.43 "(10x-13x)Scalar Memory : 42.39M 30022847 31.73 (140-177)All Vector : 0.00M 176 0.00 type of vector operation ops/CPUsec actual ops Vector Logical : 0.00M 421 Vector Shift/Pop/LZ : 0.00M 28 Vector Integer Add : 0.00M 110 Vector Floating Multiply : 0.00M 24 Vector Floating Add : 0.00M 24 Vector Floating Reciprocal : 0.00M 0 Vector Memory Read : 0.00M 1511 Vector Memory Write : 0.00M 2746 Average Vector Length for all Operations : 27.64 The report shows a high value for instructions issued (134 MIPS) and a low number of ?oating-point operations per second (35 MFLOPS). This is the ?rst indication of a scalar-dominated code. There is additional con?rmation of this conclusion, since the ratio of scalar ?oating-point operations to vector ?oating-point operations is nearly in?nite. Memory references show a similar pattern. In the hold-issue conditions, the highest percentage is for S registers. This is additional con?rmation of scalar behavior. The hold-issue on functional units is almost nonexistent. In scalar codes, the hold-issue for a calculation result usually occurs on the S register that is to receive the result. The functional unit hold-issue usually contains counts only for vector functional units. In the instruction types shown, the highest rates and counts are for scalar ?oating-point, memory, and register instructions. 004–2182–003 273Guide to Parallel Vector Applications The last section of the report shows trivial counts for vector operations. B.2.2 Highly Vectorized Code The code used in this example is the same as that in the scalar code example shown in the previous section. In this example, it was compiled with vectorization enabled. CPU seconds : 0.05 CP executing : 13168700 "Million inst/sec (MIPS) : 28.48 Instructions : 1562882 Avg. clock periods/inst : 8.43 % CP holding issue : 83.19 % CP holding issue : 10955047 Inst.buffer fetches/sec : 0.01M Inst.buf. fetches: 382 "Floating ops/sec : 455.81M F.P. ops : 25010114 "Vector Floating ops/sec : 455.81M Vec F.P. ops : 25010040 "CPU mem. references/sec : 547.68M actual refs : 30050955 avg conflict/ref : 0.05 actual conflicts : 1586966 "VEC mem. references/sec : 547.34M actual refs : 30032458 B/T mem. references/sec : 0.29M actual refs : 15642 I/O mem. references/sec : 0.23M actual refs : 12450 avg conflict/ref : 0.43 actual conflicts : 5400 Hold issue condition % of all CPs actual # of CPs Waiting on A-regs & access : 2.51 331062 Waiting on S-regs & access : 0.22 29041 "Waiting on V-registers : 45.59 6003549 Waiting on B/T-registers : 0.31 40951 "Waiting on Functional Units : 42.92 5651694 Waiting on Shared Registers : 0.00 297 Waiting on Memory Ports : 10.79 1421365 Waiting on Miscellaneous : 1.97 259927 (octal) instruction type inst./CPUsec actual inst. % of all insts. (000-004)Special : 1.12M 61281 3.92 (005-017)Branch : 1.21M 66226 4.24 (02x,030-033)A Register : 17.90M 982047 62.84 (034-037)B/T Memory : 0.05M 2705 0.17 (040-043,071-077)S Register : 0.12M 6312 0.40 (044-061)Scalar Integer : 0.11M 6309 0.40 (062-070)Scalar Floating-Point : 0.00M 74 0.00 274 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] (10x-13x)Scalar Memory : 0.05M 2855 0.18 (140-177)All Vector : 7.93M 435073 27.84 type of vector operation ops/CPUsec actual ops Vector Logical : 0.01M 409 Vector Shift/Pop/LZ : 0.00M 28 Vector Integer Add : 0.18M 10110 "Vector Floating Multiply : 364.50M 20000020 Vector Floating Add : 0.18M 10020 "Vector Floating Reciprocal : 91.13M 5000000 "Vector Memory Read : 364.68M 20009710 "Vector Memory Write : 182.66M 10022748 Average Vector Length for all Operations : 126.54 The low instruction issue count (28 MIPS) and high FLOP count (456 MFLOPS) indicate a vectorized code. The ?oating-point operations are nearly equal to the vector ?oating-point operations, as are the memory references. This set of counters con?rms that the code is vector-dominated. In the hold-issue section, there is very little hold-issue on S registers, but a high percentage on V registers. There also is a high hold-issue on functional units which, for the most part, are vector functional units. The total of all hold-issue percentages exceeds 100%. This is possible because more than one hold-issue condition can occur during any one clock period. The ?nal section of the report con?rms that the vector functional units are very active, with high counts and rates for ?oating-point multiply and reciprocal, as well as for memory references. This code performs an equal number of ?oating-point multiplies and divides. A divide requires one reciprocal and three multiplies to complete. The 5 million vector reciprocals performed, therefore, account for 15 million of the 20 million multiplies performed. The divide calculation does not chain ideally, because the three multiplies must be fed back into the ?oating-point multiply functional unit one after another. Better vector chaining occurs when results from one vector functional unit can be fed into another without having to feed back to the same unit again. Section B.2.4, page 278, shows this improved chaining. 004–2182–003 275Guide to Parallel Vector Applications B.2.3 Code That Performs in One Vector Functional Unit The code for this example is from Fortran code that primarily performs multiplications with real numbers. CPU seconds : 0.01 CP executing : 3303050 Million inst/sec (MIPS) : 43.70 Instructions : 601377 Avg. clock periods/inst : 5.49 % CP holding issue : 74.88 CP holding issue : 2473469 Inst.buffer fetches/sec : 0.03M Inst.buf. fetches: 348 Floating ops/sec : 364.04M F.P. ops : 5010114 "Vector Floating ops/sec : 364.03M Vec F.P. ops : 5010040 CPU mem. references/sec : 1093.24M actual refs : 15045956 avg conflict/ref : 0.09 actual conflicts : 1415978 "VEC mem. references/sec : 1092.26M actual refs : 15032460 B/T mem. references/sec : 0.77M actual refs : 10642 I/O mem. references/sec : 0.30M actual refs : 4096 avg conflict/ref : 1.23 actual conflicts : 5041 Hold issue condition % of all CPs actual # of CPs Waiting on A-regs & access : 5.22 172259 Waiting on S-regs & access : 0.89 29390 Waiting on V-registers : 0.26 8470 Waiting on B/T-registers : 0.74 24479 "Waiting on Functional Units : 22.91 756780 Waiting on Shared Registers : 0.01 295 "Waiting on Memory Ports : 44.45 1468227 Waiting on Miscellaneous : 4.28 141386 (octal) instruction type inst./CPUsec actual inst. % of all insts. (000-004)Special : 1.51M 20781 3.46 (005-017)Branch : 1.83M 25225 4.19 "(02x,030-033)A Register : 27.61M 380045 63.20 (034-037)B/T Memory : 0.12M 1705 0.28 (040-043,071-077)S Register : 0.42M 5812 0.97 (044-061)Scalar Integer : 0.46M 6308 1.05 (062-070)Scalar Floating-Point : 0.01M 74 0.01 (10x-13x)Scalar Memory : 0.21M 2854 0.47 (140-177)All Vector : 11.52M 158573 26.37 276 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] type of vector operation ops/CPUsec actual ops Vector Logical : 0.03M 409 Vector Shift/Pop/LZ : 0.00M 28 Vector Integer Add : 0.73M 10110 "Vector Floating Multiply : 363.30M 5000020 Vector Floating Add : 0.73M 10020 Vector Floating Reciprocal : 0.00M 0 "Vector Memory Read : 727.31M 10009711 "Vector Memory Write : 364.95M 5022749 Average Vector Length for all Operations : 126.46 This code is equivalent to the following Fortran fragment: DO 100 I=1,128 100 Z(I) = X(I) * Y(I) The report shows a high value for vector ?oating-point operations and vector memory references. The code is highly vectorized. In the hold-issue section, the code is waiting primarily for functional units (vector) and memory ports. The multiply operations are being chained from memory, rather than from another functional unit. In this case, memory can be considered to be the same as a functional unit when vector operations occur. Because the preceding algorithm performs one multiply for every three memory references, it can be supposed that the speed of memory is the limiting element of overall calculation speed. In the instruction type section, there is a dominance of A register instructions. These are used to support the vector operations in a housekeeping function. In the vector operation section, there are many ?oating-point multiplies, but few adds or reciprocals. The best FLOP rate this code can achieve is limited by the fact that only one functional unit is being used. The multiply unit thus acts much like a pipe between memory with only a small delay occurring during the multiply operations. This behavior can be contrasted with the next example (two functional units used), in which nearly the same memory rate can support almost twice as many calculations per second. This code achieves 364 MIPS. Contrast that ?op rate, however, with the next example that uses two functional units at the same time. 004–2182–003 277Guide to Parallel Vector Applications B.2.4 Code That Performs in Two Vector Functional Units The code for this example is from Fortran code that primarily performs multiply and add operations using real numbers. CPU seconds : 0.01 CP executing : 3597841 "Million inst/sec (MIPS) : 42.85 Instructions : 642377 Avg. clock periods/inst : 5.60 % CP holding issue : 74.14 CP holding issue : 2667538 Inst.buffer fetches/sec : 0.02M Inst.buf. fetches: 347 Floating ops/sec : 667.74M F.P. ops : 10010114 "Vector Floating ops/sec : 667.74M Vec F.P. ops : 10010040 CPU mem. references/sec : 1003.67M actual refs : 15045956 avg conflict/ref : 0.18 actual conflicts : 2642320 "VEC mem. references/sec : 1002.82M actual refs : 15033228 B/T mem. references/sec : 0.66M actual refs : 9874 I/O mem. references/sec : 0.01M actual refs : 223 avg conflict/ref : 0.04 actual conflicts : 10 Hold issue condition % of all CPs actual # of CPs Waiting on A-regs & access : 4.79 172278 Waiting on S-regs & access : 0.82 29360 Waiting on V-registers : 0.23 8336 Waiting on B/T-registers : 0.69 24895 "Waiting on Functional Units : 24.38 877133 Waiting on Shared Registers : 0.01 288 "Waiting on Memory Ports : 42.94 1544868 Waiting on Miscellaneous : 3.93 141341 (octal) instruction type inst./CPUsec actual inst. % of all insts. (000-004)Special : 1.39M 20781 3.24 (005-017)Branch : 1.68M 25225 3.93 (02x,030-033)A Register : 25.35M 380045 59.16 (034-037)B/T Memory : 0.11M 1705 0.27 (040-043,071-077)S Register : 0.45M 6812 1.06 (044-061)Scalar Integer : 0.45M 6808 1.06 (062-070)Scalar Floating-Point : 0.00M 74 0.01 (10x-13x)Scalar Memory : 0.19M 2854 0.44 (140-177)All Vector : 13.21M 198073 30.83 278 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] type of vector operation ops/CPUsec actual ops Vector Logical : 0.03M 409 Vector Shift/Pop/LZ : 0.00M 28 Vector Integer Add : 0.67M 10110 "Vector Floating Multiply : 333.53M 5000020 "Vector Floating Add : 334.20M 5010020 Vector Floating Reciprocal : 0.00M 0 "Vector Memory Read : 667.77M 10010479 "Vector Memory Write : 335.05M 5022749 Average Vector Length for all Operations : 126.49 This code is equivalent to the following Fortran fragment: DO 100 I=1,128 100 Z(I) = 99.99 + (X(I) * Y(I)) The report shows a FLOP rate of 668 million per second, much higher than that of the previous example. This speed is directly attributable to using two vector functional units simultaneously. The vector memory reference rate remains similar to that in the previous example. In the hold-issue section, there are similar proportions of hold-issue time spent waiting for (vector) functional units and memory. The last section of the report shows that the vector ?oating-point add and multiply functional units were able to achieve good speeds, and that these rates combine to yield a much higher overall ?oating-point calculation rate than the previous example. Combining multiple arithmetic operations into the same vectorized loop, so that more than one vector functional unit is used, allows the speed of the code to be increased considerably. This example can be contrasted with Section B.2.2, page 274, in which divide calculations were performed. This example shows much better chaining of operations, because the data is brought into registers from memory, then into one functional unit, chaining to a different unit, and then back to memory from the result registers. The divide example had to re-inject the results of two multiplies back into the multiply functional unit, causing poor chaining to occur. The example illustrates why it is best to avoid performing divides when possible. 004–2182–003 279Guide to Parallel Vector Applications B.2.5 Extensive Memory Bank Con?icts The code for this example was intentionally designed to force a high number of memory bank con?icts. The stride through memory was 16. You should try to avoid all stride values that are higher than powers of 2. A slight cost is associated with a stride of exactly 2, but the penalty then increases rapidly with larger powers of 2. The penalty for bank con?icts reaches its extreme level at stride 64 and beyond. Following is Fortran code that could cause this type of con?ict: DIMENSION X(16,1000) ... DO 100 I=1,1000 X(10,I)=X(5,I) 100 CONTINUE Contrast this with the next example, in which the memory bank con?ict has been eliminated. CPU seconds : 0.48 CP executing : 114788334 Million inst/sec (MIPS) : 0.86 Instructions : 411342 "Avg. clock periods/inst: 279.06 "% CP holding issue : 99.48 CP holding issue : 114194021 Inst.buffer fetches/sec : 0.00M Inst.buf. fetches: 374 Floating ops/sec : 0.00M F.P. ops : 114 Vector Floating ops/sec : 0.00M Vec F.P. ops : 40 CPU mem. references/sec : 68.54M actual refs : 32779247 " avg conflict/ref : 12.75 actual conflicts : 417962775 "VEC mem. references/sec: 68.52M actual refs : 32772290 B/T mem. references/sec : 0.01M actual refs : 3996 I/O mem. references/sec : 0.00M actual refs : 0 avg conflict/ref : .... actual conflicts : 0 Hold issue condition % of all CPs actual # of CPs Waiting on A-regs & access : 0.12 138252 Waiting on S-regs & access : 0.02 27353 Waiting on V-registers : 0.00 452 Waiting on B/T-registers : 0.01 6778 Waiting on Functional Units : 0.00 505 Waiting on Shared Registers : 0.00 284 "Waiting on Memory Ports : 99.33 114018985 Waiting on Miscellaneous : 0.23 258660 280 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] (octal) instruction type inst./CPUsec actual inst. % of all insts. (000-004)Special : 0.00M 202 0.05 (005-017)Branch : 0.07M 34810 8.46 (02x,030-033)A Register : 0.22M 105857 25.73 (034-037)B/T Memory : 0.00M 723 0.18 (040-043,071-077)S Register : 0.01M 4716 1.15 (044-061)Scalar Integer : 0.01M 5818 1.41 (062-070)Scalar Floating-Point : 0.00M 74 0.02 (10x-13x)Scalar Memory : 0.01M 2961 0.72 (140-177)All Vector : 0.54M 256181 62.28 type of vector operation ops/CPUsec actual ops Vector Logical : 0.00M 446 Vector Shift/Pop/LZ : 0.00M 42 Vector Integer Add : 0.00M 143 Vector Floating Multiply : 0.00M 20 Vector Floating Add : 0.00M 20 Vector Floating Reciprocal : 0.00M 0 "Vector Memory Read : 68.51M 32769539 Vector Memory Write : 0.01M 2751 Average Vector Length for all Operations : 127.93 The beginning of the report shows little ?oating-point activity. (The code used for this example performs no calculations. As with the Fortran fragment shown previously, it primarily reads data from memory.) A problem is indicated by the high percentage of clock periods holding issue, and the high number of clock periods per instruction. The low number (about 69 million) for all CPU memory references per second. This is too low when you consider that this code is referencing memory through only the vector registers. The con?ict per memory reference, the avg conflict/ref, is 12.75. This means that each memory reference incurred an average delay of more than 12 clock periods. The hold-issue section of the report shows a very high percentage of all clock periods spent waiting for memory ports. (This percentage is especially high because this code does little other work, leaving other percentages low.) The rest of the report re?ects the somewhat arti?cial nature of this example, because almost all activity is accounted for by vector memory operations. 004–2182–003 281Guide to Parallel Vector Applications B.2.6 Memory Con?ict Removed The code for this example is similar to that used in the preceding section, but with the memory con?ict removed by using a memory stride of 17, rather than 16. Total execution time is over 13 times faster than that of the previous example. Following is a Fortran fragment that shows the corrected example: C Note: First dimension is now 17 DIMENSION X(17,1000) ... DO 100 I=1,1000 X(10,I)=X(5,I) 100 CONTINUE CPU seconds : 0.04 CP executing : 9062458 Million inst/sec (MIPS) : 10.90 Instructions : 411442 "Avg. clock periods/inst : 22.03 "% CP holding issue : 93.44 CP holding issue : 8467921 Inst.buffer fetches/sec : 0.01M Inst.buf. fetches: 378 Floating ops/sec : 0.00M F.P. ops : 114 Vector Floating ops/sec : 0.00M Vec F.P. ops : 40 CPU mem. references/sec : 868.09M actual refs : 32779262 " avg conflict/ref : 0.00 actual conflicts : 5885 "VEC mem. references/sec : 867.90M actual refs : 32772299 B/T mem. references/sec : 0.11M actual refs : 3996 I/O mem. references/sec : 0.00M actual refs : 0 avg conflict/ref : .... actual conflicts : 0 Hold issue condition % of all CPs actual # of CPs Waiting on A-regs & access : 1.53 138261 Waiting on S-regs & access : 0.30 27443 Waiting on V-registers : 0.01 477 Waiting on B/T-registers : 0.07 6778 Waiting on Functional Units : 0.01 505 Waiting on Shared Registers : 0.00 284 "Waiting on Memory Ports : 91.51 8292759 Waiting on Miscellaneous : 2.85 258662 (octal) instruction type inst./CPUsec actual inst. % of all insts. 282 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] (000-004)Special : 0.01M 204 0.05 (005-017)Branch : 0.92M 34820 8.46 (02x,030-033)A Register : 2.80M 105882 25.73 (034-037)B/T Memory : 0.02M 723 0.18 (040-043,071-077)S Register : 0.13M 4734 1.15 (044-061)Scalar Integer : 0.16M 5855 1.42 (062-070)Scalar Floating-Point : 0.00M 74 0.02 (10x-13x)Scalar Memory : 0.08M 2967 0.72 (140-177)All Vector : 6.78M 256183 62.26 type of vector operation ops/CPUsec actual ops Vector Logical : 0.01M 446 Vector Shift/Pop/LZ : 0.00M 42 Vector Integer Add : 0.00M 143 Vector Floating Multiply : 0.00M 20 Vector Floating Add : 0.00M 20 Vector Floating Reciprocal : 0.00M 0 "Vector Memory Read : 867.83M 32769544 Vector Memory Write : 0.07M 2755 Average Vector Length for all Operations : 127.93 In this report, although the percentage of clock periods holding issue has been reduced by only 6% (to 93%), the count of clock periods in hold-issue has fallen greatly, reduced from 114 million to 8.5 million. The ?gure for clock periods per instruction also has dropped from 279 to 22. This means that the 13:1 speedup of the code was largely due to removing hold-issue conditions. The value for CPU memory references per second has increased from 69 million per second to 868 million per second. The code used in the example performs mostly vector reads of memory and is, therefore, memory-bound. Any optimization that improves memory access time improves overall speed. In the corrected example, most of the hold-issue time is caused by memory activity, because this is the only activity of the code. In addition, although the report shows high memory activity, the average clock periods of con?icts per memory reference has been reduced from 12.75 to nearly 0. Similarly, in the hold-issue section of the report, although the percentage of clock periods spent waiting for memory ports remains high, the count of clock periods holding issue has been greatly reduced. 004–2182–003 283Guide to Parallel Vector Applications Instruction count values have not changed signi?cantly, except for an increase in the number of scalar operations performed per second. Apparently, slow vector memory references were holding up the overall speed of scalar operations. B.2.7 Calling Overhead This example shows the effects of calling overhead on the values given by the performance monitors. In the code that was used, a Fortran main program makes 100,000 calls to a subroutine that does no real work. If your code contains many calls to routines that do little or no work, the cost of housekeeping can be considerable. CPU seconds : 0.04 CP executing : 8490778 Million inst/sec (MIPS) : 74.13 Instructions : 2622498 Avg. clock periods/inst : 3.24 "% CP holding issue : 47.70 CP holding issue : 4049977 Inst.buffer fetches/sec : 0.01M Inst.buf. fetches: 248 Floating ops/sec : 0.00M F.P. ops : 1 Vector Floating ops/sec : 0.00M Vec F.P. ops : 0 CPU mem. references/sec : 22.89M actual refs : 809857 avg conflict/ref : 0.00 actual conflicts : 320 VEC mem. references/sec : 0.11M actual refs : 4068 "B/T mem. references/sec : 22.71M actual refs : 803409 I/O mem. references/sec : 0.34M actual refs : 11969 avg conflict/ref : 0.02 actual conflicts : 275 Hold issue condition % of all CPs actual # of CPs Waiting on A-regs & access : 9.52 808272 Waiting on S-regs & access : 0.28 24084 Waiting on V-registers : 0.00 376 "Waiting on B/T-registers : 37.86 3214297 Waiting on Functional Units : 0.00 342 Waiting on Shared Registers : 0.00 272 Waiting on Memory Ports : 0.03 2158 Waiting on Miscellaneous : 0.02 1286 (octal) instruction type inst./CPUsec actual inst. % of all insts. (000-004)Special : 0.00M 167 0.01 (005-017)Branch : 11.37M 402214 15.34 (02x,030-033)A Register : 53.93M 1908028 72.76 284 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] (034-037)B/T Memory : 5.67M 200624 7.65 (040-043,071-077)S Register : 2.94M 104096 3.97 (044-061)Scalar Integer : 0.14M 4872 0.19 (062-070)Scalar Floating-Point : 0.00M 1 0.00 (10x-13x)Scalar Memory : 0.07M 2380 0.09 (140-177)All Vector : 0.00M 116 0.00 type of vector operation ops/CPUsec actual ops Vector Logical : 0.01M 232 Vector Shift/Pop/LZ : 0.00M 0 Vector Integer Add : 0.00M 33 Vector Floating Multiply : 0.00M 0 Vector Floating Add : 0.00M 0 Vector Floating Reciprocal : 0.00M 0 Vector Memory Read : 0.04M 1350 Vector Memory Write : 0.08M 2718 Average Vector Length for all Operations : 37.35 The report shows some unusual statistics. First, notice the absence of ?oating-point operations. Although the code does nothing, 48% of all clock periods are spent holding issue, with 23 million CPU memory references per second. These statistics are caused by subroutine calls. Dividing 809,857 CPU memory references by 100,000 calls gives an average of 8 memory references per call. Out of a total 8.5 million clock periods (CPs) of execution time, 4.0 million CPs are hold-issue time and 2.6 million CPs are instruction issue. The remaining 1.8 million CPs are called blank NIP (next instruction parcel) time. The blank NIP time is shown only indirectly; it is not explicitly included in any statistics. It is the time spent jumping across instruction buffers fetching words from memory to load the buffers, and processing instructions that consist of more than 1 parcel. In this example, the blank NIP time represents 21% of all clock periods. A value in this range can occur in highly scalar code. Nearly all of the CPU memory references are indicated in the report as being B/T mem. references. The rate of these references of 23 million per second is extremely high. If this rate exceeds 1 million per second, it is cause for concern. 004–2182–003 285Guide to Parallel Vector Applications The number of instruction buffer fetches is quite low. In most examples, if much CPU time is spent in routine linkage, this number also will be outside the acceptable range of 1 million per second. See Section B.2.8, page 286, for an example of high instruction buffer fetches. The hold-issue section of the report shows that the only signi?cant hold-issue time was spent waiting for B/T registers. With a high-level language such as Fortran, the B/T registers are used primarily for the housekeeping that occurs at the beginning and end of each routine or function. B.2.8 Spaghetti Code The code for this example is similar to that used in the previous example, which did nothing but call a subroutine. In this example, however, many routines are called from all parts of memory. The principal difference between this example and the previous one is the much higher number of instruction buffer fetches (more than 5 million per second). This indicates code that is jumping excessively. CPU seconds : 1.91 CP executing : 459599735 Million inst/sec (MIPS) : 28.23 Instructions : 54054487 Avg. clock periods/inst : 8.50 % CP holding issue : 17.04 CP holding issue : 78308466 "Inst.buffer fetches/sec : 5.89M Inst.buf. fetches: 11283593 Floating ops/sec : 0.00M F.P. ops : 537 Vector Floating ops/sec : 0.00M Vec F.P. ops : 22 CPU mem. references/sec : 7.84M actual refs : 15018738 avg conflict/ref : 0.01 actual conflicts : 95654 VEC mem. references/sec : 0.00M actual refs : 4671 "B/T mem. references/sec : 7.84M actual refs : 15004729 I/O mem. references/sec : 0.61M actual refs : 1170699 avg conflict/ref : 0.09 actual conflicts : 110269 Hold issue condition % of all CPs actual # of CPs Waiting on A-regs & access : 1.29 5907886 Waiting on S-regs & access : 0.01 47035 Waiting on V-registers : 0.00 371 "Waiting on B/T-registers : 15.37 70632193 Waiting on Functional Units : 0.00 848 Waiting on Shared Registers : 0.00 420 Waiting on Memory Ports : 0.00 2388 286 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] Waiting on Miscellaneous : 0.55 2531174 (octal) instruction type inst./CPUsec actual inst. % of all insts. (000-004)Special : 1.04M 1999292 3.70 (005-017)Branch : 7.31M 14005517 25.91 (02x,030-033)A Register : 17.24M 33016997 61.08 (034-037)B/T Memory : 2.09M 4000919 7.40 (040-043,071-077)S Register : 0.53M 1010343 1.87 (044-061)Scalar Integer : 0.01M 11410 0.02 (062-070)Scalar Floating-Point : 0.00M 515 0.00 (10x-13x)Scalar Memory : 0.00M 9338 0.02 (140-177)All Vector : 0.00M 156 0.00 type of vector operation ops/CPUsec actual ops Vector Logical : 0.00M 809 Vector Shift/Pop/LZ : 0.00M 0 Vector Integer Add : 0.00M 577 Vector Floating Multiply : 0.00M 11 Vector Floating Add : 0.00M 11 Vector Floating Reciprocal : 0.00M 0 Vector Memory Read : 0.00M 1938 Vector Memory Write : 0.00M 2733 Average Vector Length for all Operations : 38.97 The report shows even more unusual statistics than the previous example. For instance, out of a total 460 million CPs of execution time, 78 million CPs are hold-issue time and 54 million CPs are instruction issue. The remaining 327 million CPs are blank NIP time. This very high amount of time accounts for 71% of all clock periods. The rate of instruction buffer fetches is high, exceeding 5 million per second. This is outside the acceptable range of 1 million per second. As in the previous example, nearly all of the memory activity is B/T mem. references, which are used for routine linkage housekeeping. Most of the time spent holding issue also is accountable to this type of memory activity. 004–2182–003 287Guide to Parallel Vector Applications B.2.9 Autotasked Example The following example shows the behavior of a code using several CPUs that has been autotasked by using the f90 command. CPU seconds : 3.88 CP executing : 931929516 Wall-Clock (RT) seconds : 1.78 RT Clock Periods : 427305348 Million inst/sec (MIPS) : 40.30 Instructions : 156468424 Avg. clock periods/inst : 5.96 % CP holding issue : 74.03 CP holding issue : 689914474 Inst.buffer fetches/sec : 0.39M Inst.buf. fetches: 1497195 Floating ops/sec : 257.53M F.P. ops : 1000000807 "Vector Floating ops/sec : 257.53M Vec F.P. ops : 1000000076 CPU mem. references/sec : 313.59M actual refs : 1217684051 avg conflict/ref : 0.07 actual conflicts : 79857119 "VEC mem. references/sec : 309.43M actual refs : 1201526222 B/T mem. references/sec : 1.00M actual refs : 3881418 I/O mem. references/sec : 0.16M actual refs : 618546 avg conflict/ref : 0.18 actual conflicts : 111199 CPU/wallclock time ratio : 2.18 Floating ops/wall sec : 561.66M "Vector Floating ops/wall sec : 561.66M CPU mem. references/wall sec : 683.92M "VEC mem. references/wall sec : 674.85M Inst/wall sec : 87.88M Hold issue condition % of all CPs actual # of CPs Waiting on A-regs & access : 4.75 44223248 Waiting on S-regs & access : 7.11 66285942 Waiting on V-registers : 26.98 251422578 Waiting on B/T-registers : 0.74 6894233 Waiting on Functional Units : 22.49 209621573 "Waiting on Shared Registers : 16.87 157236923 Waiting on Memory Ports : 5.17 48151651 Waiting on Miscellaneous : 2.70 25171269 (octal) instruction type inst./CPUsec actual inst. % of all insts. (000-004)Special : 2.59M 10051789 6.42 (005-017)Branch : 2.83M 10979622 7.02 (02x,030-033)A Register : 16.76M 65060954 41.58 (034-037)B/T Memory : 0.10M 385353 0.25 288 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] (040-043,071-077)S Register : 3.64M 14133552 9.03 (044-061)Scalar Integer : 6.25M 24274391 15.51 (062-070)Scalar Floating-Point : 0.00M 731 0.00 (10x-13x)Scalar Memory : 3.16M 12276411 7.85 (140-177)All Vector : 4.97M 19305621 12.34 type of vector operation ops/CPUsec actual ops Vector Logical : 0.30M 1159708 Vector Shift/Pop/LZ : 0.00M 33 Vector Integer Add : 0.00M 691 Vector Floating Multiply : 206.02M 800000043 Vector Floating Add : 0.00M 33 Vector Floating Reciprocal : 51.51M 200000000 Vector Memory Read : 206.24M 800855007 Vector Memory Write : 103.18M 400671215 type of vector operation ops/wall sec Vector Logical : 0.65M Vector Shift/Pop/LZ : 0.00M Vector Integer Add : 0.00M Vector Floating Mult : 449.33M Vector Floating Add : 0.00M Vector Float Recip : 112.33M Vector Memory Read : 449.81M Vector Memory Write : 225.04M Average Vector Length for all Operations : 114.10 The hpm command can measure the wall-clock time used by the command being measured, so this example shows the various statistics calculated by both total CPU time and by wall-clock time. The command observed a speedup of 2.18 for this autotasked code. The actual speedup is probably greater, but that value can be measured precisely only by using either the ja or atexpert command. If you are concerned about parallel speedup, you should use those commands. The primary statistic that sets this example apart from other examples in this section is the time spent holding issue for shared registers. Because the Autotasking system uses the semaphores and shared registers to synchronize parallel calculations, some hold-issue time can be expected. 004–2182–003 289Guide to Parallel Vector Applications B.2.10 Typical User Code The following example shows the behavior of a typical user code. This code was devised, and is used daily, in a research environment. The code shows behavior typical of real world code. CPU seconds : 256.52 CP executing : 61564418203 "Million inst/sec (MIPS) : 81.52 Instructions : 20911890509 Avg. clock periods/inst : 2.94 CP% holding issue : 55.14 CP holding issue : 33947624924 Inst.buffer fetches/sec : 0.06M Inst.buf. fetches: 15631254 "Floating ops/sec : 40.11M F.P. ops : 10288629446 "Vector Floating ops/sec : 16.91M Vec F.P. ops : 4338077728 "CPU mem. references/sec : 28.52M actual refs : 7316630726 avg conflict/ref : 0.14 actual conflicts : 1031124126 "VEC mem. references/sec : 14.21M actual refs : 3644894624 "B/T mem. references/sec : 4.80M actual refs : 1231341654 I/O mem. references/sec : 0.51M actual refs : 130266701 avg conflict/ref : 0.06 actual conflicts : 8153558 Hold issue condition % of all CPs actual # of CPs Waiting on A-regs & access : 3.10 1908294742 "Waiting on S-regs & access : 45.35 27918895976 Waiting on V-registers : 2.01 1235847809 Waiting on B/T-registers : 2.56 1574297844 Waiting on Functional Units : 1.52 937317555 Waiting on Shared Registers : 0.00 1798 Waiting on Memory Ports : 0.92 563897101 Waiting on Miscellaneous : 0.77 474435394 (octal) instruction type inst./CPUsec actual inst. % of all insts. (000-004)Special : 0.09M 22094503 0.11 (005-017)Branch : 4.10M 1052771308 5.03 (02x,030-033)A Register : 9.05M 2321288981 11.10 (034-037)B/T Memory : 0.11M 28006531 0.13 (040-043,071-077)S Register : 29.38M 7536399207 36.04 (044-061)Scalar Integer : 5.24M 1343537793 6.42 "(062-070)Scalar Floating-Point : 23.20M 5950551718 28.46 (10x-13x)Scalar Memory : 9.51M 2440394448 11.67 (140-177)All Vector : 0.85M 216846020 1.04 290 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] type of vector operation ops/CPUsec actual ops "Vector Logical : 3.37M 864016537 "Vector Shift/Pop/LZ : 1.94M 496918284 "Vector Integer Add : 2.37M 607405554 "Vector Floating Multiply : 8.05M 2065989380 "Vector Floating Add : 7.91M 2027781913 "Vector Floating Reciprocal : 0.95M 244306435 "Vector Memory Read : 9.58M 2458642233 "Vector Memory Write : 4.62M 1186252391 "Average Vector Length for all Operations : 45.89 The code is dominated by scalar operations. This conclusion can be drawn by observing a high MIPS rate (82 million per second), coupled with a low ?oating-point operation rate (40 million per second). The low vector ?op rate (17 million per second) con?rms that scalar calculations dominate. The actual number of vector ?oating-point operations performed is nearly the same as scalar operations. Vector operations appear to be performed ef?ciently, but they are slowed down by the need to wait for calculation of scalar results. This observation is con?rmed by the high hold-issue time for scalar (S) registers and the lower hold-issue time for vector registers and functional units. Because this code mixes vector and scalar, it is not clear which part of the code is performing ef?ciently and which is not. Further study of this code requires the use of a tool that can isolate which routines in the code use too much CPU time, as well as which parts of those routines use the time. After the routines and sections have been identi?ed, you can study them to determine whether their speed can be increased through vectorization and other optimization techniques. Typical tools to aid in this isolation would be Pro?ling (using the prof or profview commands), Flowtrace, Perftrace, or Jumptrace. B.2.11 Nearly Peak Speed The following example shows an arti?cial code that runs at nearly the peak machine speed for ?oating-point operations. CPU seconds : 1.13 CP executing : 272061988 Million inst/sec (MIPS) : 19.41 Instructions : 22006465 Avg. clock periods/inst : 12.36 004–2182–003 291Guide to Parallel Vector Applications % CP holding issue : 85.64 CP holding issue : 232994424 Inst.buffer fetches/sec : 0.00M Inst.buf. fetches: 550 Floating ops/sec : 1354.99M F.P. ops : 1536000769 "Vector Floating ops/sec : 1354.99M Vec F.P. ops : 1536000768 CPU mem. references/sec : 0.00M actual refs : 4948 avg conflict/ref : 0.03 actual conflicts : 147 VEC mem. references/sec : 0.00M actual refs : 3984 B/T mem. references/sec : 0.00M actual refs : 246 I/O mem. references/sec : 0.08M actual refs : 85744 avg conflict/ref : 0.03 actual conflicts : 2425 Hold issue condition % of all CPs actual # of CPs Waiting on A-regs & access : 0.00 2009 Waiting on S-regs & access : 1.84 5006459 "Waiting on V-registers : 81.88 222769210 Waiting on B/T-registers : 0.00 565 "Waiting on Functional Units : 83.80 227983274 Waiting on Shared Registers : 0.00 95 Waiting on Memory Ports : 0.00 1372 Waiting on Miscellaneous : 0.00 848 (octal) instruction type inst./CPUsec actual inst. % of all insts. (000-004)Special : 0.00M 88 0.00 (005-017)Branch : 4.41M 5000703 22.72 (02x,030-033)A Register : 0.00M 2047 0.01 (034-037)B/T Memory : 0.00M 64 0.00 (040-043,071-077)S Register : 0.00M 1118 0.01 (044-061)Scalar Integer : 4.41M 5001632 22.73 (062-070)Scalar Floating-Point : 0.00M 1 0.00 (10x-13x)Scalar Memory : 0.00M 718 0.00 (140-177)All Vector : 10.59M 12000094 54.53 type of vector operation ops/CPUsec actual ops Vector Logical : 0.00M 744 Vector Shift/Pop/LZ : 0.00M 0 Vector Integer Add : 0.00M 33 "Vector Floating Multiply : 451.66M 512000256 "Vector Floating Add : 451.66M 512000256 "Vector Floating Reciprocal : 451.66M 512000256 Vector Memory Read : 0.00M 1306 292 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] Vector Memory Write : 0.00M 2678 "Average Vector Length for all Operations : 128.00 This code tries to run the vector ?oating-point functional units at peak speed. It comes quite close, achieving 1354.99 MFLOPS (1.355 GFLOPS). The code can achieve this kind of speed because it: • Has vectors of length 128 • Makes no memory references • Schedules ?oating-point functional units and vector registers so that con?icts are minimized • Keeps the kernel loop code within one instruction buffer If any of these items were altered, such as making less ef?cient use of functional units, this calculation rate could be reduced. B.2.12 Vectorized Logical Operations (with AVL) This example shows how vector logical operations can be overlapped when the AVL unit is enabled. The companion example shows the same code, with the AVL disabled. The code used in this example performs two chained vectorized logical operations, such as AND and OR, using data read from memory, and returning results to memory. (Such operations are not typical of most application codes that run on UNICOS systems.) CPU seconds : 0.35 CP executing : 83882527 Million inst/sec (MIPS) : 46.12 Instructions : 16118303 Avg. clock periods/inst : 5.20 % CP holding issue : 71.14 CP holding issue : 59676950 Inst.buffer fetches/sec : 0.00M Inst.buf. fetches: 337 Floating ops/sec : 0.03M F.P. ops : 10126 Vector Floating ops/sec : 0.03M Vec F.P. ops : 10048 "CPU mem. references/sec : 858.69M actual refs : 300120954 avg conflict/ref : 0.11 actual conflicts : 31776208 "VEC mem. references/sec : 858.44M actual refs : 300034257 B/T mem. references/sec : 0.24M actual refs : 83850 004–2182–003 293Guide to Parallel Vector Applications I/O mem. references/sec : 0.04M actual refs : 12672 avg conflict/ref : 0.41 actual conflicts : 5163 Hold issue condition % of all CPs actual # of CPs Waiting on A-regs & access : 3.84 3220329 Waiting on S-regs & access : 0.03 27476 "Waiting on V-registers : 64.38 54003765 Waiting on B/T-registers : 0.39 331097 Waiting on Functional Units : 0.00 494 Waiting on Shared Registers : 0.00 285 "Waiting on Memory Ports : 41.67 34954111 Waiting on Miscellaneous : 2.83 2373746 (octal) instruction type inst./CPUsec actual inst. % of all insts. (000-004)Special : 2.30M 805279 5.00 (005-017)Branch : 2.37M 827722 5.14 (02x,030-033)A Register : 27.75M 9700530 60.18 (034-037)B/T Memory : 0.06M 20705 0.13 (040-043,071-077)S Register : 2.30M 804799 4.99 (044-061)Scalar Integer : 0.02M 5772 0.04 (062-070)Scalar Floating-Point : 0.00M 78 0.00 (10x-13x)Scalar Memory : 0.01M 2847 0.02 (140-177)All Vector : 11.30M 3950571 24.51 type of vector operation ops/CPUsec actual ops "Vector Logical : 572.14M 199969701 Vector Shift/Pop/LZ : 0.00M 28 Vector Integer Add : 0.03M 10110 Vector Floating Multiply : 0.00M 24 Vector Floating Add : 0.03M 10024 Vector Floating Reciprocal : 0.00M 0 Vector Memory Read : 572.26M 200011511 Vector Memory Write : 286.18M 100022746 Average Vector Length for all Operations : 126.57 The beginning of the report outputs very little ?oating-point activity, indicating the code does not have a high FLOP rate. CPU memory rate and vector memory rate indicate ef?cient code. 294 004–2182–003Interpreting Performance Statistics for Cray C90 Series Systems [B] Further down in the report shows that the high vector memory rate is coupled with a high rate of vector logical operations. Hold-issue statistics show that executing in two independent vector logical functional units is the bene?t to this example code. The primary hold-issue times are associated with vector registers and memory ports. This can be contrasted with the next example, in which only one vector logical functional unit is available. In that case, the primary hold-issue time is for the functional unit. B.2.13 Vectorized Logical Operations (AVL Disabled) This example shows how vectorized logical operations behave when the AVL unit is disabled. CPU seconds : 0.47 CP executing : 112130833 Million inst/sec (MIPS) : 34.50 Instructions : 16118303 Avg. clock periods/inst : 6.96 % CP holding issue : 78.41 CP holding issue : 87925262 Inst.buffer fetches/sec : 0.00M Inst.buf. fetches: 339 Floating ops/sec : 0.02M F.P. ops : 10126 Vector Floating ops/sec : 0.02M Vec F.P. ops : 10048 CPU mem. references/sec : 642.37M actual refs : 300120954 avg conflict/ref : 0.10 actual conflicts : 31325373 "VEC mem. references/sec : 642.18M actual refs : 300034257 B/T mem. references/sec : 0.18M actual refs : 83850 I/O mem. references/sec : 0.50M actual refs : 234720 avg conflict/ref : 0.65 actual conflicts : 153458 Hold issue condition % of all CPs actual # of CPs Waiting on A-regs & access : 2.87 3220390 Waiting on S-regs & access : 0.02 27377 "Waiting on V-registers : 24.12 27042356 Waiting on B/T-registers : 0.35 394618 "Waiting on Functional Units : 73.69 82627979 Waiting on Shared Registers : 0.00 290 Waiting on Memory Ports : 0.04 48202 Waiting on Miscellaneous : 2.12 2373734 (octal) instruction type inst./CPUsec actual inst. % of all insts. 004–2182–003 295Guide to Parallel Vector Applications (000-004)Special : 1.72M 805279 5.00 (005-017)Branch : 1.77M 827722 5.14 (02x,030-033)A Register : 20.76M 9700530 60.18 (034-037)B/T Memory : 0.04M 20705 0.13 (040-043,071-077)S Register : 1.72M 804799 4.99 (044-061)Scalar Integer : 0.01M 5772 0.04 (062-070)Scalar Floating-Point : 0.00M 78 0.00 (10x-13x)Scalar Memory : 0.01M 2847 0.02 (140-177)All Vector : 8.46M 3950571 24.51 type of vector operation ops/CPUsec actual ops "Vector Logical : 428.07M 200000421 Vector Shift/Pop/LZ : 0.00M 28 Vector Integer Add : 0.02M 10110 Vector Floating Multiply : 0.00M 24 Vector Floating Add : 0.02M 10024 Vector Floating Reciprocal : 0.00M 0 Vector Memory Read : 428.10M 200011511 Vector Memory Write : 214.08M 100022746 Average Vector Length for all Operations : 126.58 This example is the same code as the previous example. However, now only one vector logical functional unit is available. The AVL was disabled by running the program under the following command: /etc/cpu -m avloff The vector memory rate is lower (642 million per second, instead of 858) than in the previous example. In this case, results can be sent to memory through only a single functional unit, rather than chaining them through two functional units. This behavior also is re?ected in the rate of vector logical operations, dropping from 572 million per second to 428 million per second. In the hold-issue section of the report, it is clear that this code is now waiting for the vector logical functional unit more than any other resource. The code performs two logical operations, one after the other, clogging up the single functional unit that is needed to perform the operations. 296 004–2182–003Index A -A option ?owview command, 69 jumpview command, 105 perfview command, 138 profview command, 222 -a option ?owview command, 69 perfview command, 138 procview command, 201 profview command, 223 Aborted programs ?odump command, 90 perfdmp command, 161 Accelerators ATExpert, 16 Accuracy CPU time Flowtrace, 91 jumpview, 129 Perftrace, 159 determinants, 2 statistics Perftrace, 165 Pro?ling, 221 Address buckets, 220, 247 Amdahl’s Law, 51 Amount and speed of a program identifying, 9 args option hpm command, 99 ASCII interface procview command, 212 use with ATExpert, 44 use with ?owview command, 83 use with jumpview command, 127 use with perfview command, 155 use with profview command, 245 ATExpert accelerators, 16 code instrumentation, 24 command predictions for single CPU, 65 con?guration, 42 CPU speci?cation from the command line, 23 X Window System interface, 43 CRI logo, 17 demonstration ?le, 29 description, 50 determining unitasked time, 33 CASE statements, 34 Display menu, 30 displaying data ASCII interface, 44 File menu, 27 ?lter options, 29, 43 generating reports ASCII interface, 45, 46 command option, 23 X Window System interface, 39 graphing information, 36, 43 help ?le printing from the command line, 23 Help menu, 17 Information panel, 27 loop timing, 33 main window, 22 observations ASCII interface, 24 X Window System interface, 38 Options menu, 42 overhead, 19, 51, 53 overview, 19 Parallel Region Monitor, 25 parallel time, 32 peel-off menus, 16 004–2182–003 297Guide to Parallel Vector Applications performance improvement suggestions, 35 raw output ?le contents, 58 report formatting, 42 running as background process, 20 serial time displaying versus parallel, from the command line, 23 ending, 32 preceding, 32 software anomalies, 64 source ?le viewing, 41 specifying raw-format ?le ASCII interface, 44 from the command line, 23 X Window System interface, 29 speedup curves, 63 window, 26, 43 threshold test effect, 65 failure, 43 use of Amdahl’s Law with, 51 using ASCII interface with, 44 using X Window System with, 25 variable work loops, 62 version, 17, 58 Version selection, 18 atexpert command command syntax, 23 description, 3 impact, 7 interfaces, 6 interpretation with variable iterations, 66 syntax, 23 atx.raw output ?le contents, 58 Autotasking utility, 19 Autotasking example, 288 Average vector length, 271 AVL disabled, vectorized Logical operations, 295 enabled vectorized logical operations, 293 B -B option jumpview command, 106 perfview command, 137 profview command, 223 -b option profview command, 223 B/T memory references Cray C90, 265 Buckets address, 220, 247 C C code instrumentation ATExpert, 24 Flowtrace feature, 73 Jumptrace feature, 114 Pro?ling feature, 234 -C option jumpview command, 106 perfview command, 138 -c option ?owview command, 69 jumpview command, 105 perfview command, 138 procview command, 201 profview command, 222, 226 Call tree ?owview command, 70 Calling overhead Cray C90, 284 298 004–2182–003Index Code performs in one vector functional unit, 276 performs in two vector functional units, 278 spaghetti, 286 typical user, 290 vectorized, 274 Code instrumentation ATExpert, 24 Flowtrace feature, 72 Jumptrace feature, 113 Pro?ling feature, 233 command option procstat command, 204 Command syntax ATExpert, 23 ?owview command, 68 hpm command, 98 jt command, 111 jumpview command, 105 procview command, 198 prof command, 227 profview command, 222 commands option jt command, 112 Compiler debug symbols jt command, 114 jumpview command, 114 prof command, 233 profview command, 233 Con?dence level profview command, 221 Con?guration ATExpert, 42 Con?ict removed memory, 282 Con?icts extensive memory bank, 280 CPU memory references, 263 overhead, 6 time Flowtrace, 91 jumpview, 129 Perftrace, 159 routine using the most, 8 used by one routine, 8 used in library routines, 8 used in one loop or code section, 8 CPU speci?cation ATExpert from the command line, 23 X Window System interface, 43 Cray C90 Autotasking example, 288 average vector length, 271 B/T memory references, 265 calling overhead, 284 code that performs in one vector functional unit, 276 code that performs in two vector functional units, 278 CPU memory references, 263 extensive memory bank con?icts, 280 hardware performance monitor statistics examples, 272 hold-issue statistics, 270, 265 topics, 266 hpm command, 259 HPM counters, 260 I/O memory references, 264 instruction summary, 267 mega?op ratings, 263 memory con?ict removed, 282 peak speed nearly, 291 scalar code example, 272 spaghetti code, 286 typical user code, 290 vector operations, 269 vectorized code, 274 vectorized logical operations with AVL disabled, 295 with AVL enabled, 293 CRI logo 004–2182–003 299Guide to Parallel Vector Applications ATExpert, 17 D -D option ?owview command, 72 jumpview command, 106 perfview command, 141 profview command, 222 -d option hpm command, 98 procstat command, 204 profview command, 223 Demonstration ?le ATExpert, 29 ?owview, 77 perfview, 148 profview, 238 DISPLAY environment variable, 10, 14 Flowtrace feature, 93 Perftrace feature, 165 Pro?ling feature, 235 Display menu ATExpert, 30 E -E option procview command, 201 -e option ?owview command, 70 perfview command, 140 procview command, 202 profview command, 224 Ending serial time determining with ATExpert, 32 Environment variables Flowtrace feature, 90 Perftrace feature, 162 prof command, 231 Examples Autotasking, 288 enabling Pro?ling, 234 hpm output, 99 scalar code, 272 Exporting data ?owview, 70 jumpview, 106 perfview, 139 procview, 201 profview, 224 F -F option perfview command, 142 procview command, 199 prof command, 228 -f option prof command, 227 File menu ATExpert, 27 ?owview command, 76 jumpview command, 117 perfview command, 147 procview command, 206 profview command, 237 Filter options ATExpert, 29, 43 ?odump command description, 90 FLOWMARK subroutine description, 3, 84 Flowtrace feature examples, 85 impact, 7 Perftrace feature description, 157 examples, 158 Flowtrace feature additional features, 84 code instrumentation, 73 CPU time 300 004–2182–003Index accuracy, 91 description, 3, 67 DISPLAY environment variable, 93 enabling in C, 73 in Fortran, 73 environment variables, 90 ?odump command , 90 ?owview command, 67, 74 instrumenting your code, 72 output, 74 selective tracing, 84 directives, 87 examples, 87 FLOWMARK, 84 SETPLIMQ subroutine, 88 Flowtrace library signal handling, 92 ?owview demonstration ?le, 77 ?owview command command syntax, 68 description, 3 examples, 93 File menu, 76 Graphs menu, 81 interfaces, 6 options, 68, 71 producing a report, 67, 74 Reports menu, 78 sample output, 93 syntax, 68 use of vi windows with, 84 using ASCII interface with, 83 Fortran code instrumentation Flowtrace feature, 73 Jumptrace feature, 114 Pro?ling, 234 common variables, 9 Fortran 90 code instrumentation ATExpert, 24 Flowtrace, 73 hpm command, 99 Pro?ling, 234 Functional units code that performs in one vector, 276 code that performs in two vectors, 278 G -G option jumpview command, 107 -g option hpm command, 98 Graphing information ATExpert, 36 Graphs menu ?owview command, 81 jumpview command, 125 perfview command, 154 procview command, 211 profview command, 244 H -H option ?owview command, 70 jumpview command, 105 perfview command, 139 procview command, 201 profview command, 224 -h option ?owview command, 69 perfview command, 137 profview command, 222 Hardware Performance Monitor, 97 counters, 251 report groups, 143 Hardware use ef?ciency, 8 Help ?le printing 004–2182–003 301Guide to Parallel Vector Applications ATExpert from the command line, 23 Help menu ATExpert, 17 Hold-issue statistics, 270, 265 topics, 266 hpm command and Perftrace feature, 97 command syntax, 98 counters, 251 Cray C90 interpreting statistics, 259 see also Hardware Performance Monitor, 259 description, 3, 97, 98 examples, 99 multiple execution, 101 output, 99 output with the perfview command, 97 syntax, 98 using, 98 HPM statistics Cray C90, 260 hpmall command interfaces, 6 I -i option procstat command, 204 profview command, 223 I/O amount and speed, 9 memory references Cray C90, 264 Impact of tools, 6 Inaccuracy causes of, 2 Information panel ATExpert, 27 Inlining identifying routines for, 8 routines that should be considered for, 8 Instruction summary, 267 Instrumenting your code ATExpert, 24 Flowtrace feature, 72 Pro?ling feature, 233 Interfaces, 5 Introduction, 1 J -J option jt command, 111 -j option ?owview command, 70 ja command impact, 7 interfaces, 6 jt command command syntax, 111 compiler debug symbols, 114 description, 3 gathering data, 111 syntax, 111 Jumptrace feature description, 4, 103 enabling in C, 114 impact, 7 instrumenting your code, 113 jt command, 111 jumpview command, 104 jumpview command command syntax, 105 compiler debug symbols, 114 CPU time, 129 description, 3, 104 examples, 129 File menu, 117 Graphs menu, 125 302 004–2182–003Index interfaces, 6 options, 105, 109 output, 129 Reports menu, 119 symbols, 112 syntax, 105 using the ASCII interface with, 127 vi windows, 128 X defaults, 115 K -k option ?owview command, 70 L -L option ?owview command, 68 jumpview command, 106 perfview command, 137 procview command, 198 profview command, 222 -l option ?owview command, 69 perfview command, 139 procview command, 202 prof command, 227 Length average vector, 271 libperf.a library enabling Perftrace, 135 libprof.a library enabling Pro?ling, 220 Logical operations with AVL disabled, vectorized, 295 with AVL enabled, vectorized, 293 Loop timing determining with ATExpert, 33 M -M option perfview command, 138 -m option ?owview command, 69 perfview command, 140 procstat command, 204 prof command, 227 profview command, 223 Mega?op ratings Cray C90, 263 Memory amount used, 9 bank con?icts extensive, 280 con?ict removed, 282 Memory bank con?icts extensive, 280 Memory references B/T, 265 CPU, 263 I/O, 264 mtdump command interfaces, 6 N -N option ?owview command, 71 -n option procview command, 202 profview command, 226 NCPUS environment variable Flowtrace feature, 92 O -O option ?owview command, 72 004–2182–003 303Guide to Parallel Vector Applications jumpview command, 109 procview command, 201 -o option ?owview command, 71 hpm command, 98 jt command, 111 profview command, 226 Observations ATExpert ASCII interface, 24 X Window System interface, 38 Operations vector, 269 Optimizing prof command, 247 Options menu ATExpert, 42 Output ?owview command, 93 hpm command, 99 jumpview command, 129 procview command, 214 profview command, 248 user interfaces, 5 Overhead ATExpert, 19, 51, 53 calling, 284 CPU, 6 Perftrace feature, 164 P -P option procview command, 200 -p option ?owview command, 71 prof command, 228 profview command, 225 Paralled processing identifying ef?ciency of, 8 Parallel Region Monitor ATExpert, 25 Parallel regions displaying timing information, 32 Parallel time as overhead, 52 determining with ATExpert, 32 sum of, 32 Parallelism ATExpert, 25 underlying Flowtrace feature, 92 unexploited, 52 Peak speed nearly, 291 Peel-off menus ATExpert, 16 Performance accuracy causes of, 2 measurement and, 2 improvement suggestions ATExpert, 35 inaccuracy causes of, 2 measurement and, 2 measurement and accuracy, 2 tuning, 2 Perftrace feature additional features, 157 and hpm command, 97 description, 4, 135 DISPLAY environment variable, 165 enabling in C, 144 in Fortran, 144 environment variables, 162 impact, 7 overhead, 164 perfdmp command, 161 perfview command, 136 selective tracing, 157 FLOWMARK, 157 SETPLIMQ subroutine, 161 304 004–2182–003Index statistics, 164 throttle feature, 163 Perftrace library signal handling, 164 perfview demonstration ?le, 148 perfview command description, 4 examples, 190 File menu, 147 Graphs menu, 154 hpm command output, 97 interfaces, 6 options, 136, 141 output sample, 190 user-de?ned, 166 producing reports, 136 reports command-line observation, 166 command-line user-de?ned, 165 Reports menu, 149 sample output, 190 syntax, 137 user-de?ned reports examples, 181 prede?ned variable, 173 processing of variables, 178 report generation, 181 units designators, 179 using, 136 using ASCII interface with, 155 using X Window System with, 145 vi windows, 156 X defaults, 145 Preceding serial time determining with ATExpert, 32 procstat command description, 4, 197 impact, 7 interfaces, 6 syntax, 203 using, 197, 203 procview command command syntax, 198 description, 4 File menu, 206 Graphs menu, 211 interfaces, 6 options, 198, 202 output, 214 reports, 214 Reports menu, 208 syntax, 198 using, 198 using the ASCII interface with, 212 using X Window System with, 204 vi windows, 213 X defaults, 204 prof command command syntax, 227 compiler debug symbols, 233 enabling Pro?ling, 220 environment variables, 231 examples output, 228 report, 229 gathering Pro?ling data, 226 options, 227 symbols used by, 230 syntax, 227 usage considerations, 247 strategies, 247 prof library, 233 Pro?ling feature code instrumentation, 233 description, 4, 219 DISPLAY environment variable, 235 enabling, 220 examples, 234 gathering data, 226 impact, 7 libprof.a library, 220 prof command, 227 004–2182–003 305Guide to Parallel Vector Applications profview command, 220 Pro?ling segmented programs, 220 profview demonstration ?le, 238 profview command command syntax, 222 compiler debug symbols, 233 description, 4, 221 File menu, 237 Graphs menu, 244 interfaces, 6 options, 222, 225 output, 248 Reports menu, 239 syntax, 222 using ASCII interface with, 245 using X Window System with, 235 vi windows, 246 X defaults, 235 program option hpm command, 99 Program’s call tree identifying, 8 Programs pro?ling segmented, 220 R -R option ?owview command, 71 jumpview command, 107 procstat command, 203 -r option hpm command, 98 jumpview command, 105 procstat command, 203 prof command, 228 profview command, 225 Ratings mega?op, 263 raw?le option ?owview command, 71 jumpview command, 108 perfview command, 141 procview command, 202 profview command, 224 References B/T memory, 265 CPU memory, 263 I/O memory, 264 Reports formatting ATExpert, 42 generating with ATExpert, 23, 39, 45 generating with jumpview command, 111 generating with the ?owview command, 67, 74 generating with the jumpview command, 109 generating with the procview command, 214 generating with the prof command, 221, 228, 247 generating with the profview command, 221 Reports menu ?owview command, 78 jumpview command, 119 perfview command, 149 procview command, 208 profview command, 239 S -S option ?owview command, 72 jt command, 111 jumpview command, 105 procview command, 199 -s option profview command, 225 Scalar code example, 272 Segmented programs pro?ling, 220 Selective tracing Perftrace, 157 306 004–2182–003Index Serial time as overhead, 52 determining, 47 ending, 32 preceding, 32 Serial versus parallel time display ATExpert from the command line, 23 SETPLIMQ subroutine examples, 89 Flowtrace feature, 88 Perftrace feature, 161 Signal handling in the Flowtrace library, 92 in the Perftrace library, 164 Software anomalies ATExpert, 64 Source ?le viewing ATExpert, 41 Spaghetti code Cray C90, 286 Speed nearly peak , 291 Speedup curves, ATExpert, 63 window, ATExpert, 26, 43 Statistics hold-issue, 270, 265 HPM, 260 subjprog option prof command, 228 Subroutines displaying in ATExpert, 31 Summary instruction, 267 Symbols jumpview command, 112 prof command, 230 T -T option ?owview command, 70 jumpview command, 108 -t option profview command, 225 Threshold test effect ATExpert, 65 Threshold test failure ATExpert, 43 Throttle feature Perftrace, 163 time command interfaces, 6 Timing information ATExpert loops, 33 parallel regions, 32 subroutines, 31 Tools described in other manuals ftnlist, 5 hpmall, 5 hpm?op, 5 ja, 5 mtdump, 5 SECOND, 5 time, 5 xbrowse, 5 described in this manual atexpert, 3 FLOWMARK, 3 Flowtrace, 3 ?owview, 3 hpm, 3 jt, 3 Jumptrace, 3 jumpview, 3 Perftrace, 4 perfview, 4 procstat, 4 004–2182–003 307Guide to Parallel Vector Applications procview, 4 Pro?ling, 4 profview, 4 interfaces atexpert, 6 ?owview, 6 hpm, 6 hpmall, 6 ja, 6 jumpview, 6 mtdump, 6 perfview, 6 procstat, 6 procview, 6 profview, 6 time, 6 Topics hold-issue, 266 U -U option perfview command, 140, 142 -u option ?owview command, 68 perfview command, 137 Unexploited parallelism, 52 Unitasked time CASE statements determining using ATExpert, 34 determining using ATExpert, 33 User code typical, 290 User interfaces, 5 V -V option ?owview command, 71 hpm command, 98 jt command, 111 jumpview command, 106 perfview command, 140 procview command, 202 prof command, 227 profview command, 224 -v option ?owview command, 69 perfview command, 140 procstat command, 204 profview command, 225 Variable work loops ATExpert, 62 Vector functional units code that performs in one, 276 code that performs in two, 278 operations, 269 Vector length average , 271 Vectorized code, 274 logical operations with AVL disabled, 295 with AVL enabled, 293 Version selection ATExpert, 18 vi windows ?owview command, 84 jumpview command, 128 perfview command, 156 procview command, 213 profview command, 246 W -w option ?owview command, 69 procview command, 202 308 004–2182–003Index X X defaults jumpview command, 115 perfview command, 145 procview command, 204 profview command, 235 X Widgets, 13 X Window System color, 14 common options, 12 ?owview command, 74 getting started, 10 .xinitrc ?le, 10 perfview command, 145 pointing device, 15 procview command, 204 profview command, 235 resources, 13 .Xdefaults ?le, 13 troubleshooting, 14 .xhost ?le, 14 use with ATExpert, 25 -X option ?owview command, 70 jumpview command, 106 procview command, 201 profview command, 224 -x option perfview command, 140 procview command, 202 prof command, 227 Y -y option ?owview command, 71 procview command, 202 profview command, 223 Z -z option ?owview command, 72 procview command, 201 profview command, 223 004–2182–003 309 CrayDoc™ Installation and Administration Guide S–2340–21© 2001–2003 Cray Inc. All Rights Reserved. This manual or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Inc. U.S. GOVERNMENT RESTRICTED RIGHTS NOTICE The Computer Software is delivered as "Commercial Computer Software" as defined in DFARS 48 CFR 252.227-7014. All Computer Software and Computer Software Documentation acquired by or for the U.S. Government is provided with Restricted Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7014, as applicable. Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7013, as applicable. Autotasking, CF77, Cray, Cray Ada, Cray Channels, Cray Chips, CraySoft, Cray Y-MP, Cray-1, CRInform, CRI/TurboKiva, HSX, LibSci, MPP Apprentice, SSD, SuperCluster, UNICOS, UNICOS/mk, and X-MP EA are federally registered trademarks and Because no workstation is an island, CCI, CCMT, CF90, CFT, CFT2, CFT77, ConCurrent Maintenance Tools, COS, Cray Animation Theater, Cray APP, Cray C90, Cray C90D, Cray CF90, Cray C++ Compiling System, CrayDoc, Cray EL, Cray Fortran Compiler, Cray J90, Cray J90se, Cray J916, Cray J932, CrayLink, Cray MTA, Cray MTA-2, Cray MTX, Cray NQS, Cray/REELlibrarian, Cray S-MP, Cray SSD-T90, Cray SV1, Cray SV1ex, Cray SV2, Cray SX-5, Cray SX-6, Cray T90, Cray T94, Cray T916, Cray T932, Cray T3D, Cray T3D MC, Cray T3D MCA, Cray T3D SC, Cray T3E, CrayTutor, Cray X1, Cray X-MP, Cray XMS, Cray-2, CSIM, CVT, Delivering the power . . ., DGauss, Docview, EMDS, GigaRing, HEXAR, IOS, ND Series Network Disk Array, Network Queuing Environment, Network Queuing Tools, OLNET, RQS, SEGLDR, SMARTE, SUPERLINK, System Maintenance and Remote Testing Environment, Trusted UNICOS, UNICOS MAX, and UNICOS/mp are trademarks of Cray Inc. IRIX is a trademark of Silicon Graphics, Inc. Linux is a trademark of Linus Torvalds. Netscape and the Netscape N and Ship’s Wheel logos are registered trademarks of Netscape Communications Corporation in the U.S. and other countries. PostScript is a trademark of Adobe Systems, Inc. Red Hat is a trademark of Red Hat, Inc. Solaris is a trademark of Sun Microsystems, Inc. UNIX, the “X device,” X Window System, and X/Open are trademarks of The Open Group in the United States and other countries. All other trademarks are the property of their respective owners.New Features CrayDoc™ Installation and Administration Guide S–2340–21 This book documents the following new features of CrayDoc 2.0 release: • Complete rewrite of the Perl scripts, with attention to security issues • Administration now from the cdadmin command line tool • Highlighting of search results • Man pages now available in HTML format • Glossary of software terms available online • Advanced search options This book also includes revised information in Section 6.6.Record of Revision Version Description 1.0 December 2001 Original Printing. 2.0 December 2002 Updated to reflect new features in CrayDoc 2.0 release. 2.1 February 2003 Updated to include changes in Section 6.6, Upgrading from version 1.0. S–2340–21 iContents Page Preface v Ordering Documentation . . . . . . . . . . . . . . . . . . . . . . v Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . vi Reader Comments . . . . . . . . . . . . . . . . . . . . . . . . vii Introduction [1] 1 Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . 1 Software Requirements . . . . . . . . . . . . . . . . . . . . . . . 2 Apache . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Perl . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Berkeley Database and DB_File Module . . . . . . . . . . . . . . . . 3 The CGI Perl Module . . . . . . . . . . . . . . . . . . . . . . 3 The Dumper.pm Perl Module . . . . . . . . . . . . . . . . . . . . 3 Sendmail.pm and Order Form Function . . . . . . . . . . . . . . . . 4 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . 4 RAM . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Hard Drive Space . . . . . . . . . . . . . . . . . . . . . . . 5 Preparing for Installation [2] 7 Stand-alone Mode . . . . . . . . . . . . . . . . . . . . . . . . 7 Shared Mode . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Security Issues . . . . . . . . . . . . . . . . . . . . . . . . . 8 suEXEC . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Understanding permissions . . . . . . . . . . . . . . . . . . . . 9 Configuring Apache [3] 11 Apache in Stand-alone Mode . . . . . . . . . . . . . . . . . . . . . 12 S–2340–21 iiiCrayDoc™ Installation and Administration Guide Page Apache in Shared Mode . . . . . . . . . . . . . . . . . . . . . . 12 Apache Tips and Troubleshooting . . . . . . . . . . . . . . . . . . . 13 Installing CrayDoc [4] 15 Automated Installation . . . . . . . . . . . . . . . . . . . . . . . 15 Manual Installation . . . . . . . . . . . . . . . . . . . . . . . . 16 CD Contents . . . . . . . . . . . . . . . . . . . . . . . . . 16 The craydoc-config File . . . . . . . . . . . . . . . . . . . . 18 Copying and Unpacking . . . . . . . . . . . . . . . . . . . . . 21 The HTML Interface . . . . . . . . . . . . . . . . . . . . . . . 22 Cray Documentation [5] 23 Man Pages . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Administration [6] 25 Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 The cdadmin tool . . . . . . . . . . . . . . . . . . . . . . . . 26 The Search Engine . . . . . . . . . . . . . . . . . . . . . . . . 27 Contents of $ScriptAlias . . . . . . . . . . . . . . . . . . . . . . 28 Moving CrayDoc . . . . . . . . . . . . . . . . . . . . . . . . 29 Upgrading from version 1.0 . . . . . . . . . . . . . . . . . . . . . 30 Procedure 1: Upgrading to CrayDoc 2.0 . . . . . . . . . . . . . . . . 30 Example 1: . . . . . . . . . . . . . . . . . . . . . . . . . 30 Example 2: . . . . . . . . . . . . . . . . . . . . . . . . . 30 Example 3: . . . . . . . . . . . . . . . . . . . . . . . . . 31 Example 4: Moving the manuals/ directory . . . . . . . . . . . . . . . 31 Index 33 iv S–2340–21Preface This publication describes CrayDoc 2.0 release and installing and administering CrayDoc and Cray documentation. Ordering Documentation To order software documentation, contact the Cray Software Distribution Center in any of the following ways: E-mail: orderdsk@cray.com Web: http://www.cray.com/craydoc/ Click on the Cray Publications Order Form link. Telephone (inside U.S., Canada): 1–800–284–2729 (BUG CRAY), then 605–9100 Telephone (outside U.S., Canada): Contact your Cray representative, or call +1–651–605–9100 Fax: +1–651–605–9001 Mail: Software Distribution Center Cray Inc. 1340 Mendota Heights Road Mendota Heights, MN 55120–1128 USA S–2340–21 vCrayDoc™ Installation and Administration Guide Conventions The following conventions are used throughout this document: Convention Meaning command This fixed-space font denotes literal items, such as file names, pathnames, man page names, command names, and programming language elements. variable Italic typeface indicates an element that you will replace with a specific value. For instance, you may replace filename with the name datafile in your program. It also denotes a word or concept being defined. user input This bold, fixed-space font denotes literal items that the user enters in interactive sessions. Output is shown in nonbold, fixed-space font. [ ] Brackets enclose optional portions of a syntax representation for a command, library routine, system call, and so on. manpage(x) Man page section identifiers appear in parentheses after man page names. The following list describes the identifiers: 1 User commands 2 System calls 3 Library routines 4 Devices (special files) and Protocols 5 File formats 7 Miscellaneous information 8 Administrator commands vi S–2340–21Preface Reader Comments Contact us with any comments that will help us to improve the accuracy and usability of this document. Be sure to include the title and number of the document with your comments. We value your comments and will respond to them promptly. Contact us in any of the following ways: E-mail: swpubs@cray.com Telephone (inside U.S., Canada): 1–800–950–2729 (Cray Customer Support Center) Telephone (outside U.S., Canada): Contact your Cray representative, or call +1–715–726–4993 (Cray Customer Support Center) Mail: Software Publications Cray Inc. 1340 Mendota Heights Road Mendota Heights, MN 55120–1128 USA S–2340–21 viiIntroduction [1] The CrayDoc software suite is a collection of open-source software components that allow easy searching and viewing of Cray documentation from a web browser. This document guides the system administrator through the installation and maintenance of CrayDoc. This document assumes you have a good understanding of file system permissions and basic system administration skills. This document does not attempt to cover the installation and configuration of Apache or Perl, because these packages are well documented on their own. Refer to the documentation that accompanies each distribution. 1.1 Compatibility CrayDoc should run under any operating system based on UNIX or Linux and has been tested on the following systems: • Linux 2.2 • Linux 2.4 • Solaris 8 • IRIX 6.5.x The look and feel of the CrayDoc Hypertext Markup Language (HTML) user interface was designed with Netscape in mind, but it should display with any web browser, including Lynx. Both frames and non-frames versions are included. Note: On many browsers other than Netscape, you should remember to hold the SHIFT key to select multiple entries on scrolling lists. Red Hat Linux releases 6.2 and later include all the required software as part of a standard installation. We cannot emphasize enough how easy it is to use CrayDoc with a standard Red Hat Linux system. CrayDoc 2.0 was developed on a Red Hat Linux 7.2 system. Many other recent Linux distributions come with all of the required packages; be sure to check the version numbers of your Apache and Perl installations against the requirements in Section 1.2, page 2. S–2340–21 1CrayDoc™ Installation and Administration Guide 1.2 Software Requirements CrayDoc requires the following software, all of which are included on the CrayDoc Documentation Library CD: 1. Apache version 1.3.x or later 2. Perl version 5.005 or later (version 5.6.1 recommended) 3. DB_File Perl module version 1.7.2 or later 4. CGI Perl module 5. Dumper.pm Perl module 6. Sendmail.pm Perl module (for the optional Order Form function) The version of Perl on the CrayDoc CD includes the correct versions of all modules needed for CrayDoc. 1.2.1 Apache There is a copy of the Apache version 1.3.20 source distribution and an INSTALL file on the CrayDoc CD. Documentation for installing and configuring Apache is freely available on the Internet as well; Apache powers 60% of the web servers on the Internet. The Apache web site is http://httpd.apache.org/. If you decide to compile Apache yourself from source, see Section 2.3, page 8. If you already have Apache installed, see Chapter 3, page 11 for help configuring it for CrayDoc. 1.2.2 Perl The source distribution of Perl version 5.6.1 is included on the CrayDoc CD along with the Perl INSTALL file. Note: Be sure to make a note of your Perl installation path for later use in configuring CrayDoc. Perl versions 5.6.0 and later include the required Perl modules, so if you are using Perl version 5.6.0 or later, you should not need to install them separately. To verify your Perl version, type the following at a shell prompt: Perl -v 2 S–2340–21Introduction [1] If you have Perl installed on your system, you can type the following at a shell prompt to test if you already have a module installed. For example, to verify the presence of the DB_File module type: find ‘Perl -e ’print "@INC"’‘ -name ’*.pm’ -print|grep DB_File If you are using a nonstandard Perl installation or you are missing a required module, you can either upgrade your Perl installation (Perl 5.6.1 is included on the CrayDoc CD; we recommend you upgrade) or compile and install the modules individually (see the CrayDoc CD for the source code and documentation for the required Perl modules). 1.2.3 Berkeley Database and DB_File Module Installation documentation for the Berkeley Database is available from the source distribution on the CrayDoc CD. The DB_File Perl module is available within the Perl.DB_File directory of the Berkeley Database source distribution on the CD. Read the README files within the BerkeleyDB directory for installation help. Note: It is not enough to compile the Perl DB_File module. You must have the Berkeley Database libraries installed as well. See the Berkeley Database documentation for more information. 1.2.4 The CGI Perl Module The CGI Perl module is included with all Perl versions 5.005 and later. If you do not have the CGI Perl module installed, you should consider upgrading your Perl version to 5.6.1, which is included on the CrayDoc CD. 1.2.5 The Dumper.pm Perl Module The Dumper.pm Perl module is, according to the Dumper.pm documentation, part of the standard Perl installation in versions 5.005 and later. This module allows for reading and writing of the CrayDoc databases. Warning: A version of the Dumper.pm module is included on the CrayDoc CD. The included version will only work with Perl version 5.6.1 or later. Do not install this version of the Dumper.pm module unless you are running Perl 5.6.1 or later and have already verified that you do not have the Dumper.pm module already installed. See Section 1.2.2, page 2 for more on verifying your Perl modules. S–2340–21 3CrayDoc™ Installation and Administration Guide 1.2.6 Sendmail.pm and Order Form Function CrayDoc includes an optional function that allows users to send e-mail to Cray requesting printed copies of manuals. The Order Form function requires the Perl Sendmail.pm module. The Sendmail.pm module is not part of any standard Perl distribution, and so it is included separately on the CrayDoc CD. If you install CrayDoc with the included installation script (see Section 4.1, page 15), you will be prompted as to whether you want to leave the Order Form function on (it is on by default). If it is turned on, the Sendmail.pm module is automatically installed for you. 1.3 Hardware Requirements If your computer can run Perl and Apache, it should support CrayDoc. Your hardware needs will vary according to your environment, the number of users you expect to access the CrayDoc server, and what other software you are running on the computer. 1.3.1 RAM There are two aspects of RAM use to consider when setting up your CrayDoc system: • Indexing the search engine • Configuring Apache The CrayDoc search engine can require significant memory (over 200 MB) when indexing the manuals. See Section 6.3, page 27 for more information on configuring the search engine. The Apache documentation covers some aspects of improving the performance of your Apache web server. See http://httpd.apache.org/docs/misc/perf-tuning.html for more information. We recommend a minimum of 96 MB of total RAM for a Stand-alone CrayDoc installation, and 256 MB if you choose the Shared installation. 4 S–2340–21Introduction [1] 1.3.2 Hard Drive Space The entire set of Cray documentation (HTML and PDF versions) will fit on a single CD (that is less than 650 MB). A standard Red Hat Linux system and a typical CrayDoc installation will easily fit on a 2 GB hard drive. Take your backup system into consideration when making hard drive decisions. S–2340–21 5Preparing for Installation [2] Before you install CrayDoc, think carefully about the environment in which you intend to use the CrayDoc system. CrayDoc requires the Apache web server. Your site may already be using the Apache web server. CrayDoc is designed to integrate smoothly with already-existing installations of the Apache web server. Refer to the mode that is best suited to your particular needs and environment: Stand-alone CrayDoc has a dedicated Apache installation. This scenario allows for optimal security and performance. See Section 2.1, page 7 for installation instructions and more information. Shared CrayDoc shares an existing Apache installation. This scenario allows CrayDoc to coexist with other users of the Apache web server, and it requires the involvement of the administrator of your existing Apache installation. See Section 2.2, page 8 for installation instructions and more information. You must also consider security issues when selecting a mode (see Section 2.3, page 8). 2.1 Stand-alone Mode The Stand-alone installation scenario allows for the most control over CrayDoc security and performance. In this scenario, the Apache web server is configured with CrayDoc as the primary user. Examples of this scenario include: • An unused Linux-compatible computer (that is, an extra PC) that you want to devote to being a CrayDoc server • Root access to a computer that is not currently running an Apache web server For this documentation, we will assume you have a Red Hat Linux 7.2 system running on an average, off-the-shelf PC. The average installation time for a new Red Hat system is about 60–90 minutes, which includes the installation of Apache 1.3.12 and Perl 5.6.0 with all the necessary modules. Once you have Apache and Perl successfully installed, the next step is deciding where you will install CrayDoc. In a Stand-alone scenario, it is up to you S–2340–21 7CrayDoc™ Installation and Administration Guide to decide where on your file system you put the CrayDoc installation. We recommend only that you verify that you will have enough space on your file system to accommodate the Cray documentation (see Section 1.3.2, page 5). Verify that your Apache web server is running by pointing a web browser at your CrayDoc server. If you are using a web browser running on the CrayDoc server machine, you should be able to point your browser at: http://localhost/ and you should see the Apache test page (or whatever home page you may have replaced it with). If you are using a web browser running on a machine other than the CrayDoc server, you should be able to point your browser at the host name of your CrayDoc server. For example, if your CrayDoc server’s host name is craydocserv, you should be able to see the Apache test page at http://craydocserv/. If you cannot see the Apache test page, you probably have a problem with your Apache installation. Verify that the httpd daemon is running by logging in to the CrayDoc server and typing: ps -ef | grep httpd See the Apache documentation for more troubleshooting help. When you have successfully installed Apache, you can configure it for use with Stand-alone mode (Section 3.1, page 12). 2.2 Shared Mode Shared mode is the most flexible installation option because it allows your CrayDoc installation to coexist peacefully with other users of the Apache web server. However, Shared mode requires a more intimate understanding of Apache configuration. See Section 3.2, page 12 for more details. 2.3 Security Issues Security issues also are important to consider when deciding whether you install in Stand-alone or Shared mode. Stand-alone mode offers more security but requires that CrayDoc be the main use of the Apache web server. Shared mode offers more flexibility but requires coordinated effort between the Apache and CrayDoc administrators and careful attention to file system permissions. A thorough understanding of file system permissions is essential for installing CrayDoc successfully in Shared mode. 8 S–2340–21Preparing for Installation [2] 2.3.1 suEXEC The suEXEC is one option to consider for Stand-alone mode. If you are concerned about the possibility of someone abusing your Apache installation via the CrayDoc system and you are installing CrayDoc in Stand-alone mode, you may want to consider compiling Apache with the suEXEC option. This option allows CGI commands to run under a different user ID than the user ID of the calling Apache web server. You can read the Apache documentation on the suEXEC option at http://httpd.apache.org/docs/suexec.html. Here is an example configure command line to compile in suEXEC support on a Red Hat Linux system. prompt> ./configure --with-layout=Red Hat --enable-suexec --suexec-uidmin=100 \ --suexec-gidmin=100 --suexec-docroot="/home/httpd/html" --suexec-caller=www ! Caution: Be certain that the --suexec-caller value equals the User directive value in your Apache httpd.conf file. CrayDoc relies on file system permissions for its security, so if you have set up CrayDoc according to the Stand-alone instructions in Section 3.1, page 12, you probably do not need to compile the suEXEC option. However, if you do compile the suEXEC option into Apache, your CrayDoc permissions will need to be set accordingly. See Section 2.3.2, page 9 for more information. 2.3.2 Understanding permissions You should only give as much permission as is absolutely necessary for CGI scripts to be executed and for HTML and PDF files to be read. This means that you should know, first of all, which user and group ids (UID and GID) your CGI scripts execute with when they are called by the Apache server. The executing UID or GID must have permission to execute your CrayDoc CGI scripts and read your CrayDoc library files. In a Stand-alone installation, this means you can set permissions on your CrayDoc files very tightly (for example, 500 if the file is owned by the executing UID). In a Shared installation, file permissions must be set so that the executing UID has executable permissions to the CrayDoc scripts in $ScriptAlias, read permissions to all the .db and .dmp files in $ScriptAlias, and read permissions to all the documents in $library and $manlibrary. For more details on the $ScriptAlias, $manlibrary and $library variables, see Section 4.2.2, page 18. S–2340–21 9CrayDoc™ Installation and Administration Guide Note: CrayDoc 2.0 requires that all the CGI scripts and databases need only to be readable and executable by the executing UID or GID. All administration is done by the cdadmin tool, so the executing UID of the cdadmin tool must have write permission to $ScriptAlias and $DocumentRoot directories, but the Apache user does not need to have write permission. 10 S–2340–21Configuring Apache [3] Apache is highly configurable, and your site may have special configuration needs. See the Apache documentation on the CrayDoc CD or at http://httpd.apache.org/docs/ for more details. Whether you install in Stand-alone or Shared mode, CrayDoc requires certain information from your Apache configuration file. These settings are found in the Apache httpd.conf file: ServerName The name returned to the web browser by the Apache web server. Example: craydocserv.foo.com DocumentRoot The root path of your HTML documents. ScriptAlias The root path of your CGI scripts. User The name or UID under which the web server will execute CGI processes. Group The name or GID under which the web server will execute CGI processes. ! Caution: The User and Group settings are very important for your installation. They must refer to a real user and a real group on your system. Unless your Apache installation was compiled with the suEXEC option (see Section 2.3, page 8), all CGI processes will run as the user and group specified in your httpd.conf file, which means that the same user or group must have correct permissions to all your CrayDoc files. In Stand-alone mode, we recommend that you create a specific system user and group for use with Apache and CrayDoc. Something appropriate and easy to remember (like user craydoc and group craydoc) is recommended. This will give you the most control and security over your installation. In Shared mode, you must know the User and Group settings in your Apache configuration so that you can verify that your CGI scripts are executable and your files readable by the web server. For security reasons, you should not make your CGI scripts world-executable, and depending on your particular installation, you may not want to make your documentation world-readable either. See Section 2.3.2, page 9 for more details. S–2340–21 11CrayDoc™ Installation and Administration Guide 3.1 Apache in Stand-alone Mode Usually, you must be logged in as the root user to configure Apache, though your particular installation may vary. First, make sure the Apache httpd daemon is not running: prompt> killall httpd Second, edit your Apache httpd.conf configuration file, paying particular attention to the five settings listed at the beginning of this chapter. The httpd.conf file is well commented; refer to it and the Apache documentation for more details. Third, close and save your httpd.conf file and then start the httpd daemon to test your changes: prompt> httpd If you get any errors when starting your httpd daemon, go back and check your httpd.conf file. You should now be ready to install the CrayDoc server. See Section 4.1, page 15. 3.2 Apache in Shared Mode If you choose to install CrayDoc on a system with a pre-existing Apache installation, you will need to consult with your local Apache administrator about the best way to install CrayDoc. Because we cannot anticipate your particular Apache configuration, you should read the Stand-alone mode (Section 3.1, page 12), Permissions (Section 2.3.2, page 9), and Manual Installation (Section 4.2, page 16) sections thoroughly before attempting a Shared-mode installation. Once installed, CrayDoc in Shared mode works the same way as in Stand-alone mode, but you must understand how your existing Apache system is configured in order to install CrayDoc. You have two basic options in Shared mode: install CrayDoc under a user’s account or in a subdirectory under the main Apache DocumentRoot. You may also consider using a VirtualHost directive in Apache in conjunction with either of the above two options. See the Apache documentation for pros and cons of each scenario. Because of the infinite possible Apache configurations, this document does not provide examples for every possible scenario. The install script will attempt to 12 S–2340–21Configuring Apache [3] accommodate your particular configuration (Section 4.1, page 15), but you may find that you need to install CrayDoc manually (Section 4.2, page 16). Just as with Stand-alone mode, the easiest way to install CrayDoc is with the included install script. See Section 4.1, page 15. 3.3 Apache Tips and Troubleshooting Some common troubleshooting tips for the httpd.conf file for installing in Shared mode follow. CrayDoc requires Apache ExecCGI permissions for the CrayDoc $ScriptAlias directory (see Section 4.2.2, page 18 and Section 2.3.2, page 9 for more details). The ScriptAliasMatch directive is an easy way to do this. If, for example, you want to install CrayDoc under a user’s public_html/cgi-bin/ directory, add the following line to your httpd.conf file in the mod_alias section: ScriptAliasMatch ^/~([^/]*)/cgi-bin/(.*) "/ pathtouserhome/$1/public_html/cgi-bin/$2" A slightly more secure option, since only the user may use it to execute scripts, is the ScriptAlias directive: ScriptAlias /~user/cgi-bin/ /pathtouserhome/user/public_html/cgi-bin/ Either directive also requires the Directory directive to allow users to access that directory: AllowOverride None Options None Order allow,deny Allow from all See the Apache documentation for more help in troubleshooting and configuring Apache. S–2340–21 13Installing CrayDoc [4] Before you install CrayDoc, you need to have made note of some important items: • The location of your httpd.conf file • The five key values from your httpd.conf file (see Chapter 3, page 11) • In which mode you will install CrayDoc (see Chapter 2, page 7) 4.1 Automated Installation The easiest way to install CrayDoc is to mount the CD and run the install script. Usually, this will require you to be logged in as the root user, since most systems allow only the root user to mount the CD. Also, the install script will attempt to chown the installed files, and some systems allow only the root user to chown files. prompt> mount cdrommountpoint prompt> cd cdrommountpoint prompt> ./install where cdrommountpoint is the mount point of your CD drive. On a Linux system, this is usually /mnt/cdrom. ! Caution: Your CrayDoc settings are closely tied to your Apache settings, so make a note of the relevant Apache settings before you run the install script (see Section 3.1, page 12 for more information about these settings). During the install, knowing the location of your httpd.conf file is necessary only if you are doing a Stand-alone mode install, since the install script uses this file to determine your default CrayDoc settings. On a Linux system, this is installed by default at /etc/httpd/conf/httpd.conf and that is where the install script will look for it by default. Note: You will be prompted during the install process for which mode you want to use. If you indicate Shared mode, you will be prompted at some point during the install process for a subdirectory value. The subdirectory is the value that should be appended to the ServerName for your main HTML URL. This value might be a user’s directory or a subdirectory under the main DocumentRoot or some other value depending on your particular Apache configuration. During the install process you will be asked to verify the URLs for your CrayDoc installation. Check these carefully before accepting them. S–2340–21 15CrayDoc™ Installation and Administration Guide If you want more information on the specifics of the CrayDoc settings, or for help in installing CrayDoc manually without using the install script, see Section 4.2, page 16. The install script will guide you through the process of installing your CrayDoc server software. Once the install script completes successfully, you are ready to install the Cray documentation. Proceed to Chapter 5, page 23. 4.2 Manual Installation If you decide to install CrayDoc manually, either because you cannot use or do not wish to use the install script, you should have a thorough understanding of how CrayDoc works and fits together. We recommend that you become familiar with your Apache configuration, especially the httpd.conf file, and that you have a good understanding of file system permissions. This section describes the details of the CrayDoc components and how you can install and configure them. You should read Section 1.2, page 2, Chapter 2, page 7, and Chapter 3, page 11 before proceeding any further. 4.2.1 CD Contents Depending on your system, you may need to be the root user in order to mount the CD. At the prompt enter: prompt> mount cdrommountpoint 16 S–2340–21Installing CrayDoc [4] where cdrommountpoint is your system’s CD mount point. On Linux systems, this is usually /mnt/cdrom. Change to the CD directory and look at the contents: prompt> cd cdrommountpoint prompt> ls -1FC BerkeleyDB/ LICENSE.CrayDoc PerlFect/ README apache/ configure.pl craydoc/ install manpages/ manuals/ Perl/ Perl_mods/ releases/ The CD contains: BerkeleyDB Directory containing source code for the Berkeley Database. If you are using Perl 5.6.0 or later, you should not need this code. Otherwise, it is provided in case you need to compile and install the Berkeley Database files, which include the required Perl DB_File module. See the documentation in this directory for more details, or see the Berkeley Database web site at http://www.sleepycat.com/. LICENSE.CrayDoc ASCII file with legal information. If you install CrayDoc manually, read this file. README ASCII file with general information and a how-to for the anxious-to-install. apache Directory containing source code for the Apache 1.3.20 distribution. See the included documentation or the Apache web site http://httpd.apache.org/docs/ for more information. configure.pl Perl script for use with the install script. craydoc Directory containing the CGI scripts, HTML interface, and supporting files. S–2340–21 17CrayDoc™ Installation and Administration Guide install Bourne shell script for automated installation of CrayDoc. This script simply finds your Perl location and then calls configure.pl. manuals Directory containing all the Cray book files. manpages Directory containing all the Cray man page files. Perl Directory containing source code for Perl 5.6.1. This version of Perl includes all the modules required for CrayDoc. Perl_mods Directory containing the Data/Dumper.pm and Mail/Sendmail.pm Perl modules. releases Directory containing ASCII file(s) of the packages included on your CrayDoc CD. Each file contains necessary information about each of the documents included with the package. Once you have determined that you have the necessary software, and you have configured Apache correctly for either Stand-alone (Section 3.1, page 12) or Shared mode (Section 3.2, page 12), you will want to create your CrayDoc configuration file, copy your CrayDoc files to the correct locations, and customize your HTML interface, as described in the following sections. 4.2.2 The craydoc-config File If you install your CrayDoc software suite manually, you need to create a configuration file similar to the following example. 18 S–2340–21Installing CrayDoc [4] Note: In Perl scripts, as in shell scripts, lines beginning with a pound sign (#) are comments and are ignored. #!/usr/bin/Perl # CrayDoc 2.0 Configuration File # written by configure.pl Thu Oct 31 15:00:02 CST 2002 # OS: Linux # OSversion: 2.4.7-10 # Hostname: somehost # Perl: /usr/bin/Perl # Install: StandAlone # CD-ROM Location: /mnt/cdrom $DocumentRoot = "/var/www/html"; $ScriptURL = "http://somehost.foo.com/cgi-bin"; $ScriptAlias = "/var/www/cgi-bin"; $User = "someuser"; $Group = "somegroup"; $cgi_umask = "770"; $html_umask = "775"; $path2cd = "/mnt/cdrom"; $URLRoot = "http://somehost.foo.com"; $autohomepage = "1"; $splashpage = "frontpage.html"; $orderform = "yes"; $booksdmp = "$ScriptAlias/books.dmp"; $collectionsdmp = "$ScriptAlias/collections.dmp"; $packagesdmp = "$ScriptAlias/packages.dmp"; $library = "$DocumentRoot/manuals"; $manlibrary = "$DocumentRoot/manpages"; $TxtSearchURL = "$ScriptURL/search/search.pl"; $CrayDocCGI = "$ScriptURL/craydoc.cgi"; $ManPageCGI = "$ScriptURL/manpage_display.cgi"; $GlossCGI = "$ScriptURL/glossary.cgi"; $LibraryRoot = "$URLRoot/manuals"; $Icons = "$URLRoot/img"; $pl2prod_dmp = "$ScriptAlias/pl2prod.dmp"; $maninfo_dmp = "$ScriptAlias/manpages.dmp"; $gloss_dmp = "$ScriptAlias/glossary.dmp"; $SMTP = "localhost"; Here is a list of the settings required and what they mean: S–2340–21 19CrayDoc™ Installation and Administration Guide DocumentRoot The full file path from where Apache will be serving files. Example: /var/www/html ScriptAlias The full file path where your scripts reside, similar to DocumentRoot. Example: /var/www/cgi User Who should own all your CrayDoc scripts and files. Group Which group all your CrayDoc scripts and files should belong to. Note: Your User and Group settings should be coordinated with your Apache User and Group settings, so that the Apache web server has appropriate permissions to all your CGI scripts and files. See Section 2.3.2, page 9 for more details. cgi_umask The permissions, in chmod numerical format, of the executable scripts. html_umask The permissions, in chmod numerical format, of the non-executable files (HTML, PDF, etc). URLRoot The URL (in Shared mode, with your specific subdirectory appended). Example: http://craydocserv/~username/ or http://craydocserv/craydoc/. ScriptURL Complete URL for your CGI scripts. path2cd Your CD mount point. autohomepage Turned on by default. This option generates a dynamic home page for each book. If commented out, links to manuals will default to the Apache DirectoryIndex setting (usually index.html or index.htm). See Section 6.1, page 25 for more on other options for the book home page. splashpage Set to frontpage.html by default. This is the main home page for CrayDoc. You can change this setting to point at a different HTML page or customize frontpage.html for your site (see Section 4.2.4, page 22). orderformfunction Turned on by default. You may turn this off by not installing the orderform.cgi script and by commenting out this option. 20 S–2340–21Installing CrayDoc [4] Once you have created your craydoc-config file, copy it to the correct location. cp craydoc-config $ScriptAlias/ Now you are ready to copy the necessary files from the CD. 4.2.3 Copying and Unpacking If installing CrayDoc manually, you must copy the necessary files from the CrayDoc CD to the correct locations, as indicated in your craydoc-config file. To make this easier, use the same installation map that the install script uses: .install.dmp. This file is located at the root level of the CD. It is written in the format of a Perl hash. The map is a list of the file paths in relation to the root level of the CD. You can interpret where to copy each file this way: • If a file path begins with html/, copy it to $DocumentRoot • If a file path begins with img/, copy it to $DocumentRoot/img • If a file path begins with modules/, copy it to $ScriptAlias/modules • If a file path begins with search/, copy it to $ScriptAlias/search • If a file path is simply a file name with no prepended path, copy it to $ScriptAlias • If a file path begins with tools/, copy it to your administration tool directory. See Section 6.1, page 25 for more on the administration tools. ! Caution: File permissions for these files will vary according to your particular configuration, but they should match whatever values you set in the $cgi_umask and $html_umask variables. See Section 2.3.2, page 9 for more information. Once you have copied the files to the correct locations and changed permissions and ownership appropriately, you are ready to install Cray documentation. Proceed to Chapter 5, page 23. S–2340–21 21CrayDoc™ Installation and Administration Guide 4.2.4 The HTML Interface The CrayDoc HTML interface is mostly generated dynamically by the various CGI scripts. However, there are a few static HTML pages that you may edit. The main one is frontpage.html, which is what users first see when they connect to the CrayDoc server. You may edit this file to suit your particular needs, using any text editor. 22 S–2340–21Cray Documentation [5] Cray delivers two kinds of documentation: books (also known as manuals) and man pages. Books are available in both HTML and PDF formats and are delivered on the CrayDoc CD. Man pages are available in both formatted (also known as catman) ASCII and HTML formats. Catman ASCII man pages are normally installed automatically along with your software on your Cray system and are accessed with the man command. HTML man pages are a new feature of CrayDoc version 2.0 and are delivered on the CrayDoc CD. We recommend that when you install Cray documentation on your CrayDoc system, you choose to install both man page and book packages for each software release, so that the complete set of documentation for each release is available through the CrayDoc user interface. You may choose to use the documentation without installing the CrayDoc server software, but you will not be able to search all the HTML files without the server software. Whether you choose to install the documentation on a CrayDoc server or simply view it locally from the CD, you should understand the organization of the documentation. Man pages and books are structured differently, so this chapter discusses their organization. 5.1 Man Pages Man pages describe uniquely identifiable components of your software, such as commands, libraries, and system calls. Each software release may contain one or more sets of man pages, which are grouped together into packages. A package is a set of documentation that supports a specific release of a software product, for example, C/C++ 4.2 or UNICOS/mp 2.0. Each man page is identified by four pieces of data: name The name of the man page, usually the same as the software component it documents. volume number The kind of component the man page documents. See the conventions section, "Conventions," page . product name Name of the product that the man page supports. This usually, but not always, corresponds to the package name. product version Version of the product that the man page supports. This usually, but not always, corresponds to the package name. S–2340–21 23CrayDoc™ Installation and Administration Guide When you view man page listings in the CrayDoc user interface, each man page should have both an implementation and a package name listed next to it. The implementation information is the product name and version with which the man page is associated. On the CrayDoc CD, all the available man pages are located in the manpages directory. You should use the cdadmin tool to install man pages. See Section 6.2, page 26 for more information. 5.2 Books Cray delivers HTML versions of most books and PDF versions of all books. Each book is identified by a publication number. On the CrayDoc CD, all the PDF and HTML versions of the books are located in the manuals directory in subdirectories named for the publication number of each book. Books, like man pages, are grouped together into packages. A book package is a collection of the books that support a particular version of a software product. 24 S–2340–21Administration [6] This chapter addresses the administration of your CrayDoc system, including installation and removal of Cray documentation packages. 6.1 Tools When you installed your CrayDoc server software, you should have installed a number of administration tools as well. Warning: The following scripts, with the exception of cdadmin, are provided as-is, as a service to the administrator, with no implied support from Cray. Use them at your own risk. The following tools should have been installed: autohomepage.pl Perl script to create custom index.html pages for your book directories. By default, this script uses the craydoc.pm Perl module, installed in $ScriptAlias/modules, to create an HTML document showing the contents of a book directory in a style consistent with the rest of the CrayDoc user interface. You can modify either the autohomepage.pl script, the index.html page that is produced, or the craydoc.pm module itself to change the look and feel of your book directory home page. cdadmin The main administration tool for the CrayDoc server system. See Section 6.2, page 26 for a complete description of how to use this tool. cdbackup.sh A simple Bourne shell script for making a backup of your most important databases. check.index.cron Perl script intended to be run from a cron job. check.index.cron will compare the timestamps of all your installed documentation against the timestamp of your search index database, and warn you if you have documents that are not indexed. Very useful as an added precaution against forgetting to re-index your search engine after installing or removing documentation. S–2340–21 25CrayDoc™ Installation and Administration Guide 6.2 The cdadmin tool The cdadmin Perl script is the administration tool for your CrayDoc system. Unlike CrayDoc 1.0, version 2.0 is administered entirely from the command line using the cdadmin tool. For security reasons, you should install cdadmin outside the $ScriptAlias file path. The administration password is stored in plain text in the cdadmin file, so you should protect the file with tight permissions. Only the user who administers the CrayDoc system should have read or execute permissions to the file. You can start the cdadmin tool from the command line, just like any other command: path_to_file/cdadmin You will be prompted to enter the administration password, and then you should see this menu: Please select a mode. [1] Install Packages [2] Remove Packages [3] Remove a Book [4] Remove a Man Page [5] Reindex the Search Engine [6] List Installed Documents by Package [7] List All Installed Books [8] List All Installed Man Pages [9] Install Custom Documents [10] Exit NOTE: This tool uses ’more’ to page results. Enter mode number: Note: The paging tool is set to more by default. You can edit this to use either your PAGER setting as defined in your shell environment, or some other paging tool on your system (like less), by changing the $pager variable in the interactive.pm Perl module. See Section 6.4, page 28 for the location of the interactive.pm module. Consult your local man page for more on more. The cdadmin tool was designed to guide you through the process of installing and removing both packages of documents and individual documents. You can 26 S–2340–21Administration [6] also reindex the search engine, which you should do each time you are finished removing or installing documents. At any time you may type q or e to exit the cdadmin tool. Note: The cdadmin tool will allow you to install your own, local documentation on the CrayDoc server, so that you can make custom documents available to your users. Select the Install Custom Documents mode to create a new, local publication number, title and brief summary. You may install documents of any format you wish, but only HTML files will be indexed by the search engine. Any HTML files you install should be placed inside a directory named html-pubnumber, where pubnumber is the local publication number you created for your documents. The html-pubnumber directory should be a subdirectory of the directory $library/pubnumber. The cdadmin will create these directories for you as part of the Install Custom Documents mode. 6.3 The Search Engine The CrayDoc search engine is based on the PerlFect Search 3.30 package (http://Perlfect.com/freescripts/search/) and modified under the auspices of the GNU General Public License (http://Perlfect.com/freescripts/GPL). Note: PerlFect is not responsible for the modifications that Cray has made to their software and will not support your CrayDoc search engine. Direct any questions or concerns directly to Cray. The CrayDoc search engine is highly configurable. The version that is installed with CrayDoc is optimized for use with CrayDoc, but you may find that you want to change some of the configuration to suit your particular needs. The relevant file is located at $ScriptAlias/search/conf.pl, which may be edited with any text editor. ! Caution: Modifying the conf.pl file may have lots of unforeseen consequences, particularly in the time it takes to reindex your library and the speed and accuracy of your searches. We suggest that you make a backup copy or comment out the original lines when you make changes, so that you can easily return to a version of the conf.pl file that is known to work well with CrayDoc. If you change certain settings in the conf.pl file, you must reindex your documentation in order for the changes to take effect. These settings are annotated as such in the conf.pl file. In addition, you should always reindex your library if you add or remove manuals. S–2340–21 27CrayDoc™ Installation and Administration Guide The easiest way to reindex your site is with the cdadmin tool. See Section 6.2, page 26 for more information. 6.4 Contents of $ScriptAlias The following contents of your $ScriptAlias directory reflect what is installed: books.dmp The main database for all your installed books. This file must be readable by the executing CGI UID (see Section 2.3.2, page 9). collections.dmp All the Cray publication categories data which allows searching according to Intended Audience or Hardware Product. This file must be readable by the executing CGI UID (see Section 2.3.2, page 9). craydoc-config Main CrayDoc configuration file. See Section 4.2.2, page 18 for more details. craydoc.cgi Main CrayDoc CGI script. This script interfaces with all the databases and allows you to use your CrayDoc system. glossary.cgi Glossary search and display CGI script. glossary.dmp Glossary database. manpage_display.cgi Man page search and display CGI script. manpages.dmp Man page database. modules/ Directory for all CrayDoc Perl modules. This directory contains the following modules: Mail/ Enclosing folder for the optional Perl module Sendmail.pm used by the optional orderform.cgi script to send e-mail to Cray. CrayDB.pm Functions for interacting with the CrayDoc databases. CrayDocGlossary.pm Functions for the glossary.cgi script. 28 S–2340–21Administration [6] craydoc.pm Functions used by all the CrayDoc CGI scripts. interactive.pm Common functions for the install script and the cdadmin tool. orderform.cgi An optional script that allows the user to request a printed copy of a Cray publication via e-mail. This script requires that your CrayDoc server have network access to a SMTP server that can relay e-mail to Cray. packages.dmp All the software releases for which you have installed documentation. This file must be readable and writable by the executing CGI UID (see Section 2.3.2, page 9). pl2prod.dmp A database that maps all man page product names and versions to their respective file path abbreviations. This file must be readable and writable by the executing CGI UID (see Section 2.3.2, page 9). search/ The directory for the CrayDoc search engine; it contains all the searching, indexing, and index data files. See Section 6.3, page 27 for more details. 6.5 Moving CrayDoc If at any point you need to move your CrayDoc installation to a different machine with a different operating system, you can preserve your existing documentation without being forced to reinstall from the original CDs. You should be able to use the tar command to bundle up all your existing files, move them to the new location, extract them, and then change your craydoc-config file to reflect the new configuration. You will then need to reindex the search engine, because the databases it creates are platform-specific. S–2340–21 29CrayDoc™ Installation and Administration Guide 6.6 Upgrading from version 1.0 This section guides you through the process of upgrading your CrayDoc server software from version 1.0 to 2.0. The safest way to upgrade your server software is to rename your existing CGI and HTML directories (the values of $ScriptAlias and $DocumentRoot, respectively) and create new directories for your 2.0 files with the same names as the old directories. You can then test the installation, and if necessary, revert to the old (1.0) directories if you enounter problems with the new 2.0 installation. Using this method, the URL remains the same for your users. ! Caution: Several of the installed CrayDoc 2.0 files require hard-coded path values, which the install script automatically performs for you. If you install CrayDoc manually, you must edit these files by hand. The values that require hard-coded paths are indicated with an HTML comment tag: , where somevalue is an absolute path. The path values depend on your craydoc-config configuration. The files that require these values are: craydoc.cgi, glossary.cgi, manpage_display.cgi in $ScriptAlias, all the administration tools (including cdadmin), and the main index.html file in $DocumentRoot. You should have a copy of your existing 1.0 craydoc-config file available for reference. Procedure 1: Upgrading to CrayDoc 2.0 1. Rename your existing $ScriptAlias and $DocumentRoot directories. Example 1: prompt> mv /var/www/docs /var/www/docs.10 prompt> mv /var/www/cgi-bin /var/www/cgi-bin.10 2. Create new directories with the same names as your old $ScriptAlias and $DocumentRoot directories. Example 2: prompt> mkdir /var/www/docs prompt> mkdir /var/www/cgi-bin 3. Mount the CrayDoc CD, and install the 2.0 software with the install script. When prompted, be sure to use the same directory values that you used for your 1.0 installation. For more information about the install process, see Section 4.2, page 16. 30 S–2340–21Administration [6] 4. Copy your 1.0 books.dmp and packages.dmp files from your renamed 1.0 $ScriptAlias directory to the new 2.0 $ScriptAlias directory. Verify that they have the correct permissions. See Section 2.3.2, page 9 for more information about permissions. Example 3: prompt> cp /var/www/cgi-bin.10/packages.dmp /var/www/cgi-bin/ prompt> cp /var/www/cgi-bin.10/books.dmp /var/www/cgi-bin/ 5. Open a web browser and point it at the URL of your new 2.0 installation. If you can see the CrayDoc welcome page, then you have installed 2.0 correctly and should proceed to the next step. Otherwise, verify that you have configured the 2.0 craydoc-config file correctly, and that your permissions are set correctly. Often, the Apache error log is a good place to check for error messages. See the Apache documentation for how to access the error log. See Section 4.2.2, page 18 for more information about the craydoc-config file. 6. At this point, your 2.0 server software is installed correctly, and you need to move your existing documentation to the new $DocumentRoot location. Example 4: Moving the manuals/ directory prompt> mv /var/www/docs.10/manuals /var/www/docs/ You should now be able to point a web browser at your original CrayDoc URL and view the documentation you already had installed in version 1.0. If any of the links are broken, verify that the new 2.0 craydoc-config settings are correct. S–2340–21 31Index A administration tools, 25 Apache source code, 2 B Berkeley Database source code, 3 C cdadmin tool, 26 Compatibility, 1 D Dumper.pm module, 3 H Hardware requirements, 4 I installing local custom documents, 27 M Manual installation craydoc-config, 18 map of file locations, 21 Mode Shared, 8 Stand-alone, 7 P Perl source code, 2 S Security, 8 permissions, 9, 11 suEXEC, 9 Sendmail.pm module, 4 Software requirements, 2 S–2340–21 33 Comparing Binaries Between Cray Linux Environment (CLE) Systems, Standalone Whiteboxes, and ESLogin Nodes Published Date 11/20/09 Abstract Users have shown an interest in understanding all potential differences between a binary generated on a Cray XT login node and a binary generated on either a standalone SuSE Linux whitebox or an ESlogin node installed with the Cray Application Developer's Environment and its supplement for standalone machines. This paper documents likely, known sources of differences between these binaries and how to detect and avoid them.© 2009 Cray Inc. All Rights Reserved. This manual or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Inc. U.S. GOVERNMENT RESTRICTED RIGHTS NOTICE The Computer Software is delivered as "Commercial Computer Software" as defined in DFARS 48 CFR 252.227-7014. All Computer Software and Computer Software Documentation acquired by or for the U.S. Government is provided with Restricted Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7014, as applicable. Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227 7013, as applicable. Cray, LibSci, UNICOS and UNICOS/mk are federally registered trademarks and Active Manager, Cray Apprentice2, Cray C++ Compiling System, Cray Fortran Compiler, Cray SeaStar, Cray SeaStar2, Cray SHMEM, Cray Threadstorm, Cray X1, Cray X1E, Cray X2, Cray XD1, Cray XMT, Cray XT, Cray XT3, Cray XT4, CrayDoc, CRInform, Libsci, RapidArray, UNICOS/lc and UNICOS/mp are trademarks of Cray Inc. PGI is a registered trademark of The Portland Group.Comparing Binaries Between Cray Linux Environment (CLE) Systems, Standalone Whiteboxes, and ESLogin Nodes Table of Contents 1. Introduction 2. Types of Binary Di erences 3. Evaluating Binary Di erences 4. Setting PGI Core Runtime Libraries 5. Conclusion S-0019-10 Cray Inc. 3Comparing Binaries Between Cray Linux Environment (CLE) Systems, Standalone Whiteboxes, and ESLogin Nodes Introduction Users have shown an interest in understanding all potential differences between a binary generated on a Cray XT login node and a binary generated on either a standalone SuSE Linux whitebox or an ESlogin node installed with the Cray Application Developer's Environment and its supplement for standalone machines. This paper documents likely, known sources of differences between these binaries and how to detect and avoid them. Types of Binary Differences The most common source of binary differences we have observed in-house is caused by library version mismatches between the XT and the standalone machine. It is vitally important to keep the software on both machines up-to-date with each other if you want comparable binaries. We have made our security patches available specifically for the xt-sysroot package in order to ensure that the SLES software versions exactly match between your XT and your standalone development machine. Use the `module list` command on each machine to compare library versions before you decide that the software on the standalone machine is not capable of producing nearly identical binaries. The wrong modulefile might be loaded or set as default on one of the two systems. Differing symbol versions in the two libraries would explain the difference in binaries. Even when the library versions and security patch levels exactly match between two systems, there are other potential sources of binary mismatch. Within the standard ELF headers, every binary contains a date-and-time stamp that almost never matches any other binary perfectly. Due to this timestamp field, you can almost never produce a perfect byte-for-byte match using separate link steps on two different machines. Therefore, we consider this to be an acceptable difference. Also, the locations of dynamic libraries can differ when linking libraries dynamically using our new dynamic shared library support. You can observe this by executing the `ldd` command on the affected binary. On a standalone machine, dynamic links first search in the /opt/cray/xt-sysroot path for dynamic library files that normally would be located in /usr/lib64, /usr/lib, /lib64, and /lib. This allows Cray to provide an alternate set of dynamic shared objects which perfectly matches with that OS version level of XT CLE. When executing on a Cray XT machine, these binaries would need to satisfy those libraries dependencies using their usual location rather than the xt-sysroot path embedded in the binary. With a properly set LD_LIBRARY_PATH, this should usually not be a problem, but it is an area of concern to be aware of. Evaluating Binary Differences Discovering and diagnosing a substantial binary difference requires tools that provide more information than printing an `ls -l` file listing and glancing at the binary size or running the `md5sum` or `cksum` utilities. For one thing, the observed file size can vary based upon the underlying filesystem. More importantly, these methods do not explain the reasons for the binary difference in a useful manner. Explained below are some alternative choices of tools for understanding the cause of a binary difference. 4 Cray Inc. S-0019-10Comparing Binaries Between Cray Linux Environment (CLE) Systems, Standalone Whiteboxes, and ESLogin Nodes First, check the load map to make sure that all libraries were loaded from the proper locations on each machine. To do this, execute your application's compile step with added command line options to pass the “-M” option to the linker: “-Wl,-M”. The compiler should then send a load map to standard output that you can save and compare by redirecting standard output to a file. This load map file shows which library on the host machine is being used to satisfy each symbol in the program. On a whitebox or ESlogin node, the xt-sysroot path provides a mirror of the standard libraries on the XT login node, so most libraries in the loadmap should be obtained from the xt-sysroot path. Second, the `readelf` utility can parse the binary into useful, easy-to-comprehend sections that can be used for comparison purposes. Using the “-a” or all option, the produced output shows the lengths and names of each section, header, and symbol in the application binary file along with other useful information. By performing a graphical `diff` of the two readelf outputs, you can quickly discover which symbols and sections differ. If the symbols differ in size, there is a library mismatch. For dynamically-linked binaries, the `ldd` command can be helpful to show which library paths are likely to be used as the runtime link search path. The `ldd` listing uses LD_LIBRARY_PATH, rpath information from the a.out ELF header, and ld_conf settings in that order to locate a runtime path. Missing path references are listed by the tool and must be accounted for by either loading a module or supplying an LD_LIBRARY_PATH. Finally, you can disassemble the binary with a disassembler utility such as diStorm64 or udis86. Often, the disassembled output is more difficult to read than a loadmap or readelf file because the standard operating system code used to control program startup and termination are encoded into machine code without helpful variable or symbol names to clue in the user to what each code segment accomplishes. Library symbols are not easy to distinguish without their symbol names. However, the disassembler output helps to understand how significant the binary differences are in terms of actual code differences. Setting PGI Core Runtime Libraries During the linking phase with each compiler, a few standard GCC object files are linked into the binary, called the runtime core libraries. Many compilers within the SLES10-based Programming Environment use the GCC 4.1.2 version of these files. They are provided with the GCC 4.1.2 compiler package that is already identical to what is installed on your XT, so they do not normally require any special configuration to avoid binary differences. However, the Portland Group compiler sets its own location for these files that should be overridden by hand using a 'siterc' configuration file. In addition to runtime core libraries, PGI compilers use the SuSE gcc runtime at /usr/lib64/gcc for linking during a build. For SLES10-based systems, this is GCC 4.1.2. Defining this PGI path in a 'siterc' file to use the xtsysroot path ensures that a standalone machine behaves like a service node during compilation and linking. If there are differences between the standalone system and the Cray XT with respect to the runtime core libraries or SuSE gcc runtime, then it is critical to create a 'siterc' file for PGI compilations. This file must be placed in a S-0019-10 Cray Inc. 5Comparing Binaries Between Cray Linux Environment (CLE) Systems, Standalone Whiteboxes, and ESLogin Nodes PGI bin directory that is used exclusively for compiling for the XT. Below is an example of a PGI 'siterc' file which uses your currently loaded xt-sysroot path: ## siterc ...... ## Forces use of /opt/cray/xt-sysroot directory instead of /usr/lib64. ## This only works when xt-sysroot is loaded. When xt-sysroot is ## not loaded, sysroot_gcc_ver is unset and the GCCDIR path is not correct. variable sysroot_dir is environment(SYSROOT_DIR); variable sysroot_gcc_ver is environment(SYSROOT_GCC_VER); set GCCDIR=$sysroot_dir/usr/lib64/gcc/x86_64-suse-linux/$sysroot_gcc_ver; set DEFSTDOBJDIR=$sysroot_dir/usr/lib64; set LCRTI=$STDOBJDIR/crti.o; set LCRTN=$STDOBJDIR/crtn.o; set LCRT1=$STDOBJDIR/crt1.o; set LCRTE=$GCCDIR/crtend.o; set LCRTB=$GCCDIR/crtbegin.o; Conclusion Although there have been known differences observed in the past between the login node and standalone machines, Cray is working hard to minimize all differences that come from our released library packages. We have created a merged Cray Application Developer's Environment release for both XT and standalone systems so that library symbols should not differ between an XT and non-XT system running the same version of CADE, CADES, and licensed products. We have also created an xt-sysroot package for standalone machines for the purpose of providing an identical library environment to the installed version of XT CLE. While we occasionally spotted minor differences during development, we have corrected most that we have found. We are working hard at Cray to replicate the XT library environment completely and accurately, so that all library path and operating system differences are fully accounted for. 6 Cray Inc. S-0019-10 TM Cray Application Developer's Environment User's Guide S–2396–601© 2004–2011 Cray Inc. All Rights Reserved. This document or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Inc. The gnulicinfo(7) man page contains the Open Source Software licenses (the "Licenses"). Your use of this software release constitutes your acceptance of the License terms and conditions. U.S. GOVERNMENT RESTRICTED RIGHTS NOTICE The Computer Software is delivered as "Commercial Computer Software" as de?ned in DFARS 48 CFR 252.227-7014. All Computer Software and Computer Software Documentation acquired by or for the U.S. Government is provided with Restricted Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7014, as applicable. Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7013, as applicable. Cray, LibSci, and PathScale are federally registered trademarks and Active Manager, Cray Apprentice2, Cray Apprentice2 Desktop, Cray C++ Compiling System, Cray CX, Cray CX1, Cray CX1-iWS, Cray CX1-LC, Cray CX1000, Cray CX1000-C, Cray CX1000-G, Cray CX1000-S, Cray CX1000-SC, Cray CX1000-SM, Cray CX1000-HN, Cray Fortran Compiler, Cray Linux Environment, Cray SHMEM, Cray X1, Cray X1E, Cray X2, Cray XD1, Cray XE, Cray XEm, Cray XE5, Cray XE5m, Cray XE6, Cray XE6m, Cray XK6, Cray XMT, Cray XR1, Cray XT, Cray XTm, Cray XT3, Cray XT4, Cray XT5, Cray XT5 h , Cray XT5m, Cray XT6, Cray XT6m, CrayDoc, CrayPort, CRInform, ECOphlex, Gemini, Libsci, NodeKARE, RapidArray, SeaStar, SeaStar2, SeaStar2+, The Way to Better Science, Threadstorm, and UNICOS/lc are trademarks of Cray Inc. AMD Opteron is a trademark of Advanced Micro Devices, Inc.. GNU is a trademark of The Free Software Foundation. Intel is a trademark of Intel Corporation or its subsidiaries in the United States and other countries. Java is a trademark of Oracle and/or its af?liates. Linux is a trademark of Linus Torvalds. Lustre is a trademark of Oracle and/or its af?liates. Other names may be trademarks of their respective owners. Moab and TORQUE are trademarks of Adaptive Computing Enterprises, Inc. MySQL is a trademark of Oracle and/or its af?liates. Opteron is a trademark of Advanced Micro Devices, Inc. PBS and PBS Professional are trademarks of Altair Engineering, Inc. and are protected under U.S. and international law and treaties. PETSc is a trademark of Copyright (C) 1995-2004 University of Chicago. PGI is a trademark of The Portland Group Compiler Technology, STMicroelectronics, Inc. PathScale is a trademark of PathScale Inc. Platform is a trademark of Platform Computing Corporation. Panasas and PanFS are trademarks or registered trademarks of Panasas, Inc. in the United States and other countries. RSA is a trademark of RSA Security Inc. SecurID is a trademark of RSA Security Inc. TotalView and TotalView Technologies are registered trademarks of Rogue Wave Software, Inc. NVIDIA, CUDA, and Tesla are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Windows is a trademark of Microsoft Corporation. UNIX, the “X device,” X Window System, and X/Open are trademarks of The Open Group in the United States and other countries. All other trademarks are the property of their respective owners. RECORD OF REVISION S–2396–601 Published August 2011 Supports general availability (GA) release of Cray XT, Cray XE, and Cray XK systems running Cray Linux Environment (CLE) release 3.0 or later. S–2396–60 Published July 2011 Supports general availability (GA) release of Cray XT, Cray XE, and Cray XK systems running Cray Linux Environment (CLE) release 3.0 or later. 5.0 Published June 2010 Restructured document now packaged with Cray Application Developer's Environment (CADE) release 5.0 and supporting Cray XT and Cray XE systems running Cray Linux Environment (CLE) release 2.2 or later.2.2 Published July 2009 Supports general availability (GA) release of Cray XT systems running the Cray XT Programming Environment and CLE 2.2 releases. 2.1 Published November 2008 Supports general availability (GA) release of Cray XT systems running the Cray XT Programming Environment and CLE 2.1 releases. 2.1 Published July 2008 Supports limited availability (LA) release of Cray XT systems running the Cray XT Programming Environment and CLE 2.1 releases. 2.0 Published October 2007 Supports general availability (GA) release of Cray XT systems running the Cray XT Programming Environment 2.0 and UNICOS/lc 2.0 releases. 2.0 Published June 2007 Supports limited availability (LA) release of Cray XT systems running the Cray XT Programming Environment 2.0 and UNICOS/lc 2.0 releases. 1.5 Published November 2006 Supports general availability (GA) release of Cray XT systems running the Cray XT Programming Environment 1.5 and UNICOS/lc 1.5 releases. 1.5 Published August 2006 Supports limited availability (LA) release of Cray XT systems running the Cray XT Programming Environment 1.5 and Cray Linux Environment (CLE) 1.5 releases. 1.4 Published April 2006 Supports Cray XT3 systems running the Cray XT3 Programming Environment 1.4 and UNICOS/lc 1.4 releases. 1.3 Published November 2005 Supports Cray XT3 systems running the Cray XT3 Programming Environment 1.3 and UNICOS/lc 1.3 releases. 1.2 Published August 2005 Supports Cray XT3 systems running the Cray XT3 Programming Environment 1.2 and UNICOS/lc 1.2 releases. 1.1 Published June 2005 Supports Cray XT3 systems running the Cray XT3 Programming Environment 1.1 and UNICOS/lc 1.1 releases. 1.0 Published March 2005 Draft documentation to support Cray XT3 limited-availability systems. 1.0 Published December 2004 Draft documentation to support Cray XT3 early-production systems.Changes to this Document Cray Application Developer's Environment User's Guide S–2396–601 This rewrite of the Cray Application Developer's Environment User's Guide supports the 6.01 and later releases of the Cray Application Developer's Environment (CADE). S–2396–601 Added information • Chapter 7, formerly "Performance Analysis," is renamed Chapter 7, Optimizing Code on page 73, and has been expanded to include information about Using iobuf on page 73, Improving MPI I/O on page 74, and Using Compiler Optimizations on page 75. Revised information • Minor typographical corrections have been made throughout this guide. In particular, coding errors which caused invalid links to appear in the HTML version of this guide have been corrected. S–2396–60 Added information • Preliminary information regarding support for Cray XK systems has been added throughout this guide. • Preliminary information regarding support for AMD Interlagos processors has been added through this guide. • Use of Cray CPU and network-type targeting modules on Cray systems is described in Using Targeting Modules on page 27. Use of targeting modules on standalone Linux workstations is described in About Cross-compilers on page 43. • A new library of eigensolvers, the Cray Adaptive Simple Eigensolvers (CASE), has been added. For more information, see Cray Adaptive Simple Eigensolvers (CASE) on page 52. • The default behavior of MPI and SHMEM modules has been changed signi?cantly. For more information, see MPT on page 59. • Abnormal Termination Processing (ATP) is now enabled by default. For more information, see Using Abnormal Termination Processing (ATP) on page 65. Revised information • Information regarding the use of hugepages has been revised and expanded. For more information, see Hugepages on page 61. • Beginning with Cray Scienti?c Libraries Release 11.0.00, the way in which dynamic libraries are found at execution time has changed. For more information, see New in LibSci Release 11.0 on page 47. • Beginning with PETSc Release 3.1.08, the Third-Party Scienti?c Libraries (TPSL) module is loaded automatically along with the PETSc module. This change affects both PETSc and Trilinos, and is described in PETSc on page 56 and Trilinos on page 58.Deleted information • Support for Cray Linux Environment (CLE) 2.1 is discontinued. Therefore all references to CLE 2.1 have been removed from this guide. • Beginning with Cray Scienti?c Libraries Release 11.0.00, Cray no longer provides a version of LibSci optimized to support the PathScale Compiler Suite.Contents Page Introduction [1] 11 1.1 What You Must Know About Your Site . . . . . . . . . . . . . . . . . . 11 1.1.1 Cray XT, Cray XE, or Cray XK? . . . . . . . . . . . . . . . . . . 12 1.1.2 How Many Cores? . . . . . . . . . . . . . . . . . . . . . . 12 1.1.3 Which Operating System? . . . . . . . . . . . . . . . . . . . . 13 1.1.4 What is a Compute Node? . . . . . . . . . . . . . . . . . . . . 14 1.1.5 Which File System? . . . . . . . . . . . . . . . . . . . . . . 14 1.1.6 Which Batch System? . . . . . . . . . . . . . . . . . . . . . 15 1.1.7 What is the hostname? . . . . . . . . . . . . . . . . . . . . . 15 1.2 Logging In . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.2.1 UNIX or Linux Users . . . . . . . . . . . . . . . . . . . . . 15 1.2.2 Windows Users . . . . . . . . . . . . . . . . . . . . . . . 16 1.2.3 Apple Macintosh Users . . . . . . . . . . . . . . . . . . . . . 19 1.3 Navigating the File Systems . . . . . . . . . . . . . . . . . . . . . 20 Using Modules [2] 23 2.1 What is Loaded Now? . . . . . . . . . . . . . . . . . . . . . . 24 2.2 What is Available? . . . . . . . . . . . . . . . . . . . . . . . 24 2.3 Loading and Unloading Module?les . . . . . . . . . . . . . . . . . . . 25 2.4 Swapping Module?les . . . . . . . . . . . . . . . . . . . . . . 26 2.5 Using Targeting Modules . . . . . . . . . . . . . . . . . . . . . . 27 2.6 Release Notes and Module Help . . . . . . . . . . . . . . . . . . . . 28 2.7 For More Information . . . . . . . . . . . . . . . . . . . . . . . 29 Batch Systems and Program Execution [3] 31 3.1 Interactive Mode . . . . . . . . . . . . . . . . . . . . . . . . 32 3.1.1 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.2 Batch Mode . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.3 Using aprun . . . . . . . . . . . . . . . . . . . . . . . . . 35 S–2396–601 7Cray Application Developer's Environment User's Guide Page Using Compilers [4] 37 4.1 About Compiler Drivers . . . . . . . . . . . . . . . . . . . . . . 37 4.1.1 Bypassing the Compiler Drivers . . . . . . . . . . . . . . . . . . 38 4.2 About C/C++ Floating Point Data Types . . . . . . . . . . . . . . . . . 38 4.3 About the Cray Compiling Environment (CCE) . . . . . . . . . . . . . . . . 39 4.3.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 39 4.4 About PGI Compilers . . . . . . . . . . . . . . . . . . . . . . . 40 4.4.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 40 4.5 About Intel Compilers . . . . . . . . . . . . . . . . . . . . . . 41 4.5.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 41 4.6 About PathScale Compilers . . . . . . . . . . . . . . . . . . . . . 41 4.6.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 42 4.7 About GNU Compilers . . . . . . . . . . . . . . . . . . . . . . 42 4.7.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 42 4.7.2 Known Warnings . . . . . . . . . . . . . . . . . . . . . . 42 4.8 About the Chapel Parallel Programming Language . . . . . . . . . . . . . . . 43 4.9 About Cross-compilers . . . . . . . . . . . . . . . . . . . . . . 43 Libraries [5] 47 5.1 Basic Scienti?c Libraries (LibSci) . . . . . . . . . . . . . . . . . . . 47 5.1.1 New in LibSci Release 11.0 . . . . . . . . . . . . . . . . . . . . 47 5.1.2 Basic LibSci Components . . . . . . . . . . . . . . . . . . . . 48 5.1.3 BLAS and LAPACK . . . . . . . . . . . . . . . . . . . . . . 48 5.1.3.1 Notes . . . . . . . . . . . . . . . . . . . . . . . . 49 5.1.4 BLACS and ScaLAPACK . . . . . . . . . . . . . . . . . . . . 50 5.1.4.1 Notes . . . . . . . . . . . . . . . . . . . . . . . . 51 5.1.5 Iterative Re?nement Toolkit (IRT) . . . . . . . . . . . . . . . . . . 51 5.1.6 Cray Adaptive Simple Eigensolvers (CASE) . . . . . . . . . . . . . . . 52 5.1.7 Fourier Transformations . . . . . . . . . . . . . . . . . . . . . 52 5.1.7.1 CRAFFT . . . . . . . . . . . . . . . . . . . . . . . 53 5.1.7.2 FFTW . . . . . . . . . . . . . . . . . . . . . . . . 53 5.1.7.3 ACML . . . . . . . . . . . . . . . . . . . . . . . . 54 5.2 Fast Math Intrinsics . . . . . . . . . . . . . . . . . . . . . . . 54 5.3 PETSc . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 5.3.1 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 57 5.4 Trilinos . . . . . . . . . . . . . . . . . . . . . . . . . . 58 5.5 MPT . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 5.5.1 Using MPI and SHMEM Modules . . . . . . . . . . . . . . . . . . 59 8 S–2396–601Contents Page 5.5.1.1 CLE 4.0 Systems . . . . . . . . . . . . . . . . . . . . . 60 5.5.1.2 CLE 3.1 Systems . . . . . . . . . . . . . . . . . . . . . 60 5.5.1.3 CLE 3.0 Systems . . . . . . . . . . . . . . . . . . . . . 61 5.5.1.4 CLE 2.2 Systems . . . . . . . . . . . . . . . . . . . . . 61 5.6 Hugepages . . . . . . . . . . . . . . . . . . . . . . . . . . 61 5.6.1 Cray XT Usage . . . . . . . . . . . . . . . . . . . . . . . 62 5.6.2 Cray XE Usage . . . . . . . . . . . . . . . . . . . . . . . 62 Debugging Code [6] 63 6.1 Cray Debugger Support Tools . . . . . . . . . . . . . . . . . . . . 63 6.1.1 Using lgdb . . . . . . . . . . . . . . . . . . . . . . . . 64 6.1.2 Using Abnormal Termination Processing (ATP) . . . . . . . . . . . . . . 65 6.1.2.1 On CLE 3.0 or Later . . . . . . . . . . . . . . . . . . . . 65 6.1.2.2 On CLE 2.2 or Earlier . . . . . . . . . . . . . . . . . . . . 66 6.1.3 Using MRNet . . . . . . . . . . . . . . . . . . . . . . . 66 6.1.4 Using STAT . . . . . . . . . . . . . . . . . . . . . . . . 66 6.2 Using Cray Fast-Track Debugging . . . . . . . . . . . . . . . . . . . 67 6.2.1 Supported Compilers and Debuggers . . . . . . . . . . . . . . . . . 68 6.3 About Core Files . . . . . . . . . . . . . . . . . . . . . . . . 68 6.4 Using TotalView . . . . . . . . . . . . . . . . . . . . . . . . 68 6.4.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 70 6.5 Using DDT . . . . . . . . . . . . . . . . . . . . . . . . . 70 6.5.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 71 Optimizing Code [7] 73 7.1 Improving I/O . . . . . . . . . . . . . . . . . . . . . . . . . 73 7.1.1 Using iobuf . . . . . . . . . . . . . . . . . . . . . . . 73 7.1.2 Improving MPI I/O . . . . . . . . . . . . . . . . . . . . . . 74 7.2 Using Compiler Optimizations . . . . . . . . . . . . . . . . . . . . 75 7.2.1 Cray Compiling Environment (CCE) . . . . . . . . . . . . . . . . . 75 7.3 Using the Cray Performance Analysis Tools . . . . . . . . . . . . . . . . 76 7.3.1 Instrumenting the Program . . . . . . . . . . . . . . . . . . . . 78 7.3.2 Collecting Data . . . . . . . . . . . . . . . . . . . . . . . 78 7.3.3 Analyzing Data . . . . . . . . . . . . . . . . . . . . . . . 78 7.3.4 For More Information . . . . . . . . . . . . . . . . . . . . . 79 7.4 Using Cray Apprentice2 . . . . . . . . . . . . . . . . . . . . . . 79 7.5 Using PAPI . . . . . . . . . . . . . . . . . . . . . . . . . 80 S–2396–601 9Cray Application Developer's Environment User's Guide Page Figures Figure 1. Selecting SSH Protocol . . . . . . . . . . . . . . . . . . . . 17 Figure 2. Enabling X11 Forwarding . . . . . . . . . . . . . . . . . . . 18 Figure 3. Logging In . . . . . . . . . . . . . . . . . . . . . . . 19 Tables Table 1. aprun Versus qsub Options . . . . . . . . . . . . . . . . . . 36 Table 2. C/C++ Data Type Sizes . . . . . . . . . . . . . . . . . . . . 38 Table 3. Cray Compiler Basics . . . . . . . . . . . . . . . . . . . . . 39 Table 4. PGI Compiler Basics . . . . . . . . . . . . . . . . . . . . . 40 Table 5. Intel Composer Basics . . . . . . . . . . . . . . . . . . . . 41 Table 6. PathScale Compiler Basics . . . . . . . . . . . . . . . . . . . 41 Table 7. GNU Compiler Basics . . . . . . . . . . . . . . . . . . . . 42 Table 8. LibSci Basics . . . . . . . . . . . . . . . . . . . . . . . 48 Table 9. Fast_mv Basics . . . . . . . . . . . . . . . . . . . . . . 54 Table 10. Fast_mv Math Intrinsics . . . . . . . . . . . . . . . . . . . . 55 Table 11. PETSc Basics . . . . . . . . . . . . . . . . . . . . . . 56 Table 12. Trilinos Basics . . . . . . . . . . . . . . . . . . . . . . 58 Table 13. MPT Basics . . . . . . . . . . . . . . . . . . . . . . . 59 Table 14. lgdb Basics . . . . . . . . . . . . . . . . . . . . . . . 64 Table 15. atp Basics . . . . . . . . . . . . . . . . . . . . . . . 65 Table 16. MRNet Basics . . . . . . . . . . . . . . . . . . . . . . 66 Table 17. STAT Basics . . . . . . . . . . . . . . . . . . . . . . . 66 Table 18. TotalView Basics . . . . . . . . . . . . . . . . . . . . . 68 Table 19. DDT Basics . . . . . . . . . . . . . . . . . . . . . . . 70 Table 20. IOBUF Basics . . . . . . . . . . . . . . . . . . . . . . 73 Table 21. MPI I/O Basics . . . . . . . . . . . . . . . . . . . . . . 74 Table 22. Performance Analysis Basics . . . . . . . . . . . . . . . . . . 76 10 S–2396–601Introduction [1] This guide describes the software environment and tools used to develop, debug, and run applications on Cray XT, Cray XE, and Cray XK systems. It is intended as a general overview and introduction to the Cray system for new users and application programmers. This guide is intended to be used in conjunction with Workload Management and Application Placement for the Cray Linux Environment (S–2496), which describes the Application Level Placement Scheduler (ALPS) and aprun command in considerably greater detail. The information contained in this guide is of necessity fairly high-level and generalized, as the Cray platform supports a wide variety of hardware nodes as well as many different compilers, debuggers, and other software tools. System hardware and software con?gurations therefore vary considerably from site to site, so for speci?c information about your site and its installed hardware, software, and usage policies, always contact your site administrator. 1.1 What You Must Know About Your Site The Cray XT, Cray XE, and Cray XK systems are: • massively parallel (MPP) • distributed memory • non-uniform memory access (NUMA) • commodity processor-based supercomputers These systems are designed to scale to immense size and run applications requiring high-performance, large-scale processing, high network bandwidth, and complex inter-processor communications. All systems use 64-bit AMD Opteron processors as the basic computational engines, although the exact number of cores per AMD Opteron varies from site to site and sometimes even from cabinet to cabinet, depending on your site's installation, expansion, and upgrade history. S–2396–601 11Cray Application Developer's Environment User's Guide 1.1.1 Cray XT, Cray XE, or Cray XK? From the programmer's point of view, the most signi?cant differences between the systems are: • Cray XT systems use SeaStar or SeaStar2+ application-speci?c integrated circuits (ASICs) to manage inter-processor communications • Cray XE systems use Gemini ASICs to manage inter-processor communications • Cray XK systems also use Gemini ASICs for inter-processor communications, but combine AMD Interlagos CPUs and NVIDIA Tesla GPUs on compute nodes Because of the differences in the network ASICs and accompanying network APIs, Cray XT and Cray XE systems use different versions of the MPI and SHMEM libraries, which offer different sets of optional controls to the application developer. Cray XK systems use the same MPI and SHMEM libraries as Cray XE systems. The differences between the versions of MPI and SHMEM are discussed in more detail in MPT on page 59. Cray XE and Cray XK systems support the Generic Network Interface (GNI) and Distributed Shared Memory Application (DMAPP) APIs, which offer the programmer API-level access to the advanced features of the Gemini network interconnect. The Gemini GNI and DMAPP APIs are discussed in detail in Using the GNI and DMAPP APIs (S–2446). 1.1.2 How Many Cores? There are signi?cant differences are between models. • Cray XT4 systems use one dual- or quad-core ("Barcelona") AMD Opteron CPU per compute node • Cray XT5 and Cray XE5 systems use two four-core ("Shanghai") or six-core ("Istanbul") AMD Opteron CPUs per compute node • Cray XT6 and Cray XE6 systems use two eight- or twelve-core ("Magny-Cours") or two sixteen-core ("Interlagos") AMD Opteron CPUs per compute node • Cray XK systems use one sixteen-core ("Interlagos") AMD Opteron CPU and one NVIDIA Tesla GPU per compute node Because of these hardware differences, you can invoke different options when compiling and executing your programs. These differences are discussed in more detail in Workload Management and Application Placement for the Cray Linux Environment (S–2496). 12 S–2396–601Introduction [1] 1.1.3 Which Operating System? All current Cray systems run the Cray Linux Environment (CLE) operating system on the login nodes and a lightweight kernel called CNL on the compute nodes. Some of the options available to application developers vary depending on which version of CLE is currently running on the system. • Cray XK systems run CLE release 4.0 or later. • Cray XE5 and Cray XE6 systems run CLE release 3.1 or later. • Cray XT6 and Cray XT6m systems run CLE release 3.0 or later. • All other systems may be running CLE release 2.2, 3.0, 3.1, or later. If you are not certain which release your site is using, check the MOTD (message of the day) when you log in. If the information is not there, there are two other ways to determine the CLE release number. • On CLE 2.x systems, check to see which version of the xt-os module is loaded by default. (For more information about using modules, see Chapter 2, Using Modules on page 23.) If no xt-os module is present, your system is running CLE release 3.0 or later. • On CLE 3.0 and later systems, cat the contents of the /etc/opt/cray/release/clerelease ?le. This returns the CLE release and update number. If this ?le or directory does not exist, your system is running CLE release 2.x. Note: CLE 2.x is based on SLES 10. CLE 3.x and CLE 4.x are based on SLES 11. S–2396–601 13Cray Application Developer's Environment User's Guide 1.1.4 What is a Compute Node? From the application developer's point of view, a Cray system is a tightly integrated network of thousands of nodes, all specialized for different purposes. While some are dedicated to administrative or networking functions and therefore off-limits to application programmers, the two you will deal with most often are: • login nodes — The node you access when you ?rst log in to the system. Login nodes offer the full Cray Linux Environment (CLE) operating system, are used for basic development tasks such as editing ?les and compiling code, generally have access to the network ?le system, and are shared resources that may be used concurrently by multiple users. Login nodes are also sometimes called service nodes. • compute nodes — The nodes on which production jobs are executed. Compute nodes run a lightweight operating system called Compute Node Linux (CNL), can be accessed only by submitting jobs through the batch management system (typically PBS Pro, Moab/TORQUE, or Platform LSF), generally have access only to the high-performance parallel ?le system (typically Lustre or Panasas), and are dedicated resources, exclusively yours for the duration of the batch reservation. When new users ?rst begin working on the Cray system, this difference between login and compute nodes can be confusing. Remember, when you ?rst log in to the system, you are placed on a login node. You cannot execute parallel programs on the login node, nor can you directly access ?les stored on the high-performance parallel ?le system. Instead, use your site's batch system to place parallel programs on the compute nodes, either from the login node or from a mount-point on the parallel ?le system. Note: You can execute serial (single-process) programs on login nodes, but executing large or long-running serial programs on login nodes is discouraged, as login nodes are shared resources. 1.1.5 Which File System? All Cray systems require the use of a high-performance parallel ?le system. Most sites currently use the Lustre File System, although others such as the Panasas PanFS are also supported. All examples shown in this guide were developed on a Lustre ?le system using Lustre commands. Before copying any examples from this guide verbatim, verify which ?le system your site uses and what your site's policies are regarding home directories, scratch space, disk quotas, backup policies, and so on. Then if required, adjust the instructions accordingly. 14 S–2396–601Introduction [1] 1.1.6 Which Batch System? Cray systems typically operate under the control of a batch system such as PBS Pro, OpenPBS, Moab/TORQUE Resource Manager, or Platform LSF. All examples shown in this guide were developed using either PBS Pro or Moab/TORQUE. Before copying any examples from this guide verbatim, verify which batch system your site uses and if required, adjust the instructions accordingly. 1.1.7 What is the hostname? All Cray systems have a unique hostname. While it is possible to log in to a given system by using its IP address (e.g., 127.0.0.1), it is easier if you use the hostname. The form of the hostname varies from site to site. Some sites use a single word; e.g., narwhal. Others use a fully quali?ed URL; e.g., narwhal.univ-alaska-barrow.edu. You should be told which name and form to use when your user account is set up. 1.2 Logging In User account setup and authentication policies vary widely from site to site. In general, you must contact your site administrator to get a login account on the system. Any site-speci?c security or authentication policies (such as, say, the correct use of an RSA SecurID token, if provided) should be explained to you at that time. Once your user account is created, log in to the Cray system using SSH (Secure Shell), protocol version 2. SSH is a remote login program that encrypts all communications between the client and host and replaces the earlier telnet, rlogin, and rsh programs. 1.2.1 UNIX or Linux Users If you use a UNIX or Linux workstation, the ssh utility is generally available at any command line and documented in the ssh(1) man page. To log in to the Cray system, enter: % ssh -X hostname The -X option enables X11 display forwarding. Automatic forward of X11 windows is highly recommended as many application development tools use GUI displays. S–2396–601 15Cray Application Developer's Environment User's Guide On some systems, you may be required to enter your user ID as well. This can be done in several different ways. For example: % ssh -X -luserID hostname Or % ssh -X userID@hostname In any case, after you SSH to the system, you may have to answer one or more RSA or password challenges, and then you are logged into the system. A series of system status and MOTD (message of the day) messages may display, after which you are placed in your home directory on a login node. /users/userID> You are now ready to begin working. Jump to Navigating the File Systems on page 20. 1.2.2 Windows Users If you use a Windows personal computer, you ?rst need to obtain and install a client program for your system that supports SSH protocol 2, such as PuTTY for Windows. Your system administrator should be able to provide a list of accepted clients. 16 S–2396–601Introduction [1] You may need to con?gure your client to support SSH protocol 2 and X11 forwarding. For example, if you are using PuTTY, you may need to click SSH in the left pane to see the preferred SSH protocol version: Figure 1. Selecting SSH Protocol Verify that the Preferred SSH protocol version is set to 2. S–2396–601 17Cray Application Developer's Environment User's Guide Then click X11 in the left pane to view the SSH X11 forwarding options: Figure 2. Enabling X11 Forwarding If necessary, click the Enable X11 forwarding checkbox. 18 S–2396–601Introduction [1] Then click Session in the left pane to return to the Basic options window. Figure 3. Logging In Enter the hostname in the Host Name ?eld and click the Open button to begin your SSH session. You may need to enter your userID and answer one or more RSA or password challenges, and then you are logged into the system. A series of system status and MOTD (message of the day) messages may display, after which you are placed in your home directory on a login node. /users/userID> You are now ready to begin working on the Cray system. 1.2.3 Apple Macintosh Users The Apple Macintosh OS X operating system is based on UNIX. Therefore, to log in to the Cray system, open the Terminal application, and then use the ssh command to connect to the Cray system. % ssh -X hostname S–2396–601 19Cray Application Developer's Environment User's Guide The -X option enables X11 display forwarding with X11 security extension restrictions. Automatic forward of X11 windows is highly recommended as many application development tools use GUI displays. Note: The version of SSH found in OS X also supports the -Y argument, as well as the -X argument. The -Y argument enables "trusted" X11 forwarding and may work better than -X for some users. On some systems, you may be required to enter your user ID as well. This can be done in several different ways. For example: % ssh -X -luserID hostname Or % ssh -X userID@hostname In any case, after you SSH to the system, you may have to answer one or more RSA or password challenges, and then you are logged into the system. A series of system status and MOTD (message of the day) messages may display, after which you are placed in your home directory on a login node. /users/userID> You are now ready to begin working on the Cray system. 1.3 Navigating the File Systems When you ?rst log in to the Cray system, you are placed in your home directory on a login node. /users/userID> At this point you have access to all the features and functions of the full Cray Linux Environment (CLE) operating system, such as the sftp and scp commands, and also typically to your full network ?le system. On most systems your home directory on the login node is de?ned as the environment variable $HOME, and this variable can be used in any ?le system command. For example, to return to your home directory from any other location in the ?le system(s), enter this command: > cd $HOME Remember, you can edit ?les, manipulate ?les, compile code, execute serial (single-process) programs, and otherwise work in your home directory on the login node. However, you cannot execute parallel programs on the login node. Parallel programs must be run on the compute nodes, under the control of the batch system, and generally while mounted on the high-performance parallel ?le system. To do this, you must ?rst identify the nids (node IDs) of the ?le system mount points. On the Lustre ?le system, this can be done in one of two ways. 20 S–2396–601Introduction [1] Either enter the df -t lustre command to ?nd the Lustre nodes and get a summary report on disk usage: users/userID> df -t lustre Filesystem 1K-blocks Used Available Use% Mounted on 8@ptl:/narwhalnid8 8998913280 6946443260 1595348672 82% /lus/nid00008 Or enter the lfs df command to get more detailed information: users/userID> lfs df UUID 1K-blocks Used Available Use% Mounted on nid00008_mds_UUID 179181084 2675664 166265604 1% /lus/nid00008[MDT:0] ost0_UUID 1124864160 895207088 172517160 79% /lus/nid00008[OST:0] ost1_UUID 1124864160 838067380 229656540 74% /lus/nid00008[OST:1] ost2_UUID 1124864160 826599428 241124820 73% /lus/nid00008[OST:2] ost3_UUID 1124864160 827914052 239801932 73% /lus/nid00008[OST:3] ost4_UUID 1124864160 964324672 103398548 85% /lus/nid00008[OST:4] ost5_UUID 1124864160 932986208 134738024 82% /lus/nid00008[OST:5] ost6_UUID 1124864160 832715148 235009164 74% /lus/nid00008[OST:6] ost7_UUID 1124864160 828631656 239092572 73% /lus/nid00008[OST:7] filesystem summary: 8998913280 6946445632 1595338760 77% /lus/nid00008 Note: The above commands are speci?c to the Lustre high-speed parallel ?le system. If your site uses a different ?le system, adjust the instructions accordingly. In either case, in this example, the Lustre mount point is /lus/nid00008. If you cd to this mount point: users/userID> cd /lus/nid00008 Directory: /lus/nid00008 /lus/nid00008> you are now on the high-performance parallel ?le system. At this point you can edit ?les, manipulate ?les, compile code, and so on, but you can also execute programs on the compute nodes, generally by using the batch system. S–2396–601 21Cray Application Developer's Environment User's Guide 22 S–2396–601Using Modules [2] The Cray system uses the Modules environment management package to support dynamic modi?cation of the user environment via module?les. Each module?le contains all the information needed to con?gure the shell for a particular application. To make major changes in your user environment, such as switching to a different compiler or a different version of a library, use the appropriate Modules commands to select the desired module?les. The advantage in using Modules is that you are not required to specify explicit paths for different executable versions or to set the $MANPATH and other environment variables manually. Instead, all the information required in order to use a given piece of software is embedded in the module?le and set automatically when you load the module?le. In general, the simplest way to make certain that the elements of your application development environment function correctly together is by using the Modules software and trusting it to keep track of paths and environment variables, and avoiding embedding speci?c directory paths into your startup ?les, make?les, and scripts. S–2396–601 23Cray Application Developer's Environment User's Guide 2.1 What is Loaded Now? When you ?rst log in to the Cray system, a set of site-speci?c default modules is loaded. This set varies depending on system hardware, operating system release level, site policies, and installed software. To see which modules are currently loaded on your system, use the module list command. users/yourname> module list Currently Loaded Modulefiles: 1) modules/3.2.6.6 2) nodestat/2.2-1.0400.25564.5.2.gem 3) sdb/1.0-1.0400.27358.12.17.gem 4) MySQL/5.0.64-1.0000.2899.19.2 5) lustre-cray_gem_s/1.8.4_2.6.32.36_0.5.2_1.0400.5941.3.1-1.0400.27356.3.114 6) udreg/2.3.1-1.0400.3052.6.12.gem 7) ugni/2.2-1.0400.3340.9.29.gem 8) gni-headers/2.1-1.0400.3411.8.1.gem 9) dmapp/3.1-1.0400.3387.13.25.gem 10) xpmem/0.1-2.0400.27172.6.5.gem 11) Base-opts/1.0.2-1.0400.25719.5.1.gem 12) xtpe-network-gemini 13) pgi/11.6.0 14) totalview-support/1.1.2 15) xt-totalview/8.9.1 16) xt-libsci/10.5.02 17) pmi/2.1.2-1.0000.8396.13.5.gem 18) xt-asyncpe/5.00.59 19) atp/1.2.0 20) PrgEnv-pgi/4.0.12A 21) pbs/10.4.0.101257 22) xtpe-mc12 23) xt-mpich2/5.3.0 This list breaks down into three groups: operating system modules, programming environment modules, and support modules. For example, the xtpe-mc12 module indicates that this development environment is set up to develop code for use on 12-core Magny-Cours processors, while the PrgEnv-pgi module indicates that PGI compiler suite is currently loaded. 2.2 What is Available? To see which module?les are available on your system, enter this command: % module avail [string] [-subset?ag] 24 S–2396–601Using Modules [2] The module avail command produces an alphabetical listing of every module?le in your module use path and has no option for "grepping." Therefore, it is usually more useful to use the command with an string argument. For example, if you are looking for a speci?c version of the PGI compiler suite, you would enter this command: users/yourname> module avail PrgEnv-pgi -------------------------------------- /opt/modulefiles -------------------------------------- PrgEnv-pgi/3.1.35 PrgEnv-pgi/3.1.37E PrgEnv-pgi/3.1.61 PrgEnv-pgi/3.1.37AA PrgEnv-pgi/3.1.37G PrgEnv-pgi/4.0.12A(default) PrgEnv-pgi/3.1.37C PrgEnv-pgi/3.1.49A One module is usually designated as the default version. Whether this is the most recent version of this module depends on your site policies. Some sites always make the newest version the default, while others wait until after the new version has been tested and proven bug- and dependency-free. Whenever a newer version of a module is installed, the older versions continue to remain available, unless the site administrator has explicitly chosen to delete them. The [-subset?ag] option lets you list a subset of available modules. The following ?ags may be used alone or in combinations: -U List user modules -D List the current default modules -T List tool modules (debuggers, performance analysis utilities, and the like) -L List library modules (see Chapter 5, Libraries on page 47) -P List Programming Environment (compiler) modules -X List CPU and network targeting modules (Barcelona, Magny-Cours, Interlagos, and the like) 2.3 Loading and Unloading Module?les If a module?le is not already loaded, use the module load command to load it. users/yourname> module load module?le This command loads the currently de?ned default version of the module, unless you specify otherwise. For example, if the modules listed in What is Available? on page 24 are present on your system but PrgEnv-pgi is not already loaded, then: users/yourname> module load PrgEnv-pgi loads PrgEnv-pgi/4.0.12A(default) module. To load a non-default version of the module, you must specify the full module name. users/yourname> module load PrgEnv-pgi/3.1.61 S–2396–601 25Cray Application Developer's Environment User's Guide However, if a module is already loaded, then you must ?rst unload the currently loaded module before loading a different version. For example, to change from the default to version of the PGI compiler suite to another version, use the module unload command to remove the version currently loaded. users/yourname> module unload PrgEnv-pgi Modules may be linked and related. If you enter the module list command now and compare the results to those shown in What is Loaded Now? on page 24: users/yourname> module list Currently Loaded Modulefiles: 1) modules/3.2.6.6 2) nodestat/2.2-1.0400.25564.5.2.gem 3) sdb/1.0-1.0400.27358.12.17.gem 4) MySQL/5.0.64-1.0000.2899.19.2 5) lustre-cray_gem_s/1.8.4_2.6.32.36_0.5.2_1.0400.5941.3.1-1.0400.27356.3.114 6) udreg/2.3.1-1.0400.3052.6.12.gem 7) ugni/2.2-1.0400.3340.9.29.gem 8) gni-headers/2.1-1.0400.3411.8.1.gem 9) dmapp/3.1-1.0400.3387.13.25.gem 10) xpmem/0.1-2.0400.27172.6.5.gem 11) Base-opts/1.0.2-1.0400.25719.5.1.gem 12) xtpe-network-gemini 13) pbs/10.4.0.101257 14) xtpe-mc12 15) xt-mpich2/5.3.0 You can see that along with PrgEnv-pgi, the pgi, totalview-support, xt-totalview, xt-libsci, pmi, xt-asyncpe, and atp modules were also unloaded. If you now load a different version of the PrgEnv-pgi module: users/yourname> module load PrgEnv-pgi/3.1.61 Then use the module list command again, you can see that the associated modules have been reloaded automatically, but this time using the versions that are appropriate for use with the 3.1.61 version of the PGI compiler. 2.4 Swapping Module?les Alternatively, you can use the module swap or module switch command to unload one module and load the comparable module. For example, to switch from using the PGI Compiler Suite to the Cray Compiling Environment, enter this command: users/yourname> module swap PrgEnv-pgi PrgEnv-cray 26 S–2396–601Using Modules [2] If you then list the modules that are loaded after this swap and compare them to the results shown in What is Loaded Now? on page 24. users/yourname> module list Currently Loaded Modulefiles: 1) modules/3.2.6.6 2) nodestat/2.2-1.0400.25564.5.2.gem 3) sdb/1.0-1.0400.27358.12.17.gem 4) MySQL/5.0.64-1.0000.2899.19.2 5) lustre-cray_gem_s/1.8.4_2.6.32.36_0.5.2_1.0400.5941.3.1-1.0400.27356.3.114 6) udreg/2.3.1-1.0400.3052.6.12.gem 7) ugni/2.2-1.0400.3340.9.29.gem 8) gni-headers/2.1-1.0400.3411.8.1.gem 9) dmapp/3.1-1.0400.3387.13.25.gem 10) xpmem/0.1-2.0400.27172.6.5.gem 11) Base-opts/1.0.2-1.0400.25719.5.1.gem 12) xtpe-network-gemini 13) pbs/10.4.0.101257 14) xtpe-mc12 15) PrgEnv-cray/4.0.12A 16) atp/1.2.0 17) xt-asyncpe/5.00.59 18) rca/1.0.0-2.0400.27173.6.23.gem 19) pmi/2.1.2-1.0000.8396.13.5.gem 20) xt-libsci/10.5.02 21) acml/4.4.0 22) xt-totalview/8.9.1 23) totalview-support/1.1.2 24) cce/7.4.1.103 You can see that a different set of supporting modules have been loaded automatically. 2.5 Using Targeting Modules The targeting modules deserve special mention. To see which targeting modules are available on your system, use the module avail -X command. It returns a list like this, which shows the CPU and network-type modules currently available. ---------------------------- /opt/cray/xt-asyncpe/default/modulefiles ---------------------------- xtpe-barcelona xtpe-mc12 xtpe-network-seastar xtpe-interlagos xtpe-mc8 xtpe-shanghai xtpe-istanbul xtpe-network-gemini xtpe-target-native If you are working on a Cray system, your default environment should load the CPU and network-type modules that are correct for your hardware. For example, if you have a Cray XE6 system with eight-core Magny-Cours compute nodes, your default environment should include the xtpe-network-gemini and xtpe-mc8 modules, and no others. If the correct modules are loaded, do not change them. S–2396–601 27Cray Application Developer's Environment User's Guide However, if you are working on a standalone Linux workstation and developing executable code which will then be moved to and run on a Cray system, always make certain that your local development environment contains the correct targeting modules for the Cray system on which you plan to run your code. For example, code compiled with the wrong CPU module loaded, or with the xtpe-network-seastar module instead of the xtpe-network-gemini module loaded, will not run correctly on a Cray XE system. For more information about using standalone Linux workstations to develop code, see About Cross-compilers on page 43. Note: Alternatively, if your site has a heterogeneous system with more than one type of compute node (for example, a Cray XE6 system with both Magny-Cours and Interlagos compute nodes), load the targeting module for the type of compute node on which you intend to execute your code, and then make certain your job is placed only on the speci?ed type of compute node. For more information about job placement, see Workload Management and Application Placement for the Cray Linux Environment (S–2496). Note: The xtpe-target-native module is not a CPU or network-type module and used only in limited and special cases. For more information, see Bypassing the Compiler Drivers on page 38. 2.6 Release Notes and Module Help Most modules on the Cray system include module help that is speci?c to the module. The exact content of the module help varies from vendor to vendor and release to release, but generally includes release notes and late-breaking news, such as lists of bugs ?xed in the release, known dependencies and limitations, and usage information received too late to be documented elsewhere. You can view the module help at any time for any module currently installed on the system. The module does not need to be loaded in order for you to view the module help. To access the module help, use the module help command. For example, to see the module help associated with the default PGI module, enter this command: users/yourname> module help pgi Note: Make certain you specify the exact module name (and if not the default, the module version) that you want. For example, module help PrgEnv-pgi and module help pgi display different information. 28 S–2396–601Using Modules [2] 2.7 For More Information The Modules commands are documented in the module(1) and modulefiles(4) man pages. A summary of Modules commands can be displayed by entering the module help command. users/yourname> module help Modules Release 3.2.6.6 2007-02-14 (Copyright GNU GPL v2 1991): Usage: module [ switches ] [ subcommand ] [subcommand-args ] Switches: -H|--help this usage info -V|--version modules version & configuration options -f|--force force active dependency resolution -t|--terse terse format avail and list format -l|--long long format avail and list format -h|--human readable format avail and list format -v|--verbose enable verbose messages -s|--silent disable verbose messages -c|--create create caches for avail and apropos -i|--icase case insensitive -u|--userlvl set user level to (nov[ice],exp[ert],adv[anced]) Available SubCommands and Args: + add|load modulefile [modulefile ...] + rm|unload modulefile [modulefile ...] + switch|swap [modulefile1] modulefile2 + display|show modulefile [modulefile ...] + avail [modulefile [modulefile ...]] + use [-a|--append] dir [dir ...] + unuse dir [dir ...] + update + refresh + purge + list + clear + help [modulefile [modulefile ...]] + whatis [modulefile [modulefile ...]] + apropos|keyword string + initadd modulefile [modulefile ...] + initprepend modulefile [modulefile ...] + initrm modulefile [modulefile ...] + initswitch modulefile1 modulefile2 + initlist + initclear Different versions of the Modules software are in use on different sites. Accordingly, the module command arguments and options available on your site may vary from those shown here. S–2396–601 29Cray Application Developer's Environment User's Guide 30 S–2396–601Batch Systems and Program Execution [3] User applications are always launched on compute nodes using the application launcher, aprun, which submits applications to the Application Level Placement Scheduler (ALPS) for placement and execution. The ALPS service is both very powerful and highly ?exible, and a thorough discussion of it is beyond the scope of this manual. For more detailed information about ALPS and aprun, see the intro_alps(1) and aprun(1) man pages, and Workload Management and Application Placement for the Cray Linux Environment (S–2496). At most sites, access to the compute node resources is managed by a batch control system, typically PBS Pro, Moab/TORQUE, or Platform LSF. Users run jobs either by using the qsub command to submit a job script (or the equivalent command for their batch control system), or else by using the qsub command (or its equivalent) to request an interactive session within the context of the batch system, and then using the aprun command to run the application within the interactive session. In either case, running an application typically involves these steps. 1. Determine what system resources you will need. Generally, this means deciding how many cores and/or compute nodes you need for your job. 2. Use the apstat command to determine whether the resources you need are available. This is very important when you are planning to run in an interactive session. This is not as important if you are submitting a job script, as the batch system will keep your job script in the queue until the resources become available to run it. 3. Translate your resource request into the appropriate batch system (i.e., qsub) and aprun command options, which are not necessarily the same. If running a batch job, modify your script accordingly. 4. Use the batch command either to submit your job script or to request an interactive session. 5. If using an interactive session, use the aprun command to launch your application. S–2396–601 31Cray Application Developer's Environment User's Guide 3.1 Interactive Mode Interactive mode is typically used for debugging or optimizing code, but not for running production code. For example, to begin an interactive session on a system using PBS Pro, use the qsub -I command. users/yourname> qsub -IVl mppwidth=cores Useful qsub options include: -I Start an interactive session. -A account Charge the time to account. -q debug Run in the debug queue. -V Import any environment variables that were set in the user's shell. -l resource_type=speci?cation Specify the resources you want to reserve for this session. For example, you can use this option to reserve 24 compute cores for one hour. -l mppwidth=24,walltime=1:00:00 The -l resource_type=speci?cation argument supports a large number of options which are described in the qsub(1B) man page and expanded upon in the pbs_resources(7B) man page, and described in greater detail with examples in Workload Management and Application Placement for the Cray Linux Environment (S–2496). After you have launched an interactive session, use the aprun command to launch your application. When you are ?nished, enter logout to exit the batch system and return to the normal command line. 3.1.1 Notes • You must use the -l mppwidth= option to specify at least one core when you start the interactive session. If you do not, your request for an interactive session will pause inde?nitely. • Pay attention to your ?le system mount points. You must be on the high-performance parallel ?le system (for example, if you are using the Lustre ?le system, /lus/nid00008/yourname) in order to launch jobs on the compute nodes. However, when you launch an interactive batch session, you are by default placed in your home directory, which typically is on a login node. (For example, /ufs/users/home/yourname.) You may need to cd back to the parallel ?le system after launching an interactive session. 32 S–2396–601Batch Systems and Program Execution [3] • After you launch an interactive batch session, a number of environment variables are automatically de?ned and exported to the job, as described on the qsub(1B) man page. For example, the environment variable $PBS_O_WORKDIR is set to the directory from which the batch job was submitted. This can be a handy way to return to your mount point if you cd to the parallel ?le system before invoking the interactive batch session. • The qsub and aprun commands use different options to perform similar functions. These differences are touched on lightly in Using aprun on page 35 and described in detail in Workload Management and Application Placement for the Cray Linux Environment (S–2496). • When you launch an interactive batch session, you must request the maximum number of resources you expect to use. Once a batch session begins, you can only use fewer resources than initially requested. You cannot use the aprun command to use more resources than you reserved using the qsub command. 3.2 Batch Mode Production jobs are typically run in batch mode. Batch scripts are shell scripts containing ?ags and commands to be interpreted by a shell and are used to run a set of commands in sequence. For example, to run a batch script using PBS Pro, use the qsub command. users/yourname> qsub [-l resource_type=speci?cation] jobscript The [-l resource_type=speci?cation] arguments are described in the pbs_resources(7B) man page. A typical batch script might look like this: 1: #!/bin/bash 2: #PBS -A account 3: #PBS -N job_name 4: #PBS -j oe 5: #PBS -l walltime=1:00:00,mppwidth=192 6: 7: cd $PBS_O_WORKDIR 8: date 9: aprun -n 192 ./a.out > my_output_?le 2>&1 S–2396–601 33Cray Application Developer's Environment User's Guide Parsing this script line-by-line, it would be interpreted as follows: 1. Invoke the shell to use to interpret the script. 2. Specify the account to which this time is billed. 3. Assign the job a name to use on messages and output. 4. Join the job's STDOUT and STDERR into STDOUT. Note: While your job is running, STDOUT and STDERR are written to a ?le or ?les in a system directory and the output is copied to your submission directory only after the job completes. Specifying the -j oe option here and redirecting the output to a ?le in line 9 makes it possible for you to view STDOUT and STDERR while the job is running. For more information about the -j option, see the qsub(1B) man page. 5. Reserve 192 processing elements for one hour. 6. This line is blank and is ignored. 7. cd to the submission directory, which presumably is a mount point on the Lustre ?le system. 8. Run the date command. 9. Execute the executable ?le a.out on 192 processing elements and redirect any output to a ?le. After the job is submitted using the qsub command, it goes into the queue, where it waits until the requested resources become available. When they do, the job is launched on the head node of the allocated resources, and it runs until either it reaches its planned completion or until the wall clock time (if speci?ed) is up. While the job is in the queue, a number of optional commands are available. qstat Show the status of the job queue. This command is available at any time, whether or not you have a job in the queue. qdel job_id Delete job job_id regardless of its current state and remove it from the queue. qhold job_id Place a non-running job on hold. The job remains in the queue but will not execute. This command cannot be used once the job begins running. qrls job_id Release a job that is on hold. 34 S–2396–601Batch Systems and Program Execution [3] qalter job_id Alter the characteristics—name, account, number of requested cores, and so on—of a job in the queue. This command cannot be used once the job begins running. showq (Moab only) Similar to qstat but providing more detail. checkjob job_id (Moab only) Check the status of a job currently in the queue. showstart job_id (Moab only) Show the estimated start time for a job in the queue. showbf (Moab only) Show the current back?ll. This can help you to build small jobs that can be back?lled immediately while you are waiting for the resources to become available for your larger jobs. For more information about batch scripts, see your batch system's user documentation. 3.3 Using aprun The aprun utility launches applications on compute nodes. The utility submits applications to the Application Level Placement Scheduler (ALPS) for placement and execution, forwards the login node environment to the assigned compute nodes, forwards signals, and manages the stdin, stdout, and stderr streams. Verify that you are in a directory mounted on the high-speed parallel ?le system before using the aprun command. In simplest form, the aprun command looks like this: % aprun -n x ./program_name The aprun command supports a large number of options that provide you with a high degree of control over just exactly how your job is placed and executed on the compute nodes. At a minimum, you must use the -n option to specify the number of cores on which to run the job. Note: Remember, you use aprun within the context of a batch session and the maximum size of the job is determined by the resources you requested when you launched the batch session. You cannot use the aprun command to use more resources than you reserved using the qsub command. The aprun and qsub commands support comparable but differently named options. This table lists some of the more commonly used aprun options and their qsub equivalents. S–2396–601 35Cray Application Developer's Environment User's Guide Table 1. aprun Versus qsub Options aprun Option qsub -l Option Description -n 4 -l mppwidth=4 Width (number of PEs) -d 2 -l mppdepth=2 Depth (number of CPUs hosting OpenMP threads) -N 1 -l mppnppn=1 Number of PEs per node -L 5,6,7 -l mppnodes=\"5,6,7\" Candidate node List -m 1000 -l mppmem=1000 Memory per PE Note: The -B option forces aprun to inherit the values associated with the -n, -d, -N, and -m options from the batch session. The aprun command exits with an error if you specify any of these options and the -B option at the same time. A full discussion of aprun options is beyond the scope of this manual. For more information see the aprun(1) man page, and for detailed explanations and examples, see Workload Management and Application Placement for the Cray Linux Environment (S–2496). Also, note that the behavior of aprun and the aprun command options supported may vary depending on which version of the CLE operating system is installed on your system. 36 S–2396–601Using Compilers [4] The Cray system supports a variety of compilers from a variety of vendors, and support for new compilers and languages is being added on an ongoing basis. The GNU Fortran, C, and C++ compilers are supplied with all systems, while all other compilers are available as optional and separately licensed add-ons. At present, the following compilers from the following vendors are supported on Cray systems. • Cray Inc., Cray Compiling Environment (CCE) (Fortran, C, and C++) • The Portland Group, Parallel Fortran, C, and C++ • Intel Inc., Intel Composer (Fortran and C++) • PathScale Inc., PathScale Compiler Suite (Fortran and C++) • Chapel Parallel Programming Language The compilers available on your system depend on which products your site administration has chosen to license and install. The different compilers have different strengths and weaknesses, and their relative strengths and weaknesses are constantly changing as new revisions and updates are released. 4.1 About Compiler Drivers Because of the multiplicity of possible compilers, Cray supplies compiler drivers, wrapper scripts, and disambiguation man pages. No matter which vendor's compiler module is loaded, always use one of the following commands to invoke the compiler. ftn Invokes the Fortran compiler, regardless of which compiler module is currently loaded. This command links in the fundamental libraries required in order to produce code that can be executed on the Cray compute nodes. For more information, see the ftn(1) man page. cc Invokes the C compiler, regardless of which compiler module is currently loaded. This command links in the fundamental header ?les and libraries required in order to produce code that can be executed on the Cray compute nodes. For more information, see the cc(1) man page. S–2396–601 37Cray Application Developer's Environment User's Guide CC Invokes the C++ compiler, regardless of which compiler module is currently loaded. This command links in the fundamental header ?les and libraries required in order to produce code that can be executed on the Cray compute nodes. For more information, see the CC(1) man page. Note that while you always use one of the above commands (either on the command line or in your make ?les) to invoke the compiler, the arguments used with the commands vary according to which compiler module is loaded. For example, the arguments and options supported by the PGI Fortran compiler are different from those supported by the Cray Fortran compiler. Regardless of which compiler module you have loaded, do not use the native compiler commands. For example, if you are using the PGI compiler suite, do not use the pgf95 command to invoke the Fortran compiler. If you do so, your code may appear to compile and link successfully, but it will be linked to the wrong libraries and the resulting program can be executed on login nodes only; it cannot be executed on compute nodes. 4.1.1 Bypassing the Compiler Drivers In special cases, you may want to bypass the compiler drivers and use the native compiler commands. Do not do so. Instead, load the special targeting module, xtpe-target-native, and then continue to use the ftn, cc, and CC commands as before. The xtpe-target-native module enables the compiler driver commands to function as native compiler commands—for example, if the PGI programming is loaded, the ftn command works as if it is pgf95—but eliminates all default library links, while also preventing any linking to incorrect libraries. To restore normal compiler driver behavior, unload the xtpe-target-native module. 4.2 About C/C++ Floating Point Data Types The C/C++ compilers differ on the size of the long double data type. The PGI and CCE compilers de?ne long double as being 8 bytes. All other compilers de?ne the long double as being 16 bytes. Table 2. C/C++ Data Type Sizes Data Type Size in Bytes unsigned char 1 signed char 1 unsigned short 2 signed short 2 38 S–2396–601Using Compilers [4] Data Type Size in Bytes unsigned int 4 signed int 4 unsigned long 8 signed long 8 unsigned long long 8 signed long long 8 ?oat 4 double 8 long double 8 or 16, depending on compiler char * 8 enum 4 4.3 About the Cray Compiling Environment (CCE) Table 3. Cray Compiler Basics Module: PrgEnv-cray Command: ftn, cc, CC Compiler-speci?c man pages: crayftn(1), craycc(1), crayCC(1) Note: Compiler-speci?c man pages are available only when the compiler module is loaded. Online help: None provided Documentation: Cray Fortran Reference Manual, Cray C and C++ Reference Manual 4.3.1 Known Limitations If you use the Cray Fortran compiler with the PETSc (Portable, Extensible Toolkit for Scienti?c Computation) library, either add the directive !dir$ PREPROCESS EXPAND_MACROS to the source code or add the -F option to the ftn command line. S–2396–601 39Cray Application Developer's Environment User's Guide 4.4 About PGI Compilers Table 4. PGI Compiler Basics Module: PrgEnv-pgi Command: ftn, cc, CC Compiler-speci?c man pages: pgf95(1), pgcc(1), pgCC(1) Note: Compiler-speci?c man pages are available only when the compiler module is loaded. Online help: pgf95 -help, pgcc -help, pgCC -help, Documentation: /opt/pgi/version/linux86-64/version/doc 4.4.1 Known Limitations • At this time, PGI compilers do not produce code optimized for AMD Interlagos processors. Code compiled with the PGI compilers can be expected to suffer performance degradation if executed on compute nodes with Interlagos CPUs. This limitation is expected to be removed in a future release. • The PGI compilers are not able to handle template-based libraries such as Tpetra. • When linking in ACML routines, you must compile and link all program units with -Mcache_align or an aggregate option that incorporates -Mcache_align such as fastsse. • The -Mconcur (auto-concurrentization of loops) option is not supported on Cray systems. • The -mprof=mpi, -Mmpi, and -Mscalapack options are not supported. • The PGI debugger, PGDBG, is not supported on Cray systems. • The PGI pro?ling tools, pgprof and pgcollect, are not supported on Cray systems. • The PGI Compiler Suite does not support the UPC or Coarray Fortran parallel programming models. 40 S–2396–601Using Compilers [4] 4.5 About Intel Compilers Table 5. Intel Composer Basics Module: PrgEnv-intel Command: ftn, cc, CC Compiler-speci?c man pages: ifort(1), fpp(1), icc(1), icpc(1) Note: Compiler-speci?c man pages are available only when the compiler module is loaded. Online help: ifort --help, icc --help Documentation: /opt/intel/Compiler/version/Documentation/language 4.5.1 Known Limitations • The Intel Compiler Suite (Intel Composer) must be installed in the default location. The optional Intel C/C++ only installation is not supported because the Intel Fortran run time libraries are required by Cray libraries such as libsci when using the Intel compiler. • The Intel Composer does not support the UPC or Coarray Fortran parallel programming models. 4.6 About PathScale Compilers Table 6. PathScale Compiler Basics Module: PrgEnv-pathscale Command: ftn, cc, CC Compiler-speci?c man pages: pathf95(1), pathf90(1), pathcc(1), pathCC(1), eko(7) Note: Compiler-speci?c man pages are available only when the compiler module is loaded. Online help: pathf95 --help, pathf90 --help, pathcc --help, pathCC --help Documentation: /opt/pathscale/share/doc/version S–2396–601 41Cray Application Developer's Environment User's Guide 4.6.1 Known Limitations • The PathScale Compiler Suite does not support the UPC or Coarray Fortran parallel programming models. • The PETSc 3.1.0.8 library of parallel linear and nonlinear solvers does not support the PathScale Compiler Suite. Use PETSc 3.0.x instead. • Beginning with PathScale release 4.0.9, Cray no longer provides PathScale-speci?c versions of the Cray Scienti?c Libraries (LibSci), or of PETSc, Trilinos, or other third-party scienti?c libraries (TPSL). Users are advised to obtain third-party scienti?c libraries directly from the respective library developers. 4.7 About GNU Compilers Table 7. GNU Compiler Basics Module: PrgEnv-gnu Command: ftn, cc, CC Compiler-speci?c man pages: gfortran(1), gcc(1), g++(1) Note: Compiler-speci?c man pages are available only when the compiler module is loaded. Online help: gfortran --help, gcc --help, g++ --help 4.7.1 Known Limitations The GNU compilers do not support the UPC or Coarray Fortran parallel programming models. 4.7.2 Known Warnings Code compiled with gfortran using the options --whole-archive,-lpthread gets the following warning message issued by libpthread.a(sem_open.o): warning: the use of 'mktemp' is dangerous, better use 'mkstemp' The --whole-archive option is necessary to avoid a runtime segmentation fault when using OpenMP libraries. This warning can be safely ignored. 42 S–2396–601Using Compilers [4] 4.8 About the Chapel Parallel Programming Language Chapel is a new parallel programming language being developed by Cray Inc. as part of the DARPA-led High Productivity Computing Systems program (HPCS). Chapel is designed to improve the productivity of high-end computer users while also serving as a portable parallel programming model that can be used on commodity clusters or desktop multicore systems. Chapel strives to vastly improve the programmability of large-scale parallel computers while matching or beating the performance and portability of current programming models like MPI. Chapel supports a multithreaded execution model via high-level abstractions for data parallelism, task parallelism, concurrency, and nested parallelism. Chapel's locale type enables users to specify and reason about the placement of data and tasks on a target architecture in order to tune for locality. Chapel supports global-view data aggregates with user-de?ned implementations, permitting operations on distributed data structures to be expressed in a natural manner. In contrast to many previous higher-level parallel languages, Chapel is designed around a multiresolution philosophy, permitting users to initially write very abstract code and then incrementally add more detail until they are as close to the machine as their needs require. Chapel supports code reuse and rapid prototyping via object-oriented design, type inference, and features for generic programming. Chapel was designed from ?rst principles rather than by extending an existing language. It is an imperative block-structured language, designed to be easy to learn for users of C, C++, Fortran, Java, Perl, Matlab, and other popular languages. While Chapel builds on concepts and syntax from many previous languages, its parallel features are most directly in?uenced by ZPL, High-Performance Fortran (HPF), and the Cray MTA extensions to C and Fortran. For more information about Chapel, see: http://chapel.cray.com. 4.9 About Cross-compilers The Cray system supports using standalone Linux workstations as code development platforms. When the Cray Application Developer's Environment (CADE) is installed on a suitable Linux system, programs can be written and compiled on the Linux system and then exported to the Cray system for subsequent execution, debugging, and optimization. S–2396–601 43Cray Application Developer's Environment User's Guide You cannot simply install CADE on any Linux system, however. Before you begin, note the following considerations. • The Linux system must meet the minimum hardware requirements listed in the Cray Application Developer's Environment Installation Guide. • The Linux system must be running a version of the SLES operating system corresponding to the Cray system for which you are developing code. If the Cray system is running CLE 2.2, the Linux system must be running SLES 10. If the Cray system is running CLE 3.0 or later, the Linux system must be running SLES 11. If you attempt to cross operating system versions, your code will compile successfully on the Linux system but not run on the Cray system. • The behavior of CADE when installed on Linux distributions other than SLES 10 or SLES 11 is untested and therefore unsupported. • With the exception of the GNU compilers, all compilers are licensed and installed separately. See your site administrator or compiler vendor for further information about license management. • Along with CADE, you must install the Cray Application Developer's Environment Supplement (CADES, or sometimes CADEsup), which requires satisfying a number of dependencies that are contingent on the compilers you are installing. These dependencies are described in the Cray Application Developer's Environment Installation Guide. • You must install and con?gure modules on the Linux system. After CADE and your compilers are installed and con?gured on your Linux system, follow these steps to begin developing code. 1. Load the xtpe-network module corresponding to the Cray system for which you intend to develop code. If you are developing for a Gemini system (Cray XE5, Cray XE6, or Cray XK6), load the xtpe-network-gemini module. If you are developing for a SeaStar system (all others), load the xtpe-network-seastar module. Note: On the actual Cray system, the network type is typically set by default and transparent to the user. When working on a standalone Linux system, you must set this manually. If the correct xtpe-network module is not selected, your code may appear to compile successfully on the Linux system but will not run on the Cray system. 44 S–2396–601Using Compilers [4] 2. Load the xtpe-processor module corresponding to the compute nodes on the Cray system for which you intend to develop code. Your choices are: • xtpe-barcelona (quad core, Cray XT4) • xtpe-shanghai (quad core, Cray XT5 or Cray XE5) • xtpe-istanbul (six cores) • xtpe-mc8 (eight cores) • xtpe-mc12 (twelve cores) • xtpe-interlagos (sixteen cores) Note: On the actual Cray system, the processor type is typically set by default and transparent to the user. When working on a standalone Linux system, you must set this manually. If you select a processor type with fewer cores than are actually present on the Cray compute nodes, your code will not make full use of the Cray system resources and may either run slowly or cause con?icts with aprun options and placement. If you select a processor type with more cores than are actually present on the Cray compute nodes, your application may appear to compile successfully but will not run. 3. (Cray XK system users only) Load the xtpe-accelerator module that is appropriate for the compute nodes on the Cray system for which you intend to develop code. Currently, the only supported option is xtpe-accel-nvidia20. Note: The accelerator target is not set by default on either the standalone Linux system or the Cray system. You must set this manually. This module sets compiler options required to compile applications for the accelerator target and loads CUDA. 4. Load the PrgEnv-vendor module containing your compiler of choice. You are now ready to begin writing, editing, and compiling code. Remember, however, that you must move your code to mount point on the Cray system (for example, /lus/nid00008) before you can run, debug, or optimize it. S–2396–601 45Cray Application Developer's Environment User's Guide 46 S–2396–601Libraries [5] Cray provides a large variety of libraries to support application development and interprocess communications on Cray systems. New libraries are being ported to the Cray system on an ongoing basis. The following libraries can be used with all compilers currently supported on the Cray system, except where noted in Chapter 4, Using Compilers on page 37. 5.1 Basic Scienti?c Libraries (LibSci) Cray LibSci is a collection of numerical routines optimized for best performance on Cray systems. All programming environment modules load xt-libsci by default, except where noted. When possible, you should use calls to the Cray LibSci routines in your code in place of calls to public-domain or user-written versions. 5.1.1 New in LibSci Release 11.0 Beginning with Scienti?c Libraries Release 11.0.0, the way in which dynamic libraries are found at execution time is changed. In previous releases, RPATH was included in the dynamic executable, with a path to the libraries to be used at execution time. Beginning with release 11.0.0, RPATH is no longer used by default. Instead, at execution time, dynamic libraries are found in /opt/cray/lib64. The version de?ned by the xt-libsci module is not used. This change is not expected to have noticeable consequences for most users, and is intended to improve overall system performance when using dynamic linking at high PE counts. However, in the event that this change does appear to have undesirable consequences, users can restore the previous dynamic linking behavior either by setting the environment variable CRAY_ADD_RPATH to yes, or else by swapping to an xt-libsci module earlier than release 11.0.0. Statically linked applications are not affected by this change. S–2396–601 47Cray Application Developer's Environment User's Guide 5.1.2 Basic LibSci Components Table 8. LibSci Basics Module: xt-libsci Man pages: intro_libsci(3s), intro_blas1(3s), intro_blas2(3s), intro_blas3(3s), intro_blacs(3s), intro_lapack(3s), intro_scalapack(3s), intro_irt(3), intro_case(3s)intro_crafft(3s), intro_fft(3s), intro_fftw2(3), intro_fftw3(3) Note: Library-speci?c man pages are available only when the associated module is loaded. The Cray LibSci collection contains the following Scienti?c Libraries. • BLAS (Basic Linear Algebra Subroutines) • BLACS (Basic Linear Algebra Communication Subprograms) • LAPACK (Linear Algebra Routines) • ScaLAPACK (Scalable LAPACK) • FFT (Fast Fourier Transform Routines) • FFTW2 (the Fastest Fourier Transforms in the West, release 2) • FFTW3 (the Fastest Fourier Transforms in the West, release 3) In addition, the Cray LibSci collection contains three libraries developed by Cray. • IRT (Iterative Re?nement Toolkit) • CASE (Cray Adaptive Sparse Eigensolvers) • CRAFFT (Cray Adaptive Fast Fourier Transform Routines) 5.1.3 BLAS and LAPACK The BLAS (Basic Linear Algebra Subroutines) library contains three levels of optimized subroutines. Level 1 BLAS perform the following types of basic vector-vector operations: • Dot products and various vector norms • Scaling, copying, swapping, and computing linear combination of vector • Generate or apply plane or modi?ed plane rotations 48 S–2396–601Libraries [5] For more information, see the intro_blas1(3s) man page. Level 2 BLAS perform matrix-vector operations, and generally produce improved code performance when inlined. For more information about Level 2 BLAS, see the intro_blas2(3s) man page. Level 3 BLAS perform matrix-matrix operations. For more information about Level 3 BLAS, see the intro_blas3(3s) man page. LAPACK is a public domain library of subroutines for solving dense linear algebra problems, including the following: • Systems of linear equations • Linear least squares problems • Eigenvalue problems • Singular value decomposition (SVD) problems LAPACK is the successor to the older LINPACK and EISPACK packages. It extends the functionality of these packages by including equilibration, iterative re?nement, error bounds, and driver routines for linear systems, routines for computing and reordering the Schur factorization, and condition estimation routines for eigenvalue problems. Performance issues are addressed by implementing the most computationally-intensive algorithms using Level 2 and Level 3 BLAS. For more information about LAPACK, see the intro_lapack(3s) man page. 5.1.3.1 Notes • The BLAS and LAPACK libraries include routines from the 64-bit libGoto library from the University of Texas. • BLAS library behavior is dependent on the xtpe-processor module. At most sites this module is typically loaded by default and transparent to the user. However, if your site has multiple types of compute nodes, or if you are working in a Linux cross-compiling environment, it may be necessary to load the xtpe-processor module corresponding to the compute nodes on the Cray system for which you intend to develop code in order to obtain best performance. Your choices are: – xtpe-barcelona (quad core, Cray XT4) – xtpe-shanghai (quad core, Cray XT5) – xtpe-istanbul (six cores) – xtpe-mc8 (eight cores) S–2396–601 49Cray Application Developer's Environment User's Guide – xtpe-mc12 (twelve cores) – xtpe-interlagos (sixteen cores) Note: If you select a processor type with fewer cores than are actually present on the Cray compute nodes, your code will not make full use of the Cray system resources and may either run slowly or cause con?icts with aprun options and placement. If you select a processor type with more cores than are actually present on the Cray compute nodes, your application may appear to compile successfully but will not run. • If you require a C interface to BLAS and LAPACK but want to use Cray LibSci BLAS or LAPACK routines, use the Fortran interfaces. • To obtain threading behavior, set OMP_NUM_THREADS, as described in BLACS and ScaLAPACK on page 50. • You can access the Fortran interfaces from a C program by adding an underscore to the respective routine names and passing arguments by reference (rather than by value). For example, you can call the dgetrf() function as follows: dgetrf_(&uplo, &m, &n, a, &lda, ipiv, work, &lwork, &info); • C programmers using the Fortran interface must order arrays in Fortran column-major order. 5.1.4 BLACS and ScaLAPACK The BLACS (Basic Linear Algebra Communication Subprograms) library is a package of routines that provide the same functionality for message-passing linear algebra communication as the Basic Linear Algebra Subprograms (BLAS) provide for linear algebra computation. With these two packages, software for dense linear algebra can use calls to BLAS for computation and calls to BLACS for communication. The BLACS consist of communication primitives routines, global reduction routines, and support routines. For more information about BLACS, see the intro_blacs(3s) man page. The ScaLAPACK (Scalable LAPACK) library uses BLACS primitives to provide optimized routines for solving real or complex general, triangular, or positive de?nite distributed systems; for reducing distributed matrices to condensed form and an eigenvalue problem solver for real symmetric distributed matrices; and to perform basic operations involving distributed matrices and vectors. LU and Cholesky routines in ScaLAPACK have been modi?ed to allow the user to choose an underlying broadcast algorithm during run-time. It can be done either via an environment variable, or by calling a helper routine in a program. For more information about ScaLAPACK, see the intro_scalapack(3s) man page. 50 S–2396–601Libraries [5] 5.1.4.1 Notes • Some ScaLAPACK routines require the Basic Linear Algebra Communication Subprograms (BLACS) to be initialized. This can be done through a call to BLACS_GRIDINIT. Also, each distributed array that is passed as an argument to a ScaLAPACK routine requires a descriptor, which is set through a call to DESCINIT. • The ScaLAPACK and BLACS libraries can be used in MPI and SHMEM applications. Cray LibSci also supports hybrid MPI/ScaLAPACK applications, which use threaded BLAS on a compute node and MPI between nodes. To use ScaLAPACK in a hybrid application: 1. Adjust the process grid dimensions in ScaLAPACK to account for the decrease in BLACS nodes. 2. Ensure that the number of BLACS processes required is equal to the number of nodes required, not the number of cores. 3. Set the OMP_NUM_THREADS environment variable. 5.1.5 Iterative Re?nement Toolkit (IRT) The Iterative Re?nement Toolkit (IRT) is a library of Fortran subroutines that provides solutions to linear systems using 32-bit factorizations while preserving accuracy through mixed-precision iterative re?nement. IRT exploits the fact that single-precision solvers can be up to twice as fast as double-precision solvers, and uses an iterative re?nement process to obtain solutions accurate to double-precision. IRT includes both serial and parallel implementations of the LU and Cholesky algorithms, and serial versions of the QR algorithm for real and complex matrices. IRT includes the following features: • Sophisticated stopping criteria • Potential minimization of forward error • Ability to return error bounds • Return an estimate of the condition number of matrix A • Return to the double-precision factorization-and-solve process if IRT cannot obtain a solution S–2396–601 51Cray Application Developer's Environment User's Guide IRT provides two interfaces: • Benchmarking interface. The benchmarking interface routines replace the high-level drivers of LAPACK and ScaLAPACK. The names of the benchmark API routines are identical to their LAPACK or ScaLAPACK counterparts or replace calls to successive factorization and solver routines. This allows you to use the IRT process without modifying your application. For example, the IRT dgesv() routine replaces either the LAPACK dgesv() routine or the LAPACK dgetrf() and dgetrs() routines. To use the benchmarking interface, set the IRT_USE_SOLVERS environment variable to 1. Note: Use this interface with caution; calls to the LAPACK LU, QR or Cholesky routines are intercepted and the IRT is used instead. • Expert interface. The expert interface routines give you greater control of the iterative re?nement process and provide details about the success or failure of the process. The format of advanced API calls is: call irt_factorization-method_data-type_processing-mode(arguments) such as: call irt_po_real_parallel(arguments). For more information about IRT, see the intro_irt(3) man page. 5.1.6 Cray Adaptive Simple Eigensolvers (CASE) CASE is a collection of simpli?ed interfaces into high-performance LAPACK- and ScaLAPACK-style routines that calculate the eigenvalues and eigenvectors of a symmetric or hermitian matrix. CASE is written in Fortran but has interfaces for C/C++ users. CASE is designed for C/C++ or Fortran users who are looking for a fast and simple way to calculate the eigenvalues and/or eigenvectors of a matrix and would otherwise call ScaLAPACK or LAPACK eigensolver routines. CASE does this by providing an easy interface for calling the ScaLAPACK and LAPACK eigensolvers that requires only a few simple arguments. From these arguments, CASE interfaces construct the sometimes complicated calling sequences and work arrays required when calling LAPACK and ScaLAPACK eigensolvers directly. CASE is provided for serial or parallel problems, with single- or double-precision real or complex data types. For more information about CASE, see the intro_case(3s) man page. 5.1.7 Fourier Transformations Fast Fourier transforms are handled either by using CRAFFT and FFTW or by using ACML. The intro_fft(3s) man page is a disambiguation page. 52 S–2396–601Libraries [5] 5.1.7.1 CRAFFT CRAFFT is a library of Fortran subroutines that compute the discrete Fourier transform in one, two, or three dimensions; of arbitrary input size; and of both real and complex data. CRAFFT provides a simpli?ed interface to several FFT libraries and allows automatic, dynamic selection of the fastest FFT kernel. CRAFFT also provides routines for two- and three-dimensional distributed FFT. Note: CRAFFT requires that module fftw/3.2.0 or higher be loaded. To use CRAFFT, add a Fortran use crafft statement to any source ?le that calls a CRAFFT routine, and then call crafft_init() before any other CRAFFT routine. This sets up the CRAFFT library for run-time use. You have a choice of how much planning of FFT kernels you wish to perform. You can set the CRAFFT_PLANNER environment variable to 0, 1 or 2 before execution. Alternately, you can use the crafft_set_planner() and crafft_get_planner() subroutines to alter and query, respectively, the value of CRAFFT_PLANNER during program execution. For more information about using CRAFFT, see the intro_crafft(3s) man page. 5.1.7.2 FFTW The Programming Environment includes versions 2.1.5 and 3.2.x of the Fastest Fourier Transform in the West (FFTW) library. FFTW is a C subroutine library with Fortran interfaces for computing the discrete Fourier transform in one or more dimensions, of arbitrary input size, and of both real and complex data (as well as of even/odd data, such as the discrete cosine/sine transforms). The Fast Fourier Transform algorithm is applied for many problem sizes. Up through LibSci version 10.3.2, loading the LibsSci module automatically loaded the fftw/3.1.1 module. Beginning with xt-libsci modules 10.3.3, FFTW is no longer loaded automatically when you load the LibSci module. Distributed-memory parallel FFTs are available only in FFTW 2.1.5.1. The FFTW 3.2.x and FFTW 2.1.5.1 modules cannot be loaded at the same time. You must ?rst unload the other module, if already loaded, before loading the desired one. For example, if you have loaded the FFTW 3.2.x library and want to use FFTW 2.1.5.1 instead, use: % module swap fftw/3.2.2.1 fftw/2.1.5.1 For more information about FFTW, see the intro_fftw2(3) and intro_fftw3(3) man pages. S–2396–601 53Cray Application Developer's Environment User's Guide 5.1.7.3 ACML The AMD Core Math Library (ACML) module is no longer loaded as part of the default PrgEnv environment. BLAS and LAPACK functionality is now provided by Cray LibSci. However, if you need ACML for FFT functions, math functions, or random number generators, you can load the library using the acml module: % module load acml ACML includes: • A suite of Fast Fourier Transform (FFT) routines for real and complex data • Fast scalar, vector, and array math transcendental library routines optimized for high performance • A comprehensive random number generator suite: – Base generators plus a user-de?ned generator – Distribution generators – Multiple-stream support ACML's internal timing facility uses the clock() function. If you run an application on compute nodes that uses the plan feature of FFTs, underlying timings will be done using the native version of clock(). On CNL, clock() returns the sum of user and system CPU times. 5.2 Fast Math Intrinsics Table 9. Fast_mv Basics Module: libfast Man pages: intro_fast_mv(3), Note: Library-speci?c man pages are available only when the associated module is loaded. Fast_mv is a library of high-performance math intrinsic functions. The functions can be used in PGI, CCE, GNU, and PathScale applications. You can use these functions without changing your programs, or you can call them directly. Note: The use of Fast_mv with Intel compilers is not currently supported. To use Fast_mv functions, load the libfast module: % module load libfast 54 S–2396–601Libraries [5] Currently, the library contains the following functions. Table 10. Fast_mv Math Intrinsics Function Name Description cos() 64-bit cosine function exp() 64-bit exponential function (the constant e raised to a power) expf() 32-bit exponential function (the constant e raised to a power) fastcos() scalar 64-bit cosine fastsin() scalar 64-bit sine fastsincos() scalar 64-bit sine and cosine frda_exp() 64-bit exponential function for all elements in an array frda_log() array version of the 64-bit base e logarithm frsa_expf() 32-bit exponential function for all elements in an array frsa_logf() array version of the 32-bit base e logarithm log() 64-bit logarithm (base e) function logf() 32-bit logarithm (base e) function sin() 64-bit sine function sincos() 64-bit sine and cosine function vrda_log() array version of the 64-bit base e logarithm __vrd4_log() four-element version of the 64-bit base e logarithm vrsa_logf() array version of the 32-bit base e logarithm For more information, see the intro_fast_mv(3) man page. S–2396–601 55Cray Application Developer's Environment User's Guide 5.3 PETSc Table 11. PETSc Basics Modules: petsc, petsc-complex, tpsl Man pages: intro_petsc(3s) Note: Library-speci?c man pages are available only when the associated module is loaded. Website: http://www.mcs.anl.gov/petsc/petsc-as/ PETSc (Portable, Extensible, Toolkit for Scienti?c Computation) is an open source library of parallel linear and nonlinear equation solvers intended for use in large-scale C, C++, or Fortran applications. PETSc uses standard MPI functions for all message-passing communication. The PETSc module is dependent on the xt-libsci and xt-asyncpe modules. Make certain these modules are loaded before using PETSc. When you load the petsc module, the Third-party Scienti?c Libraries (tpsl) module is automatically loaded as well, to provide access to the libraries required to support PETSc. Note: Always use the tpsl module that is linked to the PETSc module. PETSc and Trilinos are asynchronous products and may at times use different versions of the TPSL libraries. PETSc provides many of the mechanisms needed for parallel applications, such as simple parallel matrix and vector assembly routines that allow the overlap of communication and computation. In addition, PETSc includes support for parallel distributed arrays useful for ?nite difference methods, such as: • Parallel vectors, including code for communicating ghost points • Parallel matrices, including several sparse storage formats • Scalable parallel preconditioners • Krylov subspace methods • Parallel Newton-based nonlinear solvers • Parallel time-stepping ordinary differential equation (ODE) solvers 56 S–2396–601Libraries [5] The following packages are included in PETSc/TPSL. • MUMPS 4.9.2. MUMPS (MUltifrontal Massively Parallel sparse direct Solver) is a package of parallel, sparse, direct linear-system solvers based on a multifrontal algorithm. For further information, see http://graal.ens-lyon.fr/MUMPS/. • SuperLU 4.1. SuperLU is a sequential version of SuperLU_dist (not included with petsc-complex), and a sequential incomplete LU preconditioner that can accelerate the convergence of Krylov subspace interative solvers. For further information, see http://crd.lbl.gov/~xiaoye/SuperLU/. • SuperLU_dist 2.5. SuperLU_dist is a package of parallel, sparse, direct linear-system solvers (available in Cray LibSci). For further information, see http://crd.lbl.gov/~xiaoye/SuperLU/. • ParMETIS 3.2. ParMETIS (Parallel Graph Partitioning and Fill-reducing Matrix Ordering) is a library of routines that partition unstructured graphs and meshes and compute ?ll-reducing orderings of sparse matrices. For further information, see http://glaros.dtc.umn.edu/gkhome/views/metis/. • HYPRE 2.7.0. HYPRE is a library of high-performance preconditioners that use parallel multigrid methods for both structured and unstructured grid problems (not included with petsc-complex). For further information, see http://www.llnl.gov/CASC/linear_solvers/. • SUNDIALS 2.4.0 (SUite of Nonlinear and DIfferential/ALgebraic equation Solvers) consists of 5 solvers: CVODE, CVODES, IDA, IDAS, and KINSOL. In addition, SUNDIALS provides a MATLAB interface to CVODES, IDAS, and KINSOL that is called sundialsTB. For further information, see https://computation.llnl.gov/casc/sundials/main.html. • Scotch 5.1.1. Scotch is a software package and libraries for sequential and parallel graph partitioning, static mapping, sparse matrix block ordering, and sequential mesh and hypergraph partitioning. For further information, see http://www.labri.fr/perso/pelegrin/scotch/. Note: Although you can access these packages individually, Cray supports their use only through the PETSc interface. 5.3.1 Notes • If you use PETSc with the Cray Fortran compiler, either add the directive !dir$ PREPROCESS EXPAND_MACROS to the source code or add the -F option to the ftn command line. • The PathScale Compiler Suite does not support PETSc 3.1.0.8. If you are using the PathScale compilers, use PETSc 3.0. S–2396–601 57Cray Application Developer's Environment User's Guide • The solvers in Cray PETSc 3.1 are heavily optimized using the Cray Adaptive Sparse Kernels (CASK) library. CASK is an auto-tuned library within the Cray PETSc package that is transparent to the application developer, but improves the performance of most PETSc iterative solvers. You can expect the largest performance improvements when using blocked matrices (BAIJ or SBAIJ), but may also see large gains when using standard compressed sparse row (CSR) AIJ PETSc matrices. 5.4 Trilinos Table 12. Trilinos Basics Modules: trilinos, tpsl Man pages: intro_trilinos(1), Note: Library-speci?c man pages are available only when the associated module is loaded. Website: http://trilinos.sandia.gov/ Trilinos is a separate module, comparable to PETSc, that provides abstract, object-oriented interfaces to established libraries such as Metis/ParMetis, SuperLU, Aztec, BLAS, and LAPACK. Trilinos also includes a set of Cray Adaptive Sparse Kernels (CASK) that perform SpMV, and include optimized versions of single- and multiple-vector matrix vector multiplies. The Trilinos module is dependent on the xt-libsci and xt-asyncpe modules. Make certain these modules are loaded before using Trilinos. When you load the trilinos module, the Third-party Scienti?c Libraries (tpsl) module is automatically loaded as well, to provide access to the libraries required to support Trilinos. Note: Always use the tpsl module that is linked to the Trilinos module. Trilinos and PETSc are asynchronous products and may at times use different versions of the TPSL libraries. To use the Trilinos packages, load your compiling environment of choice, and then load the Trilinos module. % module load trilinos After you load the Trilinos module, all header and library locations are set automatically and you are ready to compile your code. No Trilinos-speci?c linking information is required on the command line. 58 S–2396–601Libraries [5] If linking to more than one Trilinos package, the libraries are linked automatically in the correct order of package dependency. For more information about link order, see http://trilinos.sandia.gov/packages/interoperability.html. 5.5 MPT Table 13. MPT Basics Modules: xt-mpich2, xt-shmem, xt-mpt (deprecated) Man pages: intro_mpi(3), intro_shmem(3) Note: Library-speci?c man pages are available only when the associated module is loaded. Documentation: Getting Started on MPI I/O (S–2490) The Cray Message Passing Toolkit (MPT) consists of two components. • MPI • SHMEM Support for MPI and SHMEM varies signi?cantly depending on whether you are using a Cray XE or Cray XK system with Gemini interconnect or a Cray XT system with SeaStar interconnect, and whether your code uses static or dynamic libraries. Because of these issues, Cray provides a variety of different MPI and SHMEM modules. These modules are hardware-dependent, so your system administrator should install only the modules that support your hardware on your system. Nonetheless, there are very signi?cant differences in the ways that MPI and SHMEM are implemented on Cray XE/Cray XK and Cray XT systems. These differences are re?ected in the functions that are available, and most visibly in the environment variables available on the two types of systems. To see which functions and environment variables are supported on your system, always check the intro_mpi(3) or intro_shmem(3) man pages. 5.5.1 Using MPI and SHMEM Modules Cray has changed the way that MPI and SHMEM are supported and the default module con?gurations. These changes were implemented in different releases of CLE and affect your usage of the MPT-related modules. S–2396–601 59Cray Application Developer's Environment User's Guide 5.5.1.1 CLE 4.0 Systems No MPT-related modules are loaded by default. • If your code uses MPI code only, load the xt-mpich2 module before compiling or linking. This ensures that your code is linked using the -lmpich option. • If your code uses SHMEM code only, load the xt-shmem module before compiling or linking. This ensures that your code is linked using the -lsma option. • If your code uses both MPI and SHMEM, load both the xt-mpich2 and xt-shmem modules. Your code will be linked using both the -lmpich and -lsma options. • There is no need to differentiate between code that uses static libraries and code that uses dynamic or shared libraries when selecting and using MPI or SHMEM modules. The xt-mpt module is provided to support legacy code and make ?les but is deprecated and planned to be removed in a future release. 5.5.1.2 CLE 3.1 Systems By default, all Cray programming environment modules (PrgEnv-compiler) load the xt-mpich2 module. This module supports MPI code only. • If your code uses MPI code only, verify that the xt-mpich2 module is loaded before compiling or linking. This ensures that your code is linked using the -lmpich option. • If your code uses SHMEM code only, unload the xt-mpich2 module and load the xt-shmem module before compiling or linking. This ensures that your code is linked using the -lsma option. • If your code uses both MPI and SHMEM, verify that both the xt-mpich2 and xt-shmem modules are loaded. Your code will be linked using both the -lmpich and -lsma options. • There is no need to differentiate between code that uses static libraries and code that uses dynamic or shared libraries when selecting and using MPI or SHMEM modules. The xt-mpt module is provided to support legacy code and make ?les but is deprecated and planned to be removed in a future release. 60 S–2396–601Libraries [5] 5.5.1.3 CLE 3.0 Systems By default, all Cray programming environment modules load the xt-mpt module. The xt-mpt module supports both MPI and SHMEM code, but causes the compiler drivers to link code using both the -lmpich and -lsma options. This can cause undesirable run time dependencies when using dynamic or shared libraries. • If your code uses static libraries only, use the xt-mpt module. There is no need to differentiate between MPI and SHMEM code. • If your code uses dynamic libraries and MPI code only, unload the xt-mpt module and load the xt-mpich2 module before compiling or linking. This ensures that your code is linked using the -lmpich option. • If your code uses dynamic libraries and SHMEM code only, unload the xt-mpt module and load the xt-shmem module before compiling or linking. This ensures that your code is linked using the -lsma option. • If your code uses dynamic libraries and both MPI and SHMEM, unload the xt-mpt module and load both the xt-mpich2 and xt-shmem modules. Your code will be linked using both the -lmpich and -lsma options. 5.5.1.4 CLE 2.2 Systems By default, all Cray programming environment modules load the xt-mpt module. Depending on which version of MPT is installed on your system, the xt-mpich2 and xt-shmem modules may not be available. Be advised that if the xt-mpt module is loaded, the compiler drivers will link code using both the -lmpich and -lsma options. This can cause undesirable run time dependencies when using dynamic or shared libraries. 5.6 Hugepages Hugepages are virtual memory pages which are bigger than the default base page size of 4KB. Hugepages can improve memory performance for common access patterns on large data sets. Access to hugepages is provided through a virtual ?le system called hugetlbfs. Every ?le on this ?le system is backed by huge pages and is directly accessed with mmap() or read(). The libhugetlbfs library allows an application to use huge pages more easily than it could by directly accessing the hugetlbfs ?le system. A user may use libhugetlbfs to back application text and data segments. Due to differing memory management mechanisms on Cray XT and Cray XE systems, the implementation of the libhugetlbfs library differs on these two architectures. S–2396–601 61Cray Application Developer's Environment User's Guide 5.6.1 Cray XT Usage Nodes do not have huge pages allocated by default. To use hugepages, link an application with the libhugetlbfs library. At run-time, de?ne the environment variable HUGETLB_MORECORE=yes. The application launcher, aprun, must be told that a given application wants to use huge pages. Specify a per-PE huge page memory requirement on the aprun invocation line using the [-m size[h|hs] ] option. For more information see the aprun(1) man page, and Workload Management and Application Placement for the Cray Linux Environment (S–2496). 5.6.2 Cray XE Usage By default, the running system is con?gured to have huge pages available. Pre-allocated huge pages are reserved inside the kernel and cannot be used for other purposes. On Cray XE systems, PGAS, SHMEM and MPI applications are likely to require the usage of huge pages for static data and/or the heap. Modules craype-hugepages2m and craype-hugepages8m (released in xt_asyncpe 4.8) set the necessary link options and environment variables (e.g., HUGETLB_DEFAULT_PAGE_SIZE, HUGETLB_MORECORE, HUGETLB_ELFMAP) to facilitate the usage of 2 or 8 MB huge pages, respectively. It is not required to use the -m option on the aprun command on the Cray XE system to allocate huge pages, because the kernel allows the dynamic creation of huge pages. However, it is advisable to specify this option and preallocate an appropriate number of huge pages, when memory requirements are known, to reduce operating system overhead. For more detailed information and examples see the intro_hugepages(1), and aprun(1) man pages, and Workload Management and Application Placement for the Cray Linux Environment (S–2496). 62 S–2396–601Debugging Code [6] The Cray system supports a variety of debugging options ranging from simple command-line debuggers to separately licensed third-party GUI tools and capable of performing a variety of tasks ranging from analyzing core ?les to setting breakpoints and debugging running parallel programs. As a rule, your code must be compiled using the -g command line option before you can use any of the debuggers to produce meaningful information. However, if you are using both a compiler and a debugger that support Fast-Track Debugging, the -g option is replaced by using the -G fast option. Note: The PGI debugger, PGDBG, is not supported on Cray systems. 6.1 Cray Debugger Support Tools Cray provides a collection of basic debugging packages that are referred to collectively as the Cray Debugger Support Tools and installed as a single rpm, but loaded and used as individual modules. These packages are: • lgdb: a modi?ed version of gdb that interfaces with aprun and works on Cray system compute nodes • atp: a system that monitors user applications and replaces the core dump with a more comprehensive stack backtrace and analysis • MRNet: a software overlay network that provides multicast and reduction communications for parallel and distributed tools and systems • STAT: stack trace analysis tool S–2396–601 63Cray Application Developer's Environment User's Guide 6.1.1 Using lgdb Table 14. lgdb Basics Module: xt-lgdb Command: lgdb Man page: lgdb(1) Note: Tool-speci?c man pages are available only when the associated module is loaded. Online help: lgdb --help The lgdb command is used to launch an application and related gdb server processes on remote nodes for debugging purposes. After lgdb is launched, it provides directions regarding how to run the gdb debugger and attach to application processes. When using lgdb on a Cray system, remember that you use lgdb to launch an instance of aprun, which in turn launches the application to be debugged. You cannot use lgdb to launch an application directly. Example 1. Launching and debugging a single-rank application $ lgdb --pes=0 --command="aprun -n1 ./myapp" --lib=/lib64/libthread_db.so.1 You must specify the number of PEs (ranks) to which you want to attach gdbservers. Since this is a single-rank application, specify 0 for the rank. The --lib=path_to_library is optional and needs to be speci?ed only for certain systems, such as X86_64 Linux, where the libthread_db library is required. Example 2. Attaching to an already-running application $ lgdb --pes=0 --pid=aprun_pid --lib=/lib64/libthread_db.so.1 Remember, you use lgdb to launch an instance of aprun, which in turn launches the application to be debugged. Therefore to attach to an already-running application, you must specify the pid of the instance of aprun that launched the application, not the application itself. For more information, see the lgdb(1) man page. 64 S–2396–601Debugging Code [6] 6.1.2 Using Abnormal Termination Processing (ATP) Table 15. atp Basics Module: atp Commands: ataprun, atpFrontend (deprecated) Man page: intro_atp(1) Note: Tool-speci?c man pages are available only when the associated module is loaded. Online help: None required Abnormal Termination Processing (ATP) monitors user applications. When the atp module is loaded and ATP is enabled, ATP is launched when a job is started and delivers a heuristically determined set of core ?les in the event of an application crash. If an application takes a system trap, ATP performs analysis on the dying application. All stack backtraces of the application processes are gathered into a merged stack backtrace tree and written to disk as the ?le, atpMergedBT.dot. The stack backtrace tree for the ?rst process to die is sent to stderr as is the number of the signal that caused the application to fail. The atpMergedBT.dot ?le can be viewed with statview, (the Stack Trace Analysis Tool viewer). The merged stack backtrace tree provides a concise yet comprehensive view of what the application was doing at the time of its termination. Note: The statview command is available only when the stat module is loaded. ATP is designed to analyze failing applications. It does not play any role with commands. That is, an application must use a supported parallel programming model, such as MPI, SHMEM, OpenMP, CAF, or UPC, in order to bene?t from ATP analysis. When the atp module is loaded, ATP sets the MPICH_ABORT_ON_ERROR, SHMEM_ABORT_ON_ERROR, and DMAPP_ABORT_ON_ERROR environment variables. This enables MPI, SHMEM, and DMAPP applications to raise a signal when they discover usage errors—rather than only printing to stderr and exiting—which therefore enables ATP to notice the problem and perform its analysis. Note: Using ATP disables core dumping. 6.1.2.1 On CLE 3.0 or Later On CLE 3.0 and later systems, when the atp module is loaded, ATP is launched automatically whenever a job is launched by using the aprun command. Do not use the atpaprun or atpFrontend commands on CLE 3.0 or later systems. These commands are for use on CLE 2.2 or earlier systems only. S–2396–601 65Cray Application Developer's Environment User's Guide 6.1.2.2 On CLE 2.2 or Earlier On CLE 2.2 and earlier systems, ATP must be invoked manually. This is done by loading the atp module and then using either the atpaprun command to launch your application, or after the application has been launched by using the atpFrontend command to attach to an already-running application. Example 3. Launching an application and invoking ATP atpaprun aprun_parameters Example 4. Attaching ATP to an already-running application atpFrontend application_PID For more information, see the intro_atp(1) man page. 6.1.3 Using MRNet Table 16. MRNet Basics Module: mrnet Online help: http://www.paradyn.org/mrnet/ MRNet is a customizable, high-throughput communication software system for parallel tools and applications with a master/slave architecture. MRNet reduces the cost of these tools' activities by incorporating a tree-based overlay network (TBON) of processes between the tool's front-end and back-ends. MRNet uses the TBON to distribute many important tool communication and computation activities, reducing analysis time and keeping tool front-end loads manageable. MRNet-based tools send data between front-end and back-ends on logical ?ows of data called streams. MRNet internal processes use ?lters to synchronize and aggregate data sent to the tool's front-end. Using ?lters to manipulate data in parallel as it passes through the network, MRNet can ef?ciently compute averages, sums, and other more complex aggregations on back-end data. 6.1.4 Using STAT Table 17. STAT Basics Module: stat Commands: statview Online help: http://www.paradyn.org/STAT/STAT.html 66 S–2396–601Debugging Code [6] STAT (Stack Trace Analysis Tool) gathers and merges the stack traces from a parallel application's processes and produces 2D spatial and 3D spatial-temporal call graphs that encode the calling behavior of the application processes in the form of a pre?x tree. The 2D graph represents a single snapshot of the entire application, while the 3D form represents a series of snapshots from the application taken over time. 6.2 Using Cray Fast-Track Debugging Normally, code must be compiled using the -g option before it can be debugged using a conventional debugger. The -g option typically disables all compiler optimizations, producing an executable that contains full DWARF information and can be breakpointed, stepped-through, or paused and restarted anywhere, but at the cost of a much larger and far slower-running program. Cray Fast-Track Debugging signi?cantly increases the speed of the debugging process by producing executables that both contain full DWARF information and run at optimized-code speed. Essentially this is done by producing two parallel executables: one that is fully optimized, and another that is not. While this combined executable is considerably larger than a normal executable, when it is executed under the control of a debugger that supports fast-track debugging, it runs at optimized-code speed until it hits a break point—at which time it switches to the unoptimized code, and allows you to set breakpoints, examine registers, pause, resume, and step through the code as if the entire program was compiled using the -g option. As far as the debugger is concerned, there are no user interface changes, aside from the possibility that you might pursue a backtrace far enough back to begin seeing internal names instead of user names for variables. (For example, instead of foo, you might see debug$foo.) All the work involved in using fast-track debugging is done on the compiler side. Procedure 1. Using Cray Fast-Track Debugging Using Cray Fast-Track Debugging is a two-step process, requiring both a compiler and a debugger that support fast-track debugging. The steps are: 1. Compile your program using your compiler's fast-track debugging option. For example, if you are using the Cray Fortran compiler, compile and link your code using the -G fast option: users/yourname> ftn -Gfast myapp.f 2. Execute your program using your debugger's normal control method. For example, if you are using the lgdb command line debugger to debug a single-rank application: $ lgdb --pes=0 --command="aprun -n1 ./myapp" S–2396–601 67Cray Application Developer's Environment User's Guide Once the debugging session is launched, fast-track debugging is transparent to the user. Sections of the code that contain breakpoints execute slowly, as if compiled using the -g option. Other sections of the code that do not contain breakpoints execute at normal speed, as if compiled using normal optimizations. 6.2.1 Supported Compilers and Debuggers At this time, Cray Fast-Track Debugging is supported on the front end by the Cray Compiling Environment (CCE) compilers. On the back end, Cray Fast-Track Debugging is supported by the lgdb and Allinea DDT debuggers. 6.3 About Core Files When an application fails, one core ?le is generated for the ?rst failing process. If a ?le named core already exists in the current working directory, it is overwritten. On large MPP systems where an application might be running thousands of processes, a conventional core ?le may not indicate the actual cause of the failure. For this reason Cray systems support Abnormal Termination Processing (ATP). If ATP is enabled, a failing application generates a heuristically determined set of data, and in place of a core ?le, all stack backtraces are gathered into a merged stack backtrace tree and written to disk as the ?le, atpMergedBT.dot. The stack backtrace tree for the ?rst process to die is sent to stderr as is the number of the signal that caused the application to fail. For more information about ATP, see Using Abnormal Termination Processing (ATP) on page 65. 6.4 Using TotalView Table 18. TotalView Basics Module: xt-totalview Commands: totalview, totalviewcli Man page: totalview(1) Note: Tool-speci?c man pages are available only when the associated module is loaded. 68 S–2396–601Debugging Code [6] Online help: The TotalView GUI contains an extensive HTML online help system but requires that $TV_HTMLHELP_VIEWER be de?ned before use. For more information, see the Totalview documentation. Documentation: /opt/totalview/version/doc TotalView is an optional product from TotalView Technologies, LLC, that provides source-level debugging of applications running on multiple compute nodes. TotalView is compatible with the Cray, PGI, GCC, PathScale, and Intel compilers. TotalView can be launched in either or two modes: in GUI mode (using the totalview command), or in command-line mode (using the totalviewcli command). TotalView is typically run interactively. If your site has not designated any compute nodes for interactive processing, use the qsub -I command to reserve the number of compute nodes you want to use in interactive mode. Example 5. Using TotalView to control program execution To debug an application on the Cray system, use TotalView to launch aprun, which in turn launches the application to be debugged. users/yourname> totalview aprun -a [aprun_arguments] ./myapp [myapp_arguments] The -a option is a TotalView option indicating that the arguments that follow apply to aprun, not TotalView. Example 6. Debugging a core ?le To use TotalView to examine a core ?le, use the totalview command to launch the GUI. Then, in the New Program window, click the Open a core ?le button and use the browse functions to ?nd, select, and open the core ?le you want to examine. Example 7. Attaching TotalView to a running process To attach TotalView to a running process, you must be logged in to the same login node that you used to launch the process, and then you must attach to the instance of aprun that was used to launch the process, not the process itself. To do so: 1. Use the totalview command to launch the GUI. 2. In the New Program window, click the Attach to process button. The list of processes currently running displays. 3. Select the instance of aprun that you want, and click OK. TotalView displays a process window showing both aprun and the program threads that were launched by that instance of aprun. S–2396–601 69Cray Application Developer's Environment User's Guide 6.4.1 Known Limitations The TotalView debugging suite for Cray systems differs in functionality from the standard TotalView implementation. It does not support: • Debugging MPI_Spawn(), OpenMP, or Cray SHMEM programs. • Compiled EVAL points and expressions. • Type transformations for the PGI C++ compiler standard template library collection classes. • Exception handling for the PGI C++ compiler run time library. • Spawning a process onto the compute processors. • Machine partitioning schemes, gang scheduling, or batch systems. 6.5 Using DDT Table 19. DDT Basics Module: ddt Commands: ddt Man page: ddt(1) Note: Tool-speci?c man pages are available only when the associated module is loaded. Online help: The DDT GUI includes an extensive online help system accessible by selecting Help from the menu. If you have problems displaying the help, see the DDT User Guide for information about con?guring X-Windows forwarding and VNC connections. Documentation: /opt/cray/ddt/version/doc DDT is an optional product from Allinea Software, and is a scalable debugger with a graphical user interface. It can be used to debug Fortran, C, and C++ programs, including MPI and OpenMP code, and to launch and debug programs, attach to already running programs, or open and debug core ?les. DDT is compatible with the Cray, PGI, GCC, PathScale, and Intel compilers. The DDT GUI requires either X-Windows forwarding or VNC in order to work. DDT can be used either within an interactive shell or via the batch system. Submission through a batch system requires the use of template ?les that specify the batch parameters. Sample template ?les are found in /opt/cray/ddt/version/templates. 70 S–2396–601Debugging Code [6] In an interactive shell, the fastest way to launch DDT is by entering the ddt command: users/yourname> ddt Assuming X-Windows forwarding is con?gured correctly, the main program window displays, along with a pop-up menu offering the following options. • Run and Debug a Program • Debug a Multi-Process Non-MPI Program • Attach to a Running Program • Open a Core File • Restore a Checkpoint • Cancel Select the option you want to use, or Cancel to close the pop-up menu and proceed to the DDT main window. All the above options are also available through the Sessions menu Run option. 6.5.1 Known Limitations DDT does not support UPC. DDT has a number of defaults that affect batch queue submission behavior. When you begin a DDT session, either by selecting Run and Debug a Program from the pop-up Welcome menu or by selecting Run from the Session menu in the DDT main window, the Queue Submission Mode window displays. If you use a batch queuing system such as PBS Pro, always verify the Queue Submission Parameters before proceeding. In particular, verify that the default Queue name and Procs Per Node match your system's con?guration. The default Queue name is generally site-speci?c, while the Procs Per Node value must match your system's processor types: dual-core, quad-core, and so on. If you need to change the Queue Submission Parameters, click the Change button on the Queue Submission Mode window to do so for the duration of the current session. Alternatively, you can create a template ?le that stores your preferred parameters. Instructions for creating and using template ?les are provided in the DDT User Guide. To change your system's default Queue Submission Parameters for all users, contact your site administrator. Default con?guration information is stored in the /opt/cray/ddt/version/default-config.ddt ?le. This information includes the name of the default template, which currently is /opt/cray/ddt/version/templates/xt4.qtf. The actual default queue submission parameters are speci?ed in the default template ?le. S–2396–601 71Cray Application Developer's Environment User's Guide 72 S–2396–601Optimizing Code [7] After your code is compiled, debugged, and capable of running to completion or planned termination, you can begin looking for ways in which to improve execution speed. In general, the opportunities for optimization fall into three categories, which require progressively more programmer effort. These categories are: • Improving overall I/O • Improving use of compiler-generated optimizations • Analyzing code behavior and rewriting code to optimize performance 7.1 Improving I/O 7.1.1 Using iobuf Table 20. IOBUF Basics Module: iobuf Man page: iobuf(3) Environment variable: IOBUF_PARAMS IOBUF is an I/O buffering library that can reduce the I/O wait time for programs that read or write large ?les sequentially. IOBUF intercepts standard I/O calls such as read and open and replaces the stdio (glibc, libio) layer of buffering with an additional layer of buffering, thus improving program performance by enabling asynchronous prefetching and caching of ?le data. IOBUF can also gather runtime statistics and print a summary report of I/O activity for each ?le. S–2396–601 73Cray Application Developer's Environment User's Guide In general, no program source changes are needed in order to take advantage of IOBUF. Instead, IOBUF is implemented by following these steps: 1. Load the IOBUF module: % module load iobuf 2. Recompile (or optionally, relink) the program. 3. Set the IOBUF_PARAMS environment variable as needed. % setenv IOBUF_PARAMS='*:verbose' Note: The simplest parameter speci?cation is: % setenv IOBUF_PARAMS='*' This setting matches all ?les and enables buffering with default parameters. 4. Execute the program. If a memory allocation error occurs, buffering is reduced or disabled for that ?le and a diagnostic is printed to stderr. When the ?le is opened, a single buffer is allocated if buffering is enabled. The allocation of additional buffers is done when a buffer is needed. When a ?le is closed, its buffers are freed. The behavior of IOBUF is controlled by the use of environment variables, the most signi?cant of which is IOBUF_PARAMS. If this environment variable is not set, the default state is no buffering and the I/O call is passed on to the next layer without intervention. For more information about valid IOBUF_PARAMS parameters and their usage, see the iobuf(3) man page. 7.1.2 Improving MPI I/O Table 21. MPI I/O Basics Module: xt-mpich2 Man page: intro_mpi(3) Environment variables: MPICH_MPIIO_HINTS, MPICH_RANK_REORDER_METHOD, others Documentation: Getting Started on MPI I/O When working with MPI code, one of the most effective ways to realize signi?cant improvements in program execution speed is by ?ne-tuning MPI rank placement and I/O usage. The Cray Message Passing Toolkit (MPT) provides more than forty environment variables designed to help you do just that, the two most signi?cant of 74 S–2396–601Optimizing Code [7] which are MPICH_MPIIO_HINTS and MPICH_RANK_REORDER_METHOD. For a listing of the MPI environment variables and their valid values and uses, see the intro_mpi(3) man page. A full discussion of MPI I/O optimization is beyond the scope of this document. For more information on this subject, including detailed explanations and examples, see Getting Started on MPI I/O (S–2490). Optimizing MPI rank placement can require considerably more detailed analysis. Alternately, you can use Cray Performance Analysis Tools to instrument your program to study MPI behavior, and then to generate suggested MPI rank reordering information. For more details, see the intro_craypat(1), pat_build(1), and pat_report(1) man pages, and Using Cray Performance Analysis Tools (S–2376). 7.2 Using Compiler Optimizations This section collects some of the more common tips and tricks for getting even better-performing code out of the compilers. This section will be expanded as information is developed. 7.2.1 Cray Compiling Environment (CCE) The Cray Fortran and C/C++ compilers are optimizing compilers that perform substantial analysis during compilation and generate highly optimized code automatically. The Cray compilers also support a large number of command-line arguments that enable you to exert manual control over compiler optimizations, and ?ne-tune the behavior of the compiler. For more detailed information about the Cray Fortran, C, and C++ compiler command-line arguments, see the crayftn(1), craycc(1), and crayCC(1) man pages, respectively. Two of the most useful compiler command-line arguments are the Fortran -rd and C/C++ -h list=m options, which cause the compiler to generate annotated loopmark listings showing which optimizations were performed where in the code. Together with the -h negmsgs options, which generate listings showing which potential optimizations were not performed, and why, these arguments can help you zero-in on areas in your code that are compiling without error, but not with maximum ef?ciency. For more detailed information about generating and reading loopmark listings, see the Cray Fortran Reference Manual (S–3901) and Cray C and C++ Reference Manual (S–2179). The Cray compilers also support a large number of pragmas and directives that enable you to exert manual control over compiler optimization behavior. In many cases, code that is not optimizing well can be corrected without substantial changes to the code itself, but simply by applying the right pragmas or directives. S–2396–601 75Cray Application Developer's Environment User's Guide For more information about Cray compiler pragmas and directives, see the intro_directives(1) man page. 7.3 Using the Cray Performance Analysis Tools Table 22. Performance Analysis Basics Module: perftools Commands: pat_build, pat_report, app2 Man pages: intro_craypat(1), pat_build(1), pat_report(1), pat_help(1), app2(1), intro_papi(3), hwpc(5), nwpc(5) Note: Tool-speci?c man pages are available only when the associated module is loaded. Online help: CrayPat includes an extensive online help system that features many examples and the answers to many frequently asked questions. To access the help system, enter pat_help at the command line. Documentation: Using Cray Performance Analysis Tools Note: The PGI pro?ling tools, pgprof and pgcollect, are not supported on Cray systems. After you have compiled and debugged your program, you are ready to begin analyzing its performance. The Cray Performance Analysis Tools are a suite of optional utilities that enable you to capture and analyze performance data generated during the execution of your program in order to help you to ?nd answers to two fundamental questions: How fast is my program running? and How can I make it run faster? The Cray Performance Analysis Tools suite consists of three components: • CrayPat: the program instrumentation, data capture, and basic text reporting tool • Cray Apprentice2: the graphical analysis and data visualization tool • PAPI: the Performance Application Programming Interface To begin working with the performance analysis tools, ?rst load your programming environment of choice, and then load the perftools module. users/yourname> module load perftools The performance analysis process consists of three basic steps. 76 S–2396–601Optimizing Code [7] 1. Instrument your program, to specify what kind of data you want to collect under what conditions. 2. Execute your instrumented program, to generate and capture the desired data. 3. Analyze the resulting data. Accordingly, CrayPat consists of the following major components: • pat_build, the utility used to instrument programs • The CrayPat run time environment, which collects the speci?ed performance data • pat_report, the ?rst-level analysis tool used to produce text reports or export data for more sophisticated analysis • pat_help, the command-line driven online help system All CrayPat components, including the man pages and help system, are available only when the perftools module is loaded. For successful results, the perftools module must be loaded before you compile the program to be instrumented, instrument the program, execute the instrumented program, or generate a report. If you want to instrument a program that was compiled before the perftools module was loaded, you may under some circumstances ?nd that re-linking is suf?cient, but as a rule it's best to load the perftools module and then recompile. When instrumenting a program, CrayPat requires that the object (.o) ?les created during compilation be present, as well as the library (.a) ?les, if any. However, most compilers automatically delete the .o and .a ?les when working with single source ?les and compiling and linking in a single step, therefore it is good practice to compile and link in separate steps and use the compiler command line option to preserve these ?les. For example, if you are using the Cray Compiling Environment (CCE) Fortran compiler, compile using either of these command line options: > ftn -c source?le.f Alternatively: > ftn -h keepfiles source?le.f Then link the object ?les to create the executable program: > ftn -o executable source?le.o S–2396–601 77Cray Application Developer's Environment User's Guide 7.3.1 Instrumenting the Program After the program is compiled and linked, use the pat_build command to instrument the program for performance analysis. In simplest form, pat_build is used like this: > pat_build -O apa executable This produces a copy of your original program, which is named executable+pat (for example, a.out+pat) and instrumented for the default experiment. Your original executable remains untouched. The pat_build command supports a large number of options and directives, including an API that enables you to instrument speci?ed regions of your code. These options and directives are summarized in the pat_build(1) man page and documented more extensively in Using Cray Performance Analysis Tools. 7.3.2 Collecting Data Instrumented programs are executed just like any other program; either by using the aprun command if your site permits interactive sessions or by using your system's batch commands. CrayPat supports more than ?fty optional run time environment variables that enable you to control instrumented program behavior and data collection during execution. For example, if you use the C shell and want to collect data in detail rather than in aggregate, consider setting the PAT_RT_SUMMARY environment variable to 0 (off) before launching your program. /lus/nid00008> setenv PAT_RT_SUMMARY 0 Doing so records data with timestamps, which makes additional reports available in Cray Apprentice2, but at the cost of potentially much larger data ?le sizes and somewhat increased overhead. The CrayPat run time environment variables are summarized in the intro_craypat(1) man page and documented more extensively in Using Cray Performance Analysis Tools. 7.3.3 Analyzing Data Assuming your instrumented program runs to completion or planned termination, CrayPat outputs one or more data ?les. The exact number, location, and content of the data ?le(s) varies depending on the nature of your program, the type of experiment for which it was instrumented, and the run time environment variable settings in effect at the time of program execution. 78 S–2396–601Optimizing Code [7] All initial data ?les are output in .xf format, with a generated ?le name consisting of your original program name, plus pat, plus the execution process ID number, plus a code string indicating the type of data contained within the ?le. Depending on the program run and the types of data collected, CrayPat output may consist of either a single .xf data ?le or a directory containing multiple .xf data ?les. If the program was instrumented with the -O apa option, a ?le with the suf?x .apa is also generated. This ?le is a customized template for this program and is created for use with future instrumentation experiments. To begin analyzing the captured data, use the pat_report command. In simplest form, it looks like this: /lus/nid00008> pat_report myprog+pat+PID-nodes.xf The pat_report command accepts either a ?le or directory name as input and processes the .xf ?le(s) to generate a text report. In addition, it also exports the .xf data to a single .ap2 ?le, which is both a self-contained archive that can be reopened later using the pat_report command and the exported-data ?le format used by Cray Apprentice2. The pat_report command provides more than thirty prede?ned report templates, as well as a large variety of user-con?gurable options. These reports and options are summarized in the pat_report(1) man page and documented more extensively in Using Cray Performance Analysis Tools. 7.3.4 For More Information In addition to Using Cray Performance Analysis Tools and the intro_craypat(1), pat_build(1), and pat_report(1) man pages, there is a substantial amount of information, including an FAQ and examples, in the CrayPat online help system. The help system is accessible whenever the perftools module is loaded; to access the help system, enter pat_help at the command line. For more information about using the help system, see the pat_help(1) man page. 7.4 Using Cray Apprentice2 Cray Apprentice2 is an optional GUI tool that is used to visualize and manipulate the performance analysis data captured during program execution. Cray Apprentice2 can be run either on the Cray system or, optionally, on a standalone Linux desktop machine. Cray Apprentice2 can display a wide variety of reports and graphs, depending on the type of program being analyzed, the way in which the program was instrumented for data capture, and the data that was collected during program execution. S–2396–601 79Cray Application Developer's Environment User's Guide Cray Apprentice2 is not directly integrated with CrayPat. You cannot launch Cray Apprentice2 from within CrayPat, nor can you set up or run performance analysis experiments from within Cray Apprentice2. Rather, use CrayPat ?rst, to instrument your program and capture performance analysis data, and then use Cray Apprentice2 to visualize and explore the resulting data ?les. The number and appearance of the reports that can be generated using Cray Apprentice2 is determined by the kind and quantity of data captured during program execution, which in turn is determined by the way in which the program was instrumented and the environment variables in effect at the time of program execution. For example, changing the PAT_RT_SUMMARY environment variable to 0 before executing the instrumented program nearly doubles the number of reports available when analyzing the resulting data in Cray Apprentice2. To run Cray Apprentice2, load the perftools module, if it is not already loaded. users/yourname> module load perftools Then use the app2 command to launch Cray Apprentice2. users/yourname> app2 [data?le.ap2] & Note: Cray Apprentice2 requires that your workstation be con?gured to host X Window System sessions. If the app2 command returns an "unable to open display" error, contact your system administrator for help in con?guring X Window System hosting and forwarding. At this point the GUI takes over. If you speci?ed a data ?le name with the app2 command, the ?le is opened and parsed and the Overview report is displayed. If you did not specify a data ?le name, the Open File window opens and you can use standard GUI tools to browse through the ?le system and select the data ?le you want to open. For more information about using Cray Apprentice2, see the app2(1) man page, the Cray Apprentice2 help system, and Using Cray Performance Analysis Tools. 7.5 Using PAPI The Performance API (PAPI) is a standard API for accessing microprocessor registers. CrayPat uses PAPI to interface to the Cray system hardware; therefore the module papi is normally loaded as part of the perftools module. 80 S–2396–601Optimizing Code [7] The interface between PAPI and CrayPat is normally transparent to the user. However, advanced users may want to bypass CrayPat and work with PAPI directly. In this case, you must unload the perftools module and then reload only the papi module. > module unload perftools > module load papi Note: CrayPat and direct PAPI commands cannot be used at the same time. Therefore, pat_build does not allow you to instrument a program that has also been instrumented using calls to PAPI functions. For more information about PAPI, see the intro_papi(3) and papi_counters(5) man pages. To see what sorts of information PAPI is capable of capturing on your system, use the papi_avail and papi_native_avail commands. Note: Effective with PAPI version 3.7.2, The PAPI Group has discontinued providing man pages for individual PAPI functions. Additional information about using PAPI is available through the PAPI website, at http://icl.cs.utk.edu/papi/. S–2396–601 81 TM Cray Application Developer's Environment User's Guide S–2396–60© 2004–2011 Cray Inc. All Rights Reserved. This document or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Inc. The gnulicinfo(7) man page contains the Open Source Software licenses (the "Licenses"). Your use of this software release constitutes your acceptance of the License terms and conditions. U.S. GOVERNMENT RESTRICTED RIGHTS NOTICE The Computer Software is delivered as "Commercial Computer Software" as de?ned in DFARS 48 CFR 252.227-7014. All Computer Software and Computer Software Documentation acquired by or for the U.S. Government is provided with Restricted Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7014, as applicable. Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7013, as applicable. Cray, LibSci, and PathScale are federally registered trademarks and Active Manager, Cray Apprentice2, Cray Apprentice2 Desktop, Cray C++ Compiling System, Cray CX, Cray CX1, Cray CX1-iWS, Cray CX1-LC, Cray CX1000, Cray CX1000-C, Cray CX1000-G, Cray CX1000-S, Cray CX1000-SC, Cray CX1000-SM, Cray CX1000-HN, Cray Fortran Compiler, Cray Linux Environment, Cray SHMEM, Cray X1, Cray X1E, Cray X2, Cray XD1, Cray XE, Cray XEm, Cray XE5, Cray XE5m, Cray XE6, Cray XE6m, Cray XK6, Cray XMT, Cray XR1, Cray XT, Cray XTm, Cray XT3, Cray XT4, Cray XT5, Cray XT5 h , Cray XT5m, Cray XT6, Cray XT6m, CrayDoc, CrayPort, CRInform, ECOphlex, Gemini, Libsci, NodeKARE, RapidArray, SeaStar, SeaStar2, SeaStar2+, The Way to Better Science, Threadstorm, and UNICOS/lc are trademarks of Cray Inc. AMD Opteron is a trademark of Advanced Micro Devices, Inc.. GNU is a trademark of The Free Software Foundation. Intel is a trademark of Intel Corporation or its subsidiaries in the United States and other countries. Java is a trademark of Oracle and/or its af?liates. Linux is a trademark of Linus Torvalds. Lustre is a trademark of Oracle and/or its af?liates. Other names may be trademarks of their respective owners. Moab and TORQUE are trademarks of Adaptive Computing Enterprises, Inc. MySQL is a trademark of Oracle and/or its af?liates. Opteron is a trademark of Advanced Micro Devices, Inc. PBS and PBS Professional are trademarks of Altair Engineering, Inc. and are protected under U.S. and international law and treaties. PETSc is a trademark of Copyright (C) 1995-2004 University of Chicago. PGI is a trademark of The Portland Group Compiler Technology, STMicroelectronics, Inc. PathScale is a trademark of PathScale Inc. Platform is a trademark of Platform Computing Corporation. Panasas and PanFS are trademarks or registered trademarks of Panasas, Inc. in the United States and other countries. RSA is a trademark of RSA Security Inc. SecurID is a trademark of RSA Security Inc. TotalView and TotalView Technologies are registered trademarks of Rogue Wave Software, Inc. NVIDIA, CUDA, and Tesla are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Windows is a trademark of Microsoft Corporation. UNIX, the “X device,” X Window System, and X/Open are trademarks of The Open Group in the United States and other countries. All other trademarks are the property of their respective owners. RECORD OF REVISION S–2396–60 Published July 2011 Supports general availability (GA) release of Cray XT, Cray XE, and Cray XK systems running Cray Linux Environment (CLE) release 3.0 or later. 5.0 Published June 2010 Restructured document now packaged with Cray Application Developer's Environment (CADE) release 5.0 and supporting Cray XT and Cray XE systems running Cray Linux Environment (CLE) release 2.2 or later. 2.2 Published July 2009 Supports general availability (GA) release of Cray XT systems running the Cray XT Programming Environment and CLE 2.2 releases.2.1 Published November 2008 Supports general availability (GA) release of Cray XT systems running the Cray XT Programming Environment and CLE 2.1 releases. 2.1 Published July 2008 Supports limited availability (LA) release of Cray XT systems running the Cray XT Programming Environment and CLE 2.1 releases. 2.0 Published October 2007 Supports general availability (GA) release of Cray XT systems running the Cray XT Programming Environment 2.0 and UNICOS/lc 2.0 releases. 2.0 Published June 2007 Supports limited availability (LA) release of Cray XT systems running the Cray XT Programming Environment 2.0 and UNICOS/lc 2.0 releases. 1.5 Published November 2006 Supports general availability (GA) release of Cray XT systems running the Cray XT Programming Environment 1.5 and UNICOS/lc 1.5 releases. 1.5 Published August 2006 Supports limited availability (LA) release of Cray XT systems running the Cray XT Programming Environment 1.5 and Cray Linux Environment (CLE) 1.5 releases. 1.4 Published April 2006 Supports Cray XT3 systems running the Cray XT3 Programming Environment 1.4 and UNICOS/lc 1.4 releases. 1.3 Published November 2005 Supports Cray XT3 systems running the Cray XT3 Programming Environment 1.3 and UNICOS/lc 1.3 releases. 1.2 Published August 2005 Supports Cray XT3 systems running the Cray XT3 Programming Environment 1.2 and UNICOS/lc 1.2 releases. 1.1 Published June 2005 Supports Cray XT3 systems running the Cray XT3 Programming Environment 1.1 and UNICOS/lc 1.1 releases. 1.0 Published March 2005 Draft documentation to support Cray XT3 limited-availability systems. 1.0 Published December 2004 Draft documentation to support Cray XT3 early-production systems.Changes to this Document Cray Application Developer's Environment User's Guide S–2396–60 This rewrite of the Cray Application Developer's Environment User's Guide supports the 6.0 release of the Cray Application Developer's Environment (CADE). S–2396–60 Added information • Preliminary information regarding support for Cray XK systems has been added throughout this guide. • Preliminary information regarding support for AMD Interlagos processors has been added through this guide. • Use of Cray CPU and network-type targeting modules on Cray systems is described in Using Targeting Modules on page 28. Use of targeting modules on standalone Linux workstations is described in About Cross-compilers on page 43. • A new library of eigensolvers, the Cray Adaptive Simple Eigensolvers (CASE), has been added. For more information, see Cray Adaptive Simple Eigensolvers (CASE) on page 52. • The default behavior of MPI and SHMEM modules has been changed signi?cantly. For more information, see MPT on page 59. • Abnormal Termination Processing (ATP) is now enabled by default. For more information, see Using Abnormal Termination Processing (ATP) on page 65. Revised information • Information regarding the use of hugepages has been revised and expanded. For more information, see Hugepages on page 61. • Beginning with Cray Scienti?c Libraries Release 11.0.00, the way in which dynamic libraries are found at execution time has changed. For more information, see New in LibSci Release 11.0 on page 47. • Beginning with PETSc Release 3.1.08, the Third-Party Scienti?c Libraries (TPSL) module is loaded automatically along with the PETSc module. This change affects both PETSc and Trilinos, and is described in PETSc on page 56 and Trilinos on page 58. Deleted information • Support for Cray Linux Environment (CLE) 2.1 is discontinued. Therefore all references to CLE 2.1 have been removed from this guide. • Beginning with Cray Scienti?c Libraries Release 11.0.00, Cray no longer provides a version of LibSci optimized to support the PathScale Compiler Suite.Contents Page Introduction [1] 11 1.1 What You Must Know About Your Site . . . . . . . . . . . . . . . . . . 11 1.1.1 Cray XT, Cray XE, or Cray XK? . . . . . . . . . . . . . . . . . . 12 1.1.2 How Many Cores? . . . . . . . . . . . . . . . . . . . . . . 12 1.1.3 Which Operating System? . . . . . . . . . . . . . . . . . . . . 13 1.1.4 What is a Compute Node? . . . . . . . . . . . . . . . . . . . . 14 1.1.5 Which File System? . . . . . . . . . . . . . . . . . . . . . . 14 1.1.6 Which Batch System? . . . . . . . . . . . . . . . . . . . . . 15 1.1.7 What is the hostname? . . . . . . . . . . . . . . . . . . . . . 15 1.2 Logging In . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.2.1 UNIX or Linux Users . . . . . . . . . . . . . . . . . . . . . 15 1.2.2 Windows Users . . . . . . . . . . . . . . . . . . . . . . . 17 1.2.3 Apple Macintosh Users . . . . . . . . . . . . . . . . . . . . . 20 1.3 Navigating the File Systems . . . . . . . . . . . . . . . . . . . . . 21 Using Modules [2] 23 2.1 What is Loaded Now? . . . . . . . . . . . . . . . . . . . . . . 24 2.2 What is Available? . . . . . . . . . . . . . . . . . . . . . . . 25 2.3 Loading and Unloading Module?les . . . . . . . . . . . . . . . . . . . 26 2.4 Swapping Module?les . . . . . . . . . . . . . . . . . . . . . . 27 2.5 Using Targeting Modules . . . . . . . . . . . . . . . . . . . . . . 28 2.6 Release Notes and Module Help . . . . . . . . . . . . . . . . . . . . 28 2.7 For More Information . . . . . . . . . . . . . . . . . . . . . . . 30 Batch Systems and Program Execution [3] 31 3.1 Interactive Mode . . . . . . . . . . . . . . . . . . . . . . . . 32 3.1.1 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.2 Batch Mode . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.3 Using aprun . . . . . . . . . . . . . . . . . . . . . . . . . 35 S–2396–60 7Cray Application Developer's Environment User's Guide Page Using Compilers [4] 37 4.1 About Compiler Drivers . . . . . . . . . . . . . . . . . . . . . . 37 4.1.1 Bypassing the Compiler Drivers . . . . . . . . . . . . . . . . . . 38 4.2 About C/C++ Floating Point Data Types . . . . . . . . . . . . . . . . . 38 4.3 About the Cray Compiling Environment (CCE) . . . . . . . . . . . . . . . . 39 4.3.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 39 4.4 About PGI Compilers . . . . . . . . . . . . . . . . . . . . . . . 40 4.4.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 40 4.5 About Intel Compilers . . . . . . . . . . . . . . . . . . . . . . 41 4.5.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 41 4.6 About PathScale Compilers . . . . . . . . . . . . . . . . . . . . . 41 4.6.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 41 4.7 About GNU Compilers . . . . . . . . . . . . . . . . . . . . . . 42 4.7.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 42 4.7.2 Known Warnings . . . . . . . . . . . . . . . . . . . . . . 42 4.8 About the Chapel Parallel Programming Language . . . . . . . . . . . . . . . 43 4.9 About Cross-compilers . . . . . . . . . . . . . . . . . . . . . . 43 Libraries [5] 47 5.1 Basic Scienti?c Libraries (LibSci) . . . . . . . . . . . . . . . . . . . 47 5.1.1 New in LibSci Release 11.0 . . . . . . . . . . . . . . . . . . . . 47 5.1.2 Basic LibSci Components . . . . . . . . . . . . . . . . . . . . 48 5.1.3 BLAS and LAPACK . . . . . . . . . . . . . . . . . . . . . . 48 5.1.3.1 Notes . . . . . . . . . . . . . . . . . . . . . . . . 49 5.1.4 BLACS and ScaLAPACK . . . . . . . . . . . . . . . . . . . . 50 5.1.4.1 Notes . . . . . . . . . . . . . . . . . . . . . . . . 51 5.1.5 Iterative Re?nement Toolkit (IRT) . . . . . . . . . . . . . . . . . . 51 5.1.6 Cray Adaptive Simple Eigensolvers (CASE) . . . . . . . . . . . . . . . 52 5.1.7 Fourier Transformations . . . . . . . . . . . . . . . . . . . . . 52 5.1.7.1 CRAFFT . . . . . . . . . . . . . . . . . . . . . . . 53 5.1.7.2 FFTW . . . . . . . . . . . . . . . . . . . . . . . . 53 5.1.7.3 ACML . . . . . . . . . . . . . . . . . . . . . . . . 54 5.2 Fast Math Intrinsics . . . . . . . . . . . . . . . . . . . . . . . 54 5.3 PETSc . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 5.3.1 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 57 5.4 Trilinos . . . . . . . . . . . . . . . . . . . . . . . . . . 58 5.5 MPT . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 5.5.1 Using MPI and SHMEM Modules . . . . . . . . . . . . . . . . . . 59 8 S–2396–60Contents Page 5.5.1.1 CLE 4.0 Systems . . . . . . . . . . . . . . . . . . . . . 59 5.5.1.2 CLE 3.1 Systems . . . . . . . . . . . . . . . . . . . . . 60 5.5.1.3 CLE 3.0 Systems . . . . . . . . . . . . . . . . . . . . . 60 5.5.1.4 CLE 2.2 Systems . . . . . . . . . . . . . . . . . . . . . 61 5.6 Hugepages . . . . . . . . . . . . . . . . . . . . . . . . . . 61 5.6.1 Cray XT Usage . . . . . . . . . . . . . . . . . . . . . . . 61 5.6.2 Cray XE Usage . . . . . . . . . . . . . . . . . . . . . . . 62 Debugging Code [6] 63 6.1 Cray Debugger Support Tools . . . . . . . . . . . . . . . . . . . . 63 6.1.1 Using lgdb . . . . . . . . . . . . . . . . . . . . . . . . 64 6.1.2 Using Abnormal Termination Processing (ATP) . . . . . . . . . . . . . . 65 6.1.2.1 On CLE 3.0 or Later . . . . . . . . . . . . . . . . . . . . 65 6.1.2.2 On CLE 2.2 or Earlier . . . . . . . . . . . . . . . . . . . . 66 6.1.3 Using MRNet . . . . . . . . . . . . . . . . . . . . . . . 66 6.1.4 Using STAT . . . . . . . . . . . . . . . . . . . . . . . . 66 6.2 Using Cray Fast-Track Debugging . . . . . . . . . . . . . . . . . . . 67 6.2.1 Supported Compilers and Debuggers . . . . . . . . . . . . . . . . . 67 6.3 About Core Files . . . . . . . . . . . . . . . . . . . . . . . . 68 6.4 Using TotalView . . . . . . . . . . . . . . . . . . . . . . . . 68 6.4.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 69 6.5 Using DDT . . . . . . . . . . . . . . . . . . . . . . . . . 70 6.5.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 71 Performance Analysis [7] 73 7.1 Using CrayPat . . . . . . . . . . . . . . . . . . . . . . . . . 74 7.1.1 Instrumenting the Program . . . . . . . . . . . . . . . . . . . . 74 7.1.2 Collecting Data . . . . . . . . . . . . . . . . . . . . . . . 75 7.1.3 Analyzing Data . . . . . . . . . . . . . . . . . . . . . . . 75 7.1.4 For More Information . . . . . . . . . . . . . . . . . . . . . 76 7.2 Using Cray Apprentice2 . . . . . . . . . . . . . . . . . . . . . . 76 7.3 Using PAPI . . . . . . . . . . . . . . . . . . . . . . . . . 77 Figures Figure 1. Selecting SSH Protocol . . . . . . . . . . . . . . . . . . . . 17 Figure 2. Enabling X11 Forwarding . . . . . . . . . . . . . . . . . . . 18 Figure 3. Logging In . . . . . . . . . . . . . . . . . . . . . . . 19 S–2396–60 9Cray Application Developer's Environment User's Guide Page Tables Table 1. aprun versus qsub Options . . . . . . . . . . . . . . . . . . 36 Table 2. C/C++ Data Type Sizes . . . . . . . . . . . . . . . . . . . . 38 Table 3. Cray Compiler Basics . . . . . . . . . . . . . . . . . . . . . 39 Table 4. PGI Compiler Basics . . . . . . . . . . . . . . . . . . . . . 40 Table 5. Intel Composer Basics . . . . . . . . . . . . . . . . . . . . 41 Table 6. PathScale Compiler Basics . . . . . . . . . . . . . . . . . . . 41 Table 7. GNU Compiler Basics . . . . . . . . . . . . . . . . . . . . 42 Table 8. LibSci Basics . . . . . . . . . . . . . . . . . . . . . . . 48 Table 9. Fast_mv Basics . . . . . . . . . . . . . . . . . . . . . . 54 Table 10. Fast_mv Math Intrinsics . . . . . . . . . . . . . . . . . . . . 55 Table 11. PETSc Basics . . . . . . . . . . . . . . . . . . . . . . 56 Table 12. Trilinos Basics . . . . . . . . . . . . . . . . . . . . . . 58 Table 13. MPT Basics . . . . . . . . . . . . . . . . . . . . . . . 59 Table 14. lgdb Basics . . . . . . . . . . . . . . . . . . . . . . . 64 Table 15. atp Basics . . . . . . . . . . . . . . . . . . . . . . . 65 Table 16. TotalView Basics . . . . . . . . . . . . . . . . . . . . . 68 Table 17. DDT Basics . . . . . . . . . . . . . . . . . . . . . . . 70 Table 18. Performance Analysis Basics . . . . . . . . . . . . . . . . . . 73 10 S–2396–60Introduction [1] This guide describes the software environment and tools used to develop, debug, and run applications on Cray XT, Cray XE, and Cray XK systems. It is intended as a general overview and introduction to the Cray system for new users and application programmers. This guide is intended to be used in conjunction with Workload Management and Application Placement for the Cray Linux Environment (S–2496), which describes the Application Level Placement Scheduler (ALPS) and aprun command in considerably greater detail. The information contained in this guide is of necessity fairly high-level and generalized, as the Cray platform supports a wide variety of hardware nodes as well as many different compilers, debuggers, and other software tools. System hardware and software con?gurations therefore vary considerably from site to site, so for speci?c information about your site and its installed hardware, software, and usage policies, always contact your site administrator. 1.1 What You Must Know About Your Site The Cray XT, Cray XE, and Cray XK systems are: • massively parallel (MPP) • distributed memory • non-uniform memory access (NUMA) • commodity processor-based supercomputers These systems are designed to scale to immense size and run applications requiring high-performance, large-scale processing, high network bandwidth, and complex inter-processor communications. All systems use 64-bit AMD Opteron processors as the basic computational engines, although the exact number of cores per AMD Opteron varies from site to site and sometimes even from cabinet to cabinet, depending on your site's installation, expansion, and upgrade history. S–2396–60 11Cray Application Developer's Environment User's Guide 1.1.1 Cray XT, Cray XE, or Cray XK? From the programmer's point of view, the most signi?cant differences between the systems are: • Cray XT systems use SeaStar or SeaStar2+ application-speci?c integrated circuits (ASICs) to manage inter-processor communications • Cray XE systems use Gemini ASICs to manage inter-processor communications • Cray XK systems also use Gemini ASICs for inter-processor communications, but combine AMD Interlagos CPUs and NVIDIA Tesla GPUs on compute nodes Because of the differences in the network ASICs and accompanying network APIs, Cray XT and Cray XE systems use different versions of the MPI and SHMEM libraries, which offer different sets of optional controls to the application developer. Cray XK systems use the same MPI and SHMEM libraries as Cray XE systems. The differences between the versions of MPI and SHMEM are discussed in more detail in MPT on page 59. Cray XE and Cray XK systems support the Generic Network Interface (GNI) and Distributed Shared Memory Application (DMAPP) APIs, which offer the programmer API-level access to the advanced features of the Gemini network interconnect. The Gemini GNI and DMAPP APIs are discussed in detail in Using the GNI and DMAPP APIs (S–2446). 1.1.2 How Many Cores? The next most signi?cant differences are between models. • Cray XT4 systems use one dual- or quad-core ("Barcelona") AMD Opteron CPU per compute node • Cray XT5 and Cray XE5 systems use two four-core ("Shanghai") or six-core ("Istanbul") AMD Opteron CPUs per compute node • Cray XT6 and Cray XE6 systems use two eight- or twelve-core ("Magny-Cours") or two sixteen-core ("Interlagos") AMD Opteron CPUs per compute node • Cray XK systems use one sixteen-core ("Interlagos") AMD Opteron CPU and one NVIDIA Tesla GPU per compute node Because of these hardware differences, you can invoke different options when compiling and executing your programs. These differences are discussed in more detail in Workload Management and Application Placement for the Cray Linux Environment (S–2496). 12 S–2396–60Introduction [1] 1.1.3 Which Operating System? All current Cray systems run the Cray Linux Environment (CLE) operating system on the login nodes and a lightweight kernel called CNL on the compute nodes. Some of the options available to application developers vary depending on which version of CLE is currently running on the system. • Cray XK systems run CLE release 4.0 or later. • Cray XE5 and Cray XE6 systems run CLE release 3.1 or later. • Cray XT6 and Cray XT6m systems run CLE release 3.0 or later. • All other systems may be running CLE release 2.2, 3.0, 3.1, or later. If you are not certain which release your site is using, check the MOTD when you log in. If the information is not there, there are two other ways to determine the CLE release number. • On CLE 2.x systems, check to see which version of the xt-os module is loaded by default. (For more information about using modules, see Chapter 2, Using Modules on page 23.) If no xt-os module is present, your system is running CLE release 3.0 or later. • On CLE 3.0 and later systems, cat the contents of the /etc/opt/cray/release/clerelease ?le. This returns the CLE release and update number. If this ?le or directory does not exist, your system is running CLE release 2.x. Note: CLE 2.x is based on SLES 10. CLE 3.x and CLE 4.x are based on SLES 11. S–2396–60 13Cray Application Developer's Environment User's Guide 1.1.4 What is a Compute Node? From the application developer's point of view, a Cray system is a tightly integrated network of thousands of nodes, all specialized for different purposes. While some are dedicated to administrative or networking functions and therefore off-limits to application programmers, the two you will deal with most often are: • login nodes — The node you access when you ?rst log in to the system. Login nodes offer the full Cray Linux Environment (CLE) operating system, are used for basic development tasks such as editing ?les and compiling code, generally have access to the network ?le system, and are shared resources that may be used concurrently by multiple users. Login nodes are also sometimes called service nodes. • compute nodes — The nodes on which production jobs are executed. Compute nodes run a lightweight operating system called Compute Node Linux (CNL), can be accessed only by submitting jobs through the batch management system (typically PBS Pro, Moab/TORQUE, or Platform LSF), generally have access only to the high-performance parallel ?le system (typically Lustre or Panasas), and are dedicated resources, exclusively yours for the duration of the batch reservation. When new users ?rst begin working on the Cray system, this difference between login and compute nodes can be confusing. Remember, when you ?rst log in to the system, you are placed on a login node. You cannot execute parallel programs on the login node, nor can you directly access ?les stored on the high-performance parallel ?le system. Instead, use your site's batch system to place parallel programs on the compute nodes, either from the login node or from a mount-point on the parallel ?le system. Note: You can execute serial (single-process) programs on login nodes, but executing large or long-running serial programs on login nodes is discouraged, as login nodes are shared resources. 1.1.5 Which File System? All Cray systems require the use of a high-performance parallel ?le system. Most sites currently use the Lustre File System, although others such as the Panasas PanFS are also supported. All examples shown in this guide were developed on a Lustre ?le system using Lustre commands. Before copying any examples from this guide verbatim, verify which ?le system your site uses and what your site's policies are regarding home directories, scratch space, disk quotas, backup policies, and so on. Then if required, adjust the instructions accordingly. 14 S–2396–60Introduction [1] 1.1.6 Which Batch System? Cray systems typically operate under the control of a batch system such as PBS Pro, OpenPBS, Moab/TORQUE Resource Manager, or Platform LSF. All examples shown in this guide were developed using either PBS Pro or Moab/TORQUE. Before copying any examples from this guide verbatim, verify which batch system your site uses and if required, adjust the instructions accordingly. 1.1.7 What is the hostname? All Cray systems have a unique hostname. While it is possible to log in to a given system by using its IP address (e.g., 127.0.0.1), it's easier if you use the hostname. The form of the hostname varies from site to site. Some sites use a single word; e.g., narwhal. Others use a fully quali?ed URL; e.g., narwhal.univ-alaska-barrow.edu. You should be told which name and form to use when your user account is set up. 1.2 Logging In User account setup and authentication policies vary widely from site to site. In general, you must contact your site administrator to get a login account on the system. Any site-speci?c security or authentication policies (such as, say, the correct use of an RSA SecurID token, if provided) should be explained to you at that time. Once your user account is created, log in to the Cray system using SSH (Secure Shell), protocol version 2. SSH is a remote login program that encrypts all communications between the client and host and replaces the earlier telnet, rlogin, and rsh programs. 1.2.1 UNIX or Linux Users If you use a UNIX or Linux workstation, the ssh utility is generally available at any command line and documented in the ssh(1) man page. To log in to the Cray system, enter: % ssh -X hostname The -X option enables X11 display forwarding. Automatic forward of X11 windows is highly recommended as many application development tools use GUI displays. S–2396–60 15Cray Application Developer's Environment User's Guide On some systems, you may be required to enter your user ID as well. This can be done in several different ways. For example: % ssh -X -luserID hostname Or % ssh -X userID@hostname In any case, after you SSH to the system, you may have to answer one or more RSA or password challenges, and then you are logged into the system. A series of system status and MOTD (message of the day) messages may display, after which you are placed in your home directory on a login node. /users/userID> You are now ready to begin working. Jump to Navigating the File Systems on page 21. 16 S–2396–60Introduction [1] 1.2.2 Windows Users If you use a Windows personal computer, you ?rst need to obtain and install a client program for your system that supports SSH protocol 2, such as PuTTY for Windows. Your system administrator should be able to provide a list of accepted clients. You may need to con?gure your client to support SSH protocol 2 and X11 forwarding. For example, if you are using PuTTY, you may need to click SSH in the left pane to see the preferred SSH protocol version: Figure 1. Selecting SSH Protocol Verify that the Preferred SSH protocol version is set to 2. S–2396–60 17Cray Application Developer's Environment User's Guide Then click X11 in the left pane to view the SSH X11 forwarding options: Figure 2. Enabling X11 Forwarding If necessary, click the Enable X11 forwarding checkbox. 18 S–2396–60Introduction [1] Then click Session in the left pane to return to the Basic options window. Figure 3. Logging In Enter the hostname in the Host Name ?eld and click the Open button to begin your SSH session. You may need to enter your userID and answer one or more RSA or password challenges, and then you are logged into the system. A series of system status and MOTD (message of the day) messages may display, after which you are placed in your home directory on a login node. /users/userID> You are now ready to begin working. S–2396–60 19Cray Application Developer's Environment User's Guide 1.2.3 Apple Macintosh Users The Apple Macintosh OS X operating system is based on UNIX. Therefore, to log in to the Cray system, open the Terminal application, and then use the ssh command to connect to the Cray system. % ssh -X hostname The -X option enables X11 display forwarding with X11 security extension restrictions. Automatic forward of X11 windows is highly recommended as many application development tools use GUI displays. Note: The version of SSH found in OS X also supports the -Y argument, as well as the -X argument. The -Y argument enables "trusted" X11 forwarding and may work better than -X for some users. On some systems, you may be required to enter your user ID as well. This can be done in several different ways. For example: % ssh -X -luserID hostname Or % ssh -X userID@hostname In any case, after you SSH to the system, you may have to answer one or more RSA or password challenges, and then you are logged into the system. A series of system status and MOTD (message of the day) messages may display, after which you are placed in your home directory on a login node. /users/userID> You are now ready to begin working on the Cray system. 20 S–2396–60Introduction [1] 1.3 Navigating the File Systems When you ?rst log in to the Cray system, you are placed in your home directory on a login node. /users/userID> At this point you have access to all the features and functions of the full Cray Linux Environment (CLE) operating system, such as the sftp and scp commands, and also typically to your full network ?le system. On most systems your home directory on the login node is de?ned as the environment variable $HOME, and this variable can be used in any ?le system command. For example, to return to your home directory from any other location in the ?le system(s), enter this command: > cd $HOME Remember, you can edit ?les, manipulate ?les, compile code, execute serial (single-process) programs, and otherwise work in your home directory on the login node. However, you cannot execute parallel programs on the login node. Parallel programs must be run on the compute nodes, under the control of the batch system, and generally while mounted on the high-performance parallel ?le system. To do this, you must ?rst identify the nids (node IDs) of the ?le system mount points. On the Lustre ?le system, this can be done in one of two ways. Either enter the df -t lustre command to ?nd the Lustre nodes and get a summary report on disk usage: users/userID> df -t lustre Filesystem 1K-blocks Used Available Use% Mounted on 8@ptl:/narwhalnid8 8998913280 6946443260 1595348672 82% /lus/nid00008 Or enter the lfs df command to get more detailed information: users/userID> lfs df UUID 1K-blocks Used Available Use% Mounted on nid00008_mds_UUID 179181084 2675664 166265604 1% /lus/nid00008[MDT:0] ost0_UUID 1124864160 895207088 172517160 79% /lus/nid00008[OST:0] ost1_UUID 1124864160 838067380 229656540 74% /lus/nid00008[OST:1] ost2_UUID 1124864160 826599428 241124820 73% /lus/nid00008[OST:2] ost3_UUID 1124864160 827914052 239801932 73% /lus/nid00008[OST:3] ost4_UUID 1124864160 964324672 103398548 85% /lus/nid00008[OST:4] ost5_UUID 1124864160 932986208 134738024 82% /lus/nid00008[OST:5] ost6_UUID 1124864160 832715148 235009164 74% /lus/nid00008[OST:6] ost7_UUID 1124864160 828631656 239092572 73% /lus/nid00008[OST:7] filesystem summary: 8998913280 6946445632 1595338760 77% /lus/nid00008 Note: The above commands are speci?c to the Lustre high-speed parallel ?le system. If your site uses a different ?le system, adjust the instructions accordingly. S–2396–60 21Cray Application Developer's Environment User's Guide In either case, in this example, the Lustre mount point is /lus/nid00008. If you cd to this mount point: users/userID> cd /lus/nid00008 Directory: /lus/nid00008 /lus/nid00008> you are now on the high-performance parallel ?le system. At this point you can edit ?les, manipulate ?les, compile code, and so on, but you can also execute programs on the compute nodes, generally by using the batch system. 22 S–2396–60Using Modules [2] The Cray system uses the Modules environment management package to support dynamic modi?cation of the user environment via module?les. Each module?le contains all the information needed to con?gure the shell for a particular application. To make major changes in your user environment, such as switching to a different compiler or a different version of a library, use the appropriate Modules commands to select the desired module?les. The advantage in using Modules is that you are not required to specify explicit paths for different executable versions or to set the $MANPATH and other environment variables manually. Instead, all the information required in order to use a given piece of software is embedded in the module?le and set automatically when you load the module?le. In general, the simplest way to make certain that the elements of your application development environment function correctly together is by using the Modules software and trusting it to keep track of paths and environment variables, and avoiding embedding speci?c directory paths into your startup ?les, make?les, and scripts. S–2396–60 23Cray Application Developer's Environment User's Guide 2.1 What is Loaded Now? When you ?rst log in to the Cray system, a set of site-speci?c default modules is loaded. This set varies depending on system hardware, operating system release level, site policies, and installed software. To see which modules are currently loaded on your system, use the module list command. users/yourname> module list Currently Loaded Modulefiles: 1) modules/3.2.6.6 2) nodestat/2.2-1.0400.25564.5.2.gem 3) sdb/1.0-1.0400.27358.12.17.gem 4) MySQL/5.0.64-1.0000.2899.19.2 5) lustre-cray_gem_s/1.8.4_2.6.32.36_0.5.2_1.0400.5941.3.1-1.0400.27356.3.114 6) udreg/2.3.1-1.0400.3052.6.12.gem 7) ugni/2.2-1.0400.3340.9.29.gem 8) gni-headers/2.1-1.0400.3411.8.1.gem 9) dmapp/3.1-1.0400.3387.13.25.gem 10) xpmem/0.1-2.0400.27172.6.5.gem 11) Base-opts/1.0.2-1.0400.25719.5.1.gem 12) xtpe-network-gemini 13) pgi/11.6.0 14) totalview-support/1.1.2 15) xt-totalview/8.9.1 16) xt-libsci/10.5.02 17) pmi/2.1.2-1.0000.8396.13.5.gem 18) xt-asyncpe/5.00.59 19) atp/1.2.0 20) PrgEnv-pgi/4.0.12A 21) pbs/10.4.0.101257 22) xtpe-mc12 23) xt-mpich2/5.3.0 This list breaks down into three groups: operating system modules, programming environment modules, and support modules. For example, the xtpe-mc12 module indicates that this development environment is set up to develop code for use on 12-core Magny-Cours processors, while the PrgEnv-pgi module indicates that PGI compiler suite is currently loaded. 24 S–2396–60Using Modules [2] 2.2 What is Available? To see which module?les are available on your system, enter this command: % module avail [string] [-subset?ag] The module avail command produces an alphabetical listing of every module?le in your module use path and has no option for "grepping." Therefore, it is usually more useful to use the command with an string argument. For example, if you are looking for a speci?c version of the PGI compiler suite, you would enter this command: users/yourname> module avail PrgEnv-pgi -------------------------------------- /opt/modulefiles -------------------------------------- PrgEnv-pgi/3.1.35 PrgEnv-pgi/3.1.37E PrgEnv-pgi/3.1.61 PrgEnv-pgi/3.1.37AA PrgEnv-pgi/3.1.37G PrgEnv-pgi/4.0.12A(default) PrgEnv-pgi/3.1.37C PrgEnv-pgi/3.1.49A One module is usually designated as the default version. Whether this is the most recent version of this module depends on your site policies. Some sites always make the newest version the default, while others wait until after the new version has been tested and proven bug- and dependency-free. Whenever a newer version of a module is installed, the older versions continue to remain available, unless the site administrator has explicitly chosen to delete them. The [-subset?ag] option lets you list a subset of available modules. The following ?ags may be used alone or in combinations: -U List user modules -D List the current default modules -T List tool modules (debuggers, performance analysis utilities, and the like) -L List library modules (see Chapter 5, Libraries on page 47) -P List Programming Environment (compiler) modules -X List CPU and network targeting modules (Barcelona, Magny-Cours, Interlagos, and the like) S–2396–60 25Cray Application Developer's Environment User's Guide 2.3 Loading and Unloading Module?les If a module?le is not already loaded, use the module load command to load it. users/yourname> module load module?le This command loads the currently de?ned default version of the module, unless you specify otherwise. For example, if the modules listed in What is Available? on page 25 are present on your system but PrgEnv-pgi is not already loaded, then: users/yourname> module load PrgEnv-pgi loads PrgEnv-pgi/4.0.12A(default) module. To load a non-default version of the module, you must specify the full module name. users/yourname> module load PrgEnv-pgi/3.1.61 However, if a module is already loaded, then you must ?rst unload the currently loaded module before loading a different version. For example, to change from the default to version of the PGI compiler suite to another version, use the module unload command to remove the version currently loaded. users/yourname> module unload PrgEnv-pgi Modules may be linked and related. If you enter the module list command now and compare the results to those shown in What is Loaded Now? on page 24: users/yourname> module list Currently Loaded Modulefiles: 1) modules/3.2.6.6 2) nodestat/2.2-1.0400.25564.5.2.gem 3) sdb/1.0-1.0400.27358.12.17.gem 4) MySQL/5.0.64-1.0000.2899.19.2 5) lustre-cray_gem_s/1.8.4_2.6.32.36_0.5.2_1.0400.5941.3.1-1.0400.27356.3.114 6) udreg/2.3.1-1.0400.3052.6.12.gem 7) ugni/2.2-1.0400.3340.9.29.gem 8) gni-headers/2.1-1.0400.3411.8.1.gem 9) dmapp/3.1-1.0400.3387.13.25.gem 10) xpmem/0.1-2.0400.27172.6.5.gem 11) Base-opts/1.0.2-1.0400.25719.5.1.gem 12) xtpe-network-gemini 13) pbs/10.4.0.101257 14) xtpe-mc12 15) xt-mpich2/5.3.0 You can see that along with PrgEnv-pgi, the pgi, totalview-support, xt-totalview, xt-libsci, pmi, xt-asyncpe, and atp modules were also unloaded. If you now load a different version of the PrgEnv-pgi module: users/yourname> module load PrgEnv-pgi/3.1.61 Then use the module list command again, you can see that the associated modules have been reloaded automatically, but this time using the versions that are appropriate for use with the 3.1.61 version of the PGI compiler. 26 S–2396–60Using Modules [2] 2.4 Swapping Module?les Alternatively, you can use the module swap or module switch command to unload one module and load the comparable module. For example, to switch from using the PGI Compiler Suite to the Cray Compiling Environment, enter this command: users/yourname> module swap PrgEnv-pgi PrgEnv-cray If you then list the modules that are loaded after this swap and compare them to the results shown in What is Loaded Now? on page 24. users/yourname> module list Currently Loaded Modulefiles: 1) modules/3.2.6.6 2) nodestat/2.2-1.0400.25564.5.2.gem 3) sdb/1.0-1.0400.27358.12.17.gem 4) MySQL/5.0.64-1.0000.2899.19.2 5) lustre-cray_gem_s/1.8.4_2.6.32.36_0.5.2_1.0400.5941.3.1-1.0400.27356.3.114 6) udreg/2.3.1-1.0400.3052.6.12.gem 7) ugni/2.2-1.0400.3340.9.29.gem 8) gni-headers/2.1-1.0400.3411.8.1.gem 9) dmapp/3.1-1.0400.3387.13.25.gem 10) xpmem/0.1-2.0400.27172.6.5.gem 11) Base-opts/1.0.2-1.0400.25719.5.1.gem 12) xtpe-network-gemini 13) pbs/10.4.0.101257 14) xtpe-mc12 15) PrgEnv-cray/4.0.12A 16) atp/1.2.0 17) xt-asyncpe/5.00.59 18) rca/1.0.0-2.0400.27173.6.23.gem 19) pmi/2.1.2-1.0000.8396.13.5.gem 20) xt-libsci/10.5.02 21) acml/4.4.0 22) xt-totalview/8.9.1 23) totalview-support/1.1.2 24) cce/7.4.1.103 You can see that a different set of supporting modules have been loaded automatically. S–2396–60 27Cray Application Developer's Environment User's Guide 2.5 Using Targeting Modules The targeting modules deserve special mention. To see which targeting modules are available on your system, use the module avail -X command. It returns a list like this, which shows the CPU and network-type modules currently available. ---------------------------- /opt/cray/xt-asyncpe/default/modulefiles ---------------------------- xtpe-barcelona xtpe-mc12 xtpe-network-seastar xtpe-interlagos xtpe-mc8 xtpe-shanghai xtpe-istanbul xtpe-network-gemini xtpe-target-native If you are working on a Cray system, your default environment should load the CPU and network-type modules that are correct for your hardware. For example, if you have a Cray XE6 system with eight-core Magny-Cours compute nodes, your default environment should include the xtpe-network-gemini and xtpe-mc8 modules, and no others. If the correct modules are loaded, don't change them. However, if you are working on a standalone Linux workstation and developing executable code which will then be moved to and run on a Cray system, always make certain that your local development environment contains the correct targeting modules for the Cray system on which you plan to run your code. For example, code compiled with the wrong CPU module loaded, or with the xtpe-network-seastar module instead of the xtpe-network-gemini module loaded, will not run correctly on a Cray XE system. For more information about using standalone Linux workstations to develop code, see About Cross-compilers on page 43. Note: Alternatively, if your site has a heterogeneous system with more than one type of compute node (for example, a Cray XE6 system with both Magny-Cours and Interlagos compute nodes), load the targeting module for the type of compute node on which you intend to execute your code, and then make certain your job is placed only on the speci?ed type of compute node. For more information about job placement, see Workload Management and Application Placement for the Cray Linux Environment (S–2496). Note: The xtpe-target-native module is not a CPU or network-type module and used only in limited and special cases. For more information, see Using Targeting Modules on page 28. 2.6 Release Notes and Module Help Most modules on the Cray system include module help that is speci?c to the module. The exact content of the module help varies from vendor to vendor and release to release, but generally includes release notes and late-breaking news, such as lists of bugs ?xed in the release, known dependencies and limitations, and usage information received too late to be documented elsewhere. 28 S–2396–60Using Modules [2] You can view the module help at any time for any module currently installed on the system. The module does not need to be loaded in order for you to view the module help. To access the module help, use the module help command. For example, to see the module help associated with the default PGI module, enter this command: users/yourname> module help pgi Note: Make certain you specify the exact module name (and if not the default, the module version) that you want. For example, module help PrgEnv-pgi and module help pgi display different information. S–2396–60 29Cray Application Developer's Environment User's Guide 2.7 For More Information The Modules commands are documented in the module(1) and modulefiles(4) man pages. A summary of Modules commands can be displayed by entering the module help command. users/yourname> module help Modules Release 3.2.6.6 2007-02-14 (Copyright GNU GPL v2 1991): Usage: module [ switches ] [ subcommand ] [subcommand-args ] Switches: -H|--help this usage info -V|--version modules version & configuration options -f|--force force active dependency resolution -t|--terse terse format avail and list format -l|--long long format avail and list format -h|--human readable format avail and list format -v|--verbose enable verbose messages -s|--silent disable verbose messages -c|--create create caches for avail and apropos -i|--icase case insensitive -u|--userlvl set user level to (nov[ice],exp[ert],adv[anced]) Available SubCommands and Args: + add|load modulefile [modulefile ...] + rm|unload modulefile [modulefile ...] + switch|swap [modulefile1] modulefile2 + display|show modulefile [modulefile ...] + avail [modulefile [modulefile ...]] + use [-a|--append] dir [dir ...] + unuse dir [dir ...] + update + refresh + purge + list + clear + help [modulefile [modulefile ...]] + whatis [modulefile [modulefile ...]] + apropos|keyword string + initadd modulefile [modulefile ...] + initprepend modulefile [modulefile ...] + initrm modulefile [modulefile ...] + initswitch modulefile1 modulefile2 + initlist + initclear Different versions of the Modules software are in use on different sites. Accordingly, the module command arguments and options available on your site may vary from those shown here. 30 S–2396–60Batch Systems and Program Execution [3] User applications are always launched on compute nodes using the application launcher, aprun, which submits applications to the Application Level Placement Scheduler (ALPS) for placement and execution. The ALPS service is both very powerful and highly ?exible, and a thorough discussion of it is beyond the scope of this manual. For more detailed information about ALPS and aprun, see the intro_alps(1) and aprun(1) man pages, and Workload Management and Application Placement for the Cray Linux Environment (S–2496). At most sites, access to the compute node resources is managed by a batch control system, typically PBS Pro, Moab/TORQUE, or Platform LSF. Users run jobs either by using the qsub command to submit a job script (or the equivalent command for their batch control system), or else by using the qsub command (or its equivalent) to request an interactive session within the context of the batch system, and then using the aprun command to run the application within the interactive session. In either case, running an application typically involves these steps. 1. Determine what system resources you will need. Generally, this means deciding how many cores and/or compute nodes you need for your job. 2. Use the apstat command to determine whether the resources you need are available. This is very important when you are planning to run in an interactive session. This is not as important if you are submitting a job script, as the batch system will keep your job script in the queue until the resources become available to run it. 3. Translate your resource request into the appropriate batch system (i.e., qsub) and aprun command options, which are not necessarily the same. If running a batch job, modify your script accordingly. 4. Use the batch command either to submit your job script or to request an interactive session. 5. If using an interactive session, use the aprun command to launch your application. S–2396–60 31Cray Application Developer's Environment User's Guide 3.1 Interactive Mode Interactive mode is typically used for debugging or optimizing code, but not for running production code. For example, to begin an interactive session on a system using PBS Pro, use the qsub -I command. users/yourname> qsub -IVl mppwidth=cores Useful qsub options include: -I Start an interactive session -A account Charge the time to account. -q debug Run in the debug queue. -V Import any environment variables that were set in the user's shell. -l resource_type=speci?cation Specify the resources you want to reserve for this session. For example, you can use this option to reserve 24 compute cores for one hour. -l mppwidth=24,walltime=1:00:00 The -l resource_type=speci?cation argument supports a large number of options which are described in the qsub(1B) man page and expanded upon in the pbs_resources(7B) man page, and described in greater detail with examples in Workload Management and Application Placement for the Cray Linux Environment (S–2496). After you have launched an interactive session, use the aprun command to launch your application. When you are ?nished, enter logout to exit the batch system and return to the normal command line. 3.1.1 Notes • You must use the -l mppwidth= option to specify at least one core when you start the interactive session. If you do not, your request for an interactive session will pause inde?nitely. • Pay attention to your ?le system mount points. You must be on the high-performance parallel ?le system (for example, if you are using the Lustre ?le system, /lus/nid00008/yourname) in order to launch jobs on the compute nodes. However, when you launch an interactive batch session, you are by default placed in your home directory, which typically is on a login node. (For example, /ufs/users/home/yourname.) You may need to cd back to the parallel ?le system after launching an interactive session. 32 S–2396–60Batch Systems and Program Execution [3] • After you launch an interactive batch session, a number of environment variables are automatically de?ned and exported to the job, as described on the qsub(1B) man page. For example, the environment variable $PBS_O_WORKDIR is set to the directory from which the batch job was submitted. This can be a handy way to return to your mount point if you cd to the parallel ?le system before invoking the interactive batch session. • The qsub and aprun commands use different options to perform similar functions. These differences are touched on lightly in Using aprun on page 35 and described in detail in Workload Management and Application Placement for the Cray Linux Environment (S–2496). • When you launch an interactive batch session, you must request the maximum number of resources you expect to use. Once a batch session begins, you can only use fewer resources than initially requested. You cannot use the aprun command to use more resources than you reserved using the qsub command. 3.2 Batch Mode Production jobs are typically run in batch mode. Batch scripts are shell scripts containing ?ags and commands to be interpreted by a shell and are used to run a set of commands in sequence. For example, to run a batch script using PBS Pro, use the qsub command. users/yourname> qsub [-l resource_type=speci?cation] jobscript The [-l resource_type=speci?cation] arguments are described in the pbs_resources(7B) man page. A typical batch script might look like this: 1: #!/bin/bash 2: #PBS -A account 3: #PBS -N job_name 4: #PBS -j oe 5: #PBS -l walltime=1:00:00,mppwidth=192 6: 7: cd $PBS_O_WORKDIR 8: date 9: aprun -n 192 ./a.out > my_output_?le 2>&1 S–2396–60 33Cray Application Developer's Environment User's Guide Parsing this script line-by-line, it would be interpreted as follows: 1. Invoke the shell to use to interpret the script. 2. Specify the account to which this time is billed. 3. Assign the job a name to use on messages and output. 4. Join the job's STDOUT and STDERR into STDOUT. Note: While your job is running, STDOUT and STDERR are written to a ?le or ?les in a system directory and the output is copied to your submission directory only after the job completes. Specifying the -j oe option here and redirecting the output to a ?le in line 9 makes it possible for you to view STDOUT and STDERR while the job is running. For more information about the -j option, see the qsub(1B) man page. 5. Reserve 192 processing elements for one hour. 6. This line is blank and is ignored. 7. cd to the submission directory, which presumably is a mount point on the Lustre ?le system. 8. Run the date command. 9. Execute the executable ?le a.out on 192 processing elements and redirect any output to a ?le. After the job is submitted using the qsub command, it goes into the queue, where it waits until the requested resources become available. When they do, the job is launched on the head node of the allocated resources, and it runs until either it reaches its planned completion or until the wall clock time (if speci?ed) is up. While the job is in the queue, a number of optional commands are available. qstat Show the status of the job queue. This command is available at any time, whether or not you have a job in the queue. qdel job_id Delete job job_id regardless of its current state and remove it from the queue. qhold job_id Place a non-running job on hold. The job remains in the queue but will not execute. This command cannot be used once the job begins running. qrls job_id Release a job that is on hold. 34 S–2396–60Batch Systems and Program Execution [3] qalter job_id Alter the characteristics—name, account, number of requested cores, and so on—of a job in the queue. This command cannot be used once the job begins running. showq (Moab only) Similar to qstat but providing more detail. checkjob job_id (Moab only) Check the status of a job currently in the queue. showstart job_id (Moab only) Show the estimated start time for a job in the queue. showbf (Moab only) Show the current back?ll. This can help you to build small jobs that can be back?lled immediately while you are waiting for the resources to become available for your larger jobs. For more information about batch scripts, see your batch system's user documentation. 3.3 Using aprun The aprun utility launches applications on compute nodes. The utility submits applications to the Application Level Placement Scheduler (ALPS) for placement and execution, forwards the login node environment to the assigned compute nodes, forwards signals, and manages the stdin, stdout, and stderr streams. Verify that you are in a directory mounted on the high-speed parallel ?le system before using the aprun command. In simplest form, the aprun command looks like this: % aprun -n x ./program_name The aprun command supports a large number of options that provide you with a high degree of control over just exactly how your job is placed and executed on the compute nodes. At a minimum, you must use the -n option to specify the number of cores on which to run the job. Note: Remember, you use aprun within the context of a batch session and the maximum size of the job is determined by the resources you requested when you launched the batch session. You cannot use the aprun command to use more resources than you reserved using the qsub command. The aprun and qsub commands support comparable but differently named options. This table lists some of the more commonly used aprun options and their qsub equivalents. S–2396–60 35Cray Application Developer's Environment User's Guide Table 1. aprun versus qsub Options aprun Option qsub -l Option Description -n 4 -l mppwidth=4 Width (number of PEs) -d 2 -l mppdepth=2 Depth (number of CPUs hosting OpenMP threads) -N 1 -l mppnppn=1 Number of PEs per node -L 5,6,7 -l mppnodes=\"5,6,7\" Candidate node List -m 1000 -l mppmem=1000 Memory per PE Note: The -B option forces aprun to inherit the values associated with the -n, -d, -N, and -m options from the batch session. The aprun command exits with an error if you specify any of these options and the -B option at the same time. A full discussion of aprun options is beyond the scope of this manual. For more information see the aprun(1) man page, and for detailed explanations and examples, see Workload Management and Application Placement for the Cray Linux Environment (S–2496). Also, note that the behavior of aprun and the aprun command options supported may vary depending on which version of the CLE operating system is installed on your system. 36 S–2396–60Using Compilers [4] The Cray system supports a variety of compilers from a variety of vendors and support for new compilers and languages is being added on an ongoing basis. The GNU Fortran, C, and C++ compilers are supplied with all systems, while all other compilers are available as optional and separately licensed add-ons. At present, the following compilers from the following vendors are supported on Cray systems. • Cray Inc., Cray Compiling Environment (CCE) (Fortran, C, and C++) • The Portland Group, Parallel Fortran, C, and C++ • Intel Inc., Intel Composer (Fortran and C++) • PathScale Inc., PathScale Compiler Suite (Fortran and C++) • Chapel Parallel Programming Language The compilers available on your system depend on which products your site administration has chosen to license and install. The different compilers have different strengths and weaknesses, and their relative strengths and weaknesses are constantly changing as new revisions and updates are released. 4.1 About Compiler Drivers Because of the multiplicity of possible compilers, Cray supplies compiler drivers, wrapper scripts, and disambiguation man pages. No matter which vendor's compiler module is loaded, always use one of the following commands to invoke the compiler. ftn Invokes the Fortran compiler, regardless of which compiler module is currently loaded. This command links in the fundamental libraries required in order to produce code that can be executed on the Cray compute nodes. For more information, see the ftn(1) man page. cc Invokes the C compiler, regardless of which compiler module is currently loaded. This command links in the fundamental header ?les and libraries required in order to produce code that can be executed on the Cray compute nodes. For more information, see the cc(1) man page. S–2396–60 37Cray Application Developer's Environment User's Guide CC Invokes the C++ compiler, regardless of which compiler module is currently loaded. This command links in the fundamental header ?les and libraries required in order to produce code that can be executed on the Cray compute nodes. For more information, see the CC(1) man page. Note that while you always use one of the above commands (either on the command line or in your make ?les) to invoke the compiler, the arguments used with the commands vary according to which compiler module is loaded. For example, the arguments and options supported by the PGI Fortran compiler are different from those supported by the Cray Fortran compiler. Regardless of which compiler module you have loaded, do not use the native compiler commands. For example, if you are using the PGI compiler suite, do not use the pgf95 command to invoke the Fortran compiler. If you do so, your code may appear to compile and link successfully, but it will be linked to the wrong libraries and the resulting program can be executed on login nodes only; it cannot be executed on compute nodes. 4.1.1 Bypassing the Compiler Drivers In special cases, you may want to bypass the compiler drivers and use the native compiler commands. Do not do so. Instead, load the special targeting module, xtpe-target-native, and then continue to use the ftn, cc, and CC commands as before. The xtpe-target-native module enables the compiler driver commands to function as native compiler commands—for example, if the PGI programming is loaded, the ftn command works as if it is pgf95—but eliminates all default library links, while also preventing any linking to incorrect libraries. To restore normal compiler driver behavior, unload the xtpe-target-native module. 4.2 About C/C++ Floating Point Data Types The C/C++ compilers differ on the size of the long double data type. The PGI and CCE compilers de?ne long double as being 8 bytes. All other compilers de?ne the long double as being 16 bytes. Table 2. C/C++ Data Type Sizes Data Type Size in Bytes unsigned char 1 signed char 1 unsigned short 2 signed short 2 38 S–2396–60Using Compilers [4] Data Type Size in Bytes unsigned int 4 signed int 4 unsigned long 8 signed long 8 unsigned long long 8 signed long long 8 ?oat 4 double 8 long double 8 or 16, depending on compiler char * 8 enum 4 4.3 About the Cray Compiling Environment (CCE) Table 3. Cray Compiler Basics Module: PrgEnv-cray Command: ftn, cc, CC Compiler-speci?c man pages: crayftn(1), craycc(1), crayCC(1) Note: Compiler-speci?c man pages are available only when the compiler module is loaded. Online help: None provided Documentation: Cray Fortran Reference Manual, Cray C and C++ Reference Manual 4.3.1 Known Limitations If you use the Cray Fortran compiler with the PETSc (Portable, Extensible Toolkit for Scienti?c Computation) library, either add the directive !dir$ PREPROCESS EXPAND_MACROS to the source code or add the -F option to the ftn command line. S–2396–60 39Cray Application Developer's Environment User's Guide 4.4 About PGI Compilers Table 4. PGI Compiler Basics Module: PrgEnv-pgi Command: ftn, cc, CC Compiler-speci?c man pages: pgf95(1), pgcc(1), pgCC(1) Note: Compiler-speci?c man pages are available only when the compiler module is loaded. Online help: pgf95 -help, pgcc -help, pgCC -help, Documentation: /opt/pgi/version/linux86-64/version/doc 4.4.1 Known Limitations • At this time, PGI compilers do not produce code optimized for AMD Interlagos processors. Code compiled with the PGI compilers can be expected to suffer performance degradation if executed on compute nodes with Interlagos CPUs. This limitation is expected to be removed in the near future. • The PGI compilers are not able to handle template-based libraries such as Tpetra. • When linking in ACML routines, you must compile and link all program units with -Mcache_align or an aggregate option that incorporates -Mcache_align such as fastsse. • The -Mconcur (auto-concurrentization of loops) option is not supported on Cray systems. • The -mprof=mpi, -Mmpi, and -Mscalapack options are not supported. • The PGI debugger, PGDBG, is not supported on Cray systems. • The PGI pro?ling tools, pgprof and pgcollect, are not supported on Cray systems. • The PGI Compiler Suite does not support the UPC or Coarray Fortran parallel programming models. 40 S–2396–60Using Compilers [4] 4.5 About Intel Compilers Table 5. Intel Composer Basics Module: PrgEnv-intel Command: ftn, cc, CC Compiler-speci?c man pages: ifort(1), fpp(1), icc(1), icpc(1) Note: Compiler-speci?c man pages are available only when the compiler module is loaded. Online help: ifort --help, icc --help Documentation: /opt/intel/Compiler/version/Documentation/language 4.5.1 Known Limitations • The Intel Compiler Suite (Intel Composer) must be installed in the default location. The optional Intel C/C++ only installation is not supported because the Intel Fortran run time libraries are required by Cray libraries such as libsci when using the Intel compiler. • The Intel Composer does not support the UPC or Coarray Fortran parallel programming models. 4.6 About PathScale Compilers Table 6. PathScale Compiler Basics Module: PrgEnv-pathscale Command: ftn, cc, CC Compiler-speci?c man pages: pathf95(1), pathf90(1), pathcc(1), pathCC(1), eko(7) Note: Compiler-speci?c man pages are available only when the compiler module is loaded. Online help: pathf95 --help, pathf90 --help, pathcc --help, pathCC --help Documentation: /opt/pathscale/share/doc/version 4.6.1 Known Limitations • The PathScale Compiler Suite does not support the UPC or Coarray Fortran parallel programming models. S–2396–60 41Cray Application Developer's Environment User's Guide • The PETSc 3.1.0.8 library of parallel linear and nonlinear solvers does not support the PathScale Compiler Suite. Use PETSc 3.0.x instead. • The TPSL 1.1.00 library of third-party scienti?c libraries does not support the PathScale Compiler Suite. • Beginning the PathScale release 4.0, Cray no longer provides a PathScale-speci?c version of the Cray Scienti?c Libraries. Users are advised to obtain third-party scienti?c libraries directly from the respective library developers. 4.7 About GNU Compilers Table 7. GNU Compiler Basics Module: PrgEnv-gnu Command: ftn, cc, CC Compiler-speci?c man pages: gfortran(1), gcc(1), g++(1) Note: Compiler-speci?c man pages are available only when the compiler module is loaded. Online help: gfortran --help, gcc --help, g++ --help 4.7.1 Known Limitations The GNU compilers do not support the UPC or Coarray Fortran parallel programming models. 4.7.2 Known Warnings Code compiled with gfortran using the options --whole-archive,-lpthread gets the following warning message issued by libpthread.a(sem_open.o): warning: the use of 'mktemp' is dangerous, better use 'mkstemp' The --whole-archive option is necessary to avoid a runtime segmentation fault when using OpenMP libraries. This warning can be safely ignored. 42 S–2396–60Using Compilers [4] 4.8 About the Chapel Parallel Programming Language Chapel is a new parallel programming language being developed by Cray Inc. as part of the DARPA-led High Productivity Computing Systems program (HPCS). Chapel is designed to improve the productivity of high-end computer users while also serving as a portable parallel programming model that can be used on commodity clusters or desktop multicore systems. Chapel strives to vastly improve the programmability of large-scale parallel computers while matching or beating the performance and portability of current programming models like MPI. Chapel supports a multithreaded execution model via high-level abstractions for data parallelism, task parallelism, concurrency, and nested parallelism. Chapel's locale type enables users to specify and reason about the placement of data and tasks on a target architecture in order to tune for locality. Chapel supports global-view data aggregates with user-de?ned implementations, permitting operations on distributed data structures to be expressed in a natural manner. In contrast to many previous higher-level parallel languages, Chapel is designed around a multiresolution philosophy, permitting users to initially write very abstract code and then incrementally add more detail until they are as close to the machine as their needs require. Chapel supports code reuse and rapid prototyping via object-oriented design, type inference, and features for generic programming. Chapel was designed from ?rst principles rather than by extending an existing language. It is an imperative block-structured language, designed to be easy to learn for users of C, C++, Fortran, Java, Perl, Matlab, and other popular languages. While Chapel builds on concepts and syntax from many previous languages, its parallel features are most directly in?uenced by ZPL, High-Performance Fortran (HPF), and the Cray MTA extensions to C and Fortran. For more information about Chapel, see: http://chapel.cray.com. 4.9 About Cross-compilers The Cray system supports using standalone Linux workstations as code development platforms. When the Cray Application Developer's Environment (CADE) is installed on a suitable Linux system, programs can be written and compiled on the Linux system and then exported to the Cray system for subsequent execution, debugging, and optimization. S–2396–60 43Cray Application Developer's Environment User's Guide You cannot simply install CADE on any Linux system, however. Before you begin, note the following considerations. • The Linux system must meet the minimum hardware requirements listed in the Cray Application Developer's Environment Installation Guide. • The Linux system must be running a version of the SLES operating system corresponding to the Cray system for which you are developing code. If the Cray system is running CLE 2.2, the Linux system must be running SLES 10. If the Cray system is running CLE 3.0 or later, the Linux system must be running SLES 11. If you attempt to cross operating system versions, your code will compile successfully on the Linux system but not run on the Cray system. • The behavior of CADE when installed on Linux distributions other than SLES 10 or SLES 11 is untested and therefore unsupported. • With the exception of the GNU compilers, all compilers are licensed and installed separately. See your site administrator or compiler vendor for further information about license management. • Along with CADE, you must install the Cray Application Developer's Environment Supplement (CADES, or sometimes CADEsup), which requires satisfying a number of dependencies that are contingent on the compilers you are installing. These dependencies are described in the Cray Application Developer's Environment Installation Guide. • You must install and con?gure modules on the Linux system. After CADE and your compilers are installed and con?gured on your Linux system, follow these steps to begin developing code. 1. Load the xtpe-network module corresponding to the Cray system for which you intend to develop code. If you are developing for a Gemini system (Cray XE5, Cray XE6, or Cray XK6), load the xtpe-network-gemini module. If you are developing for a SeaStar system (all others), load the xtpe-network-seastar module. Note: On the actual Cray system, the network type is typically set by default and transparent to the user. When working on a standalone Linux system, you must set this manually. If the correct xtpe-network module is not selected, your code may appear to compile successfully on the Linux system but will not run on the Cray system. 44 S–2396–60Using Compilers [4] 2. Load the xtpe-processor module corresponding to the compute nodes on the Cray system for which you intend to develop code. Your choices are: • xtpe-barcelona (quad core, Cray XT4) • xtpe-shanghai (quad core, Cray XT5 or Cray XE5) • xtpe-istanbul (six cores) • xtpe-mc8 (eight cores) • xtpe-mc12 (twelve cores) • xtpe-interlagos (sixteen cores) Note: On the actual Cray system, the processor type is typically set by default and transparent to the user. When working on a standalone Linux system, you must set this manually. If you select a processor type with fewer cores than are actually present on the Cray compute nodes, your code will not make full use of the Cray system resources and may either run slowly or cause con?icts with aprun options and placement. If you select a processor type with more cores than are actually present on the Cray compute nodes, your application may appear to compile successfully but will not run. 3. (Cray XK system users only) Load the xtpe-accelerator module that is appropriate for the compute nodes on the Cray system for which you intend to develop code. Currently, the only supported option is xtpe-accel-nvidia20. Note: The accelerator target is not set by default on either the standalone Linux system or the Cray system. You must set this manually. This module sets compiler options required to compile applications for the accelerator target and loads CUDA. 4. Load the PrgEnv-vendor module containing your compiler of choice. You are now ready to begin writing, editing, and compiling code. Remember, however, that you must move your code to mount point on the Cray system (for example, /lus/nid00008) before you can run, debug, or optimize it. S–2396–60 45Cray Application Developer's Environment User's Guide 46 S–2396–60Libraries [5] Cray provides a large variety of libraries to support application development and interprocess communications on Cray systems. New libraries are being ported to the Cray system on an ongoing basis. The following libraries can be used with all compilers currently supported on the Cray system, except where noted in Chapter 4, Using Compilers on page 37. 5.1 Basic Scienti?c Libraries (LibSci) Cray LibSci is a collection of numerical routines optimized for best performance on Cray systems. All programming environment modules load xt-libsci by default, except where noted. When possible, you should use calls to the Cray LibSci routines in your code in place of calls to public-domain or user-written versions. 5.1.1 New in LibSci Release 11.0 Beginning with Scienti?c Libraries Release 11.0.0, the way in which dynamic libraries are found at execution time is changed. In previous releases, RPATH was included in the dynamic executable, with a path to the libraries to be used at execution time. Beginning with release 11.0.0, RPATH is no longer used by default. Instead, at execution time, dynamic libraries are found in /opt/cray/lib64. The version de?ned by the xt-libsci module is not used. This change is not expected to have noticeable consequences for most users, and is intended to improve overall system performance when using dynamic linking at high PE counts. However, in the event that this change does appear to have undesirable consequences, users can restore the previous dynamic linking behavior either by setting the environment variable CRAY_ADD_RPATH to yes, or else by swapping to an xt-libsci module earlier than release 11.0.0. Statically linked applications are not affected by this change. S–2396–60 47Cray Application Developer's Environment User's Guide 5.1.2 Basic LibSci Components Table 8. LibSci Basics Module: xt-libsci Man pages: intro_libsci(3s), intro_blas1(3s), intro_blas2(3s), intro_blas3(3s), intro_blacs(3s), intro_lapack(3s), intro_scalapack(3s), intro_irt(3), intro_case(3s)intro_crafft(3s), intro_fft(3s), intro_fftw2(3), intro_fftw3(3) Note: Library-speci?c man pages are available only when the associated module is loaded. The Cray LibSci collection contains the following Scienti?c Libraries. • BLAS (Basic Linear Algebra Subroutines) • BLACS (Basic Linear Algebra Communication Subprograms) • LAPACK (Linear Algebra Routines) • ScaLAPACK (Scalable LAPACK) • FFT (Fast Fourier Transform Routines) • FFTW2 (the Fastest Fourier Transforms in the West, release 2) • FFTW3 (the Fastest Fourier Transforms in the West, release 3) In addition, the Cray LibSci collection contains three libraries developed by Cray. • IRT (Iterative Re?nement Toolkit) • CASE (Cray Adaptive Sparse Eigensolvers) • CRAFFT (Cray Adaptive Fast Fourier Transform Routines) 5.1.3 BLAS and LAPACK The BLAS (Basic Linear Algebra Subroutines) library contains three levels of optimized subroutines. Level 1 BLAS perform the following types of basic vector-vector operations: • Dot products and various vector norms • Scaling, copying, swapping, and computing linear combination of vector • Generate or apply plane or modi?ed plane rotations 48 S–2396–60Libraries [5] For more information, see the intro_blas1(3s) man page. Level 2 BLAS perform matrix-vector operations, and generally produce improved code performance when inlined. For more information about Level 2 BLAS, see the intro_blas2(3s) man page. Level 3 BLAS perform matrix-matrix operations. For more information about Level 3 BLAS, see the intro_blas3(3s) man page. LAPACK is a public domain library of subroutines for solving dense linear algebra problems, including the following: • Systems of linear equations • Linear least squares problems • Eigenvalue problems • Singular value decomposition (SVD) problems LAPACK is the successor to the older LINPACK and EISPACK packages. It extends the functionality of these packages by including equilibration, iterative re?nement, error bounds, and driver routines for linear systems, routines for computing and reordering the Schur factorization, and condition estimation routines for eigenvalue problems. Performance issues are addressed by implementing the most computationally-intensive algorithms using Level 2 and Level 3 BLAS. For more information about LAPACK, see the intro_lapack(3s) man page. 5.1.3.1 Notes • The BLAS and LAPACK libraries include routines from the 64-bit libGoto library from the University of Texas. • BLAS library behavior is dependent on the xtpe-processor module. At most sites this module is typically loaded by default and transparent to the user. However, if your site has multiple types of compute nodes, or if you are working in a Linux cross-compiling environment, it may be necessary to load the xtpe-processor module corresponding to the compute nodes on the Cray system for which you intend to develop code in order to obtain best performance. Your choices are: – xtpe-barcelona (quad core, Cray XT4) – xtpe-shanghai (quad core, Cray XT5) – xtpe-istanbul (six cores) – xtpe-mc8 (eight cores) – xtpe-mc12 (twelve cores) S–2396–60 49Cray Application Developer's Environment User's Guide – xtpe-interlagos (sixteen cores) Note: If you select a processor type with fewer cores than are actually present on the Cray compute nodes, your code will not make full use of the Cray system resources and may either run slowly or cause con?icts with aprun options and placement. If you select a processor type with more cores than are actually present on the Cray compute nodes, your application may appear to compile successfully but will not run. • If you require a C interface to BLAS and LAPACK but want to use Cray LibSci BLAS or LAPACK routines, use the Fortran interfaces. • To obtain threading behavior, set OMP_NUM_THREADS, as described in BLACS and ScaLAPACK on page 50. • You can access the Fortran interfaces from a C program by adding an underscore to the respective routine names and passing arguments by reference (rather than by value). For example, you can call the dgetrf() function as follows: dgetrf_(&uplo, &m, &n, a, &lda, ipiv, work, &lwork, &info); • C programmers using the Fortran interface must order arrays in Fortran column-major order. 5.1.4 BLACS and ScaLAPACK The BLACS (Basic Linear Algebra Communication Subprograms) library is a package of routines that provide the same functionality for message-passing linear algebra communication as the Basic Linear Algebra Subprograms (BLAS) provide for linear algebra computation. With these two packages, software for dense linear algebra can use calls to BLAS for computation and calls to BLACS for communication. The BLACS consist of communication primitives routines, global reduction routines, and support routines. For more information about BLACS, see the intro_blacs(3s) man page. The ScaLAPACK (Scalable LAPACK) library uses BLACS primitives to provide optimized routines for solving real or complex general, triangular, or positive de?nite distributed systems; for reducing distributed matrices to condensed form and an eigenvalue problem solver for real symmetric distributed matrices; and to perform basic operations involving distributed matrices and vectors. LU and Cholesky routines in ScaLAPACK have been modi?ed to allow the user to choose an underlying broadcast algorithm during run-time. It can be done either via an environment variable, or by calling a helper routine in a program. For more information about ScaLAPACK, see the intro_scalapack(3s) man page. 50 S–2396–60Libraries [5] 5.1.4.1 Notes • Some ScaLAPACK routines require the Basic Linear Algebra Communication Subprograms (BLACS) to be initialized. This can be done through a call to BLACS_GRIDINIT. Also, each distributed array that is passed as an argument to a ScaLAPACK routine requires a descriptor, which is set through a call to DESCINIT. • The ScaLAPACK and BLACS libraries can be used in MPI and SHMEM applications. Cray LibSci also supports hybrid MPI/ScaLAPACK applications, which use threaded BLAS on a compute node and MPI between nodes. To use ScaLAPACK in a hybrid application: 1. Adjust the process grid dimensions in ScaLAPACK to account for the decrease in BLACS nodes. 2. Ensure that the number of BLACS processes required is equal to the number of nodes required, not the number of cores. 3. Set the OMP_NUM_THREADS environment variable. 5.1.5 Iterative Re?nement Toolkit (IRT) The Iterative Re?nement Toolkit (IRT) is a library of Fortran subroutines that provides solutions to linear systems using 32-bit factorizations while preserving accuracy through mixed-precision iterative re?nement. IRT exploits the fact that single-precision solvers can be up to twice as fast as double-precision solvers, and uses an iterative re?nement process to obtain solutions accurate to double-precision. IRT includes both serial and parallel implementations of the LU and Cholesky algorithms, and serial versions of the QR algorithm for real and complex matrices. IRT includes the following features: • Sophisticated stopping criteria • Potential minimization of forward error • Ability to return error bounds • Return an estimate of the condition number of matrix A • Return to the double-precision factorization-and-solve process if IRT cannot obtain a solution S–2396–60 51Cray Application Developer's Environment User's Guide IRT provides two interfaces: • Benchmarking interface. The benchmarking interface routines replace the high-level drivers of LAPACK and ScaLAPACK. The names of the benchmark API routines are identical to their LAPACK or ScaLAPACK counterparts or replace calls to successive factorization and solver routines. This allows you to use the IRT process without modifying your application. For example, the IRT dgesv() routine replaces either the LAPACK dgesv() routine or the LAPACK dgetrf() and dgetrs() routines. To use the benchmarking interface, set the IRT_USE_SOLVERS environment variable to 1. Note: Use this interface with caution; calls to the LAPACK LU, QR or Cholesky routines are intercepted and the IRT is used instead. • Expert interface. The expert interface routines give you greater control of the iterative re?nement process and provide details about the success or failure of the process. The format of advanced API calls is: call irt_factorization-method_data-type_processing-mode(arguments) such as: call irt_po_real_parallel(arguments). For more information about IRT, see the intro_irt(3) man page. 5.1.6 Cray Adaptive Simple Eigensolvers (CASE) CASE is a collection of simpli?ed interfaces into high-performance LAPACK- and ScaLAPACK-style routines that calculate the eigenvalues and eigenvectors of a symmetric or hermitian matrix. CASE is written in Fortran but has interfaces for C/C++ users. CASE is designed for C/C++ or Fortran users who are looking for a fast and simple way to calculate the eigenvalues and/or eigenvectors of a matrix and would otherwise call ScaLAPACK or LAPACK eigensolver routines. CASE does this by providing an easy interface for calling the ScaLAPACK and LAPACK eigensolvers that requires only a few simple arguments. From these arguments, CASE interfaces construct the sometimes complicated calling sequences and work arrays required when calling LAPACK and ScaLAPACK eigensolvers directly. CASE is provided for serial or parallel problems, with single- or double-precision real or complex data types. For more information about CASE, see the intro_case(3s) man page. 5.1.7 Fourier Transformations Fast Fourier transforms are handled either by using CRAFFT and FFTW or by using ACML. The intro_fft(3s) man page is a disambiguation page. 52 S–2396–60Libraries [5] 5.1.7.1 CRAFFT CRAFFT is a library of Fortran subroutines that compute the discrete Fourier transform in one, two, or three dimensions; of arbitrary input size; and of both real and complex data. CRAFFT provides a simpli?ed interface to several FFT libraries and allows automatic, dynamic selection of the fastest FFT kernel. CRAFFT also provides routines for two- and three-dimensional distributed FFT. Note: CRAFFT requires that module fftw/3.2.0 or higher be loaded. To use CRAFFT, add a Fortran use crafft statement to any source ?le that calls a CRAFFT routine, and then call crafft_init() before any other CRAFFT routine. This sets up the CRAFFT library for run-time use. You have a choice of how much planning of FFT kernels you wish to perform. You can set the CRAFFT_PLANNER environment variable to 0, 1 or 2 before execution. Alternately, you can use the crafft_set_planner() and crafft_get_planner() subroutines to alter and query, respectively, the value of CRAFFT_PLANNER during program execution. For more information about using CRAFFT, see the intro_crafft(3s) man page. 5.1.7.2 FFTW The Programming Environment includes versions 2.1.5 and 3.2.x of the Fastest Fourier Transform in the West (FFTW) library. FFTW is a C subroutine library with Fortran interfaces for computing the discrete Fourier transform in one or more dimensions, of arbitrary input size, and of both real and complex data (as well as of even/odd data, such as the discrete cosine/sine transforms). The Fast Fourier Transform algorithm is applied for many problem sizes. Up through LibSci version 10.3.2, loading the LibsSci module automatically loaded the fftw/3.1.1 module. Beginning with xt-libsci modules 10.3.3, FFTW is no longer loaded automatically when you load the LibSci module. Distributed-memory parallel FFTs are available only in FFTW 2.1.5.1. The FFTW 3.2.x and FFTW 2.1.5.1 modules cannot be loaded at the same time. You must ?rst unload the other module, if already loaded, before loading the desired one. For example, if you have loaded the FFTW 3.2.x library and want to use FFTW 2.1.5.1 instead, use: % module swap fftw/3.2.2.1 fftw/2.1.5.1 For more information about FFTW, see the intro_fftw2(3) and intro_fftw3(3) man pages. S–2396–60 53Cray Application Developer's Environment User's Guide 5.1.7.3 ACML The AMD Core Math Library (ACML) module is no longer loaded as part of the default PrgEnv environment. BLAS and LAPACK functionality is now provided by Cray LibSci. However, if you need ACML for FFT functions, math functions, or random number generators, you can load the library using the acml module: % module load acml ACML includes: • A suite of Fast Fourier Transform (FFT) routines for real and complex data • Fast scalar, vector, and array math transcendental library routines optimized for high performance • A comprehensive random number generator suite: – Base generators plus a user-de?ned generator – Distribution generators – Multiple-stream support ACML's internal timing facility uses the clock() function. If you run an application on compute nodes that uses the plan feature of FFTs, underlying timings will be done using the native version of clock(). On CNL, clock() returns the sum of user and system CPU times. 5.2 Fast Math Intrinsics Table 9. Fast_mv Basics Module: libfast Man pages: intro_fast_mv(3), Note: Library-speci?c man pages are available only when the associated module is loaded. Fast_mv is a library of high-performance math intrinsic functions. The functions can be used in PGI, CCE, GNU, and PathScale applications. You can use these functions without changing your programs, or you can call them directly. Note: The use of Fast_mv with Intel compilers is not currently supported. To use Fast_mv functions, load the libfast module: % module load libfast 54 S–2396–60Libraries [5] Currently, the library contains the following functions. Table 10. Fast_mv Math Intrinsics Function Name Description cos() 64-bit cosine function exp() 64-bit exponential function (the constant e raised to a power) expf() 32-bit exponential function (the constant e raised to a power) fastcos() scalar 64-bit cosine fastsin() scalar 64-bit sine fastsincos() scalar 64-bit sine and cosine frda_exp() 64-bit exponential function for all elements in an array frda_log() array version of the 64-bit base e logarithm frsa_expf() 32-bit exponential function for all elements in an array frsa_logf() array version of the 32-bit base e logarithm log() 64-bit logarithm (base e) function logf() 32-bit logarithm (base e) function sin() 64-bit sine function sincos() 64-bit sine and cosine function vrda_log() array version of the 64-bit base e logarithm __vrd4_log() four-element version of the 64-bit base e logarithm vrsa_logf() array version of the 32-bit base e logarithm For more information, see the intro_fast_mv(3) man page. S–2396–60 55Cray Application Developer's Environment User's Guide 5.3 PETSc Table 11. PETSc Basics Modules: petsc, petsc-complex, tpsl Man pages: intro_petsc(3s) Note: Library-speci?c man pages are available only when the associated module is loaded. Website: http://www.mcs.anl.gov/petsc/petsc-as/ PETSc (Portable, Extensible, Toolkit for Scienti?c Computation) is an open source library of parallel linear and nonlinear equation solvers intended for use in large-scale C, C++, or Fortran applications. PETSc uses standard MPI functions for all message-passing communication. The PETSc module is dependent on the xt-libsci and xt-asyncpe modules. Make certain these modules are loaded before using PETSc. When you load the petsc module, the Third-party Scienti?c Libraries (tpsl) module is automatically loaded as well, to provide access to the libraries required to support PETSc. Note: Always use the tpsl module that is linked to the PETSc module. PETSc and Trilinos are asynchronous products and may at times use different versions of the TPSL libraries. PETSc provides many of the mechanisms needed for parallel applications, such as simple parallel matrix and vector assembly routines that allow the overlap of communication and computation. In addition, PETSc includes support for parallel distributed arrays useful for ?nite difference methods, such as: • Parallel vectors, including code for communicating ghost points • Parallel matrices, including several sparse storage formats • Scalable parallel preconditioners • Krylov subspace methods • Parallel Newton-based nonlinear solvers • Parallel time-stepping ordinary differential equation (ODE) solvers 56 S–2396–60Libraries [5] The following packages are included in PETSc/TPSL. • MUMPS 4.9.2. MUMPS (MUltifrontal Massively Parallel sparse direct Solver) is a package of parallel, sparse, direct linear-system solvers based on a multifrontal algorithm. For further information, see http://graal.ens-lyon.fr/MUMPS/. • SuperLU 4.1. SuperLU is a sequential version of SuperLU_dist (not included with petsc-complex), and a sequential incomplete LU preconditioner that can accelerate the convergence of Krylov subspace interative solvers. For further information, see http://crd.lbl.gov/~xiaoye/SuperLU/. • SuperLU_dist 2.5. SuperLU_dist is a package of parallel, sparse, direct linear-system solvers (available in Cray LibSci). For further information, see http://crd.lbl.gov/~xiaoye/SuperLU/. • ParMETIS 3.2. ParMETIS (Parallel Graph Partitioning and Fill-reducing Matrix Ordering) is a library of routines that partition unstructured graphs and meshes and compute ?ll-reducing orderings of sparse matrices. For further information, see http://glaros.dtc.umn.edu/gkhome/views/metis/. • HYPRE 2.7.0. HYPRE is a library of high-performance preconditioners that use parallel multigrid methods for both structured and unstructured grid problems (not included with petsc-complex). For further information, see http://www.llnl.gov/CASC/linear_solvers/. • SUNDIALS 2.4.0 (SUite of Nonlinear and DIfferential/ALgebraic equation Solvers) consists of 5 solvers: CVODE, CVODES, IDA, IDAS, and KINSOL. In addition, SUNDIALS provides a MATLAB interface to CVODES, IDAS, and KINSOL that is called sundialsTB. For further information, see https://computation.llnl.gov/casc/sundials/main.html. • Scotch 5.1.1. Scotch is a software package and libraries for sequential and parallel graph partitioning, static mapping, sparse matrix block ordering, and sequential mesh and hypergraph partitioning. For further information, see http://www.labri.fr/perso/pelegrin/scotch/. Note: Although you can access these packages individually, Cray supports their use only through the PETSc interface. 5.3.1 Notes • If you use PETSc with the Cray Fortran compiler, either add the directive !dir$ PREPROCESS EXPAND_MACROS to the source code or add the -F option to the ftn command line. • The PathScale Compiler Suite does not support PETSc 3.1.0.8. If you are using the PathScale compilers, use PETSc 3.0. • The solvers in Cray PETSc 3.1 are heavily optimized using the Cray Adaptive Sparse Kernels (CASK) library. CASK is an auto-tuned library within the Cray S–2396–60 57Cray Application Developer's Environment User's Guide PETSc package that is transparent to the application developer, but improves the performance of most PETSc iterative solvers. You can expect the largest performance improvements when using blocked matrices (BAIJ or SBAIJ), but may also see large gains when using standard compressed sparse row (CSR) AIJ PETSc matrices. 5.4 Trilinos Table 12. Trilinos Basics Modules: trilinos, tpsl Man pages: intro_trilinos(1), Note: Library-speci?c man pages are available only when the associated module is loaded. Website: http://trilinos.sandia.gov/ Trilinos is a separate module, comparable to PETSc, that provides abstract, object-oriented interfaces to established libraries such as Metis/ParMetis, SuperLU, Aztec, BLAS, and LAPACK. Trilinos also includes a set of Cray Adaptive Sparse Kernels (CASK) that perform SpMV, and include optimized versions of single- and multiple-vector matrix vector multiplies. The Trilinos module is dependent on the xt-libsci and xt-asyncpe modules. Make certain these modules are loaded before using Trilinos. When you load the trilinos module, the Third-party Scienti?c Libraries (tpsl) module is automatically loaded as well, to provide access to the libraries required to support Trilinos. Note: Always use the tpsl module that is linked to the Trilinos module. Trilinos and PETSc are asynchronous products and may at times use different versions of the TPSL libraries. To use the Trilinos packages, load your compiling environment of choice, and then load the Trilinos module. % module load trilinos After you load the Trilinos module, all header and library locations are set automatically and you are ready to compile your code. No Trilinos-speci?c linking information is required on the command line. If linking to more than one Trilinos package, the libraries are linked automatically in the correct order of package dependency. For more information about link order, see see http://trilinos.sandia.gov/packages/interoperability.html. 58 S–2396–60Libraries [5] 5.5 MPT Table 13. MPT Basics Modules: xt-mpich2, xt-shmem, xt-mpt (deprecated) Man pages: intro_mpi(3), intro_shmem(3) Note: Library-speci?c man pages are available only when the associated module is loaded. The Cray Message Passing Toolkit (MPT) consists of two components. • MPI • SHMEM Support for MPI and SHMEM varies signi?cantly depending on whether you are using a Cray XE or Cray XK system with Gemini interconnect or a Cray XT system with SeaStar interconnect, and whether your code uses static or dynamic libraries. Because of these issues, Cray provides a variety of different MPI and SHMEM modules. These modules are hardware-dependent, so your system administrator should install only the modules that support your hardware on your system. Nonetheless, there are very signi?cant differences in the ways that MPI and SHMEM are implemented on Cray XE/Cray XK and Cray XT systems. These differences are re?ected in the functions that are available, and most visibly in the environment variables available on the two types of systems. To see which functions and environment variables are supported on your system, always check the intro_mpi(3) or intro_shmem(3) man pages. 5.5.1 Using MPI and SHMEM Modules Cray has changed the way that MPI and SHMEM are supported and the default module con?gurations. These changes were implemented in different releases of CLE and affect your usage of the MPT-related modules. 5.5.1.1 CLE 4.0 Systems No MPT-related modules are loaded by default. • If your code uses MPI code only, load the xt-mpich2 module before compiling or linking. This ensures that your code is linked using the -lmpich option. • If your code uses SHMEM code only, load the xt-shmem module before compiling or linking. This ensures that your code is linked using the -lsma option. S–2396–60 59Cray Application Developer's Environment User's Guide • If your code uses both MPI and SHMEM, load both the xt-mpich2 and xt-shmem modules. Your code will be linked using both the -lmpich and -lsma options. • There is no need to differentiate between code that uses static libraries and code that uses dynamic or shared libraries when selecting and using MPI or SHMEM modules. The xt-mpt module is provided to support legacy code and make ?les but is deprecated and planned to be removed in a future release. 5.5.1.2 CLE 3.1 Systems By default, all Cray programming environment modules (PrgEnv-compiler) load the xt-mpich2 module. This module supports MPI code only. • If your code uses MPI code only, verify that the xt-mpich2 module is loaded before compiling or linking. This ensures that your code is linked using the -lmpich option. • If your code uses SHMEM code only, unload the xt-mpich2 module and load the xt-shmem module before compiling or linking. This ensures that your code is linked using the -lsma option. • If your code uses both MPI and SHMEM, verify that both the xt-mpich2 and xt-shmem modules are loaded. Your code will be linked using both the -lmpich and -lsma options. • There is no need to differentiate between code that uses static libraries and code that uses dynamic or shared libraries when selecting and using MPI or SHMEM modules. The xt-mpt module is provided to support legacy code and make ?les but is deprecated and planned to be removed in a future release. 5.5.1.3 CLE 3.0 Systems By default, all Cray programming environment modules load the xt-mpt module. The xt-mpt module supports both MPI and SHMEM code, but causes the compiler drivers to link code using both the -lmpich and -lsma options. This can cause undesirable run time dependencies when using dynamic or shared libraries. • If your code uses static libraries only, use the xt-mpt module. There is no need to differentiate between MPI and SHMEM code. • If your code uses dynamic libraries and MPI code only, unload the xt-mpt module and load the xt-mpich2 module before compiling or linking. This ensures that your code is linked using the -lmpich option. 60 S–2396–60Libraries [5] • If your code uses dynamic libraries and SHMEM code only, unload the xt-mpt module and load the xt-shmem module before compiling or linking. This ensures that your code is linked using the -lsma option. • If your code uses dynamic libraries and both MPI and SHMEM, unload the xt-mpt module and load both the xt-mpich2 and xt-shmem modules. Your code will be linked using both the -lmpich and -lsma options. 5.5.1.4 CLE 2.2 Systems By default, all Cray programming environment modules load the xt-mpt module. Depending on which version of MPT is installed on your system, the xt-mpich2 and xt-shmem modules may not be available. Be advised that if the xt-mpt module is loaded, the compiler drivers will link code using both the -lmpich and -lsma options. This can cause undesirable run time dependencies when using dynamic or shared libraries. 5.6 Hugepages Hugepages are virtual memory pages which are bigger than the default base page size of 4KB. Huge pages can improve memory performance for common access patterns on large data sets. Access to huge pages is provided through a virtual ?le system called hugetlbfs. Every ?le on this ?le system is backed by huge pages and is directly accessed with mmap() or read(). The libhugetlbfs library allows an application to use huge pages more easily than it could by directly accessing the hugetlbfs ?le system. A user may use libhugetlbfs to back application text and data segments. Due to differing memory management mechanisms on Cray XT and Cray XE systems, the implementation of the libhugetlbfs library differs on these two architectures. 5.6.1 Cray XT Usage Nodes do not have huge pages allocated by default. To use hugepages, link an application with the libhugetlbfs library. At run-time, de?ne the environment variable HUGETLB_MORECORE=yes. The application launcher, aprun, must be told that a given application wants to use huge pages. Specify a per-PE huge page memory requirement on the aprun invocation line using the [-m size[h|hs] ] option. For more information see the aprun(1) man page, and Workload Management and Application Placement for the Cray Linux Environment (S–2496). S–2396–60 61Cray Application Developer's Environment User's Guide 5.6.2 Cray XE Usage By default, the running system is con?gured to have huge pages available. Pre-allocated huge pages are reserved inside the kernel and cannot be used for other purposes. On Cray XE systems, PGAS, SHMEM and MPI applications are likely to require the usage of huge pages for static data and/or the heap. Modules craype-hugepages2m and craype-hugepages8m (released in xt_asyncpe 4.8) set the necessary link options and environment variables (e.g. HUGETLB_DEFAULT_PAGE_SIZE, HUGETLB_MORECORE, HUGETLB_ELFMAP) to facilitate the usage of 2 or 8 MB huge pages, respectively. It is not required to use the -m option on the aprun command on the Cray XE system to allocate huge pages, because the kernel allows the dynamic creation of huge pages. However, it is advisable to specify this option and preallocate an appropriate number of huge pages, when memory requirements are known, to reduce operating system overhead. For more detailed information and examples see the intro_hugepages(1), and aprun(1) man pages, and Workload Management and Application Placement for the Cray Linux Environment (S–2496). 62 S–2396–60Debugging Code [6] The Cray system supports a variety of debugging options ranging from simple command-line debuggers to separately licensed third-party GUI tools and capable of performing a variety of tasks ranging from analyzing core ?les to setting breakpoints and debugging running parallel programs. As a rule, your code must be compiled using the -g command line option before you can use any of the debuggers to produce meaningful information. However, if you are using both a compiler and a debugger that support Fast-Track Debugging, the -g option is replaced by using the -G fast option. Note: The PGI debugger, PGDBG, is not supported on Cray systems. 6.1 Cray Debugger Support Tools Cray provides a collection of basic debugging packages that are referred to collectively as the Cray Debugger Support Tools and installed as a single rpm, but loaded and used as individual modules. These packages are: • lgdb: a modi?ed version of gdb that interfaces with aprun and works on Cray system compute nodes • atp: a system that monitors user applications and replaces the core dump with a more comprehensive stack backtrace and analysis • MRNet: a software overlay network that provides multicast and reduction communications for parallel and distributed tools and systems • STAT: stack trace analysis tool S–2396–60 63Cray Application Developer's Environment User's Guide 6.1.1 Using lgdb Table 14. lgdb Basics Module: xt-lgdb Command: lgdb Man page: lgdb(1) Note: Tool-speci?c man pages are available only when the associated module is loaded. Online help: lgdb --help The lgdb command is used to launch an application and related gdb server processes on remote nodes for debugging purposes. After lgdb is launched, it provides directions regarding how to run the gdb debugger and attach to application processes. When using lgdb on a Cray system, remember that you use lgdb to launch an instance of aprun, which in turn launches the application to be debugged. You cannot use lgdb to launch an application directly. Example 1. Launching and debugging a single-rank application $ lgdb --pes=0 --command="aprun -n1 ./myapp" --lib=/lib64/libthread_db.so.1 You must specify the number of PEs (ranks) to which you want to attach gdbservers. Since this is a single-rank application, specify 0 for the rank. The --lib=path_to_library is optional and needs to be speci?ed only for certain systems, such as X86_64 Linux, where the libthread_db library is required. Example 2. Attaching to an already-running application $ lgdb --pes=0 --pid=aprun_pid --lib=/lib64/libthread_db.so.1 Remember, you use lgdb to launch an instance of aprun, which in turn launches the application to be debugged. Therefore to attach to an already-running application, you must specify the pid of the instance of aprun that launched the application, not the application itself. For more information, see the lgdb(1) man page. 64 S–2396–60Debugging Code [6] 6.1.2 Using Abnormal Termination Processing (ATP) Table 15. atp Basics Module: atp Commands: ataprun, atpFrontend (deprecated) Man page: intro_atp(1) Note: Tool-speci?c man pages are available only when the associated module is loaded. Online help: None required Abnormal Termination Processing (ATP) monitors user applications. When the atp module is loaded and ATP is enabled, ATP is launched when a job is started and delivers a heuristically determined set of core ?les in the event of an application crash. If an application takes a system trap, ATP performs analysis on the dying application. All stack backtraces of the application processes are gathered into a merged stack backtrace tree and written to disk as the ?le, atpMergedBT.dot. The stack backtrace tree for the ?rst process to die is sent to stderr as is the number of the signal that caused the application to fail. The atpMergedBT.dot ?le can be viewed with statview, (the Stack Trace Analysis Tool viewer). The merged stack backtrace tree provides a concise yet comprehensive view of what the application was doing at the time of its termination. ATP is designed to analyze failing applications. It does not play any role with commands. That is, an application must use a supported parallel programming model, such as MPI, SHMEM, OpenMP, CAF, or UPC, in order to bene?t from ATP analysis. When the atp module is loaded, ATP sets the MPICH_ABORT_ON_ERROR, SHMEM_ABORT_ON_ERROR, and DMAPP_ABORT_ON_ERROR environment variables. This enables MPI, SHMEM, and DMAPP applications to raise a signal when they discover usage errors—rather than only printing to stderr and exiting—which therefore enables ATP to notice the problem and perform its analysis. Note: Using ATP disables core dumping. 6.1.2.1 On CLE 3.0 or Later On CLE 3.0 and later systems, when the atp module is loaded, ATP is launched automatically whenever a job is launched by using the aprun command. Do not use the atpaprun or atpFrontend commands on CLE 3.0 or later systems. These commands are for use on CLE 2.2 or earlier systems only. S–2396–60 65Cray Application Developer's Environment User's Guide 6.1.2.2 On CLE 2.2 or Earlier On CLE 2.2 and earlier systems, ATP must be invoked manually. This is done by loading the atp module and then using either the atpaprun command to launch your application, or after the application has been launched by using the atpFrontend command to attach to an already-running application. Example 3. Launching an application and invoking ATP atpaprun aprun_parameters Example 4. Attaching ATP to an already-running application atpFrontend application_PID For more information, see the intro_atp(1) man page. 6.1.3 Using MRNet MRNet is a customizable, high-throughput communication software system for parallel tools and applications with a master/slave architecture. MRNet reduces the cost of these tools' activities by incorporating a tree-based overlay network (TBON) of processes between the tool's front-end and back-ends. MRNet uses the TBON to distribute many important tool communication and computation activities, reducing analysis time and keeping tool front-end loads manageable. MRNet-based tools send data between front-end and back-ends on logical ?ows of data called streams. MRNet internal processes use ?lters to synchronize and aggregate data sent to the tool's front-end. Using ?lters to manipulate data in parallel as it passes through the network, MRNet can ef?ciently compute averages, sums, and other more complex aggregations on back-end data. For more information, see: http://www.paradyn.org/mrnet/. 6.1.4 Using STAT STAT (Stack Trace Analysis Tool) gathers and merges the stack traces from a parallel application's processes and produces 2D spatial and 3D spatial-temporal call graphs that encode the calling behavior of the application processes in the form of a pre?x tree. The 2D graph represents a single snapshot of the entire application, while the 3D form represents a series of snapshots from the application taken over time. For more information, see: http://www.paradyn.org/STAT/STAT.html. 66 S–2396–60Debugging Code [6] 6.2 Using Cray Fast-Track Debugging Normally code must be compiled using the -g option before it can be debugged using a conventional debugger. The -g option typically disables all compiler optimizations, producing an executable that contains full DWARF information and can be breakpointed, stepped-through, or paused and restarted anywhere, but at the cost of a much larger and far slower-running program. Cray Fast-Track Debugging signi?cantly increases the speed of the debugging process by producing executables that both contain full DWARF information and run at optimized-code speed. Essentially this is done by producing two parallel executables: one that is fully optimized, and another that is not. While this combined executable is considerably larger than a normal executable, when it is executed under the control of a debugger that supports fast-track debugging, it runs at optimized-code speed until it hits a break point—at which time it switches to the unoptimized code, and allows you to set breakpoints, examine registers, pause, resume, and step through the code as if the entire program was compiled using the -g option. As far as the debugger is concerned, there are no user interface changes, aside from the possibility that you might pursue a backtrace far enough back to begin seeing internal names instead of user names for variables. (For example, instead of foo, you might see debug$foo.) All the work involved in using fast-track debugging is done on the compiler side. Procedure 1. Using Cray Fast-Track Debugging Using Cray Fast-Track Debugging is a two-step process, requiring both a compiler and a debugger that support fast-track debugging. The steps are: 1. Compile your program using your compiler's fast-track debugging option. For example, if you are using the Cray Fortran compiler, compile and link your code using the -G fast option: users/yourname> ftn -Gfast myapp.f 2. Execute your program using your debugger's normal control method. For example, if you are using the lgdb command line debugger to debug a single-rank application: $ lgdb --pes=0 --command="aprun -n1 ./myapp" Once the debugging session is launched, fast-track debugging is transparent to the user. Sections of the code that contain breakpoints execute slowly, as if compiled using the -g option. Other sections of the code that do not contain breakpoints execute at normal speed, as if compiled using normal optimizations. 6.2.1 Supported Compilers and Debuggers At this time, Cray Fast-Track Debugging is supported on the front end by the Cray Compiling Environment (CCE) compilers. S–2396–60 67Cray Application Developer's Environment User's Guide On the back end, Cray Fast-Track Debugging is supported by the lgdb and Allinea DDT debuggers. 6.3 About Core Files When an application fails, one core ?le is generated for the ?rst failing process. If a ?le named core already exists in the current working directory, it is overwritten. On large MPP systems where an application might be running thousands of processes, a conventional core ?le may not indicate the actual cause of the failure. For this reason Cray systems support Abnormal Termination Processing (ATP). If ATP is enabled, a failing application generates a heuristically determined set of data, and in place of a core ?le, all stack backtraces are gathered into a merged stack backtrace tree and written to disk as the ?le, atpMergedBT.dot. The stack backtrace tree for the ?rst process to die is sent to stderr as is the number of the signal that caused the application to fail. For more information about ATP, see Using Abnormal Termination Processing (ATP) on page 65. 6.4 Using TotalView Table 16. TotalView Basics Module: xt-totalview Commands: totalview, totalviewcli Man page: totalview(1) Note: Tool-speci?c man pages are available only when the associated module is loaded. Online help: The TotalView GUI contains an extensive HTML online help system but requires that $TV_HTMLHELP_VIEWER be de?ned before use. For more information, see the Totalview documentation. Documentation: /opt/totalview/version/doc TotalView is an optional product from TotalView Technologies, LLC, that provides source-level debugging of applications running on multiple compute nodes. TotalView is compatible with the Cray, PGI, GCC, PathScale, and Intel compilers. 68 S–2396–60Debugging Code [6] TotalView can be launched in either or two modes: in GUI mode (using the totalview command), or in command-line mode (using the totalviewcli command). TotalView is typically run interactively. If your site has not designated any compute nodes for interactive processing, use the qsub -I command to reserve the number of compute nodes you want to use in interactive mode. Example 5. Using TotalView to control program execution To debug an application on the Cray system, use TotalView to launch aprun, which in turn launches the application to be debugged. users/yourname> totalview aprun -a [aprun_arguments] ./myapp [myapp_arguments] The -a option is a TotalView option indicating that the arguments that follow apply to aprun, not TotalView. Example 6. Debugging a core ?le To use TotalView to examine a core ?le, use the totalview command to launch the GUI. Then, in the New Program window, click the Open a core ?le button and use the browse functions to ?nd, select, and open the core ?le you want to examine. Example 7. Attaching TotalView to a running process To attach TotalView to a running process, you must be logged in to the same login node that you used to launch the process, and then you must attach to the instance of aprun that was used to launch the process, not the process itself. To do so: 1. Use the totalview command to launch the GUI. 2. In the New Program window, click the Attach to process button. The list of processes currently running displays. 3. Select the instance of aprun that you want, and click OK. TotalView displays a process window showing both aprun and the program threads that were launched by that instance of aprun. 6.4.1 Known Limitations The TotalView debugging suite for Cray systems differs in functionality from the standard TotalView implementation. It does not support: • Debugging MPI_Spawn(), OpenMP, or Cray SHMEM programs. • Compiled EVAL points and expressions. • Type transformations for the PGI C++ compiler standard template library collection classes. S–2396–60 69Cray Application Developer's Environment User's Guide • Exception handling for the PGI C++ compiler run time library. • Spawning a process onto the compute processors. • Machine partitioning schemes, gang scheduling, or batch systems. 6.5 Using DDT Table 17. DDT Basics Module: ddt Commands: ddt Man page: ddt(1) Note: Tool-speci?c man pages are available only when the associated module is loaded. Online help: The DDT GUI includes an extensive online help system accessible by selecting Help from the menu. If you have problems displaying the help, see the DDT User Guide for information about con?guring X-Windows forwarding and VNC connections. Documentation: /opt/cray/ddt/version/doc DDT is an optional product from Allinea Software, and is a scalable debugger with a graphical user interface. It can be used to debug Fortran, C, and C++ programs, including MPI and OpenMP code, and to launch and debug programs, attach to already running programs, or open and debug core ?les. DDT is compatible with the Cray, PGI, GCC, PathScale, and Intel compilers. The DDT GUI requires either X-Windows forwarding or VNC in order to work. DDT can be used either within an interactive shell or via the batch system. Submission through a batch system requires the use of template ?les that specify the batch parameters. Sample template ?les are found in /opt/cray/ddt/version/templates. In an interactive shell, the fastest way to launch DDT is by entering the ddt command: users/yourname> ddt 70 S–2396–60Debugging Code [6] Assuming X-Windows forwarding is con?gured correctly, the main program window displays, along with a pop-up menu offering the following options. • Run and Debug a Program • Debug a Multi-Process Non-MPI Program • Attach to a Running Program • Open a Core File • Restore a Checkpoint • Cancel Select the option you want to use, or Cancel to close the pop-up menu and proceed to the DDT main window. All the above options are also available through the Sessions menu Run option. 6.5.1 Known Limitations DDT does not support UPC. DDT has a number of defaults that affect batch queue submission behavior. When you begin a DDT session, either by selecting Run and Debug a Program from the pop-up Welcome menu or by selecting Run from the Session menu in the DDT main window, the Queue Submission Mode window displays. If you use a batch queuing system such as PBS Pro, always verify the Queue Submission Parameters before proceeding. In particular, verify that the default Queue name and Procs Per Node match your system's con?guration. The default Queue name is generally site-speci?c, while the Procs Per Node value must match your system's processor types: dual-core, quad-core, and so on. If you need to change the Queue Submission Parameters, click the Change button on the Queue Submission Mode window to do so for the duration of the current session. Alternatively, you can create a template ?le that stores your preferred parameters. Instructions for creating and using template ?les are provided in the DDT User Guide. To change your system's default Queue Submission Parameters for all users, contact your site administrator. Default con?guration information is stored in the /opt/cray/ddt/version/default-config.ddt ?le. This information includes the name of the default template, which currently is /opt/cray/ddt/version/templates/xt4.qtf. The actual default queue submission parameters are speci?ed in the default template ?le. S–2396–60 71Cray Application Developer's Environment User's Guide 72 S–2396–60Performance Analysis [7] Table 18. Performance Analysis Basics Module: perftools Commands: pat_build, pat_report, app2 Man pages: intro_craypat(1), pat_build(1), pat_report(1), pat_help(1), app2(1), intro_papi(3), hwpc(5), nwpc(5) Note: Tool-speci?c man pages are available only when the associated module is loaded. Online help: CrayPat includes an extensive online help system that features many examples and the answers to many frequently asked questions. To access the help system, enter pat_help at the command line. Documentation: Using Cray Performance Analysis Tools After you have compiled and debugged your program, you are ready to begin analyzing its performance. The Cray Performance Analysis Tools are a suite of optional utilities that enable you to capture and analyze performance data generated during the execution of your program in order to help you to ?nd answers to two fundamental questions: How fast is my program running? and How can I make it run faster? The Cray Performance Analysis Tools suite consists of three components: • CrayPat: the program instrumentation, data capture, and basic text reporting tool • Cray Apprentice2: the graphical analysis and data visualization tool • PAPI: the Performance Application Programming Interface To begin working with the performance analysis tools, ?rst load your programming environment of choice, and then load the perftools module. users/yourname> module load perftools The performance analysis process consists of three basic steps. 1. Instrument your program, to specify what kind of data you want to collect under what conditions. 2. Execute your instrumented program, to generate and capture the desired data. 3. Analyze the resulting data. S–2396–60 73Cray Application Developer's Environment User's Guide Accordingly, CrayPat consists of the following major components: • pat_build, the utility used to instrument programs • the CrayPat run time environment, which collects the speci?ed performance data • pat_report, the ?rst-level analysis tool used to produce text reports or export data for more sophisticated analysis • pat_help, the command-line driven online help system All CrayPat components, including the man pages and help system, are available only when the perftools module is loaded. 7.1 Using CrayPat For successful results, the perftools module must be loaded before you compile the program to be instrumented, instrument the program, execute the instrumented program, or generate a report. If you want to instrument a program that was compiled before the perftools module was loaded, you may under some circumstances ?nd that re-linking is suf?cient, but as a rule it's best to load the perftools module and then recompile. When instrumenting a program, CrayPat requires that the object (.o) ?les created during compilation be present, as well as the library (.a) ?les, if any. However, most compilers automatically delete the .o and .a ?les when working with single source ?les and compiling and linking in a single step, therefore it is good practice to compile and link in separate steps and use the compiler command line option to preserve these ?les. For example, if you are using the Cray Compiling Environment (CCE) Fortran compiler, compile using either of these command line options: > ftn -c source?le.f Alternatively: > ftn -h keepfiles source?le.f Then link the object ?les to create the executable program: > ftn -o executable source?le.o 7.1.1 Instrumenting the Program After the program is compiled and linked, use the pat_build command to instrument the program for performance analysis. In simplest form, pat_build is used like this: > pat_build -O apa executable This produces a copy of your original program, which is named executable+pat (for example, a.out+pat) and instrumented for the default experiment. Your original executable remains untouched. 74 S–2396–60Performance Analysis [7] The pat_build command supports a large number of options and directives, including an API that enables you to instrument speci?ed regions of your code. These options and directives are summarized in the pat_build(1) man page and documented more extensively in Using Cray Performance Analysis Tools. 7.1.2 Collecting Data Instrumented programs are executed just like any other program; either by using the aprun command if your site permits interactive sessions or by using your system's batch commands. CrayPat supports more than ?fty optional run time environment variables that enable you to control instrumented program behavior and data collection during execution. For example, if you use the C shell and want to collect data in detail rather than in aggregate, consider setting the PAT_RT_SUMMARY environment variable to 0 (off) before launching your program. /lus/nid00008> setenv PAT_RT_SUMMARY 0 Doing so records data with timestamps, which makes additional reports available in Cray Apprentice2, but at the cost of potentially much larger data ?le sizes and somewhat increased overhead. The CrayPat run time environment variables are summarized in the intro_craypat(1) man page and documented more extensively in Using Cray Performance Analysis Tools. 7.1.3 Analyzing Data Assuming your instrumented program runs to completion or planned termination, CrayPat outputs one or more data ?les. The exact number, location, and content of the data ?le(s) varies depending on the nature of your program, the type of experiment for which it was instrumented, and the run time environment variable settings in effect at the time of program execution. All initial data ?les are output in .xf format, with a generated ?le name consisting of your original program name, plus pat, plus the execution process ID number, plus a code string indicating the type of data contained within the ?le. Depending on the program run and the types of data collected, CrayPat output may consist of either a single .xf data ?le or a directory containing multiple .xf data ?les. If the program was instrumented with the -O apa option, a ?le with the suf?x .apa is also generated. This ?le is a customized template for this program and is created for use with future instrumentation experiments. To begin analyzing the captured data, use the pat_report command. In simplest form, it looks like this: /lus/nid00008> pat_report myprog+pat+PID-nodes.xf S–2396–60 75Cray Application Developer's Environment User's Guide The pat_report command accepts either a ?le or directory name as input and processes the .xf ?le(s) to generate a text report. In addition, it also exports the .xf data to a single .ap2 ?le, which is both a self-contained archive that can be reopened later using the pat_report command and the exported-data ?le format used by Cray Apprentice2. The pat_report command provides more than thirty prede?ned report templates, as well as a large variety of user-con?gurable options. These reports and options are summarized in the pat_report(1) man page and documented more extensively in Using Cray Performance Analysis Tools. 7.1.4 For More Information In addition to Using Cray Performance Analysis Tools and the intro_craypat(1), pat_build(1), and pat_report(1) man pages, there is a substantial amount of information, including an FAQ and examples, in the CrayPat online help system. The help system is accessible whenever the perftools module is loaded; to access the help system, enter pat_help at the command line. For more information about using the help system, see the pat_help(1) man page. 7.2 Using Cray Apprentice2 Cray Apprentice2 is an optional GUI tool that is used to visualize and manipulate the performance analysis data captured during program execution. Cray Apprentice2 can be run either on the Cray system or, optionally, on a standalone Linux desktop machine. Cray Apprentice2 can display a wide variety of reports and graphs, depending on the type of program being analyzed, the way in which the program was instrumented for data capture, and the data that was collected during program execution. Cray Apprentice2 is not directly integrated with CrayPat. You cannot launch Cray Apprentice2 from within CrayPat, nor can you set up or run performance analysis experiments from within Cray Apprentice2. Rather, use CrayPat ?rst, to instrument your program and capture performance analysis data, and then use Cray Apprentice2 to visualize and explore the resulting data ?les. The number and appearance of the reports that can be generated using Cray Apprentice2 is determined by the kind and quantity of data captured during program execution, which in turn is determined by the way in which the program was instrumented and the environment variables in effect at the time of program execution. For example, changing the PAT_RT_SUMMARY environment variable to 0 before executing the instrumented program nearly doubles the number of reports available when analyzing the resulting data in Cray Apprentice2. To run Cray Apprentice2, load the perftools module, if it is not already loaded. users/yourname> module load perftools 76 S–2396–60Performance Analysis [7] Then use the app2 command to launch Cray Apprentice2. users/yourname> app2 [data?le.ap2] & Note: Cray Apprentice2 requires that your workstation be con?gured to host X Window System sessions. If the app2 command returns an "unable to open display" error, contact your system administrator for help in con?guring X Window System hosting and forwarding. At this point the GUI takes over. If you speci?ed a data ?le name with the app2 command, the ?le is opened and parsed and the Overview report is displayed. If you did not specify a data ?le name, the Open File window opens and you can use standard GUI tools to browse through the ?le system and select the data ?le you want to open. For more information about using Cray Apprentice2, see the app2(1) man page, the Cray Apprentice2 help system, and Using Cray Performance Analysis Tools. 7.3 Using PAPI The Performance API (PAPI) is a standard API for accessing microprocessor registers. CrayPat uses PAPI to interface to the Cray system hardware; therefore the module papi is normally loaded as part of the perftools module. The interface between PAPI and CrayPat is normally transparent to the user. However, advanced users may want to bypass CrayPat and work with PAPI directly. In this case, you must unload the perftools module and then reload only the papi module. > module unload perftools > module load papi Note: CrayPat and direct PAPI commands cannot be used at the same time. Therefore, pat_build does not allow you to instrument a program that has also been instrumented using calls to PAPI functions. For more information about PAPI, see the intro_papi(3) and papi_counters(5) man pages. To see what sorts of information PAPI is capable of capturing on your system, use the papi_avail and papi_native_avail commands. Note: Effective with PAPI version 3.7.2, The PAPI Group has discontinued providing man pages for individual PAPI functions. Additional information about using PAPI is available through the PAPI website, at http://icl.cs.utk.edu/papi/. S–2396–60 77 TM Cray Application Developer's Environment User's Guide S–2396–50© 2004–2010 Cray Inc. All Rights Reserved. This document or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Inc. The gnulicinfo(7) man page contains the Open Source Software licenses (the "Licenses"). Your use of this software release constitutes your acceptance of the License terms and conditions. U.S. GOVERNMENT RESTRICTED RIGHTS NOTICE The Computer Software is delivered as "Commercial Computer Software" as de?ned in DFARS 48 CFR 252.227-7014. All Computer Software and Computer Software Documentation acquired by or for the U.S. Government is provided with Restricted Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7014, as applicable. Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7013, as applicable. Cray, LibSci, PathScale, and UNICOS are federally registered trademarks and Active Manager, Baker, Cascade, Cray Apprentice2, Cray Apprentice2 Desktop, Cray C++ Compiling System, Cray CX, Cray CX1, Cray CX1-iWS, Cray CX1-LC, Cray CX1000, Cray CX1000-C, Cray CX1000-G, Cray CX1000-S, Cray CX1000-SC, Cray CX1000-SM, Cray CX1000-HN, Cray Fortran Compiler, Cray Linux Environment, Cray SHMEM, Cray X1, Cray X1E, Cray X2, Cray XD1, Cray XE, Cray XE6, Cray XMT, Cray XR1, Cray XT, Cray XTm, Cray XT3, Cray XT4, Cray XT5, Cray XT5 h , Cray XT5m, Cray XT6, Cray XT6m, CrayDoc, CrayPort, CRInform, ECOphlex, Gemini, Libsci, NodeKARE, RapidArray, SeaStar, SeaStar2, SeaStar2+, Threadstorm, UNICOS/lc, UNICOS/mk, and UNICOS/mp are trademarks of Cray Inc. AMD Opteron is a trademark of Advanced Micro Devices, Inc.. GNU is a trademark of The Free Software Foundation. Intel is a trademark of Intel Corporation or its subsidiaries in the United States and other countries. Java is a trademark of Oracle and/or its af?liates. Linux is a trademark of Linus Torvalds. Lustre is a trademark of Oracle and/or its af?liates. Moab and TORQUE are trademarks of Adaptive Computing Enterprises, Inc. MySQL is a trademark of Oracle and/or its af?liates. Opteron is a trademark of Advanced Micro Devices, Inc. PBS Professional is a trademark of Altair Grid Technologies. PETSc is a trademark of Copyright (C) 1995-2004 University of Chicago. PGI is a trademark of The Portland Group Compiler Technology, STMicroelectronics, Inc. Platform is a trademark of Platform Computing Corporation. RSA is a trademark of RSA Security Inc. SecurID is a trademark of RSA Security Inc. TotalView is a trademark of TotalView Technology, LLC. Windows is a trademark of Microsoft Corporation. UNIX, the “X device,” X Window System, and X/Open are trademarks of The Open Group in the United States and other countries. All other trademarks are the property of their respective owners. Version 1.0 Published December 2004 Draft documentation to support Cray XT3 early-production systems. Version 1.0 Published March 2005 Draft documentation to support Cray XT3 limited-availability systems. Version 1.1 Published June 2005 Supports Cray XT3 systems running the Cray XT3 Programming Environment 1.1 and UNICOS/lc 1.1 releases. Version 1.2 Published August 2005 Supports Cray XT3 systems running the Cray XT3 Programming Environment 1.2 and UNICOS/lc 1.2 releases. Version 1.3 Published November 2005 Supports Cray XT3 systems running the Cray XT3 Programming Environment 1.3 and UNICOS/lc 1.3 releases. Version 1.4 Published April 2006 Supports Cray XT3 systems running the Cray XT3 Programming Environment 1.4 and UNICOS/lc 1.4 releases.Version 1.5 Published August 2006 Supports limited availability (LA) release of Cray XT systems running the Cray XT Programming Environment 1.5 and Cray Linux Environment (CLE)1.5 releases. Version 1.5 Published November 2006 Supports general availability (GA) release of Cray XT systems running the Cray XT Programming Environment 1.5 and UNICOS/lc 1.5 releases. Version 2.0 Published June 2007 Supports limited availability (LA) release of Cray XT systems running the Cray XT Programming Environment 2.0 and UNICOS/lc 2.0 releases. Version 2.0 Published October 2007 Supports general availability (GA) release of Cray XT systems running the Cray XT Programming Environment 2.0 and UNICOS/lc 2.0 releases. Version 2.1 Published July 2008 Supports limited availability (LA) release of Cray XT systems running the Cray XT Programming Environment and CLE 2.1 releases. Version 2.1 Published November 2008 Supports general availability (GA) release of Cray XT systems running the Cray XT Programming Environment and CLE 2.1 releases. Version 2.2 Published July 2009 Supports general availability (GA) release of Cray XT systems running the Cray XT Programming Environment and CLE 2.2 releases. Version 5.0 Published June 2010 Restructured document now packaged with Cray Application Developer's Environment (CADE) release 5.0 and supporting Cray XT and Cray XE systems running Cray Linux Environment (CLE) release 2.2 or later.New Features Cray Application Developer's Environment User's Guide S–2396–50 CADE 5.0 For CADE release 5.0 this guide has been completely redesigned and rewritten, with more emphasis on the kinds of information users new to Cray systems need in order to get started and more links to external sources of more information. The detailed aprun, ALPS, batch system, and other OS-dependent information has been moved to a new guide, Workload Management and Application Placement for the Cray Linux Environment, which is packaged with the operating system.Contents Page Introduction [1] 11 1.1 What You Must Know About Your Site . . . . . . . . . . . . . . . . . . 11 1.1.1 Cray XE or Cray XT? . . . . . . . . . . . . . . . . . . . . . 12 1.1.2 How Many Cores? . . . . . . . . . . . . . . . . . . . . . . 12 1.1.3 Which Operating System? . . . . . . . . . . . . . . . . . . . . 13 1.1.4 What Is a Compute Node? . . . . . . . . . . . . . . . . . . . . 13 1.1.5 Which File System? . . . . . . . . . . . . . . . . . . . . . . 14 1.1.6 Which Batch System? . . . . . . . . . . . . . . . . . . . . . 14 1.1.7 What Is the hostname? . . . . . . . . . . . . . . . . . . . . . 14 1.2 Logging In . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.2.1 UNIX or Linux Users . . . . . . . . . . . . . . . . . . . . . 15 1.2.2 Windows, Macintosh, or Other Users . . . . . . . . . . . . . . . . . 16 1.3 Navigating the File Systems . . . . . . . . . . . . . . . . . . . . . 19 Using Modules [2] 21 2.1 What Is Loaded Now? . . . . . . . . . . . . . . . . . . . . . . 21 2.2 What Is Available? . . . . . . . . . . . . . . . . . . . . . . . 22 2.3 Loading and Unloading Module?les . . . . . . . . . . . . . . . . . . . 22 2.4 Swapping Module?les . . . . . . . . . . . . . . . . . . . . . . 23 2.5 Release Notes and Module Help . . . . . . . . . . . . . . . . . . . . 24 2.6 For More Information . . . . . . . . . . . . . . . . . . . . . . . 26 Batch Systems and Program Execution [3] 27 3.1 Interactive Mode . . . . . . . . . . . . . . . . . . . . . . . . 28 3.1.1 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.2 Batch Mode . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.3 Using aprun . . . . . . . . . . . . . . . . . . . . . . . . . 31 Using Compilers [4] 33 4.1 About Compiler Drivers . . . . . . . . . . . . . . . . . . . . . . 33 S–2396–50 7Cray Application Developer’s Environment User’s Guide Page 4.2 About C/C++ Floating Point Data Types . . . . . . . . . . . . . . . . . 34 4.3 About PGI Compilers . . . . . . . . . . . . . . . . . . . . . . . 35 4.3.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 35 4.4 About the Cray Compiling Environment (CCE) . . . . . . . . . . . . . . . . 36 4.4.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 36 4.5 About PathScale Compilers . . . . . . . . . . . . . . . . . . . . . 36 4.5.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 37 4.6 About Intel Compilers . . . . . . . . . . . . . . . . . . . . . . 37 4.6.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 37 4.7 About GNU Compilers . . . . . . . . . . . . . . . . . . . . . . 37 4.7.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 38 4.8 About the Chapel Parallel Programming Language . . . . . . . . . . . . . . . 38 4.9 About Cross-compilers . . . . . . . . . . . . . . . . . . . . . . 38 Libraries [5] 41 5.1 Scienti?c Libraries . . . . . . . . . . . . . . . . . . . . . . . 41 5.1.1 BLAS and LAPACK . . . . . . . . . . . . . . . . . . . . . . 42 5.1.1.1 Notes . . . . . . . . . . . . . . . . . . . . . . . . 43 5.1.2 BLACS and ScaLAPACK . . . . . . . . . . . . . . . . . . . . 44 5.1.2.1 Notes . . . . . . . . . . . . . . . . . . . . . . . . 44 5.1.3 IRT . . . . . . . . . . . . . . . . . . . . . . . . . . 45 5.1.4 FFT . . . . . . . . . . . . . . . . . . . . . . . . . . 46 5.1.4.1 CRAFFT . . . . . . . . . . . . . . . . . . . . . . . 46 5.1.4.2 FFTW . . . . . . . . . . . . . . . . . . . . . . . . 46 5.1.4.3 ACML . . . . . . . . . . . . . . . . . . . . . . . . 47 5.2 About the Fast_mv Library . . . . . . . . . . . . . . . . . . . . . 47 5.3 PETSc . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 5.3.1 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 50 5.4 Trilinos . . . . . . . . . . . . . . . . . . . . . . . . . . 50 5.5 MPT . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 5.5.1 Static Libraries . . . . . . . . . . . . . . . . . . . . . . . 52 5.5.2 Dynamic Libraries . . . . . . . . . . . . . . . . . . . . . . 52 Debugging Code [6] 53 6.1 Cray Debugger Support Tools . . . . . . . . . . . . . . . . . . . . 53 6.1.1 Using lgdb . . . . . . . . . . . . . . . . . . . . . . . . 54 6.1.2 Using Abnormal Termination Processing (ATP) . . . . . . . . . . . . . . 55 6.1.2.1 On CLE 3.0 or Later . . . . . . . . . . . . . . . . . . . . 55 8 S–2396–50Contents Page 6.1.2.2 On CLE 2.2 or Earlier . . . . . . . . . . . . . . . . . . . . 56 6.1.3 Using MRNet (Deferred implementation) . . . . . . . . . . . . . . . . 56 6.1.4 Using STAT (Deferred implementation) . . . . . . . . . . . . . . . . 56 6.2 Using Fast-Track Debugging . . . . . . . . . . . . . . . . . . . . . 57 6.2.1 Supported Compilers and Debuggers . . . . . . . . . . . . . . . . . 58 6.3 About Core Files . . . . . . . . . . . . . . . . . . . . . . . . 58 6.4 Using TotalView . . . . . . . . . . . . . . . . . . . . . . . . 58 6.4.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 60 6.5 Using DDT . . . . . . . . . . . . . . . . . . . . . . . . . 60 6.5.1 Known Limitations . . . . . . . . . . . . . . . . . . . . . . 61 Performance Analysis [7] 63 7.1 Using CrayPat . . . . . . . . . . . . . . . . . . . . . . . . . 64 7.1.1 Instrumenting the Program . . . . . . . . . . . . . . . . . . . . 65 7.1.2 Collecting Data . . . . . . . . . . . . . . . . . . . . . . . 65 7.1.3 Analyzing Data . . . . . . . . . . . . . . . . . . . . . . . 65 7.1.4 For more information . . . . . . . . . . . . . . . . . . . . . 66 7.2 Using Cray Apprentice2 . . . . . . . . . . . . . . . . . . . . . . 66 7.3 Using PAPI . . . . . . . . . . . . . . . . . . . . . . . . . 67 Glossary 69 Figures Figure 1. Selecting SSH Protocol . . . . . . . . . . . . . . . . . . . . 16 Figure 2. Enabling X11 Forwarding . . . . . . . . . . . . . . . . . . . 17 Figure 3. Logging In . . . . . . . . . . . . . . . . . . . . . . . 18 Tables Table 1. aprun versus qsub Options . . . . . . . . . . . . . . . . . . 32 Table 2. C/C++ Data Type Sizes . . . . . . . . . . . . . . . . . . . . 34 Table 3. PGI Compiler Basics . . . . . . . . . . . . . . . . . . . . . 35 Table 4. Cray Compiler Basics . . . . . . . . . . . . . . . . . . . . . 36 Table 5. PathScale Compiler Basics . . . . . . . . . . . . . . . . . . . 36 Table 6. Intel Compiler Basics . . . . . . . . . . . . . . . . . . . . . 37 Table 7. GNU Compiler Basics . . . . . . . . . . . . . . . . . . . . 37 Table 8. LibSci Basics . . . . . . . . . . . . . . . . . . . . . . . 41 Table 9. Fast_mv Basics . . . . . . . . . . . . . . . . . . . . . . 47 Table 10. PETSc Basics . . . . . . . . . . . . . . . . . . . . . . 49 Table 11. Trilinos Basics . . . . . . . . . . . . . . . . . . . . . . 50 S–2396–50 9Cray Application Developer’s Environment User’s Guide Page Table 12. MPT Basics . . . . . . . . . . . . . . . . . . . . . . . 51 Table 13. lgdb Basics . . . . . . . . . . . . . . . . . . . . . . . 54 Table 14. atp Basics . . . . . . . . . . . . . . . . . . . . . . . 55 Table 15. TotalView Basics . . . . . . . . . . . . . . . . . . . . . 58 Table 16. DDT Basics . . . . . . . . . . . . . . . . . . . . . . . 60 Table 17. Performance Analysis Basics . . . . . . . . . . . . . . . . . . 63 10 S–2396–50Introduction [1] Welcome to the Cray Application Developer's Environment (CADE) user's guide. This guide describes the software environment and tools used to develop, debug, and run applications on Cray XT and Cray XE systems. It is intended as a general overview and introduction to the Cray system for new users and application programmers. The information contained in this guide is of necessity fairly high-level and generalized, as the Cray platform supports a wide variety of hardware nodes as well as many different compilers, debuggers, and other software tools. System hardware and software con?gurations therefore vary considerably from site to site, so for speci?c information about your site and its installed hardware, software, and usage policies, always contact your site administrator. 1.1 What You Must Know About Your Site The Cray XT and Cray XE systems are: • massively parallel (MPP) • distributed memory • non-uniform memory access (NUMA) • commodity processor-based Supercomputers designed to scale to immense size and run applications requiring high-performance, large-scale processing, high network bandwidth, and complex inter-processor communications. Both series of systems use 64-bit AMD Opteron processors as the basic computational engines, although the exact number of cores per AMD Opteron varies from site to site and sometimes even from cabinet to cabinet, depending on your site's installation, expansion, and upgrade history. S–2396–50 11Cray Application Developer’s Environment User’s Guide 1.1.1 Cray XE or Cray XT? From the programmer's point of view, the most signi?cant difference between the two systems is: • Cray XE systems use Cray Gemini ASICs to manage inter-processor communications • Cray XT systems use Cray SeaStar or Cray SeaStar2+ application-speci?c integrated circuits (ASICs) to manage inter-processor communications Because of this hardware difference, the two systems use different versions of the MPI and SHMEM libraries, which offer different sets of optional controls to the application developer. Cray XE systems also support the Generic Network Interface (GNI) and Distributed Shared Memory Application (DMAPP) APIs. which offer the programmer API-level access to the advanced features of the Gemini interconnect. The differences between the Cray XT and Cray XE versions of MPI and SHMEM are discussed in more detail in MPT on page 51. The Gemini GNI and DMAPP APIs are discussed in detail in Using the GNI and DMAPP APIs. 1.1.2 How Many Cores? The next most signi?cant differences are between models. • Cray XT4 systems use one dual- or quad-core ("Barcelona") AMD Opteron CPU per compute node • Cray XT5 and Cray XE5 systems use two four-core ("Shanghai") or six-core ("Istanbul") AMD Opteron CPUs per compute node • Cray XT6 and Cray XE6 systems use two eight- or twelve-core ("Magny-Cours") AMD Opteron CPUs per compute node Because of these hardware differences, you can invoke different options when compiling and executing your programs. These differences are discussed in more detail in Workload Management and Application Placement for the Cray Linux Environment. 12 S–2396–50Introduction [1] 1.1.3 Which Operating System? All current Cray systems run the Cray Linux Environment (CLE) operating system on the login nodes and a lightweight kernel called CNL on the compute nodes. Some of the options available to application developers vary depending on which version of CLE is currently running on the system. • Cray XE5 and Cray XE6 systems run CLE release 3.1 or later. CLE 3.x is based on SLES 11. • Cray XT6 and Cray XT6m systems run CLE release 3.0 or later. CLE 3.0 is based on SLES 11. • All other systems may be running CLE release 2.1, 2.2, 3.0, or 3.1. If you are not certain which release your site is using, check the MOTD when you log in, or check to see which version of the xt-os module is loaded by default. (For more information about using modules, see Chapter 2, Using Modules on page 21.) CLE 2.x is based on SLES 10. Note: The Catamount compute node operating system is no longer supported by Cray. 1.1.4 What Is a Compute Node? From the application developer's point of view, a modern Cray system is a tightly integrated network of thousands of nodes, all specialized for different purposes. While some are dedicated to administrative or networking functions and therefore off-limits to application programmers, the two you will deal with most often are: • login nodes — The node you access when you ?rst log in to the system. Login nodes offer the full Cray Linux Environment (CLE) operating system, are used for basic development tasks such as editing ?les and compiling code, generally have access to the network ?le system, and are shared resources that may be used concurrently by multiple users. • compute nodes — The nodes on which production jobs are executed. Compute nodes run a lightweight operating system called Compute Node Linux (CNL), can be accessed only by submitting jobs through the batch management system (typically PBS or TORQUE Resource Manager), generally have access only to the high-performance parallel ?le system (typically Lustre), and are dedicated resources, exclusively yours for the duration of the batch reservation. When new users ?rst begin working on the Cray system, this difference between login and compute nodes can be a source of confusion. Remember, when you ?rst log in to the system, you are placed on a login node. You cannot execute parallel programs on the login node, nor can you directly access ?les stored on the high-performance parallel ?le system. S–2396–50 13Cray Application Developer’s Environment User’s Guide Instead, you must use your site's batch system to place parallel programs on the compute nodes, either from the login node or from a mount-point on the parallel ?le system. Note: You can execute serial (single-process) programs on login nodes, but executing large or long-running serial programs on login nodes is discouraged, as login nodes are shared resources. 1.1.5 Which File System? All Cray XT and Cray XE systems require the use of a high-performance parallel ?le system. Most sites currently use the Lustre File System, and all examples shown in this guide were developed on a Lustre ?le system using Lustre commands. Before copying any examples from this guide verbatim, verify which ?le system your site uses and what your site's policies are regarding home directories, scratch space, disk quotas, backup policies, and so on. Then if required, adjust the instructions accordingly. 1.1.6 Which Batch System? Cray XT and Cray XE systems typically operate under the control of a batch system such as PBS, OpenPBS, PBS Professional, TORQUE Resource Manager, or the Moab Cluster Suite. All examples shown in this guide were developed using PBS Pro and/or Moab/TORQUE. Before copying any examples from this guide verbatim, verify which batch system your site uses and if required, adjust the instructions accordingly. 1.1.7 What Is the hostname? All Cray XT and Cray XE systems have a unique hostname. While it is possible to log in to a given system by using its IP address (e.g., 127.0.0.1), it's easier if you use the hostname. The form of the hostname varies from site to site. Some sites use a single word; e.g., narwhal. Others use a fully quali?ed URL; e.g., narwhal.univ-alaska-barrow.edu. You should be told which name and form to use when your user account is set up. 1.2 Logging In User account setup and authentication policies vary widely from site to site. In general, you must contact your site administrator to get a login account on the system. Any site-speci?c security or authentication policies (such as, say, the correct use of an RSA SecurID token, if provided) should be explained to you at that time. 14 S–2396–50Introduction [1] Once your user account is created, log in to the Cray system using SSH (Secure Shell), protocol version 2. SSH is a remote login program that encrypts all communications between the client and host and replaces the earlier telnet, rlogin, and rsh programs. 1.2.1 UNIX or Linux Users If you use a UNIX or Linux workstation, the ssh utility is generally available at any command line and documented in the ssh(1) man page. To log in to the Cray system, enter: % ssh -X hostname The -X option enables X11 display forwarding. Automatic forward of X11 windows is highly recommended as many of the application development tools use GUI displays. On some systems, you may be required to enter your user ID as well. This can be done in several different ways. For example: % ssh -X -luserID hostname Or % ssh -X userID@hostname In any case, after you SSH to the system, you may have to answer one or more RSA or password challenges, and then you are logged into the system. A series of system status and MOTD (message of the day) type messages may display, after which you are placed in your home directory on a login node. /users/userID> You are now ready to begin working. Jump to Navigating the File Systems on page 19. S–2396–50 15Cray Application Developer’s Environment User’s Guide 1.2.2 Windows, Macintosh, or Other Users If you use a Windows, Macintosh, or other personal computer, you ?rst need to obtain and install a client program for your system that supports SSH protocol 2, such as PuTTY for Windows. Your system administrator should be able to provide a list of accepted clients. You may need to con?gure your client to support SSH protocol 2 and X11 forwarding. For example, if you are using PuTTY, you may need to click SSH in the left pane to set the preferred SSH protocol version: Figure 1. Selecting SSH Protocol 16 S–2396–50Introduction [1] Click X11 in the left pane and check the box to enable X11 forwarding: Figure 2. Enabling X11 Forwarding S–2396–50 17Cray Application Developer’s Environment User’s Guide Then click Session in the left pane to return to the Basic options window. Figure 3. Logging In Enter the hostname in the Host Name ?eld and click the Open button to begin your SSH session. You may need to enter your userID and answer one or more RSA or password challenges, and then you are logged into the system. A series of system status and MOTD (message of the day) type messages may display, after which you are placed in your home directory on a login node. /users/userID> You are now ready to begin working. 18 S–2396–50Introduction [1] 1.3 Navigating the File Systems When you ?rst log in to the Cray system, you are placed in your home directory on a login node. /users/userID> At this point you have access to all the features and functions of the full Cray Linux Environment (CLE) operating system, such as the sftp and scp commands, and also typically to your full network ?le system. On most systems your home directory on the login node is de?ned as the environment variable $HOME, and this variable can be used in any ?le system command. For example, to return to your home directory from any other location in the ?le system(s), enter this command: > cd $HOME Remember, you can edit ?les, manipulate ?les, compile code, execute serial (single-process) programs, and otherwise work in your home directory on the login node. However, you cannot execute parallel programs on the login node. Parallel programs must be run on the compute nodes, under the control of the batch system, and generally while mounted on the high-performance parallel ?le system. To do this, you must ?rst identify the nids (node IDs) of the ?le system mount points. On the Lustre ?le system, this can be done in one of two ways. Either enter the df -t lustre command to ?nd the Lustre nodes and get a summary report on disk usage: users/userID> df -t lustre Filesystem 1K-blocks Used Available Use% Mounted on 8@ptl:/narwhalnid8 8998913280 6946443260 1595348672 82% /lus/nid00008 Or enter the lfs df command to get more detailed information: users/userID> lfs df UUID 1K-blocks Used Available Use% Mounted on nid00008_mds_UUID 179181084 2675664 166265604 1% /lus/nid00008[MDT:0] ost0_UUID 1124864160 895207088 172517160 79% /lus/nid00008[OST:0] ost1_UUID 1124864160 838067380 229656540 74% /lus/nid00008[OST:1] ost2_UUID 1124864160 826599428 241124820 73% /lus/nid00008[OST:2] ost3_UUID 1124864160 827914052 239801932 73% /lus/nid00008[OST:3] ost4_UUID 1124864160 964324672 103398548 85% /lus/nid00008[OST:4] ost5_UUID 1124864160 932986208 134738024 82% /lus/nid00008[OST:5] ost6_UUID 1124864160 832715148 235009164 74% /lus/nid00008[OST:6] ost7_UUID 1124864160 828631656 239092572 73% /lus/nid00008[OST:7] filesystem summary: 8998913280 6946445632 1595338760 77% /lus/nid00008 Note: The above commands are speci?c to the Lustre high-speed parallel ?le system. If your site uses a different ?le system, adjust the instructions accordingly. S–2396–50 19Cray Application Developer’s Environment User’s Guide In either case, in this example, the Lustre mount point is /lus/nid00008. If you cd to this mount point: users/userID> cd /lus/nid00008 Directory: /lus/nid00008 /lus/nid00008> you are now on the high-performance parallel ?le system. At this point you can edit ?les, manipulate ?les, compile code, and so on, but you can also execute programs on the compute nodes, generally by using the batch system. 20 S–2396–50Using Modules [2] The Cray system uses the Modules environment management package to support dynamic modi?cation of the user environment via module?les. Each module?le contains all the information needed to con?gure the shell for a particular application. To make major changes in your user environment, such as switching to a different compiler or a different version of a library, use the appropriate Modules commands to select the desired module?les. The advantage in using Modules is that you are not required to specify explicit paths for different executable versions or to set the $MANPATH and other environment variables manually. Instead, all the information required in order to use a given piece of software is embedded in the module?le and set automatically when you load the module?le. In general, the simplest way to make certain that the elements of your application development environment function correctly together is by using the Modules software and trusting it to keep track of paths and environment variables, and avoiding embedding speci?c directory paths into your startup ?les, make?les, and scripts. 2.1 What Is Loaded Now? When you ?rst log in to the Cray system, a set of site-speci?c default modules is loaded. This set varies depending on system hardware, operating system release level, site policies, and installed software. To see which modules are currently loaded on your system, use the module list command. users/yourname> module list Currently Loaded Modulefiles: 1) modules/3.1.6.5 12) Base-opts/2.2.48B 2) xtpe-target-cnl 13) pgi/10.4.0 3) xt-service/2.2.48B 14) totalview-support/1.1.0 4) xt-os/2.2.48B 15) xt-totalview/8.8.0 5) xt-boot/2.2.48B 16) xt-libsci/10.4.4 6) xt-lustre-ss/2.2.48B_1.6.5 17) pmi/1.0-1.0000.7628.10.2.ss 7) cray/job/1.5.5-0.1_2.0202.19481.53.6 18) xt-mpt/4.1.1 8) cray/csa/3.0.0-1_2.0202.19602.75.1 19) xt-pe/2.2.48B 9) cray/account/1.0.0-2.0202.19482.49.3 20) xt-asyncpe/3.9.39 10) cray/projdb/1.0.0-1.0202.19483.52.1 21) PrgEnv-pgi/2.2.48B 11) pbs/default 22) cray/MySQL/5.0.64-1.0202.2899.21.1 S–2396–50 21Cray Application Developer’s Environment User’s Guide This list is not as complicated as it may look. Essentially, it breaks down into three groups: operating system modules, programming environment modules, and support modules. For example, the xt-os module indicates that this system is running the CLE release 2.2 operating system, while the PrgEnv-pgi module indicates that PGI compiler suite is currently loaded. 2.2 What Is Available? To see which module?les are available on your system, use the module avail command. Note: The module avail command produces an alphabetical listing of every module?le in your module use path and has no option for "grepping." For this reason it is usually more useful to use the command with an argument. For example, if you are looking for a speci?c version of the PGI compiler suite, enter the following command. users/yourname> module avail PrgEnv-pgi ---------------------------------- /opt/modulefiles ---------------------------------- PrgEnv-pgi/2.2.26 PrgEnv-pgi/2.2.31 PrgEnv-pgi/2.2.37 PrgEnv-pgi/2.2.27 PrgEnv-pgi/2.2.33 PrgEnv-pgi/2.2.38 PrgEnv-pgi/2.2.28 PrgEnv-pgi/2.2.34 PrgEnv-pgi/2.2.39 PrgEnv-pgi/2.2.29 PrgEnv-pgi/2.2.35 PrgEnv-pgi/2.2.41 PrgEnv-pgi/2.2.30 PrgEnv-pgi/2.2.36 PrgEnv-pgi/2.2.48B(default) One module is generally designated as the default version. Whether this is the most recent version of this module depends on your site policies. Some sites always make the newest version the default, while others wait until after the new version has been tested and proven bug- and dependency-free. Whenever a newer version of a module is installed, the older versions continue to remain available, unless the site administrator has explicitly chosen to delete them. 2.3 Loading and Unloading Module?les If a module?le is not already loaded, use the module load command to load it. users/yourname> module load module?le This command loads the currently de?ned default version of the module, unless you specify otherwise. For example, if the modules listed in What Is Available? on page 22 are present on your system but PrgEnv-pgi is not already loaded, then: users/yourname> module load PrgEnv-pgi loads PrgEnv-pgi/2.2.48B. To load a non-default version of the module—for example, release 2.2.39—you must specify the full module name. users/yourname> module load PrgEnv-pgi/2.2.39 22 S–2396–50Using Modules [2] However, if a module is already loaded, then you must ?rst unload the currently loaded module before loading a different version. For example, to change from the default to version of the PGI compiler suite to version 2.2.39, use the module unload command to remove the version currently loaded. users/yourname> module unload PrgEnv-pgi Modules may be linked and related. If you enter the module list command now and compare the results to those shown in What Is Loaded Now? on page 21: users/yourname> module list Currently Loaded Modulefiles: 1) modules/3.1.6.5 8) cray/csa/3.0.0-1_2.0202.19602.75.1 2) xtpe-target-cnl 9) cray/account/1.0.0-2.0202.19482.49.3 3) xt-service/2.2.48B 10) cray/projdb/1.0.0-1.0202.19483.52.1 4) xt-os/2.2.48B 11) pbs/default 5) xt-boot/2.2.48B 12) Base-opts/2.2.48B 6) xt-lustre-ss/2.2.48B_1.6.5 13) cray/MySQL/5.0.64-1.0202.2899.21.1 7) cray/job/1.5.5-0.1_2.0202.19481.53.6 You can see that along with PrgEnv-pgi, the pgi, totalview-support, xt-totalview, xt-libsci, pmi, xt-mpt, xt-pe, and xt-asyncpe modules were also unloaded. If you now load a different version of the PrgEnv-pgi module: users/yourname> module load PrgEnv-pgi/2.2.39 Then use the module list command again, you can see that the eight associated modules have been reloaded automatically, but this time using the versions that are appropriate for use with the 2.2.39 version of the PGI compiler. 2.4 Swapping Module?les Alternatively, you can use the module swap or module switch command to unload one module and load the comparable module. For example, to switch from using the PGI Compiler Suite to the Cray Compiling Environment, enter this command: users/yourname> module swap PrgEnv-pgi PrgEnv-cray S–2396–50 23Cray Application Developer’s Environment User’s Guide If you then list the modules that are loaded after this swap and compare them to the results shown in What Is Loaded Now? on page 21. users/yourname> module list Currently Loaded Modulefiles: 1) modules/3.1.6.5 12) Base-opts/2.2.48B 2) xtpe-target-cnl 13) cray/MySQL/5.0.64-1.0202.2899.21.1 3) xt-service/2.2.48B 14) PrgEnv-cray/1.0.1 4) xt-os/2.2.48B 15) xt-asyncpe/3.9.39 5) xt-boot/2.2.48B 16) xt-mpt/4.1.1 6) xt-lustre-ss/2.2.48B_1.6.5 17) pmi/1.0-1.0000.7628.10.2.ss 7) cray/job/1.5.5-0.1_2.0202.19481.53.6 18) xt-pe/2.2.48B 8) cray/csa/3.0.0-1_2.0202.19602.75.1 19) xt-libsci/10.4.4 9) cray/account/1.0.0-2.0202.19482.49.3 20) acml/4.4.0 10) cray/projdb/1.0.0-1.0202.19483.52.1 21) cce/7.2.3 11) pbs/default You can see that a different set of supporting modules have been loaded automatically. 2.5 Release Notes and Module Help Most module?les on the Cray system include module help that is speci?c to the module?le. The exact content of the module help varies from vendor to vendor and release to release, but it generally includes release notes and late-breaking news, such as lists of bugs ?xed in the release, known dependencies and limitations, and usage information received too late to be documented elsewhere. You can view the module help at any time for any module currently installed on the system. The module does not need to be loaded in order for you to view the module help. To access the module help, use the module help command. For example, to see the module help associated with the default PGI module, enter this command: users/yourname> module help pgi Note: Make certain you specify the exact module name that you want. For example, module help PrgEnv-pgi and module help pgi display different information. 24 S–2396–50Using Modules [2] The module help content is displayed. pgi 10.5.0 ========== Release Date: May 20, 2010 Purpose: -------- Features of PGI 10.5.0 are documented at: http://www.pgroup.com/doc/pgiwsrn105.pdf The following bugs are fixed in the PGI 10.5.0 release. 751661 PGI C/C++ handles non-conforming pgm with broken perfectly nested loops - C (no msg) & C++ (signal 11) [TPR 16167] 755930 PGI OpenMP pgf90 gets incorrect output for REDUCTION with ALLOCATABLE array as variable. [16507] 757052 PGI OpenMP pgf90 incorrect output from threadprivate(A) when A is allocatable. [16593] 757192 PGI OpenMP pgf90 !$omp do lastprivate(A) returns incorrect output. [16607] 758613 PGI OpenMP pgf90 incorrect output from sections lastprivate(a) where a is allocatable [16808] Product and OS Dependencies: ---------------------------- pgi 10.5.0 is supported on Cray XT systems running the UNICOS/lc 2.0 CNL and CLE 2.1 CNL or later operating systems. NOTE: Licensing change: PGI issued licensing policy has changed. The license will work with existing PGI compilers and all new compilers released prior to a specified date. A new license will be required to use any compilers released after the expiration date. Documentation: -------------- PGI User's Guide, /opt/pgi/10.5.0/linux86-64/10.5/doc/pgi10ug.pdf PGI Tools Guide, /opt/pgi/10.5.0/linux86-64/10.5/doc/pgi10tools.pdf PGI Workstation 10.5 Release Notes, /opt/pgi/10.5.0/linux86-64/10.5/doc/pgiwsrn105.pdf PGI Fortran Reference, /opt/pgi/10.5.0/linux86-64/10.5/doc/pgifortref.pdf PGI Workstation 10.5 Installation Guide, /opt/pgi/10.5.0/linux86-64/10.5/doc/pgiwsinstall105.pdf Installation: ------------- env CRAY_INSTALL_DEFAULT=1 rpm -ihv --oldpackage pgi-10.5.0-1.x86_64.rpm S–2396–50 25Cray Application Developer’s Environment User’s Guide 2.6 For More Information The Modules commands are documented in the module(1) and modulefiles(4) man pages. A summary of Modules commands can be displayed by entering the module help command. users/yourname> module help Modules Release 3.1.6.5 (Copyright GNU GPL v2 1991): Available Commands and Usage: + add|load modulefile [modulefile ...] + rm|unload modulefile [modulefile ...] + switch|swap modulefile1 modulefile2 + display|show modulefile [modulefile ...] + avail [modulefile [modulefile ...]] + use [-a|--append] dir [dir ...] + unuse dir [dir ...] + update + purge + list + clear + help [modulefile [modulefile ...]] + whatis [modulefile [modulefile ...]] + apropos|keyword string + initadd modulefile [modulefile ...] + initprepend modulefile [modulefile ...] + initrm modulefile [modulefile ...] + initswitch modulefile1 modulefile2 + initlist + initclear Different versions of the Modules software are in use on different sites. Accordingly, the module command arguments and options available on your site may vary from those shown here. 26 S–2396–50Batch Systems and Program Execution [3] User applications are always launched on the compute nodes using the application launcher, aprun, which submits applications to the Application Level Placement Scheduler (ALPS) for placement and execution. The ALPS service is both very powerful and highly ?exible, and a thorough discussion of it is beyond the scope of this manual. For more detailed information about ALPS and aprun, see the intro_alps(1) and aprun(1) man pages and Workload Management and Application Placement for the Cray Linux Environment. At most sites access to the compute node resources is managed by a batch control system, typically PBS Pro or Moab/TORQUE. Users run jobs either by using the qsub command to submit a job script, or else by using the qsub command to request an interactive session within the context of the batch system and then using the aprun command to run the application within the interactive session. In either case, running an application typically involves these steps. 1. Determine what system resources you will need. Generally, this means deciding how many cores and/or compute nodes you need for your job. 2. Use the apstat command to determine whether the resources you need are available. This is very important when you are planning to run in an interactive session. This is not as important if you are submitting a job script, as the batch system will keep your job script in the queue until the resources become available to run it. 3. Translate your resource request into the appropriate qsub and aprun command options, which are not necessarily the same. If running a batch job, modify your script accordingly. 4. Use the qsub command either to submit your job script or request an interactive session. 5. If using an interactive session, use the aprun command to launch your application. S–2396–50 27Cray Application Developer’s Environment User’s Guide 3.1 Interactive Mode Interactive mode is typically used for debugging or optimizing code but not for running production code. To begin an interactive session, use the qsub -I command. users/yourname> qsub -IVl mppwidth=cores Useful qsub options include: -I Start an interactive session -A account Charge the time to account. -q debug Run in the debug queue. -V Import any environment variables that were set in the user's shell. -l resource_type=speci?cation Specify the resources you want to reserve for this session. For example, you can use this option to reserve 24 compute cores for one hour. -l mppwidth=24,walltime=1:00:00 The -l resource_type=speci?cation argument supports a large number of options which are described in the qsub(1B) man page and expanded upon in the pbs_resources(7B) man page, and described in greater detail with examples in Workload Management and Application Placement for the Cray Linux Environment. After you have launched an interactive session, use the aprun command to launch your application. When you are ?nished, enter logout to exit the batch system and return to the normal command line. 3.1.1 Notes • You must use the -l mppwidth= option to specify at least one core when you start the interactive session. If you do not, your request for an interactive session will pause inde?nitely. • Pay attention to your ?le system mount points. You must be on the high-performance parallel ?le system (for example, if you are using the Lustre ?le system, /lus/nid00008/yourname) in order to launch jobs on the compute nodes. However, when you launch an interactive batch session, you are by default placed in your home directory, which is typically on a login node. (For example, /ufs/users/home/yourname.) You may need to cd back to the parallel ?le system after launching an interactive session. 28 S–2396–50Batch Systems and Program Execution [3] • After you launch an interactive batch session, a number of environment variables are automatically de?ned and exported to the job, as described on the qsub(1B) man page. For example, the environment variable $PBS_O_WORKDIR is set to the directory from which the batch job was submitted. This can be a handy way to return to your mount point if you cd to the parallel ?le system before invoking the interactive batch session. • The qsub and aprun commands use different options to perform similar functions. These differences are touched on lightly in Using aprun on page 31 and described in detail in Workload Management and Application Placement for the Cray Linux Environment. • When you launch an interactive batch session, you must request the maximum number of resources you expect to use. Once a batch session begins, you can only use fewer resources than initially requested. You cannot use the aprun command to use more resources than you reserved using the qsub command. 3.2 Batch Mode Production jobs are typically run in batch mode. Batch scripts are shell scripts containing ?ags and commands to be interpreted by a shell and are used to run a set of commands in sequence. To run a batch script, use the qsub command. users/yourname> qsub [-l resource_type=speci?cation] jobscript The [-l resource_type=speci?cation] arguments are described in the pbs_resources(7B) man page. A typical batch script might look like this: 1: #!/bin/bash 2: #PBS -A account 3: #PBS -N job_name 4: #PBS -j oe 5: #PBS -l walltime=1:00:00,size=192 6: 7: cd $PBS_O_WORKDIR 8: date 9: aprun -n 192 ./a.out > my_output_?le 2>&1 S–2396–50 29Cray Application Developer’s Environment User’s Guide Parsing this script line-by-line, it would be interpreted as follows: 1. Invoke the shell to use to interpret the script. 2. Specify the account to which this time is billed. 3. Assign the job a name to use on messages and output. 4. Combine the job's STDOUT and STDERR. Note: While your job is running, STDOUT and STDERR are written to a ?le or ?les in a system directory and the output is copied to your submission directory only after the job completes. Specifying the -j oe option here and redirecting the output to a ?le in line 9 makes it possible for you to view STDOUT and STDERR while the job is running. 5. Reserve 192 cores for one hour. 6. This line is blank and is ignored. 7. cd to the submission directory, which presumably is a mount point on the Lustre ?le system. 8. Run the date command. 9. Execute the executable ?le a.out on 192 cores, and redirect any output to a ?le. After the job is submitted using the qsub command it goes into the queue, where it waits until the requested resources become available. When they do, the job is launched on the head node of the allocated resources, and it runs until either it reaches its planned completion or until the wallclock time (if speci?ed) is up. While the job is in the queue, a number of optional commands are available. qstat Show the status of the job queue. This command is available at any time, whether or not you have a job in the queue. qdel job_id Delete job job_id regardless of its current state and remove it from the queue. qhold job_id Place a non-running job on hold. The job remains in the queue but will not execute. This command cannot be used once the job begins running. qrls job_id Release a job that is on hold. 30 S–2396–50Batch Systems and Program Execution [3] qalter job_id Alter the characteristics—name, account, number of requested cores, and so on—of a job in the queue. This command cannot be used once the job begins running. showq (Moab only) Similar to qstat but providing more detail. checkjob job_id (Moab only) Check the status of a job currently in the queue. showstart job_id (Moab only) Show the estimated start time for a job in the queue. showbf (Moab only) Show the current back?ll. This can help you to build small jobs that can be back?lled immediately while you are waiting for the resources to become available for your larger jobs. For more information about batch scripts, see your batch system's user documentation. 3.3 Using aprun The aprun utility launches applications on compute nodes. The utility submits applications to the Application Level Placement Scheduler (ALPS) for placement and execution, forwards the login node environment to the assigned compute nodes, forwards signals, and manages the stdin, stdout, and stderr streams. Verify that you are in a directory mounted on the high-speed parallel ?le system before using the aprun command. In simplest form, the aprun command looks like this: % aprun -n x ./program_name The aprun command supports a large number of options that provide you with a high degree of control over just exactly how your job is placed and executed on the compute nodes. At a minimum, you must use the -n option to specify the number of cores on which to run the job. Note: Remember, you use aprun within the context of a batch session and the maximum size of the job is determined by the resources you requested when you launched the batch session. You cannot use the aprun command to use more resources than you reserved using the qsub command. The aprun and qsub commands support comparable but differently named options. This table lists some of the more commonly used aprun options and their qsub equivalents. S–2396–50 31Cray Application Developer’s Environment User’s Guide Table 1. aprun versus qsub Options aprun Option qsub -l Option Description -n 4 -l mppwidth=4 Width (number of PEs) -d 2 -l mppdepth=2 Depth (number of CPUs hosting OpenMP threads) -N 1 -l mppnppn=1 Number of PEs per node -L 5,6,7 -l mppnodes=\"5,6,7\" Candidate node List -m 1000 -l mppmem=1000 Memory per PE Note: The -B option forces aprun to inherit the values associated with the -n, -d, -N, and -m options from the batch session. The aprun command exits with an error if you specify any of these options and the -B option at the same time. A full discussion of aprun options is beyond the scope of this manual. For more information see the aprun(1) man page, and for detailed explanations and examples, see Workload Management and Application Placement for the Cray Linux Environment. 32 S–2396–50Using Compilers [4] The Cray system supports a variety of compilers from a variety of vendors and support for new compilers and languages is being added on an ongoing basis. The GNU Fortran, C, and C++ compilers are supplied with all systems, while all other compilers are available as optional and separately licensed add-ons. At present, the following compilers from the following vendors are supported on Cray systems. • The Portland Group, Parallel Fortran, C, and C++ • Cray Inc., Cray Compiling Environment (CCE) (Fortran, C, and C++) • PathScale Inc., PathScale Compiler Suite (Fortran and C++) • Intel Inc., Intel Compiler Suite (Fortran and C++) • Chapel Parallel Programming Language The compilers available on your system depend on which products your site administration has chosen to license and install. The different compilers have different strengths and weaknesses, and their relative strengths and weaknesses are constantly changing as new revisions and updates are released. To date we have found that no one compiler is always the best for all possible applications under all possible circumstances, nor does any one compiler always produce faster executable code than any other. 4.1 About Compiler Drivers Because of the multiplicity of possible compilers, Cray supplies compiler drivers, wrapper scripts, and disambiguation man pages. No matter which vendor's compiler module is loaded, always use one of the following commands to invoke the compiler. ftn Invokes the Fortran compiler, regardless of which compiler module is currently loaded. This command links in the fundamental libraries required in order to produce code that can be executed on the Cray compute nodes. For more information, see the ftn(1) man page. cc Invokes the C compiler, regardless of which compiler module is currently loaded. This command links in the fundamental header ?les and libraries required in order to produce code that can be executed on the Cray compute nodes. For more information, see the cc(1) man page. S–2396–50 33Cray Application Developer’s Environment User’s Guide CC Invokes the C++ compiler, regardless of which compiler module is currently loaded. This command links in the fundamental header ?les and libraries required in order to produce code that can be executed on the Cray compute nodes. For more information, see the CC(1) man page. Note that while you always use one of the above commands (either on the command line or in your make ?les) to invoke the compiler, the arguments used with the commands vary according to which compiler module is loaded. For example, the arguments and options supported by the PGI Fortran compiler are different from those supported by the Cray Fortran compiler. Regardless of which compiler module you have loaded, do not use the native compiler commands. For example, if you are using the PGI compiler suite, do not use the pgf95 command to invoke the Fortran compiler. If you do so, your code may appear to compile and link successfully, but it will be linked to the wrong libraries and the resulting program can be executed on login nodes only; it cannot be executed on Cray compute nodes. 4.2 About C/C++ Floating Point Data Types The C/C++ compilers differ on the size of the long double data type. The PGI and CCE compilers de?ne long double as being 8 bytes. All other compilers de?ne the long double as being 16 bytes. Table 2. C/C++ Data Type Sizes Data Type Size in Bytes unsigned char 1 signed char 1 unsigned short 2 signed short 2 unsigned int 4 signed int 4 unsigned long 8 signed long 8 unsigned long long 8 signed long long 8 ?oat 4 34 S–2396–50Using Compilers [4] Data Type Size in Bytes double 8 long double 8 or 16, depending on compiler char * 8 enum 4 4.3 About PGI Compilers Table 3. PGI Compiler Basics Module: PrgEnv-pgi Command: ftn, cc, CC Compiler-speci?c man pages: pgf95(1), pgcc(1), pgCC(1) Note: Compiler-speci?c man pages are available only when the compiler module is loaded. Online help: pgf95 -help, pgcc -help, pgCC -help, Documentation: /opt/pgi/version/linux86-64/version/doc 4.3.1 Known Limitations • When linking in ACML routines, you must compile and link all program units with -Mcache_align or an aggregate option that incorporates -Mcache_align such as fastsse. • The -Mconcur (auto-concurrentization of loops) option is not supported on Cray systems. • The -mprof=mpi, -Mmpi, and -Mscalapack options are not supported. • The PGI debugger, PGDBG, is not supported on Cray systems. • The PGI Compiler Suite does not support the UPC or Coarray Fortran parallel programming models. S–2396–50 35Cray Application Developer’s Environment User’s Guide 4.4 About the Cray Compiling Environment (CCE) Table 4. Cray Compiler Basics Module: PrgEnv-cray Command: ftn, cc, CC Compiler-speci?c man pages: crayftn(1), craycc(1), crayCC(1) Note: Compiler-speci?c man pages are available only when the compiler module is loaded. Online help: None provided Documentation: Cray Fortran Reference Manual, Cray C and C++ Reference Manual 4.4.1 Known Limitations If you use the Cray Fortran compiler with the PETSc (Portable, Extensible Toolkit for Scienti?c Computation) library, either add the directive !dir$ PREPROCESS EXPAND_MACROS to the source code or add the -F option to the ftn command line. 4.5 About PathScale Compilers Table 5. PathScale Compiler Basics Module: PrgEnv-pathscale Command: ftn, cc, CC Compiler-speci?c man pages: pathf95(1), pathf90(1), pathcc(1), pathCC(1), eko(7) Note: Compiler-speci?c man pages are available only when the compiler module is loaded. Online help: pathf95 --help, pathf90 --help, pathcc --help, pathCC --help Documentation: /opt/pathscale/share/doc/version 36 S–2396–50Using Compilers [4] 4.5.1 Known Limitations • The PathScale Compiler Suite does not support the UPC or Coarray Fortran parallel programming models. • The PETSc 3.1.0 library of parallel linear and nonlinear solvers does not support the PathScale Compiler Suite. Use PETSc 3.0.x instead. 4.6 About Intel Compilers Table 6. Intel Compiler Basics Module: PrgEnv-intel Command: ftn, cc, CC Compiler-speci?c man pages: ifort(1), fpp(1), icc(1), icpc(1) Note: Compiler-speci?c man pages are available only when the compiler module is loaded. Online help: ifort --help, icc --help Documentation: /opt/intel/Compiler/version/Documentation/language 4.6.1 Known Limitations • The Intel Compiler Suite must be installed in the default location. The optional Intel C/C++ only installation is not supported because the Intel Fortran run time libraries are required by Cray libraries such as libsci when using the Intel compiler. • The Intel Compiler Suite does not support the UPC or Coarray Fortran parallel programming models. 4.7 About GNU Compilers Table 7. GNU Compiler Basics Module: PrgEnv-gnu Command: ftn, cc, CC Compiler-speci?c man pages: gfortran(1), gcc(1), g++(1) Note: Compiler-speci?c man pages are available only when the compiler module is loaded. Online help: gfortran --help, gcc --help, g++ --help S–2396–50 37Cray Application Developer’s Environment User’s Guide 4.7.1 Known Limitations The GNU compilers do not support the UPC or Coarray Fortran parallel programming models. 4.8 About the Chapel Parallel Programming Language Chapel is a new parallel programming language being developed by Cray Inc. as part of the DARPA-led High Productivity Computing Systems program (HPCS). Chapel is designed to improve the productivity of high-end computer users while also serving as a portable parallel programming model that can be used on commodity clusters or desktop multicore systems. Chapel strives to vastly improve the programmability of large-scale parallel computers while matching or beating the performance and portability of current programming models like MPI. Chapel supports a multithreaded execution model via high-level abstractions for data parallelism, task parallelism, concurrency, and nested parallelism. Chapel's locale type enables users to specify and reason about the placement of data and tasks on a target architecture in order to tune for locality. Chapel supports global-view data aggregates with user-de?ned implementations, permitting operations on distributed data structures to be expressed in a natural manner. In contrast to many previous higher-level parallel languages, Chapel is designed around a multiresolution philosophy, permitting users to initially write very abstract code and then incrementally add more detail until they are as close to the machine as their needs require. Chapel supports code reuse and rapid prototyping via object-oriented design, type inference, and features for generic programming. Chapel was designed from ?rst principles rather than by extending an existing language. It is an imperative block-structured language, designed to be easy to learn for users of C, C++, Fortran, Java, Perl, Matlab, and other popular languages. While Chapel builds on concepts and syntax from many previous languages, its parallel features are most directly in?uenced by ZPL, High-Performance Fortran (HPF), and the Cray MTA extensions to C and Fortran. For more information about Chapel, see: http://chapel.cray.com. 4.9 About Cross-compilers The Cray system supports using standalone Linux workstations as code development platforms. When the Cray Application Developer's Environment (CADE) is installed on a suitable Linux system, programs can be written and compiled on the Linux system and then exported to the Cray system for subsequent execution, debugging, and optimization. 38 S–2396–50Using Compilers [4] You cannot simply install CADE on any Linux system, however. Before you begin, note the following considerations. • The Linux system must meet the minimum hardware requirements listed in the Cray Application Developer's Environment Supplement Installation Guide. • The Linux system must be running a version of the SLES operating system corresponding to the Cray system for which you are developing code. If the Cray system is running CLE 2.1 or 2.2, the Linux system must be running SLES 10. If the Cray system is running CLE 3.0 or 3.1, the Linux system must be running SLES 11. If you attempt to cross operating system versions, your code will compile successfully on the Linux system but it will not run on the Cray system. • The behavior of CADE when installed on Linux distributions other than SLES 10 or SLES 11 is untested and therefore unsupported. • With the exception of the GNU compilers, all compilers are licensed and installed separately. See your site administrator or compiler vendor for further information about license management. • Along with CADE, you must install the Cray Application Developer's Environment Supplement (CADES, or sometimes CADEsup), which requires satisfying a number of dependencies that are contingent on the compilers you are installing. These dependencies are described in the Cray Application Developer's Environment Supplement Installation Guide. • You must install and con?gure modules on the Linux system. After CADE, CADES, and your compilers are installed and con?gured on your Linux system, follow these steps to begin developing code. 1. Load the xtpe-network module corresponding to the Cray system for which you intend to develop code. If you are developing for a Gemini system (Cray XE5 or Cray XE6), load the xtpe-network-gemini module. If you are developing for a SeaStar system (all others), load the xtpe-network-seastar module. Note: On the actual Cray system, the network type is typically set by default and transparent to the user. When working on a standalone Linux system, you must set this manually. If the correct xtpe-network module is not selected, your code may appear to compile successfully on the Linux system but will not run on the Cray system. S–2396–50 39Cray Application Developer’s Environment User’s Guide 2. Load the xtpe-processor module corresponding to the compute nodes on the Cray system for which you intend to develop code. Your choices are: • xtpe-barcelona (quad core) • xtpe-shanghai (quad core) • xtpe-istanbul (six cores) • xtpe-mc8 (eight cores) • xtpe-mc12 (twelve cores) Note: On the actual Cray system, the processor type is typically set by default and transparent to the user. When working on a standalone Linux system, you must set this manually. If you select a processor type with fewer cores than are actually present on the Cray compute nodes, your code will not make full use of the Cray system resources and may either run slowly or cause con?icts with aprun options and placement. If you select a processor type with more cores than are actually present on the Cray compute nodes, your application may appear to compile successfully but will not run. 3. Load the PrgEnv-vendor module containing your compiler of choice. You are now ready to begin writing, editing, and compiling code. Remember, however, that you must move your code to mount point on the Cray system (for example, /lus/nid00008) before you can run, debug, or optimize it. 40 S–2396–50Libraries [5] Cray provides a large variety of libraries and functions to support application development and interprocess communications on Cray systems. New libraries are being ported to the Cray system on an ongoing basis. The following libraries support all compilers currently supported on the Cray system, except where noted. 5.1 Scienti?c Libraries Table 8. LibSci Basics Module: xt-libsci Man pages: intro_libsci(3s), intro_blas1(3s), intro_blas2(3s), intro_blas3(3s), intro_blacs(3s), intro_lapack(3s), intro_scalapack(3s), intro_irt(3), intro_crafft(3s), intro_fft(3s), intro_fftw2(3), intro_fftw3(3) Note: Library-speci?c man pages are available only when the associated module is loaded. Cray LibSci is a collection of numerical routines optimized for best performance on Cray systems. All programming environment modules load xt-libsci by default. When possible, you should use calls to the Cray LibSci routines in your code in place of calls to public-domain or user-written versions. The Cray LibSci collection contains the following sub-libraries. • BLAS (Basic Linear Algebra Subroutines) • BLACS (Basic Linear Algebra Communication Subprograms) • LAPACK (Linear Algebra Routines) • ScaLAPACK (Scalable LAPACK) • IRT (Iterative Re?nement Toolkit) S–2396–50 41Cray Application Developer’s Environment User’s Guide • CRAFFT (Cray Adaptive Fast Fourier Transform Routines) • FFT (Fast Fourier Transform Routines) • FFTW2 (the Fastest Fourier Transforms in the West, release 2) • FFTW3 (the Fastest Fourier Transforms in the West, release 3) 5.1.1 BLAS and LAPACK The BLAS (Basic Linear Algebra Subroutines) library contains three levels of optimized subroutines. Level 1 BLAS perform the following types of basic vector-vector operations: • Dot products and various vector norms • Scaling, copying, swapping, and computing linear combination of vector • Generate or apply plane or modi?ed plane rotations For more information, see the intro_blas1(3s) man page. Level 2 BLAS perform matrix-vector operations, and generally produce improved code performance when inlined. For more information about Level 2 BLAS, see the intro_blas2(3s) man page. Level 3 BLAS perform matrix-matrix operations. For more information about Level 3 BLAS, see the intro_blas3(3s) man page. LAPACK is a public domain library of subroutines for solving dense linear algebra problems, including the following: • Systems of linear equations • Linear least squares problems • Eigenvalue problems • Singular value decomposition (SVD) problems LAPACK is the successor to the older LINPACK and EISPACK packages. It extends the functionality of these packages by including equilibration, iterative re?nement, error bounds, and driver routines for linear systems, routines for computing and reordering the Schur factorization, and condition estimation routines for eigenvalue problems. Performance issues are addressed by implementing the most computationally-intensive algorithms using Level 2 and Level 3 BLAS. For more information about LAPACK, see the intro_lapack(3s) man page. 42 S–2396–50Libraries [5] 5.1.1.1 Notes • The BLAS and LAPACK libraries include routines from the 64-bit libGoto library from the University of Texas. • BLAS library behavior is dependent on the xtpe-processor module. At most sites this module is typically loaded by default and transparent to the user. However, if your site has multiple types of compute nodes, or if you are working in a Linux cross-compiling environment, it may be necessary to load the xtpe-processor module corresponding to the compute nodes on the Cray system for which you intend to develop code in order to obtain best performance. Your choices are: – xtpe-barcelona (quad core) – xtpe-shanghai (quad core) – xtpe-istanbul (six cores) – xtpe-mc8 (eight cores) – xtpe-mc12 (twelve cores) Note: If you select a processor type with fewer cores than are actually present on the Cray compute nodes, your code will not make full use of the Cray system resources and may either run slowly or cause con?icts with aprun options and placement. If you select a processor type with more cores than are actually present on the Cray compute nodes, your application may appear to compile successfully but will not run. • If you require a C interface to BLAS and LAPACK but want to use Cray LibSci BLAS or LAPACK routines, use the Fortran interfaces. • To obtain threading behavior, set OMP_NUM_THREADS, as described in BLACS and ScaLAPACK on page 44. • You can access the Fortran interfaces from a C program by adding an underscore to the respective routine names and passing arguments by reference (rather than by value). For example, you can call the dgetrf() function as follows: dgetrf_(&uplo, &m, &n, a, &lda, ipiv, work, &lwork, &info); • C programmers using the Fortran interface must order arrays in Fortran column-major order. S–2396–50 43Cray Application Developer’s Environment User’s Guide 5.1.2 BLACS and ScaLAPACK The BLACS (Basic Linear Algebra Communication Subprograms) library is a package of routines that provide the same functionality for message-passing linear algebra communication as the Basic Linear Algebra Subprograms (BLAS) provide for linear algebra computation. With these two packages, software for dense linear algebra can use calls to BLAS for computation and calls to BLACS for communication. The BLACS consist of communication primitives routines, global reduction routines, and support routines. For more information about BLACS, see the intro_blacs(3s) man page. The ScaLAPACK (Scalable LAPACK) library uses BLACS primitives to provide optimized routines for solving real or complex general, triangular, or positive de?nite distributed systems; for reducing distributed matrices to condensed form and an eigenvalue problem solver for real symmetric distributed matrices; and to perform basic operations involving distributed matrices and vectors. For more information about ScaLAPACK, see the intro_scalapack(3s) man page. 5.1.2.1 Notes • Some ScaLAPACK routines require the Basic Linear Algebra Communication Subprograms (BLACS) to be initialized. This can be done through a call to BLACS_GRIDINIT. Also, each distributed array that is passed as an argument to a ScaLAPACK routine requires a descriptor, which is set through a call to DESCINIT. • The ScaLAPACK and BLACS libraries can be used in MPI and SHMEM applications. Cray LibSci also supports hybrid MPI/ScaLAPACK applications, which use threaded BLAS on a compute node and MPI between nodes. To use ScaLAPACK in a hybrid application: 1. Adjust the process grid dimensions in ScaLAPACK to account for the decrease in BLACS nodes. 2. Ensure that the number of BLACS processes required is equal to the number of nodes required, not the number of cores. 3. Set the OMP_NUM_THREADS environment variable. 44 S–2396–50Libraries [5] 5.1.3 IRT The Iterative Re?nement Toolkit (IRT) is a library of Fortran subroutines that provides solutions to linear systems using 32-bit factorizations while preserving accuracy through mixed-precision iterative re?nement. IRT exploits the fact that single-precision solvers can be up to twice as fast as double-precision solvers, and uses an iterative re?nement process to obtain solutions accurate to double-precision. IRT includes both serial and parallel implementations of the LU and Cholesky algorithms, and serial versions of the QR algorithm for real and complex matrices. IRT includes the following features: • Sophisticated stopping criteria • Potential minimization of forward error • Ability to return error bounds • Return an estimate of the condition number of matrix A • Return to the double-precision factorization-and-solve process if IRT cannot obtain a solution IRT provides two interfaces: • Benchmarking interface. The benchmarking interface routines replace the high-level drivers of LAPACK and ScaLAPACK. The names of the benchmark API routines are identical to their LAPACK or ScaLAPACK counterparts or replace calls to successive factorization and solver routines. This allows you to use the IRT process without modifying your application. For example, the IRT dgesv() routine replaces either the LAPACK dgesv() routine or the LAPACK dgetrf() and dgetrs() routines. To use the benchmarking interface, set the IRT_USE_SOLVERS environment variable to 1. Note: Use this interface with caution; calls to the LAPACK LU, QR or Cholesky routines are intercepted and the IRT is used instead. • Expert interface. The expert interface routines give you greater control of the iterative re?nement process and provide details about the success or failure of the process. The format of advanced API calls is: call irt_factorization-method_data-type_processing-mode(arguments) such as: call irt_po_real_parallel(arguments). For more information about IRT, see the intro_irt(3) man page. S–2396–50 45Cray Application Developer’s Environment User’s Guide 5.1.4 FFT Fast Fourier transforms are handled either by using CRAFFT and FFTW or by using ACML. The intro_fft(3s) man page is a disambiguation page. 5.1.4.1 CRAFFT CRAFFT is a library of Fortran subroutines that compute the discrete Fourier transform in one, two, or three dimensions; of arbitrary input size; and of both real and complex data. CRAFFT provides a simpli?ed interface to several FFT libraries and allows automatic, dynamic selection of the fastest FFT kernel. CRAFFT also provides routines for two- and three-dimensional distributed FFT. Note: CRAFFT requires that module fftw/3.2.0 or higher be loaded. To use CRAFFT, add a Fortran use crafft statement to any source ?le that calls a CRAFFT routine, and then call crafft_init() before any other CRAFFT routine. This sets up the CRAFFT library for run-time use. You have a choice of how much planning of FFT kernels you wish to perform. You can set the CRAFFT_PLANNER environment variable to 0, 1 or 2 before execution. Alternately, you can use the crafft_set_planner() and crafft_get_planner() subroutines to alter and query, respectively, the value of CRAFFT_PLANNER during program execution. For more information about using CRAFFT, see the intro_crafft(3s) man page. 5.1.4.2 FFTW The Programming Environment includes versions 2.1.5 and 3.2.x of the Fastest Fourier Transform in the West (FFTW) library. FFTW is a C subroutine library with Fortran interfaces for computing the discrete Fourier transform in one or more dimensions, of arbitrary input size, and of both real and complex data (as well as of even/odd data, such as the discrete cosine/sine transforms). The Fast Fourier Transform algorithm is applied for many problem sizes. Up through LibSci version 10.3.2, loading the LibsSci module automatically loaded the fftw/3.1.1 module. Beginning with xt-libsci modules 10.3.3, FFTW is no longer loaded automatically when you load the LibSci module. Distributed-memory parallel FFTs are available only in FFTW 2.1.5.1. 46 S–2396–50Libraries [5] The FFTW 3.2.x and FFTW 2.1.5.1 modules cannot be loaded at the same time. You must ?rst unload the other module, if already loaded, before loading the desired one. For example, if you have loaded the FFTW 3.2.x library and want to use FFTW 2.1.5.1 instead, use: % module swap fftw/3.2.1 fftw/2.1.5.1 For more information about FFTW, see the intro_fftw2(3) and intro_fftw3(3) man pages. 5.1.4.3 ACML The AMD Core Math Library (ACML) module is no longer loaded as part of the default PrgEnv environment. BLAS and LAPACK functionality is now provided by Cray LibSci. However, if you need ACML for FFT functions, math functions, or random number generators, you can load the library using the acml module: % module load acml ACML includes: • A suite of Fast Fourier Transform (FFT) routines for real and complex data • Fast scalar, vector, and array math transcendental library routines optimized for high performance • A comprehensive random number generator suite: – Base generators plus a user-de?ned generator – Distribution generators – Multiple-stream support ACML's internal timing facility uses the clock() function. If you run an application on compute nodes that uses the plan feature of FFTs, underlying timings will be done using the native version of clock(). On CNL, clock() returns the sum of user and system CPU times. 5.2 About the Fast_mv Library Table 9. Fast_mv Basics Module: libfast Man pages: intro_fast_mv(3), Note: Library-speci?c man pages are available only when the associated module is loaded. S–2396–50 47Cray Application Developer’s Environment User’s Guide Fast_mv is a library of high-performance math intrinsic functions. The functions can be used in PGI, CCE, GNU, and PathScale applications. You can use these functions without changing your programs, or you can call them directly. Note: The use of Fast_mv with Intel compilers is not currently supported. To use Fast_mv functions, load the libfast module: % module load libfast Currently, the library contains: • cos(), the 64-bit cosine function • exp(), the 64-bit exponential function (the constant e raised to a power) • expf(), the 32-bit exponential function (the constant e raised to a power) • fastcos(), the scalar 64-bit cosine • fastsin(), the scalar 64-bit sine • fastsincos(), the scalar 64-bit sine and cosine • frda_exp(), the 64-bit exponential function for all elements in an array • frda_log(), the array version of the 64-bit base e logarithm • frsa_expf(), the 32-bit exponential function for all elements in an array • frsa_logf(), the array version of the 32-bit base e logarithm • log(), the 64-bit logarithm (base e) function • logf(), the 32-bit logarithm (base e) function • sin(), the 64-bit sine function • sincos(), the 64-bit sine and cosine function • vrda_log(), the array version of the 64-bit base e logarithm • __vrd4_log(), the four-element version of the 64-bit base e logarithm • vrsa_logf(), the array version of the 32-bit base e logarithm For more information, see the intro_fast_mv(3) man page. 48 S–2396–50Libraries [5] 5.3 PETSc Table 10. PETSc Basics Module: petsc, petsc-complex Man pages: intro_petsc(3s), Note: Library-speci?c man pages are available only when the associated module is loaded. Web site: http://www.mcs.anl.gov/petsc/petsc-as/ PETSc (Portable, Extensible, Toolkit for Scienti?c Computation) is an open source library of parallel linear and nonlinear equation solvers intended for use in large-scale C, C++, or Fortran applications. PETSc uses standard MPI functions for all message-passing communication. PETSc provides many of the mechanisms needed for parallel applications, such as simple parallel matrix and vector assembly routines that allow the overlap of communication and computation. In addition, PETSc includes support for parallel distributed arrays useful for ?nite difference methods, such as: • Parallel vectors, including code for communicating ghost points • Parallel matrices, including several sparse storage formats • Scalable parallel preconditioners • Krylov subspace methods • Parallel Newton-based nonlinear solvers • Parallel time-stepping ordinary differential equation (ODE) solvers The following packages are included in PETSc 3.1. • MUMPS 4.9.2. MUMPS (MUltifrontal Massively Parallel sparse direct Solver) is a package of parallel, sparse, direct linear-system solvers based on a multifrontal algorithm. For further information, see http://graal.ens-lyon.fr/MUMPS/. • SuperLU 4.0. SuperLU is a sequential version of SuperLU_dist (not included with petsc-complex), and a sequential incomplete LU preconditioner that can accelerate the convergence of Krylov subspace interative solvers. For further information, see http://crd.lbl.gov/~xiaoye/SuperLU/. • SuperLU_dist 2.3. SuperLU_dist is a package of parallel, sparse, direct linear-system solvers (available in Cray LibSci). For further information, see http://crd.lbl.gov/~xiaoye/SuperLU/. S–2396–50 49Cray Application Developer’s Environment User’s Guide • ParMETIS 3.1. ParMETIS (Parallel Graph Partitioning and Fill-reducing Matrix Ordering) is a library of routines that partition unstructured graphs and meshes and compute ?ll-reducing orderings of sparse matrices. For further information, see http://glaros.dtc.umn.edu/gkhome/views/metis/. • HYPRE 2.6.0b. HYPRE is a library of high-performance preconditioners that use parallel multigrid methods for both structured and unstructured grid problems (not included with petsc-complex). For further information, see http://www.llnl.gov/CASC/linear_solvers/. Note: Although you can access these packages individually, Cray supports their use only through the PETSc interface. 5.3.1 Notes • If you use PETSc with the Cray Fortran compiler, either add the directive !dir$ PREPROCESS EXPAND_MACROS to the source code or add the -F option to the ftn command line. • The PathScale Compiler Suite does not support PETSc 3.1. If you are using the PathScale compilers, use PETSc 3.0. • The solvers in Cray PETSc 3.1 are heavily optimized using the Cray Adaptive Sparse Kernels (CASK) library. CASK is an auto-tuned library within the Cray PETSc package that is transparent to the application developer, but improves the performance of most PETSc iterative solvers. You can expect the largest performance improvements when using blocked matrices (BAIJ or SBAIJ), but may also see large gains when using standard compressed sparse row (CSR) AIJ PETSc matrices. 5.4 Trilinos Table 11. Trilinos Basics Module: trilinos Man pages: intro_trilinos(1), Note: Library-speci?c man pages are available only when the associated module is loaded. Web site: http://trilinos.sandia.gov/ 50 S–2396–50Libraries [5] Trilinos is a separate module that provides abstract, object-oriented interfaces to established libraries such as PETSc, Metis/ParMetis, SuperLU, Aztec, BLAS, and LAPACK. Trilinos also includes a set of Cray Adaptive Sparse Kernels (CASK) that perform SpMV, and include optimized versions of single- and multiple-vector matrix vector multiplies. The Trilinos module is dependent on the petsc, xt-libsci, and xt-asyncpe modules. Make certain these modules are loaded before using Trilinos. To use the Trilinos packages, load your compiling environment of choice, and then load the Trilinos module. % module load trilinos After you load the Trilinos module, all header and library locations are set automatically and you are ready to compile your code. No Trilinos-speci?c linking information is required on the command line. If linking to more than one Trilinos package, the libraries are linked automatically in the correct order of package dependency. For more information about link order, see see http://trilinos.sandia.gov/packages/interoperability.html. 5.5 MPT Table 12. MPT Basics Module: xt-mpt, xt-mpich2, xt-shmem Man pages: intro_mpi(3), intro_shmem(3) Note: Library-speci?c man pages are available only when the associated module is loaded. The Cray Message Passing Toolkit (MPT) consists of two components. • MPI • SHMEM Support for MPI and SHMEM varies signi?cantly depending on whether you are using a Cray XE system with Gemini interconnect or a Cray XT system with SeaStar interconnect, and whether your code uses static or dynamic libraries. Because of these issues, Cray provides a variety of different MPI and SHMEM modules. These modules are hardware-dependent, so your system administrator will install only the modules that support your hardware on your system. S–2396–50 51Cray Application Developer’s Environment User’s Guide Nonetheless, there are very signi?cant differences in the ways that MPI and SHMEM are implemented on Cray XE and Cray XT systems, and these differences are re?ected in the functions that are available, and most visibly in the environment variables that are available on the two types of systems. To see which functions and environment variables are supported on your system, always check the intro_mpi(3) and intro_shmem(3) man pages. Furthermore, there are differences in the ways that MPI and SHMEM are implemented in order to support code with static or dynamic libraries. 5.5.1 Static Libraries If your code uses static libraries, load the xt-mpt module. > module load xt-mpt Regardless of whether you are working on an Cray XE and Cray XT system, always use the xt-mpt module with static libraries. 5.5.2 Dynamic Libraries If your code uses dynamic, shared libraries, unload the xt-mpt module and then load either the xt-mpich2 or xt-shmem module, depending on whether your code uses MPI or SHMEM message-passing. > module unload xt-mpt > module load [xt-mpich2|xt-shmem] Regardless of whether you are working on an Cray XE and Cray XT system, always use either the xt-mpich2 or xt-shmem module with dynamic libraries. 52 S–2396–50Debugging Code [6] The Cray system supports a variety of debugging options ranging from simple command-line debuggers to separately licensed third-party GUI tools, and capable of performing a variety of tasks ranging from analyzing core ?les to setting breakpoints and debugging running parallel programs. As a rule, your code must be compiled using the -g command line option before you can use any of the debuggers to produce meaningful information. However, if you are using both a compiler and a debugger that support Fast-Track Debugging, the -g option is replaced by using the -G fast option. Note: The PGI debugger, PGDBG, is not supported on Cray systems. 6.1 Cray Debugger Support Tools Cray provides a collection of basic debugging packages that are referred to collectively as the Cray Debugger Support Tools and installed as a single rpm, but loaded and used as individual modules. These packages are: • lgdb: a modi?ed version of gdb that interfaces with aprun and works on Cray system compute nodes • atp: an optional system that monitors user applications and replaces the core dump with a more comprehensive stack backtrace and analysis • MRNet: a software overlay network that provides multicast and reduction communications for parallel and distributed tools and systems (Deferred implementation) • STAT: stack trace analysis tool (Deferred implementation) S–2396–50 53Cray Application Developer’s Environment User’s Guide 6.1.1 Using lgdb Table 13. lgdb Basics Module: xt-lgdb Command: lgdb Man page: lgdb(1), Note: Tool-speci?c man pages are available only when the associated module is loaded. Online help: lgdb --help The lgdb command is used to launch an application and related gdb server processes on remote nodes for debugging purposes. After lgdb is launched, it provides directions regarding how to run the gdb debugger and attach to application processes. When using lgdb on a Cray system, remember that you use lgdb to launch an instance of aprun, which in turn launches the application to be debugged. You cannot use lgdb to launch an application directly. Example 1. Launching and debugging a single-rank application $ lgdb --pes=0 --command="aprun -n1 ./myapp" --lib=/lib64/libthread_db.so.1 You must specify the number of PEs (ranks) to which you want to attach gdbservers. Since this is a single-rank application, specify 0 for the rank. The --lib=path_to_library is optional and needs to be speci?ed only for certain systems, such as X86_64 Linux, where the libthread_db library is required. Example 2. Attaching to an already-running application $ lgdb --pes=0 --pid=aprun_pid --lib=/lib64/libthread_db.so.1 Remember, you use lgdb to launch an instance of aprun, which in turn launches the application to be debugged. Therefore to attach to an already-running application, you specify the pid of the instance of aprun that launched the application, not the application itself. For more information, see the lgdb(1) man page. 54 S–2396–50Debugging Code [6] 6.1.2 Using Abnormal Termination Processing (ATP) Table 14. atp Basics Module: atp Commands: ataprun, atpFrontend Man page: intro_atp(1), Note: Tool-speci?c man pages are available only when the associated module is loaded. Online help: None required Abnormal Termination Processing (ATP) is an optional system that monitors user applications. When the atp module is loaded, ATP is launched when a job is started and delivers a heuristically determined set of core ?les in the event of an application crash. If an application takes a system trap, ATP performs analysis on the dying application. All stack backtraces of the application processes are gathered into a merged stack backtrace tree and written to disk as the ?le, atpMergedBT.dot. The stack backtrace tree for the ?rst process to die is sent to stderr as is the number of the signal that caused the application to fail. The atpMergedBT.dot ?le can be viewed with statview, (the Stack Trace Analysis Tool viewer) or alternatively with the ?le viewer dotty, which can be found on most Linux systems. The merged stack backtrace tree provides a concise yet comprehensive view of what the application was doing at the time of its termination. ATP is designed to analyze failing applications. It does not play any role with commands. That is, an application must use a supported parallel programming model, such as MPI, SHMEM, OpenMP, CAF, or UPC, in order to bene?t from ATP analysis. When the atp module is loaded, ATP sets the MPICH_ABORT_ON_ERROR, SHMEM_ABORT_ON_ERROR, and DMAPP_ABORT_ON_ERROR environment variables. This enables MPI, SHMEM, and DMAPP applications to raise a signal when they discover usage errors—rather than only printing to stderr and exiting—which therefore enables ATP to notice the problem and perform its analysis. Note: Using ATP disables core dumping. 6.1.2.1 On CLE 3.0 or Later On CLE 3.0 and later systems, if the atp module is loaded, ATP is launched automatically whenever a job is launched by using the aprun command. Do not use the atpaprun or atpFrontend commands on CLE 3.0 or later systems. These commands are for use on CLE 2.2 or earlier systems only. S–2396–50 55Cray Application Developer’s Environment User’s Guide 6.1.2.2 On CLE 2.2 or Earlier On CLE 2.2 and earlier systems, ATP must be invoked manually. This is done by loading the atp module and then using either the atpaprun command to launch your application, or after the application has been launched by using the atpFrontend command to attach to an already-running application. Example 3. Launching an application and invoking ATP atpaprun aprun_parameters Example 4. Attaching ATP to an already-running application atpFrontend application_PID For more information, see the intro_atp(1) man page. 6.1.3 Using MRNet (Deferred implementation) MRNet is a customizable, high-throughput communication software system for parallel tools and applications with a master/slave architecture. MRNet reduces the cost of these tools' activities by incorporating a tree-based overlay network (TBON) of processes between the tool's front-end and back-ends. MRNet uses the TBON to distribute many important tool communication and computation activities, reducing analysis time and keeping tool front-end loads manageable. MRNet-based tools send data between front-end and back-ends on logical ?ows of data called streams. MRNet internal processes use ?lters to synchronize and aggregate data sent to the tool's front-end. Using ?lters to manipulate data in parallel as it passes through the network, MRNet can ef?ciently compute averages, sums, and other more complex aggregations on back-end data. For more information, see: http://www.paradyn.org/mrnet/. 6.1.4 Using STAT (Deferred implementation) STAT (Stack Trace Analysis Tool) gathers and merges the stack traces from a parallel application's processes and produces 2D spatial and 3D spatial-temporal call graphs that encode the calling behavior of the application processes in the form of a pre?x tree. The 2D graph represents a single snapshot of the entire application, while the 3D form represents a series of snapshots from the application taken over time. For more information, see: http://www.paradyn.org/STAT/STAT.html. 56 S–2396–50Debugging Code [6] 6.2 Using Fast-Track Debugging Normally code must be compiled using the -g option before it can be debugged using a conventional debugger. The -g option typically disables all compiler optimizations, producing an executable that contains full DWARF information and can be breakpointed, stepped-through, or paused and restarted anywhere, but at the cost of a much larger and far slower-running program. Cray Fast-Track Debugging signi?cantly increases the speed of the debugging process by producing executables that both contain full DWARF information and run at optimized-code speed. Essentially this is done by producing two parallel executables: one that is fully optimized, and another that is not. While this combined executable is considerably larger than a normal executable, when it is executed under the control of a debugger that supports fast-track debugging, it runs at optimized-code speed until it hits a break point—at which time it switches to the unoptimized code, and allows you to set breakpoints, examine registers, pause, resume, and step through the code as if the entire program was compiled using the -g option. As far as the debugger is concerned, there are no user interface changes, aside from the possibility that you might pursue a backtrace far enough back to begin seeing internal names instead of user names for variables. (For example, instead of foo, you might see debug$foo.) All the work involved in using fast-track debugging is done on the compiler side. Example 5. Using Cray Fast-Track Debugging Using Cray Fast-Track Debugging is a two-step process, requiring both a compiler and a debugger that support fast-track debugging. The steps are: 1. Compile your program using your compiler's fast-track debugging option. For example, if you are using the Cray Fortran compiler, compile and link your code using the -G fast option: users/yourname> ftn -Gfast myapp.f 2. Execute your program using your debugger's normal control method. For example, if you are using the lgdb command line debugger to debug a single-rank application: $ lgdb --pes=0 --command="aprun -n1 ./myapp" Once the debugging session is launched, fast-track debugging is transparent to the user. Sections of the code that contain breakpoints execute slowly, as if compiled using the -g option. Other sections of the code that do not contain breakpoints execute at normal speed, as if compiled using normal optimizations. S–2396–50 57Cray Application Developer’s Environment User’s Guide 6.2.1 Supported Compilers and Debuggers At this time, Cray Fast-Track Debugging is supported on the front end by the Cray Compiling Environment (CCE) compilers. On the back end, Cray Fast-Track Debugging is supported by the lgdb and Allinea DDT debuggers. 6.3 About Core Files When an application fails, one core ?le is generated for the ?rst failing process. If a ?le named core already exists in the current working directory, it is overwritten. On large MPP systems where an application might be running thousands of processes, a conventional core ?le may not indicate the actual cause of the failure. For this reason Cray systems support Abnormal Termination Processing (ATP). If ATP is enabled, a failing application generates a heuristically determined set of data, and in place of a core ?le, all stack backtraces are gathered into a merged stack backtrace tree and written to disk as the ?le, atpMergedBT.dot. The stack backtrace tree for the ?rst process to die is sent to stderr as is the number of the signal that caused the application to fail. For more information about ATP, see Using Abnormal Termination Processing (ATP) on page 55. 6.4 Using TotalView Table 15. TotalView Basics Module: xt-totalview Commands: totalview, totalviewcli Man page: totalview(1), Note: Tool-speci?c man pages are available only when the associated module is loaded. Online help: The TotalView GUI contains an extensive HTML online help system but requires that $TV_HTMLHELP_VIEWER be de?ned before use. For more information, see the Totalview documentation. Documentation: /opt/totalview/version/doc TotalView is an optional product from TotalView Technologies, LLC, that provides source-level debugging of applications running on multiple compute nodes. TotalView is compatible with the Cray, PGI, GCC, PathScale, and Intel compilers. 58 S–2396–50Debugging Code [6] TotalView can be launched in either or two modes: in GUI mode (using the totalview command), or in command-line mode (using the totalviewcli command). TotalView is typically run interactively. If your site has not designated any compute nodes for interactive processing, use the qsub -I command to reserve the number of compute nodes you want to use in interactive mode. Example 6. Using TotalView to control program execution To debug an application on the Cray system, use TotalView to launch aprun, which in turn launches the application to be debugged. users/yourname> totalview aprun -a [aprun_arguments] ./myapp [myapp_arguments] The -a option is a TotalView option indicating that the arguments that follow apply to aprun, not TotalView. Example 7. Debugging a core ?le To use TotalView to examine a core ?le, use the totalview command to launch the GUI. Then, in the New Program window, click the Open a core ?le button and use the browse functions to ?nd, select, and open the core ?le you want to examine. Example 8. Attaching TotalView to a running process To attach TotalView to a running process, you must be logged in to the same login node that you used to launch the process, and then you must attach to the instance of aprun that was used to launch the process, not the process itself. To do so: 1. Use the totalview command to launch the GUI. 2. In the New Program window, click the Attach to process button. The list of processes currently running displays. 3. Select the instance of aprun that you want, and click OK. TotalView displays a process window showing both aprun and the program threads that were launched by that instance of aprun. S–2396–50 59Cray Application Developer’s Environment User’s Guide 6.4.1 Known Limitations The TotalView debugging suite for Cray systems differs in functionality from the standard TotalView implementation. It does not support: • Debugging MPI_Spawn(), OpenMP, or Cray SHMEM programs. • Compiled EVAL points and expressions. • Type transformations for the PGI C++ compiler standard template library collection classes. • Exception handling for the PGI C++ compiler run time library. • Spawning a process onto the compute processors. • Machine partitioning schemes, gang scheduling, or batch systems. 6.5 Using DDT Table 16. DDT Basics Module: ddt Commands: ddt Man page: ddt(1), Note: Tool-speci?c man pages are available only when the associated module is loaded. Online help: The DDT GUI includes an extensive online help system accessible by selecting Help from the menu. If you have problems displaying the help, see the DDT User Guide for information about con?guring X-Windows forwarding and VNC connections. Documentation: /opt/cray/ddt/version/doc DDT is an optional product from Allinea Software, and is a scalable debugger with a graphical user interface. It can be used to debug Fortran, C, and C++ programs, including MPI and OpenMP code, and to launch and debug programs, attach to already running programs, or open and debug core ?les. DDT is compatible with the Cray, PGI, GCC, PathScale, and Intel compilers. The DDT GUI requires either X-Windows forwarding or VNC in order to work. DDT can be used either within an interactive shell or via the batch system. Submission through a batch system requires the use of template ?les that specify the batch parameters. Sample template ?les are found in /opt/cray/ddt/version/templates. 60 S–2396–50Debugging Code [6] In an interactive shell, the fastest way to launch DDT is by entering the ddt command: users/yourname> ddt Assuming X-Windows forwarding is con?gured correctly, the main program window displays, along with a pop-up menu offering the following options. • Run and Debug a Program • Debug a Multi-Process Non-MPI Program • Attach to a Running Program • Open a Core File • Restore a Checkpoint • Cancel Select the option you want to use, or Cancel to close the pop-up menu and proceed to the DDT main window. All the above options are also available through the Sessions menu Run option. 6.5.1 Known Limitations DDT has a number of defaults that affect batch queue submission behavior. When you begin a DDT session, either by selecting Run and Debug a Program from the pop-up Welcome menu or by selecting Run from the Session menu in the DDT main window, the Queue Submission Mode window displays. If you use a batch queuing system such as PBS Pro, always verify the Queue Submission Parameters before proceeding. In particular, verify that the default Queue name and Procs Per Node match your system's con?guration. The default Queue name is generally site-speci?c, while the Procs Per Node value must match your system's processor types: dual-core, quad-core, and so on. If you need to change the Queue Submission Parameters, click the Change button on the Queue Submission Mode window to do so for the duration of the current session. Alternatively, you can create a template ?le that stores your preferred parameters. Instructions for creating and using template ?les are provided in the DDT User Guide. To change your system's default Queue Submission Parameters for all users, contact your site administrator. Default con?guration information is stored in the /opt/cray/ddt/version/default-config.ddt ?le. This information includes the name of the default template, which currently is /opt/cray/ddt/version/templates/xt4.qtf. The actual default queue submission parameters are speci?ed in the default template ?le. S–2396–50 61Cray Application Developer’s Environment User’s Guide 62 S–2396–50Performance Analysis [7] Table 17. Performance Analysis Basics Module: perftools Commands: pat_build, pat_report, app2 Man pages: intro_craypat(1), pat_build(1), pat_report(1), pat_help(1), app2(1), intro_papi(3), Note: Tool-speci?c man pages are available only when the associated module is loaded. Online help: CrayPat includes an extensive online help system that features many examples and the answers to many frequently asked questions. To access the help system, enter pat_help at the command line. Documentation: Using Cray Performance Analysis Tools After you have compiled and debugged your program, you are ready to begin analyzing its performance. The Cray Performance Analysis Tools are a suite of optional utilities that enable you to capture and analyze performance data generated during the execution of your program in order to help you to ?nd answers to two fundamental questions: How fast is my program running? and How can I make it run faster? The Cray Performance Analysis Tools suite consists of three components: • CrayPat: the program instrumentation, data capture, and basic text reporting tool • Cray Apprentice2: the graphical analysis and data visualization tool • PAPI: the Performance Application Programming Interface Beginning with release 5.1.0, the Cray Performance Analysis Tools are bundled into a single module. To begin working with the performance analysis tools, ?rst load your programming environment of choice, and then load the perftools module. users/yourname> module load perftools S–2396–50 63Cray Application Developer’s Environment User’s Guide The performance analysis process consists of three basic steps. 1. Instrument your program, to specify what kind of data you want to collect under what conditions. 2. Execute your instrumented program, to generate and capture the desired data. 3. Analyze the resulting data. Accordingly, CrayPat consists of the following major components: • pat_build, the utility used to instrument programs • the CrayPat run time environment, which collects the speci?ed performance data • pat_report, the ?rst-level analysis tool used to produce text reports or export data for more sophisticated analysis • pat_help, the command-line driven online help system All CrayPat components, including the man pages and help system, are available only when the perftools module is loaded. 7.1 Using CrayPat For successful results, the perftools module must be loaded before you compile the program to be instrumented, instrument the program, execute the instrumented program, or generate a report. If you want to instrument a program that was compiled before the perftools module was loaded, you may under some circumstances ?nd that re-linking is suf?cient, but as a rule it's best to load the perftools module and then recompile. When instrumenting a program, CrayPat requires that the object (.o) ?les created during compilation be present, as well as the library (.a) ?les, if any. However, most compilers automatically delete the .o and .a ?les when working with single source ?les and compiling and linking in a single step, therefore it is good practice to compile and link in separate steps and use the compiler command line option to preserve these ?les. For example, if you are using the Cray Compiling Environment (CCE) Fortran compiler, compile using either of these command line options: > ftn -c source?le.f Alternatively: > ftn -h keepfiles source?le.f Then link the object ?les to create the executable program: > ftn -o executable source?le.o 64 S–2396–50Performance Analysis [7] 7.1.1 Instrumenting the Program After the program is compiled and linked, use the pat_build command to instrument the program for performance analysis. In simplest form, pat_build is used like this: > pat_build -O apa executable This produces a copy of your original program, which is named executable+pat (for example, a.out+pat) and instrumented for the default experiment. Your original executable remains untouched. The pat_build command supports a large number of options and directives, including an API that enables you to instrument speci?ed regions of your code. These options and directives are summarized in the pat_build(1) man page and documented more extensively in Using Cray Performance Analysis Tools. 7.1.2 Collecting Data Instrumented programs are executed just like any other program; either by using the aprun command if your site permits interactive sessions or by using your system's batch commands. CrayPat supports more than ?fty optional run time environment variables that enable you to control instrumented program behavior and data collection during execution. For example, if you use the C shell and want to collect data in detail rather than in aggregate, consider setting the PAT_RT_SUMMARY environment variable to 0 (off) before launching your program. /lus/nid00008> setenv PAT_RT_SUMMARY 0 Doing so records data with timestamps, which makes additional reports available in Cray Apprentice2, but at the cost of potentially much larger data ?le sizes and somewhat increased overhead. The CrayPat run time environment variables are summarized in the intro_craypat(1) man page and documented more extensively in Using Cray Performance Analysis Tools. 7.1.3 Analyzing Data Assuming your instrumented program runs to completion or planned termination, CrayPat outputs one or more data ?les. The exact number, location, and content of the data ?le(s) varies depending on the nature of your program, the type of experiment for which it was instrumented, and the run time environment variable settings in effect at the time of program execution. S–2396–50 65Cray Application Developer’s Environment User’s Guide All initial data ?les are output in .xf format, with a generated ?le name consisting of your original program name, plus pat, plus the execution process ID number, plus a code string indicating the type of data contained within the ?le. Depending on the program run and the types of data collected, CrayPat output may consist of either a single .xf data ?le or a directory containing multiple .xf data ?les. If the program was instrumented with the -O apa option, a ?le with the suf?x .apa is also generated. This ?le is a customized template for this program and is created for use with future instrumentation experiments. To begin analyzing the captured data, use the pat_report command. In simplest form, it looks like this: /lus/nid00008> pat_report myprog+pat+PIDem-n.xf The pat_report command accepts either a ?le or directory name as input and processes the .xf ?le(s) to generate a text report. In addition, it also exports the .xf data to a single .ap2 ?le, which is both a self-contained archive that can be reopened later using the pat_report command and the exported-data ?le format used by Cray Apprentice2. The pat_report command provides more than thirty prede?ned report templates, as well as a large variety of user-con?gurable options. These reports and options are summarized in the pat_report(1) man page and documented more extensively in Using Cray Performance Analysis Tools. 7.1.4 For more information In addition to Using Cray Performance Analysis Tools and the intro_craypat(1), pat_build(1), and pat_report(1) man pages, there is a substantial amount of information, including an FAQ and examples, in the CrayPat online help system. The help system is accessible whenever the perftools module is loaded; to access the help system, enter pat_help at the command line. For more information about using the help system, see the pat_help(1) man page. 7.2 Using Cray Apprentice2 Cray Apprentice2 is an optional GUI tool that is used to visualize and manipulate the performance analysis data captured during program execution. Cray Apprentice2 can be run either on the Cray system or, optionally, on a standalone Linux desktop machine. Cray Apprentice2 can display a wide variety of reports and graphs, depending on the type of program being analyzed, the way in which the program was instrumented for data capture, and the data that was collected during program execution. 66 S–2396–50Performance Analysis [7] Cray Apprentice2 is not directly integrated with CrayPat. You cannot launch Cray Apprentice2 from within CrayPat, nor can you set up or run performance analysis experiments from within Cray Apprentice2. Rather, use CrayPat ?rst, to instrument your program and capture performance analysis data, and then use Cray Apprentice2 to visualize and explore the resulting data ?les. The number and appearance of the reports that can be generated using Cray Apprentice2 is determined by the kind and quantity of data captured during program execution, which in turn is determined by the way in which the program was instrumented and the environment variables in effect at the time of program execution. For example, changing the PAT_RT_SUMMARY environment variable to 0 before executing the instrumented program nearly doubles the number of reports available when analyzing the resulting data in Cray Apprentice2. To run Cray Apprentice2, load the perftools module, if it is not already loaded. users/yourname> module load perftools Then use the app2 command to launch Cray Apprentice2. users/yourname> app2 [data?le.ap2] & Note: Cray Apprentice2 requires that your workstation be con?gured to host X Window System sessions. If the app2 command returns an "unable to open display" error, contact your system administrator for help in con?guring X Window System hosting and forwarding. At this point the GUI takes over. If you speci?ed a data ?le name with the app2 command, the ?le is opened and parsed and the Overview report is displayed. If you did not specify a data ?le name, the Open File window opens and you can use standard GUI tools to browse through the ?le system and select the data ?le you want to open. For more information about using Cray Apprentice2, see the app2(1) man page, the Cray Apprentice2 help system, and Using Cray Performance Analysis Tools. 7.3 Using PAPI The Performance API (PAPI) is a standard API for accessing microprocessor registers. CrayPat uses PAPI to interface to the Cray system hardware; therefore the module xt-papi is normally loaded as part of the perftools module. The interface between PAPI and CrayPat is normally transparent to the user. However, advanced users may want to bypass CrayPat and work with PAPI directly. In this case, you must unload the perftools module and then reload only the xt-papi module. > module unload perftools > module load xt-papi S–2396–50 67Cray Application Developer’s Environment User’s Guide Note: CrayPat and direct PAPI commands cannot be used at the same time. Therefore, pat_build does not allow you to instrument a program that has also been instrumented using calls to PAPI functions. For more information about PAPI, see the intro_papi(3) and papi_counters(5) man pages. To see what sorts of information PAPI is capable of capturing on your system, use the papi_avail and papi_native_avail commands. Note: Effective with PAPI version 3.7.2, The PAPI Group has discontinued providing man pages for individual PAPI functions. Additional information about using PAPI is available through the PAPI website, at http://icl.cs.utk.edu/papi/. 68 S–2396–50Glossary blade 1) A ?eld-replaceable physical entity. A service blade consists of AMD Opteron sockets, memory, Cray network application-speci?c integrated circuit (ASIC) chips, PCI cards, and a blade control processor. A compute blade consists of AMD Opteron sockets, memory, Cray network application-speci?c integrated circuit (ASIC) chips, and a blade control processor. 2) From a system management perspective, a logical grouping of nodes and blade control processor that monitors the nodes on that blade. Catamount The operating system kernel developed by Sandia National Laboratories and implemented to run on Cray XT single-core compute nodes. See also Catamount Virtual Node (CVN); compute node. Catamount Virtual Node (CVN) The Catamount kernel enhanced to run on dual-core Cray XT compute nodes. CLE The operating system for Cray XE and Cray XT systems. CNL The CLE compute node kernel. CNL provides a set of supported system calls. CNL provides many of the operating system functions available through the service nodes, although some functionality has been removed to improve performance and reduce memory usage by the system. compute node A node that runs application programs. A compute node performs only computation; system services cannot run on compute nodes. Compute nodes run a speci?ed kernel to support either scalar or vector applications. See also node; service node. S–2396–50 69Cray Application Developer’s Environment User’s Guide deferred implementation The label used to introduce information about a feature that will not be implemented until a later release. login node The service node that provides a user interface and services for compiling and running applications. module See blade. multicore A processor that combines multiple independent execution engines ("cores"), each with its own cache and cache controller. node For CLE systems, the logical group of processor(s), memory, and network components acting as a network end point on the system interconnection network. service node A node that performs support functions for applications and system services. Service nodes run a version of SLES and perform specialized functions. There are six types of prede?ned service nodes: login, IO, network, boot, database, and syslog. 70 S–2396–50 AMD Core Math Library (ACML) Version 4.3.0 Copyright c 2003-2009 Advanced Micro Devices, Inc., Numerical Algorithms Group Ltd. AMD, the AMD Arrow logo, AMD Opteron, AMD Athlon and combinations thereof are trademarks of Advanced Micro Devices, Inc.i Short Contents 1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 General Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3 BLAS: Basic Linear Algebra Subprograms . . . . . . . . . . . . . 19 4 LAPACK: Package of Linear Algebra Subroutines . . . . . . . . 20 5 Fast Fourier Transforms (FFTs) . . . . . . . . . . . . . . . . . . . . 24 6 Random Number Generators. . . . . . . . . . . . . . . . . . . . . . . 75 7 ACML MV: Fast Math and Fast Vector Math Library . . . . 163 8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 Subject Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 Routine Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231ii Table of Contents 1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 General Information . . . . . . . . . . . . . . . . . . . . . . . 2 2.1 Determining the best ACML version for your system . . . . . . . . . . . 2 2.2 Accessing the Library (Linux). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2.1 Accessing the Library under Linux using GNU gfortran/gcc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2.2 Accessing the Library under Linux using PGI compilers pgf77/pgf90/pgcc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2.3 Accessing the Library under Linux using Open64 compilers openf95/opencc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2.4 Accessing the Library under Linux using the NAGWare f95 compiler. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2.5 Accessing the Library under Linux using the Intel ifort compiler. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2.6 Accessing the Library under Linux using Absoft af90 . . . . . . 7 2.2.7 Accessing the Library under Linux using compilers other than GNU, PGI, Open64, NAGWare, Intel or Absoft . . . . . . . . . . . . . 7 2.3 Accessing the Library (Microsoft Windows) . . . . . . . . . . . . . . . . . . . 8 2.3.1 Accessing the Library under 32-bit Windows using PGI compilers pgf77/pgf90/Microsoft C . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3.2 Accessing the Library under 32-bit Windows using Microsoft C or Intel Fortran. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3.3 Accessing the Library under 32-bit Windows using the Compaq Visual Fortran compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3.4 Accessing the Library under 32-bit Windows using the Salford FTN95 compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3.5 Accessing the Library under 64-bit Windows using PGI compilers pgf77/pgf90/pgcc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3.6 Accessing the Library under 64-bit Windows using Microsoft C or Intel Fortran. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.4 Accessing the Library (Sun compiler under Solaris or Linux). . . 12 2.4.1 Accessing the Library using the Sun compiler under Solaris or Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5 ACML FORTRAN and C interfaces . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.6 ACML variants using 64-bit integer (INTEGER*8) arguments. . 14 2.7 Library Version and Build Information . . . . . . . . . . . . . . . . . . . . . . . 15 2.8 Library Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.9 Example programs calling ACML. . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.10 Example ACML programs demonstrating performance . . . . . . . 16 2.11 ACML Memory Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3 BLAS: Basic Linear Algebra Subprograms . . 19iii 4 LAPACK: Package of Linear Algebra Subroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.1 Introduction to LAPACK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.2 Reference sources for LAPACK. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.3 LAPACK block sizes, ILAENV and ILAENVSET. . . . . . . . . . . . . 21 4.4 IEEE exceptions and LAPACK. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 5 Fast Fourier Transforms (FFTs) . . . . . . . . . . . 24 5.1 Introduction to FFTs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 5.1.1 Transform de?nitions and Storage for Complex Data . . . . . 24 5.1.2 Transform de?nitions and Storage for Real Data . . . . . . . . . 25 5.1.3 E?ciency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 5.1.4 Default and Generated Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 5.2 FFTs on Complex Sequences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5.2.1 FFT of a single sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 ZFFT1D Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 CFFT1D Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 ZFFT1DX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 CFFT1DX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 5.2.2 FFT of multiple complex sequences . . . . . . . . . . . . . . . . . . . . . 34 ZFFT1M Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 CFFT1M Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 ZFFT1MX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 CFFT1MX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.2.3 2D FFT of two-dimensional arrays of data . . . . . . . . . . . . . . . 43 ZFFT2D Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 CFFT2D Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 ZFFT2DX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 CFFT2DX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 5.2.4 3D FFT of three-dimensional arrays of data . . . . . . . . . . . . . 52 ZFFT3D Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 CFFT3D Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 ZFFT3DX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 CFFT3DX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 ZFFT3DY Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 CFFT3DY Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.3 FFTs on real and Hermitian data sequences . . . . . . . . . . . . . . . . . . 66 5.3.1 FFT of single sequences of real data. . . . . . . . . . . . . . . . . . . . . 67 DZFFT Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 SCFFT Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 5.3.2 FFT of multiple sequences of real data . . . . . . . . . . . . . . . . . . 69 DZFFTM Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 SCFFTM Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 5.3.3 FFT of single Hermitian sequences. . . . . . . . . . . . . . . . . . . . . . 71 ZDFFT Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 CSFFT Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 5.3.4 FFT of multiple Hermitian sequences. . . . . . . . . . . . . . . . . . . . 73 ZDFFTM Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73iv CSFFTM Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 6 Random Number Generators . . . . . . . . . . . . . . 75 6.1 Base Generators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 6.1.1 Initialization of the Base Generators . . . . . . . . . . . . . . . . . . . . 76 DRANDINITIALIZE / SRANDINITIALIZE . . . . . . . . . . . . . . . . . . . . . . 78 DRANDINITIALIZEBBS / SRANDINITIALIZEBBS . . . . . . . . . . . . . . . 81 6.1.2 Calling the Base Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 DRANDBLUMBLUMSHUB / SRANDBLUMBLUMSHUB . . . . . . . . . . . . . . . . . 83 6.1.3 Basic NAG Generator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 6.1.4 Wichmann-Hill Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 6.1.5 Mersenne Twister. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 6.1.6 L’Ecuyer’s Combined Recursive Generator. . . . . . . . . . . . . . . 85 6.1.7 Blum-Blum-Shub Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 6.1.8 User Supplied Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 DRANDINITIALIZEUSER / SRANDINITIALIZEUSER . . . . . . . . . . . . . 87 UINI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 UGEN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 6.2 Multiple Streams. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 6.2.1 Using Di?erent Seeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6.2.2 Using Di?erent Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6.2.3 Skip Ahead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 DRANDSKIPAHEAD / SRANDSKIPAHEAD . . . . . . . . . . . . . . . . . . . . . . . . 92 6.2.4 Leap Frogging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 DRANDLEAPFROG / SRANDLEAPFROG . . . . . . . . . . . . . . . . . . . . . . . . . . 95 6.3 Distribution Generators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 6.3.1 Continuous Univariate Distributions. . . . . . . . . . . . . . . . . . . . . 97 DRANDBETA / SRANDBETA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 DRANDCAUCHY / SRANDCAUCHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 DRANDCHISQUARED / SRANDCHISQUARED . . . . . . . . . . . . . . . . . . . . . 101 DRANDEXPONENTIAL / SRANDEXPONENTIAL. . . . . . . . . . . . . . . . . . . 103 DRANDF / SRANDF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 DRANDGAMMA / SRANDGAMMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 DRANDGAUSSIAN / DRANDGAUSSIAN . . . . . . . . . . . . . . . . . . . . . . . . . 109 DRANDLOGISTIC / SRANDLOGISTIC . . . . . . . . . . . . . . . . . . . . . . . . . 111 DRANDLOGNORMAL / SRANDLOGNORMAL . . . . . . . . . . . . . . . . . . . . . . . 113 DRANDSTUDENTST / SRANDSTUDENTST . . . . . . . . . . . . . . . . . . . . . . . 115 DRANDTRIANGULAR / SRANDTRIANGULAR . . . . . . . . . . . . . . . . . . . . . 117 DRANDUNIFORM / SRANDUNIFORM. . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 DRANDVONMISES / SRANDVONMISES . . . . . . . . . . . . . . . . . . . . . . . . . 121 DRANDWEIBULL / SRANDWEIBULL. . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 6.3.2 Discrete Univariate Distributions . . . . . . . . . . . . . . . . . . . . . . 125 DRANDBINOMIAL / SRANDBINOMIAL . . . . . . . . . . . . . . . . . . . . . . . . . 125 DRANDGEOMETRIC / SRANDGEOMETRIC . . . . . . . . . . . . . . . . . . . . . . . 127 DRANDHYPERGEOMETRIC / SRANDHYPERGEOMETRIC . . . . . . . . . . . . 129 DRANDNEGATIVEBINOMIAL / SRANDNEGATIVEBINOMIAL. . . . . . . . 131 DRANDPOISSON / SRANDPOISSON. . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 DRANDDISCRETEUNIFORM / SRANDDISCRETEUNIFORM . . . . . . . . . . 135v DRANDGENERALDISCRETE / SRANDGENERALDISCRETE . . . . . . . . . . 137 DRANDBINOMIALREFERENCE / SRANDBINOMIALREFERENCE . . . . . 139 DRANDGEOMETRICREFERENCE / SRANDGEOMETRICREFERENCE . . . 141 DRANDHYPERGEOMETRICREFERENCE / SRANDHYPERGEOMETRICREFERENCE . . . . . . . . . . . . . . . . . . . . . 143 DRANDNEGATIVEBINOMIALREFERENCE / SRANDNEGATIVEBINOMIALREFERENCE . . . . . . . . . . . . . . . . . . . 145 DRANDPOISSONREFERENCE / SRANDPOISSONREFERENCE. . . . . . . . 147 6.3.3 Continuous Multivariate Distributions. . . . . . . . . . . . . . . . . . 149 DRANDMULTINORMAL / SRANDMULTINORMAL. . . . . . . . . . . . . . . . . . . 149 DRANDMULTISTUDENTST / SRANDMULTISTUDENTST . . . . . . . . . . . . 151 DRANDMULTINORMALR / SRANDMULTINORMALR . . . . . . . . . . . . . . . . 153 DRANDMULTISTUDENTSTR / SRANDMULTISTUDENTSTR . . . . . . . . . . 155 DRANDMULTINORMALREFERENCE / SRANDMULTINORMALREFERENCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 DRANDMULTISTUDENTSTREFERENCE / SRANDMULTISTUDENTSTREFERENCE . . . . . . . . . . . . . . . . . . . . . 159 6.3.4 Discrete Multivariate Distributions. . . . . . . . . . . . . . . . . . . . . 161 DRANDMULTINOMIAL / SRANDMULTINOMIAL. . . . . . . . . . . . . . . . . . . 161 7 ACML MV: Fast Math and Fast Vector Math Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 7.1 Introduction to ACML MV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 7.1.1 Terminology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 7.1.2 Weak Aliases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 7.1.3 De?ned Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 7.2 Fast Basic Math Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 fastcos: fast double precision Cosine . . . . . . . . . . . . . . . . . . . . . . . . . . 165 fastcosf: fast single precision Cosine . . . . . . . . . . . . . . . . . . . . . . . . . . 166 fastexp: fast double precision exponential function . . . . . . . . . . . . . 166 fastexpf: fast single precision exponential function . . . . . . . . . . . . . 168 fastlog: fast double precision natural logarithm function. . . . . . . . 169 fastlogf: fast single precision natural logarithm function . . . . . . . . 170 fastlog10: fast double precision base-10 logarithm function . . . . . 171 fastlog10f: fast single precision base-10 logarithm function . . . . . . 172 fastlog2: fast double precision base-2 logarithm function. . . . . . . . 173 fastlog2f: fast single precision base-2 logarithm function . . . . . . . . 174 fastpow: fast double precision power function. . . . . . . . . . . . . . . . . . 175 fastpowf: fast single precision power function . . . . . . . . . . . . . . . . . . 177 fastsin: fast double precision Sine. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 fastsinf: fast single precision Sine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 fastsincos: fast double precision Sine and Cosine . . . . . . . . . . . . . . . 181 fastsincosf: fast single precision Sine and Cosine . . . . . . . . . . . . . . . 182 7.3 Fast Vector Math Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 vrd2 cos: Two-valued double precision Cosine . . . . . . . . . . . . . . . . . 183 vrd4 cos: Four-valued double precision Cosine . . . . . . . . . . . . . . . . . 184 vrda cos: Array double precision Cosine . . . . . . . . . . . . . . . . . . . . . . 185 vrs4 cosf: Four-valued single precision Cosine . . . . . . . . . . . . . . . . . 186vi vrsa cosf: Array single precision Cosine . . . . . . . . . . . . . . . . . . . . . . . 187 vrd2 exp: Two-valued double precision exponential function. . . . 188 vrd4 exp: Four-valued double precision exponential function . . . 189 vrda exp: Array double precision exponential function . . . . . . . . . 190 vrs4 expf: Four-valued single precision exponential function . . . . 191 vrs8 expf: Eight-valued single precision exponential function . . . 192 vrsa expf: Array single precision exponential function. . . . . . . . . . 193 vrd2 log: Two-valued double precision natural logarithm. . . . . . . 194 vrd4 log: Four-valued double precision natural logarithm. . . . . . . 195 vrda log: Array double precision natural logarithm . . . . . . . . . . . . 196 vrs4 logf: Four-valued single precision natural logarithm . . . . . . . 197 vrs8 logf: Eight-valued single precision natural logarithm . . . . . . 198 vrsa logf: Array single precision natural logarithm. . . . . . . . . . . . . 199 vrd2 log10: Two-valued double precision base-10 logarithm. . . . . 200 vrd4 log10: Four-valued double precision base-10 logarithm . . . . 201 vrda log10: Array double precision base-10 logarithm . . . . . . . . . . 202 vrs4 log10f: Four-valued single precision base-10 logarithm . . . . . 203 vrs8 log10f: Eight-valued single precision base-10 logarithm . . . . 204 vrsa log10f: Array single precision base-10 logarithm. . . . . . . . . . . 205 vrd2 log2: Two-valued double precision base-2 logarithm. . . . . . . 206 vrd4 log2: Four-valued double precision base-2 logarithm . . . . . . 207 vrda log2: Array double precision base-2 logarithm . . . . . . . . . . . . 208 vrs4 log2f: Four-valued single precision base-2 logarithm . . . . . . . 209 vrs8 log2f: Eight-valued single precision base-2 logarithm . . . . . . 210 vrsa log2f: Array single precision base-2 logarithm. . . . . . . . . . . . . 211 vrs4 powf: Four-valued single precision power function . . . . . . . . . 212 vrsa powf: Array single precision power function . . . . . . . . . . . . . . 213 vrs4 powxf: Four-valued single precision power function with constant y. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 vrsa powxf: Array single precision power function, constant y . . 216 vrd2 sin: Two-valued double precision Sine . . . . . . . . . . . . . . . . . . . 218 vrd4 sin: Four-valued double precision Sine . . . . . . . . . . . . . . . . . . . 219 vrda sin: Array double precision Sine . . . . . . . . . . . . . . . . . . . . . . . . . 220 vrs4 sinf: Four-valued single precision Sine . . . . . . . . . . . . . . . . . . . . 221 vrsa sinf: Array single precision Sine . . . . . . . . . . . . . . . . . . . . . . . . . 222 vrd2 sincos: Two-valued double precision Sine and Cosine. . . . . . 223 vrda sincos: Array double precision Sine and Cosine . . . . . . . . . . . 224 vrs4 sincosf: Four-valued single precision Sine and Cosine . . . . . . 225 vrsa sincosf: Array single precision Sine and Cosine. . . . . . . . . . . . 226 8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 Subject Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 Routine Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231Chapter 1: Introduction 1 1 Introduction The AMD Core Math Library (ACML) is a set of numerical routines tuned speci?cally for AMD64 platform processors (including Opteron TM and Athlon TM 64 ). The routines, which are available via both FORTRAN 77 and C interfaces, include: • BLAS - Basic Linear Algebra Subprograms (including Sparse Level 1 BLAS); • LAPACK - A comprehensive package of higher level linear algebra routines; • FFT - a set of Fast Fourier Transform routines for real and complex data; • RNG - a set of random number generators and statistical distribution functions. The BLAS and LAPACK routines provide a portable and standard set of interfaces for common numerical linear algebra operations that allow code containing calls to these routines to be readily ported across platforms. Full documentation for the BLAS and LAPACK are available online. This manual will, therefore, be restricted to providing brief descriptions of the BLAS and LAPACK and providing links to their documentation and other materials (see Chapter 3 [The BLAS], page 19 and see Chapter 4 [LAPACK], page 20). The FFT is an implementation of the Discrete Fourier Transform (DFT) that makes use of symmetries in the de?nition to reduce the number of operations required from O(n*n) to O(n*log n) when the sequence length, n, is the product of small prime factors; in particular, when n is a power of 2. Despite the popularity and widespread use of FFT algorithms, the de?nition of the DFT is not su?ciently precise to prescribe either the forward and backward directions (these are sometimes interchanged), or the scaling factor associated with the forward and backward transforms (the combined forward and backward transforms may only reproduce the original sequence by following a prescribed scaling). Currently, there is no agreed standard API for FFT routines. Hardware vendors usually provide a set of high performance FFTs optimized for their systems: no two vendors employ the same interfaces for their FFT routines. The ACML provides a set of FFT routines, optimized for AMD64 processors, using an ACML-speci?c set of interfaces. The functionality, interfaces and use of the ACML FFT routines are described below (see Chapter 5 [Fast Fourier Transforms], page 24). The RNG is a comprehensive set of statistical distribution functions which are founded on various underlying uniform distribution generators (base generators) including WichmannHill and an implementation of the Mersenne Twister. In addition there are hooks which allow you to supply your own preferred base generator if it is not already included in ACML. All RNG functionality and interfaces are described below (see Chapter 6 [Random Number Generators], page 75). Chapter 2 [General Information], page 2 provides details on: • how to link a user program to the ACML; • FORTRAN and C interfaces to ACML routines; • how to obtain the ACML version and build information; • how to access the ACML documentation. A supplementary library of fast math and fast vector math functions (ACML MV) is also provided with some 64-bit versions of ACML. Some of the functions included in ACML MV are not callable from high-level languages, but must be called via assembly language; the documentation of ACML MV (see Chapter 7 [Fast Vector Math Library], page 163) gives details for each individual routine.Chapter 2: General Information 2 2 General Information 2.1 Determining the best ACML version for your system ACML comes in versions for 64-bit and 32-bit processors, running both Linux and Microsoft Windows R operating systems. To use the following tables, you will need to know answers to these questions: • Are you running a 64-bit operating system (on AMD64 hardware such as Opteron or Athlon64)? Or are you running a 32-bit operating system? • Is the operating system Linux or Microsoft Windows? • Do you have the GNU compilers (gfortran/gcc) or compatible compilers (compilers such as Absoft that are interoperable with the GNU compilers) installed? • Do you have the PGI compilers (pgf77/pgf90/pgcc) installed? • Do you have the Open64 compilers (openf95/opencc) installed? • Do you have the NAGWare compiler (f95) installed? • Do you have the Sun compiler (f95) installed? • On a 32-bit Windows machine, do you have Microsoft C, or PGI Visual Fortran, or Intel Fortran, or compatible compilers installed? • Do you have a single processor system or a multiprocessor (SMP) system? The single processor version of ACML can be run on an SMP machine and vice versa, but (if you have the right compilers) it is more e?cient to run the version appropriate to the machine. • If you’re on a 32-bit machine, does it support Streaming SIMD Extension instructions (SSE and SSE2)? The ACML installation includes a binary utility that can help you ?nd an answer to the last question. The utility lies in directory util, and is named cpuid.exe. It interrogates the processor to determine whether SSE and SSE2 instructions exist. util/cpuid.exe Under a Linux operating system, another way of ?nding out the answer to the last question is to look at the special ?le /proc/cpuinfo, and see what appears under the “?ags” label. Try this command: cat /proc/cpuinfo | grep flags If the list of ?ags includes the ?ag “sse” then your machine supports SSE instructions. If it also includes “sse2” then your machine supports SSE2 instructions. If your machine supports these instructions, it is better to use a version of ACML which was built to take advantage of them, for reasons of good performance. The method of examining /proc/cpuinfo can also be used under Microsoft Windows if you have the Cygwin UNIX-like tools installed (see http://www.cygwin.com/) and run a bash shell. Note that AMD64 machines always support both SSE and SSE2 instructions, under both Linux and Windows. Older (32-bit) AMD chips may support SSE but not SSE2, or neither SSE nor SSE2 instructions. Other manufacturers’ hardware may or may not support SSE or SSE2. If you link to a version of ACML that was built to use SSE or SSE2 instructions, and your machine does not in fact support them, it is likely that your program will halt dueChapter 2: General Information 3 to encountering an “illegal instruction” - you may or may not be noti?ed of this by the operating system. For 32-bit machines, older versions of ACML (ACML 3.1.0 and earlier) came in variants suitable for hardware without SSE/SSE2 instructions (Streaming SIMD Extensions). This is no longer the case, and if you have older 32-bit hardware that does not support SSE/SSE2, and wish to use ACML, you must continue to use an older version. Once you have answered the questions above, use these tables to decide which version of ACML to link against. Linux 64-bit   Number of processors Compilers ACML install directory Single processor PGI pgf77/pgf90/pgcc acml4.3.0/pgi64 ” GNU gfortran/gcc or compat. acml4.3.0/gfortran64 ” Open64 openf95/opencc acml4.3.0/open64_64 ” Intel Fortran acml4.3.0/ifort64 ” NAGWare f95 acml4.3.0/nag64 ” Sun f95 acml4.3.0/sun64 ” Absoft (use gfortran ACML) acml4.3.0/gfortran64 Multi processor or core PGI pgf77/pgf90/pgcc acml4.3.0/pgi64_mp ” GNU gfortran/gcc or compat. acml4.3.0/gfortran64_mp ” Intel Fortran acml4.3.0/ifort64_mp ” Sun f95 acml4.3.0/sun64_mp ” Absoft (use gfortran ACML) acml4.3.0/gfortran64_mp Linux 32-bit   Number of processors Compilers ACML install directory Single processor PGI pgf77 / pgf90 / pgcc acml4.3.0/pgi32 ” GNU gfortran / gcc or compat. acml4.3.0/gfortran32 ” Open64 openf95 / opencc acml4.3.0/open64_32 ” Intel Fortran acml4.3.0/ifort32 ” NAGWare f95 acml4.3.0/nag32 ” Sun f95 acml4.3.0/sun32 ” Absoft (use gfortran ACML) acml4.3.0/gfortran32 Multi processor or core PGI pgf77 / pgf90 / pgcc acml4.3.0/pgi32_mp ” GNU gfortran / gcc or compat. acml4.3.0/gfortran32_mp ” Intel Fortran acml4.3.0/ifort32_mp ” Sun f95 acml4.3.0/sun32_mp ” Absoft (use gfortran ACML) acml4.3.0/gfortran32_mp Chapter 2: General Information 4 Microsoft Windows 64-bit   Number of processors Compilers ACML install directory Single processor PGI pgf77/pgf90/pgcc/MSC acml4.3.0/win64 ” Intel Fortran/Microsoft C acml4.3.0/ifort64 Multi processor or core PGI pgf77/pgf90/pgcc/MSC acml4.3.0/win64_mp ” Intel Fortran/Microsoft C acml4.3.0/ifort64_mp Microsoft Windows 32-bit   Number of processors Compilers ACML install directory Single processor PGI pgf77/pgf90/Microsoft C acml4.3.0/pgi32 ” Intel Fortran/Microsoft C acml4.3.0/ifort32 Multi processor or core PGI pgf77/pgf90/Microsoft C acml4.3.0/pgi32_mp ” Intel Fortran/Microsoft C acml4.3.0/ifort32_mp 2.2 Accessing the Library (Linux) 2.2.1 Accessing the Library under Linux using GNU gfortran/gcc If the Linux 64-bit gfortran version of ACML was installed in the default directory, /opt/acml4.3.0/gfortran64, then the command: gfortran -m64 driver.f -L/opt/acml4.3.0/gfortran64/lib -lacml can be used to compile the program driver.f and link it to the ACML. The ACML Library is supplied in both static and shareable versions, libacml.a and libacml.so, respectively. By default, the commands given above will link to the shareable version of the library, libacml.so, if that exists in the directory speci?ed. Linking with the static library can be forced either by using the compiler ?ag -static, e.g. gfortran -m64 driver.f -L/opt/acml4.3.0/gfortran64/lib -static -lacml -lrt or by inserting the name of the static library explicitly in the command line, e.g. gfortran -m64 driver.f /opt/acml4.3.0/gfortran64/lib/libacml.a -lrt Notice that if the application program has been linked to the shareable ACML Library, then before running the program, the environment variable LD_LIBRARY_PATH must be set. Assuming that libacml.so was installed in the directory /opt/acml4.3.0/gfortran64/lib, then LD_LIBRARY_PATH may be set by, for example, the C-shell command setenv LD_LIBRARY_PATH /opt/acml4.3.0/gfortran64/lib (See the man page for ld(1) for more information about LD_LIBRARY_PATH.) The command gfortran -m32 driver.f -L/opt/acml4.3.0/gfortran32/lib -lacml will compile and link a 32-bit program with a 32-bit ACML. If you have an SMP machine and want to take best advantage of it, link against the gfortran OpenMP version of ACML like this:Chapter 2: General Information 5 gfortran -fopenmp -m64 driver.f -L/opt/acml4.3.0/gfortran64_mp/lib -lacml_mp gfortran -fopenmp -m32 driver.f -L/opt/acml4.3.0/gfortran32_mp/lib -lacml_mp Note that the directories and library names involved now include the su?x mp. To compile and link a 64-bit C program with a 64-bit ACML, invoke gcc -m64 -I/opt/acml4.3.0/gfortran64/include driver.c -L/opt/acml4.3.0/gfortran64/lib -lacml -lgfortran The switch "-I/opt/acml4.3.0/gfortran64/include" tells the compiler to search the directory /opt/acml4.3.0/gfortran64/include for the ACML C header ?le acml.h, which should be included by driver.c. Note that it is necessary to add the gfortran compiler run-time library -lgfortran when linking the program. 2.2.2 Accessing the Library under Linux using PGI compilers pgf77/pgf90/pgcc Similar commands apply for the PGI versions of ACML. For example, pgf77 -tp=k8-64 -Mcache_align driver.f -L/opt/acml4.3.0/pgi64/lib -lacml pgf77 -tp=k8-32 -Mcache_align driver.f -L/opt/acml4.3.0/pgi32/lib -lacml will compile driver.f and link it to the ACML using 64-bit and 32-bit versions respectively. In the example above we are linking with the single-processor PGI version of ACML. If you have an SMP machine and want to take best advantage of it, link against the PGI OpenMP version of ACML like this: pgf77 -tp=k8-64 -mp -Mcache_align driver.f -L/opt/acml4.3.0/pgi64_mp/lib -lacml_mp pgf77 -tp=k8-32 -mp -Mcache_align driver.f -L/opt/acml4.3.0/pgi32_mp/lib -lacml_mp Note that the directories and library names involved now include the su?x mp. The -mp ?ag is important - it tells pgf77 to link with the appropriate compiler OpenMP run-time library. Without it you might get an "unresolved symbol" message at link time. The -Mcache align ?ag is also important - it tells the compiler to align objects on cache-line boundaries. The commands pgcc -c -tp=k8-64 -mp -Mcache_align -I/opt/acml4.3.0/pgi64_mp/include driver.c pgcc -tp=k8-64 -mp -Mcache_align driver.o -L/opt/acml4.3.0/pgi64_mp/lib -lacml_mp -lpgftnrtl -lm will compile driver.c and link it to the 64-bit ACML. Again, the -mp ?ag is important if you are linking to the PGI OpenMP version of ACML. The C compiler is instructed to search the directory /opt/acml4.3.0/pgi64 mp/include for the ACML C header ?le acml.h, which should be included by driver.c, by using the switch "-I/opt/acml4.3.0/pgi64 mp/include". Note that in the example we add the libraries -lpgftnrtl and -lm to the link command, so that required PGI compiler run-time libraries are found. Note that since ACML version 3.5.0, all PGI 64-bit variants are compiled with the PGI -Mlarge arrays switch to allow use of larger data arrays (see PGI compiler documentation for more information). The special ’large array’ variants that were distributed with earlier versions of ACML are therefore no longer required.Chapter 2: General Information 6 2.2.3 Accessing the Library under Linux using Open64 compilers openf95/opencc Similar commands apply for the Open64 versions of ACML. For example, openf95 driver.f -L/opt/acml4.3.0/open64_64/lib -lacml will compile driver.f and link it to the ACML using the 64-bit version. The commands opencc -c -I/opt/acml4.3.0/open64_64/include driver.c opencc driver.o -L/opt/acml4.3.0/open64_64/lib -lacml -lfortran will compile driver.c and link it to the 64-bit ACML. The switch -I/opt/acml4.3.0/open64_64/include tells the C compiler to search directory /opt/acml4.3.0/open64 64/include for the ACML C header ?le acml.h, which should be included by driver.c. Note that in the example we add the library -lfortran to the link command, so that the required Open64 compiler run-time library is found. 2.2.4 Accessing the Library under Linux using the NAGWare f95 compiler Similar commands apply for the NAGware f95 versions of ACML. For example, f95 driver.f -L/opt/acml4.3.0/nag64/lib -lacml f95 -32 driver.f -L/opt/acml4.3.0/nag32/lib -lacml will compile driver.f and link it to the ACML using the 64-bit version and 32-bit version respectively. 2.2.5 Accessing the Library under Linux using the Intel ifort compiler Similar commands apply for the Intel ifort versions of ACML. For example, ifort driver.f -L/opt/acml4.3.0/ifort64/lib -lacml will compile driver.f and link it to the ACML using the 64-bit version. The commands gcc -c -I/opt/acml4.3.0/ifort64/include driver.c ifort -nofor-main driver.o -L/opt/acml4.3.0/ifort64/lib -lacml will compile driver.c and link it to the 64-bit ACML. The switch -I/opt/acml4.3.0/ifort64/include tells the C compiler to search directory /opt/acml4.3.0/ifort64/include for the ACML C header ?le acml.h, which should be included by driver.c. Note that in the example we link the C program using the ifort compiler with the -nofor-main switch, so that required ifort compiler run-time libraries are found. If you have an SMP machine and want to take best advantage of it, link against the ifort OpenMP version of ACML like this: ifort -openmp driver.f -L/opt/acml4.3.0/ifort64_mp/lib -lacml_mp ifort -openmp driver.f -L/opt/acml4.3.0/ifort32_mp/lib -lacml_mp Note that the directories and library names involved now include the su?x mp. The -openmp ?ag is important - it tells ifort to link with the appropriate compiler OpenMP run-time library. Without it you might get an "unresolved symbol" message at link time.Chapter 2: General Information 7 2.2.6 Accessing the Library under Linux using Absoft af90 The Absoft compiler af90 is compatible with the GNU compiler gfortran version of ACML, so long as the appropriate gfortran run-time libraries are installed on your system. If the Linux 64-bit gfortran version of ACML was installed in the default directory, /opt/acml4.3.0/gfortran64, then the command: af90 -m64 driver.f -L/opt/acml4.3.0/gfortran64/lib -lacml -lgfortran can be used to compile the program driver.f and link it to the ACML. Note that -gfortran links to the gfortran run-time library, which must be installed on your system. The ACML Library is supplied in both static and shareable versions, libacml.a and libacml.so, respectively. By default, the commands given above will link to the shareable version of the library, libacml.so, if that exists in the directory speci?ed. Linking with the static library can be forced either by using the compiler ?ag -static, e.g. af90 -m64 driver.f -L/opt/acml4.3.0/gfortran64/lib -static \ -lacml -lgfortran -lrt or by inserting the name of the static library explicitly in the command line, e.g. af90 -m64 driver.f /opt/acml4.3.0/gfortran64/lib/libacml.a -lgfortran -lrt Notice that if the application program has been linked to the shareable ACML Library, then before running the program, the environment variable LD_LIBRARY_PATH must be set. Assuming that libacml.so was installed in the directory /opt/acml4.3.0/gfortran64/lib, then LD_LIBRARY_PATH may be set by, for example, the C-shell command setenv LD_LIBRARY_PATH /opt/acml4.3.0/gfortran64/lib (See the man page for ld(1) for more information about LD_LIBRARY_PATH.) The command af90 -m32 driver.f -L/opt/acml4.3.0/gfortran32/lib -lacml -lgfortran will compile and link a 32-bit program with a 32-bit ACML. If you have an SMP machine and want to take best advantage of it, link against the gfortran OpenMP version of ACML like this: af90 -m64 driver.f -L/opt/acml4.3.0/gfortran64_mp/lib -lacml_mp -lgfortran -lgomp af90 -m32 driver.f -L/opt/acml4.3.0/gfortran32_mp/lib -lacml_mp -lgfortran -lgomp Note that the directories and library names involved now include the su?x mp. Also note that it is necessary to link to the gfortran run-time libraries -lgfortran -lgomp, both of which must be installed on your system. 2.2.7 Accessing the Library under Linux using compilers other than GNU, PGI, Open64, NAGWare, Intel or Absoft It may be possible to link to some versions of ACML using compilers other than those already mentioned, if they are compatible with one of the other versions. If you do this, it may be necessary to link to the run-time library of the compiler used to build the ACML you link to, in order to satisfy run-time symbols. Since doing this is very compiler-speci?c, we give no further details here.Chapter 2: General Information 8 2.3 Accessing the Library (Microsoft Windows) 2.3.1 Accessing the Library under 32-bit Windows using PGI compilers pgf77/pgf90/Microsoft C To use the 32-bit Windows PGI version of ACML, use a command like pgf77 -Mdll -Munix driver.f "c:\AMD\acml4.3.0\pgi32\lib\libacml_dll.lib" where libacml dll.lib is the import library for the ACML DLL. Note that it is important to use the compiler switch -Munix in order to tell the compiler to use the same calling convention as was used to build ACML. In the example above we are linking with the single-processor PGI version of ACML. If you have an SMP machine and want to take best advantage of it, link against the PGI OpenMP version of ACML like this: pgf77 -Mdll -Munix -mp driver.f "c:\AMD\acml4.3.0\pgi32_mp\lib\libacml_mp_dll.lib" Note that the directories and library names involved now include the su?x mp. For the OpenMP version of ACML, if you link to the static library libacml mp.lib rather than the DLL import library libacml mp dll.lib, you will need to use the PGI compiler ?ag -mp in order to tell the compiler to link with the appropriate compiler OpenMP run-time library. Without it you might get an "unresolved symbol" message at link time. This should not be necessary when linking to the ACML DLL because the DLL itself knows that it depends on the run-time library; but using the -mp ?ag in any case will do no harm. To compile and link a C program using the Microsoft C command line compiler, cl, the commands cl "-Ic:\AMD\acml4.3.0\pgi32\include" /MD driver.c "c:\AMD\acml4.3.0\pgi32\lib\libacml_dll.lib" cl "-Ic:\AMD\acml4.3.0\pgi32_mp\include" /MD driver.c "c:\AMD\acml4.3.0\pgi32_mp\lib\libacml_mp_dll.lib" will link against the single-threaded DLL and multi-threaded versions of ACML respectively. 2.3.2 Accessing the Library under 32-bit Windows using Microsoft C or Intel Fortran To use the 32-bit Windows MSC/Intel Fortran version of ACML, use a command like ifort /threads /libs:dll driver.f "c:\AMD\acml4.3.0\ifort32\lib\libacml_dll.lib" where libacml dll.lib is the import library for the ACML DLL. In the example above we are linking with the single-processor ifort version of ACML. If you have an SMP machine and want to take best advantage of it, link against the ifort OpenMP version of ACML like this:Chapter 2: General Information 9 ifort /libs:dll -Qopenmp driver.f c:\acml4.3.0\ifort32_mp\lib\libacml_mp_dll.lib Note that the directories and library names involved now include the su?x mp. For the OpenMP version of ACML, if you link to the static library libacml mp.lib rather than the DLL import library libacml mp dll.lib, you will need to use the ifort compiler ?ag -Qopenmp in order to tell the compiler to link with the appropriate compiler OpenMP run-time library. Without it you might get an "unresolved symbol" message at link time. This should not be necessary when linking to the ACML DLL because the DLL itself knows that it depends on the run-time library; but using the -Qopenmp ?ag in any case will do no harm. To compile and link a C program using the Microsoft C command line compiler, cl, the commands cl "-Ic:\AMD\acml4.3.0\ifort32\include" /MD driver.c "c:\AMD\acml4.3.0\ifort32\lib\libacml_dll.lib" cl "-Ic:\AMD\acml4.3.0\ifort32_mp\include" /MD driver.c "c:\AMD\acml4.3.0\ifort32_mp\lib\libacml_mp_dll.lib" will link against the single-threaded DLL and multi-threaded versions of ACML respectively. ACML can also be linked from inside a development environment such as Microsoft Visual Studio or Visual Studio.NET. Again, it is important to get compilation options correct. The directory acml4.3.0\ifort32\examples\Projects contains a few sample Visual Studio project directories showing how this can be done. Note that in both examples above we linked to a DLL version of ACML, and so before running the resulting programs the environment variable PATH must be set to include the location of the DLL. For example, assuming that libacml dll.dll was installed in "c:\AMD\acml4.3.0\ifort32\lib", PATH may be set by, for example, the DOS command PATH="c:\AMD\acml4.3.0\ifort32\lib";%PATH% Alternatively, the PATH environment variable may be set in the system category of the Windows control panel. ACML also comes as a static (non-DLL) library, named libacml.lib, in the same directory as the DLL. If you link to the static library instead of the DLL import library then there is no need to set the PATH. 2.3.3 Accessing the Library under 32-bit Windows using the Compaq Visual Fortran compiler The win32 Intel Fortran variant of ACML can be used with the Compaq Visual Fortran compiler as follows: f90 /iface:cref,nomixed_str_len_arg /threads /libs:dll driver.f "c:\AMD\acml4.3.0\ifort32\lib\libacml_dll.lib" where f90 is the Compaq Visual Fortran command line compiler and libacml dll.lib is the import library for the ACML DLL. The switch /iface:cref,nomixed str len arg used on the f90 compiler command line is important - it tells the compiler to use a calling convention equivalent to the default Intel Fortran calling convention, rather than the default cvf stdcall calling convention. If you forget to use this switch your program is likely to crash on execution.Chapter 2: General Information 10 2.3.4 Accessing the Library under 32-bit Windows using the Salford FTN95 compiler The win32 Intel Fortran variant of ACML can be used with the Salford ftn95 compiler as follows: ftn95 driver.f The resulting object ?le can be linked using the Salford linker, slink, for example like this: slink driver.obj install_dir\libacml_dll.dll where install dir is the location of the DLL. The full pathname of install dir should be speci?ed to the DLL and should be enclosed within quotes if it contains spaces. It is worth emphasising that the linker should link directly against the DLL itself, not the libacml dll.lib import library. 2.3.5 Accessing the Library under 64-bit Windows using PGI compilers pgf77/pgf90/pgcc Under 64-bit versions of Windows, ACML 4.3.0 comes as a static (.LIB) library or a DLL. To link with the 64-bit Windows DLL library PGI version of ACML, in a DOS command prompt use a command like pgf77 -Mdll driver.f c:/acml4.3.0/win64/lib/libacml_dll.lib where libacml dll.lib is the import library for the DLL. In the example above we are linking with the single-processor WIN64 version of ACML. If you have an SMP machine and want to take best advantage of it, link against the WIN64 OpenMP version of ACML like this: pgf77 -Mdll -mp driver.f c:/acml4.3.0/win64_mp/lib/libacml_mp_dll.lib Note that the directories and library names involved now include the su?x mp. For the OpenMP version of ACML, if you link to the static library libacml mp.lib rather than the DLL import library libacml mp dll.lib, you will need to use the PGI compiler ?ag -mp in order to tell the compiler to link with the appropriate compiler OpenMP run-time library. Without it you might get an "unresolved symbol" message at link time. This should not be necessary when linking to the ACML DLL because the DLL itself knows that it depends on the run-time library; but using the -mp ?ag in any case will do no harm. Note that the performance of OpenMP code produced with the PGI WIN64 compilers depends on environment variables named MP BIND and MP SPIN, which control how multiple threads behave (see PGI compiler documentation for discussion of these variables). For ACML, empirical experiments show that higher values of MP SPIN than the default are likely to give better performance. We recommend that users set MP BIND=yes and MP SPIN=100000000. Under WIN64, to compile and link a C program, the commands pgcc -Mdll driver.c -Ic:/acml4.3.0/win64/include c:/acml4.3.0/win64/lib/libacml_dll.lib pgcc -Mdll -mp driver.c -Ic:/acml4.3.0/win64_mp/include c:/acml4.3.0/win64_mp/lib/libacml_mp_dll.lib will link against the single-threaded DLL and multi-threaded versions of ACML respectively. To use the Microsoft C command line compiler, cl, use commands like this:Chapter 2: General Information 11 cl driver.c -Ic:/acml4.3.0/win64/include c:/acml4.3.0/win64/lib/libacml_dll.lib cl driver.c -Ic:/acml4.3.0/win64_mp/include c:/acml4.3.0/win64_mp/lib/libacml_mp_dll.lib for single- and multi-threaded ACML variants respectively. 2.3.6 Accessing the Library under 64-bit Windows using Microsoft C or Intel Fortran Under 64-bit versions of Windows, ACML 4.3.0 comes as a static (.LIB) library or a DLL. To link with the 64-bit Windows DLL library Intel Fortran version of ACML, in a DOS command prompt use a command like ifort /libs:dll driver.f c:\acml4.3.0\ifort64\lib\libacml_dll.lib where libacml dll.lib is the import library for the DLL. In the example above we are linking with the single-processor ifort version of ACML. If you have an SMP machine and want to take best advantage of it, link against the ifort OpenMP version of ACML like this: ifort /libs:dll -Qopenmp driver.f c:\acml4.3.0\win64_mp\lib\libacml_mp_dll.lib Note that the directories and library names involved now include the su?x mp. For the OpenMP version of ACML, if you link to the static library libacml mp.lib rather than the DLL import library libacml mp dll.lib, you will need to use the ifort compiler ?ag -Qopenmp in order to tell the compiler to link with the appropriate compiler OpenMP run-time library. Without it you might get an "unresolved symbol" message at link time. This should not be necessary when linking to the ACML DLL because the DLL itself knows that it depends on the run-time library; but using the -Qopenmp ?ag in any case will do no harm. Under WIN64, to compile and link a C program using the Microsoft C command line compiler, cl, the commands cl driver.c -Ic:/acml4.3.0/ifort64/include c:/acml4.3.0/ifort64/lib/libacml_dll.lib cl driver.c -Ic:/acml4.3.0/ifort64_mp/include c:/acml4.3.0/ifort64_mp/lib/libacml_mp_dll.lib will link against the single-threaded DLL and multi-threaded versions of ACML respectively.Chapter 2: General Information 12 2.4 Accessing the Library (Sun compiler under Solaris or Linux) 2.4.1 Accessing the Library using the Sun compiler under Solaris or Linux If the Sun f95 64-bit version of ACML was installed in the default directory, /opt/acml4.3.0/sun64, then the command: f95 -m64 driver.f -L/opt/acml4.3.0/sun64/lib -lacml can be used to compile the program driver.f and link it to the ACML. The ACML Library is supplied in both static and shareable versions, libacml.a and libacml.so, respectively. By default, the commands given above will link to the shareable version of the library, libacml.so, if that exists in the directory speci?ed. Linking with the static library can be forced either by using the compiler ?ag -Bstatic, e.g. f95 -m64 driver.f -L/opt/acml4.3.0/sun64/lib -Bstatic -lacml or by inserting the name of the static library explicitly in the command line, e.g. f95 -m64 driver.f /opt/acml4.3.0/sun64/lib/libacml.a Notice that if the application program has been linked to the shareable ACML Library, then before running the program, the environment variable LD_LIBRARY_PATH must be set, for example, by the C-shell command: setenv LD_LIBRARY_PATH /opt/acml4.3.0/sun64/lib where it is assumed that libacml.so was installed in the directory /opt/acml4.3.0/sun64/lib (see the man page for ld(1) for more information about LD_LIBRARY_PATH.). The command f95 -m32 driver.f -L/opt/acml4.3.0/sun32/lib -lacml will compile and link a 32-bit program with a 32-bit ACML. To compile and link a 64-bit C program with a 64-bit ACML, invoke cc -m64 -I/opt/acml4.3.0/sun64/include driver.c -L/opt/acml4.3.0/sun64/lib -lacml -lfsu -lsunmath -lm The switch "-I/opt/acml4.3.0/sun64/include" tells the compiler to search the directory /opt/acml4.3.0/sun64/include for the ACML C header ?le acml.h, which should be included by driver.c. Note that it is necessary to add the Sun compiler run-time libraries -lfsu -lsunmath -lm when linking the program. If you have an SMP machine and want to take best advantage of it, link against the Sun OpenMP version of ACML like this: f95 -xopenmp -m64 driver.f -L/opt/acml4.3.0/sun64_mp/lib -lacml_mp f95 -xopenmp -m32 driver.f -L/opt/acml4.3.0/sun32_mp/lib -lacml_mp Note that the directories and library names involved now include the su?x mp. The -xopenmp ?ag is important - it tells f95 to link with the appropriate compiler OpenMP run-time library. Without it you might get an "unresolved symbol" message at link time. The commandChapter 2: General Information 13 cc -xopenmp -m64 -I/opt/acml4.3.0/sun64/include driver.c -L/opt/acml4.3.0/sun64/lib -lacml_mp -lfsu -lsunmath -lm -lmtsk will compile driver.c and link it to the 64-bit ACML. Again, the -xopenmp ?ag is important if you are linking to the OpenMP version of ACML. The C compiler is instructed to search the directory /opt/acml4.3.0/sun64 mp/include for the ACML C header ?le acml.h, which should be included by driver.c, by using the switch "-I/opt/acml4.3.0/sun64 mp/include". Note that in the example we add the libraries -lfsu -lsunmath -lm -lmtsk to the link command, so that required compiler run-time libraries are found. 2.5 ACML FORTRAN and C interfaces All routines in ACML come with both FORTRAN and C interfaces. The FORTRAN interfaces typically follow the relevant standard (e.g. LAPACK, BLAS). Here we document how a C programmer should call ACML routines. In C code that uses ACML routines, be sure to include the header ?le , which contains function prototypes for all ACML C interfaces. The header ?le also contains C prototypes for FORTRAN interfaces, thus the C programmer could call the FORTRAN interfaces from C, though there is little reason to do so. C interfaces to ACML routines di?er from FORTRAN interfaces in the following major respects: • The FORTRAN interface names are appended by an underscore (except for the Windows 32-bit and 64-bit Microsoft C/Intel Fortran version of ACML, where FORTRAN interface names are distinguished from C by being upper case rather than lower case - this is the default for the Intel Fortran compiler) • The C interfaces contain no workspace arguments; all workspace memory is allocated internally. • Scalar input arguments are passed by value in C interfaces. FORTRAN interfaces pass all arguments (except for character string length arguments that are normally hidden from FORTRAN programmers) by reference. • Most arguments that are passed as character string pointers to FORTRAN interfaces are passed by value as single characters to C interfaces. The character string length arguments of FORTRAN interfaces are not required in the C interfaces. • Unlike FORTRAN, C has no native complex data type. ACML C routines which operate on complex data use the types complex and doublecomplex de?ned in for single and double precision computations respectively. Some of the programs in the ACML examples directory (see Section 2.9 [Examples], page 16) make use of these types. It is important to note that in both the FORTRAN and C interfaces, 2-dimensional arrays are assumed to be stored in column-major order. e.g. the matrix A =  1.0 2.0 3.0 4.0  would be stored in memory as 1.0, 3.0, 2.0, 4.0. This storage order corresponds to a FORTRAN-style 2-D array declaration A(2,2), but not to an array declared as a[2][2] in C which would be stored in row-major order as 1.0, 2.0, 3.0, 4.0.Chapter 2: General Information 14 As an example, compare the FORTRAN and C interfaces of LAPACK routine dsytrf as implemented in ACML. FORTRAN: void dsytrf_(char *uplo, int *n, double *a, int *lda, int *ipiv, double *work, int *lwork, int *info, int uplo_len); C: void dsytrf(char uplo, int n, double *a, int lda, int *ipiv, int *info); C code calling both the above variants might look like this: double *a; int *ipiv; double *work; int n, lda, lwork, info; /* Assume that all arrays and variables are allocated and initialized as required by dsytrf. */ /* Call the FORTRAN version of dsytrf. The first argument is a character string, and the last argument is the length of that string. The input scalar arguments n, lda and lwork, as well as the output scalar argument info, are all passed by reference. */ dsytrf_("Upper", &n, a, &lda, ipiv, work, &lwork, &info, 5); /* Call the C version of dsytrf. The first argument is a character, workspace is not required, and input scalar arguments n and lda are passed by value. Output scalar argument info is passed by reference. */ dsytrf(’U’, n, a, lda, ipiv, &info); 2.6 ACML variants using 64-bit integer (INTEGER*8) arguments Where compilers support, through the use of switches, the automatic promotion of regular INTEGER (32-bit) arguments to INTEGER*8 (64-bit) arguments, ACML variants exist to use this facility. This means that if you have a 64-bit Fortran program using INTEGER*8 variables, or a 64-bit C program using 8-byte long variables, there is an ACML version that you can use. This applies to 64-bit ACML versions built with PGI, Open64, gfortran, Intel and NAG compilers. The INTEGER*8 versions of these libraries are distinguished from the usual versions by having the string “ int64” as part of the name of the directory under which ACML is installed. Thus, for example, if the regular PGI 64-bit library is in a directory named pgi64, then the INTEGER*8 version will be installed in directory pgi64 int64. For these ACML variants, all ACML documentation that mentions arguments of Fortran type INTEGER or C type int should be read as INTEGER*8 or long respectively. It is important to ensure that if you have INTEGER*8 variables in your code, you link to the int64 variant, and not otherwise. Unexpected program crashes are likely to occur if you link to the wrong version.Chapter 2: General Information 15 2.7 Library Version and Build Information This document is applicable to version 4.3.0 of ACML. The utility routine acmlversion can be called to obtain the major, minor and patch version numbers of the installed ACML. This routine returns three integers; the major, minor and patch version numbers, respectively. The utility routine acmlinfo can be called to obtain information on the compiler used to build ACML, the version of the compiler, and the options used for building the Library. This subroutine takes no arguments and prints the information to the current standard output. FORTRAN speci?cations: ACMLVERSION (MAJOR, MINOR, PATCH) [SUBROUTINE] MAJOR, MINOR, PATCH [INTEGER] ACMLINFO () [SUBROUTINE] C speci?cations: void acmlversion (int *major, int *minor, int *patch); [function] void acmlinfo (void); [function] 2.8 Library Documentation The /Doc subdirectory of the top ACML installation directory, (e.g. /opt/acml4.3.0/Doc under Linux, or c:\AMD\acml4.3.0\Doc under Windows), should contain this document in the following formats: • Printed Manual / PDF format – acml.pdf • Info Pages – acml.info (Linux only) • Html – html/index.html • Plain text – acml.txt Under Linux the info ?le can be read using info after updating the environment variable INFOPATH to include the doc subdirectory of the ACML installation directory, e.g. % setenv INFOPATH ${INFOPATH}:/opt/acml4.3.0/Doc % info acml or simply by using the full name of the ?le: % info /opt/acml4.3.0/Doc/acml.infoChapter 2: General Information 16 2.9 Example programs calling ACML The /examples subdirectory of the top ACML installation directory (for example, possible default locations are /opt/acml4.3.0/pgi64/examples under Linux, or, under windows, c:\acml4.3.0\win64\examples), contains example programs showing how to call the ACML, along with a GNUmake?le to build and run them. Examples of calling both FORTRAN and C interfaces are included. They may be used as an ACML installation test. Depending on where your copy of the ACML is installed, and which compiler and ?ags you wish to use, it may be necessary to modify some variables in the GNUmake?le before using it. The 32-bit Windows versions of ACML assume that you have the Cygwin UNIX-like tools installed, and can use the make command that comes with them to build the examples. For the 64-bit Windows version of ACML, it is not necessary to have the Cygwin tools. The examples directory contains a bat script, acmlexample.bat, which can be used to run one of the example programs. Another bat script, acmlallexamples.bat, builds and runs all the examples in the directory. Alternatively, if you do have the Cygwin tools installed, you can use the GNUmake?le to build the examples. If you need more example programs showing how to call LAPACK routines from Fortran, we refer you to this web page: http://www.nag.com/lapack/ Here you will ?nd examples for all double precision LAPACK driver routines, and all of these should work when linked with ACML. Note that as well as the example programs themselves, it is necessary to download and compile a small amount of utility code used by the programs. See the web page for detailed instructions. 2.10 Example ACML programs demonstrating performance The /examples/performance subdirectory of the top ACML installation directory (for example, possible default locations are /opt/acml4.3.0/pgi64/examples/performance under Linux, or c:\acml4.3.0\win64\examples\performance under windows) contains several timing programs designed to show the performance of ACML when running on your machine. Again, a GNUmake?le may be used to build and run them. Depending on where your copy of the ACML is installed, and which compiler and ?ags you wish to use, it may be necessary to modify some variables in the GNUmake?le before using it. The 32- and 64-bit Windows versions of ACML assume that you have the Cygwin UNIXlike tools installed, and can use the make command that comes with them to build the examples. In addition, the GNUmake?le uses the gnuplot plotting program to display graphs of the timing results. If you do not have gnuplot installed, the timing programs will still run and show their results, but you will see no graph plots. Under linux, gnuplot may come with your linux distribution, but you may need to explicitly ask for it to be installed. Note that version 4.0 or later of gnuplot is required. The gnuplot program is also available for Windows machines. See http://www.gnuplot.info for more information. If you are on an SMP (multiprocessor) machine and have installed an OpenMP version of the ACML, then in the examples/performance directory a command such asChapter 2: General Information 17 % make OMP_NUM_THREADS=5 will run the timing programs on P processors, where P = 1, 2, 4, 5; i.e., P equals an integer power of 2 and also equals OMP NUM THREADS if this value is not a power of 2. The results for a particular routine are concatenated into one ?le. gnuplot then shows on one graph for each routine the results of varying the number of processors for that routine. Setting OMP NUM THREADS in this way is not useful if you are not on an SMP machine or are not using an OpenMP version of ACML. Neither is it useful to set OMP NUM THREADS to a value higher than the number of processors (or processor cores) on your machine. A way to ?nd the number of processors (or cores) under linux is to examine the special ?le /proc/cpuinfo which has an entry for every core. Not all routines in ACML are SMP parallelized, so in this context the OMP NUM THREADS setting only applies to those examples, including time c?t2d.f90, time dgemm.f90 and time dgetrf.f90, which are for parallelized routines. The other timing programs run on one thread regardless of the setting of OMP NUM THREADS. In all cases, timing graphs can be viewed without regenerating timing results by typing the command % make plots Note that all results generated by timing programs will vary depending on the load on your machine at run time. 2.11 ACML Memory Usage Many ACML routines make use of allocated memory as temporary workspace during computation. Normally this workspace is freed as soon as it is no longer required, just before exit from the ACML routine that allocated it. In general this allocatiom and freeing of memory does not make much di?erence to the performance of ACML. However, in some cases it can make a di?erence; for example, where a kernel routine which allocates memory is repeatedly called by a higher level routine to solve subproblems. This applies particularly to some LAPACK routines which make many calls to the matrix-matrix multiply routine DGEMM. For this reason, for experimental purposes we have introduced into ACML a system whereby the memory allocation mechanism used by DGEMM may avoid freeing memory, instead re-using memory allocated previously. Currently this new scheme has only been implemented in multi-threaded (OpenMP) versions of ACML where it is likely to make most di?erence. Furthermore it currently only works in the Linux versions of ACML, though it may be extended to Windows versions later. By default, the new scheme is turned o?, so if you wish to try it you need to explicitly switch it on. This can be accomplished by means of an environment variable. The variable is named ACML_FAST_MALLOC, and can be set under Linux like this export ACML_FAST_MALLOC=1 (if you use bash or a similar shell), or like this: setenv ACML_FAST_MALLOC 1 (if you use csh, tcsh or similar). At run-time, programs linked with ACML which use DGEMM, directly or indirectly, will then use the new scheme.Chapter 2: General Information 18 Since with this scheme memory is not always freed, there is an obvious penalty to be paid - some of your machine’s memory will be tied up while your program is running and will not be available for other use. There is a limit to the amount of memory that will be assigned in this way, and by default that limit is set to 64 chunks of size 10 Megabytes (actually 10,000,000 bytes) each. This means that the most memory that will be used is 640 Megabytes - whether or not it is all actually used depends on the number of threads used by your program (whether or not you create the threads yourself, since ACML may itself create them). Each DGEMM thread will use one of the 64 chunks of memory. On modern machines it might be considered that this is not a large amount of memory to be used in this way, but if you don’t want to accept the default values, you can change them by the use of further environment variables. These are ACML_FAST_MALLOC_CHUNK_ SIZE which can be used to change the default chunk size from 10 Megabytes to any other positive value, given in bytes, and ACML_FAST_MALLOC_MAX_CHUNKS which sets the maximum number of chunks that may be used. You should use care when changing these values - if you accidentally set them too large and your program uses enough threads you may ?nd that your program fails due to overallocation. Note that if it turns out that ACML needs to use more than the amount of memory that has been set aside in this way, it falls back onto normal allocation and freeing of memory. Thus there should be be no problem if you set too small an amount of memory to be used by the scheme. There is one ?nal environment variable relevant to the fast memory allocation scheme. This is ACML_FAST_MALLOC_DEBUG. If you set this environment variable to any value, then ACML will print various messages showing the memory being used under the scheme. Note that other memory may still be allocated and freed in the normal way by ACML - such allocation will not lead to messages. This environment variable is designed to let you verify that the scheme is actually being used. In normal use you would de?nitely want ACML_ FAST_MALLOC_DEBUG not to be set.Chapter 3: BLAS: Basic Linear Algebra Subprograms 19 3 BLAS: Basic Linear Algebra Subprograms The BLAS are a set of well de?ned basic linear algebra operations ([1], [2], [3]). These operations are subdivided into three groups: • Level 1: operations acting on vectors only (e.g. dot product) • Level 2: matrix-vector operations (e.g. matrix-vector multiplication) • Level 3: matrix-matrix operations (e.g. matrix-matrix multiplication) E?cient machine-speci?c implementations of the BLAS are available for many modern high-performance computers. The implementation of higher level linear algebra algorithms on these systems depends critically on the use of the BLAS as building blocks. AMD provides, as part of the ACML, an implementation of the BLAS optimized for performance on AMD64 processors. For any information relating to the BLAS please refer to the BLAS FAQ: http://www.netlib.org/blas/faq.html ACML also includes interfaces to the extensions to Level 1 BLAS known as the sparse BLAS. These routines perform operations on a sparse vector x which is stored in compressed form and a vector y in full storage form. See reference [4] for more information.Chapter 4: LAPACK: Package of Linear Algebra Subroutines 20 4 LAPACK: Package of Linear Algebra Subroutines 4.1 Introduction to LAPACK LAPACK ([5]) is a library of FORTRAN 77 subroutines for solving commonly occurring problems in numerical linear algebra. LAPACK components can solve systems of linear equations, linear least squares problems, eigenvalue problems and singular value problems. Dense and banded matrices are provided for, but not general sparse matrices. In all areas, similar functionality is provided for real and complex matrices. LAPACK routines are written so that as much as possible of the computations is performed by calls to the BLAS. The e?ciency of LAPACK routines depends, in large part, on the e?ciency of the BLAS being called. Block algorithms are employed wherever possible to maximize the use of calls to level 3 BLAS, which generally run faster than lower level BLAS due to the high number of operations per memory access. The performance of some of the LAPACK routines has been further improved by reworking the computational algorithms. Some of the LAPACK routines contained in ACML are therefore based on code that is di?erent from the LAPACK sources available in the public domain. In all these cases the algorithmic and numerical properties of the original LAPACK routines have been strictly preserved. Furthermore, key LAPACK routines have been treated using OpenMP to take advantage of multiple processors when running on SMP machines. Your application will automatically bene?t when you link with the OpenMP versions of ACML. 4.2 Reference sources for LAPACK The LAPACK homepage can be accessed on the World Wide Web via the URL address: http://www.netlib.org/lapack/ The on-line version of the Lapack User’s Guide, Third Edition ([5]) is available from this homepage, or directly using the URL: http://www.netlib.org/lapack/lug/index.html The standard source code is available for download from netlib, with separate distributions for UNIX/Linux and Windows R installations: http://www.netlib.org/lapack/lapack.tgz http://www.netlib.org/lapack/lapack-pc.zip A list of known problems, bugs, and compiler errors for LAPACK, as well as an errata list for the LAPACK User’s Guide ([5]), is maintained on netlib http://www.netlib.org/lapack/release_notes A LAPACK FAQ (Frequently Asked Questions) ?le can also be accessed via the LAPACK homepage http://www.netlib.org/lapack/faq.htmlChapter 4: LAPACK: Package of Linear Algebra Subroutines 21 4.3 LAPACK block sizes, ILAENV and ILAENVSET As described in Section 6.2 of the LAPACK User’s Guide, block sizes and other parameters used by various LAPACK routines are returned by the LAPACK inquiry function ILAENV. In ACML, values returned by ILAENV have been chosen to achieve very good performance on a wide variety of hardware and problem sizes. In general it is unlikely that you will want or need to be concerned with these parameters. However, in some cases it may be that a default value returned by ILAENV is not optimal for your particular hardware and problem size. Following the advice in the LAPACK User’s Guide may enable you to choose a better value in some circumstances. For convenience, ACML includes a subroutine which allows you to override default values returned by ILAENV if you have superior knowledge. The routine is named ILAENVSET and has the following speci?cation. ILAENVSET (ISPEC,NAME,OPTS,N1,N2,N3,N4,NVALUE,INFO) [SUBROUTINE] INTEGER ISPEC [Input] On input: ISPEC speci?es the parameter to be set (see Section 6.2 of the LAPACK User’s Guide for details). CHARACTER*(*) NAME [Input] On input: NAME speci?es the name of the LAPACK subroutine for which the parameter is to be set. CHARACTER*(*) OPTS [Input] On input: OPTS is a character string of options to the subroutine. INTEGER N1, N2, N3, N4 [Input] On input: N1, N2, N3 and N4 are problem dimensions. A value of -1 means that the dimension is unused or irrelevant. INTEGER NVALUE [Input] On input: NVALUE is the value to be set for the parameter speci?ed by ISPEC. This value will be retrieved by any future call of ILAENV with similar arguments, including the call of ILAENV coming directly from the routine speci?ed by argument NAME. In most cases, but not all, the value set will apply irrespective of the values of arguments OPTS, N1, N2, N3 and N4. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. All arguments of ILAENVSET apart from the last two, NVALUE and INFO, are identical to the arguments of ILAENV. ILAENVSET should be called before you call the LAPACK routine in question. It should be noted that not all LAPACK routines make use of the ILAENV mechanism (because not all routines use blocked algorithms or require other tuning parameters). Calls of ILAENVSET with argument NAME set to the name of such a routine will fail with INFO=0. In addition, the ACML versions of some important routines that do use blocked algorithms, such as the QR factorization routine DGEQRF, bypass ILAENV because they make use of a di?erent tuning system which is independent of standard LAPACK. For all such routines,Chapter 4: LAPACK: Package of Linear Algebra Subroutines 22 ILAENVSET can still be called with no error exit, but calls will have no e?ect on performance of the routine. Below we give examples of how to call ILAENVSET in both FORTRAN and C. Example (FORTRAN code):   INTEGER ILO, IHI, INFO, N, NS CHARACTER COMPZ, JOB INTEGER ILAENV EXTERNAL ILAENV, ILAENVSET JOB = ’E’ COMPZ = ’I’ N = 512 ILO = 1 IHI = 512 C Check the default shift parameter (ISPEC=4) used by DHSEQR NS = ILAENV(4, ’DHSEQR’, JOB//COMPZ, N, ILO, IHI, -1) WRITE (*,*) ’Default NS = ’, NS C Set a new value 5 for the shift parameter CALL ILAENVSET(4, ’DHSEQR’, JOB//COMPZ, N, ILO, IHI, -1, 5, INFO) C Then check the shift parameter again NS = ILAENV(4, ’DHSEQR’, JOB//COMPZ, N, ILO, IHI, -1) WRITE (*,*) ’Revised NS = ’, NS END Example (C code):   #include #include int main(void) { int n=512, ilo=1, ihi=512, ns, info; char compz = ’I’, job = ’E’, opts[3]; opts[0] = job; opts[1] = compz; opts[2] = ’\0’; /* Check the default shift parameter (ISPEC=4) used by DHSEQR */ ns = ilaenv(4, "DHSEQR", opts, n, ilo, ihi, -1); printf("Default ns = %d\n", ns); /* Set a new value 5 for the shift parameter */ ilaenvset(4, "DHSEQR", opts, n, ilo, ihi, -1, 5, &info); /* Then check the shift parameter again */ ns = ilaenv(4, "DHSEQR", opts, n, ilo, ihi, -1); printf("Revised ns = %d\n", ns); return 0; } Chapter 4: LAPACK: Package of Linear Algebra Subroutines 23 4.4 IEEE exceptions and LAPACK Some LAPACK eigensystem routines (namely CHEEVR, DSTEVR, DSYEVR, SSTEVR, SSYEVR, ZHEEVR) are able to take advantage of a faster algorithm when the full eigenspectrum is requested on machines which conform to the IEEE-754 ?oating point standard [14]. Normal execution of the faster algorithm (implemented by LAPACK routines SSTEGR and DSTEGR, which are called by the routines mentioned above) may create NaNs and in?nities and hence may abort due to a ?oating point exception in environments which do not handle NaNs and in?nities in the IEEE standard default manner. This may depend upon the compiler ?ags used to compile and link the main program. The LAPACK routine ILAENV, called with ISPEC = 10 or 11, states whether or not NaNs or in?nities respectively will cause a trap. In ACML, by default ILAENV assumes that NaNs and in?nities cause traps, even if this reduces the performance of the eigensystem routines. This is because it is not possible in general to reliably check whether they do trap or not at run-time. The intention is to ensure that these routines always function correctly, irrespective of how the main program calling ACML is compiled. However, if your main program is compiled in such a way that NaNs and in?nities do not cause traps, the ACML-speci?c routine ILAENVSET (see Section 4.3 [ILAENVILAENVSET], page 21) may be used to override the default operative mode of ILAENV, and allow the xxxEVR routines to use the faster xSTEGR algorithm when calculating the full eigenspectrum. When used for this purpose, ILAENVSET should be called as follows: CALL ILAENVSET(10,’X’,’X’,0,0,0,0,1,INFO) CALL ILAENVSET(11,’X’,’X’,0,0,0,0,1,INFO) (or the C equivalent). It is important to note that if you use ILAENVSET in this way before calling an xxxEVR routine, but your program does trap on IEEE exceptions, then there is a chance that your program will terminate unexpectedly. You should consult the documentation for the compiler you are using to ?nd out whether there are compiler ?ags controlling this.Chapter 5: Fast Fourier Transforms (FFTs) 24 5 Fast Fourier Transforms (FFTs) 5.1 Introduction to FFTs There are two main types of Discrete Fourier Transform (DFT): • routines for the transformation of complex data: in the ACML, these routines have names beginning with ZFFT or CFFT, for double and single precision, respectively; • routines for the transformation of real to complex data and vice versa: in the ACML the names for the former begin with DZFFT or SCFFT, for double and single precision, respectively; the names for the latter begin with ZDFFT or CSFFT. The following subsections provide de?nitions of the DFT for complex and real data types, and some guidelines on the e?cient use of the ACML FFT routines. 5.1.1 Transform de?nitions and Storage for Complex Data The simplest transforms to describe are those performed on sequences of complex data. Such data are stored as arrays of type complex. The result of a complex FFT is also a complex sequence of the same length and, for the simple interfaces, is written back to the original array. Where multiple (m, say), same-length sequences (of length n) of complex data are to be transformed, the sequences are held in a single complex array; in the simple interfaces the array will be of length m * n containing m end-to-end sequences and the results of the m FFTs are returned in the original array. Expert interfaces are provided which give: greater ?exibility in the storage of the original data and results, user provided scaling, and whether results should be written to a separate array or not. The de?nition of a complex DFT used here is given by: x˜j = 1 v n nX-1 k=0 xk exp  ±i 2pjk n  for j = 0, 1, . . . , n - 1 where xk are the complex data to be transformed, ˜xj are the transformed data, and the sign of ± determines the direction of the transform: (-) for forward and (+) for backward. Note that, in this de?nition, both directional transforms have the same scaling and performing both consecutively recovers the original data; this is the prescribed scaling provided in the simple FFT interfaces, whereas, in the expert interfaces, the scaling factor must be supplied by the user. For the simple interfaces, a two dimensional array of complex data, with m rows and n columns is stored in the same order as a set of n sequences of length m (as described above). That is, column elements are stored contiguously and the ?rst element of the next column follows the last element of the current column. In the expert interfaces, column elements may be separated by a ?xed step length (increment) while row elements may be separated by a second increment; if the ?rst increment is 1 and the second increment is m then we have the same storage as in the simple interface. The de?nition of a complex 2D DFT used here is given by: x˜jp = 1 v m * n mX-1 l=0 nX-1 k=0 xkl exp  ±i 2pjk n  exp  ±i 2ppl m  for j = 0, 1, . . . , n - 1 and l = 0, 1, . . . , m - 1, where xkl are the complex data to be transformed, ˜xjp are the transformed data, and the sign of ± determines the direction of the transform.Chapter 5: Fast Fourier Transforms (FFTs) 25 5.1.2 Transform de?nitions and Storage for Real Data The DFT of a sequence of real data results in a special form of complex sequence known as a Hermitian sequence. The symmetries de?ning such a sequence mean that it can be fully represented by a set of n real values, where n is the length of the original real sequence. It is therefore conventional for the array containing the real sequence to be overwritten by such a representation of the transformed Hermitian sequence. If the original sequence is purely real valued, i.e. zj = xj , then the de?nition of the real DFT used here is given by: z˜j = aj + ibj = 1 v n nX-1 k=0 xk exp  -i 2pjk n  for j = 0, 1, . . . , n - 1 where xk are the real data to be transformed, ˜zj are the transformed complex data. In full complex representation, the Hermitian sequence would be a sequence of n complex values Z(i) for i = 0, 1, ..., n - 1, where Z(n - j) is the complex conjugate of Z(j) for j = 1, 2, ...,(n-1)/2; Z(0) is real valued; and, if n is even, Z(n/2) is real valued. In ACML, the representation of Hermitian sequences used on output from DZFFT routines and on input to ZDFFT routines is as follows: let X be an array of length N and with ?rst index 0, • X(i) contains the real part of Z(i) for i = 0, ..., N/2 • X(N - i) contains the imaginary part of Z(i) for i = 1, ...,(N - 1)/2 Also, given a Hermitian sequence, the discrete transform can be written as: xj = 1 v n ? ?a0 + 2 n/X2-1 k=1  ak cos  2pjk n  - bk sin  2pjk n  + an/2 ? ? where an/2 = 0 if n is odd, and ˜zk = ak + ibk is the Hermitian sequence to be transformed. Note that, in the above de?nitions, both transforms have the same (negative) sign in the exponent; performing both consecutively does not recover the original data. To recover original real data, or otherwise to perform an inverse transform on a set of Hermitian data, the Hermitian data must be conjugated prior to performing the transform (i.e. changing the sign of the stored imaginary parts). 5.1.3 E?ciency The e?ciency of the FFT is maximized by choosing the sequence length to be a power of 2. Good e?ciency can also be achieved when the sequence length has small prime factors, up to a factor 13; however, the time taken for an FFT increases as the size of the prime factor increases.Chapter 5: Fast Fourier Transforms (FFTs) 26 5.1.4 Default and Generated Plans For those FFT routines that can be initialized prior to computing the FFTs, the initialization can be performed in one of two ways. In either case, initialization involves the storing of the factorization of N, and the twiddle factors associated with this factorization, in the communication array COMM. The simpler way to initialize is by setting the argument MODE to zero. This means that a default plan, for the given input dimensions, is used to calculate the FFT. This has the advantage that the initialization phase is very quick and is generally a small fraction of the time required to perform the FFT computation. However, for some problem dimensions the default plan may not be optimal, especially where there is a mixture of prime factors. Under some circumstances, optimality of performance of an FFT computation may be crucial. For example, where a very large number of FFTs are to be performed on problems of a ?xed size (e.g. N remains the same), then it is best to initialize by setting the argument MODE to 100. This will time a number of plans (this number can be quite large when N has a signi?cant number of prime factors) and initialize using the plan with the best time. Using this form of initialization can, potentially, lead to signi?cant improvements in the performance of the FFT computation for the given dimensions. Where problem dimensions will not change over a number of runs of a program, the communication array could, for example, be written out to a ?le during an initialization run, and then read in from the same ?le on subsequent computation runs. This would be e?ective for problem dimensions that have a large number of possible plans (factor orderings and groupings) and therefore take a signi?cant amount of time to ?nd the optimal plan. Please consult the individual FFT routine documents to determine whether plan generation is enabled.Chapter 5: Fast Fourier Transforms (FFTs) 27 5.2 FFTs on Complex Sequences 5.2.1 FFT of a single sequence The routines documented here compute the discrete Fourier transform (DFT) of a sequence of complex numbers in either single or double precision arithmetic. The DFT is computed using a highly-e?cient FFT algorithm. There are two sets of interfaces available: simple drivers and expert drivers. The simple drivers perform in-place transforms on data held contiguously in memory using a ?xed scaling factor; these are simpler to use and are su?- cient for many problems. The expert drivers o?er greater ?exibility by including a number of additional arguments. These allow you to control: the scaling factor applied; whether the result should be output to a separate vector; and, the increments used in storing successive elements of both the input sequence and the result.Chapter 5: Fast Fourier Transforms (FFTs) 28 ZFFT1D Routine Documentation ZFFT1D (MODE,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZFFT1D. On input: • MODE=0 : only default initializations (speci?c to N) are performed; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT1D. • MODE=1 : a backward (reverse) transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT1D. • MODE=-2 : initializations and a forward transform are performed. • MODE=2 : initializations and a backward transform are performed. • MODE=100 : similar to MODE=0; only initializations are performed, but ?rst a plan is generated. This plan is chosen based on the fastest FFT computation for a subset of all possible plans. INTEGER N [Input] On input: N is the length of the complex sequence X COMPLEX*16 X(N) [Input/Output] On input: X contains the complex sequence of length N to be transformed. On output: X contains the transformed sequence. COMPLEX*16 COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL ZFFT1D(0,N,X,COMM,INFO) CALL ZFFT1D(-1,N,X,COMM,INFO) CALL ZFFT1D(-1,N,Y,COMM,INFO) DO 10 I = 1, N X(I) = X(I)*DCONJG(Y(I)) 10 CONTINUE CALL ZFFT1D(1,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 29 CFFT1D Routine Documentation CFFT1D (MODE,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by CFFT1D. On input: • MODE=0 : only default initializations (speci?c to N) are performed; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward transform is performed. Initializations are assumed to have been performed by a prior call to CFFT1D. • MODE=1 : a backward (reverse) transform is performed. Initializations are assumed to have been performed by a prior call to CFFT1D. • MODE=-2 : (default) initializations and a forward transform are performed. • MODE=2 : (default) initializations and a backward transform are performed. • MODE=100 : similar to MODE=0; only initializations are performed, but ?rst a plan is generated. This plan is chosen based on the fastest FFT computation for a subset of all possible plans. INTEGER N [Input] On input: N is the length of the complex sequence X COMPLEX X(N) [Input/Output] On input: X contains the complex sequence of length N to be transformed. On output: X contains the transformed sequence. COMPLEX COMM(5*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL CFFT1D(0,N,X,COMM,INFO) CALL CFFT1D(-1,N,X,COMM,INFO) CALL CFFT1D(-1,N,Y,COMM,INFO) DO 10 I = 1, N X(I) = X(I)*CONJG(Y(I)) 10 CONTINUE CALL CFFT1D(1,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 30 ZFFT1DX Routine Documentation ZFFT1DX (MODE,SCALE,INPL,N,X,INCX,Y,INCY,COMM, [SUBROUTINE] INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZFFT1DX. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT1DX. • MODE=1 : a backward (reverse) transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT1DX. • MODE=-2 : (default) initializations and a forward transform are performed. • MODE=2 : (default) initializations and a backward transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. DOUBLE PRECISION SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequence LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequence; otherwise the output sequence is returned in Y. INTEGER N [Input] On input: N is the number of elements to be transformed COMPLEX*16 X(1+(N-1)*INCX) [Input/Output] On input: X contains the complex sequence of length N to be transformed, with the ith element stored in X(1+(i-1)*INCX). On output: if INPL is .TRUE. then X contains the transformed sequence in the same locations as on input; otherwise X remains unchanged. INTEGER INCX [Input] On input: INCX is the increment used to store successive elements of a sequence in X. Constraint: INCX > 0. COMPLEX*16 Y(1+(N-1)*INCY) [Output] On output: if INPL is .FALSE. then Y contains the transformed sequence, with the ith element stored in Y(1+(i-1)*INCY); otherwise Y is not referenced.Chapter 5: Fast Fourier Transforms (FFTs) 31 INTEGER INCY [Input] On input: INCY is the increment used to store successive elements of a sequence in Y. If INPL is .TRUE. then INCY is not referenced. Constraint: INCY > 0. COMPLEX*16 COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Forward FFTs are performed unscaled and in-place on contiguous C vectors X and Y following initialization. Manipulations on C resultant Fourier coefficients are stored in X which is then C transformed back. C SCALE = 1.0D0 INPL = .TRUE. CALL ZFFT1DX(0,SCALE,INPL,N,X,1,DUM,1,COMM,INFO) CALL ZFFT1DX(-1,SCALE,INPL,N,X,1,DUM,1,COMM,INFO) CALL ZFFT1DX(-1,SCALE,INPL,N,Y,1,DUM,1,COMM,INFO) DO 10 I = 1, N X(I) = X(I)*DCONJG(Y(I))/DBLE(N) 10 CONTINUE CALL ZFFT1DX(1,SCALE,INPL,N,X,1,DUM,1,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 32 CFFT1DX Routine Documentation CFFT1DX (MODE,SCALE,INPL,N,X,INCX,Y,INCY,COMM, [SUBROUTINE] INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by CFFT1DX. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward transform is performed. Initializations are assumed to have been performed by a prior call to CFFT1DX. • MODE=1 : a backward (reverse) transform is performed. Initializations are assumed to have been performed by a prior call to CFFT1DX. • MODE=-2 : (default) initializations and a forward transform are performed. • MODE=2 : (default) initializations and a backward transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. REAL SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequence LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequence; otherwise the output sequence is returned in Y. INTEGER N [Input] On input: N is the number of elements to be transformed COMPLEX X(1+(N-1)*INCX) [Input/Output] On input: X contains the complex sequence of length N to be transformed, with the ith element stored in X(1+(i-1)*INCX). On output: if INPL is .TRUE. then X contains the transformed sequence in the same locations as on input; otherwise X remains unchanged. INTEGER INCX [Input] On input: INCX is the increment used to store successive elements of a sequence in X. Constraint: INCX > 0. COMPLEX Y(1+(N-1)*INCY) [Output] On output: if INPL is .FALSE. then Y contains the transformed sequence, with the ith element stored in Y(1+(i-1)*INCY); otherwise Y is not referenced.Chapter 5: Fast Fourier Transforms (FFTs) 33 INTEGER INCY [Input] On input: INCY is the increment used to store successive elements of a sequence in Y. If INPL is .TRUE. then INCY is not referenced. Constraint: INCY > 0. COMPLEX COMM(5*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Forward FFTs are performed unscaled and in-place on contiguous C vectors X and Y following initialization. Manipulations on C resultant Fourier coefficients are stored in X which is then C transformed back. C SCALE = 1.0 INPL = .TRUE. CALL CFFT1DX(0,SCALE,INPL,N,X,1,DUM,1,COMM,INFO) CALL CFFT1DX(-1,SCALE,INPL,N,X,1,DUM,1,COMM,INFO) CALL CFFT1DX(-1,SCALE,INPL,N,Y,1,DUM,1,COMM,INFO) DO 10 I = 1, N X(I) = X(I)*CONJG(Y(I))/REAL(N) 10 CONTINUE CALL CFFT1DX(1,SCALE,INPL,N,X,1,DUM,1,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 34 5.2.2 FFT of multiple complex sequences The routines documented here compute the discrete Fourier transforms (DFTs) of a number of sequences of complex numbers in either single or double precision arithmetic. The sequences must all have the same length. The DFTs are computed using a highly-e?cient FFT algorithm. There are two sets of interfaces available: simple drivers and expert drivers. The simple drivers perform in-place transforms on data held contiguously in memory using a ?xed scaling factor; these are simpler to use and are su?cient for many problems. The expert drivers o?er greater ?exibility by including a number of additional arguments. These allow you to control: the scaling factor applied; whether the result should be output to a separate vector; the increments used in storing successive elements of a given sequence (for both input and output sequences); and the increments used in storing corresponding elements in successive sequences (for both input and output).Chapter 5: Fast Fourier Transforms (FFTs) 35 ZFFT1M Routine Documentation ZFFT1M (MODE,M,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZFFT1M. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : forward transforms are performed. Initializations are assumed to have been performed by a prior call to ZFFT1M. • MODE=1 : backward (reverse) transforms are performed. Initializations are assumed to have been performed by a prior call to ZFFT1M. • MODE=-2 : (default) initializations and forward transforms are performed. • MODE=2 : (default) initializations and backward transforms are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. INTEGER M [Input] On input: M is the number of sequences to be transformed. INTEGER N [Input] On input: N is the length of the complex sequences in X COMPLEX*16 X(N*M) [Input/Output] On input: X contains the M complex sequences of length N to be transformed. Element i of sequence j is stored in location i + (j - 1) * N of X. On output: X contains the transformed sequences. COMPLEX*16 COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 5: Fast Fourier Transforms (FFTs) 36 Example:   CALL ZFFT1M(0,1,N,X,COMM,INFO) CALL ZFFT1M(-1,2,N,X,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*DCONJG(X(I,2)) X(I,2) = DCMPLX(0.0D0,1.0D0)*X(I,2) 10 CONTINUE CALL ZFFT1M(1,2,N,X(1,2),COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 37 CFFT1M Routine Documentation CFFT1M (MODE,M,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by CFFT1M. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : forward transforms are performed. Initializations are assumed to have been performed by a prior call to CFFT1M. • MODE=1 : backward (reverse) transforms are performed. Initializations are assumed to have been performed by a prior call to CFFT1M. • MODE=-2 : (default) initializations and forward transforms are performed. • MODE=2 : (default) initializations and backward transforms are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. INTEGER M [Input] On input: M is the number of sequences to be transformed. INTEGER N [Input] On input: N is the length of the complex sequences in X COMPLEX X(N*M) [Input/Output] On input: X contains the M complex sequences of length N to be transformed. Element i of sequence j is stored in location i + (j - 1) * N of X. On output: X contains the transformed sequences. COMPLEX COMM(5*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 5: Fast Fourier Transforms (FFTs) 38 Example:   CALL CFFT1M(0,1,N,X,COMM,INFO) CALL CFFT1M(-1,2,N,X,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*CONJG(X(I,2)) X(I,2) = CMPLX(0.0D0,1.0D0)*X(I,2) 10 CONTINUE CALL CFFT1M(1,2,N,X(1,2),COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 39 ZFFT1MX Routine Documentation ZFFT1MX (MODE,SCALE,INPL,NSEQ,N,X,INCX1,INCX2, [SUBROUTINE] Y,INCY1,INCY2,COMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZFFT1MX. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT1MX. • MODE=1 : a backward (reverse) transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT1MX. • MODE=-2 : (default) initializations and a forward transform are performed. • MODE=2 : (default) initializations and a backward transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. DOUBLE PRECISION SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER NSEQ [Input] On input: NSEQ is the number of sequences to be transformed INTEGER N [Input] On input: N is the number of elements in each sequence to be transformed COMPLEX*16 X(1+(N-1)*INCX1+(NSEQ-1)*INCX2) [Input/Output] On input: X contains the NSEQ complex sequences of length N to be transformed; the ith element of sequence j is stored in X(1+(i-1)*INCX1+(j- 1)*INCX2). On output: if INPL is .TRUE. then X contains the transformed sequences in the same locations as on input; otherwise X remains unchanged. INTEGER INCX1 [Input] On input: INCX1 is the increment used to store successive elements of a given sequence in X (INCX1=1 for contiguous data). Constraint: INCX1 > 0.Chapter 5: Fast Fourier Transforms (FFTs) 40 INTEGER INCX2 [Input] On input: INCX2 is the increment used to store corresponding elements of successive sequences in X (INCX2=N for contiguous data). Constraint: INCX2 > 0. COMPLEX*16 Y(1+(N-1)*INCY1+(NSEQ-1)*INCY2) [Output] On output: if INPL is .FALSE. then Y contains the transformed sequences with the ith element of sequence j stored in Y(1+(i-1)*INCY1+(j-1)*INCY2); otherwise Y is not referenced. INTEGER INCY1 [Input] On input: INCY1 is the increment used to store successive elements of a given sequence in Y. If INPL is .TRUE. then INCY1 is not referenced. Constraint: INCY1 > 0. INTEGER INCY2 [Input] On input: INCY2 is the increment used to store corresponding elements of successive sequences in Y (INCY2=N for contiguous data). If INPL is .TRUE. then INCY2 is not referenced. Constraint: INCY2 > 0. COMPLEX*16 COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Forward FFTs are performed unscaled and in-place on two C contiguous vectors stored in the first two columns of X. C Manipulations are stored in 2nd and 3rd columns of X which are C then transformed back. C COMPLEX *16 X(N,3) SCALE = 1.0D0 INPL = .TRUE. CALL ZFFT1MX(0,SCALE,INPL,2,N,X,1,N,DUM,1,N,COMM,INFO) CALL ZFFT1MX(-1,SCALE,INPL,2,N,X,1,N,DUM,1,N,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*DCONJG(X(I,2))/DBLE(N) X(I,2) = DCMPLX(0.0D0,1.0D0)*X(I,2)/DBLE(N) 10 CONTINUE CALL ZFFT1MX(1,SCALE,INPL,2,N,X(1,2),1,N,DUM,1,N,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 41 CFFT1MX Routine Documentation CFFT1MX (MODE,SCALE,INPL,NSEQ,N,X,INCX1,INCX2, [SUBROUTINE] Y,INCY1,INCY2,COMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by CFFT1MX. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward transform is performed. Initializations are assumed to have been performed by a prior call to CFFT1MX. • MODE=1 : a backward (reverse) transform is performed. Initializations are assumed to have been performed by a prior call to CFFT1MX. • MODE=-2 : (default) initializations and a forward transform are performed. • MODE=2 : (default) initializations and a backward transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. REAL SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER NSEQ [Input] On input: NSEQ is the number of sequences to be transformed INTEGER N [Input] On input: N is the number of elements in each sequence to be transformed COMPLEX X(1+(N-1)*INCX1+(NSEQ-1)*INCX2) [Input/Output] On input: X contains the NSEQ complex sequences of length N to be transformed; the ith element of sequence j is stored in X(1+(i-1)*INCX1+(j- 1)*INCX2). On output: if INPL is .TRUE. then X contains the transformed sequences in the same locations as on input; otherwise X remains unchanged. INTEGER INCX1 [Input] On input: INCX1 is the increment used to store successive elements of a given sequence in X (INCX1=1 for contiguous data). Constraint: INCX1 > 0.Chapter 5: Fast Fourier Transforms (FFTs) 42 INTEGER INCX2 [Input] On input: INCX2 is the increment used to store corresponding elements of successive sequences in X (INCX2=N for contiguous data). Constraint: INCX2 > 0. COMPLEX Y(1+(N-1)*INCY1+(NSEQ-1)*INCY2) [Output] On output: if INPL is .FALSE. then Y contains the transformed sequences with the ith element of sequence j stored in Y(1+(i-1)*INCY1+(j-1)*INCY2); otherwise Y is not referenced. INTEGER INCY1 [Input] On input: INCY1 is the increment used to store successive elements of a given sequence in Y. If INPL is .TRUE. then INCY1 is not referenced. Constraint: INCY1 > 0. INTEGER INCY2 [Input] On input: INCY2 is the increment used to store corresponding elements of successive sequences in Y (INCY2=N for contiguous data). If INPL is .TRUE. then INCY2 is not referenced. Constraint: INCY2 > 0. COMPLEX COMM(5*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Forward FFTs are performed unscaled and in-place on two C contiguous vectors stored in the first two columns of X. C Manipulations are stored in 2nd and 3rd columns of X which are C then transformed back. C COMPLEX X(N,3) SCALE = 1.0 INPL = .TRUE. CALL CFFT1MX(0,SCALE,INPL,2,N,X,1,N,DUM,1,N,COMM,INFO) CALL CFFT1MX(-1,SCALE,INPL,2,N,X,1,N,DUM,1,N,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*CONJG(X(I,2))/REAL(N) X(I,2) = CMPLX(0.0D0,1.0D0)*X(I,2)/REAL(N) 10 CONTINUE CALL CFFT1MX(1,SCALE,INPL,2,N,X(1,2),1,N,DUM,1,N,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 43 5.2.3 2D FFT of two-dimensional arrays of data The routines documented here compute the two-dimensional discrete Fourier transforms (DFT) of a two-dimensional array of complex numbers in either single or double precision arithmetic. The 2D DFT is computed using a highly-e?cient FFT algorithm. There are two sets of interfaces available: simple drivers and expert drivers. The simple drivers perform in-place transforms on data held contiguously in memory using a ?xed scaling factor; these are simpler to use and are su?cient for many problems. The expert drivers o?er greater ?exibility by including a number of additional arguments. These allow you to control: the scaling factor applied; whether the result should be output to a separate array; the increments used in storing successive elements in each dimension (for both input and output); and the facility to not perform a ?nal transposition. This ?nal facility is useful for those cases where a forward and backward transform are to be applied with some data manipulations in between; here two whole transpositions can be saved.Chapter 5: Fast Fourier Transforms (FFTs) 44 ZFFT2D Routine Documentation ZFFT2D (MODE,M,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the direction of transform to be performed by ZFFT2D. On input: • MODE=-1 : forward 2D transform is performed. • MODE=1 : backward (reverse) 2D transform is performed. INTEGER M [Input] On input: M is the number of rows in the 2D array of data to be transformed. If X is declared as a 2D array then M is the ?rst dimension of X. INTEGER N [Input] On input: N is the number of columns in the 2D array of data to be transformed. If X is declared as a 2D array then M is the second dimension of X. COMPLEX*16 X(M*N) [Input/Output] On input: X contains the M by N complex 2D array to be transformed. Element ij is stored in location i + (j - 1) * M of X. On output: X contains the transformed sequence. COMPLEX*16 COMM(M*N+3*(M+N)+100) [Input/Output] COMM is a communication array used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL ZFFT2D(-1,M,N,X,COMM,INFO) DO 20 J = 1, N DO 10 I = 1, MIN(J-1,M) X(I,J) = 0.5D0*(X(I,J) + X(J,I)) X(J,I) = DCONJG(X(I,J)) 10 CONTINUE 20 CONTINUE CALL ZFFT2D(1,M,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 45 CFFT2D Routine Documentation CFFT2D (MODE,M,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the direction of transform to be performed by CFFT2D. On input: • MODE=-1 : a forward 2D transform is performed. • MODE=1 : a backward (reverse) 2D transform is performed. INTEGER M [Input] On input: M is the number of rows in the 2D array of data to be transformed. If X is declared as a 2D array then M is the ?rst dimension of X. INTEGER N [Input] On input: N is the number of columns in the 2D array of data to be transformed. If X is declared as a 2D array then M is the second dimension of X. COMPLEX X(M*N) [Input/Output] On input: X contains the M by N complex 2D array to be transformed. Element ij is stored in location i + (j - 1) * M of X. On output: X contains the transformed sequence. COMPLEX COMM(M*N+5*(M+N)) [Input/Output] COMM is a communication array used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL CFFT2D(-1,M,N,X,COMM,INFO) DO 20 J = 1, N DO 10 I = 1, MIN(J-1,M) X(I,J) = 0.5D0*(X(I,J) + X(J,I)) X(J,I) = CONJG(X(I,J)) 10 CONTINUE 20 CONTINUE CALL CFFT2D(1,M,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 46 ZFFT2DX Routine Documentation ZFFT2DX (MODE,SCALE,LTRANS,INPL,M,N,X,INCX1, [SUBROUTINE] INCX2,Y,INCY1,INCY2,COMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZFFT2DX. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward 2D transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT2DX. • MODE=1 : a backward (reverse) 2D transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT2DX. • MODE=-2 : (default) initializations and a forward 2D transform are performed. • MODE=2 : (default) initializations and a backward 2D transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the values of N and M) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the values of N and M. DOUBLE PRECISION SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL LTRANS [Input] On input: if LTRANS is .TRUE. then a normal ?nal transposition is performed internally to return transformed data consistent with the values for arguments INPL, INCX1, INCX2, INCY1 and INCY2. If LTRANS is .FALSE. then the ?nal transposition is not performed explicitly; the storage format on output is determined by whether the output data is stored contiguously or not – please see the output speci?cations for X and Y for details. LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER M [Input] On input: M is the ?rst dimension of the 2D transform. INTEGER N [Input] On input: N is the second dimension of the 2D transform.Chapter 5: Fast Fourier Transforms (FFTs) 47 COMPLEX*16 X(1+(M-1)*INCX1+(N-1)*INCX2) [Input/Output] On input: X contains the M by N complex 2D data array to be transformed; the (ij)th element is stored in X(1+(i-1)*INCX1+(j-1)*INCX2). On output: if INPL is .TRUE. then X contains the transformed data, either in the same locations as on input when LTRANS=.TRUE.; in locations X((i- 1)*N+j) when LTRANS=.FALSE., INCX1=1 and INCX2=M; and otherwise in the same locations as on input. If INPL is .FALSE. X remains unchanged. INTEGER INCX1 [Input] On input: INCX1 is the increment used to store, in X, successive elements in the ?rst dimension (INCX1=1 for contiguous data). Constraint: INCX1 > 0. INTEGER INCX2 [Input] On input: INCX2 is the increment used to store, in X, successive elements in the second dimension (INCX2=M for contiguous data). Constraint: INCX2 > 0; INCX2 > (M-1)*INCX1 if N > 1. COMPLEX*16 Y(1+(M-1)*INCY1+(N-1)*INCY2) [Output] On output: if INPL is .FALSE. then Y contains the transformed data. If LTRANS=.TRUE. then the (ij)th data element is stored in Y(1+(i- 1)*INCY1+(j-1)*INCY2); if LTRANS=.FALSE., INCY1=1 and INCY2=N then the (ij)th data element is stored in Y((i-1)*N+j); and otherwise the (ij)th element is stored in Y(1+(i-1)*INCY1+(j-1)*INCY2). If INPL is .TRUE. then Y is not referenced. INTEGER INCY1 [Input] On input: INCY1 is the increment used to store successive elements in the ?rst dimension in Y (INCY1=1 for contiguous data). If INPL is .TRUE. then INCY1 is not referenced. Constraint: INCY1 > 0. INTEGER INCY2 [Input] On input: INCY2 is the increment used to store successive elements in the second dimension in Y (for contiguous data, INCY2=M when LTRANS is .TRUE. or INCY2=N when LTRANS is .FALSE.). If INPL is .TRUE. then INCY2 is not referenced. Constraints: INCY2 > 0; INCY2 > (M-1)*INCY1 if N > 1 and LTRANS is .TRUE.; INCY2 = N if M > 1 and LTRANS is .FALSE.. COMPLEX*16 COMM(M*N+3*M+3*N+200) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same dimensions M and N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 5: Fast Fourier Transforms (FFTs) 48 Example:   C Forward 2D FFT is performed unscaled, without final transpose C and out-of-place on data stored in array X and output to Y. C Manipulations are stored in vector Y which is then transformed C back, with scaling, into the first M rows of X. C COMPLEX *16 X(M,N), Y(N,M) SCALE = 1.0D0 INPL = .FALSE. LTRANS = .FALSE. CALL ZFFT2DX(0,SCALE,LTRANS,INPL,M,N,X,1,M,Y,1,N,COMM,INFO) CALL ZFFT2DX(-1,SCALE,LTRANS,INPL,M,N,X,1,M,Y,1,N,COMM,INFO) DO 20 I = M DO 10 J = 1, N Y(J,I) = 0.5D0*Y(J,I)*EXP(0.001D0*(I+J-2)) 10 CONTINUE 20 CONTINUE SCALE = 1.0D0/DBLE(M*N) CALL ZFFT2DX(1,SCALE,LTRANS,INPL,N,M,Y,1,N,X,1,M,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 49 CFFT2DX Routine Documentation CFFT2DX (MODE,SCALE,LTRANS,INPL,M,N,X,INCX1, [SUBROUTINE] INCX2,Y,INCY1,INCY2,COMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by CFFT2DX. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward 2D transform is performed. Initializations are assumed to have been performed by a prior call to CFFT2DX. • MODE=1 : a backward (reverse) 2D transform is performed. Initializations are assumed to have been performed by a prior call to CFFT2DX. • MODE=-2 : (default) initializations and a forward 2D transform are performed. • MODE=2 : (default) initializations and a backward 2D transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the values of N and M) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the values of N and M. REAL SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL LTRANS [Input] On input: if LTRANS is .TRUE. then a normal ?nal transposition is performed internally to return transformed data consistent with the values for arguments INPL, INCX1, INCX2, INCY1 and INCY2. If LTRANS is .FALSE. then the ?nal transposition is not performed explicitly; the storage format on output is determined by whether the output data is stored contiguously or not – please see the output speci?cations for X and Y for details. LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER M [Input] On input: M is the ?rst dimension of the 2D transform. INTEGER N [Input] On input: N is the second dimension of the 2D transform.Chapter 5: Fast Fourier Transforms (FFTs) 50 COMPLEX X(1+(M-1)*INCX1+(N-1)*INCX2) [Input/Output] On input: X contains the M by N complex 2D data array to be transformed; the (ij)th element is stored in X(1+(i-1)*INCX1+(j-1)*INCX2). On output: if INPL is .TRUE. then X contains the transformed data, either in the same locations as on input when LTRANS=.TRUE.; in locations X((i- 1)*N+j) when LTRANS=.FALSE., INCX1=1 and INCX2=M; and otherwise in the same locations as on input. If INPL is .FALSE. X remains unchanged. INTEGER INCX1 [Input] On input: INCX1 is the increment used to store, in X, successive elements in the ?rst dimension (INCX1=1 for contiguous data). Constraint: INCX1 > 0. INTEGER INCX2 [Input] On input: INCX2 is the increment used to store, in X, successive elements in the second dimension (INCX2=M for contiguous data). Constraint: INCX2 > 0; INCX2 > (M-1)*INCX1 if N > 1. COMPLEX Y(1+(M-1)*INCY1+(N-1)*INCY2) [Output] On output: if INPL is .FALSE. then Y contains the transformed data. If LTRANS=.TRUE. then the (ij)th data element is stored in Y(1+(i- 1)*INCY1+(j-1)*INCY2); if LTRANS=.FALSE., INCY1=1 and INCY2=N then the (ij)th data element is stored in Y((i-1)*N+j); and otherwise the (ij)th element is stored in Y(1+(i-1)*INCY1+(j-1)*INCY2). If INPL is .TRUE. then Y is not referenced. INTEGER INCY1 [Input] On input: INCY1 is the increment used to store successive elements in the ?rst dimension in Y (INCY1=1 for contiguous data). If INPL is .TRUE. then INCY1 is not referenced. Constraint: INCY1 > 0. INTEGER INCY2 [Input] On input: INCY2 is the increment used to store successive elements in the second dimension in Y (for contiguous data, INCY2=M when LTRANS is .TRUE. or INCY2=N when LTRANS is .FALSE.). If INPL is .TRUE. then INCY2 is not referenced. Constraints: INCY2 > 0; INCY2 > (M-1)*INCY1 if N > 1 and LTRANS is .TRUE.; INCY2 = N if M > 1 and LTRANS is .FALSE.. COMPLEX COMM(M*N+5*M+5*N+200) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same dimensions M and N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 5: Fast Fourier Transforms (FFTs) 51 Example:   C Forward 2D FFT is performed unscaled, without final transpose C and out-of-place on data stored in array X and output to Y. C Manipulations are stored in vector Y which is then transformed C back, with scaling, into the first M rows of X. C COMPLEX X(M,N), Y(N,M) SCALE = 1.0 INPL = .FALSE. LTRANS = .FALSE. CALL CFFT2DX(0,SCALE,LTRANS,INPL,M,N,X,1,M,Y,1,N,COMM,INFO) CALL CFFT2DX(-1,SCALE,LTRANS,INPL,M,N,X,1,M,Y,1,N,COMM,INFO) DO 20 I = M DO 10 J = 1, N Y(J,I) = 0.5*Y(J,I)*EXP(-0.001*REAL(I+J-2)) IY = IY + 1 10 CONTINUE 20 CONTINUE SCALE = 1.0/REAL(M*N) CALL CFFT2DX(1,SCALE,LTRANS,INPL,N,M,Y,1,N,X,1,M,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 52 5.2.4 3D FFT of three-dimensional arrays of data The routines documented here compute the three-dimensional discrete Fourier transforms (DFT) of a three-dimensional array of complex numbers in either single or double precision arithmetic. The 3D DFT is computed using a highly-e?cient FFT algorithm. Please note that at Release 2.7 of ACML it was necessary to modify slightly the interfaces of two of the expert FFT drivers introduced at Release 2.2 of ACML. The two routines are CFFT3DX and ZFFT3DX. The changes were required to permit the optimization of these routines by adding an initialization stage which can then use the plan generator (MODE=100) to select the optimal plan. User codes that called CFFT3DX or ZFFT3DX using a release of ACML prior to 2.7 will need to be modi?ed in one of two ways. Calls to CFFT3DX/ZFFT3DX with MODE = -1 or 1 can be ?xed for ACML Release 2.7 and later by either: • preceding the call with a call setting MODE = 0 (default initialization), or MODE = 100 (initialization using plan generator); or, • doubling the MODE argument value to MODE = -2 or 2 respectively (thus incorporating default initialization). Additionally, the minimum length of the communication (work)space arrays in CFFT3DX and ZFFT3DX has been increased by 100 to allow for plan storage. Please consult the individual routine documents for full details on their use.Chapter 5: Fast Fourier Transforms (FFTs) 53 ZFFT3D Routine Documentation ZFFT3D (MODE,L,M,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the direction of transform to be performed by ZFFT3D. On input: • MODE=-1 : forward 3D transform is performed. • MODE=1 : backward (reverse) 3D transform is performed. INTEGER L [Input] On input: the length of the ?rst dimension of the 3D array of data to be transformed. If X is declared as a 3D array then L is the ?rst dimension of X. INTEGER M [Input] On input: the length of the second dimension of the 3D array of data to be transformed. If X is declared as a 3D array then M is the second dimension of X. INTEGER N [Input] On input: the length of the third dimension of the 3D array of data to be transformed. If X is declared as a 3D array then N is the third dimension of X. COMPLEX*16 X(L*M*N) [Input/Output] On input: X contains the L by M by N complex 3D array to be transformed. Element ijk is stored in location i + (j - 1) * L + (k - 1) * L * M of X. On output: X contains the transformed sequence. COMPLEX*16 COMM(L*M*N+3*(L+M+N)) [Input/Output] COMM is a communication array used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL ZFFT3D(-1,L,M,N,X,COMM,INFO) DO 30 K = 1, N DO 20 J = 1, M DO 10 I = 1, L X(I,J) = X(I,J)*EXP(-0.001D0*DBLE(I+J+K)) 10 CONTINUE 20 CONTINUE 30 CONTINUE CALL ZFFT3D(1,L,M,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 54 CFFT3D Routine Documentation CFFT3D (MODE,L,M,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the direction of transform to be performed by CFFT3D. On input: • MODE=-1 : forward 3D transform is performed. • MODE=1 : backward (reverse) 3D transform is performed. INTEGER L [Input] On input: the length of the ?rst dimension of the 3D array of data to be transformed. If X is declared as a 3D array then L is the ?rst dimension of X. INTEGER M [Input] On input: the length of the second dimension of the 3D array of data to be transformed. If X is declared as a 3D array then M is the second dimension of X. INTEGER N [Input] On input: the length of the third dimension of the 3D array of data to be transformed. If X is declared as a 3D array then N is the third dimension of X. COMPLEX X(L*M*N) [Input/Output] On input: X contains the L by M by N complex 3D array to be transformed. Element ijk is stored in location i + (j - 1) * L + (k - 1) * L * M of X. On output: X contains the transformed sequence. COMPLEX COMM(5*(L+M+N)+4) [Input/Output] COMM is a communication array used as temporary store. Note that the amount of store explicitly required here is less than in some versions prior to this release (version 4.1 and older). Some further workspace will be allocated internally; the amount of allocated memory requested will be MAX(L*N+N, L*M + L + M, N). INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL CFFT3D(-1,L,M,N,X,COMM,INFO) DO 30 K = 1, N DO 20 J = 1, M DO 10 I = 1, L X(I,J) = X(I,J)*EXP(-0.001D0*REAL(I+J+K)) 10 CONTINUE 20 CONTINUE 30 CONTINUE CALL CFFT3D(1,L,M,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 55 ZFFT3DX Routine Documentation ZFFT3DX (MODE,SCALE,LTRANS,INPL,L,M,N,X,Y, [SUBROUTINE] COMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZFFT3DX. On input: • MODE=0 : only initializations (speci?c to the values of L, M and N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward 3D transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT3DX. • MODE=1 : a backward (reverse) 3D transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT3DX. • MODE=-2 : (default) initializations and a forward 3D transform are performed. • MODE=2 : (default) initializations and a backward 3D transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the values of L, M and M) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the values of L, M and N. DOUBLE PRECISION SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL LTRANS [Input] On input: if LTRANS is .TRUE. then a normal ?nal transposition is performed internally to return transformed data using the same storage format as the input data. If LTRANS is .FALSE. then the ?nal transposition is not performed and transformed data is stored, in X or Y, in transposed form. LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER L [Input] On input: L is the ?rst dimension of the 3D transform. INTEGER M [Input] On input: M is the second dimension of the 3D transform. INTEGER N [Input] On input: N is the third dimension of the 3D transform.Chapter 5: Fast Fourier Transforms (FFTs) 56 COMPLEX*16 X(L*M*N) [Input/Output] On input: X contains the L by M by N complex 3D data array to be transformed; the (ijk)th element is stored in X(i+(j-1)*L+(k-1)*L*M). On output: if INPL is .TRUE. then X contains the transformed data, either in the same locations as on input when LTRANS=.TRUE.; or in locations X(k+(j-1)*N+(i-1)*N*M) when LTRANS=.FALSE. If INPL is .FALSE. X remains unchanged. COMPLEX*16 Y(L*M*N) [Output] On output: if INPL is .FALSE. then Y contains the three-dimensional transformed data. If LTRANS=.TRUE. then the (ijk)th data element is stored in Y(i+(j-1)*L+(k-1)*L*M); otherwise, the (ijk)th data element is stored in Y(k+(j- 1)*N+(i-1)*N*M). If INPL is .TRUE. then Y is not referenced. COMPLEX*16 COMM(L*M*N+3*(L+M+N)+300) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence dimensions. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Forward 3D FFT is performed unscaled, without final transpose C and out-of-place on data stored in array X and output to Y. C Manipulations are stored in vector Y which is then transformed C back, with scaling, into the first M rows of X. C COMPLEX *16 X(L*M*N), Y(L*M*N) SCALE = 1.0D0 INPL = .FALSE. LTRANS = .FALSE. CALL ZFFT3DX(0,SCALE,LTRANS,INPL,L,M,N,X,Y,COMM,INFO) CALL ZFFT3DX(-1,SCALE,LTRANS,INPL,L,M,N,X,Y,COMM,INFO) IY = 1 DO 20 I = 1, L DO 40 J = 1, M DO 10 K = 1, N Y(IY) = Y(IY)*EXP(-0.001D0*DBLE(I+J+K-3)) IY = IY + 1 10 CONTINUE 20 CONTINUE SCALE = 1.0D0/DBLE(L*M*N) CALL ZFFT3DX(1,SCALE,LTRANS,INPL,N,M,L,Y,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 57 CFFT3DX Routine Documentation CFFT3DX (MODE,SCALE,LTRANS,INPL,L,M,N,X,Y, [SUBROUTINE] COMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by CFFT3DX. On input: • MODE=0 : only initializations (speci?c to the values of L, M and N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward 3D transform is performed. Initializations are assumed to have been performed by a prior call to CFFT3DX. • MODE=1 : a backward (reverse) 3D transform is performed. Initializations are assumed to have been performed by a prior call to CFFT3DX. • MODE=-2 : (default) initializations and a forward 3D transform are performed. • MODE=2 : (default) initializations and a backward 3D transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the values of L, M and M) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the values of L, M and N. REAL SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL LTRANS [Input] On input: if LTRANS is .TRUE. then a normal ?nal transposition is performed internally to return transformed data using the same storage format as the input data. If LTRANS is .FALSE. then the ?nal transposition is not performed and transformed data is stored, in X or Y, in transposed form. LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER L [Input] On input: L is the ?rst dimension of the 3D transform. INTEGER M [Input] On input: M is the second dimension of the 3D transform. INTEGER N [Input] On input: N is the third dimension of the 3D transform.Chapter 5: Fast Fourier Transforms (FFTs) 58 COMPLEX X(L*M*N) [Input/Output] On input: X contains the L by M by N complex 3D data array to be transformed; the (ijk)th element is stored in X(i+(j-1)*L+(k-1)*L*M). On output: if INPL is .TRUE. then X contains the transformed data, either in the same locations as on input when LTRANS=.TRUE.; or in locations X(k+(j-1)*N+(i-1)*N*M) when LTRANS=.FALSE. If INPL is .FALSE. X remains unchanged. COMPLEX Y(L*M*N) [Output] On output: if INPL is .FALSE. then Y contains the three-dimensional transformed data. If LTRANS=.TRUE. then the (ijk)th data element is stored in Y(i+(j-1)*L+(k-1)*L*M); otherwise, the (ijk)th data element is stored in Y(k+(j- 1)*N+(k-1)*N*M). If INPL is .TRUE. then Y is not referenced. COMPLEX COMM(*) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence dimensions. The remainder is used as temporary store. The amount of store required depends on the values of the arguments L, M, N and LTRANS. If LTRANS=.TRUE. then for a genuine 3D transform (all of L, M, N greater than 1) the dimension of COMM need only be 5*(L+M+N) + 150; in this case some further workspace will be allocated internally; the amount of allocated memory requested will be MAX(L*N+N, L*M + L + M, N). If LTRANS=.FALSE. then for a genuine 3D transform the workspace requirement is considerably more to allow for the storage of an intermediate 3D transposed array; in this case the dimension of COMM must be at least L*M*N+5*(L+M+N)+150. It is recommended that the appropriate 1D or 2D FFT routine be called when at least one of L, M or N is 1. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:Chapter 5: Fast Fourier Transforms (FFTs) 59   C Forward 3D FFT is performed unscaled, without final transpose C and out-of-place on data stored in array X and output to Y. C Manipulations are stored in vector Y which is then transformed C back, with scaling, into the first M rows of X. C SCALE = 1.0 INPL = .FALSE. LTRANS = .FALSE. CALL CFFT3DX(0,SCALE,LTRANS,INPL,L,M,N,X,Y,COMM,INFO) CALL CFFT3DX(-1,SCALE,LTRANS,INPL,L,M,N,X,Y,COMM,INFO) IY = 1 DO 20 I = 1, L DO 40 J = 1, M DO 10 K = 1, N Y(IY) = Y(IY)*EXP(-0.001*REAL(I+J+K-3)) IY = IY + 1 10 CONTINUE 20 CONTINUE SCALE = 1.0/REAL(L*M*N) CALL CFFT3DX(1,SCALE,LTRANS,INPL,N,M,L,Y,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 60 ZFFT3DY Routine Documentation ZFFT3DY (MODE,SCALE,INPL,L,M,N,X, [SUBROUTINE] INCX1,INCX2,INCX3,Y,INCY1,INCY2,INCY3,COMM,LCOMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZFFT3DY. On input: • MODE=0 : only initializations (speci?c to the values of L, M and N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward 3D transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT3DY. • MODE=1 : a backward (reverse) 3D transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT3DY. • MODE=-2 : (default) initializations and a forward 3D transform are performed. • MODE=2 : (default) initializations and a backward 3D transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the values of L, M and M) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the values of L, M and N. REAL SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER L [Input] On input: L is the ?rst dimension of the 3D transform. INTEGER M [Input] On input: M is the second dimension of the 3D transform. INTEGER N [Input] On input: N is the third dimension of the 3D transform. COMPLEX*16 X(*) [Input/Output] On input: X contains the L by M by N complex 3D data array to be transformed; the (ijk)th element is stored in X(1+(i-1)*INCX1+(j-1)*INCX2+(k- 1)*INCX3). On output: if INPL is .TRUE. then X contains the transformed data in the same locations as on input. If INPL is .FALSE. X remains unchanged.Chapter 5: Fast Fourier Transforms (FFTs) 61 INTEGER INCX1 [Input] On input: INCX1 is the step in index of X between successive data elements in the ?rst dimension of the 3D data. Usually INCX1=1 so that succesive elements in the ?rst dimension are stored contiguously. Constraint: INCX1 > 0. INTEGER INCX2 [Input] On input: INCX2 is the step in index of X between successive data elements in the second dimension of the 3D data. For completely contiguous data (no gaps in X) INCX2 should be set to L. Constraint: INCX2 > 0; INCX2 > (L-1)*INCX1 if max(M,N) > 1. INTEGER INCX3 [Input] On input: INCX3 is the step in index of X between successive data elements in the third dimension of the 3D data. For completely contiguous data (no gaps in X) INCX3 should be set to L*M. Constraint: INCX3 > 0; INCX3 > (L-1)*INCX1+(M-1)*INCX2 if N > 1. COMPLEX*16 Y(*) [Output] On output: if INPL is .FALSE. then Y contains the three-dimensional transformed data. If LTRANS=.TRUE. then the the (ijk)th element is stored in Y(1+(i-1)*INCY1+(j-1)*INCY2+(k-1)*INCY3). If INPL is .TRUE. then Y is not referenced. INTEGER INCY1 [Input] On input: if INPL is .FALSE. then INCY1 is the step in index of Y between successive data elements in the ?rst dimension of the 3D transformed data. Usually INCY1=1 so that succesive elements in the ?rst dimension are stored contiguously. If INPL is .TRUE. then INCY1 is not referenced. Constraint: If INPL is .FALSE. then INCY1 > 0. INTEGER INCY2 [Input] On input: if INPL is .FALSE. then INCY2 is the step in index of Y between successive data elements in the second dimension of the 3D transformed data. For completely contiguous data (no gaps in Y) INCY2 should be set to L. Constraint: INCY2 > 0 if INPL is .FALSE.; INCY2 > (L-1)*INCY1, if INPL is .FALSE. and max(M,N) > 1. INTEGER INCY3 [Input] On input: if INPL is .FALSE. then INCY3 is the step in index of Y between successive data elements in the third dimension of the 3D transformed data. For completely contiguous data (no gaps in Y) INCY3 should be set to L*M. Constraint: INCY3 > 0 if INPL is .FALSE.; INCY3 > (L-1)*INCY1+(M-1)*INCY2, if INPL is .FALSE. and N > 1.Chapter 5: Fast Fourier Transforms (FFTs) 62 COMPLEX*16 COMM(LCOMM) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence dimensions. The remainder is used as temporary store; if this is not su?cient for the requirements of the routine then temporary storage space will be dynamically allocated internally. INTEGER LCOMM [Input] On input: LCOMM is the length of the communication array COMM. The amount of internal dynamic allocation of temporary storage can be reduced signi?cantly by declaring COMM to be of length at least L*M*N + 4*(L+M+N) + 300. Constraint: LCOMM > 3*(L+M+N) + 150. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Forward 3D FFT is performed unscaled and in-place, on the leading C 10x10x10 submatrix of a larger 100x100x100 array of data. C The result is transformed back with scaling. C SCALE = 1.0D0 INPL = .TRUE. L = 10 M = 10 N = 10 LCOMM = 2000000 CALL ZFFT3DY(0,SCALE,INPL,L,M,N,X,1,100,10000,Y,1,1,1, * COMM,LCOMM,INFO) CALL ZFFT3DY(-1,SCALE,INPL,L,M,N,X,1,100,10000,Y,1,1,1, * COMM,LCOMM,INFO) IY = 1 DO 20 I = 1, L DO 40 J = 1, M DO 10 K = 1, N X(I,J,K) = X(I,J,K)*EXP(-1.0D-3*DBLE(I+J+K-3)) 10 CONTINUE 20 CONTINUE SCALE = 1.0/DBLE(L*M*N) CALL ZFFT3DY(1,SCALE,INPL,L,M,N,X,1,100,10000,Y,1,1,1, * COMM,LCOMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 63 CFFT3DY Routine Documentation CFFT3DY (MODE,SCALE,INPL,L,M,N,X, [SUBROUTINE] INCX1,INCX2,INCX3,Y,INCY1,INCY2,INCY3,COMM,LCOMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by CFFT3DY. On input: • MODE=0 : only initializations (speci?c to the values of L, M and N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward 3D transform is performed. Initializations are assumed to have been performed by a prior call to CFFT3DY. • MODE=1 : a backward (reverse) 3D transform is performed. Initializations are assumed to have been performed by a prior call to CFFT3DY. • MODE=-2 : (default) initializations and a forward 3D transform are performed. • MODE=2 : (default) initializations and a backward 3D transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the values of L, M and M) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the values of L, M and N. REAL SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER L [Input] On input: L is the ?rst dimension of the 3D transform. INTEGER M [Input] On input: M is the second dimension of the 3D transform. INTEGER N [Input] On input: N is the third dimension of the 3D transform. COMPLEX X(*) [Input/Output] On input: X contains the L by M by N complex 3D data array to be transformed; the (ijk)th element is stored in X(1+(i-1)*INCX1+(j-1)*INCX2+(k- 1)*INCX3). On output: if INPL is .TRUE. then X contains the transformed data in the same locations as on input. If INPL is .FALSE. X remains unchanged.Chapter 5: Fast Fourier Transforms (FFTs) 64 INTEGER INCX1 [Input] On input: INCX1 is the step in index of X between successive data elements in the ?rst dimension of the 3D data. Usually INCX1=1 so that succesive elements in the ?rst dimension are stored contiguously. Constraint: INCX1 > 0. INTEGER INCX2 [Input] On input: INCX2 is the step in index of X between successive data elements in the second dimension of the 3D data. For completely contiguous data (no gaps in X) INCX2 should be set to L. Constraint: INCX2 > 0; INCX2 > (L-1)*INCX1 if max(M,N) > 1. INTEGER INCX3 [Input] On input: INCX3 is the step in index of X between successive data elements in the third dimension of the 3D data. For completely contiguous data (no gaps in X) INCX3 should be set to L*M. Constraint: INCX3 > 0; INCX3 > (L-1)*INCX1+(M-1)*INCX2 if N > 1. COMPLEX Y(*) [Output] On output: if INPL is .FALSE. then Y contains the three-dimensional transformed data. If LTRANS=.TRUE. then the the (ijk)th element is stored in Y(1+(i-1)*INCY1+(j-1)*INCY2+(k-1)*INCY3). If INPL is .TRUE. then Y is not referenced. INTEGER INCY1 [Input] On input: if INPL is .FALSE. then INCY1 is the step in index of Y between successive data elements in the ?rst dimension of the 3D transformed data. Usually INCY1=1 so that succesive elements in the ?rst dimension are stored contiguously. If INPL is .TRUE. then INCY1 is not referenced. Constraint: If INPL is .FALSE. then INCY1 > 0. INTEGER INCY2 [Input] On input: if INPL is .FALSE. then INCY2 is the step in index of Y between successive data elements in the second dimension of the 3D transformed data. For completely contiguous data (no gaps in Y) INCY2 should be set to L. Constraint: INCY2 > 0 if INPL is .FALSE.; INCY2 > (L-1)*INCY1, if INPL is .FALSE. and max(M,N) > 1. INTEGER INCY3 [Input] On input: if INPL is .FALSE. then INCY3 is the step in index of Y between successive data elements in the third dimension of the 3D transformed data. For completely contiguous data (no gaps in Y) INCY3 should be set to L*M. Constraint: INCY3 > 0 if INPL is .FALSE.; INCY3 > (L-1)*INCY1+(M-1)*INCY2, if INPL is .FALSE. and N > 1.Chapter 5: Fast Fourier Transforms (FFTs) 65 COMPLEX COMM(LCOMM) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence dimensions. The remainder is used as temporary store; if this is not su?cient for the requirements of the routine then temporary storage space will be dynamically allocated internally. INTEGER LCOMM [Input] On input: LCOMM is the length of the communication array COMM. The amount of internal dynamic allocation of temporary storage is dependent on the values of the increment arguments for arrays X and Y. The amount is minimized when the increments for the output array are 1, L and L*M respectively, since this represents a contiguous array in which calculations can be performedd inplace. Constraint: If min(L,M,N) > 1, LCOMM >= 5*(L+M+N) + 150 (otherwise see see [CFFT2DX], page 49). INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Forward 3D FFT is performed unscaled and in-place, on the leading C 10x10x10 submatrix of a larger 100x100x100 array of data. C The result is transformed back with scaling. C SCALE = 1.0 INPL = .TRUE. L = 10 M = 10 N = 10 LCOMM = 2000000 CALL CFFT3DY(0,SCALE,INPL,L,M,N,X,1,100,10000,Y,1,1,1, * COMM,LCOMM,INFO) CALL CFFT3DY(-1,SCALE,INPL,L,M,N,X,1,100,10000,Y,1,1,1, * COMM,LCOMM,INFO) IY = 1 DO 20 I = 1, L DO 40 J = 1, M DO 10 K = 1, N X(I,J,K) = X(I,J,K)*EXP(-0.001*REAL(I+J+K-3)) 10 CONTINUE 20 CONTINUE SCALE = 1.0/REAL(L*M*N) CALL CFFT3DY(1,SCALE,INPL,L,M,N,X,1,100,10000,Y,1,1,1, * COMM,LCOMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 66 5.3 FFTs on real and Hermitian data sequences The routines documented here compute discrete Fourier transforms (DFTs) of sequences of real numbers or of Hermitian sequences in either single or double precision arithmetic. The DFTs are computed using a highly-e?cient FFT algorithm. Hermitian sequences are represented in a condensed form that is described in Section 5.1 [Introduction to FFTs], page 24. The DFT of a real sequence results in a Hermitian sequence; the DFT of a Hermitian sequence is a real sequence. Please note that prior to Release 2.0 of ACML the routine ZDFFT/CSFFT and ZDFFTM/CSFFTM returned results that were scaled by a factor 0.5 compared with the currently returned results.Chapter 5: Fast Fourier Transforms (FFTs) 67 5.3.1 FFT of single sequences of real data DZFFT Routine Documentation DZFFT (MODE,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by DZFFT. On input: • MODE=0 : only default initializations (speci?c to N) are performed; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=1 : a real transform is performed. Initializations are assumed to have been performed by a prior call to DZFFT. • MODE=2 : (default) initializations and a real transform are performed. • MODE=100 : similar to MODE=0; only initializations are performed, but ?rst a plan is generated. This plan is chosen based on the fastest FFT computation for a subset of all possible plans. INTEGER N [Input] On input: N is the length of the real sequence X DOUBLE PRECISION X(N) [Input/Output] On input: X contains the real sequence of length N to be transformed. On output: X contains the transformed Hermitian sequence. DOUBLE PRECISION COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL DZFFT(0,N,X,COMM,INFO) CALL DZFFT(1,N,X,COMM,INFO) DO 10 I = N/2+2, N X(I) = -X(I) 10 CONTINUE CALL ZDFFT(2,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 68 SCFFT Routine Documentation SCFFT (MODE,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by SCFFT. On input: • MODE=0 : only default initializations (speci?c to N) are performed; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=1 : a real transform is performed. Initializations are assumed to have been performed by a prior call to SCFFT. • MODE=2 : (default) initializations and a real transform are performed. • MODE=100 : similar to MODE=0; only initializations are performed, but ?rst a plan is generated. This plan is chosen based on the fastest FFT computation for a subset of all possible plans. INTEGER N [Input] On input: N is the length of the real sequence X REAL X(N) [Input/Output] On input: X contains the real sequence of length N to be transformed. On output: X contains the transformed Hermitian sequence. REAL COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL SCFFT(0,N,X,COMM,INFO) CALL SCFFT(1,N,X,COMM,INFO) DO 10 I = N/2+2, N X(I) = -X(I) 10 CONTINUE CALL CSFFT(2,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 69 5.3.2 FFT of multiple sequences of real data DZFFTM Routine Documentation DZFFTM (M,N,X,COMM,INFO) [SUBROUTINE] INTEGER M [Input] On input: M is the number of sequences to be transformed. INTEGER N [Input] On input: N is the length of the real sequences in X DOUBLE PRECISION X(N*M) [Input/Output] On input: X contains the M real sequences of length N to be transformed. Element i of sequence j is stored in location i + (j - 1) * N of X. On output: X contains the transformed Hermitian sequences. DOUBLE PRECISION COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL DZFFTM(1,N,X,COMM,INFO) CALL DZFFTM(2,N,X,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*X(N-I+1,2) 10 CONTINUE CALL ZDFFTM(2,N,X(1,3),COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 70 SCFFTM Routine Documentation SCFFTM (M,N,X,COMM,INFO) [SUBROUTINE] INTEGER M [Input] On input: M is the number of sequences to be transformed. INTEGER N [Input] On input: N is the length of the real sequences in X REAL X(N*M) [Input/Output] On input: X contains the M real sequences of length N to be transformed. Element i of sequence j is stored in location i + (j - 1) * N of X. On output: X contains the transformed Hermitian sequences. REAL COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL SCFFTM(1,N,X,COMM,INFO) CALL SCFFTM(2,N,X,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*X(N-I+1,2) 10 CONTINUE CALL CSFFTM(1,N,X(1,3),COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 71 5.3.3 FFT of single Hermitian sequences ZDFFT Routine Documentation ZDFFT (MODE,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZDFFT. On input: • MODE=0 : only initializations (speci?c to the values of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=1. • MODE=1 : a real transform is performed. Initializations are assumed to have been performed by a prior call to ZDFFT. • MODE=2 : (default) initializations and a real transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. INTEGER N [Input] On input: N is length of the sequence in X DOUBLE PRECISION X(N) [Input/Output] On input: X contains the Hermitian sequence of length N to be transformed. On output: X contains the transformed real sequence. DOUBLE PRECISION COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL DZFFT(0,N,X,COMM,INFO) CALL DZFFT(1,N,X,COMM,INFO) DO 10 I = N/2+2, N X(I) = -X(I) 10 CONTINUE CALL ZDFFT(2,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 72 CSFFT Routine Documentation CSFFT (MODE,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by CSFFT. On input: • MODE=0 : only initializations (speci?c to the values of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=1. • MODE=1 : a real transform is performed. Initializations are assumed to have been performed by a prior call to CSFFT. • MODE=2 : (default) initializations and a real transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. INTEGER N [Input] On input: N is the length of the sequence in X REAL X(N) [Input/Output] On input: X contains the Hermitian sequence of length N to be transformed. On output: X contains the transformed real sequence. REAL COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL SCFFT(0,N,X,COMM,INFO) CALL SCFFT(1,N,X,COMM,INFO) DO 10 I = N/2+2, N X(I) = -X(I) 10 CONTINUE CALL CSFFT(2,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 73 5.3.4 FFT of multiple Hermitian sequences ZDFFTM Routine Documentation ZDFFTM (M,N,X,COMM,INFO) [SUBROUTINE] INTEGER M [Input] On input: M is the number of sequences to be transformed. INTEGER N [Input] On input: N is the length of the sequences in X DOUBLE PRECISION X(N*M) [Input/Output] On input: X contains the M Hermitian sequences of length N to be transformed. Element i of sequence j is stored in location i + (j - 1) * N of X. On output: X contains the transformed real sequences. DOUBLE PRECISION COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL DZFFTM(1,N,X,COMM,INFO) CALL DZFFTM(2,N,X,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*X(N-I+1,2) 10 CONTINUE CALL ZDFFTM(1,N,X(1,3),COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 74 CSFFTM Routine Documentation CSFFTM (M,N,X,COMM,INFO) [SUBROUTINE] INTEGER M [Input] On input: M is the number of sequences to be transformed. INTEGER N [Input] On input: N is the length of the sequences in X REAL X(N*M) [Input/Output] On input: X contains the M Hermitian sequences of length N to be transformed. Element i of sequence j is stored in location i + (j - 1) * N of X. On output: X contains the transformed real sequences. REAL COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL SCFFTM(1,N,X,COMM,INFO) CALL SCFFTM(2,N,X,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*X(N-I+1,2) 10 CONTINUE CALL CSFFTM(1,N,X(1,3),COMM,INFO) Chapter 6: Random Number Generators 75 6 Random Number Generators Within the context of this document, a base random number generator (BRNG) is a mathematical algorithm that, given an initial state, produces a sequence (or stream) of variates (or values) uniformly distributed over the semi-open interval (0,1]. The period of the BRNG is de?ned as the maximum number of values that can be generated before the sequence starts to repeat. The initial state of a BRNG is often called the seed. Note that this de?nition means that the value 1.0 may be returned, but the value 0.0 will not. A pseudo-random number generator (PRNG) is a BRNG that produces a stream of variates that are independent and statistically indistinguishable from a random sequence. A PRNG has several advantages over a true random number generator in that the generated sequence is repeatable, has known mathematical properties and is usually much quicker to generate. A quasi-random number generator (QRNG) is similar to a PRNG, however the variates generated are not statistically independent, rather they are designed to give a more even distribution in multidimensional space. Many books on statistics and computer science have good introductions to PRNGs and QRNGs, see for example Knuth [6] or Banks [7]. All of the BRNGs supplied in the ACML are PRNGs. In addition to standard PRNGs some applications require cryptologically secure generators. A PRNG is said to be cryptologically secure if there is no polynomial-time algorithm which, on input of the ?rst l bits of the output sequence can predict the (l + 1)st bit of the sequence with probability signi?cantly greater than 0.5. This is equivalent to saying there exists no polynomial-time algorithm that can correctly distinguish between an output sequence from the PRNG and a truly random sequence of the same length with probability signi?cantly greater than 0.5 [8]. A distribution generator is a routine that takes variates generated from a BRNG and transforms them into variates from a speci?ed distribution, for example the Gaussian (Normal) distribution. The ACML contains ?ve base generators, (Section 6.1 [Base Generators], page 75), and twenty-three distribution generators (Section 6.3 [Distribution Generators], page 97). In addition users can supply a custom built generator as the base generator for all of the distribution generators (Section 6.1.8 [User Supplied Generators], page 86). The base generators were tested using the Big Crush, Small Crush and Pseudo Diehard test suites from the TestU01 software library [15]. 6.1 Base Generators The ?ve base generators (BRNGs) supplied with the ACML are; the NAG basic generator [9], a series of Wichmann-Hill generators [10], the Mersenne Twister [11], L’Ecuyer’s combined recursive generator MRG32k3a [12] and the Blum-Blum-Shub generator [8]. Some of the generators have been slightly modi?ed from their usual form to make them consistent between themselves. For instance, the Wichmann-Hill generators in standard form may return exactly 0.0 but not exactly 1.0. In ACML we return 1.0-x to convert the value x into the semi-open interval (0, 1] without a?ecting any other randomness properties. The original Mersenne Twister algorithm returns an exact zero about one time in a few billion; the ACML implementation returns a tiny non-zero number as surrogate for zero.Chapter 6: Random Number Generators 76 If a single stream of variates is required it is recommended that the Mersenne Twister (Section 6.1.5 [Mersenne Twister], page 84) base generator is used. This generator combines speed with good statistical properties and an extremely long period. The NAG basic generator (Section 6.1.3 [Basic NAG Generator], page 83) is another quick generator suitable for generating a single stream. However it has a shorter period than the Mersenne Twister and being a linear congruential generator, its statistical properties are not as good. If 273 or fewer multiple streams, with a period of up to 2 80 are required then it is recommended that the Wichmann-Hill generators are used (Section 6.1.4 [Wichmann-Hill Generator], page 84). For more streams or multiple streams with a longer period it is recommended that the L’Ecuyer combined recursive generator (Section 6.1.6 [L’Ecuyer’s Combined Recursive Generator], page 85) is used in combination with the skip ahead routine (Section 6.2.3 [Skip Ahead], page 91). Generating multiple streams of variates by skipping ahead is generally quicker than generating the streams using the leap frog method. More details on multiple streams can be found in Section 6.2 [Multiple Streams], page 90. The Blum-Blum-Shub generator (Section 6.1.7 [Blum-Blum-Shub Generator], page 85) should only be used if a cryptologically secure generator is required. This generator is extremely slow and has poor statistical properties when used as a base generator for any of the distributional generators. 6.1.1 Initialization of the Base Generators A random number generator must be initialized before use. Three routines are supplied within the ACML for this purpose: DRANDINITIALIZE, DRANDINITIALIZEBBS and DRANDINITIALIZEUSER (see [DRANDINITIALIZE], page 78, [DRANDINITIALIZEBBS], page 81 and [DRANDINITIALIZEUSER], page 87, respectively). Of these, DRANDINITIALIZE is used to initialize all of the supplied base generators, DRANDINITIALIZEBBS supplies an alternative interface to DRANDINITIALIZE for the Blum-Blum-Shub generator, and DRANDINITIALIZEUSER allows the user to register and initialize their own base generator. Both double and single precision versions of all RNG routines are supplied. Double precision names are pre?xed by DRAND, and single precision by SRAND. Note that if a generator has been initialized using the relevant double precision routine, then the double precision versions of the distribution generators must also be used, and vice versa. This even applies to generators with no double or single precision parameters; for example, a call of DRANDDISCRETEUNIFORM must be preceded by a call to one of the double precision initializers (typically DRANDINITIALIZE). No utilities for saving, retrieving or copying the current state of a generator have been provided. All of the information on the current state of a generator (or stream, if multiple streams are being used) is stored in the integer array STATE and as such this array can be treated as any other integer array, allowing for easy copying, restoring etc. The statistical properties of a sequence of random numbers are only guaranteed within the sequence, and not between sequences provided by the same generator. Therefore it is likely that repeated initialization will render the numbers obtained less, rather than more, independent. In most cases there should only be a single call to one of the initialization routines, per application, and this call must be made before any variates are generated. One example of where multiple initialization may be required is brie?y touched upon in Section 6.2 [Multiple Streams], page 90.Chapter 6: Random Number Generators 77 In order to initialize the Blum-Blum-Shub generator a number of additional parameters, as well as an initial state (seed), are required. Although this generator can be initialized through the DRANDINITIALIZE routine it is recommended that the DRANDINITIALIZEBBS routine is used instead.Chapter 6: Random Number Generators 78 DRANDINITIALIZE / SRANDINITIALIZE Initialize one of the ?ve supplied base generators; NAG basic generator, Wichmann-Hill generator, Mersenne Twister, L’Ecuyer’s combined recursive generator (MRG32k3a) or the Blum-Blum-Shub generator. (Note that SRANDINITIALIZE is the single precision version of DRANDINITIALIZE. The argument lists of both routines are identical except that any double precision arguments of DRANDINITIALIZE are replaced in SRANDINITIALIZE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDINITIALIZE (GENID,SUBID,SEED,LSEED,STATE, [SUBROUTINE] LSTATE,INFO) INTEGER GENID [Input] On input: a numerical code indicating which of the ?ve base generators to initialize. • 1 = NAG basic generator (Section 6.1.3 [Basic NAG Generator], page 83). • 2 = Wichmann-Hill generator (Section 6.1.4 [Wichmann-Hill Generator], page 84). • 3 = Mersenne Twister (Section 6.1.5 [Mersenne Twister], page 84). • 4 = L’Ecuyer’s Combined Recursive generator (Section 6.1.6 [L’Ecuyer’s Combined Recursive Generator], page 85). • 5 = Blum-Blum-Shub generator (Section 6.1.7 [Blum-Blum-Shub Generator], page 85). Constraint: 1= GENID = 5. INTEGER SUBID [Input] On input: if GENID = 2, then SUBID indicates which of the 273 WichmannHill generators to use. If GENID = 5 then SUBID indicates the number of bits to use (v) from each of iteration of the Blum-Blum-Shub generator. In all other cases SUBID is not referenced. Constraint: If GENID = 2 then 1= SUBID = 273 . INTEGER SEED(LSEED) [Input] On input: if GENID= 5 , then 6 SEED is a vector of initial values for the base generator. These values must be positive integers. The number of values required depends on the base generator being used. The NAG basic generator requires one initial value, the Wichmann-Hill generator requires four initial values, the L’Ecuyer combined recursive generator requires six initial values and the Mersenne Twister requires 624 initial values. If the number of seeds required by the chosen generator is > LSEED then SEED(1) is used to initialize the NAG basic generator. This is then used to generate all of the remaining seed values required. In general it is best not to set all the elements of SEED to anything too obvious, such as a single repeated value or a simple sequence. Using such a seed array may lead to several similar values being created in a row when the generator is subsequently called. This is particularly true for the Mersenne Twister generator.Chapter 6: Random Number Generators 79 In order to initialize the Blum-Blum-Shub generator two large prime values, p and q are required as well as an initial value s. As p, q and s can be of an arbitrary size, these values are expressed as a polynomial in B, where B = 2 24 . For example, p can be factored into a polynomial of order lp, with p = p1 + p2B + p3B2 + · · · + plpBlp-1 . The elements of SEED should then be set to the following: • SEED(1) = lp • SEED(2) to SEED(lp + 1) = p1 to plp • SEED(lp + 2) = lq • SEED(lp + 3) to SEED(lp + lq + 2) = q1 to qlq • SEED(lp + lq + 3) = ls • SEED(lp + lq + 4) to SEED(lp + lq + ls + 3) = s1 to sls Constraint: If GENID= 5 then 6 SEED(i) > 0, i = 1, 2, · · ·. If GENID = 5 then SEED must take the values described above. INTEGER LSEED [Input/Output] On input: either the length of the seed vector, SEED, or a value = 0 . On output: if LSEED= 0 on input, then LSEED is set to the number of initial values required by the selected generator, and the routine returns. Otherwise LSEED is left unchanged. INTEGER STATE(LSTATE) [Output] On output: the state vector required by all of the supplied distributional and base generators. INTEGER LSTATE [Input/Output] On input: either the length of the state vector, STATE, or a value = 0 . On output: if LSTATE= 0 on input, then LSTATE is set to the minimum length of the state vector STATE for the base generator chosen, and the routine returns. Otherwise LSTATE is left unchanged. Constraint: LSTATE= 0 or the minimum length for the chosen base generator, given by: • GENID = 1: LSTATE= 16, • GENID = 2: LSTATE= 20, • GENID = 3: LSTATE= 633, • GENID = 4: LSTATE= 61, • GENID = 5: LSTATE= lp + lq + ls + 6, where lp, lq and ls are the order of the polynomials used to express the parameters p, q and s respectively. INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then either, or both of LSEED and / or LSTATE have been set to the required length for vectors SEED and STATE respectively. Of the two variables LSEED and LSTATE, only those which had an input value = 0 will have been set. The STATE vector will not have been initialized. If INFO = 0 then the state vector, STATE, has been successfully initialized.Chapter 6: Random Number Generators 80 Example:   C Generate 100 values from the Beta distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A,B DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Beta distribution CALL DRANDBETA(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 81 DRANDINITIALIZEBBS / SRANDINITIALIZEBBS Alternative initialization routine for the Blum-Blum-Shub generator. Unlike the other base generators supplied with the ACML, the Blum-Blum-Shub generator requires two additional parameters, p and q as well as an initial state, s. The parameters p, q and s can be of an arbitrary size. In order to avoid over?ow these values are expressed as a polynomial in B, where B = 2 24 . For example, p can be factored into a polynomial of order lp, with p = p1 + p2B + p3B2 + · · · + plpBlp-1 , similarly q = q1 + q2B + q3B2 + · · · + qlqBlq-1 and s = s1 + s2B + s3B2 + · · · + slsBls-1 . (Note that SRANDINITIALIZEBBS is the single precision version of DRANDINITIALIZEBBS. The argument lists of both routines are identical except that any double precision arguments of DRANDINITIALIZEBBS are replaced in SRANDINITIALIZEBBS by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDINITIALIZEBBS (NBITS,LP,P,LQ,Q,LS,S,STATE,LSTATE, [SUBROUTINE] INFO) INTEGER NBITS [Input] On input: the number of bits, v, to use from each iteration of the BlumBlum-Shub generator. If NBITS < 1 then NBITS = 1. If NBITS > 15 then NBITS = 15. INTEGER LP [Input] On input: the order of the polynomial used to express p (lp). Constraint: 1 = LP = 25. INTEGER P(LP) [Input] On input: the coe?cients of the polynomial used to express p. P(i) = pi , i = 1 to lp. Constraint: 0 = P (i) < 2 24 INTEGER LQ [Input] On input: the order of the polynomial used to express q (lq). Constraint: 1 = LQ = 25. INTEGER Q(LQ) [Input] On input: the coe?cients of the polynomial used to express q. Q(i) = qi , i = 1 to lq. Constraint: 0 = Q (i) < 2 24 INTEGER LS [Input] On input: the order of the polynomial used to express s (ls). Constraint: 1 = LS = 25. INTEGER S(LS) [Input] On input: the coe?cients of the polynomial used to express s. S(i) = si , i = 1 to ls. Constraint: 0 = S (i) < 2 24 INTEGER STATE(*) [Output] On output: the initial state for the Blum-Blum-Shub generator with parameters P,Q,S and NBITS.Chapter 6: Random Number Generators 82 INTEGER LSTATE [Input/Output] On input: either the length of the state vector, STATE, or a value = 0 . On output: if LSTATE= 0 on input, then LSTATE is set to the minimum length of the state vector STATE for the parameters chosen, and the routine returns. Otherwise LSTATE is left unchanged. Constraint: LSTATE= 0 or LSTATE = lp + lq + ls + 6 INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LSTATE has been set to the required length for the STATE vector. If INFO = 0 then the state vector, STATE, has been successfully initialized. 6.1.2 Calling the Base Generators With the exception of the Blum-Blum-Shub generator, there are no interfaces for direct access to the base generators. All of the base generators return variates uniformly distributed over the semi-open interval (0, 1]. This functionality can be accessed using the uniform distributional generator DRANDUNIFORM, with parameter A = 0.0 and parameter B = 1.0 (see [DRANDUNIFORM], page 119). The base generator used is, as usual, selected during the initialization process (see Section 6.1.1 [Initialization of the Base Generators], page 76). To directly access the Blum-Blum-Shub generator, use the routine DRANDBLUMBLUMSHUB.Chapter 6: Random Number Generators 83 DRANDBLUMBLUMSHUB / SRANDBLUMBLUMSHUB Allows direct access to the bit stream generated by the Blum-Blum-Shub generator. (Note that SRANDBLUMBLUMSHUB is the single precision version of DRANDBLUMBLUMSHUB. The argument lists of both routines are identical except that any double precision arguments of DRANDBLUMBLUMSHUB are replaced in SRANDBLUMBLUMSHUB by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDBLUMBLUMSHUB (N,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. The total number of bits generated is 24N. Constraint: N= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDBLUMBLUMSHUB STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector holding the bit stream. The least signi?cant 24 bits of each of the X(i) contain the bit stream as generated by the Blum-Blum-Shub generator. The least signi?cant bit of X(1) is the ?rst bit generated, the second least signi?cant bit of X(1) is the second bit generated etc. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. 6.1.3 Basic NAG Generator The NAG basic generator is a linear congruential generator (LCG) and, like all LCGs, has the form: xi = a1xi-1 mod m1, ui = xi m1 , where the ui , i = 1, 2, · · · form the required sequence. The NAG basic generator takes a1 = 13 13 and m1 = 2 59 , which gives a period of approximately 2 57 . This generator has been part of the NAG numerical library [9] since Mark 6 and as such has been widely used. It su?ers from no known problems, other than those due to the lattice structure inherent in all LCGs, and, even though the period is relatively short compared to many of the newer generators, it is su?ciently large for many practical problems.Chapter 6: Random Number Generators 84 6.1.4 Wichmann-Hill Generator The Wichmann-Hill [10] base generator uses a combination of four linear congruential generators (LCGs) and has the form: wi = a1wi-1 mod m1 xi = a2xi-1 mod m2 yi = a3yi-1 mod m3 zi = a4zi-1 mod m4 ui = ( wi m1 + xi m2 + yi m3 + zi m4 ) mod 1, where the ui , i = 1, 2, · · · form the required sequence. There are 273 sets of parameters, {ai , mi : i = 1, 2, 3, 4}, to choose from. These values have been selected so that the resulting generators are independent and have a period of approximately 2 80 [10]. 6.1.5 Mersenne Twister The Mersenne Twister [11] is a twisted generalized feedback shift register generator. The algorithm is as follows: • Set some arbitrary initial values x1, x2, · · · , xr, each consisting of w bits. • Letting A =  0 Iw-1 aw aw-1 · · · a1  , where Iw-1 is the (w - 1) × (w - 1) identity matrix and each of the ai , i = 1 to w take a value of either 0 or 1 (i.e. they can be represented as bits). De?ne xi+r = (xi+s ? (x (w:(l+1)) i |x (l:1) i+1 )A), where x (w:(l+1)) i |x (l:1) i+1 indicates the concatenation of the most signi?cant (upper) w - l bits of xi and the least signi?cant (lower) l bits of xi+1. • Perform the following operations sequentially: z = xi+r ? (xi+r  t1) z = z ? ((z  t2) AND m1) z = z ? ((z  t3) AND m2) z = z ? (z  t4) ui+r = z/(2 w - 1), where t1, t2, t3 and t4 are integers and m1 and m2 are bit-masks and “ t” and “ t” represent a t bit shift right and left respectively, ? is bit-wise exclusively or (xor) operation and “AND” is a bit-wise and operation.Chapter 6: Random Number Generators 85 The ui+r : i = 1, 2, · · · then form a pseudo-random sequence, with ui ? (0, 1), for all i. This implementation of the Mersenne Twister uses the following values for the algorithmic constants: w = 32 a = 0x9908b0df l = 31 r = 624 s = 397 t1 = 11 t2 = 7 t3 = 15 t4 = 18 m1 = 0x9d2c5680 m2 = 0xefc60000 where the notation 0xDD · · · indicates the bit pattern of the integer whose hexadecimal representation is DD · · ·. This algorithm has a period length of approximately 2 19,937 - 1 and has been shown to be uniformly distributed in 623 dimensions. 6.1.6 L’Ecuyer’s Combined Recursive Generator The base generator referred to as L’Ecuyer’s combined recursive generator is referred to as MRG32k3a in [12] and combines two multiple recursive generators: xi = a11xi-1 + a12xi-2 + a13xi-3 mod m1 yi = a21yi-1 + a22yi-2 + a23yi-3 mod m2 zi = xi - yi mod m1 ui = zi m1 , where the ui , i = 1, 2, · · · form the required sequence and a11 = 0, a12 = 1403580, a13 = -810728, m1 = 2 32 - 209, a21 = 527612, a22 = 0, a23 = -1370589 and m2 = 2 32 - 22853. Combining the two multiple recursive generators (MRG) results in sequences with better statistical properties in high dimensions and longer periods compared with those generated from a single MRG. The combined generator described above has a period length of approximately 2 191 6.1.7 Blum-Blum-Shub Generator The Blum-Blum-Shub pseudo random number generator is cryptologically secure under the assumption that the quadratic residuosity problem is intractable [8]. The algorithm consists of the following: • Generate two large and distinct primes, p and q, each congruent to 3 mod 4. De?ne m = pq. • Select a seed s taking a value between 1 and m - 1, such that the greatest common divisor between s and m is 1.Chapter 6: Random Number Generators 86 • Let x0 = s 2 mod m. For i = 1, 2, · · · generate: xi = x 2 i-1 mod m zi = v least signi?cant bits of xi where v= 1 . • The bit-sequence z1, z2, z3, · · · is then the output sequence used. 6.1.8 User Supplied Generators All of the distributional generators described in Section 6.3 [Distribution Generators], page 97 require a base generator which returns a uniformly distributed value in the semiopen interval (0, 1] and ACML includes several such generators (as detailed in Section 6.1 [Base Generators], page 75). However, for greater ?exibility, the ACML routines allow the user to register their own base generator function. This user-supplied generator then becomes the base generator for all of the distribution generators. A user supplied generator comes in the form of two routines, one to initialize the generator and one to generate a set of uniformly distributed values in the semi-open interval (0, 1]. These two routines can be named anything, but are referred to as UINI for the initialization routine and UGEN for the generation routine in the following documentation. In order to register a user supplied generator a call to DRANDINITIALIZEUSER must be made. Once registered the generator can be accessed and used in the same manner as the ACML supplied base generators. The speci?cations for DRANDINTIALIZEUSER, UINI and UGEN are given below. See the ACML example programs drandinitializeuser_example.f and drandinitializeuser_c_example.c (Section 2.9 [Examples], page 16) to understand how to use these routines in ACML.Chapter 6: Random Number Generators 87 DRANDINITIALIZEUSER / SRANDINITIALIZEUSER Registers a user supplied base generator so that it can be used with the ACML distributional generators. (Note that SRANDINITIALIZEUSER is the single precision version of DRANDINITIALIZEUSER. The argument lists of both routines are identical except that any double precision arguments of DRANDINITIALIZEUSER are replaced in SRANDINITIALIZEUSER by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDINITIALIZEUSER (UINI,UGEN,GENID,SUBID,SEED,LSEED, [SUBROUTINE] STATE,LSTATE,INFO) SUBROUTINE UINI [Input] On input: routine that will be used to initialize the user supplied generator, UGEN. SUBROUTINE UGEN [Input] On input: user supplied base generator. INTEGER GENID [Input] On input: parameter is passed directly to UINI. Its function therefore depends on that routine. INTEGER SUBID [Input] On input: parameter is passed directly to UINI. Its function therefore depends on that routine. INTEGER SEED(LSEED) [Input] On input: parameter is passed directly to UINI. Its function therefore depends on that routine. INTEGER LSEED [Input/Output] On input: length of the vector SEED. This parameter is passed directly to UINI and therefore its required value depends on that routine. On output: whether LSEED changes will depend on UINI. INTEGER STATE(LSTATE) [Output] On output: the state vector required by all of the supplied distributional generators. The value of STATE returned by UINI has some housekeeping elements appended to the end before being returned by DRANDINITIALIZEUSER. See Section 6.1.8 [User Supplied Generators], page 86 for details about the form of STATE. INTEGER LSTATE [Input/Output] On input: length of the vector STATE. This parameter is passed directly to UINI and therefore its required value depends on that routine. On output: whether LSTATE changes will depend on UINI. If LSTATE= 0 then it is assumed that a request for the required length of STATE has been made. The value of LSTATE returned from UINI is therefore adjusted to allow for housekeeping elements to be added to the end of the STATE vector. This results in the value of LSTATE returned by DRANDINITIALIZEUSER being 3 larger than that returned by UINI.Chapter 6: Random Number Generators 88 INTEGER INFO [Output] On output: INFO is an error indicator. DRANDINITIALIZEUSER will return a value of -6 if the value of LSTATE is between 1 and 3. Otherwise INFO is passed directly back from UINI. It is recommended that the value of INFO returned by UINI is kept consistent with the rest of the ACML, that is if INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then either, or both of LSEED and / or LSTATE have been set to the required length for vectors SEED and STATE respectively and the STATE vector has not have been initialized. If INFO = 0 then the state vector, STATE, has been successfully initialized. Example:   C Generate 100 values from the Uniform distribution using C a user supplied base generator INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,NSKIP,SEED(1),STATE(LSTATE) INTEGER X(N) DOUBLE PRECISION A,B C Set the seed SEED(1) = 1234 C Set the distributional parameters A = 0.0D0 B = 1.0D0 C Initialize the base generator. Here ACMLRNGNB0GND is a user C supplied generator and ACMLRNGNB0INI its initializer CALL DRANDINITIALIZEUSER(ACMLRNGNB0INI,ACMLRNGNB0GND,1,0,SEED, * LSEED,STATE,LSTATE,INFO) C Generate N variates from the Univariate distribution CALL DRANDUNIFORM(N,A,B,STATE,X,LDX,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 89 UINI Speci?cation for a user supplied initialization routine. UINI (GENID,SUBID,SEED,LSEED,STATE,LSTATE,INFO) [SUBROUTINE] INTEGER GENID [Input] On input: the ID associated with the generator. It may be used for anything you like. INTEGER SUBID [Input] On input: the sub-ID associated with the generator. It may be used for anything you like. INTEGER SEED(LSEED) [Input] On input: an array containing the initial seed for your generator. INTEGER LSEED [Input/Output] On input: either the size of the SEED array, or a value < 1. On output: if LSEED < 1 on entry, LSEED must be set to the required size of the SEED array. This allows a caller of UINI to query the required size. INTEGER STATE(LSTATE) [Output] On output: if LSTATE < 1 on entry, STATE should be unchanged. Otherwise, STATE is a state vector holding internal details required by your generator. On exit from UINI, the array STATE must hold the following information: STATE(1) = ESTATE, where ESTATE is your minimum allowed size of array STATE. STATE(2) = MAGIC, where MAGIC is a magic number of your own choice. This can be used by your routine UGEN as a check that UINI has previously been called. STATE(3) = GENID STATE(4) = SUBID STATE(5) ... STATE(ESTATE-1) = internal state values required by your generator routine UGEN; for example, the current value of your seed. STATE(ESTATE) = MAGIC, i.e. the same value as STATE(2). INTEGER LSTATE [Input/Output] On input: either the size of the STATE array, or a value < 1. On output: if LSTATE < 1 on entry, LSTATE should be set to the required size of the STATE array, i.e. the value ESTATE as described above. This allows the caller of UINI to query the required size. Constraint: either LSTATE < 1 or LSTATE= EST AT E . INTEGER INFO [Output] On output: an error code, to be used in whatever way you wish; for example to ?ag an incorrect argument to UINI. If no error is encountered, UINI must set INFO to 0.Chapter 6: Random Number Generators 90 UGEN Speci?cation for a user supplied base generator. UGEN (N,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: the number of random numbers to be generated. INTEGER STATE(*) [Input/Output] On input: the internal state of your generator. DOUBLE PRECISION X(N) [Output] On output: the array of N uniform distributed random numbers, each in the semi-open interval (0.0, 1.0] - i.e. 1.0 is a legitimate return value, but 0.0 is not. INTEGER INFO [Output] On output: a ?ag which you can use to signal an error in the call of UGEN - for example, if UGEN is called without being initialized by UINI. 6.2 Multiple Streams It is often advantageous to be able to generate variates from multiple, independent, streams. For example when running a simulation in parallel on several processors. There are four ways of generating multiple streams using the routines available in the ACML: • (a) Using di?erent seeds • (b) Using di?erent sequences • (c) Block-splitting or skipping ahead • (d) Leap frogging The four methods are detailed in the following sections. Of the four, (a) should be avoided in most cases, (b) is only really of any practical use when using the Wichmann-Hill generator, and is then still limited to 273 streams. Both block-splitting and leap-frogging work using the sequence from a single generator, both guarantee that the di?erent sequences will not overlap and both can be scaled to an arbitrary number of streams. Leap-frogging requires no a-priori knowledge about the number of variates being generated, whereas block-splitting requires the user to know (approximately) the maximum number of variates required from each stream. Block-splitting requires no a-priori information on the number of streams required. In contrast leap-frogging requires the user to know the maximum number of streams required, prior to generating the ?rst value. It is known that, dependent on the number of streams required, leap-frogging can lead to sequences with poor statistical properties, especially when applied to linear congruential generators (see Section 6.2.4 [Leap Frogging], page 94 for a brief explanation). In addition, for more complicated generators like a L’Ecuyer’s multiple recursive generator leap-frogging can increase the time required to generate each variate compared to block-splitting. The additional time required by block-splitting occurs at the initialization stage, and not at the variate generation stage. Therefore in most instances block-splitting would be the preferred method for generating multiple sequences.Chapter 6: Random Number Generators 91 6.2.1 Using Di?erent Seeds A di?erent sequence of variates can be generated from the same base generator by initializing the generator using a di?erent set of seeds. Of the four methods for creating multiple streams described here, this is the least satisfactory. As mentioned in Section 6.1.1 [Initialization of the Base Generators], page 76, the statistical properties of the base generators are only guaranteed within sequences, not between sequences. For example, sequences generated from di?erent starting points may overlap if the initial values are not far enough apart. The potential for overlapping sequences is reduced if the period of the generator being used is large. Although there is no guarantee of the independence of the sequences, due to its extremely large period, using the Mersenne Twister with random starting values is unlikely to lead to problems, especially if the number of sequences required is small. This is the only way in which multiple sequences can be generated with the ACML using the Mersenne Twister as the base generator. If the statistical properties of di?erent sequences must be provable then one of the other methods should be adopted. 6.2.2 Using Di?erent Generators Independent sequences of variates can be generated using di?erent base generators for each sequence. For example, sequence 1 can be generated using the NAG basic generator, sequence 2 using the L’Ecuyer’s Combined Recursive generator, sequence 3 using the Mersenne Twister. The Wichmann-Hill generator implemented in the ACML is in fact a series of 273 independent generators. The particular sub-generator being used can be selected using the SUBID variable (see [DRANDINITIALIZE], page 78 for details). Therefore, in total, 277 independent streams can be generated with each using an independent generator (273 Wichmann-Hill generators, and 4 additional base generators). 6.2.3 Skip Ahead Independent sequences of variates can be generated from a single base generator through the use of block-splitting, or skipping-ahead. This method consists of splitting the sequence into k non-overlapping blocks, each of length n, where n is larger than the maximum number of variates required from any of the sequences. For example: x1, x2, · · · , xn, block 1 xn+1, xn+2, · · · , x2n, block 2 x2n+1, x2n+2, · · · , x3n, block 3 etc where x1, x2, · · · is the sequence produced by the generator of interest. Each of the k blocks provide an independent sequence. The block splitting algorithm therefore requires the sequence to be advanced a large number of places. Due to their form this can be done e?ciently for linear congruential generators and multiple congruential generators. The ACML provides block-splitting for the NAG Basic generator, the Wichmann-Hill generators and L’Ecuyer’s Combined Recursive generator.Chapter 6: Random Number Generators 92 DRANDSKIPAHEAD / SRANDSKIPAHEAD Advance a generator N places. (Note that SRANDSKIPAHEAD is the single precision version of DRANDSKIPAHEAD. The argument lists of both routines are identical except that any double precision arguments of DRANDSKIPAHEAD are replaced in SRANDSKIPAHEAD by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDSKIPAHEAD (N,STATE,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of places to skip ahead. Constraint: N= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDSKIPAHEAD STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: The STATE vector for a generator that has been advanced N places. Constraint: The STATE vector must be for either the NAG basic, WichmannHill or L’Ecuyer Combined Recursive base generators. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 93 Example:   C Generate 3 * 100 values from the Uniform distribution C Multiple streams generated using the Skip Ahead method INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,NSKIP INTEGER SEED(1),STATE1(LSTATE),STATE2(LSTATE),STATE3(LSTATE) INTEGER X1(N),X2(N),X3(N) DOUBLE PRECISION A,B C Set the seed SEED(1) = 1234 C Set the distributional parameters A = 0.0D0 B = 1.0D0 C Initialize the STATE1 vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE1,LSTATE,INFO) C Copy the STATE1 vector into other state vectors DO 20 I = 1,LSTATE STATE2(I) = STATE1(I) STATE3(I) = STATE1(I) 20 CONTINUE C Calculate how many places we want to skip, this C should be >> than the number of variates we C wish to generate from each stream NSKIP = N * N C Advance each stream, first does not need changing CALL DRANDSKIPAHEAD(NSKIP,STATE2,INFO) CALL DRANDSKIPAHEAD(2*NSKIP,STATE3,INFO) C Generate 3 sets of N variates from the Univariate distribution CALL DRANDUNIFORM(N,A,B,STATE1,X1,LDX,INFO) CALL DRANDUNIFORM(N,A,B,STATE2,X2,LDX,INFO) CALL DRANDUNIFORM(N,A,B,STATE3,X3,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) X1(I),X2(I),X3(I) 40 CONTINUE Chapter 6: Random Number Generators 94 6.2.4 Leap Frogging Independent sequences of variates can be generated from a single base generator through the use of leap-frogging. This method involves splitting the sequence from a single generator into k disjoint subsequences. For example: Subsequence 1 : x1, xk+1, x2k+1, · · · Subsequence 2 : x2, xk+2, x2k+2, · · · . . . Subsequence k : xk, x2k, x3k, · · · each subsequence is then provides an independent stream. The leap-frog algorithm therefore requires the generation of every kth variate of a sequence. Due to their form this can be done e?ciently for linear congruential generators and multiple congruential generators. The ACML provides leap-frogging for the NAG Basic generator, the Wichmann-Hill generators and L’Ecuyer’s Combined Recursive generator. As an illustrative example, a brief description of the algebra behind the implementation of the leap-frog algorithm (and block-splitting algorithm) for a linear congruential generator (LCG) will be given. A linear congruential generator has the form xi+1 = a1xi mod m1. The recursive nature of a LCG means that xi+v = a1xi+v-1 mod m1 = a1(a1xi+v-2 mod m1) mod m1 = a 2 1xi+v-2 mod m1 = a v 1xi mod m1 The sequence can be quickly advanced v places by multiplying the current state (xi) by a v 1 mod m1, hence allowing block-splitting. Leap-frogging is implemented by using a k 1 , where k is the number of streams required, in place of a1 in the standard LCG recursive formula. In a linear congruential generator the multiplier a1 is constructed so that the generator has good statistical properties in, for example, the spectral test. When using leap-frogging to construct multiple streams this multiplier is replaced with a k 1 , and there is no guarantee that this new multiplier will have suitable properties especially as the value of k depends on the number of streams required and so is likely to change depending on the application. This problem can be emphasised by the lattice structure of LCGs. Note that, due to rounding, a sequence generated using leap-frogging and a sequence constructed by taking every kth value from a set of variates generated without leap-frogging may di?er slightly. These di?erences should only a?ect the least signi?cant digit.Chapter 6: Random Number Generators 95 DRANDLEAPFROG / SRANDLEAPFROG Amend a generator so that it will generate every Kth value. (Note that SRANDLEAPFROG is the single precision version of DRANDLEAPFROG. The argument lists of both routines are identical except that any double precision arguments of DRANDLEAPFROG are replaced in SRANDLEAPFROG by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDLEAPFROG (N,K,STATE,INFO) [SUBROUTINE] INTEGER N [Input] On input: total number of streams being used. Constraint: N> 0. INTEGER K [Input] On input: number of the current stream Constraint: 0< K = N. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDLEAPFROG STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: The STATE vector for a generator that has been advanced K - 1 places and will return every Nth value. Constraint: The STATE array must be for either the NAG basic, WichmannHill or L’Ecuyer Combined Recursive base generators. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 96 Example:   C Generate 3 * 100 values from the Uniform distribution C Multiple streams generated using the Leap Frog method INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO INTEGER SEED(1),STATE1(LSTATE),STATE2(LSTATE),STATE3(LSTATE) INTEGER X1(N),X2(N),X3(N) DOUBLE PRECISION A,B C Set the seed SEED(1) = 1234 C Set the distributional parameters A = 0.0D0 B = 1.0D0 C Initialize the STATE1 vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE1,LSTATE,INFO) C Copy the STATE1 vector into other state vectors DO 20 I = 1,LSTATE STATE2(I) = STATE1(I) STATE3(I) = STATE1(I) 20 CONTINUE C Update each stream so they generate every 3rd value CALL DRANDLEAPFROG(3,1,STATE1,INFO) CALL DRANDLEAPFROG(3,2,STATE2,INFO) CALL DRANDLEAPFROG(3,3,STATE3,INFO) C Generate 3 sets of N variates from the Univariate distribution CALL DRANDUNIFORM(N,A,B,STATE1,X1,LDX,INFO) CALL DRANDUNIFORM(N,A,B,STATE2,X2,LDX,INFO) CALL DRANDUNIFORM(N,A,B,STATE3,X3,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) X1(I),X2(I),X3(I) 40 CONTINUE Chapter 6: Random Number Generators 97 6.3 Distribution Generators 6.3.1 Continuous Univariate Distributions DRANDBETA / SRANDBETA Generates a vector of random variates from a beta distribution with probability density function, f(X), where: f(X) = G(A + B) G(A)G(B) XA-1 (1 - X) B-1 if 0 = X = 1 and A, B > 0.0, otherwise f(X) = 0. (Note that SRANDBETA is the single precision version of DRANDBETA. The argument lists of both routines are identical except that any double precision arguments of DRANDBETA are replaced in SRANDBETA by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDBETA (N,A,B,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION A [Input] On input: ?rst parameter for the distribution. Constraint: A> 0. DOUBLE PRECISION B [Input] On input: second parameter for the distribution. Constraint: B> 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDBETA STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 98 Example:   C Generate 100 values from the Beta distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A,B DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Beta distribution CALL DRANDBETA(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 99 DRANDCAUCHY / SRANDCAUCHY Generates a vector of random variates from a Cauchy distribution with probability density function, f(X), where: f(X) = 1 pB(1 + ( X-A B ) 2 ) (Note that SRANDCAUCHY is the single precision version of DRANDCAUCHY. The argument lists of both routines are identical except that any double precision arguments of DRANDCAUCHY are replaced in SRANDCAUCHY by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDCAUCHY (N,A,B,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION A [Input] On input: median of the distribution. DOUBLE PRECISION B [Input] On input: semi-quartile range of the distribution. Constraint: B= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDCAUCHY STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 100 Example:   C Generate 100 values from the Cauchy distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A,B DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Cauchy distribution CALL DRANDCAUCHY(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 101 DRANDCHISQUARED / SRANDCHISQUARED Generates a vector of random variates from a ? 2 distribution with probability density function, f(X), where: f(X) = X ? 2 -1 e - X 2 2 ? 2 ( ? 2 - 1)! , if X > 0, otherwise f(X) = 0. Here ? is the degrees of freedom, DF. (Note that SRANDCHISQUARED is the single precision version of DRANDCHISQUARED. The argument lists of both routines are identical except that any double precision arguments of DRANDCHISQUARED are replaced in SRANDCHISQUARED by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDCHISQUARED (N,DF,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER DF [Input] On input: degrees of freedom of the distribution. Constraint: DF> 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDCHISQUARED STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 102 Example:   C Generate 100 values from the Chi-squared distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER DF DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) DF C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Chi-squared distribution CALL DRANDCHISQUARED(N,DF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 103 DRANDEXPONENTIAL / SRANDEXPONENTIAL Generates a vector of random variates from an exponential distribution with probability density function, f(X), where: f(X) = e - X A A if X > 0, otherwise f(X) = 0. (Note that SRANDEXPONENTIAL is the single precision version of DRANDEXPONENTIAL. The argument lists of both routines are identical except that any double precision arguments of DRANDEXPONENTIAL are replaced in SRANDEXPONENTIAL by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDEXPONENTIAL (N,A,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION A [Input] On input: exponential parameter. Constraint: A= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDEXPONENTIAL STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 104 Example:   C Generate 100 values from the Exponential distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Exponential distribution CALL DRANDEXPONENTIAL(N,A,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 105 DRANDF / SRANDF Generates a vector of random variates from an F distribution, also called the Fisher’s variance ratio distribution, with probability density function, f(X), where: f(X) = ( µ+?-2 2 )!X µ 2 -1 µ µ 2 ( µ 2 - 1)!( ? 2 - 1)!(1 + µX ? ) µ+? 2 ? µ 2 , if X > 0, otherwise f(X) = 0. Here µ is the ?rst degrees of freedom, (DF1) and ? is the second degrees of freedom, (DF2). (Note that SRANDF is the single precision version of DRANDF. The argument lists of both routines are identical except that any double precision arguments of DRANDF are replaced in SRANDF by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDF (N,DF1,DF2,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER DF1 [Input] On input: ?rst degrees of freedom. Constraint: DF1= 0. INTEGER DF2 [Input] On input: second degrees of freedom. Constraint: DF2= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDF STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 106 Example:   C Generate 100 values from the F distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER DF1,DF2 DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) DF1,DF2 C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the F distribution CALL DRANDF(N,DF1,DF2,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 107 DRANDGAMMA / SRANDGAMMA Generates a vector of random variates from a Gamma distribution with probability density function, f(X), where: f(X) = XA-1 e - X B BAG(A) , if X = 0 and A, B > 0.0, otherwise f(X) = 0. (Note that SRANDGAMMA is the single precision version of DRANDGAMMA. The argument lists of both routines are identical except that any double precision arguments of DRANDGAMMA are replaced in SRANDGAMMA by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDGAMMA (N,A,B,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION A [Input] On input: ?rst parameter of the distribution. Constraint: A> 0. DOUBLE PRECISION B [Input] On input: second parameter of the distribution. Constraint: B> 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDGAMMA STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 108 Example:   C Generate 100 values from the Gamma distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A,B DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Gamma distribution CALL DRANDGAMMA(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 109 DRANDGAUSSIAN / DRANDGAUSSIAN Generates a vector of random variates from a Gaussian distribution with probability density function, f(X), where: f(X) = e - (X-µ) 2 2s2 s v 2p . Here µ is the mean, (XMU) and s 2 the variance, (VAR) of the distribution. (Note that SRANDGAUSSIAN is the single precision version of DRANDGAUSSIAN. The argument lists of both routines are identical except that any double precision arguments of DRANDGAUSSIAN are replaced in SRANDGAUSSIAN by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDGAUSSIAN (N,XMU,VAR,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION XMU [Input] On input: mean of the distribution. DOUBLE PRECISION VAR [Input] On input: variance of the distribution. Constraint: VAR= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDGAUSSIAN STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 110 Example:   C Generate 100 values from the Gaussian distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION XMU,VAR DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) XMU,VAR C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Gaussian distribution CALL DRANDGAUSSIAN(N,XMU,VAR,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 111 DRANDLOGISTIC / SRANDLOGISTIC Generates a vector of random variates from a logistic distribution with probability density function, f(X), where: f(X) = e (X-A) B B(1 + e (X-A) B ) 2 . (Note that SRANDLOGISTIC is the single precision version of DRANDLOGISTIC. The argument lists of both routines are identical except that any double precision arguments of DRANDLOGISTIC are replaced in SRANDLOGISTIC by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDLOGISTIC (N,A,B,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION A [Input] On input: mean of the distribution. DOUBLE PRECISION B [Input] On input: spread of the distribution. B = v 3s/p where s is the standard deviation of the distribution. Constraint: B> 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDLOGISTIC STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 112 Example:   C Generate 100 values from the Logistic distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A,B DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Logistic distribution CALL DRANDLOGISTIC(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 113 DRANDLOGNORMAL / SRANDLOGNORMAL Generates a vector of random variates from a lognormal distribution with probability density function, f(X), where: f(X) = e - (log X-µ) 2 2s2 Xs v 2p , if X > 0, otherwise f(X) = 0. Here µ is the mean, (XMU) and s 2 the variance, (VAR) of the underlying Gaussian distribution. (Note that SRANDLOGNORMAL is the single precision version of DRANDLOGNORMAL. The argument lists of both routines are identical except that any double precision arguments of DRANDLOGNORMAL are replaced in SRANDLOGNORMAL by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDLOGNORMAL (N,XMU,VAR,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION XMU [Input] On input: mean of the underlying Gaussian distribution. DOUBLE PRECISION VAR [Input] On input: variance of the underlying Gaussian distribution. Constraint: VAR= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDLOGNORMAL STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 114 Example:   C Generate 100 values from the Lognormal distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION XMU,VAR DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) XMU,VAR C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Lognormal distribution CALL DRANDLOGNORMAL(N,XMU,VAR,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 115 DRANDSTUDENTST / SRANDSTUDENTST Generates a vector of random variates from a Students T distribution with probability density function, f(X), where: f(X) = (?-1) 2 ! ( ? 2 )! v p?(1 + X2 ? ) (?+1) 2 . Here ? is the degrees of freedom, DF. (Note that SRANDSTUDENTST is the single precision version of DRANDSTUDENTST. The argument lists of both routines are identical except that any double precision arguments of DRANDSTUDENTST are replaced in SRANDSTUDENTST by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDSTUDENTST (N,DF,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER DF [Input] On input: degrees of freedom. Constraint: DF> 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDSTUDENTST STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 116 Example:   C Generate 100 values from the Students T distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER DF DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) DF C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Students T distribution CALL DRANDSTUDENTST(N,DF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 117 DRANDTRIANGULAR / SRANDTRIANGULAR Generates a vector of random variates from a Triangular distribution with probability density function, f(X), where: f(X) = 2(X - XMIN) (XMAX - XMIN)(XMED - XMIN) , if XMIN < X = XMED, else f(X) = 2(XMAX - X) (XMAX - XMIN)(XMAX - XMED) , if XMED < X = XMAX, otherwise f(X) = 0. (Note that SRANDTRIANGULAR is the single precision version of DRANDTRIANGULAR. The argument lists of both routines are identical except that any double precision arguments of DRANDTRIANGULAR are replaced in SRANDTRIANGULAR by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDTRIANGULAR (N,XMIN,XMED,XMAX,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION XMIN [Input] On input: minimum value for the distribution. DOUBLE PRECISION XMED [Input] On input: median value for the distribution. Constraint: XMIN= XMED = XMAX. DOUBLE PRECISION XMAX [Input] On input: maximum value for the distribution. Constraint: XMAX= XMIN. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDTRIANGULAR STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 118 Example:   C Generate 100 values from the Triangular distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION XMIN,XMAX,XMED DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) XMIN,XMAX,XMED C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Triangular distribution CALL DRANDTRIANGULAR(N,XMIN,XMAX,XMED,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 119 DRANDUNIFORM / SRANDUNIFORM Generates a vector of random variates from a Uniform distribution with probability density function, f(X), where: f(X) = 1 B - A . (Note that SRANDUNIFORM is the single precision version of DRANDUNIFORM. The argument lists of both routines are identical except that any double precision arguments of DRANDUNIFORM are replaced in SRANDUNIFORM by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDUNIFORM (N,A,B,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION A [Input] On input: minimum value for the distribution. DOUBLE PRECISION B [Input] On input: maximum value for the distribution. Constraint: B= A. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDUNIFORM STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 120 Example:   C Generate 100 values from the Uniform distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A,B DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Uniform distribution CALL DRANDUNIFORM(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 121 DRANDVONMISES / SRANDVONMISES Generates a vector of random variates from a Von Mises distribution with probability density function, f(X), where: f(X) = e ? cos X 2pI0(?) where X is reduced modulo 2p so that it lies between ±p, and ? is the concentration parameter VK. (Note that SRANDVONMISES is the single precision version of DRANDVONMISES. The argument lists of both routines are identical except that any double precision arguments of DRANDVONMISES are replaced in SRANDVONMISES by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDVONMISES (N,VK,,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION VK [Input] On input: concentration parameter. Constraint: VK> 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDVONMISES STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 122 Example:   C Generate 100 values from the Von Mises distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION VK DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) VK C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Von Mises distribution CALL DRANDVONMISES(N,VK,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 123 DRANDWEIBULL / SRANDWEIBULL Generates a vector of random variates from a Weibull distribution with probability density function, f(X), where: f(X) = AXA-1 e - XA B B , if X > 0, otherwise f(X) = 0. (Note that SRANDWEIBULL is the single precision version of DRANDWEIBULL. The argument lists of both routines are identical except that any double precision arguments of DRANDWEIBULL are replaced in SRANDWEIBULL by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDWEIBULL (N,A,B,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION A [Input] On input: shape parameter for the distribution. Constraint: A> 0. DOUBLE PRECISION B [Input] On input: scale parameter for the distribution. Constraint: B> 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDWEIBULL STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 124 Example:   C Generate 100 values from the Weibull distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A,B DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Weibull distribution CALL DRANDWEIBULL(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 125 6.3.2 Discrete Univariate Distributions DRANDBINOMIAL / SRANDBINOMIAL Generates a vector of random variates from a Binomial distribution with probability, f(X), de?ned by: f(X) = M!P X (1 - P) (M-X) X!(M - 1)! , X = 0, 1, · · · , M (Note that SRANDBINOMIAL is the single precision version of DRANDBINOMIAL. The argument lists of both routines are identical except that any double precision arguments of DRANDBINOMIAL are replaced in SRANDBINOMIAL by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDBINOMIAL (N,M,P,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER M [Input] On input: number of trials. Constraint: M= 0. DOUBLE PRECISION P [Input] On input: probability of success. Constraint: 0= P < 1. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDBINOMIAL STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 126 Example:   C Generate 100 values from the Binomial distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER M DOUBLE PRECISION P INTEGER X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,P C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Binomial distribution CALL DRANDBINOMIAL(N,M,P,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 127 DRANDGEOMETRIC / SRANDGEOMETRIC Generates a vector of random variates from a Geometric distribution with probability, f(X), de?ned by: f(X) = P(1 - P) (X-1) , X = 1, 2, · · · (Note that SRANDGEOMETRIC is the single precision version of DRANDGEOMETRIC. The argument lists of both routines are identical except that any double precision arguments of DRANDGEOMETRIC are replaced in SRANDGEOMETRIC by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDGEOMETRIC (N,P,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION P [Input] On input: distribution parameter. Constraint: 0= P < 1. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDGEOMETRIC STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 128 Example:   C Generate 100 values from the Geometric distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION P INTEGER X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) P C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Geometric distribution CALL DRANDGEOMETRIC(N,P,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 129 DRANDHYPERGEOMETRIC / SRANDHYPERGEOMETRIC Generates a vector of random variates from a Hypergeometric distribution with probability, f(X), de?ned by: f(X) = s!m!(p - s)!(p - m)! X!(s - X)!(m - X)!(p - m - s + X)!p! , if X = max(0, m + s - p), · · · ,min(l, m), otherwise f(X) = 0. Here p is the size of the population, (NP), s is the size of the sample taken from the population, (NS) and m is the number of labeled, or speci?ed, items in the population, (M). (Note that SRANDHYPERGEOMETRIC is the single precision version of DRANDHYPERGEOMETRIC. The argument lists of both routines are identical except that any double precision arguments of DRANDHYPERGEOMETRIC are replaced in SRANDHYPERGEOMETRIC by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDHYPERGEOMETRIC (N,NP,NS,M,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER NP [Input] On input: size of population. Constraint: NP= 0. INTEGER NS [Input] On input: size of sample being taken from population. Constraint: 0= NS = NP. INTEGER M [Input] On input: number of speci?ed items in the population. Constraint: 0= M = NP. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDHYPERGEOMETRIC STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 130 Example:   C Generate 100 values from the Hypergeometric distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER NP,NS,M INTEGER X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) NP,NS,M C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Hypergeometric distribution CALL DRANDHYPERGEOMETRIC(N,NP,NS,M,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 131 DRANDNEGATIVEBINOMIAL / SRANDNEGATIVEBINOMIAL Generates a vector of random variates from a Negative Binomial distribution with probability f(X) de?ned by: f(X) = (M + X - 1)!P X (1 - P) M X!(M - 1)! , X = 0, 1, · · · (Note that SRANDNEGATIVEBINOMIAL is the single precision version of DRANDNEGATIVEBINOMIAL. The argument lists of both routines are identical except that any double precision arguments of DRANDNEGATIVEBINOMIAL are replaced in SRANDNEGATIVEBINOMIAL by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDNEGATIVEBINOMIAL (N,M,P,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER M [Input] On input: number of failures. Constraint: M= 0. DOUBLE PRECISION P [Input] On input: probability of success. Constraint: 0= P < 1. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDNEGATIVEBINOMIAL STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 132 Example:   C Generate 100 values from the Negative Binomial distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER M DOUBLE PRECISION P INTEGER X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,P C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Negative Binomial distribution CALL DRANDNEGATIVEBINOMIAL(N,M,P,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 133 DRANDPOISSON / SRANDPOISSON Generates a vector of random variates from a Poisson distribution with probability f(X) de?ned by: f(X) = ? X e -? X! , X = 0, 1, · · · , where ? is the mean of the distribution, LAMBDA. (Note that SRANDPOISSON is the single precision version of DRANDPOISSON. The argument lists of both routines are identical except that any double precision arguments of DRANDPOISSON are replaced in SRANDPOISSON by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDPOISSON (N,LAMBDA,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER M [Input] On input: number of failures. Constraint: M= 0. DOUBLE PRECISION LAMBDA [Input] On input: mean of the distribution. Constraint: LAMBDA= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDPOISSON STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 134 Example:   C Generate 100 values from the Poisson distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION LAMBDA INTEGER X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) LAMBDA C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Poisson distribution CALL DRANDPOISSON(N,LAMBDA,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 135 DRANDDISCRETEUNIFORM / SRANDDISCRETEUNIFORM Generates a vector of random variates from a Uniform distribution with probability f(X) de?ned by: f(X) = 1 (B - A) , X = A, A + 1, · · · , B (Note that SRANDDISCRETEUNIFORM is the single precision version of DRANDDISCRETEUNIFORM. The argument lists of both routines are identical except that any double precision arguments of DRANDDISCRETEUNIFORM are replaced in SRANDDISCRETEUNIFORM by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDDISCRETEUNIFORM (N,A,B,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER A [Input] On input: minimum for the distribution. INTEGER B [Input] On input: maximum for the distribution. Constraint: B= A. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDDISCRETEUNIFORM STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 136 Example:   C Generate 100 values from the Uniform distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER A,B INTEGER X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Uniform distribution CALL DRANDDISCRETEUNIFORM(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 137 DRANDGENERALDISCRETE / SRANDGENERALDISCRETE Takes a reference vector initialized via one of DRANDBINOMIALREFERENCE, DRANDGEOMETRIC REFERENCE, DRANDHYPERGEOMETRICREFERENCE, DRANDNEGATIVEBINOMIALREFERENCE, DRAND POISSONREFERENCE and generates a vector of random variates from it. (Note that SRANDGENERALDISCRETE is the single precision version of DRANDGENERALDISCRETE. The argument lists of both routines are identical except that any double precision arguments of DRANDGENERALDISCRETE are replaced in SRANDGENERALDISCRETE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDGENERALDISCRETE (N,REF,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION REF(*) [Input] On input: reference vector generated by one of the following: DRANDBINOMIALREFERENCE, DRANDGEOMETRICREFERENCE, DRANDHYPERGEOMETRICREFERENCE, DRANDNEGATIVEBINOMIALREFERENCE, DRANDPOISSONREFERENCE. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDGENERALDISCRETE STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 138 Example:   C Generate 100 values from the Binomial distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER M DOUBLE PRECISION P INTEGER X(N) INTEGER LREF DOUBLE PRECISION REF(1000) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,P C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDBINOMIALREFERENCE(M,P,REF,LREF,INFO) C Generate N variates from the Binomial distribution CALL DRANDGENERALDISCRETE(N,REF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 139 DRANDBINOMIALREFERENCE / SRANDBINOMIALREFERENCE Initializes a reference vector for use with DRANDGENERALDISCRETE. Reference vector is for a Binomial distribution with probability, f(X), de?ned by: f(X) = M!P X (1 - P) (M-X) X!(M - 1)! , X = 0, 1, · · · , M (Note that SRANDBINOMIALREFERENCE is the single precision version of DRANDBINOMIALREFERENCE. The argument lists of both routines are identical except that any double precision arguments of DRANDBINOMIALREFERENCE are replaced in SRANDBINOMIALREFERENCE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDBINOMIALREFERENCE (M,P,REF,LREF,INFO) [SUBROUTINE] INTEGER M [Input] On input: number of trials. Constraint: M= 0. DOUBLE PRECISION P [Input] On input: probability of success. Constraint: 0= P < 1. DOUBLE PRECISION REF(LREF) [Output] On output: if INFO returns with a value of 0 then REF contains reference information required to generate values from a Binomial distribution using DRANDGENERALDISCRETE. INTEGER LREF [Input/Output] On input: either the length of the reference vector REF, or -1. On output: if LREF = -1 on input, then LREF is set to the recommended length of the reference vector and the routine returns. Otherwise LREF is left unchanged. INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LREF has been set to the recommended length for the reference vector REF. If INFO = 0 then the reference vector, REF, has been successfully initialized.Chapter 6: Random Number Generators 140 Example:   C Generate 100 values from the Binomial distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER M DOUBLE PRECISION P INTEGER X(N) INTEGER LREF DOUBLE PRECISION REF(1000) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,P C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDBINOMIALREFERENCE(M,P,REF,LREF,INFO) C Generate N variates from the Binomial distribution CALL DRANDGENERALDISCRETE(N,REF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 141 DRANDGEOMETRICREFERENCE / SRANDGEOMETRICREFERENCE Initializes a reference vector for use with DRANDGENERALDISCRETE. Reference vector is for a Geometric distribution with probability, f(X), de?ned by: f(X) = P(1 - P) (X-1) , X = 1, 2, · · · (Note that SRANDGEOMETRICREFERENCE is the single precision version of DRANDGEOMETRICREFERENCE. The argument lists of both routines are identical except that any double precision arguments of DRANDGEOMETRICREFERENCE are replaced in SRANDGEOMETRICREFERENCE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDGEOMETRICREFERENCE (P,REF,LREF,INFO) [SUBROUTINE] DOUBLE PRECISION P [Input] On input: distribution parameter. Constraint: 0= P < 1. DOUBLE PRECISION REF(LREF) [Output] On output: if INFO returns with a value of 0 then REF contains reference information required to generate values from a Geometric distribution using DRANDGENERALDISCRETE. INTEGER LREF [Input/Output] On input: either the length of the reference vector REF, or -1. On output: if LREF = -1 on input, then LREF is set to the recommended length of the reference vector and the routine returns. Otherwise LREF is left unchanged. INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LREF has been set to the recommended length for the reference vector REF. If INFO = 0 then the reference vector, REF, has been successfully initialized.Chapter 6: Random Number Generators 142 Example:   C Generate 100 values from the Geometric distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION P INTEGER X(N) INTEGER LREF DOUBLE PRECISION REF(1000) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) P C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDGEOMETRICREFERENCE(P,REF,LREF,INFO) C Generate N variates from the Geometric distribution CALL DRANDGENERALDISCRETE(N,REF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 143 DRANDHYPERGEOMETRICREFERENCE / SRANDHYPERGEOMETRICREFERENCE Initializes a reference vector for use with DRANDGENERALDISCRETE. Reference vector is for a Hypergeometric distribution with probability, f(X), de?ned by: f(X) = s!m!(p - s)!(p - m)! X!(s - X)!(m - X)!(p - m - s + X)!p! , if X = max(0, m + s - p), · · · ,min(l, m), otherwise f(X) = 0. Here p is the size of the population, (NP), s is the size of the sample taken from the population, (NS) and m is the number of labeled, or speci?ed, items in the population, (M). (Note that SRANDHYPERGEOMETRICREFERENCE is the single precision version of DRANDHYPERGEOMETRICREFERENCE. The argument lists of both routines are identical except that any double precision arguments of DRANDHYPERGEOMETRICREFERENCE are replaced in SRANDHYPERGEOMETRICREFERENCE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDHYPERGEOMETRICREFERENCE (NP,NS,M,REF,LREF,INFO) [SUBROUTINE] INTEGER NP [Input] On input: size of population. Constraint: NP= 0. INTEGER NS [Input] On input: size of sample being taken from population. Constraint: 0= NS = NP. INTEGER M [Input] On input: number of speci?ed items in the population. Constraint: 0= M = NP. DOUBLE PRECISION REF(LREF) [Output] On output: if INFO returns with a value of 0 then REF contains reference information required to generate values from a Hypergeometric distribution using DRANDGENERALDISCRETE. INTEGER LREF [Input/Output] On input: either the length of the reference vector REF, or -1. On output: if LREF = -1 on input, then LREF is set to the recommended length of the reference vector and the routine returns. Otherwise LREF is left unchanged. INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LREF has been set to the recommended length for the reference vector REF. If INFO = 0 then the reference vector, REF, has been successfully initialized.Chapter 6: Random Number Generators 144 Example:   C Generate 100 values from the Hypergeometric distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER NP, NS,M INTEGER X(N) INTEGER LREF DOUBLE PRECISION REF(1000) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) NP, NS,M C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDHYPERGEOMETRICREFERENCE(NP, NS,M,REF,LREF,INFO) C Generate N variates from the Hypergeometric distribution CALL DRANDGENERALDISCRETE(N,REF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 145 DRANDNEGATIVEBINOMIALREFERENCE / SRANDNEGATIVEBINOMIALREFERENCE Initializes a reference vector for use with DRANDGENERALDISCRETE. Reference vector is for a Negative Binomial distribution with probability f(X) de?ned by: f(X) = (M + X - 1)!P X (1 - P) M X!(M - 1)! , X = 0, 1, · · · (Note that SRANDNEGATIVEBINOMIALREFERENCE is the single precision version of DRANDNEGATIVEBINOMIALREFERENCE. The argument lists of both routines are identical except that any double precision arguments of DRANDNEGATIVEBINOMIALREFERENCE are replaced in SRANDNEGATIVEBINOMIALREFERENCE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDNEGATIVEBINOMIALREFERENCE (M,P,REF,LREF,INFO) [SUBROUTINE] INTEGER M [Input] On input: number of failures. Constraint: M= 0. DOUBLE PRECISION P [Input] On input: probability of success. Constraint: 0= P < 1. DOUBLE PRECISION REF(LREF) [Output] On output: if INFO returns with a value of 0 then REF contains reference information required to generate values from a Negative Binomial distribution using DRANDGENERALDISCRETE. INTEGER LREF [Input/Output] On input: either the length of the reference vector REF, or -1. On output: if LREF = -1 on input, then LREF is set to the recommended length of the reference vector and the routine returns. Otherwise LREF is left unchanged. INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LREF has been set to the recommended length for the reference vector REF. If INFO = 0 then the reference vector, REF, has been successfully initialized.Chapter 6: Random Number Generators 146 Example:   C Generate 100 values from the Negative Binomial distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER M DOUBLE PRECISION P INTEGER X(N) INTEGER LREF DOUBLE PRECISION REF(1000) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,P C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDNEGATIVEBINOMIALREFERENCE(M,P,REF,LREF,INFO) C Generate N variates from the Negative Binomial distribution CALL DRANDGENERALDISCRETE(N,REF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 147 DRANDPOISSONREFERENCE / SRANDPOISSONREFERENCE Initializes a reference vector for use with DRANDGENERALDISCRETE. Reference vector is for a Poisson distribution with probability f(X) de?ned by: f(X) = ? X e -? X! , X = 0, 1, · · · , where ? is the mean of the distribution, LAMBDA. (Note that SRANDPOISSONREFERENCE is the single precision version of DRANDPOISSONREFERENCE. The argument lists of both routines are identical except that any double precision arguments of DRANDPOISSONREFERENCE are replaced in SRANDPOISSONREFERENCE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDPOISSONREFERENCE (LAMBDA,REF,LREF,INFO) [SUBROUTINE] INTEGER M [Input] On input: number of failures. Constraint: M= 0. DOUBLE PRECISION LAMBDA [Input] On input: mean of the distribution. Constraint: LAMBDA= 0. DOUBLE PRECISION REF(LREF) [Output] On output: if INFO returns with a value of 0 then REF contains reference information required to generate values from a Poisson distribution using DRANDGENERALDISCRETE. INTEGER LREF [Input/Output] On input: either the length of the reference vector REF, or -1. On output: if LREF = -1 on input, then LREF is set to the recommended length of the reference vector and the routine returns. Otherwise LREF is left unchanged. INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LREF has been set to the recommended length for the reference vector REF. If INFO = 0 then the reference vector, REF, has been successfully initialized.Chapter 6: Random Number Generators 148 Example:   C Generate 100 values from the Poisson distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION LAMBDA INTEGER X(N) INTEGER LREF DOUBLE PRECISION REF(1000) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) LAMBDA C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDPOISSONREFERENCE(LAMBDA,REF,LREF,INFO) C Generate N variates from the Poisson distribution CALL DRANDGENERALDISCRETE(N,REF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 149 6.3.3 Continuous Multivariate Distributions DRANDMULTINORMAL / SRANDMULTINORMAL Generates an array of random variates from a Multivariate Normal distribution with probability density function, f(X), where: f(X) = s |C-1 | (2p)M e -(X-µ) T C -1 (X-µ) , where µ is the vector of means, XMU. (Note that SRANDMULTINORMAL is the single precision version of DRANDMULTINORMAL. The argument lists of both routines are identical except that any double precision arguments of DRANDMULTINORMAL are replaced in SRANDMULTINORMAL by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDMULTINORMAL (N,M,XMU,C,LDC,STATE,X,LDX,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER M [Input] On input: number of dimensions for the distribution. Constraint: M= 1. DOUBLE PRECISION XMU(M) [Input] On input: vector of means for the distribution. DOUBLE PRECISION C(LDC,M) [Input] On input: variance / covariance matrix for the distribution. INTEGER LDC [Input] On input: leading dimension of C in the calling routine. Constraint: LDC= M. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDMULTINORMAL STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(LDX,M) [Output] On output: matrix of variates from the speci?ed distribution. INTEGER LDX [Input] On input: leading dimension of X in the calling routine. Constraint: LDX= N. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 150 Example:   C Generate 100 values from the C Multivariate Normal distribution INTEGER LSTATE,N, MM PARAMETER (LSTATE=16,N=100,MM=10) INTEGER I,J,INFO,SEED(1),STATE(LSTATE) INTEGER LDC,LDX,M DOUBLE PRECISION X(N,MM),XMU(MM),C(MM,MM) C Set array sizes LDC = MM LDX = N C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M READ(5,*) (XMU(I),I=1,M) DO 20 I = 1,M READ(5,*) (C(I,J),J=1,M) 20 CONTINUE C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the C Multivariate Normal distribution CALL DRANDMULTINORMAL(N,M,XMU,C,LDC,STATE,X,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) (X(I,J),J=1,M) 40 CONTINUE Chapter 6: Random Number Generators 151 DRANDMULTISTUDENTST / SRANDMULTISTUDENTST Generates an array of random variates from a Multivariate Students T distribution with probability density function, f(X), where: f(X) = G  (?+M) 2  (p?) m 2 G( ? 2 )|C| 1 2 1 + (X - µ) T C -1 (X - µ) ? !- (?+M) 2 , where µ is the vector of means, XMU and ? is the degrees of freedom, DF. (Note that SRANDMULTISTUDENTST is the single precision version of DRANDMULTISTUDENTST. The argument lists of both routines are identical except that any double precision arguments of DRANDMULTISTUDENTST are replaced in SRANDMULTISTUDENTST by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDMULTISTUDENTST (N,M,DF,XMU,C,LDC,STATE,X,LDX,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER M [Input] On input: number of dimensions for the distribution. Constraint: M= 1. INTEGER DF [Input] On input: degrees of freedom. Constraint: DF> 2. DOUBLE PRECISION XMU(M) [Input] On input: vector of means for the distribution. DOUBLE PRECISION C(LDC,M) [Input] On input: matrix de?ning the variance / covariance for the distribution. The variance / covariance matrix is given by ? ?-2C, where ? are the degrees of freedom, DF. INTEGER LDC [Input] On input: leading dimension of C in the calling routine. Constraint: LDC= M. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDMULTISTUDENTST STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(LDX,M) [Output] On output: matrix of variates from the speci?ed distribution.Chapter 6: Random Number Generators 152 INTEGER LDX [Input] On input: leading dimension of X in the calling routine. Constraint: LDX= N. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Generate 100 values from the C Multivariate Students T distribution INTEGER LSTATE,N, MM PARAMETER (LSTATE=16,N=100,MM=10) INTEGER I,J,INFO,SEED(1),STATE(LSTATE) INTEGER LDC,LDX,M,DF DOUBLE PRECISION X(N,MM),XMU(MM),C(MM,MM) C Set array sizes LDC = MM LDX = N C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,DF READ(5,*) (XMU(I),I=1,M) DO 20 I = 1,M READ(5,*) (C(I,J),J=1,M) 20 CONTINUE C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the C Multivariate Students T distribution CALL DRANDMULTISTUDENTST(N,M,DF,XMU,C,LDC,STATE,X,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) (X(I,J),J=1,M) 40 CONTINUE Chapter 6: Random Number Generators 153 DRANDMULTINORMALR / SRANDMULTINORMALR Generates an array of random variates from a Multivariate Normal distribution using a reference vector initialized by DRANDMULTINORMALREFERENCE. (Note that SRANDMULTINORMALR is the single precision version of DRANDMULTINORMALR. The argument lists of both routines are identical except that any double precision arguments of DRANDMULTINORMALR are replaced in SRANDMULTINORMALR by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDMULTINORMALR (N,REF,STATE,X,LDX,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION REF(*) [Input] On input: a reference vector generated by DRANDMULTINORMALREFERENCE. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDMULTINORMALR STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(LDX,M) [Output] On output: matrix of variates from the speci?ed distribution. INTEGER LDX [Input] On input: leading dimension of X in the calling routine. Constraint: LDX= N. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 154 Example:   C Generate 100 values from the C Multivariate Normal distribution INTEGER LSTATE,N, MM PARAMETER (LSTATE=16,N=100,MM=10) INTEGER I,J,INFO,SEED(1),STATE(LSTATE) INTEGER LDC,LDX,M DOUBLE PRECISION X(N,MM),XMU(MM),C(MM,MM) INTEGER LREF DOUBLE PRECISION REF(1000) C Set array sizes LDC = MM LDX = N C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M READ(5,*) (XMU(I),I=1,M) DO 20 I = 1,M READ(5,*) (C(I,J),J=1,M) 20 CONTINUE C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDMULTINORMALREFERENCE(M,XMU,C,LDC,REF,LREF,INFO) C Generate N variates from the C Multivariate Normal distribution CALL DRANDMULTINORMALR(N,REF,STATE,X,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) (X(I,J),J=1,M) 40 CONTINUE Chapter 6: Random Number Generators 155 DRANDMULTISTUDENTSTR / SRANDMULTISTUDENTSTR Generates an array of random variates from a Multivariate Students T distribution using a reference vector initialized by DRANDMULTISTUDENTSTREFERENCE. (Note that SRANDMULTISTUDENTSTR is the single precision version of DRANDMULTISTUDENTSTR. The argument lists of both routines are identical except that any double precision arguments of DRANDMULTISTUDENTSTR are replaced in SRANDMULTISTUDENTSTR by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDMULTISTUDENTSTR (N,REF,STATE,X,LDX,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION REF(*) [Input] On input: a reference vector generated by DRANDMULTISTUDENTSTREFERENCE. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDMULTISTUDENTSTR STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(LDX,M) [Output] On output: matrix of variates from the speci?ed distribution. INTEGER LDX [Input] On input: leading dimension of X in the calling routine. Constraint: LDX= N. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 156 Example:   C Generate 100 values from the C Multivariate Students T distribution INTEGER LSTATE,N, MM PARAMETER (LSTATE=16,N=100,MM=10) INTEGER I,J,INFO,SEED(1),STATE(LSTATE) INTEGER LDC,LDX,M,DF DOUBLE PRECISION X(N,MM),XMU(MM),C(MM,MM) INTEGER LREF DOUBLE PRECISION REF(1000) C Set array sizes LDC = MM LDX = N C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,DF READ(5,*) (XMU(I),I=1,M) DO 20 I = 1,M READ(5,*) (C(I,J),J=1,M) 20 CONTINUE C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDMULTISTUDENTSTREFERENCE(M,DF,XMU,C,LDC,REF,LREF,INFO) C Generate N variates from the C Multivariate Students T distribution CALL DRANDMULTISTUDENTSTR(N,REF,STATE,X,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) (X(I,J),J=1,M) 40 CONTINUE Chapter 6: Random Number Generators 157 DRANDMULTINORMALREFERENCE / SRANDMULTINORMALREFERENCE Initializes a reference vector for use with DRANDMULTINORMALR. Reference vector is for a Multivariate Normal distribution with probability density function, f(X), where: f(X) = s |C-1 | (2p)M e -(X-µ) T C -1 (X-µ) , where µ is the vector of means, XMU. (Note that SRANDMULTINORMALREFERENCE is the single precision version of DRANDMULTINORMALREFERENCE. The argument lists of both routines are identical except that any double precision arguments of DRANDMULTINORMALREFERENCE are replaced in SRANDMULTINORMALREFERENCE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDMULTINORMALREFERENCE (M,XMU,C,LDC,REF,LREF,INFO) [SUBROUTINE] INTEGER M [Input] On input: number of dimensions for the distribution. Constraint: M= 1. DOUBLE PRECISION XMU(M) [Input] On input: vector of means for the distribution. DOUBLE PRECISION C(LDC,M) [Input] On input: variance / covariance matrix for the distribution. INTEGER LDC [Input] On input: leading dimension of C in the calling routine. Constraint: LDC= M. DOUBLE PRECISION REF(LREF) [Output] On output: if INFO returns with a value of 0 then REF contains reference information required to generate values from a Multivariate Normal distribution using DRANDMULTINORMALR. INTEGER LREF [Input/Output] On input: either the length of the reference vector REF, or -1. On output: if LREF = -1 on input, then LREF is set to the recommended length of the reference vector and the routine returns. Otherwise LREF is left unchanged. INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LREF has been set to the recommended length for the reference vector REF. If INFO = 0 then the reference vector, REF, has been successfully initialized.Chapter 6: Random Number Generators 158 Example:   C Generate 100 values from the C Multivariate Normal distribution INTEGER LSTATE,N, MM PARAMETER (LSTATE=16,N=100,MM=10) INTEGER I,J,INFO,SEED(1),STATE(LSTATE) INTEGER LDC,LDX,M DOUBLE PRECISION X(N,MM),XMU(MM),C(MM,MM) INTEGER LREF DOUBLE PRECISION REF(1000) C Set array sizes LDC = MM LDX = N C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M READ(5,*) (XMU(I),I=1,M) DO 20 I = 1,M READ(5,*) (C(I,J),J=1,M) 20 CONTINUE C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDMULTINORMALREFERENCE(M,XMU,C,LDC,REF,LREF,INFO) C Generate N variates from the C Multivariate Normal distribution CALL DRANDMULTINORMALR(N,REF,STATE,X,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) (X(I,J),J=1,M) 40 CONTINUE Chapter 6: Random Number Generators 159 DRANDMULTISTUDENTSTREFERENCE / SRANDMULTISTUDENTSTREFERENCE Initializes a reference vector for use with DRANDMULTISTUDENTSTR. Reference vector is for a Multivariate Students T distribution with probability density function, f(X), where: f(X) = G  (?+M) 2  (p?) m 2 G( ? 2 )|C| 1 2 1 + (X - µ) T C -1 (X - µ) ? !- (?+M) 2 , where µ is the vector of means, XMU and ? is the degrees of freedom, DF. (Note that SRANDMULTISTUDENTSTREFERENCE is the single precision version of DRANDMULTISTUDENTSTREFERENCE. The argument lists of both routines are identical except that any double precision arguments of DRANDMULTISTUDENTSTREFERENCE are replaced in SRANDMULTISTUDENTSTREFERENCE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDMULTISTUDENTSREFERENCE [SUBROUTINE] (M,DF,XMU,C,LDC,REF,LREF,INFO) INTEGER M [Input] On input: number of dimensions for the distribution. Constraint: M= 1. INTEGER DF [Input] On input: degrees of freedom. Constraint: DF> 2. DOUBLE PRECISION XMU(M) [Input] On input: vector of means for the distribution. DOUBLE PRECISION C(LDC,M) [Input] On input: matrix de?ning the variance / covariance for the distribution. The variance / covariance matrix is given by ? ?-2C, where ? are the degrees of freedom, DF. INTEGER LDC [Input] On input: leading dimension of C in the calling routine. Constraint: LDC= M. DOUBLE PRECISION REF(LREF) [Output] On output: if INFO returns with a value of 0 then REF contains reference information required to generate values from a Multivariate Students T distribution using DRANDMULTISTUDENTSTR. INTEGER LREF [Input/Output] On input: either the length of the reference vector REF, or -1. On output: if LREF = -1 on input, then LREF is set to the recommended length of the reference vector and the routine returns. Otherwise LREF is left unchanged.Chapter 6: Random Number Generators 160 INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LREF has been set to the recommended length for the reference vector REF. If INFO = 0 then the reference vector, REF, has been successfully initialized. Example:   C Generate 100 values from the C Multivariate Students T distribution INTEGER LSTATE,N, MM PARAMETER (LSTATE=16,N=100,MM=10) INTEGER I,J,INFO,SEED(1),STATE(LSTATE) INTEGER LDC,LDX,M,DF DOUBLE PRECISION X(N,MM),XMU(MM),C(MM,MM) INTEGER LREF DOUBLE PRECISION REF(1000) C Set array sizes LDC = MM LDX = N C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,DF READ(5,*) (XMU(I),I=1,M) DO 20 I = 1,M READ(5,*) (C(I,J),J=1,M) 20 CONTINUE C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDMULTISTUDENTSTREFERENCE(M,DF,XMU,C,LDC,REF,LREF,INFO) C Generate N variates from the C Multivariate Students T distribution CALL DRANDMULTISTUDENTSTR(N,REF,STATE,X,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) (X(I,J),J=1,M) 40 CONTINUE Chapter 6: Random Number Generators 161 6.3.4 Discrete Multivariate Distributions DRANDMULTINOMIAL / SRANDMULTINOMIAL Generates a matrix of random variates from a Multinomial distribution with probability, f(X), de?ned by: f(X) = M! QK i=1 Xi ! YK i=1 p Xi i , where X = {X1, X2, · · · , XK}, P = {P1, P2, · · · , PK}, PK i=1 Xi = 1 and PK i=1 Pi = 1. (Note that SRANDMULTINOMIAL is the single precision version of DRANDMULTINOMIAL. The argument lists of both routines are identical except that any double precision arguments of DRANDMULTINOMIAL are replaced in SRANDMULTINOMIAL by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDMULTINOMIAL (N,M,P,K,STATE,X,LDX,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER M [Input] On input: number of trials. Constraint: M= 0. DOUBLE PRECISION P(K) [Input] On input: vector of probabilities for each of the K possible outcomes. Constraint: 0 = Pi = 1, i = 1, 2, · · · , K, PK i=1 Pi = 1. INTEGER K [Input] On input: number of possible outcomes. Constraint: K= 2. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDBINOMIAL STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 76 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(LDX,K) [Output] On output: matrix of variates from the speci?ed distribution. INTEGER LDX [Input] On input: leading dimension of X in the calling routine. Constraint: LDX= N. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 162 Example:   C Generate 100 values from the Multinomial distribution INTEGER LSTATE,N, MK PARAMETER (LSTATE=16,N=100,MK=10) INTEGER I,J,INFO,SEED(1),STATE(LSTATE) INTEGER LDC,LDX,K,M INTEGER X(N,MK) DOUBLE PRECISION P(MK) C Set array sizes LDX = N C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) K READ(5,*) (P(I),I=1,K) C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Multinomial distribution CALL DRANDMULTINOMIAL(N,M,P,K,STATE,X,LDX,INFO) C Print the results DO 20 I = 1,N WRITE(6,*) (X(I,J),J=1,K) 20 CONTINUE Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 163 7 ACML MV: Fast Math and Fast Vector Math Library 7.1 Introduction to ACML MV ACML MV is a library which contains fast and/or vectorized versions of some familiar math library routines such as sin, cos and exp. The routines take advantage of the AMD64 architecture for performance, and so are currently only available with 64-bit versions of ACML. The routines in the library are very accurate over the range of acceptable input arguments. Some of the performance is gained by sacri?cing error handling or the acceptance of certain arguments. It is therefore the responsibility of the caller of these routines to ensure that their arguments are suitable. Furthermore, some of the routines are not callable from highlevel languages at all, but must be called via assembly language; see the documentation of individual routines for details. Hence, these routines are intended to be utilized by knowledgeable users only. 7.1.1 Terminology The individual documentation for a routine states what outputs will be returned for special arguments, and also gives an indication of performance of the routine. In general, special case arguments for any routine will cause a return value in accordance with the C99 language standard [13]. Special case arguments include NaNs and in?nities, as de?ned by the IEEE arithmetic standard [14]. In these documents, NaN means Not a Number, QNaN means Quiet NaN, and SNaN means Signalling NaN. A denormal number is a number which is very tiny (close to the machine arithmetic under- ?ow threshold) and is stored to less precision than a normal number. Due to their special nature, operations on such numbers are often very slow. While such numbers might not necessarily be regarded as special case arguments, for the sake of performance some of the ACML MV routines have been designed not to handle them. This has been noted in the documentation for each ACML MV routine. Accuracy of a routine is quoted in ulps, where ulp stands for Unit in the Last Place. Since ?oating-point numbers on a computer are limited precision approximations of mathematical numbers, not all real numbers can be represented by machine numbers, and the machine number must in general be rounded to available precision. An ulp is the distance between the two machine numbers that bracket a real number. In this document, the ulp is used as a measure of the error in a returned result when compared with the mathematically exact expected result. Because of the ?nite nature of machine arithmetic, a routine can never in general achieve accuracy of better than 0.5 ulps, and an accuracy of less than 1 ulp is good.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 164 7.1.2 Weak Aliases Some of the functions in ACML MV include a weak alias to an equivalent function in libm. For example, the fastcos function includes a weak alias to cos. If ACML MV is included in the link order before libm, then all calls to the aliased libm function name (e.g. cos) will use the equivalent ACML MV routine (e.g. fastcos). If ACML MV is included in the link order after libm, then all calls to libm functions will use the libm versions. ACML MV routines can always be accessed using their ACML MV names (e.g. fastcos), regardless of link order. 7.1.3 De?ned Types The following types are used to describe the functions contained in this chapter: m128d a pair of double precision values; m128 four single precision values.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 165 7.2 Fast Basic Math Functions This section documents the interfaces to a set of basic mathematical functions. fastcos: fast double precision Cosine double fastcos (double x) Weak alias: cos C Prototype: double fastcos (double x); Inputs: double x - the double precision input value. Outputs: Cosine of x. Fortran Function Interface: DOUBLE PRECISION FASTCOS(X) Inputs: DOUBLE PRECISION X - the double precision input value. Return Value: Cosine of X. Notes: fastcos computes the Cosine function of its argument x. This is a relaxed version of cos, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 2 ulp over most of the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 166 fastcosf: fast single precision Cosine float fastcosf (?oat x) Weak alias: cosf C Prototype: ?oat fastcosf (?oat x); Inputs: ?oat x - the single precision input value. Outputs: Single precision Cosine of x. Fortran Function Interface: REAL FASTCOSF(X) Inputs: REAL X - the single precision input value. Return Value: Cosine of X. Notes: fastcosf computes the single precision Cosine function of its argument x. This is a relaxed version of cosf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over most of the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN fastexp: fast double precision exponential function double fastexp (double x) Weak alias: exp C Prototype: double fastexp (double x); Inputs: double x - the double precision input value. Outputs: e raised to the power x (exponential of x). Fortran Function Interface:Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 167 DOUBLE PRECISION FASTEXP(X) Inputs: DOUBLE PRECISION X - the double precision input value. Return Value: e raised to the power X (exponential of X). Notes: fastexp computes the double precision exponential function of the input argument x. This is a relaxed version of exp, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -708.5 0 > 709.8 +8Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 168 fastexpf: fast single precision exponential function float fastexpf (?oat x) Weak alias: expf C Prototype: ?oat fastexpf (?oat x); Inputs: ?oat x - the single precision input value. Outputs: e raised to the power x (exponential of x). Fortran Function Interface: REAL FASTEXPF(X) Inputs: REAL X - the single precision input value. Return Value: e raised to the power X (exponential of X). Notes: fastexpf computes the single precision exponential function of the input argument x. This is a relaxed version of expf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -87.5 0 > 88 +8Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 169 fastlog: fast double precision natural logarithm function double fastlog (double x) Weak alias: log C Prototype: double fastlog (double x); Inputs: double x - the double precision input value. Outputs: The natural logarithm (base e) of x. Fortran Function Interface: DOUBLE PRECISION FASTLOG(X) Inputs: DOUBLE PRECISION X - the double precision input value. Return Value: The natural logarithm (base e) of X. Notes: fastlog computes the double precision natural logarithm of its argument x. This is a relaxed version of log, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 170 fastlogf: fast single precision natural logarithm function float fastlogf (?oat x) Weak alias: logf C Prototype: ?oat fastlogf (?oat x); Inputs: ?oat x - the single precision input value. Outputs: The natural logarithm (base e) of x. Fortran Function Interface: REAL FASTLOGF(X) Inputs: REAL X - the single precision input value. Return Value: The natural logarithm (base e) of X. Notes: fastlogf computes the single precision natural logarithm of its argument x. This is a relaxed version of logf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 171 fastlog10: fast double precision base-10 logarithm function double fastlog10 (double x) Weak alias: log10 C Prototype: double fastlog10 (double x); Inputs: double x - the double precision input value. Outputs: The base-10 logarithm of x. Fortran Function Interface: DOUBLE PRECISION FASTLOG10(X) Inputs: DOUBLE PRECISION X - the double precision input value. Return Value: The base-10 logarithm of X. Notes: fastlog10 computes the double precision base-10 logarithm of its argument x. This is a relaxed version of log10, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 172 fastlog10f: fast single precision base-10 logarithm function float fastlog10f (?oat x) Weak alias: log10f C Prototype: ?oat fastlog10f (?oat x); Inputs: ?oat x - the single precision input value. Outputs: The base-10 logarithm of x. Fortran Function Interface: REAL FASTLOG10F(X) Inputs: REAL X - the single precision input value. Return Value: The base-10 logarithm of X. Notes: fastlog10f computes the single precision base-10 logarithm of its argument x. This is a relaxed version of log10f, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 173 fastlog2: fast double precision base-2 logarithm function double fastlog2 (double x) Weak alias: log2 C Prototype: double fastlog2 (double x); Inputs: double x - the double precision input value. Outputs: The base-2 logarithm of x. Fortran Function Interface: DOUBLE PRECISION FASTLOG2(X) Inputs: DOUBLE PRECISION X - the double precision input value. Return Value: The base-2 logarithm of X. Notes: fastlog2 computes the double precision base-2 logarithm of its argument x. This is a relaxed version of log2, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 174 fastlog2f: fast single precision base-2 logarithm function float fastlog2f (?oat x) Weak alias: log2f C Prototype: ?oat fastlog2f (?oat x); Inputs: ?oat x - the single precision input value. Outputs: The base-2 logarithm of x. Fortran Function Interface: REAL FASTLOG2F(X) Inputs: REAL X - the single precision input value. Return Value: The base-2 logarithm of X. Notes: fastlog2f computes the single precision base-2 logarithm of its argument x. This is a relaxed version of log2f, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 175 fastpow: fast double precision power function double fastpow (double x, double y) Weak alias: pow C Prototype: double fastpow (double x, double y); Inputs: double x - the double precision base input value. double y - the double precision exponent input value. Outputs: x raised to the power y. Fortran Function Interface: DOUBLE PRECISION FASTPOW(X,Y) Inputs: DOUBLE PRECISION X - the base value. DOUBLE PRECISION Y - the exponent value. Return Value: X raised to the power Y. Notes: fastpow computes the x raised to the power y in double precision. This is a relaxed version of pow, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs will produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input x Input y Output ±0 y < 0, odd integer ±8 ±0 y < 0, not odd integer +8 ±0 y > 0, odd integer ±0 ±0 y > 0, not odd integer +0 -1 +8 1 +1 y (incl. NaN) 1 x (incl. Nan) ±0 1 x < 0 y, not integer QNaN |x|<1 -8 +8 |x|>1 -8 +0 |x|<1 +8 +0 |x|>1 +8 +8 -8 y < 0, odd integer -0 -8 y < 0, not odd integer +0 -8 y > 0, odd integer -8 -8 y > 0, not odd integer +8Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 176 +8 y < 0, +0 +8 y > 0, +8 NaN y nonzero, NaN x<>1 NaN, NaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 177 fastpowf: fast single precision power function float fastpowf (?oat x, ?oat y) Weak alias: powf C Prototype: ?oat fastpowf (?oat x, ?oat y); Inputs: ?oat x - the single precision base input value. ?oat y - the single precision exponent input value. Outputs: x raised to the power y. Fortran Function Interface: REAL FASTPOWF(X,Y) Inputs: REAL X - the single precision base value. REAL Y - the single precision exponent value. Return Value: X raised to the power Y. Notes: fastpowf computes the x raised to the power y in single precision. This is a relaxed version of powf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs will produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 0.5 ulp over the valid input range. Special case return values: Input x Input y Output ±0 y < 0, odd integer ±8 ±0 y < 0, not odd integer +8 ±0 y > 0, odd integer ±0 ±0 y > 0, not odd integer +0 -1 +8 1 +1 y (incl. NaN) 1 x (incl. Nan) ±0 1 x < 0 y, not integer QNaN |x|<1 -8 +8 |x|>1 -8 +0 |x|<1 +8 +0 |x|>1 +8 +8 -8 y < 0, odd integer -0 -8 y < 0, not odd integer +0 -8 y > 0, odd integer -8 -8 y > 0, not odd integer +8Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 178 +8 y < 0, +0 +8 y > 0, +8 NaN y nonzero, NaN x<>1 NaN, NaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 179 fastsin: fast double precision Sine double fastsin (double x) Weak alias: sin C Prototype: double fastsin (double x); Inputs: double x - the double precision input value. Outputs: Sine of x. Fortran Function Interface: DOUBLE PRECISION FASTSIN(X) Inputs: DOUBLE PRECISION X - the double precision input value. Return Value: Sine of X. Notes: fastsin computes the Sine function of its argument x. This is a relaxed version of sin, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 180 fastsinf: fast single precision Sine float fastsinf (?oat x) Weak alias: sinf C Prototype: ?oat fastsinf (?oat x); Inputs: ?oat x - the single precision input value. Outputs: Single precision Sine of x. Fortran Function Interface: REAL PRECISION FASTSINF(X) Inputs: REAL PRECISION X - the single precision input value. Return Value: Sine of X. Notes: fastsinf computes the Sine function of its argument x. This is a relaxed version of sinf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 181 fastsincos: fast double precision Sine and Cosine void fastsincos (double x, double *s, double *c) Weak alias: sincos C Prototype: void fastsincos (double x, double *s, double *c); Inputs: double x - the double precision input value. Outputs: double *s - Sine of x. double *c - Cosine of x. Fortran Subroutine Interface: SUBROUTINE FASTSINCOS(X,S,C) Inputs: DOUBLE PRECISION X - the double precision input value. Outputs: DOUBLE PRECISION S - Sine of X. DOUBLE PRECISION C - Cosine of X. Notes: fastsincos computes the Sine and Cosine functions of its argument x. This function can provide a signi?cant performance advantage for applications that require both the sine and cosine of an angle, such as axis and matrix rotation. This is a relaxed version of sincos, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 2 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 182 fastsincosf: fast single precision Sine and Cosine void fastsincosf (?oat x, ?oat *s, ?oat *c) Weak alias: sincosf C Prototype: void fastsincosf (?oat x, ?oat *s, ?oat *c); Inputs: ?oat x - the single precision input value. Outputs: ?oat *s - Sine of x. ?oat *c - Cosine of x. Fortran Subroutine Interface: SUBROUTINE FASTSINCOSF(X,S,C) Inputs: REAL X - the single precision input value. Outputs: REAL S - Sine of X. REAL C - Cosine of X. Notes: fastsincosf computes the Sine and Cosine functions of its argument x. This function can provide a signi?cant performance advantage for applications that require both the sine and cosine of an angle, such as axis and matrix rotation. This is a relaxed version of sincosf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 183 7.3 Fast Vector Math Functions This section documents the interfaces to a set of vector mathematical functions. vrd2 cos: Two-valued double precision Cosine __m128d __vrd2_cos ( m128d x) C Prototype: m128d vrd2 cos( m128d x); Inputs: m128d x - the double precision input value pair. Outputs: m128d y - the double precision Cosine result pair, returned in xmm0. Notes: vrd2 cos computes the Cosine function of two input arguments. This routine accepts a pair of double precision input values passed as a m128d value. The result is the double precision Cosine of both values, returned as a m128d value. This is a relaxed version of cos, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 2 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 184 vrd4 cos: Four-valued double precision Cosine __m128d,__m128d __vrd4_cos ( m128d x1, m128d x2) C Prototype: m128d vrd2 cos( m128d x); Note that this function uses a non-standard programming interface. The two m128d inputs, which contain four double precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128d is nonstandard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128d x1 - the ?rst double precision input value pair. m128d x2 - the second double precision input value pair. Outputs: m128d y1 - the ?rst double precision Cosine result pair, returned in xmm0. m128d y2 - second double precision Cosine result pair, returned in xmm1. Notes: vrd4 cos computes the Cosine function of four input arguments. This routine accepts four double precision input values passed as two m128d values. The result is the double precision Cosine of the four values, returned as two m128d values. This is a relaxed version of cos, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 2 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 185 vrda cos: Array double precision Cosine void vrda_cos (int n, double *x, double *y) C Prototype: void vrda cos (int n, double *x, double *y) Inputs: int n - the number of values in both the input and output arrays. double *x - pointer to the array of input values. double *y - pointer to the array of output values. Outputs: Cosine for each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRDA COS(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. DOUBLE PRECISION X(N) - array of double precision input values. Outputs: DOUBLE PRECISION Y(N) - array of Cosines of input values. Notes: vrda cos computes the Cosine function for each element of an array of input arguments. This routine accepts an array of double precision input values, computes cos(x) for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of cos, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 2 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 186 vrs4 cosf: Four-valued single precision Cosine __m128 __vrs4_cosf ( m128 x) C Prototype: m128 vrs4 cosf( m128 x); Inputs: m128 x - the four single precision input values. Outputs: m128 y - the four single precision Cosine results , returned in xmm0. Notes: vrs4 cosf computes the Cosine function of four input arguments. This routine accepts four single precision input values passed as a m128 value. The result is the single precision Cosine of all four values, returned as a m128 value. This is a relaxed version of cosf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 187 vrsa cosf: Array single precision Cosine void vrsa_cosf (int n, ?oat *x, ?oat *y) C Prototype: void vrsa cosf (int n, ?oat *x, ?oat *y) Inputs: int n - the number of values in both the input and output arrays. ?oat *x - pointer to the array of input values. ?oat *y - pointer to the array of output values. Outputs: Cosine for each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRSA COSF(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of single precision input values. Outputs: REAL Y(N) - array of Cosines of input values. Notes: vrsa cosf computes the Cosine function for each element of an array of input arguments. This routine accepts an array of single precision input values, computes cosf(x) for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of cosf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 188 vrd2 exp: Two-valued double precision exponential function __m128d __vrd2_exp ( m128d x) C Prototype: m128d vrd2 exp( m128d x); Inputs: m128d x - the double precision input value pair. Outputs: e raised to the power x (exponential of x). m128d y - the double precision exponent result pair, returned in xmm0. Notes: vrd2 exp computes the exponential function of two input arguments. This routine accepts a pair of double precision input values passed as a m128d value. The result is the double precision exponent of both values, returned as a m128d value. This is a relaxed version of exp, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -708.5 0 > 709.8 +8Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 189 vrd4 exp: Four-valued double precision exponential function __m128d,__m128d __vrd4_exp ( m128d x1, m128d x2) Prototype: m128d, m128d vrd4 exp( m128d x1, m128d x2); Note that this function uses a non-standard programming interface. The two m128d inputs, which contain four double precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128d is nonstandard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128d x1 - the ?rst double precision input value pair. m128d x2 - the second double precision input value pair. Outputs: m128d y1 - the ?rst double precision exponent result pair, returned in xmm0. m128d y2 - the second double precision exponent result pair, returned in xmm1. Notes: vrd4 exp computes the double precision exponential function of four input arguments. This routine accepts four double precision input values passed as two m128d values. The result is the double precision exponent of the four values, returned as two m128d values. This is a relaxed version of exp, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -708.5 0 > 709.8 +8Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 190 vrda exp: Array double precision exponential function void vrda_exp (int n, double *x, double *y) C Prototype: void vrda exp (int n, double *x, double *y) Inputs: int n - the number of values in both the input and output arrays. double *x - pointer to the array of input values. double *y - pointer to the array of output values. Outputs: e raised to the power x (exponential of x) for each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRDA EXP(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. DOUBLE PRECISION X(N) - array of double precision input values. Outputs: DOUBLE PRECISION Y(N) - array of exponentials (e raised to the power x) of input values. Notes: vrda exp computes the double precision exponential function for each element of an array of input arguments. This routine accepts an array of double precision input values, computes the e x for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of exp, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -708.5 0 > 709.8 +8Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 191 vrs4 expf: Four-valued single precision exponential function __m128 __vrs4_expf ( m128 x) C Prototype: m128 vrs4 expf( m128 x); Inputs: m128 x - the four single precision input values. Outputs: e raised to the power x (exponential of x) for each input value x. m128 y - the four single precision exponent results, returned in xmm0. Notes: vrs4 expf computes the double precision exponential function of four input arguments. This routine accepts four single precision input values passed as a m128 value. The result is the single precision exponent of the four values, returned as a m128 value. This is a relaxed version of exp, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -87.5 0 > 88 +8Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 192 vrs8 expf: Eight-valued single precision exponential function __m128,__m128 __vrs8_expf ( m128 x1, m128 x2) Prototype: m128, m128 vrs8 expf( m128 x1, m128 x2); Note that this function uses a non-standard programming interface. The two m128 inputs, which contain eight single precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128 is non-standard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128 x1 - the ?rst single precision vector of four input values. m128 x2 - the second single precision vector of four input values. Outputs: m128 y1 - the ?rst four single precision exponent results, returned in xmm0. m128 y2 - the second four single precision exponent results, returned in xmm1. Notes: vrs8 expf computes the single precision exponential function of eight input arguments. This routine accepts eight single precision input values passed as two m128 values. The result is the single precision exponent of the eight values, returned as two m128 values. This is a relaxed version of exp, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -87.5 0 > 88 +8Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 193 vrsa expf: Array single precision exponential function void vrsa_expf (int n, ?oat *x, ?oat *y) C Prototype: void vrsa expf (int n, ?oat *x, ?oat *y) Inputs: int n - the number of single precision values in both the input and output arrays. ?oat *x - pointer to the array of input values. ?oat *y - pointer to the array of output values. Outputs: e raised to the power x (exponential of x) for each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRSA EXPF(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of single precision input values. Outputs: REAL Y(N) - array of exponentials (e raised to the power x) of input values. Notes: vrsa expf computes the single precision exponential function for each element of an array of input arguments. This routine accepts an array of single precision input values, computes the e x for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of exp, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -87.5 0 > 88 +8Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 194 vrd2 log: Two-valued double precision natural logarithm __m128d __vrd2_log ( m128d x) C Prototype: m128d vrd2 log( m128d x); Inputs: m128d x - the double precision input value pair. Outputs: The natural (base e) logarithm of x. m128d y - the double precision natural logarithm result pair, returned in xmm0. Notes: vrd2 log computes the natural logarithm for each of two input arguments. This routine accepts a pair of double precision input values passed as a m128d value. The result is the double precision natural logarithm of both values, returned as a m128d value. This is a relaxed version of log, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 195 vrd4 log: Four-valued double precision natural logarithm __m128d,__m128d __vrd4_log ( m128d x1, m128d x2) Prototype: m128d, m128d vrd4 log( m128d x1, m128d x2); Note that this function uses a non-standard programming interface. The two m128d inputs, which contain four double precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128d is nonstandard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128d x1 - the ?rst double precision input value pair. m128d x2 - the second double precision input value pair. Outputs: The natural (base e) logarithm of x. m128d y1 - the ?rst double precision natural logarithm result pair, returned in xmm0. m128d y2 - the second double precision natural logarithm result pair, returned in xmm1. Notes: vrd4 log computes the natural logarithm for each of four input arguments. This routine accepts four double precision input values passed as two m128d values. The result is the double precision natural logarithm of the four values, returned as two m128d values. This is a relaxed version of log, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 196 vrda log: Array double precision natural logarithm void vrda_log (int n, double *x, double *y) C Prototype: void vrda log (int n, double *x, double *y) Inputs: int n - the number of values in both the input and output arrays. double *x - pointer to the array of input values. double *y - pointer to the array of output values. Outputs: The natural (base e) logarithm of each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRDA LOG(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. DOUBLE PRECISION X(N) - array of double precision input values. Outputs: DOUBLE PRECISION Y(N) - array of natural (base e) logarithms of input values. Notes: vrda log computes the double precision natural logarithm for each element of an array of input arguments. This routine accepts an array of double precision input values, computes the natural log for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of log, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 197 vrs4 logf: Four-valued single precision natural logarithm __m128 __vrs4_logf ( m128 x) C Prototype: m128 vrs4 logf( m128 x); Inputs: m128 x - the single precision input values. Outputs: The natural (base e) logarithm of x. m128 y - the single precision natural logarithm results, returned in xmm0. Notes: vrs4 logf computes the natural logarithm for each of four input arguments. This routine accepts four single precision input values passed as a m128 value. The result is the single precision natural logarithm of all four values, returned as a m128 value. This is a relaxed version of logf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 198 vrs8 logf: Eight-valued single precision natural logarithm __m128,__m128 __vrs8_logf ( m128 x1, m128 x2) Prototype: m128, m128 vrs8 logf( m128 x1, m128 x2); Note that this function uses a non-standard programming interface. The two m128 inputs, which contain eight single precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128 is non-standard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128 x1 - the ?rst single precision input value pair. m128 x2 - the second single precision input value pair. Outputs: The natural (base e) logarithm of x. m128 y1 - the ?rst single precision natural logarithm result pair, returned in xmm0. m128 y2 - the second single precision natural logarithm result pair, returned in xmm1. Notes: vrs8 logf computes the natural logarithm for each of eight input arguments. This routine accepts eight single precision input values passed as two m128 values. The result is the single precision natural logarithm of the eight values, returned as two m128 values. This is a relaxed version of logf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 199 vrsa logf: Array single precision natural logarithm void vrsa_logf (int n, ?oat *x, ?oat *y) C Prototype: void vrsa logf (int n, ?oat *x, ?oat *y) Inputs: int n - the number of values in both the input and output arrays. ?oat *x - pointer to the array of input values. ?oat *y - pointer to the array of output values. Outputs: The natural (base e) logarithm of each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRSA LOGF(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of single precision input values. Outputs: REAL Y(N) - array of natural (base e) logarithms of input values. Notes: vrsa logf computes the single precision natural logarithm for each element of an array of input arguments. This routine accepts an array of single precision input values, computes the natural log for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of logf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 200 vrd2 log10: Two-valued double precision base-10 logarithm __m128d __vrd2_log10 ( m128d x) C Prototype: m128d vrd2 log10( m128d x); Inputs: m128d x - the double precision input value pair. Outputs: The base-10 logarithm of x. m128d y - the double precision base-10 logarithm result pair, returned in xmm0. Notes: vrd2 log10 computes the base-10 logarithm for each of two input arguments. This routine accepts a pair of double precision input values passed as a m128d value. The result is the double precision base-10 logarithm of both values, returned as a m128d value. This is a relaxed version of log10, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 201 vrd4 log10: Four-valued double precision base-10 logarithm __m128d,__m128d __vrd4_log10 ( m128d x1, m128d x2) Prototype: m128d, m128d vrd4 log10( m128d x1, m128d x2); Note that this function uses a non-standard programming interface. The two m128d inputs, which contain four double precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128d is nonstandard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128d x1 - the ?rst double precision input value pair. m128d x2 - the second double precision input value pair. Outputs: The base-10 logarithm of x. m128d y1 - the ?rst double precision base-10 logarithm result pair, returned in xmm0. m128d y2 - the second double precision base-10 logarithm result pair, returned in xmm1. Notes: vrd4 log10 computes the base-10 logarithm for each of four input arguments. This routine accepts four double precision input values passed as two m128d values. The result is the double precision base-10 logarithm of the four values, returned as two m128d values. This is a relaxed version of log10, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 202 vrda log10: Array double precision base-10 logarithm void vrda_log10 (int n, double *x, double *y) C Prototype: void vrda log10 (int n, double *x, double *y) Inputs: int n - the number of values in both the input and output arrays. double *x - pointer to the array of input values. double *y - pointer to the array of output values. Outputs: The base-10 logarithm of each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRDA LOG10(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. DOUBLE PRECISION X(N) - array of double precision input values. Outputs: DOUBLE PRECISION Y(N) - array of base-10 logarithms of input values. Notes: vrda log10 computes the double precision base-10 logarithm for each element of an array of input arguments. This routine accepts an array of double precision input values, computes the base- 10 log for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of log10, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 203 vrs4 log10f: Four-valued single precision base-10 logarithm __m128 __vrs4_log10f ( m128 x) Prototype: m128 vrs4 log10f( m128 x); Inputs: m128 x - the four single precision inputs. Outputs: The base-10 logarithm of x. m128 y - the four single precision base-10 logarithm results, returned in xmm0. Notes: vrs4 log10f computes the base-10 logarithm for each of four input arguments. This routine accepts four single precision input values passed as a m128 value. The result is the single precision base-10 logarithm of the four values, returned as a m128 value. This is a relaxed version of log10, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 204 vrs8 log10f: Eight-valued single precision base-10 logarithm __m128,__m128 __vrs8_log10f ( m128 x1, m128 x2) Prototype: m128, m128 vrs8 log10f( m128 x1, m128 x2); Note that this function uses a non-standard programming interface. The two m128 inputs, which contain eight single precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128 is non-standard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128 x1 - the ?rst set of four single precision input values. m128 x2 - the second set of four single precision input values. Outputs: The base-10 logarithm of x. m128 y1 - the ?rst set of four single precision base-10 logarithm results, returned in xmm0. m128 y2 - the second set of four single precision base-10 logarithm results, returned in xmm1. Notes: vrs8 log10f computes the base-10 logarithm for each of eight input arguments. This routine accepts eight single precision input values passed as two m128 values. The result is the single precision base-10 logarithm of the eight values, returned as two m128 values. This is a relaxed version of log10f, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 205 vrsa log10f: Array single precision base-10 logarithm void vrsa_log10f (int n, ?oat *x, ?oat *y) C Prototype: void vrsa log10f (int n, ?oat *x, ?oat *y) Inputs: int n - the number of values in both the input and output arrays. ?oat *x - pointer to the array of input values. ?oat *y - pointer to the array of output values. Outputs: The base-10 logarithm of each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRSA LOG10F(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of single precision input values. Outputs: REAL Y(N) - array of base-10 logarithms of input values. Notes: vrsa log10f computes the single precision base-10 logarithm for each element of an array of input arguments. This routine accepts an array of single precision input values, computes the base- 10 log for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of log10f, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 206 vrd2 log2: Two-valued double precision base-2 logarithm __m128d __vrd2_log2 ( m128d x) C Prototype: m128d vrd2 log2( m128d x); Inputs: m128d x - the double precision input value pair. Outputs: The base-2 logarithm of x. m128d y - the double precision base-2 logarithm result pair, returned in xmm0. Notes: vrd2 log2 computes the base-2 logarithm for each of two input arguments. This routine accepts a pair of double precision input values passed as a m128d value. The result is the double precision base-2 logarithm of both values, returned as a m128d value. This is a relaxed version of log2, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 207 vrd4 log2: Four-valued double precision base-2 logarithm __m128d,__m128d __vrd4_log2 ( m128d x1, m128d x2) Prototype: m128d, m128d vrd4 log2( m128d x1, m128d x2); Note that this function uses a non-standard programming interface. The two m128d inputs, which contain four double precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128d is nonstandard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128d x1 - the ?rst double precision input value pair. m128d x2 - the second double precision input value pair. Outputs: The base-2 logarithm of x. m128d y1 - the ?rst double precision base-2 logarithm result pair, returned in xmm0. m128d y2 - the second double precision base-2 logarithm result pair, returned in xmm1. Notes: vrd4 log2 computes the base-2 logarithm for each of four input arguments. This routine accepts four double precision input values passed as two m128d values. The result is the double precision base-2 logarithm of the four values, returned as two m128d values. This is a relaxed version of log2, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 208 vrda log2: Array double precision base-2 logarithm void vrda_log2 (int n, double *x, double *y) C Prototype: void vrda log2 (int n, double *x, double *y) Inputs: int n - the number of values in both the input and output arrays. double *x - pointer to the array of input values. double *y - pointer to the array of output values. Outputs: The base-2 logarithm of each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRDA LOG2(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. DOUBLE PRECISION X(N) - array of double precision input values. Outputs: DOUBLE PRECISION Y(N) - array of base-2 logarithms of input values. Notes: vrda log2 computes the double precision base-2 logarithm for each element of an array of input arguments. This routine accepts an array of double precision input values, computes the base- 2 log for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of log2, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 209 vrs4 log2f: Four-valued single precision base-2 logarithm __m128 __vrs4_log2f ( m128 x) Prototype: m128 vrs4 log2f( m128 x); Inputs: m128 x - the four single precision inputs. Outputs: The base-2 logarithm of x. m128 y - the four single precision base-2 logarithm results, returned in xmm0. Notes: vrs4 log2f computes the base-2 logarithm for each of four input arguments. This routine accepts four single precision input values passed as a m128 value. The result is the single precision base-2 logarithm of the four values, returned as a m128 value. This is a relaxed version of log2, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 210 vrs8 log2f: Eight-valued single precision base-2 logarithm __m128,__m128 __vrs8_log2f ( m128 x1, m128 x2) Prototype: m128, m128 vrs8 log2f( m128 x1, m128 x2); Note that this function uses a non-standard programming interface. The two m128 inputs, which contain eight single precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128 is non-standard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128 x1 - the ?rst set of four single precision input values. m128 x2 - the second set of four single precision input values. Outputs: The base-2 logarithm of x. m128 y1 - the ?rst set of four single precision base-2 logarithm results, returned in xmm0. m128 y2 - the second set of four single precision base-2 logarithm results, returned in xmm1. Notes: vrs8 log2f computes the base-2 logarithm for each of eight input arguments. This routine accepts eight single precision input values passed as two m128 values. The result is the single precision base-2 logarithm of the eight values, returned as two m128 values. This is a relaxed version of log2f, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 211 vrsa log2f: Array single precision base-2 logarithm void vrsa_log2f (int n, ?oat *x, ?oat *y) C Prototype: void vrsa log2f (int n, ?oat *x, ?oat *y) Inputs: int n - the number of values in both the input and output arrays. ?oat *x - pointer to the array of input values. ?oat *y - pointer to the array of output values. Outputs: The base-2 logarithm of each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRSA LOG2F(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of single precision input values. Outputs: REAL Y(N) - array of base-2 logarithms of input values. Notes: vrsa log2f computes the single precision base-2 logarithm for each element of an array of input arguments. This routine accepts an array of single precision input values, computes the base-2 log for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of log2f, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 212 vrs4 powf: Four-valued single precision power function __m128 __vrs4_powf ( m128 x, m128 y) C Prototype: m128 vrs4 powf( m128 x, m128 y); Inputs: m128 x - the single precision input base values. m128 y - the single precision input exponent values. Outputs: m128 z - the single precision results of each x raised to the y power, returned in xmm0. Notes: vrs4 powf() computes the single precision x raised to the y power for four pairs of input arguments. This routine accepts four single precision input value pairs passed as m128 values. The result is the x raised to the y power for all four input pairs, returned as a m128 value. This is a relaxed version of powf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 0.5 ulp over the valid input range. Special case return values: Input x Input y Output ±0 y < 0, odd integer ±8 ±0 y < 0, not odd integer +8 ±0 y > 0, odd integer ±0 ±0 y > 0, not odd integer +0 -1 +8 1 +1 y (incl. NaN) 1 x (incl. Nan) ±0 1 x < 0 y, not integer QNaN |x|<1 -8 +8 |x|>1 -8 +0 |x|<1 +8 +0 |x|>1 +8 +8 -8 y < 0, odd integer -0 -8 y < 0, not odd integer +0 -8 y > 0, odd integer -8 -8 y > 0, not odd integer +8 +8 y < 0, +0 +8 y > 0, +8 NaN y nonzero, NaN x<>1 NaN, NaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 213 vrsa powf: Array single precision power function void vrsa_powf (int n, ?oat *x, ?oat *y, ?oat *z) C Prototype: void vrsa powf(int n, ?oat *x, ?oat *y, ?oat *z); Inputs: ?oat *x - pointer to the array of single precision input x values. ?oat *y - pointer to the array of single precision input y values. ?oat *z - pointer to the array of single precision output values. int n - the number of single precision values in both the input and output arrays. Outputs: x raised to the y value for each array pair, ?lled into the z array. Fortran Subroutine Interface: VRSA POWF(INTEGER*4 N, REAL*4 X(), REAL*4 Y(), REAL*4 Z()) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of real x input values. REAL Y(N) - array of real y input values. Outputs: REAL Z(N) - array of real result values. Notes: vrsa powf() computes x to the y power in single precision for each pair of elements in the x and y input arrays. This routine accepts an array of single precision input x values and an arrayi of single precision input y values, computes x^y for each input value pair, and stores the result in the array pointed to by the z input. This is a relaxed version of powf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 0.5 ulp over the valid input range. Special case return values: Input x Input y Output ±0 y < 0, odd integer ±8 ±0 y < 0, not odd integer +8 ±0 y > 0, odd integer ±0 ±0 y > 0, not odd integer +0 -1 +8 1 +1 y (incl. NaN) 1 x (incl. Nan) ±0 1 x < 0 y, not integer QNaN |x|<1 -8 +8Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 214 |x|>1 -8 +0 |x|<1 +8 +0 |x|>1 +8 +8 -8 y < 0, odd integer -0 -8 y < 0, not odd integer +0 -8 y > 0, odd integer -8 -8 y > 0, not odd integer +8 +8 y < 0, +0 +8 y > 0, +8 NaN y nonzero, NaN x<>1 NaN, NaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 215 vrs4 powxf: Four-valued single precision power function with constant y __m128 __vrs4_powxf ( m128 x,?oat y) C Prototype: m128 vrs4 powxf( m128 x,?oat y); Inputs: m128 x - the single precision input base values. ?oat y - the common single precision input exponent value. Outputs: m128 z - the single precision results of each x raised to the y power, returned in xmm0. Notes: vrs4 powxf() computes the single precision x raised to the y power for four input x arguments and a constant y input value. This routine accepts four single precision input values passed as an m128 value. The y value is passed as one single precision value. The result is the x raised to the y power for all four input values, returned as a m128 value. This is a relaxed version of powxf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 0.5 ulp over the valid input range. Special case return values: Input x Input y Output ±0 y < 0, odd integer ±8 ±0 y < 0, not odd integer +8 ±0 y > 0, odd integer ±0 ±0 y > 0, not odd integer +0 -1 +8 1 +1 y (incl. NaN) 1 x (incl. Nan) ±0 1 x < 0 y, not integer QNaN |x|<1 -8 +8 |x|>1 -8 +0 |x|<1 +8 +0 |x|>1 +8 +8 -8 y < 0, odd integer -0 -8 y < 0, not odd integer +0 -8 y > 0, odd integer -8 -8 y > 0, not odd integer +8 +8 y < 0, +0 +8 y > 0, +8 NaN y nonzero, NaN x<>1 NaN, NaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 216 vrsa powxf: Array single precision power function, constant y void vrsa_powf (int n, ?oat *x, ?oat y, ?oat *z) C Prototype: void vrsa powxf(int n, ?oat *x, ?oat y, ?oat *z); Inputs: int n - the number of single precision values in both the x input and output arrays. ?oat *x - pointer to the array of single precision input x values. ?oat *z - pointer to the array of single precision output values. ?oat y - the constant single precision input y value. Outputs: x raised to the y value for each x array value, ?lled into the z array Fortran Subroutine Interface: VRSA POWF(INTEGER*4 N, REAL*4 X(), REAL*4 Y, REAL*4 Z()) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of real x input values. REAL Y - the constant single precision input y value. Outputs: REAL Z(N) - array of real result values. Notes: vrsa powxf() computes x to the y power in single precision for each element in the x input arrays, using a constant y. This routine accepts an array of single precision input x values and one single precision input y value, computes x^y for each x input value, and stores the result in the array pointed to by the z pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of powf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 0.5 ulp over the valid input range. Special case return values: Input x Input y Output ±0 y < 0, odd integer ±8 ±0 y < 0, not odd integer +8 ±0 y > 0, odd integer ±0 ±0 y > 0, not odd integer +0 -1 +8 1Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 217 +1 y (incl. NaN) 1 x (incl. Nan) ±0 1 x < 0 y, not integer QNaN |x|<1 -8 +8 |x|>1 -8 +0 |x|<1 +8 +0 |x|>1 +8 +8 -8 y < 0, odd integer -0 -8 y < 0, not odd integer +0 -8 y > 0, odd integer -8 -8 y > 0, not odd integer +8 +8 y < 0, +0 +8 y > 0, +8 NaN y nonzero, NaN x<>1 NaN, NaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 218 vrd2 sin: Two-valued double precision Sine __m128d __vrd2_sin ( m128d x) C Prototype: m128d vrd2 sin( m128d x); Inputs: m128d x - the double precision input value pair. Outputs: m128d y - the double precision Sine result pair, returned in xmm0. Notes: vrd2 sin computes the Sine function of two input arguments. This routine accepts a pair of double precision input values passed as a m128d value. The result is the double precision Sine of both values, returned as a m128d value. This is a relaxed version of sin, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 219 vrd4 sin: Four-valued double precision Sine __m128d,__m128d __vrd4_sin ( m128d x1, m128d x2) C Prototype: m128d vrd4 sin( m128d x); Note that this function uses a non-standard programming interface. The two m128d inputs, which contain four double precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128d is nonstandard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128d x1 - the ?rst double precision input value pair. m128d x2 - the second double precision input value pair. Outputs: m128d y1 - the ?rst double precision Sine result pair, returned in xmm0. m128d y2 - second double precision Sine result pair, returned in xmm1. Notes: vrd4 sin computes the Sine function of four input arguments. This routine accepts four double precision input values passed as two m128d values. The result is the double precision Sine of the four values, returned as two m128d values. This is a relaxed version of sin, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. This routine may return slightly worse than 1 ulp for very large values between 4e5 and 5e5. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 220 vrda sin: Array double precision Sine void vrda_sin (int n, double *x, double *y) C Prototype: void vrda sin (int n, double *x, double *y) Inputs: int n - the number of values in both the input and output arrays. double *x - pointer to the array of input values. double *y - pointer to the array of output values. Outputs: Sine for each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRDA SIN(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. DOUBLE PRECISION X(N) - array of double precision input values. Outputs: DOUBLE PRECISION Y(N) - array of Sines of input values. Notes: vrda sin computes the Sine function for each element of an array of input arguments. This routine accepts an array of double precision input values, computes sin(x) for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of sin, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. This routine may return slightly worse than 1 ulp for very large values between 4e5 and 5e5. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 221 vrs4 sinf: Four-valued single precision Sine __m128 __vrs4_sinf ( m128 x) C Prototype: m128 vrs4 sinf( m128 x); Inputs: m128 x - the four single precision inputs. Outputs: m128 y - the four single precision Sine results, returned in xmm0. Notes: vrs4 sinf computes the Sine function of four input arguments. This routine accepts four single precision input values passed as a m128 value. The result is the single precision Sine of the four values, returned as a m128 value. This is a relaxed version of sinf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. This routine may return slightly worse than 1 ulp for very large values between 4e5 and 5e5. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 222 vrsa sinf: Array single precision Sine void vrsa_sinf (int n, ?oat *x, ?oat *y) C Prototype: void vrsa sinf (int n, ?oat *x, ?oat *y) Inputs: int n - the number of values in both the input and output arrays. ?oat *x - pointer to the array of input values. ?oat *y - pointer to the array of output values. Outputs: Sine for each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRSA SINF(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of single precision input values. Outputs: REAL Y(N) - array of Sines of input values. Notes: vrsa sinf computes the Sine function for each element of an array of input arguments. This routine accepts an array of single precision input values, computes sin(x) for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of sinf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 223 vrd2 sincos: Two-valued double precision Sine and Cosine void __vrd2_sincos ( m128d x, m128d* S, m128d* C) C Prototype: void vrd2 sincos( m128d x, m128d* S, m128d* C)); Inputs: m128d x - the double precision input value pair. Outputs: (Sine of x and Cosine of x.) m128d *S - Pointer to the double precision Sine result pair. m128d *C - Pointer to the double precision Cosine result pair. Notes: vrd2 sincos computes the Sine and Cosine functions of two input arguments. This routine accepts a pair of double precision input values passed as a m128d value. The result is the double precision Sin and Cosine of both values, returned as a m128d value. This is a relaxed version of sincos, suitable for use with fastmath compiler ?ags or application not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 2 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 224 vrda sincos: Array double precision Sine and Cosine void vrda_sincos (int n, double *x, double *ys, double *yc) C Prototype: void vrda sincos (int n, double *x, double *ys, double *yc) Inputs: int n - the number of values in both the input and output arrays. double *x - pointer to the array of input values. double *ys - pointer to the array of sin output values. double *yc - pointer to the array of cos output values. Outputs: Sine for each x value, ?lled into the ys array. Cosine for each x value, ?lled into the yc array. Fortran Subroutine Interface: SUBROUTINE VRDA SINCOS(N,X,YS,YC) Inputs: INTEGER N - the number of values in both the input and output arrays. DOUBLE PRECISION X(N) - array of double precision input values. Outputs: DOUBLE PRECISION YS(N) - array of Sines of input values. DOUBLE PRECISION YC(N) - array of Cosines of input values. Notes: vrda sincos computes the Sine and Cosine functions for each element of an array of input arguments. This routine accepts an array of double precision input values, computes sincos(x) for each input value, and stores the results in the arrays pointed to by the ys and yc pointer inputs. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of sincos, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 2 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 225 vrs4 sincosf: Four-valued single precision Sine and Cosine void __vrs4_sincosf ( m128 x, m128* S, m128* C) C Prototype: void vrs4 sincosf( m128 x, m128* S, m128* C)); Inputs: m128 x - the single precision input value pair. Outputs: (Sine of x and Cosine of x.) m128 *S - Pointer to the single precision Sine result pair. m128 *C - Pointer to the single precision Cosine result pair. Notes: vrs4 sincosf computes the Sine and Cosine functions of four input arguments. This routine accepts four single precision input values passed as a m128 value. The result is the single precision Sin and Cosine of all four values, returned as a m128 value. This is a relaxed version of sincosf, suitable for use with fastmath compiler ?ags or application not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 226 vrsa sincosf: Array single precision Sine and Cosine void vrsa_sincosf (int n, ?oat *x, ?oat *ys, ?oat *yc) C Prototype: void vrsa sincosf (int n, ?oat *x, ?oat *ys, ?oat *yc) Inputs: int n - the number of values in both the input and output arrays. ?oat *x - pointer to the array of input values. ?oat *ys - pointer to the array of sin output values. ?oat *yc - pointer to the array of cos output values. Outputs: Sine for each x value, ?lled into the ys array. Cosine for each x value, ?lled into the yc array. Fortran Subroutine Interface: SUBROUTINE VRSA SINCOSF(N,X,YS,YC) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of single precision input values. Outputs: REAL YS(N) - array of Sines of input values. REAL YC(N) - array of Cosines of input values. Notes: vrsa sincosf computes the Sine and Cosine functions for each element of an array of input arguments. This routine accepts an array of single precision input values, computes sincos(x) for each input value, and stores the results in the arrays pointed to by the ys and yc pointer inputs. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of sincos, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaNChapter 8: References 227 8 References • [1] C.L. Lawson, R.J. Hanson, D. Kincaid, and F.T. Krogh, Basic linear algebra subprograms for Fortran usage, ACM Trans. Maths. Soft., 5 (1979), pp. 308–323. • [2] J.J. Dongarra, J. Du Croz, S. Hammarling, and R.J. Hanson, An extended set of FORTRAN basic linear algebra subroutines, ACM Trans. Math. Soft., 14 (1988), pp. 1–17. • [3] J.J. Dongarra, J. Du Croz, I.S. Du?, and S. Hammarling, A set of level 3 basic linear algebra subprograms, ACM Trans. Math. Soft., 16 (1990), pp. 1–17. • [4] David S. Dodson, Roger G. Grimes, John G. Lewis, Sparse Extensions to the FORTRAN Basic Linear Algebra Subprograms, ACM Trans. Math. Soft., 17 (1991), pp. 253–263. • [5] E. Anderson, Z. Bai, C. Bischof, S. Blackford, J. Demmel, J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, A. McKenney, and D. Sorensen, LAPACK User’s Guide, SIAM, Philidelphia, (1999). • [6] D. E. Knuth, The Art of Computer Programming Addison-Wesley, 1997. • [7] J. Banks, Handbook on Simulation, Wiley, 1998. • [8] A. Menezes, P. van Oorschot, and S. Vanstone, Handbook of Applied Cryptography, Chapter 5, CRC Press, 1996. • [9] Chapter Introduction G05 - Random Number Generators The NAG Fortran Library Manual, Mark 21 Numerical Algorithms Group, 2005. • [10] N. M. Maclaren, The generation of multiple independent sequences of pseudorandom numbers, Appl. Statist., 1989, 38, 351-359. • [11] M. Matsumoto and T. Nishimura, Mersenne twister: A 623-dimensionally equidistributed uniform pseudorandom number generator, ACM Transactions on Modelling and Computer Simulations, 1998. • [12] P. L’Ecuyer, Good parameter sets for combined multiple recursive random number generators, Operations Research, 1999, 47, 159-164. • [13] Programming languages - C - ISO/IEC 9899:1999 • [14] IEEE Standard for Binary Floating-Point Arithmetic (ANSI/IEEE Std 754-1985) • [15] P. L’Ecuyer and R. Simard, TestU01: A Software Library in ANSI C for Empirical Testing of Random Number Generators, Departement d’Informatique et de Recherche Operationnelle, Universite de Montreal, 2002. Software and user’s guide available at http://www.iro.umontreal.ca/~lecuyerSubject Index 228 Subject Index 2 2D FFT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3 3D FFT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 A accessing ACML (Absoft af90 under Linux). . . . . 7 accessing ACML (Compaq Visual Fortran under 32-bit Windows). . . . . . . . . . . . . . . . . . . . . . . . . . 9 accessing ACML (GNU gfortran/gcc under Linux) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 accessing ACML (Intel Fortran/Microsoft C under 32-bit Windows). . . . . . . . . . . . . . . . . . . . . . . . . . 8 accessing ACML (Intel Fortran/Microsoft C under 64-bit Windows). . . . . . . . . . . . . . . . . . . . . . . . . 11 accessing ACML (Intel ifort under Linux). . . . . . . 6 accessing ACML (Linux) . . . . . . . . . . . . . . . . . . . . . . 4 accessing ACML (NAGware f95 compiler under Linux). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 accessing ACML (Open64 openf95/opencc under Linux). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 accessing ACML (other compilers under Linux). . 7 accessing ACML (PGI pgf77/pgf90/Microsoft C under 32-bit Windows). . . . . . . . . . . . . . . . . . . . 8 accessing ACML (PGI pgf77/pgf90/pgcc or Microsoft C under 64-bit Windows). . . . . . . 10 accessing ACML (PGI pgf77/pgf90/pgcc under Linux). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 accessing ACML (Salford ftn95 under 32-bit Windows). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 accessing ACML (Sun f95 compiler under Solaris or Linux) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 accessing ACML (Sun f95/cc under Solaris or Linux). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 accessing ACML under Windows . . . . . . . . . . . . . . . 8 accessing the base generators . . . . . . . . . . . . . . . . . 82 ACML C Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . 13 ACML FORTRAN interfaces . . . . . . . . . . . . . . . . . 13 ACML installation test. . . . . . . . . . . . . . . . . . . . . . . 16 ACML Memory Allocation . . . . . . . . . . . . . . . . . . . 17 ACML performance examples . . . . . . . . . . . . . . . . . 16 ACML version information . . . . . . . . . . . . . . . . . . . 15 ACML MV (ACML vector math functions). . . 163 ACML MV types . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 B base generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 base generator, basic NAG generator . . . . . . . . . . 83 base generator, blum-blum-shub . . . . . . . . . . . . . . 85 base generator, calling . . . . . . . . . . . . . . . . . . . . . . . 82 base generator, de?nition . . . . . . . . . . . . . . . . . . . . . 75 base generator, initialization . . . . . . . . . . . . . . . . . . 76 base generator, L’Ecuyer’s combined recursive generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 base generator, Mersenne twister . . . . . . . . . . . . . . 84 base generator, recommendation . . . . . . . . . . . . . . 75 base generator, user supplied . . . . . . . . . . . . . . . . . 86 base generator, Wichmann-Hill . . . . . . . . . . . . . . . 84 basic NAG base generator . . . . . . . . . . . . . . . . . . . . 83 beta distribution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 binomial distribution. . . . . . . . . . . . . . . . . . . . . . . . 125 binomial distribution, using reference vector . . 139 BLAS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 blum-blum-shub generator . . . . . . . . . . . . . . . . . . . . 85 BRNG, de?nition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 C C interfaces in ACML. . . . . . . . . . . . . . . . . . . . . . . . 13 calling the base generators . . . . . . . . . . . . . . . . . . . . 82 cauchy distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 99 chi-squared distribution . . . . . . . . . . . . . . . . . . . . . 101 complex FFT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 continuous multivariate distribution, gaussian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 continuous multivariate distribution, gaussian, using reference vector . . . . . . . . . . . . . . . . . . . 153 continuous multivariate distribution, normal . . 149 continuous multivariate distribution, normal, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 153 continuous multivariate distribution, students t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 continuous multivariate distribution, students t, using reference vector . . . . . . . . . . . . . . . . . . . 155 continuous univariate distribution, beta . . . . . . . 97 continuous univariate distribution, cauchy . . . . . 99 continuous univariate distribution, chi-squared . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 continuous univariate distribution, exponential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 continuous univariate distribution, f . . . . . . . . . . 105 continuous univariate distribution, ?sher’s variance ratio distribution . . . . . . . . . . . . . . . . . . . . . . . 105 continuous univariate distribution, gamma . . . . 107 continuous univariate distribution, gaussian. . . 109 continuous univariate distribution, logistic . . . . 111 continuous univariate distribution, lognormal. . 113 continuous univariate distribution, normal . . . . 109 continuous univariate distribution, students t. . 115 continuous univariate distribution, t. . . . . . . . . . 115 continuous univariate distribution, triangular . . 117 continuous univariate distribution, uniform . . . 119 continuous univariate distribution, von mises . . 121 continuous univariate distribution, weibull . . . . 123 copying a generator . . . . . . . . . . . . . . . . . . . . . . . . . . 76 cryptologically secure, de?nition . . . . . . . . . . . . . . 75 cryptologically secure, generator . . . . . . . . . . . . . . 85Subject Index 229 D determining the best ACML version for your system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 discrete multivariate distribution, multinomial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 discrete univariate distribution, binomial . . . . . 125 discrete univariate distribution, binomial, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 139 discrete univariate distribution, geometric . . . . 127 discrete univariate distribution, geometric, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 141 discrete univariate distribution, hypergeometric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 discrete univariate distribution, hypergeometric, using reference vector . . . . . . . . . . . . . . . . . . . 143 discrete univariate distribution, negative binomial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 discrete univariate distribution, negative binomial, using reference vector . . . . . . . . . . . . . . . . . . . 145 discrete univariate distribution, poisson. . . . . . . 133 discrete univariate distribution, poisson, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 147 discrete univariate distribution, uniform . . . . . . 135 distribution generator, de?nition . . . . . . . . . . . . . . 75 E example programs . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 exponential distribution . . . . . . . . . . . . . . . . . . . . . 103 F f distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 fast basic math functions . . . . . . . . . . . . . . . . . . . . 165 Fast Fourier Transforms . . . . . . . . . . . . . . . . . . . . . . 24 feedback shift generator . . . . . . . . . . . . . . . . . . . . . . 84 FFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 FFT e?ciency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 FFT of multiple complex sequences . . . . . . . . . . . 34 FFT of multiple Hermitian sequences. . . . . . . . . . 73 FFT of multiple real sequences . . . . . . . . . . . . . . . 69 FFT of single complex sequence. . . . . . . . . . . . . . . 27 FFT of single Hermitian sequence . . . . . . . . . . . . . 71 FFT of single real sequence . . . . . . . . . . . . . . . . . . . 67 FFT plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 ?sher’s variance ratio distribution . . . . . . . . . . . . 105 FORTRAN interfaces in ACML. . . . . . . . . . . . . . . 13 G gamma distribution . . . . . . . . . . . . . . . . . . . . . . . . . 107 gaussian distribution (multivariate) . . . . . . . . . . 149 gaussian distribution (univariate) . . . . . . . . . . . . 109 gaussian distribution, multivariate, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 general information . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 generalized feedback shift generator . . . . . . . . . . . 84 generating discrete variates from a reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 geometric distribution. . . . . . . . . . . . . . . . . . . . . . . 127 geometric distribution, using reference vector . . 141 H Hermitian data sequences (FFT). . . . . . . . . . . . . . 66 hypergeometric distribution . . . . . . . . . . . . . . . . . 129 hypergeometric distribution, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 I IEEE exceptions and LAPACK . . . . . . . . . . . . . . . 23 initialization of a generator . . . . . . . . . . . . . . . . . . . 76 installation test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 INTEGER*8 arguments . . . . . . . . . . . . . . . . . . . . . . 14 introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 L L’Ecuyer’s combined recursive generator . . . . . . . 85 language interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 LAPACK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 LAPACK blocking factors . . . . . . . . . . . . . . . . . . . . 21 LAPACK reference sources . . . . . . . . . . . . . . . . . . . 20 libm names. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 library manual. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 library version information . . . . . . . . . . . . . . . . . . . 15 linear congruential generator, basic NAG generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 linear congruential generator, Wichmann-Hill . . 84 linking with ACML . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 linking with Linux ACML . . . . . . . . . . . . . . . . . . . . . 4 linking with Sun f95 ACML . . . . . . . . . . . . . . . . . . 12 linking with Windows ACML . . . . . . . . . . . . . . . . . . 8 logistic distribution . . . . . . . . . . . . . . . . . . . . . . . . . 111 lognormal distribution . . . . . . . . . . . . . . . . . . . . . . 113 M Mersenne twister . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Mersenne twister, multiple streams . . . . . . . . . . . . 91 multinomial distribution . . . . . . . . . . . . . . . . . . . . 161 multiple recursive generator, L’Ecuyer’s combined recursive generator . . . . . . . . . . . . . . . . . . . . . . 85 multiple streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 multiple streams, block splitting . . . . . . . . . . . . . . 91 multiple streams, L’Ecuyer’s combined recursive generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91, 94 multiple streams, leap frogging. . . . . . . . . . . . . . . . 94 multiple streams, Mersenne twister . . . . . . . . . . . . 91 multiple streams, NAG basic generator . . . . . 91, 94 multiple streams, skip ahead. . . . . . . . . . . . . . . . . . 91 multiple streams, using di?erent generators . . . . 91 multiple streams, using di?erent seeds . . . . . . . . . 91 multiple streams, Wichmann-Hill generator . . . 91, 94 multivariate distribution, gaussian . . . . . . . . . . . 149 multivariate distribution, gaussian, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 multivariate distribution, multinomial . . . . . . . . 161Subject Index 230 multivariate distribution, normal. . . . . . . . . . . . . 149 multivariate distribution, normal, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 multivariate distribution, students t. . . . . . . . . . 151 multivariate distribution, students t, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 155 N negative binomial distribution . . . . . . . . . . . . . . . 131 negative binomial distribution, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 normal distribution (multivariate). . . . . . . . . . . . 149 normal distribution (univariate). . . . . . . . . . . . . . 109 normal distribution, multivariate, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 P performance example programs . . . . . . . . . . . . . . . 16 period of a random number generator, de?nition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 plan, default, FFTs . . . . . . . . . . . . . . . . . . . . . . . . . . 26 plan, generated, FFTs. . . . . . . . . . . . . . . . . . . . . . . . 26 poisson distribution . . . . . . . . . . . . . . . . . . . . . . . . . 133 poisson distribution, using reference vector . . . 147 PRNG, de?nition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 pseudo-random number, de?nition . . . . . . . . . . . . 75 Q QRNG, de?nition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 quasi-random number, de?nition . . . . . . . . . . . . . . 75 R random bit stream . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 real data sequences (FFT). . . . . . . . . . . . . . . . . . . . 66 real FFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 reference vector, binomial distribution . . . . . . . . 139 reference vector, gaussian (multivariate) . . . . . . 157 reference vector, generating discrete variates from . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 reference vector, geometric distribution . . . . . . . 141 reference vector, hypergeometric distribution. . 143 reference vector, negative binomial distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 reference vector, normal (multivariate) . . . . . . . 157 reference vector, poisson distribution . . . . . . . . . 147 reference vector, students t (multivariate). . . . . 159 retrieving the state of a generator . . . . . . . . . . . . . 76 S saving the state of a generator . . . . . . . . . . . . . . . . 76 seed, de?nition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 size of integer arguments . . . . . . . . . . . . . . . . . . . . . 14 sparse BLAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 students t distribution . . . . . . . . . . . . . . . . . . . . . . 115 students t distribution (multivariate). . . . . . . . . 151 students t distribution, multivariate, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 155 T t distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 triangular distribution . . . . . . . . . . . . . . . . . . . . . . 117 U uniform distribution (continuous) . . . . . . . . . . . . 119 uniform distribution (discrete). . . . . . . . . . . . . . . 135 univariate distribution, beta . . . . . . . . . . . . . . . . . . 97 univariate distribution, binomial . . . . . . . . . . . . . 125 univariate distribution, binomial, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 univariate distribution, cauchy . . . . . . . . . . . . . . . . 99 univariate distribution, chi-squared. . . . . . . . . . . 101 univariate distribution, exponential . . . . . . . . . . 103 univariate distribution, f . . . . . . . . . . . . . . . . . . . . 105 univariate distribution, ?sher’s variance ratio. . 105 univariate distribution, gamma . . . . . . . . . . 107, 109 univariate distribution, geometric . . . . . . . . . . . . 127 univariate distribution, geometric, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 univariate distribution, hypergeometric . . . . . . . 129 univariate distribution, hypergeometric, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 143 univariate distribution, logistic. . . . . . . . . . . . . . . 111 univariate distribution, lognormal . . . . . . . . . . . . 113 univariate distribution, negative binomial. . . . . 131 univariate distribution, negative binomial, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 145 univariate distribution, normal. . . . . . . . . . . . . . . 109 univariate distribution, poisson . . . . . . . . . . . . . . 133 univariate distribution, poisson, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 univariate distribution, students t. . . . . . . . . . . . 115 univariate distribution, t . . . . . . . . . . . . . . . . . . . . 115 univariate distribution, triangular . . . . . . . . . . . . 117 univariate distribution, uniform (continuous). . 119 univariate distribution, uniform (discrete) . . . . 135 univariate distribution, von mises . . . . . . . . . . . . 121 univariate distribution, weibull. . . . . . . . . . . . . . . 123 user supplied generators . . . . . . . . . . . . . . . . . . . . . . 86 V vector math functions . . . . . . . . . . . . . . . . . . . . . . . 183 von mises distribution. . . . . . . . . . . . . . . . . . . . . . . 121 W weak aliases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 weibull distribution . . . . . . . . . . . . . . . . . . . . . . . . . 123 Wichmann-Hill base generator . . . . . . . . . . . . . . . . 84 Wichmann-Hill, multiple streams . . . . . . . . . . . . . 91Routine Index 231 Routine Index __vrd2_cos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 __vrd2_exp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 __vrd2_log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 __vrd2_log10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 __vrd2_log2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 __vrd2_sin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 __vrd2_sincos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 __vrd4_cos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 __vrd4_exp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 __vrd4_log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 __vrd4_log10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 __vrd4_log2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 __vrd4_sin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 __vrs4_cosf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 __vrs4_expf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 __vrs4_log10f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 __vrs4_log2f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 __vrs4_logf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 __vrs4_powf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 __vrs4_powxf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 __vrs4_sincosf . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 __vrs4_sinf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 __vrs8_expf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 __vrs8_log10f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 __vrs8_log2f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 __vrs8_logf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 A acmlinfo. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 ACMLINFO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 acmlversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 ACMLVERSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 C CFFT1D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 CFFT1DX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 CFFT1M . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 CFFT1MX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 CFFT2D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 CFFT2DX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 CFFT3D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 CFFT3DX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 CFFT3DY. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 CSFFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 CSFFTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 D DRANDBETA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 DRANDBINOMIAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 DRANDBINOMIALREFERENCE . . . . . . . . . . . . . . . . . . . 139 DRANDBLUMBLUMSHUB . . . . . . . . . . . . . . . . . . . . . . . . . . 83 DRANDCAUCHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 DRANDCHISQUARED . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 DRANDDISCRETEUNIFORM. . . . . . . . . . . . . . . . . . . . . . 135 DRANDEXPONENTIAL . . . . . . . . . . . . . . . . . . . . . . . . . . 103 DRANDF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 DRANDGAMMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 DRANDGAUSSIAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 DRANDGENERALDISCRETE. . . . . . . . . . . . . . . . . . . . . . 137 DRANDGEOMETRIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 DRANDGEOMETRICREFERENCE . . . . . . . . . . . . . . . . . . 141 DRANDHYPERGEOMETRIC. . . . . . . . . . . . . . . . . . . . . . . 129 DRANDHYPERGEOMETRICREFERENCE . . . . . . . . . . . . . 143 DRANDINITIALIZE . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 DRANDINITIALIZEBBS . . . . . . . . . . . . . . . . . . . . . . . . . 81 DRANDINITIALIZEUSER. . . . . . . . . . . . . . . . . . . . . . . . 87 DRANDLEAPFROG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 DRANDLOGISTIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 DRANDLOGNORMAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 DRANDMULTINOMIAL . . . . . . . . . . . . . . . . . . . . . . . . . . 161 DRANDMULTINORMAL . . . . . . . . . . . . . . . . . . . . . . . . . . 149 DRANDMULTINORMALR . . . . . . . . . . . . . . . . . . . . . . . . . 153 DRANDMULTINORMALREFERENCE . . . . . . . . . . . . . . . . 157 DRANDMULTISTUDENTSREFERENCE . . . . . . . . . . . . . . 159 DRANDMULTISTUDENTST. . . . . . . . . . . . . . . . . . . . . . . 151 DRANDMULTISTUDENTSTR. . . . . . . . . . . . . . . . . . . . . . 155 DRANDNEGATIVEBINOMIAL . . . . . . . . . . . . . . . . . . . . 131 DRANDNEGATIVEBINOMIALREFERENCE . . . . . . . . . . . 145 DRANDPOISSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 DRANDPOISSONREFERENCE . . . . . . . . . . . . . . . . . . . . 147 DRANDSKIPAHEAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 DRANDSTUDENTST . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 DRANDTRIANGULAR . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 DRANDUNIFORM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 DRANDVONMISES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 DRANDWEIBULL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 DZFFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 DZFFTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 F fastcos. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 fastcosf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 fastexp. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 fastexpf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 fastlog. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 fastlog10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 fastlog10f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 fastlog2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 fastlog2f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 fastlogf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 fastpow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 fastpowf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 fastsin. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 fastsincos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 fastsincosf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 fastsinf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180Routine Index 232 I ILAENVSET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 S SCFFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 SCFFTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 SRANDBINOMIALREFERENCE . . . . . . . . . . . . . . . . . . . 139 SRANDCHISQUARED . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 SRANDGEOMETRICREFERENCE . . . . . . . . . . . . . . . . . . 141 SRANDBETA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 SRANDBINOMIAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 SRANDBLUMBLUMSHUB . . . . . . . . . . . . . . . . . . . . . . . . . . 83 SRANDCAUCHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 SRANDDISCRETEUNIFORM. . . . . . . . . . . . . . . . . . . . . . 135 SRANDEXPONENTIAL . . . . . . . . . . . . . . . . . . . . . . . . . . 103 SRANDF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 SRANDGAMMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 SRANDGAUSSIAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 SRANDGENERALDISCRETE. . . . . . . . . . . . . . . . . . . . . . 137 SRANDGEOMETRIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 SRANDHYPERGEOMETRIC. . . . . . . . . . . . . . . . . . . . . . . 129 SRANDHYPERGEOMETRICREFERENCE . . . . . . . . . . . . . 143 SRANDINITIALIZE . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 SRANDINITIALIZEBBS . . . . . . . . . . . . . . . . . . . . . . . . . 81 SRANDINITIALIZEUSER. . . . . . . . . . . . . . . . . . . . . . . . 87 SRANDLEAPFROG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 SRANDLOGISTIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 SRANDLOGNORMAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 SRANDMULTINOMIAL . . . . . . . . . . . . . . . . . . . . . . . . . . 161 SRANDMULTINORMAL . . . . . . . . . . . . . . . . . . . . . . . . . . 149 SRANDMULTINORMALR . . . . . . . . . . . . . . . . . . . . . . . . . 153 SRANDMULTINORMALREFERENCE . . . . . . . . . . . . . . . . 157 SRANDMULTISTUDENTST. . . . . . . . . . . . . . . . . . . . . . . 151 SRANDMULTISTUDENTSTR. . . . . . . . . . . . . . . . . . . . . . 155 SRANDMULTISTUDENTSTREFERENCE . . . . . . . . . . . . . 159 SRANDNEGATIVEBINOMIAL . . . . . . . . . . . . . . . . . . . . 131 SRANDNEGATIVEBINOMIALREFERENCE . . . . . . . . . . . 145 SRANDPOISSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 SRANDPOISSONREFERENCE . . . . . . . . . . . . . . . . . . . . 147 SRANDSKIPAHEAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 SRANDSTUDENTST . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 SRANDTRIANGULAR . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 SRANDUNIFORM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 SRANDVONMISES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 SRANDWEIBULL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 U UGEN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 UINI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 V vrda_cos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 vrda_exp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 vrda_log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 vrda_log10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 vrda_log2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 vrda_sin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 vrda_sincos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 vrsa_cosf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 vrsa_expf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 vrsa_log10f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 vrsa_log2f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 vrsa_logf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 vrsa_powf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213, 216 vrsa_sincosf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 vrsa_sinf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 Z ZDFFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 ZDFFTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 ZFFT1D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 ZFFT1DX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 ZFFT1M . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 ZFFT1MX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 ZFFT2D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 ZFFT2DX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 ZFFT3D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 ZFFT3DX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 ZFFT3DY. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 AMD Core Math Library (ACML) Version 4.0.0 Copyright c 2003-2007 Advanced Micro Devices, Inc., Numerical Algorithms Group Ltd. AMD, the AMD Arrow logo, AMD Opteron, AMD Athlon and combinations thereof are trademarks of Advanced Micro Devices, Inc.i Short Contents 1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 General Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3 BLAS: Basic Linear Algebra Subprograms . . . . . . . . . . . . . 19 4 LAPACK: Package of Linear Algebra Subroutines . . . . . . . . 20 5 Fast Fourier Transforms (FFTs) . . . . . . . . . . . . . . . . . . . . 24 6 Random Number Generators. . . . . . . . . . . . . . . . . . . . . . . 74 7 ACML MV: Fast Math and Fast Vector Math Library . . . . 161 8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 Subject Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 Routine Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231ii Table of Contents 1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 General Information . . . . . . . . . . . . . . . . . . . . . . . 2 2.1 Determining the best ACML version for your system . . . . . . . . . . . 2 2.2 Accessing the Library (Linux). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2.1 Accessing the Library under Linux using GNU g77/gcc . . . . 4 2.2.2 Accessing the Library under Linux using GNU gfortran/gcc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2.3 Accessing the Library under Linux using PGI compilers pgf77/pgf90/pgcc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2.4 Accessing the Library under Linux using PathScale compilers pathf90/pathcc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2.5 Accessing the Library under Linux using the NAGWare f95 compiler. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2.6 Accessing the Library under Linux using the Intel ifort compiler. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2.7 Accessing the Library under Linux using compilers other than GNU, PGI, PathScale, NAGWare or Intel. . . . . . . . . . . . . . . . . . . 8 2.3 Accessing the Library (Microsoft Windows) . . . . . . . . . . . . . . . . . . . 8 2.3.1 Accessing the Library under 32-bit Windows using GNU g77/gcc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3.2 Accessing the Library under 32-bit Windows using PGI compilers pgf77/pgf90/Microsoft C . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3.3 Accessing the Library under 32-bit Windows using Microsoft C or Intel Fortran. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3.4 Accessing the Library under 32-bit Windows using the Compaq Visual Fortran compiler. . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.3.5 Accessing the Library under 32-bit Windows using the Salford FTN95 compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.3.6 Accessing the Library under 64-bit Windows using PGI compilers pgf77/pgf90/pgcc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.3.7 Accessing the Library under 64-bit Windows using Microsoft C or Intel Fortran. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.4 Accessing the Library (Solaris) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.4.1 Accessing the Library under Solaris . . . . . . . . . . . . . . . . . . . . . 13 2.5 ACML FORTRAN and C interfaces . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.6 ACML variants using 64-bit integer (INTEGER*8) arguments. . 15 2.7 Library Version and Build Information . . . . . . . . . . . . . . . . . . . . . . . 16 2.8 Library Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.9 Example programs calling ACML. . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.10 Example ACML programs demonstrating performance . . . . . . . 17 3 BLAS: Basic Linear Algebra Subprograms . . 19iii 4 LAPACK: Package of Linear Algebra Subroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.1 Introduction to LAPACK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.2 Reference sources for LAPACK. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.3 LAPACK block sizes, ILAENV and ILAENVSET. . . . . . . . . . . . . 21 4.4 IEEE exceptions and LAPACK. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 5 Fast Fourier Transforms (FFTs) . . . . . . . . . . . 24 5.1 Introduction to FFTs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 5.1.1 Transform de?nitions and Storage for Complex Data . . . . . 24 5.1.2 Transform de?nitions and Storage for Real Data . . . . . . . . . 25 5.1.3 E?ciency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 5.1.4 Default and Generated Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 5.2 FFTs on Complex Sequences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5.2.1 FFT of a single sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 ZFFT1D Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 CFFT1D Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 ZFFT1DX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 CFFT1DX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 5.2.2 FFT of multiple complex sequences . . . . . . . . . . . . . . . . . . . . . 34 ZFFT1M Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 CFFT1M Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 ZFFT1MX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 CFFT1MX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.2.3 2D FFT of two-dimensional arrays of data . . . . . . . . . . . . . . . 43 ZFFT2D Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 CFFT2D Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 ZFFT2DX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 CFFT2DX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 5.2.4 3D FFT of three-dimensional arrays of data . . . . . . . . . . . . . 52 ZFFT3D Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 CFFT3D Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 ZFFT3DX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 CFFT3DX Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 ZFFT3DY Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 CFFT3DY Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 5.3 FFTs on real and Hermitian data sequences . . . . . . . . . . . . . . . . . . 65 5.3.1 FFT of single sequences of real data. . . . . . . . . . . . . . . . . . . . . 66 DZFFT Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 SCFFT Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 5.3.2 FFT of multiple sequences of real data . . . . . . . . . . . . . . . . . . 68 DZFFTM Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 SCFFTM Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 5.3.3 FFT of single Hermitian sequences. . . . . . . . . . . . . . . . . . . . . . 70 ZDFFT Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 CSFFT Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5.3.4 FFT of multiple Hermitian sequences. . . . . . . . . . . . . . . . . . . . 72 ZDFFTM Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72iv CSFFTM Routine Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 6 Random Number Generators . . . . . . . . . . . . . . 74 6.1 Base Generators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 6.1.1 Initialization of the Base Generators . . . . . . . . . . . . . . . . . . . . 75 DRANDINITIALIZE / SRANDINITIALIZE . . . . . . . . . . . . . . . . . . . . . . 76 DRANDINITIALIZEBBS / SRANDINITIALIZEBBS . . . . . . . . . . . . . . . 79 6.1.2 Calling the Base Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 DRANDBLUMBLUMSHUB / SRANDBLUMBLUMSHUB . . . . . . . . . . . . . . . . . 81 6.1.3 Basic NAG Generator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 6.1.4 Wichmann-Hill Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 6.1.5 Mersenne Twister. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 6.1.6 L’Ecuyer’s Combined Recursive Generator. . . . . . . . . . . . . . . 83 6.1.7 Blum-Blum-Shub Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 6.1.8 User Supplied Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 DRANDINITIALIZEUSER / SRANDINITIALIZEUSER . . . . . . . . . . . . . 85 UINI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 UGEN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 6.2 Multiple Streams. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 6.2.1 Using Di?erent Seeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 6.2.2 Using Di?erent Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 6.2.3 Skip Ahead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 DRANDSKIPAHEAD / SRANDSKIPAHEAD . . . . . . . . . . . . . . . . . . . . . . . . 90 6.2.4 Leap Frogging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 DRANDLEAPFROG / SRANDLEAPFROG . . . . . . . . . . . . . . . . . . . . . . . . . . 93 6.3 Distribution Generators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 6.3.1 Continuous Univariate Distributions. . . . . . . . . . . . . . . . . . . . . 95 DRANDBETA / SRANDBETA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 DRANDCAUCHY / SRANDCAUCHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 DRANDCHISQUARED / SRANDCHISQUARED . . . . . . . . . . . . . . . . . . . . . . 99 DRANDEXPONENTIAL / SRANDEXPONENTIAL. . . . . . . . . . . . . . . . . . . 101 DRANDF / SRANDF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 DRANDGAMMA / SRANDGAMMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 DRANDGAUSSIAN / DRANDGAUSSIAN . . . . . . . . . . . . . . . . . . . . . . . . . 107 DRANDLOGISTIC / SRANDLOGISTIC . . . . . . . . . . . . . . . . . . . . . . . . . 109 DRANDLOGNORMAL / SRANDLOGNORMAL . . . . . . . . . . . . . . . . . . . . . . . 111 DRANDSTUDENTST / SRANDSTUDENTST . . . . . . . . . . . . . . . . . . . . . . . 113 DRANDTRIANGULAR / SRANDTRIANGULAR . . . . . . . . . . . . . . . . . . . . . 115 DRANDUNIFORM / SRANDUNIFORM. . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 DRANDVONMISES / SRANDVONMISES . . . . . . . . . . . . . . . . . . . . . . . . . 119 DRANDWEIBULL / SRANDWEIBULL. . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 6.3.2 Discrete Univariate Distributions . . . . . . . . . . . . . . . . . . . . . . 123 DRANDBINOMIAL / SRANDBINOMIAL . . . . . . . . . . . . . . . . . . . . . . . . . 123 DRANDGEOMETRIC / SRANDGEOMETRIC . . . . . . . . . . . . . . . . . . . . . . . 125 DRANDHYPERGEOMETRIC / SRANDHYPERGEOMETRIC . . . . . . . . . . . . 127 DRANDNEGATIVEBINOMIAL / SRANDNEGATIVEBINOMIAL. . . . . . . . 129 DRANDPOISSON / SRANDPOISSON. . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 DRANDDISCRETEUNIFORM / SRANDDISCRETEUNIFORM . . . . . . . . . . 133v DRANDGENERALDISCRETE / SRANDGENERALDISCRETE . . . . . . . . . . 135 DRANDBINOMIALREFERENCE / SRANDBINOMIALREFERENCE . . . . . 137 DRANDGEOMETRICREFERENCE / SRANDGEOMETRICREFERENCE . . . 139 DRANDHYPERGEOMETRICREFERENCE / SRANDHYPERGEOMETRICREFERENCE . . . . . . . . . . . . . . . . . . . . . 141 DRANDNEGATIVEBINOMIALREFERENCE / SRANDNEGATIVEBINOMIALREFERENCE . . . . . . . . . . . . . . . . . . . 143 DRANDPOISSONREFERENCE / SRANDPOISSONREFERENCE. . . . . . . . 145 6.3.3 Continuous Multivariate Distributions. . . . . . . . . . . . . . . . . . 147 DRANDMULTINORMAL / SRANDMULTINORMAL. . . . . . . . . . . . . . . . . . . 147 DRANDMULTISTUDENTST / SRANDMULTISTUDENTST . . . . . . . . . . . . 149 DRANDMULTINORMALR / SRANDMULTINORMALR . . . . . . . . . . . . . . . . 151 DRANDMULTISTUDENTSTR / SRANDMULTISTUDENTSTR . . . . . . . . . . 153 DRANDMULTINORMALREFERENCE / SRANDMULTINORMALREFERENCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 DRANDMULTISTUDENTSTREFERENCE / SRANDMULTISTUDENTSTREFERENCE . . . . . . . . . . . . . . . . . . . . . 157 6.3.4 Discrete Multivariate Distributions. . . . . . . . . . . . . . . . . . . . . 159 DRANDMULTINOMIAL / SRANDMULTINOMIAL. . . . . . . . . . . . . . . . . . . 159 7 ACML MV: Fast Math and Fast Vector Math Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 7.1 Introduction to ACML MV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 7.1.1 Terminology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 7.1.2 Weak Aliases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 7.1.3 De?ned Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 7.2 Fast Basic Math Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 fastcos: fast double precision Cosine . . . . . . . . . . . . . . . . . . . . . . . . . . 163 fastcosf: fast single precision Cosine . . . . . . . . . . . . . . . . . . . . . . . . . . 164 fastexp: fast double precision exponential function . . . . . . . . . . . . . 165 fastexpf: fast single precision exponential function . . . . . . . . . . . . . 166 fastlog: fast double precision natural logarithm function. . . . . . . . 167 fastlogf: fast single precision natural logarithm function . . . . . . . . 168 fastlog10: fast double precision base-10 logarithm function . . . . . 169 fastlog10f: fast single precision base-10 logarithm function . . . . . . 170 fastlog2: fast double precision base-2 logarithm function. . . . . . . . 171 fastlog2f: fast single precision base-2 logarithm function . . . . . . . . 172 fastpow: fast double precision power function. . . . . . . . . . . . . . . . . . 173 fastpowf: fast single precision power function . . . . . . . . . . . . . . . . . . 175 fastsin: fast double precision Sine. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 fastsinf: fast single precision Sine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 fastsincos: fast double precision Sine and Cosine . . . . . . . . . . . . . . . 179 fastsincosf: fast single precision Sine and Cosine . . . . . . . . . . . . . . . 180 7.3 Fast Vector Math Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 vrd2 cos: Two-valued double precision Cosine . . . . . . . . . . . . . . . . . 181 vrd4 cos: Four-valued double precision Cosine . . . . . . . . . . . . . . . . . 182 vrda cos: Array double precision Cosine . . . . . . . . . . . . . . . . . . . . . . 183 vrs4 cosf: Four-valued single precision Cosine . . . . . . . . . . . . . . . . . 184vi vrsa cosf: Array single precision Cosine . . . . . . . . . . . . . . . . . . . . . . . 185 vrd2 exp: Two-valued double precision exponential function. . . . 186 vrd4 exp: Four-valued double precision exponential function . . . 187 vrda exp: Array double precision exponential function . . . . . . . . . 188 vrs4 expf: Four-valued single precision exponential function . . . . 189 vrs8 expf: Eight-valued single precision exponential function . . . 190 vrsa expf: Array single precision exponential function. . . . . . . . . . 191 vrd2 log: Two-valued double precision natural logarithm. . . . . . . 192 vrd4 log: Four-valued double precision natural logarithm. . . . . . . 193 vrda log: Array double precision natural logarithm . . . . . . . . . . . . 194 vrs4 logf: Four-valued single precision natural logarithm . . . . . . . 195 vrs8 logf: Eight-valued single precision natural logarithm . . . . . . 196 vrsa logf: Array single precision natural logarithm. . . . . . . . . . . . . 197 vrd2 log10: Two-valued double precision base-10 logarithm. . . . . 198 vrd4 log10: Four-valued double precision base-10 logarithm . . . . 199 vrda log10: Array double precision base-10 logarithm . . . . . . . . . . 200 vrs4 log10f: Four-valued single precision base-10 logarithm . . . . . 201 vrs8 log10f: Eight-valued single precision base-10 logarithm . . . . 202 vrsa log10f: Array single precision base-10 logarithm. . . . . . . . . . . 203 vrd2 log2: Two-valued double precision base-2 logarithm. . . . . . . 204 vrd4 log2: Four-valued double precision base-2 logarithm . . . . . . 205 vrda log2: Array double precision base-2 logarithm . . . . . . . . . . . . 206 vrs4 log2f: Four-valued single precision base-2 logarithm . . . . . . . 207 vrs8 log2f: Eight-valued single precision base-2 logarithm . . . . . . 208 vrsa log2f: Array single precision base-2 logarithm. . . . . . . . . . . . . 209 vrs4 powf: Four-valued single precision power function . . . . . . . . . 210 vrsa powf: Array single precision power function . . . . . . . . . . . . . . 211 vrs4 powxf: Four-valued single precision power function with constant y. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 vrsa powxf: Array single precision power function, constant y . . 215 vrd2 sin: Two-valued double precision Sine . . . . . . . . . . . . . . . . . . . 217 vrd4 sin: Four-valued double precision Sine . . . . . . . . . . . . . . . . . . . 218 vrda sin: Array double precision Sine . . . . . . . . . . . . . . . . . . . . . . . . . 219 vrs4 sinf: Four-valued single precision Sine . . . . . . . . . . . . . . . . . . . . 220 vrsa sinf: Array single precision Sine . . . . . . . . . . . . . . . . . . . . . . . . . 221 vrd2 sincos: Two-valued double precision Sine and Cosine. . . . . . 222 vrda sincos: Array double precision Sine and Cosine . . . . . . . . . . . 223 vrs4 sincosf: Four-valued single precision Sine and Cosine . . . . . . 225 vrsa sincosf: Array single precision Sine and Cosine. . . . . . . . . . . . 226 8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 Subject Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 Routine Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231Chapter 1: Introduction 1 1 Introduction The AMD Core Math Library (ACML) is a set of numerical routines tuned speci?cally for AMD64 platform processors (including Opteron TM and Athlon TM 64 ). The routines, which are available via both FORTRAN 77 and C interfaces, include: • BLAS - Basic Linear Algebra Subprograms (including Sparse Level 1 BLAS); • LAPACK - A comprehensive package of higher level linear algebra routines; • FFT - a set of Fast Fourier Transform routines for real and complex data; • RNG - a set of random number generators and statistical distribution functions. The BLAS and LAPACK routines provide a portable and standard set of interfaces for common numerical linear algebra operations that allow code containing calls to these routines to be readily ported across platforms. Full documentation for the BLAS and LAPACK are available online. This manual will, therefore, be restricted to providing brief descriptions of the BLAS and LAPACK and providing links to their documentation and other materials (see Chapter 3 [The BLAS], page 19 and see Chapter 4 [LAPACK], page 20). The FFT is an implementation of the Discrete Fourier Transform (DFT) that makes use of symmetries in the de?nition to reduce the number of operations required from O(n*n) to O(n*log n) when the sequence length, n, is the product of small prime factors; in particular, when n is a power of 2. Despite the popularity and widespread use of FFT algorithms, the de?nition of the DFT is not su?ciently precise to prescribe either the forward and backward directions (these are sometimes interchanged), or the scaling factor associated with the forward and backward transforms (the combined forward and backward transforms may only reproduce the original sequence by following a prescribed scaling). Currently, there is no agreed standard API for FFT routines. Hardware vendors usually provide a set of high performance FFTs optimized for their systems: no two vendors employ the same interfaces for their FFT routines. The ACML provides a set of FFT routines, optimized for AMD64 processors, using an ACML-speci?c set of interfaces. The functionality, interfaces and use of the ACML FFT routines are described below (see Chapter 5 [Fast Fourier Transforms], page 24). The RNG is a comprehensive set of statistical distribution functions which are founded on various underlying uniform distribution generators (base generators) including WichmannHill and an implementation of the Mersenne Twister. In addition there are hooks which allow you to supply your own preferred base generator if it is not already included in ACML. All RNG functionality and interfaces are described below (see Chapter 6 [Random Number Generators], page 74). Chapter 2 [General Information], page 2 provides details on: • how to link a user program to the ACML; • FORTRAN and C interfaces to ACML routines; • how to obtain the ACML version and build information; • how to access the ACML documentation. A supplementary library of fast math and fast vector math functions (ACML MV) is also provided with some 64-bit versions of ACML. Some of the functions included in ACML MV are not callable from high-level languages, but must be called via assembly language; the documentation of ACML MV (see Chapter 7 [Fast Vector Math Library], page 161) gives details for each individual routine.Chapter 2: General Information 2 2 General Information 2.1 Determining the best ACML version for your system ACML comes in versions for 64-bit and 32-bit processors, running both Linux and Microsoft Windows R operating systems. To use the following tables, you will need to know answers to these questions: • Are you running a 64-bit operating system (on AMD64 hardware such as Opteron or Athlon64)? Or are you running a 32-bit operating system? • Is the operating system Linux or Microsoft Windows? • Do you have the GNU compilers (g77/gcc or gfortran/gcc) or compatible compilers (compilers that are interoperable with the GNU compilers) installed? • Do you have the PGI compilers (pgf77/pgf90/pgcc) installed? • Do you have the PathScale compilers (pathf90/pathcc) installed? • Do you have the NAGWare compiler (f95) installed? • On a 32-bit Windows machine, do you have Microsoft C, or PGI Visual Fortran, or Intel Fortran, or compatible compilers installed? • Do you have a single processor system or a multiprocessor (SMP) system? The single processor version of ACML can be run on an SMP machine and vice versa, but (if you have the right compilers) it is more e?cient to run the version appropriate to the machine. • If you’re on a 32-bit machine, does it support Streaming SIMD Extension instructions (SSE and SSE2)? The ACML installation includes a binary utility that can help you ?nd an answer to the last question. The utility lies in directory util, and is named cpuid.exe. It interrogates the processor to determine whether SSE and SSE2 instructions exist. util/cpuid.exe Under a Linux operating system, another way of ?nding out the answer to the last question is to look at the special ?le /proc/cpuinfo, and see what appears under the “?ags” label. Try this command: cat /proc/cpuinfo | grep flags If the list of ?ags includes the ?ag “sse” then your machine supports SSE instructions. If it also includes “sse2” then your machine supports SSE2 instructions. If your machine supports these instructions, it is better to use a version of ACML which was built to take advantage of them, for reasons of good performance. The method of examining /proc/cpuinfo can also be used under Microsoft Windows if you have the Cygwin UNIX-like tools installed (see http://www.cygwin.com/) and run a bash shell. Note that AMD64 machines always support both SSE and SSE2 instructions, under both Linux and Windows. Older (32-bit) AMD chips may support SSE but not SSE2, or neither SSE nor SSE2 instructions. Other manufacturers’ hardware may or may not support SSE or SSE2. If you link to a version of ACML that was built to use SSE or SSE2 instructions, and your machine does not in fact support them, it is likely that your program will halt dueChapter 2: General Information 3 to encountering an “illegal instruction” - you may or may not be noti?ed of this by the operating system. For 32-bit machines, older versions of ACML (ACML 3.1.0 and earlier) came in variants suitable for hardware without SSE/SSE2 instructions (Streaming SIMD Extensions). This is no longer the case, and if you have older 32-bit hardware that does not support SSE/SSE2, and wish to use ACML, you must continue to use an older version. Once you have answered the questions above, use these tables to decide which version of ACML to link against. Linux 64-bit   Number of processors Compilers ACML install directory Single processor GNU g77/gcc or compatible acml4.0.0/gnu64 ” GNU gfortran/gcc acml4.0.0/gfortran64 ” PGI pgf77/pgf90/pgcc acml4.0.0/pgi64 ” PathScale pathf90/pathcc acml4.0.0/pathscale64 ” NAGWare f95 acml4.0.0/nag64 ” Intel Fortran acml4.0.0/ifort64 Multi processor PGI pgf77/pgf90/pgcc acml4.0.0/pgi64_mp ” PathScale pathf90/pathcc acml4.0.0/pathscale64_mp ” GNU gfortran/gcc acml4.0.0/gfortran64_mp ” Intel Fortran acml4.0.0/ifort64_mp Linux 32-bit   Number of processors Compilers ACML install directory Single GNU g77 / gcc or compat. acml4.0.0/gnu32 ” GNU gfortran / gcc acml4.0.0/gfortran32 ” PGI pgf77 / pgf90 / pgcc acml4.0.0/pgi32 ” PathScale pathf90 / pathcc acml4.0.0/pathscale32 ” NAGWare f95 acml4.0.0/nag32 ” Intel Fortran acml4.0.0/ifort32 Multiple PGI pgf77 / pgf90 / pgcc acml4.0.0/pgi32_mp ” PathScale pathf90 / pathcc acml4.0.0/pathscale32_mp ” GNU gfortran / gcc acml4.0.0/gfortran32_mp ” Intel Fortran acml4.0.0/ifort32_mp Microsoft Windows 64-bit   Number of processors Compilers ACML install directory Single processor PGI pgf77/pgf90/pgcc/MSC acml4.0.0/win64 ” Intel Fortran/Microsoft C acml4.0.0/ifort64 Multi processor PGI pgf77/pgf90/pgcc/MSC acml4.0.0/win64_mp ” Intel Fortran/Microsoft C acml4.0.0/ifort64_mp Chapter 2: General Information 4 Microsoft Windows 32-bit   Number of processors Compilers ACML install directory Single GNU g77/gcc acml4.0.0/gnu32 ” PGI pgf77/pgf90/Microsoft C acml4.0.0/pgi32 ” Intel Fortran/Microsoft C acml4.0.0/ifort32 Multi PGI pgf77/pgf90/Microsoft C acml4.0.0/pgi32_mp ” Intel Fortran/Microsoft C acml4.0.0/ifort32_mp 2.2 Accessing the Library (Linux) 2.2.1 Accessing the Library under Linux using GNU g77/gcc If the Linux 64-bit g77 version of ACML was installed in the default directory, /opt/acml4.0.0/gnu64, then the command: g77 -m64 driver.f -L/opt/acml4.0.0/gnu64/lib -lacml can be used to compile the program driver.f and link it to the ACML. The ACML Library is supplied in both static and shareable versions, libacml.a and libacml.so, respectively. By default, the commands given above will link to the shareable version of the library, libacml.so, if that exists in the directory speci?ed. Linking with the static library can be forced either by using the compiler ?ag -static, e.g. g77 -m64 driver.f -L/opt/acml4.0.0/gnu64/lib -static -lacml or by inserting the name of the static library explicitly in the command line, e.g. g77 -m64 driver.f /opt/acml4.0.0/gnu64/lib/libacml.a Notice that if the application program has been linked to the shareable ACML Library, then before running the program, the environment variable LD_LIBRARY_PATH must be set, for example, by the C-shell command: setenv LD_LIBRARY_PATH /opt/acml4.0.0/gnu64/lib where it is assumed that libacml.so was installed in the directory /opt/acml4.0.0/gnu64/lib (see the man page for ld(1) for more information about LD_LIBRARY_PATH.). The command g77 -m32 driver.f -L/opt/acml4.0.0/gnu32/lib -lacml will compile and link a 32-bit program with a 32-bit ACML. To compile and link a 64-bit C program with a 64-bit ACML, invoke gcc -m64 -I/opt/acml4.0.0/gnu64/include driver.c -L/opt/acml4.0.0/gnu64/lib -lacml -lg2c The switch "-I/opt/acml4.0.0/gnu64/include" tells the compiler to search the directory /opt/acml4.0.0/gnu64/include for the ACML C header ?le acml.h, which should be included by driver.c. Note that it is necessary to add the compiler run-time library -lg2c when linking the program.Chapter 2: General Information 5 2.2.2 Accessing the Library under Linux using GNU gfortran/gcc If the Linux 64-bit gfortran version of ACML was installed in the default directory, /opt/acml4.0.0/gfortran64, then the command: gfortran -m64 driver.f -L/opt/acml4.0.0/gfortran64/lib -lacml can be used to compile the program driver.f and link it to the ACML. The ACML Library is supplied in both static and shareable versions, libacml.a and libacml.so, respectively. By default, the commands given above will link to the shareable version of the library, libacml.so, if that exists in the directory speci?ed. Linking with the static library can be forced either by using the compiler ?ag -static, e.g. gfortran -m64 driver.f -L/opt/acml4.0.0/gfortran64/lib -static -lacml or by inserting the name of the static library explicitly in the command line, e.g. gfortran -m64 driver.f /opt/acml4.0.0/gfortran64/lib/libacml.a Notice that if the application program has been linked to the shareable ACML Library, then before running the program, the environment variable LD_LIBRARY_PATH must be set. Assuming that libacml.so was installed in the directory /opt/acml4.0.0/gfortran64/lib, then LD_LIBRARY_PATH may be set by, for example, the C-shell command setenv LD_LIBRARY_PATH /opt/acml4.0.0/gfortran64/lib (See the man page for ld(1) for more information about LD_LIBRARY_PATH.) The command gfortran -m32 driver.f -L/opt/acml4.0.0/gfortran32/lib -lacml will compile and link a 32-bit program with a 32-bit ACML. If you have an SMP machine and want to take best advantage of it, link against the gfortran OpenMP version of ACML like this: gfortran -fopenmp -m64 driver.f -L/opt/acml4.0.0/gfortran64_mp/lib -lacml_mp gfortran -fopenmp -m32 driver.f -L/opt/acml4.0.0/gfortran32_mp/lib -lacml_mp Note that the directories and library names involved now include the su?x mp. To compile and link a 64-bit C program with a 64-bit ACML, invoke gcc -m64 -I/opt/acml4.0.0/gfortran64/include driver.c -L/opt/acml4.0.0/gfortran64/lib -lacml -lgfortran The switch "-I/opt/acml4.0.0/gfortran64/include" tells the compiler to search the directory /opt/acml4.0.0/gfortran64/include for the ACML C header ?le acml.h, which should be included by driver.c. Note that it is necessary to add the gfortran compiler run-time library -lgfortran when linking the program.Chapter 2: General Information 6 2.2.3 Accessing the Library under Linux using PGI compilers pgf77/pgf90/pgcc Similar commands apply for the PGI versions of ACML. For example, pgf77 -tp=k8-64 -Mcache_align driver.f -L/opt/acml4.0.0/pgi64/lib -lacml pgf77 -tp=k8-32 -Mcache_align driver.f -L/opt/acml4.0.0/pgi32/lib -lacml will compile driver.f and link it to the ACML using 64-bit and 32-bit versions respectively. In the example above we are linking with the single-processor PGI version of ACML. If you have an SMP machine and want to take best advantage of it, link against the PGI OpenMP version of ACML like this: pgf77 -tp=k8-64 -mp -Mcache_align driver.f -L/opt/acml4.0.0/pgi64_mp/lib -lacml_mp pgf77 -tp=k8-32 -mp -Mcache_align driver.f -L/opt/acml4.0.0/pgi32_mp/lib -lacml_mp Note that the directories and library names involved now include the su?x mp. The -mp ?ag is important - it tells pgf77 to link with the appropriate compiler OpenMP run-time library. Without it you might get an "unresolved symbol" message at link time. The -Mcache align ?ag is also important - it tells the compiler to align objects on cache-line boundaries. The commands pgcc -c -tp=k8-64 -mp -Mcache_align -I/opt/acml4.0.0/pgi64_mp/include driver.c pgcc -tp=k8-64 -mp -Mcache_align driver.o -L/opt/acml4.0.0/pgi64_mp/lib -lacml_mp -lpgftnrtl -lm will compile driver.c and link it to the 64-bit ACML. Again, the -mp ?ag is important if you are linking to the PGI OpenMP version of ACML. The C compiler is instructed to search the directory /opt/acml4.0.0/pgi64 mp/include for the ACML C header ?le acml.h, which should be included by driver.c, by using the switch "-I/opt/acml4.0.0/pgi64 mp/include". Note that in the example we add the libraries -lpgftnrtl and -lm to the link command, so that required PGI compiler run-time libraries are found. Note that since ACML version 3.5.0, all PGI 64-bit variants are compiled with the PGI -Mlarge arrays switch to allow use of larger data arrays (see PGI compiler documentation for more information). The special ’large array’ variants that were distributed with earlier versions of ACML are therefore no longer required. 2.2.4 Accessing the Library under Linux using PathScale compilers pathf90/pathcc Similar commands apply for the PathScale versions of ACML. For example, pathf90 driver.f -L/opt/acml4.0.0/pathscale64/lib -lacml will compile driver.f and link it to the ACML using the 64-bit version. The commands pathcc -c -I/opt/acml4.0.0/pathscale64/include driver.c pathcc driver.o -L/opt/acml4.0.0/pathscale64/lib -lacml -lpathfortran will compile driver.c and link it to the 64-bit ACML. The switchChapter 2: General Information 7 -I/opt/acml4.0.0/pathscale64/include tells the C compiler to search directory /opt/acml4.0.0/pathscale64/include for the ACML C header ?le acml.h, which should be included by driver.c. Note that in the example we add the library -lpathfortran to the link command, so that the required PathScale compiler run-time library is found. If you have an SMP machine and want to take best advantage of it, link against the PathScale OpenMP version of ACML like this: pathf90 -mp driver.f -L/opt/acml4.0.0/pathscale64_mp/lib -lacml_mp pathf90 -mp driver.f -L/opt/acml4.0.0/pathscale32_mp/lib -lacml_mp Note that the directories and library names involved now include the su?x mp. The -mp ?ag is important - it tells pathf90 to link with the appropriate compiler OpenMP run-time library. Without it you might get an "unresolved symbol" message at link time. The commands pathcc -c -mp -I/opt/acml4.0.0/pathscale64_mp/include driver.c pathcc -mp driver.o -L/opt/acml4.0.0/pathscale64_mp/lib -lacml_mp -lpathfortran will compile driver.c and link it to the 64-bit ACML. Again, the -mp ?ag is important if you are linking to the PathScale OpenMP version of ACML. The C compiler is instructed to search the directory /opt/acml4.0.0/pathscale64 mp/include for the ACML C header ?le acml.h, which should be included by driver.c, by using the switch "-I/opt/acml4.0.0/pathscale64 mp/include". Note that in the example we add the library -lpathfortran to the link command, so that a required PathScale compiler run-time library is found. 2.2.5 Accessing the Library under Linux using the NAGWare f95 compiler Similar commands apply for the NAGware f95 versions of ACML. For example, f95 driver.f -L/opt/acml4.0.0/nag64/lib -lacml f95 -32 driver.f -L/opt/acml4.0.0/nag32/lib -lacml will compile driver.f and link it to the ACML using the 64-bit version and 32-bit version respectively. 2.2.6 Accessing the Library under Linux using the Intel ifort compiler Similar commands apply for the Intel ifort versions of ACML. For example, ifort driver.f -L/opt/acml4.0.0/ifort64/lib -lacml will compile driver.f and link it to the ACML using the 64-bit version. The commands gcc -c -I/opt/acml4.0.0/ifort64/include driver.c ifort -nofor-main driver.o -L/opt/acml4.0.0/ifort64/lib -lacml will compile driver.c and link it to the 64-bit ACML. The switchChapter 2: General Information 8 -I/opt/acml4.0.0/ifort64/include tells the C compiler to search directory /opt/acml4.0.0/ifort64/include for the ACML C header ?le acml.h, which should be included by driver.c. Note that in the example we link the C program using the ifort compiler with the -nofor-main switch, so that required ifort compiler run-time libraries are found. If you have an SMP machine and want to take best advantage of it, link against the ifort OpenMP version of ACML like this: ifort -openmp driver.f -L/opt/acml4.0.0/ifort64_mp/lib -lacml_mp ifort -openmp driver.f -L/opt/acml4.0.0/ifort32_mp/lib -lacml_mp Note that the directories and library names involved now include the su?x mp. The -openmp ?ag is important - it tells ifort to link with the appropriate compiler OpenMP run-time library. Without it you might get an "unresolved symbol" message at link time. 2.2.7 Accessing the Library under Linux using compilers other than GNU, PGI, PathScale, NAGWare or Intel It may be possible to link to some versions of ACML using compilers other than those already mentioned, if they are compatible with one of the other versions. If you do this, it may be necessary to link to the run-time library of the compiler used to build the ACML you link to, in order to satisfy run-time symbols. Since doing this is very compiler-speci?c, we give no further details here. 2.3 Accessing the Library (Microsoft Windows) 2.3.1 Accessing the Library under 32-bit Windows using GNU g77/gcc Under Microsoft Windows R , for the g77/gcc version of ACML it is assumed that you have the Cygwin UNIX-like tools installed (see http://www.cygwin.com/), including the g77/gcc compiler and associated tools. Assuming you have installed the ACML in the default place, then in a DOS command prompt window, the command g77 driver.f "c:\Program Files\AMD\acml4.0.0\gnu32\lib\libacml.a" can be used to link the application program driver.f to the static library version of the ACML. The g77 version of the ACML Library is supplied in both static and shareable versions, libacml.a and libacml.dll, respectively. The command given above links to the static version of the library, libacml.a. To link to the DLL version, the command g77 driver.f "c:\Program Files\AMD\acml4.0.0\gnu32\lib\libacml.dll" can be used. Notice that if the application program has been linked to the DLL version of the ACML Library, then before running the program, the environment variable PATH must have been set to include the location of the DLL, for example by the DOS command: PATH="c:\Program Files\AMD\acml4.0.0\gnu32\lib";%PATH% where it was assumed that libacml.dll was installed in the directory "c:\Program Files\AMD\acml4.0.0\gnu32\lib". Alternatively, the PATH environment variable may be set in the system category of the Windows control panel. The commandChapter 2: General Information 9 gcc "-Ic:\Program Files\AMD\acml4.0.0\gnu32\include" driver.c "c:\Program Files\AMD\acml4.0.0\gnu32\lib\libacml.a" -lg2c will compile driver.c and link it to the 32-bit g77/gcc version of ACML. The switch "- Ic:\Program Files\AMD\acml4.0.0\gnu32\include" tells the gcc compiler to search directory "c:\Program Files\AMD\acml4.0.0\gnu32\include" for the ACML C header ?le acml.h, which should be included by driver.c. Note that it is necessary to add the compiler run-time library -lg2c when linking the program. 2.3.2 Accessing the Library under 32-bit Windows using PGI compilers pgf77/pgf90/Microsoft C To use the 32-bit Windows PGI version of ACML, use a command like pgf77 -Munix driver.f "c:\Program Files\AMD\acml4.0.0\pgi32\lib\libacml_dll.lib" where libacml dll.lib is the import library for the ACML DLL. Note that it is important to use the compiler switch -Munix in order to tell the compiler to use the same calling convention as was used to build ACML. In the example above we are linking with the single-processor PGI version of ACML. If you have an SMP machine and want to take best advantage of it, link against the PGI OpenMP version of ACML like this: pgf77 -Munix -mp driver.f "c:\Program Files\AMD\acml4.0.0\pgi32\lib\libacml_mp_dll.lib" Note that the directories and library names involved now include the su?x mp. For the OpenMP version of ACML, if you link to the static library libacml mp.lib rather than the DLL import library libacml mp dll.lib, you will need to use the PGI compiler ?ag -mp in order to tell the compiler to link with the appropriate compiler OpenMP run-time library. Without it you might get an "unresolved symbol" message at link time. This should not be necessary when linking to the ACML DLL because the DLL itself knows that it depends on the run-time library; but using the -mp ?ag in any case will do no harm. To compile and link a C program using the Microsoft C command line compiler, cl, the commands cl "-Ic:\Program Files\AMD\acml4.0.0\pgi32\include" /MD driver.c "c:\Program Files\AMD\acml4.0.0\pgi32\lib\libacml_dll.lib" cl "-Ic:\Program Files\AMD\acml4.0.0\pgi32_mp\include" /MD driver.c "c:\Program Files\AMD\acml4.0.0\pgi32_mp\lib\libacml_mp_dll.lib" will link against the single-threaded DLL and multi-threaded versions of ACML respectively.Chapter 2: General Information 10 2.3.3 Accessing the Library under 32-bit Windows using Microsoft C or Intel Fortran To use the 32-bit Windows MSC/Intel Fortran version of ACML, use a command like ifort /threads /libs:dll driver.f "c:\Program Files\AMD\acml4.0.0\ifort32\lib\libacml_dll.lib" where libacml dll.lib is the import library for the ACML DLL. In the example above we are linking with the single-processor ifort version of ACML. If you have an SMP machine and want to take best advantage of it, link against the ifort OpenMP version of ACML like this: ifort /libs:dll -Qopenmp driver.f c:\acml4.0.0\ifort32_mp\lib\libacml_mp_dll.lib Note that the directories and library names involved now include the su?x mp. For the OpenMP version of ACML, if you link to the static library libacml mp.lib rather than the DLL import library libacml mp dll.lib, you will need to use the ifort compiler ?ag -Qopenmp in order to tell the compiler to link with the appropriate compiler OpenMP run-time library. Without it you might get an "unresolved symbol" message at link time. This should not be necessary when linking to the ACML DLL because the DLL itself knows that it depends on the run-time library; but using the -Qopenmp ?ag in any case will do no harm. To compile and link a C program using the Microsoft C command line compiler, cl, the commands cl "-Ic:\Program Files\AMD\acml4.0.0\ifort32\include" /MD driver.c "c:\Program Files\AMD\acml4.0.0\ifort32\lib\libacml_dll.lib" cl "-Ic:\Program Files\AMD\acml4.0.0\ifort32_mp\include" /MD driver.c "c:\Program Files\AMD\acml4.0.0\ifort32_mp\lib\libacml_mp_dll.lib" will link against the single-threaded DLL and multi-threaded versions of ACML respectively. ACML can also be linked from inside a development environment such as Microsoft Visual Studio or Visual Studio.NET. Again, it is important to get compilation options correct. The directory acml4.0.0\ifort32\examples\Projects contains a few sample Visual Studio project directories showing how this can be done. Note that in both examples above we linked to a DLL version of ACML, and so before running the resulting programs the environment variable PATH must be set to include the location of the DLL. For example, assuming that libacml dll.dll was installed in "c:\Program Files\AMD\acml4.0.0\ifort32\lib", PATH may be set by, for example, the DOS command PATH="c:\Program Files\AMD\acml4.0.0\ifort32\lib";%PATH% Alternatively, the PATH environment variable may be set in the system category of the Windows control panel. ACML also comes as a static (non-DLL) library, named libacml.lib, in the same directory as the DLL. If you link to the static library instead of the DLL import library then there is no need to set the PATH.Chapter 2: General Information 11 2.3.4 Accessing the Library under 32-bit Windows using the Compaq Visual Fortran compiler The win32 Intel Fortran variant of ACML can be used with the Compaq Visual Fortran compiler as follows: f90 /iface:cref,nomixed_str_len_arg /threads /libs:dll driver.f "c:\Program Files\AMD\acml4.0.0\ifort32\lib\libacml_dll.lib" where f90 is the Compaq Visual Fortran command line compiler and libacml dll.lib is the import library for the ACML DLL. The switch /iface:cref,nomixed str len arg used on the f90 compiler command line is important - it tells the compiler to use a calling convention equivalent to the default Intel Fortran calling convention, rather than the default cvf stdcall calling convention. If you forget to use this switch your program is likely to crash on execution. 2.3.5 Accessing the Library under 32-bit Windows using the Salford FTN95 compiler The win32 Intel Fortran variant of ACML can be used with the Salford ftn95 compiler as follows: ftn95 driver.f The resulting object ?le can be linked using the Salford linker, slink, for example like this: slink driver.obj install_dir\libacml_dll.dll where install dir is the location of the DLL. The full pathname of install dir should be speci?ed to the DLL and should be enclosed within quotes if it contains spaces. It is worth emphasising that the linker should link directly against the DLL itself, not the libacml dll.lib import library. 2.3.6 Accessing the Library under 64-bit Windows using PGI compilers pgf77/pgf90/pgcc Under 64-bit versions of Windows, ACML 4.0.0 comes as a static (.LIB) library or a DLL. To link with the 64-bit Windows DLL library PGI version of ACML, in a DOS command prompt use a command like pgf77 -Mdll driver.f c:/acml4.0.0/win64/lib/libacml_dll.lib where libacml dll.lib is the import library for the DLL. In the example above we are linking with the single-processor WIN64 version of ACML. If you have an SMP machine and want to take best advantage of it, link against the WIN64 OpenMP version of ACML like this: pgf77 -Mdll -mp driver.f c:/acml4.0.0/win64_mp/lib/libacml_mp_dll.lib Note that the directories and library names involved now include the su?x mp. For the OpenMP version of ACML, if you link to the static library libacml mp.lib rather than the DLL import library libacml mp dll.lib, you will need to use the PGI compiler ?ag -mp in order to tell the compiler to link with the appropriate compiler OpenMP run-time library. Without it you might get an "unresolved symbol" message at link time. This should not be necessary when linking to the ACML DLL because the DLL itself knows that it depends on the run-time library; but using the -mp ?ag in any case will do no harm.Chapter 2: General Information 12 Note that the performance of OpenMP code produced with the PGI WIN64 compilers depends on environment variables named MP BIND and MP SPIN, which control how multiple threads behave (see PGI compiler documentation for discussion of these variables). For ACML, empirical experiments show that higher values of MP SPIN than the default are likely to give better performance. We recommend that users set MP BIND=yes and MP SPIN=100000000. Under WIN64, to compile and link a C program, the commands pgcc -Mdll driver.c -Ic:/acml4.0.0/win64/include c:/acml4.0.0/win64/lib/libacml_dll.lib pgcc -Mdll -mp driver.c -Ic:/acml4.0.0/win64_mp/include c:/acml4.0.0/win64_mp/lib/libacml_mp_dll.lib will link against the single-threaded DLL and multi-threaded versions of ACML respectively. To use the Microsoft C command line compiler, cl, use commands like this: cl driver.c -Ic:/acml4.0.0/win64/include c:/acml4.0.0/win64/lib/libacml_dll.lib cl driver.c -Ic:/acml4.0.0/win64_mp/include c:/acml4.0.0/win64_mp/lib/libacml_mp_dll.lib for single- and multi-threaded ACML variants respectively. 2.3.7 Accessing the Library under 64-bit Windows using Microsoft C or Intel Fortran Under 64-bit versions of Windows, ACML 4.0.0 comes as a static (.LIB) library or a DLL. To link with the 64-bit Windows DLL library Intel Fortran version of ACML, in a DOS command prompt use a command like ifort /libs:dll driver.f c:\acml4.0.0\ifort64\lib\libacml_dll.lib where libacml dll.lib is the import library for the DLL. In the example above we are linking with the single-processor ifort version of ACML. If you have an SMP machine and want to take best advantage of it, link against the ifort OpenMP version of ACML like this: ifort /libs:dll -Qopenmp driver.f c:\acml4.0.0\win64_mp\lib\libacml_mp_dll.lib Note that the directories and library names involved now include the su?x mp. For the OpenMP version of ACML, if you link to the static library libacml mp.lib rather than the DLL import library libacml mp dll.lib, you will need to use the ifort compiler ?ag -Qopenmp in order to tell the compiler to link with the appropriate compiler OpenMP run-time library. Without it you might get an "unresolved symbol" message at link time. This should not be necessary when linking to the ACML DLL because the DLL itself knows that it depends on the run-time library; but using the -Qopenmp ?ag in any case will do no harm. Under WIN64, to compile and link a C program using the Microsoft C command line compiler, cl, the commandsChapter 2: General Information 13 cl driver.c -Ic:/acml4.0.0/ifort64/include c:/acml4.0.0/ifort64/lib/libacml_dll.lib cl driver.c -Ic:/acml4.0.0/ifort64_mp/include c:/acml4.0.0/ifort64_mp/lib/libacml_mp_dll.lib will link against the single-threaded DLL and multi-threaded versions of ACML respectively. 2.4 Accessing the Library (Solaris) 2.4.1 Accessing the Library under Solaris If the Solaris 64-bit f95 version of ACML was installed in the default directory, /opt/acml4.0.0/sun64, then the command: f95 -xarch=amd64 driver.f -L/opt/acml4.0.0/sun64/lib -lacml can be used to compile the program driver.f and link it to the ACML. The ACML Library is supplied in both static and shareable versions, libacml.a and libacml.so, respectively. By default, the commands given above will link to the shareable version of the library, libacml.so, if that exists in the directory speci?ed. Linking with the static library can be forced either by using the compiler ?ag -Bstatic, e.g. f95 -xarch=amd64 driver.f -L/opt/acml4.0.0/sun64/lib -Bstatic -lacml or by inserting the name of the static library explicitly in the command line, e.g. f95 -xarch=amd64 driver.f /opt/acml4.0.0/sun64/lib/libacml.a Notice that if the application program has been linked to the shareable ACML Library, then before running the program, the environment variable LD_LIBRARY_PATH must be set, for example, by the C-shell command: setenv LD_LIBRARY_PATH /opt/acml4.0.0/sun64/lib where it is assumed that libacml.so was installed in the directory /opt/acml4.0.0/sun64/lib (see the man page for ld(1) for more information about LD_LIBRARY_PATH.). The command f95 -xarch=sse2 driver.f -L/opt/acml4.0.0/sun32/lib -lacml will compile and link a 32-bit program with a 32-bit ACML. To compile and link a 64-bit C program with a 64-bit ACML, invoke cc -xarch=amd64 -I/opt/acml4.0.0/sun64/include driver.c -L/opt/acml4.0.0/sun64/lib -lacml -lfsu -lsunmath -lm The switch "-I/opt/acml4.0.0/sun64/include" tells the compiler to search the directory /opt/acml4.0.0/sun64/include for the ACML C header ?le acml.h, which should be included by driver.c. Note that it is necessary to add the Sun compiler run-time libraries -lfsu -lsunmath -lm when linking the program. If you have an SMP machine and want to take best advantage of it, link against the Solaris OpenMP version of ACML like this: f95 -openmp -xarch=amd64 driver.f -L/opt/acml4.0.0/sun64_mp/lib -lacml_mp f95 -openmp -xarch=sse2 driver.f -L/opt/acml4.0.0/sun32_mp/lib -lacml_mp Note that the directories and library names involved now include the su?x mp. The -openmp ?ag is important - it tells f95 to link with the appropriate compiler OpenMP run-time library. Without it you might get an "unresolved symbol" message at link time. The commandChapter 2: General Information 14 cc -openmp -xarch=amd64 -I/opt/acml4.0.0/sun64/include driver.c -L/opt/acml4.0.0/sun64/lib -lacml_mp -lfsu -lsunmath -lm -lmtsk will compile driver.c and link it to the 64-bit ACML. Again, the -openmp ?ag is important if you are linking to the OpenMP version of ACML. The C compiler is instructed to search the directory /opt/acml4.0.0/sun64 mp/include for the ACML C header ?le acml.h, which should be included by driver.c, by using the switch "-I/opt/acml4.0.0/sun64 mp/include". Note that in the example we add the libraries -lfsu -lsunmath -lm -lmtsk to the link command, so that required compiler run-time libraries are found. 2.5 ACML FORTRAN and C interfaces All routines in ACML come with both FORTRAN and C interfaces. The FORTRAN interfaces typically follow the relevant standard (e.g. LAPACK, BLAS). Here we document how a C programmer should call ACML routines. In C code that uses ACML routines, be sure to include the header ?le , which contains function prototypes for all ACML C interfaces. The header ?le also contains C prototypes for FORTRAN interfaces, thus the C programmer could call the FORTRAN interfaces from C, though there is little reason to do so. C interfaces to ACML routines di?er from FORTRAN interfaces in the following major respects: • The FORTRAN interface names are appended by an underscore (except for the Windows 32-bit and 64-bit Microsoft C/Intel Fortran version of ACML, where FORTRAN interface names are distinguished from C by being upper case rather than lower case - this is the default for the Intel Fortran compiler) • The C interfaces contain no workspace arguments; all workspace memory is allocated internally. • Scalar input arguments are passed by value in C interfaces. FORTRAN interfaces pass all arguments (except for character string length arguments that are normally hidden from FORTRAN programmers) by reference. • Most arguments that are passed as character string pointers to FORTRAN interfaces are passed by value as single characters to C interfaces. The character string length arguments of FORTRAN interfaces are not required in the C interfaces. • Unlike FORTRAN, C has no native complex data type. ACML C routines which operate on complex data use the types complex and doublecomplex de?ned in for single and double precision computations respectively. Some of the programs in the ACML examples directory (see Section 2.9 [Examples], page 17) make use of these types. It is important to note that in both the FORTRAN and C interfaces, 2-dimensional arrays are assumed to be stored in column-major order. e.g. the matrix A =  1.0 2.0 3.0 4.0  would be stored in memory as 1.0, 3.0, 2.0, 4.0. This storage order corresponds to a FORTRAN-style 2-D array declaration A(2,2), but not to an array declared as a[2][2] in C which would be stored in row-major order as 1.0, 2.0, 3.0, 4.0.Chapter 2: General Information 15 As an example, compare the FORTRAN and C interfaces of LAPACK routine dsytrf as implemented in ACML. FORTRAN: void dsytrf_(char *uplo, int *n, double *a, int *lda, int *ipiv, double *work, int *lwork, int *info, int uplo_len); C: void dsytrf(char uplo, int n, double *a, int lda, int *ipiv, int *info); C code calling both the above variants might look like this: double *a; int *ipiv; double *work; int n, lda, lwork, info; /* Assume that all arrays and variables are allocated and initialized as required by dsytrf. */ /* Call the FORTRAN version of dsytrf. The first argument is a character string, and the last argument is the length of that string. The input scalar arguments n, lda and lwork, as well as the output scalar argument info, are all passed by reference. */ dsytrf_("Upper", &n, a, &lda, ipiv, work, &lwork, &info, 5); /* Call the C version of dsytrf. The first argument is a character, workspace is not required, and input scalar arguments n and lda are passed by value. Output scalar argument info is passed by reference. */ dsytrf(’U’, n, a, lda, ipiv, &info); 2.6 ACML variants using 64-bit integer (INTEGER*8) arguments Where compilers support, through the use of switches, the automatic promotion of regular INTEGER (32-bit) arguments to INTEGER*8 (64-bit) arguments, ACML variants exist to use this facility. This means that if you have a 64-bit Fortran program using INTEGER*8 variables, or a 64-bit C program using 8-byte long variables, there is an ACML version that you can use. This applies to 64-bit ACML versions built with PGI, PathScale, and gfortran compilers. The INTEGER*8 versions of these libraries are distinguished from the usual versions by having the string “ int64” as part of the name of the directory under which ACML is installed. Thus, for example, if the regular PGI 64-bit library is in a directory named pgi64, then the INTEGER*8 version will be installed in directory pgi64 int64. For these ACML variants, all ACML documentation that mentions arguments of Fortran type INTEGER or C type int should be read as INTEGER*8 or long respectively. It is important to ensure that if you have INTEGER*8 variables in your code, you link to the int64 variant, and not otherwise. Unexpected program crashes are likely to occur if you link to the wrong version.Chapter 2: General Information 16 2.7 Library Version and Build Information This document is applicable to version 4.0.0 of ACML. The utility routine acmlversion can be called to obtain the major, minor and patch version numbers of the installed ACML. This routine returns three integers; the major, minor and patch version numbers, respectively. The utility routine acmlinfo can be called to obtain information on the compiler used to build ACML, the version of the compiler, and the options used for building the Library. This subroutine takes no arguments and prints the information to the current standard output. FORTRAN speci?cations: ACMLVERSION (MAJOR, MINOR, PATCH) [SUBROUTINE] MAJOR, MINOR, PATCH [INTEGER] ACMLINFO () [SUBROUTINE] C speci?cations: void acmlversion (int *major, int *minor, int *patch); [function] void acmlinfo (void); [function] 2.8 Library Documentation The /Doc subdirectory of the top ACML installation directory, (e.g. /opt/acml4.0.0/Doc under Linux, or c:\Program Files\AMD\acml4.0.0\Doc under Windows), should contain this document in the following formats: • Printed Manual / PDF format – acml.pdf • Info Pages – acml.info (Linux only) • Html – html/index.html • Plain text – acml.txt Under Linux the info ?le can be read using info after updating the environment variable INFOPATH to include the doc subdirectory of the ACML installation directory, e.g. % setenv INFOPATH ${INFOPATH}:/opt/acml4.0.0/Doc % info acml or simply by using the full name of the ?le: % info /opt/acml4.0.0/Doc/acml.infoChapter 2: General Information 17 2.9 Example programs calling ACML The /examples subdirectory of the top ACML installation directory (for example, possible default locations are /opt/acml4.0.0/gnu64/examples under Linux, or, under windows, c:\Program Files\AMD\acml4.0.0\gnu32\examples), contains example programs showing how to call the ACML, along with a GNUmake?le to build and run them. Examples of calling both FORTRAN and C interfaces are included. They may be used as an ACML installation test. Depending on where your copy of the ACML is installed, and which compiler and ?ags you wish to use, it may be necessary to modify some variables in the GNUmake?le before using it. The 32-bit Windows versions of ACML assume that you have the Cygwin UNIX-like tools installed, and can use the make command that comes with them to build the examples. For the 64-bit Windows version of ACML, it is not necessary to have the Cygwin tools. The examples directory contains a bat script, acmlexample.bat, which can be used to run one of the example programs. Another bat script, acmlallexamples.bat, builds and runs all the examples in the directory. Alternatively, if you do have the Cygwin tools installed, you can use the GNUmake?le to build the examples. If you need more example programs showing how to call LAPACK routines from Fortran, we refer you to this web page: http://www.nag.com/lapack/ Here you will ?nd examples for all double precision LAPACK driver routines, and all of these should work when linked with ACML. Note that as well as the example programs themselves, it is necessary to download and compile a small amount of utility code used by the programs. See the web page for detailed instructions. 2.10 Example ACML programs demonstrating performance The /examples/performance subdirectory of the top ACML installation directory (for example, possible default locations are /opt/acml4.0.0/gnu64/examples/performance under Linux, or c:\Program Files\AMD\acml4.0.0\gnu32\examples\performance under windows) contains several timing programs designed to show the performance of ACML when running on your machine. Again, a GNUmake?le may be used to build and run them. Depending on where your copy of the ACML is installed, and which compiler and ?ags you wish to use, it may be necessary to modify some variables in the GNUmake?le before using it. The 32- and 64-bit Windows versions of ACML assume that you have the Cygwin UNIXlike tools installed, and can use the make command that comes with them to build the examples. In addition, the GNUmake?le uses the gnuplot plotting program to display graphs of the timing results. If you do not have gnuplot installed, the timing programs will still run and show their results, but you will see no graph plots. Under linux, gnuplot may come with your linux distribution, but you may need to explicitly ask for it to be installed. Note that version 4.0 or later of gnuplot is required. The gnuplot program is also available for Windows machines. See http://www.gnuplot.info for more information.Chapter 2: General Information 18 If you are on an SMP (multiprocessor) machine and have installed an OpenMP version of the ACML, then in the examples/performance directory a command such as % make OMP_NUM_THREADS=5 will run the timing programs on P processors, where P = 1, 2, 4, 5; i.e., P equals an integer power of 2 and also equals OMP NUM THREADS if this value is not a power of 2. The results for a particular routine are concatenated into one ?le. gnuplot then shows on one graph for each routine the results of varying the number of processors for that routine. Setting OMP NUM THREADS in this way is not useful if you are not on an SMP machine or are not using an OpenMP version of ACML. Neither is it useful to set OMP NUM THREADS to a value higher than the number of processors (or processor cores) on your machine. A way to ?nd the number of processors (or cores) under linux is to examine the special ?le /proc/cpuinfo which has an entry for every core. Not all routines in ACML are SMP parallelized, so in this context the OMP NUM THREADS setting only applies to those examples, including time c?t2d.f, time dgemm.f and time dgetrf.f, which are for parallelized routines. The other timing programs run on one thread regardless of the setting of OMP NUM THREADS. In all cases, timing graphs can be viewed without regenerating timing results by typing the command % make plots Note that all results generated by timing programs will vary depending on the load on your machine at run time.Chapter 3: BLAS: Basic Linear Algebra Subprograms 19 3 BLAS: Basic Linear Algebra Subprograms The BLAS are a set of well de?ned basic linear algebra operations ([1], [2], [3]). These operations are subdivided into three groups: • Level 1: operations acting on vectors only (e.g. dot product) • Level 2: matrix-vector operations (e.g. matrix-vector multiplication) • Level 3: matrix-matrix operations (e.g. matrix-matrix multiplication) E?cient machine-speci?c implementations of the BLAS are available for many modern high-performance computers. The implementation of higher level linear algebra algorithms on these systems depends critically on the use of the BLAS as building blocks. AMD provides, as part of the ACML, an implementation of the BLAS optimized for performance on AMD64 processors. For any information relating to the BLAS please refer to the BLAS FAQ: http://www.netlib.org/blas/faq.html ACML also includes interfaces to the extensions to Level 1 BLAS known as the sparse BLAS. These routines perform operations on a sparse vector x which is stored in compressed form and a vector y in full storage form. See reference [4] for more information.Chapter 4: LAPACK: Package of Linear Algebra Subroutines 20 4 LAPACK: Package of Linear Algebra Subroutines 4.1 Introduction to LAPACK LAPACK ([5]) is a library of FORTRAN 77 subroutines for solving commonly occurring problems in numerical linear algebra. LAPACK components can solve systems of linear equations, linear least squares problems, eigenvalue problems and singular value problems. Dense and banded matrices are provided for, but not general sparse matrices. In all areas, similar functionality is provided for real and complex matrices. LAPACK routines are written so that as much as possible of the computations is performed by calls to the BLAS. The e?ciency of LAPACK routines depends, in large part, on the e?ciency of the BLAS being called. Block algorithms are employed wherever possible to maximize the use of calls to level 3 BLAS, which generally run faster than lower level BLAS due to the high number of operations per memory access. The performance of some of the LAPACK routines has been further improved by reworking the computational algorithms. Some of the LAPACK routines contained in ACML are therefore based on code that is di?erent from the LAPACK sources available in the public domain. In all these cases the algorithmic and numerical properties of the original LAPACK routines have been strictly preserved. Furthermore, key LAPACK routines have been treated using OpenMP to take advantage of multiple processors when running on SMP machines. Your application will automatically bene?t when you link with the OpenMP versions of ACML. 4.2 Reference sources for LAPACK The LAPACK homepage can be accessed on the World Wide Web via the URL address: http://www.netlib.org/lapack/ The on-line version of the Lapack User’s Guide, Third Edition ([5]) is available from this homepage, or directly using the URL: http://www.netlib.org/lapack/lug/index.html The standard source code is available for download from netlib, with separate distributions for UNIX/Linux and Windows R installations: http://www.netlib.org/lapack/lapack.tgz http://www.netlib.org/lapack/lapack-pc.zip A list of known problems, bugs, and compiler errors for LAPACK, as well as an errata list for the LAPACK User’s Guide ([5]), is maintained on netlib http://www.netlib.org/lapack/release_notes A LAPACK FAQ (Frequently Asked Questions) ?le can also be accessed via the LAPACK homepage http://www.netlib.org/lapack/faq.htmlChapter 4: LAPACK: Package of Linear Algebra Subroutines 21 4.3 LAPACK block sizes, ILAENV and ILAENVSET As described in Section 6.2 of the LAPACK User’s Guide, block sizes and other parameters used by various LAPACK routines are returned by the LAPACK inquiry function ILAENV. In ACML, values returned by ILAENV have been chosen to achieve very good performance on a wide variety of hardware and problem sizes. In general it is unlikely that you will want or need to be concerned with these parameters. However, in some cases it may be that a default value returned by ILAENV is not optimal for your particular hardware and problem size. Following the advice in the LAPACK User’s Guide may enable you to choose a better value in some circumstances. For convenience, ACML includes a subroutine which allows you to override default values returned by ILAENV if you have superior knowledge. The routine is named ILAENVSET and has the following speci?cation. ILAENVSET (ISPEC,NAME,OPTS,N1,N2,N3,N4,NVALUE,INFO) [SUBROUTINE] INTEGER ISPEC [Input] On input: ISPEC speci?es the parameter to be set (see Section 6.2 of the LAPACK User’s Guide for details). CHARACTER*(*) NAME [Input] On input: NAME speci?es the name of the LAPACK subroutine for which the parameter is to be set. CHARACTER*(*) OPTS [Input] On input: OPTS is a character string of options to the subroutine. INTEGER N1, N2, N3, N4 [Input] On input: N1, N2, N3 and N4 are problem dimensions. A value of -1 means that the dimension is unused or irrelevant. INTEGER NVALUE [Input] On input: NVALUE is the value to be set for the parameter speci?ed by ISPEC. This value will be retrieved by any future call of ILAENV with similar arguments, including the call of ILAENV coming directly from the routine speci?ed by argument NAME. In most cases, but not all, the value set will apply irrespective of the values of arguments OPTS, N1, N2, N3 and N4. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. All arguments of ILAENVSET apart from the last two, NVALUE and INFO, are identical to the arguments of ILAENV. ILAENVSET should be called before you call the LAPACK routine in question. It should be noted that not all LAPACK routines make use of the ILAENV mechanism (because not all routines use blocked algorithms or require other tuning parameters). Calls of ILAENVSET with argument NAME set to the name of such a routine will fail with INFO=0. In addition, the ACML versions of some important routines that do use blocked algorithms, such as the QR factorization routine DGEQRF, bypass ILAENV because they make use of a di?erent tuning system which is independent of standard LAPACK. For all such routines,Chapter 4: LAPACK: Package of Linear Algebra Subroutines 22 ILAENVSET can still be called with no error exit, but calls will have no e?ect on performance of the routine. Below we give examples of how to call ILAENVSET in both FORTRAN and C. Example (FORTRAN code):   INTEGER ILO, IHI, INFO, N, NS CHARACTER COMPZ, JOB INTEGER ILAENV EXTERNAL ILAENV, ILAENVSET JOB = ’E’ COMPZ = ’I’ N = 512 ILO = 1 IHI = 512 C Check the default shift parameter (ISPEC=4) used by DHSEQR NS = ILAENV(4, ’DHSEQR’, JOB//COMPZ, N, ILO, IHI, -1) WRITE (*,*) ’Default NS = ’, NS C Set a new value 5 for the shift parameter CALL ILAENVSET(4, ’DHSEQR’, JOB//COMPZ, N, ILO, IHI, -1, 5, INFO) C Then check the shift parameter again NS = ILAENV(4, ’DHSEQR’, JOB//COMPZ, N, ILO, IHI, -1) WRITE (*,*) ’Revised NS = ’, NS END Example (C code):   #include #include int main(void) { int n=512, ilo=1, ihi=512, ns, info; char compz = ’I’, job = ’E’, opts[3]; opts[0] = job; opts[1] = compz; opts[2] = ’\0’; /* Check the default shift parameter (ISPEC=4) used by DHSEQR */ ns = ilaenv(4, "DHSEQR", opts, n, ilo, ihi, -1); printf("Default ns = %d\n", ns); /* Set a new value 5 for the shift parameter */ ilaenvset(4, "DHSEQR", opts, n, ilo, ihi, -1, 5, &info); /* Then check the shift parameter again */ ns = ilaenv(4, "DHSEQR", opts, n, ilo, ihi, -1); printf("Revised ns = %d\n", ns); return 0; } Chapter 4: LAPACK: Package of Linear Algebra Subroutines 23 4.4 IEEE exceptions and LAPACK Some LAPACK eigensystem routines (namely CHEEVR, DSTEVR, DSYEVR, SSTEVR, SSYEVR, ZHEEVR) are able to take advantage of a faster algorithm when the full eigenspectrum is requested on machines which conform to the IEEE-754 ?oating point standard [14]. Normal execution of the faster algorithm (implemented by LAPACK routines SSTEGR and DSTEGR, which are called by the routines mentioned above) may create NaNs and in?nities and hence may abort due to a ?oating point exception in environments which do not handle NaNs and in?nities in the IEEE standard default manner. This may depend upon the compiler ?ags used to compile and link the main program. The LAPACK routine ILAENV, called with ISPEC = 10 or 11, states whether or not NaNs or in?nities respectively will cause a trap. In ACML, by default ILAENV assumes that NaNs and in?nities cause traps, even if this reduces the performance of the eigensystem routines. This is because it is not possible in general to reliably check whether they do trap or not at run-time. The intention is to ensure that these routines always function correctly, irrespective of how the main program calling ACML is compiled. However, if your main program is compiled in such a way that NaNs and in?nities do not cause traps, the ACML-speci?c routine ILAENVSET (see Section 4.3 [ILAENVILAENVSET], page 21) may be used to override the default operative mode of ILAENV, and allow the xxxEVR routines to use the faster xSTEGR algorithm when calculating the full eigenspectrum. When used for this purpose, ILAENVSET should be called as follows: CALL ILAENVSET(10,’X’,’X’,0,0,0,0,1,INFO) CALL ILAENVSET(11,’X’,’X’,0,0,0,0,1,INFO) (or the C equivalent). It is important to note that if you use ILAENVSET in this way before calling an xxxEVR routine, but your program does trap on IEEE exceptions, then there is a chance that your program will terminate unexpectedly. You should consult the documentation for the compiler you are using to ?nd out whether there are compiler ?ags controlling this.Chapter 5: Fast Fourier Transforms (FFTs) 24 5 Fast Fourier Transforms (FFTs) 5.1 Introduction to FFTs There are two main types of Discrete Fourier Transform (DFT): • routines for the transformation of complex data: in the ACML, these routines have names beginning with ZFFT or CFFT, for double and single precision, respectively; • routines for the transformation of real to complex data and vice versa: in the ACML the names for the former begin with DZFFT or SCFFT, for double and single precision, respectively; the names for the latter begin with ZDFFT or CSFFT. The following subsections provide de?nitions of the DFT for complex and real data types, and some guidelines on the e?cient use of the ACML FFT routines. 5.1.1 Transform de?nitions and Storage for Complex Data The simplest transforms to describe are those performed on sequences of complex data. Such data are stored as arrays of type complex. The result of a complex FFT is also a complex sequence of the same length and, for the simple interfaces, is written back to the original array. Where multiple (m, say), same-length sequences (of length n) of complex data are to be transformed, the sequences are held in a single complex array; in the simple interfaces the array will be of length m * n containing m end-to-end sequences and the results of the m FFTs are returned in the original array. Expert interfaces are provided which give: greater ?exibility in the storage of the original data and results, user provided scaling, and whether results should be written to a separate array or not. The de?nition of a complex DFT used here is given by: x˜j = 1 v n nX-1 k=0 xk exp  ±i 2pjk n  for j = 0, 1, . . . , n - 1 where xk are the complex data to be transformed, ˜xj are the transformed data, and the sign of ± determines the direction of the transform: (-) for forward and (+) for backward. Note that, in this de?nition, both directional transforms have the same scaling and performing both consecutively recovers the original data; this is the prescribed scaling provided in the simple FFT interfaces, whereas, in the expert interfaces, the scaling factor must be supplied by the user. For the simple interfaces, a two dimensional array of complex data, with m rows and n columns is stored in the same order as a set of n sequences of length m (as described above). That is, column elements are stored contiguously and the ?rst element of the next column follows the last element of the current column. In the expert interfaces, column elements may be separated by a ?xed step length (increment) while row elements may be separated by a second increment; if the ?rst increment is 1 and the second increment is m then we have the same storage as in the simple interface. The de?nition of a complex 2D DFT used here is given by: x˜jp = 1 v m * n mX-1 l=0 nX-1 k=0 xkl exp  ±i 2pjk n  exp  ±i 2ppl m  for j = 0, 1, . . . , n - 1 and l = 0, 1, . . . , m - 1, where xkl are the complex data to be transformed, ˜xjp are the transformed data, and the sign of ± determines the direction of the transform.Chapter 5: Fast Fourier Transforms (FFTs) 25 5.1.2 Transform de?nitions and Storage for Real Data The DFT of a sequence of real data results in a special form of complex sequence known as a Hermitian sequence. The symmetries de?ning such a sequence mean that it can be fully represented by a set of n real values, where n is the length of the original real sequence. It is therefore conventional for the array containing the real sequence to be overwritten by such a representation of the transformed Hermitian sequence. If the original sequence is purely real valued, i.e. zj = xj , then the de?nition of the real DFT used here is given by: z˜j = aj + ibj = 1 v n nX-1 k=0 xk exp  -i 2pjk n  for j = 0, 1, . . . , n - 1 where xk are the real data to be transformed, ˜zj are the transformed complex data. In full complex representation, the Hermitian sequence would be a sequence of n complex values Z(i) for i = 0, 1, ..., n - 1, where Z(n - j) is the complex conjugate of Z(j) for j = 1, 2, ...,(n-1)/2; Z(0) is real valued; and, if n is even, Z(n/2) is real valued. In ACML, the representation of Hermitian sequences used on output from DZFFT routines and on input to ZDFFT routines is as follows: let X be an array of length N and with ?rst index 0, • X(i) contains the real part of Z(i) for i = 0, ..., N/2 • X(N - i) contains the imaginary part of Z(i) for i = 1, ...,(N - 1)/2 Also, given a Hermitian sequence, the discrete transform can be written as: xj = 1 v n ? ?a0 + 2 n/X2-1 k=1  ak cos  2pjk n  - bk sin  2pjk n  + an/2 ? ? where an/2 = 0 if n is odd, and ˜zk = ak + ibk is the Hermitian sequence to be transformed. Note that, in the above de?nitions, both transforms have the same (negative) sign in the exponent; performing both consecutively does not recover the original data. To recover original real data, or otherwise to perform an inverse transform on a set of Hermitian data, the Hermitian data must be conjugated prior to performing the transform (i.e. changing the sign of the stored imaginary parts). 5.1.3 E?ciency The e?ciency of the FFT is maximized by choosing the sequence length to be a power of 2. Good e?ciency can also be achieved when the sequence length has small prime factors, up to a factor 13; however, the time taken for an FFT increases as the size of the prime factor increases.Chapter 5: Fast Fourier Transforms (FFTs) 26 5.1.4 Default and Generated Plans For those FFT routines that can be initialized prior to computing the FFTs, the initialization can be performed in one of two ways. In either case, initialization involves the storing of the factorization of N, and the twiddle factors associated with this factorization, in the communication array COMM. The simpler way to initialize is by setting the argument MODE to zero. This means that a default plan, for the given input dimensions, is used to calculate the FFT. This has the advantage that the initialization phase is very quick and is generally a small fraction of the time required to perform the FFT computation. However, for some problem dimensions the default plan may not be optimal, especially where there is a mixture of prime factors. Under some circumstances, optimality of performance of an FFT computation may be crucial. For example, where a very large number of FFTs are to be performed on problems of a ?xed size (e.g. N remains the same), then it is best to initialize by setting the argument MODE to 100. This will time a number of plans (this number can be quite large when N has a signi?cant number of prime factors) and initialize using the plan with the best time. Using this form of initialization can, potentially, lead to signi?cant improvements in the performance of the FFT computation for the given dimensions. Where problem dimensions will not change over a number of runs of a program, the communication array could, for example, be written out to a ?le during an initialization run, and then read in from the same ?le on subsequent computation runs. This would be e?ective for problem dimensions that have a large number of possible plans (factor orderings and groupings) and therefore take a signi?cant amount of time to ?nd the optimal plan. Please consult the individual FFT routine documents to determine whether plan generation is enabled.Chapter 5: Fast Fourier Transforms (FFTs) 27 5.2 FFTs on Complex Sequences 5.2.1 FFT of a single sequence The routines documented here compute the discrete Fourier transform (DFT) of a sequence of complex numbers in either single or double precision arithmetic. The DFT is computed using a highly-e?cient FFT algorithm. There are two sets of interfaces available: simple drivers and expert drivers. The simple drivers perform in-place transforms on data held contiguously in memory using a ?xed scaling factor; these are simpler to use and are su?- cient for many problems. The expert drivers o?er greater ?exibility by including a number of additional arguments. These allow you to control: the scaling factor applied; whether the result should be output to a separate vector; and, the increments used in storing successive elements of both the input sequence and the result.Chapter 5: Fast Fourier Transforms (FFTs) 28 ZFFT1D Routine Documentation ZFFT1D (MODE,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZFFT1D. On input: • MODE=0 : only default initializations (speci?c to N) are performed; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT1D. • MODE=1 : a backward (reverse) transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT1D. • MODE=-2 : initializations and a forward transform are performed. • MODE=2 : initializations and a backward transform are performed. • MODE=100 : similar to MODE=0; only initializations are performed, but ?rst a plan is generated. This plan is chosen based on the fastest FFT computation for a subset of all possible plans. INTEGER N [Input] On input: N is the length of the complex sequence X COMPLEX*16 X(N) [Input/Output] On input: X contains the complex sequence of length N to be transformed. On output: X contains the transformed sequence. COMPLEX*16 COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL ZFFT1D(0,N,X,COMM,INFO) CALL ZFFT1D(-1,N,X,COMM,INFO) CALL ZFFT1D(-1,N,Y,COMM,INFO) DO 10 I = 1, N X(I) = X(I)*DCONJG(Y(I)) 10 CONTINUE CALL ZFFT1D(1,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 29 CFFT1D Routine Documentation CFFT1D (MODE,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by CFFT1D. On input: • MODE=0 : only default initializations (speci?c to N) are performed; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward transform is performed. Initializations are assumed to have been performed by a prior call to CFFT1D. • MODE=1 : a backward (reverse) transform is performed. Initializations are assumed to have been performed by a prior call to CFFT1D. • MODE=-2 : (default) initializations and a forward transform are performed. • MODE=2 : (default) initializations and a backward transform are performed. • MODE=100 : similar to MODE=0; only initializations are performed, but ?rst a plan is generated. This plan is chosen based on the fastest FFT computation for a subset of all possible plans. INTEGER N [Input] On input: N is the length of the complex sequence X COMPLEX X(N) [Input/Output] On input: X contains the complex sequence of length N to be transformed. On output: X contains the transformed sequence. COMPLEX COMM(5*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL CFFT1D(0,N,X,COMM,INFO) CALL CFFT1D(-1,N,X,COMM,INFO) CALL CFFT1D(-1,N,Y,COMM,INFO) DO 10 I = 1, N X(I) = X(I)*CONJG(Y(I)) 10 CONTINUE CALL CFFT1D(1,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 30 ZFFT1DX Routine Documentation ZFFT1DX (MODE,SCALE,INPL,N,X,INCX,Y,INCY,COMM, [SUBROUTINE] INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZFFT1DX. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT1DX. • MODE=1 : a backward (reverse) transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT1DX. • MODE=-2 : (default) initializations and a forward transform are performed. • MODE=2 : (default) initializations and a backward transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. DOUBLE PRECISION SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequence LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequence; otherwise the output sequence is returned in Y. INTEGER N [Input] On input: N is the number of elements to be transformed COMPLEX*16 X(1+(N-1)*INCX) [Input/Output] On input: X contains the complex sequence of length N to be transformed, with the ith element stored in X(1+(i-1)*INCX). On output: if INPL is .TRUE. then X contains the transformed sequence in the same locations as on input; otherwise X remains unchanged. INTEGER INCX [Input] On input: INCX is the increment used to store successive elements of a sequence in X. Constraint: INCX > 0. COMPLEX*16 Y(1+(N-1)*INCY) [Output] On output: if INPL is .FALSE. then Y contains the transformed sequence, with the ith element stored in Y(1+(i-1)*INCY); otherwise Y is not referenced.Chapter 5: Fast Fourier Transforms (FFTs) 31 INTEGER INCY [Input] On input: INCY is the increment used to store successive elements of a sequence in Y. If INPL is .TRUE. then INCY is not referenced. Constraint: INCY > 0. COMPLEX*16 COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Forward FFTs are performed unscaled and in-place on contiguous C vectors X and Y following initialization. Manipulations on C resultant Fourier coefficients are stored in X which is then C transformed back. C SCALE = 1.0D0 INPL = .TRUE. CALL ZFFT1DX(0,SCALE,INPL,N,X,1,DUM,1,COMM,INFO) CALL ZFFT1DX(-1,SCALE,INPL,N,X,1,DUM,1,COMM,INFO) CALL ZFFT1DX(-1,SCALE,INPL,N,Y,1,DUM,1,COMM,INFO) DO 10 I = 1, N X(I) = X(I)*DCONJG(Y(I))/DBLE(N) 10 CONTINUE CALL ZFFT1DX(1,SCALE,INPL,N,X,1,DUM,1,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 32 CFFT1DX Routine Documentation CFFT1DX (MODE,SCALE,INPL,N,X,INCX,Y,INCY,COMM, [SUBROUTINE] INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by CFFT1DX. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward transform is performed. Initializations are assumed to have been performed by a prior call to CFFT1DX. • MODE=1 : a backward (reverse) transform is performed. Initializations are assumed to have been performed by a prior call to CFFT1DX. • MODE=-2 : (default) initializations and a forward transform are performed. • MODE=2 : (default) initializations and a backward transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. REAL SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequence LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequence; otherwise the output sequence is returned in Y. INTEGER N [Input] On input: N is the number of elements to be transformed COMPLEX X(1+(N-1)*INCX) [Input/Output] On input: X contains the complex sequence of length N to be transformed, with the ith element stored in X(1+(i-1)*INCX). On output: if INPL is .TRUE. then X contains the transformed sequence in the same locations as on input; otherwise X remains unchanged. INTEGER INCX [Input] On input: INCX is the increment used to store successive elements of a sequence in X. Constraint: INCX > 0. COMPLEX Y(1+(N-1)*INCY) [Output] On output: if INPL is .FALSE. then Y contains the transformed sequence, with the ith element stored in Y(1+(i-1)*INCY); otherwise Y is not referenced.Chapter 5: Fast Fourier Transforms (FFTs) 33 INTEGER INCY [Input] On input: INCY is the increment used to store successive elements of a sequence in Y. If INPL is .TRUE. then INCY is not referenced. Constraint: INCY > 0. COMPLEX COMM(5*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Forward FFTs are performed unscaled and in-place on contiguous C vectors X and Y following initialization. Manipulations on C resultant Fourier coefficients are stored in X which is then C transformed back. C SCALE = 1.0 INPL = .TRUE. CALL CFFT1DX(0,SCALE,INPL,N,X,1,DUM,1,COMM,INFO) CALL CFFT1DX(-1,SCALE,INPL,N,X,1,DUM,1,COMM,INFO) CALL CFFT1DX(-1,SCALE,INPL,N,Y,1,DUM,1,COMM,INFO) DO 10 I = 1, N X(I) = X(I)*CONJG(Y(I))/REAL(N) 10 CONTINUE CALL CFFT1DX(1,SCALE,INPL,N,X,1,DUM,1,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 34 5.2.2 FFT of multiple complex sequences The routines documented here compute the discrete Fourier transforms (DFTs) of a number of sequences of complex numbers in either single or double precision arithmetic. The sequences must all have the same length. The DFTs are computed using a highly-e?cient FFT algorithm. There are two sets of interfaces available: simple drivers and expert drivers. The simple drivers perform in-place transforms on data held contiguously in memory using a ?xed scaling factor; these are simpler to use and are su?cient for many problems. The expert drivers o?er greater ?exibility by including a number of additional arguments. These allow you to control: the scaling factor applied; whether the result should be output to a separate vector; the increments used in storing successive elements of a given sequence (for both input and output sequences); and the increments used in storing corresponding elements in successive sequences (for both input and output).Chapter 5: Fast Fourier Transforms (FFTs) 35 ZFFT1M Routine Documentation ZFFT1M (MODE,M,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZFFT1M. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : forward transforms are performed. Initializations are assumed to have been performed by a prior call to ZFFT1M. • MODE=1 : backward (reverse) transforms are performed. Initializations are assumed to have been performed by a prior call to ZFFT1M. • MODE=-2 : (default) initializations and forward transforms are performed. • MODE=2 : (default) initializations and backward transforms are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. INTEGER M [Input] On input: M is the number of sequences to be transformed. INTEGER N [Input] On input: N is the length of the complex sequences in X COMPLEX*16 X(N*M) [Input/Output] On input: X contains the M complex sequences of length N to be transformed. Element i of sequence j is stored in location i + (j - 1) * N of X. On output: X contains the transformed sequences. COMPLEX*16 COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 5: Fast Fourier Transforms (FFTs) 36 Example:   CALL ZFFT1M(0,1,N,X,COMM,INFO) CALL ZFFT1M(-1,2,N,X,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*DCONJG(X(I,2)) X(I,2) = DCMPLX(0.0D0,1.0D0)*X(I,2) 10 CONTINUE CALL ZFFT1M(1,2,N,X(1,2),COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 37 CFFT1M Routine Documentation CFFT1M (MODE,M,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by CFFT1M. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : forward transforms are performed. Initializations are assumed to have been performed by a prior call to CFFT1M. • MODE=1 : backward (reverse) transforms are performed. Initializations are assumed to have been performed by a prior call to CFFT1M. • MODE=-2 : (default) initializations and forward transforms are performed. • MODE=2 : (default) initializations and backward transforms are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. INTEGER M [Input] On input: M is the number of sequences to be transformed. INTEGER N [Input] On input: N is the length of the complex sequences in X COMPLEX X(N*M) [Input/Output] On input: X contains the M complex sequences of length N to be transformed. Element i of sequence j is stored in location i + (j - 1) * N of X. On output: X contains the transformed sequences. COMPLEX COMM(5*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 5: Fast Fourier Transforms (FFTs) 38 Example:   CALL CFFT1M(0,1,N,X,COMM,INFO) CALL CFFT1M(-1,2,N,X,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*CONJG(X(I,2)) X(I,2) = CMPLX(0.0D0,1.0D0)*X(I,2) 10 CONTINUE CALL CFFT1M(1,2,N,X(1,2),COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 39 ZFFT1MX Routine Documentation ZFFT1MX (MODE,SCALE,INPL,NSEQ,N,X,INCX1,INCX2, [SUBROUTINE] Y,INCY1,INCY2,COMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZFFT1MX. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT1MX. • MODE=1 : a backward (reverse) transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT1MX. • MODE=-2 : (default) initializations and a forward transform are performed. • MODE=2 : (default) initializations and a backward transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. DOUBLE PRECISION SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER NSEQ [Input] On input: NSEQ is the number of sequences to be transformed INTEGER N [Input] On input: N is the number of elements in each sequence to be transformed COMPLEX*16 X(1+(N-1)*INCX1+(NSEQ-1)*INCX2) [Input/Output] On input: X contains the NSEQ complex sequences of length N to be transformed; the ith element of sequence j is stored in X(1+(i-1)*INCX1+(j- 1)*INCX2). On output: if INPL is .TRUE. then X contains the transformed sequences in the same locations as on input; otherwise X remains unchanged. INTEGER INCX1 [Input] On input: INCX1 is the increment used to store successive elements of a given sequence in X (INCX1=1 for contiguous data). Constraint: INCX1 > 0.Chapter 5: Fast Fourier Transforms (FFTs) 40 INTEGER INCX2 [Input] On input: INCX2 is the increment used to store corresponding elements of successive sequences in X (INCX2=N for contiguous data). Constraint: INCX2 > 0. COMPLEX*16 Y(1+(N-1)*INCY1+(NSEQ-1)*INCY2) [Output] On output: if INPL is .FALSE. then Y contains the transformed sequences with the ith element of sequence j stored in Y(1+(i-1)*INCY1+(j-1)*INCY2); otherwise Y is not referenced. INTEGER INCY1 [Input] On input: INCY1 is the increment used to store successive elements of a given sequence in Y. If INPL is .TRUE. then INCY1 is not referenced. Constraint: INCY1 > 0. INTEGER INCY2 [Input] On input: INCY2 is the increment used to store corresponding elements of successive sequences in Y (INCY2=N for contiguous data). If INPL is .TRUE. then INCY2 is not referenced. Constraint: INCY2 > 0. COMPLEX*16 COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Forward FFTs are performed unscaled and in-place on two C contiguous vectors stored in the first two columns of X. C Manipulations are stored in 2nd and 3rd columns of X which are C then transformed back. C COMPLEX *16 X(N,3) SCALE = 1.0D0 INPL = .TRUE. CALL ZFFT1MX(0,SCALE,INPL,2,N,X,1,N,DUM,1,N,COMM,INFO) CALL ZFFT1MX(-1,SCALE,INPL,2,N,X,1,N,DUM,1,N,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*DCONJG(X(I,2))/DBLE(N) X(I,2) = DCMPLX(0.0D0,1.0D0)*X(I,2)/DBLE(N) 10 CONTINUE CALL ZFFT1MX(1,SCALE,INPL,2,N,X(1,2),1,N,DUM,1,N,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 41 CFFT1MX Routine Documentation CFFT1MX (MODE,SCALE,INPL,NSEQ,N,X,INCX1,INCX2, [SUBROUTINE] Y,INCY1,INCY2,COMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by CFFT1MX. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward transform is performed. Initializations are assumed to have been performed by a prior call to CFFT1MX. • MODE=1 : a backward (reverse) transform is performed. Initializations are assumed to have been performed by a prior call to CFFT1MX. • MODE=-2 : (default) initializations and a forward transform are performed. • MODE=2 : (default) initializations and a backward transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. REAL SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER NSEQ [Input] On input: NSEQ is the number of sequences to be transformed INTEGER N [Input] On input: N is the number of elements in each sequence to be transformed COMPLEX X(1+(N-1)*INCX1+(NSEQ-1)*INCX2) [Input/Output] On input: X contains the NSEQ complex sequences of length N to be transformed; the ith element of sequence j is stored in X(1+(i-1)*INCX1+(j- 1)*INCX2). On output: if INPL is .TRUE. then X contains the transformed sequences in the same locations as on input; otherwise X remains unchanged. INTEGER INCX1 [Input] On input: INCX1 is the increment used to store successive elements of a given sequence in X (INCX1=1 for contiguous data). Constraint: INCX1 > 0.Chapter 5: Fast Fourier Transforms (FFTs) 42 INTEGER INCX2 [Input] On input: INCX2 is the increment used to store corresponding elements of successive sequences in X (INCX2=N for contiguous data). Constraint: INCX2 > 0. COMPLEX Y(1+(N-1)*INCY1+(NSEQ-1)*INCY2) [Output] On output: if INPL is .FALSE. then Y contains the transformed sequences with the ith element of sequence j stored in Y(1+(i-1)*INCY1+(j-1)*INCY2); otherwise Y is not referenced. INTEGER INCY1 [Input] On input: INCY1 is the increment used to store successive elements of a given sequence in Y. If INPL is .TRUE. then INCY1 is not referenced. Constraint: INCY1 > 0. INTEGER INCY2 [Input] On input: INCY2 is the increment used to store corresponding elements of successive sequences in Y (INCY2=N for contiguous data). If INPL is .TRUE. then INCY2 is not referenced. Constraint: INCY2 > 0. COMPLEX COMM(5*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Forward FFTs are performed unscaled and in-place on two C contiguous vectors stored in the first two columns of X. C Manipulations are stored in 2nd and 3rd columns of X which are C then transformed back. C COMPLEX X(N,3) SCALE = 1.0 INPL = .TRUE. CALL CFFT1MX(0,SCALE,INPL,2,N,X,1,N,DUM,1,N,COMM,INFO) CALL CFFT1MX(-1,SCALE,INPL,2,N,X,1,N,DUM,1,N,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*CONJG(X(I,2))/REAL(N) X(I,2) = CMPLX(0.0D0,1.0D0)*X(I,2)/REAL(N) 10 CONTINUE CALL CFFT1MX(1,SCALE,INPL,2,N,X(1,2),1,N,DUM,1,N,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 43 5.2.3 2D FFT of two-dimensional arrays of data The routines documented here compute the two-dimensional discrete Fourier transforms (DFT) of a two-dimensional array of complex numbers in either single or double precision arithmetic. The 2D DFT is computed using a highly-e?cient FFT algorithm. There are two sets of interfaces available: simple drivers and expert drivers. The simple drivers perform in-place transforms on data held contiguously in memory using a ?xed scaling factor; these are simpler to use and are su?cient for many problems. The expert drivers o?er greater ?exibility by including a number of additional arguments. These allow you to control: the scaling factor applied; whether the result should be output to a separate array; the increments used in storing successive elements in each dimension (for both input and output); and the facility to not perform a ?nal transposition. This ?nal facility is useful for those cases where a forward and backward transform are to be applied with some data manipulations in between; here two whole transpositions can be saved.Chapter 5: Fast Fourier Transforms (FFTs) 44 ZFFT2D Routine Documentation ZFFT2D (MODE,M,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the direction of transform to be performed by ZFFT2D. On input: • MODE=-1 : forward 2D transform is performed. • MODE=1 : backward (reverse) 2D transform is performed. INTEGER M [Input] On input: M is the number of rows in the 2D array of data to be transformed. If X is declared as a 2D array then M is the ?rst dimension of X. INTEGER N [Input] On input: N is the number of columns in the 2D array of data to be transformed. If X is declared as a 2D array then M is the second dimension of X. COMPLEX*16 X(M*N) [Input/Output] On input: X contains the M by N complex 2D array to be transformed. Element ij is stored in location i + (j - 1) * M of X. On output: X contains the transformed sequence. COMPLEX*16 COMM(M*N+3*(M+N)) [Input/Output] COMM is a communication array used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL ZFFT2D(-1,M,N,X,COMM,INFO) DO 20 J = 1, N DO 10 I = 1, MIN(J-1,M) X(I,J) = 0.5D0*(X(I,J) + X(J,I)) X(J,I) = DCONJG(X(I,J)) 10 CONTINUE 20 CONTINUE CALL ZFFT2D(1,M,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 45 CFFT2D Routine Documentation CFFT2D (MODE,M,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the direction of transform to be performed by CFFT2D. On input: • MODE=-1 : a forward 2D transform is performed. • MODE=1 : a backward (reverse) 2D transform is performed. INTEGER M [Input] On input: M is the number of rows in the 2D array of data to be transformed. If X is declared as a 2D array then M is the ?rst dimension of X. INTEGER N [Input] On input: N is the number of columns in the 2D array of data to be transformed. If X is declared as a 2D array then M is the second dimension of X. COMPLEX X(M*N) [Input/Output] On input: X contains the M by N complex 2D array to be transformed. Element ij is stored in location i + (j - 1) * M of X. On output: X contains the transformed sequence. COMPLEX COMM(M*N+5*(M+N)) [Input/Output] COMM is a communication array used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL CFFT2D(-1,M,N,X,COMM,INFO) DO 20 J = 1, N DO 10 I = 1, MIN(J-1,M) X(I,J) = 0.5D0*(X(I,J) + X(J,I)) X(J,I) = CONJG(X(I,J)) 10 CONTINUE 20 CONTINUE CALL CFFT2D(1,M,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 46 ZFFT2DX Routine Documentation ZFFT2DX (MODE,SCALE,LTRANS,INPL,M,N,X,INCX1, [SUBROUTINE] INCX2,Y,INCY1,INCY2,COMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZFFT2DX. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward 2D transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT2DX. • MODE=1 : a backward (reverse) 2D transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT2DX. • MODE=-2 : (default) initializations and a forward 2D transform are performed. • MODE=2 : (default) initializations and a backward 2D transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the values of N and M) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the values of N and M. DOUBLE PRECISION SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL LTRANS [Input] On input: if LTRANS is .TRUE. then a normal ?nal transposition is performed internally to return transformed data consistent with the values for arguments INPL, INCX1, INCX2, INCY1 and INCY2. If LTRANS is .FALSE. then the ?nal transposition is not performed explicitly; the storage format on output is determined by whether the output data is stored contiguously or not – please see the output speci?cations for X and Y for details. LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER M [Input] On input: M is the ?rst dimension of the 2D transform. INTEGER N [Input] On input: N is the second dimension of the 2D transform.Chapter 5: Fast Fourier Transforms (FFTs) 47 COMPLEX*16 X(1+(M-1)*INCX1+(N-1)*INCX2) [Input/Output] On input: X contains the M by N complex 2D data array to be transformed; the (ij)th element is stored in X(1+(i-1)*INCX1+(j-1)*INCX2). On output: if INPL is .TRUE. then X contains the transformed data, either in the same locations as on input when LTRANS=.TRUE.; in locations X((i- 1)*N+j) when LTRANS=.FALSE., INCX1=1 and INCX2=M; and otherwise in the same locations as on input. If INPL is .FALSE. X remains unchanged. INTEGER INCX1 [Input] On input: INCX1 is the increment used to store, in X, successive elements in the ?rst dimension (INCX1=1 for contiguous data). Constraint: INCX1 > 0. INTEGER INCX2 [Input] On input: INCX2 is the increment used to store, in X, successive elements in the second dimension (INCX2=M for contiguous data). Constraint: INCX2 > 0; INCX2 > (M-1)*INCX1 if N > 1. COMPLEX*16 Y(1+(M-1)*INCY1+(N-1)*INCY2) [Output] On output: if INPL is .FALSE. then Y contains the transformed data. If LTRANS=.TRUE. then the (ij)th data element is stored in Y(1+(i- 1)*INCY1+(j-1)*INCY2); if LTRANS=.FALSE., INCY1=1 and INCY2=N then the (ij)th data element is stored in Y((i-1)*N+j); and otherwise the (ij)th element is stored in Y(1+(i-1)*INCY1+(j-1)*INCY2). If INPL is .TRUE. then Y is not referenced. INTEGER INCY1 [Input] On input: INCY1 is the increment used to store successive elements in the ?rst dimension in Y (INCY1=1 for contiguous data). If INPL is .TRUE. then INCY1 is not referenced. Constraint: INCY1 > 0. INTEGER INCY2 [Input] On input: INCY2 is the increment used to store successive elements in the second dimension in Y (for contiguous data, INCY2=M when LTRANS is .TRUE. or INCY2=N when LTRANS is .FALSE.). If INPL is .TRUE. then INCY2 is not referenced. Constraints: INCY2 > 0; INCY2 > (M-1)*INCY1 if N > 1 and LTRANS is .TRUE.; INCY2 = N if M > 1 and LTRANS is .FALSE.. COMPLEX*16 COMM(M*N+3*M+3*N+200) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same dimensions M and N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 5: Fast Fourier Transforms (FFTs) 48 Example:   C Forward 2D FFT is performed unscaled, without final transpose C and out-of-place on data stored in array X and output to Y. C Manipulations are stored in vector Y which is then transformed C back, with scaling, into the first M rows of X. C COMPLEX *16 X(M,N), Y(N,M) SCALE = 1.0D0 INPL = .FALSE. LTRANS = .FALSE. CALL ZFFT2DX(0,SCALE,LTRANS,INPL,M,N,X,1,M,Y,1,N,COMM,INFO) CALL ZFFT2DX(-1,SCALE,LTRANS,INPL,M,N,X,1,M,Y,1,N,COMM,INFO) DO 20 I = M DO 10 J = 1, N Y(J,I) = 0.5D0*Y(J,I)*EXP(0.001D0*(I+J-2)) 10 CONTINUE 20 CONTINUE SCALE = 1.0D0/DBLE(M*N) CALL ZFFT2DX(1,SCALE,LTRANS,INPL,N,M,Y,1,N,X,1,M,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 49 CFFT2DX Routine Documentation CFFT2DX (MODE,SCALE,LTRANS,INPL,M,N,X,INCX1, [SUBROUTINE] INCX2,Y,INCY1,INCY2,COMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by CFFT2DX. On input: • MODE=0 : only initializations (speci?c to the value of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward 2D transform is performed. Initializations are assumed to have been performed by a prior call to CFFT2DX. • MODE=1 : a backward (reverse) 2D transform is performed. Initializations are assumed to have been performed by a prior call to CFFT2DX. • MODE=-2 : (default) initializations and a forward 2D transform are performed. • MODE=2 : (default) initializations and a backward 2D transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the values of N and M) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the values of N and M. REAL SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL LTRANS [Input] On input: if LTRANS is .TRUE. then a normal ?nal transposition is performed internally to return transformed data consistent with the values for arguments INPL, INCX1, INCX2, INCY1 and INCY2. If LTRANS is .FALSE. then the ?nal transposition is not performed explicitly; the storage format on output is determined by whether the output data is stored contiguously or not – please see the output speci?cations for X and Y for details. LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER M [Input] On input: M is the ?rst dimension of the 2D transform. INTEGER N [Input] On input: N is the second dimension of the 2D transform.Chapter 5: Fast Fourier Transforms (FFTs) 50 COMPLEX X(1+(M-1)*INCX1+(N-1)*INCX2) [Input/Output] On input: X contains the M by N complex 2D data array to be transformed; the (ij)th element is stored in X(1+(i-1)*INCX1+(j-1)*INCX2). On output: if INPL is .TRUE. then X contains the transformed data, either in the same locations as on input when LTRANS=.TRUE.; in locations X((i- 1)*N+j) when LTRANS=.FALSE., INCX1=1 and INCX2=M; and otherwise in the same locations as on input. If INPL is .FALSE. X remains unchanged. INTEGER INCX1 [Input] On input: INCX1 is the increment used to store, in X, successive elements in the ?rst dimension (INCX1=1 for contiguous data). Constraint: INCX1 > 0. INTEGER INCX2 [Input] On input: INCX2 is the increment used to store, in X, successive elements in the second dimension (INCX2=M for contiguous data). Constraint: INCX2 > 0; INCX2 > (M-1)*INCX1 if N > 1. COMPLEX Y(1+(M-1)*INCY1+(N-1)*INCY2) [Output] On output: if INPL is .FALSE. then Y contains the transformed data. If LTRANS=.TRUE. then the (ij)th data element is stored in Y(1+(i- 1)*INCY1+(j-1)*INCY2); if LTRANS=.FALSE., INCY1=1 and INCY2=N then the (ij)th data element is stored in Y((i-1)*N+j); and otherwise the (ij)th element is stored in Y(1+(i-1)*INCY1+(j-1)*INCY2). If INPL is .TRUE. then Y is not referenced. INTEGER INCY1 [Input] On input: INCY1 is the increment used to store successive elements in the ?rst dimension in Y (INCY1=1 for contiguous data). If INPL is .TRUE. then INCY1 is not referenced. Constraint: INCY1 > 0. INTEGER INCY2 [Input] On input: INCY2 is the increment used to store successive elements in the second dimension in Y (for contiguous data, INCY2=M when LTRANS is .TRUE. or INCY2=N when LTRANS is .FALSE.). If INPL is .TRUE. then INCY2 is not referenced. Constraints: INCY2 > 0; INCY2 > (M-1)*INCY1 if N > 1 and LTRANS is .TRUE.; INCY2 = N if M > 1 and LTRANS is .FALSE.. COMPLEX COMM(M*N+5*M+5*N+200) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same dimensions M and N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 5: Fast Fourier Transforms (FFTs) 51 Example:   C Forward 2D FFT is performed unscaled, without final transpose C and out-of-place on data stored in array X and output to Y. C Manipulations are stored in vector Y which is then transformed C back, with scaling, into the first M rows of X. C COMPLEX X(M,N), Y(N,M) SCALE = 1.0 INPL = .FALSE. LTRANS = .FALSE. CALL CFFT2DX(0,SCALE,LTRANS,INPL,M,N,X,1,M,Y,1,N,COMM,INFO) CALL CFFT2DX(-1,SCALE,LTRANS,INPL,M,N,X,1,M,Y,1,N,COMM,INFO) DO 20 I = M DO 10 J = 1, N Y(J,I) = 0.5*Y(J,I)*EXP(-0.001*REAL(I+J-2)) IY = IY + 1 10 CONTINUE 20 CONTINUE SCALE = 1.0/REAL(M*N) CALL CFFT2DX(1,SCALE,LTRANS,INPL,N,M,Y,1,N,X,1,M,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 52 5.2.4 3D FFT of three-dimensional arrays of data The routines documented here compute the three-dimensional discrete Fourier transforms (DFT) of a three-dimensional array of complex numbers in either single or double precision arithmetic. The 3D DFT is computed using a highly-e?cient FFT algorithm. Please note that at Release 2.7 of ACML it has been necessary to modify slightly the interfaces of two of the expert FFT drivers introduced at Release 2.2 of ACML. The two routines are CFFT3DX and ZFFT3DX. The changes are required to permit the optimization of these routines by adding an initialization stage which can then use the plan generator (MODE=100) to select the optimal plan. User codes that called CFFT3DX or ZFFT3DX using a release of ACML prior to 2.7 will need to be modi?ed in one of two ways. Calls to CFFT3DX/ZFFT3DX with MODE = -1 or 1 can be ?xed for ACML Release 2.7 and later by either: • preceding the call with a call setting MODE = 0 (default initialization), or MODE = 100 (initialization using plan generator); or, • doubling the MODE argument value to MODE = -2 or 2 respectively (thus incorporating default initialization). Additionally, the minimum length of the communication (work)space arrays in CFFT3DX and ZFFT3DX has been increased by 100 to allow for plan storage. Please consult the individual routine documents for full details on their use.Chapter 5: Fast Fourier Transforms (FFTs) 53 ZFFT3D Routine Documentation ZFFT3D (MODE,L,M,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the direction of transform to be performed by ZFFT3D. On input: • MODE=-1 : forward 3D transform is performed. • MODE=1 : backward (reverse) 3D transform is performed. INTEGER L [Input] On input: the length of the ?rst dimension of the 3D array of data to be transformed. If X is declared as a 3D array then L is the ?rst dimension of X. INTEGER M [Input] On input: the length of the second dimension of the 3D array of data to be transformed. If X is declared as a 3D array then M is the second dimension of X. INTEGER N [Input] On input: the length of the third dimension of the 3D array of data to be transformed. If X is declared as a 3D array then N is the third dimension of X. COMPLEX*16 X(L*M*N) [Input/Output] On input: X contains the L by M by N complex 3D array to be transformed. Element ijk is stored in location i + (j - 1) * L + (k - 1) * L * M of X. On output: X contains the transformed sequence. COMPLEX*16 COMM(L*M*N+3*(L+M+N)) [Input/Output] COMM is a communication array used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL ZFFT3D(-1,L,M,N,X,COMM,INFO) DO 30 K = 1, N DO 20 J = 1, M DO 10 I = 1, L X(I,J) = X(I,J)*EXP(-0.001D0*DBLE(I+J+K)) 10 CONTINUE 20 CONTINUE 30 CONTINUE CALL ZFFT3D(1,L,M,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 54 CFFT3D Routine Documentation CFFT3D (MODE,L,M,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the direction of transform to be performed by CFFT3D. On input: • MODE=-1 : forward 3D transform is performed. • MODE=1 : backward (reverse) 3D transform is performed. INTEGER L [Input] On input: the length of the ?rst dimension of the 3D array of data to be transformed. If X is declared as a 3D array then L is the ?rst dimension of X. INTEGER M [Input] On input: the length of the second dimension of the 3D array of data to be transformed. If X is declared as a 3D array then M is the second dimension of X. INTEGER N [Input] On input: the length of the third dimension of the 3D array of data to be transformed. If X is declared as a 3D array then N is the third dimension of X. COMPLEX X(L*M*N) [Input/Output] On input: X contains the L by M by N complex 3D array to be transformed. Element ijk is stored in location i + (j - 1) * L + (k - 1) * L * M of X. On output: X contains the transformed sequence. COMPLEX COMM(L*M*N+5*(L+M+N)) [Input/Output] COMM is a communication array used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL CFFT3D(-1,L,M,N,X,COMM,INFO) DO 30 K = 1, N DO 20 J = 1, M DO 10 I = 1, L X(I,J) = X(I,J)*EXP(-0.001D0*REAL(I+J+K)) 10 CONTINUE 20 CONTINUE 30 CONTINUE CALL CFFT3D(1,L,M,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 55 ZFFT3DX Routine Documentation ZFFT3DX (MODE,SCALE,LTRANS,INPL,L,M,N,X,Y, [SUBROUTINE] COMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZFFT3DX. On input: • MODE=0 : only initializations (speci?c to the values of L, M and N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward 3D transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT3DX. • MODE=1 : a backward (reverse) 3D transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT3DX. • MODE=-2 : (default) initializations and a forward 3D transform are performed. • MODE=2 : (default) initializations and a backward 3D transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the values of L, M and M) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the values of L, M and N. DOUBLE PRECISION SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL LTRANS [Input] On input: if LTRANS is .TRUE. then a normal ?nal transposition is performed internally to return transformed data using the same storage format as the input data. If LTRANS is .FALSE. then the ?nal transposition is not performed and transformed data is stored, in X or Y, in transposed form. LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER L [Input] On input: L is the ?rst dimension of the 3D transform. INTEGER M [Input] On input: M is the second dimension of the 3D transform. INTEGER N [Input] On input: N is the third dimension of the 3D transform.Chapter 5: Fast Fourier Transforms (FFTs) 56 COMPLEX*16 X(L*M*N) [Input/Output] On input: X contains the L by M by N complex 3D data array to be transformed; the (ijk)th element is stored in X(i+(j-1)*L+(k-1)*L*M). On output: if INPL is .TRUE. then X contains the transformed data, either in the same locations as on input when LTRANS=.TRUE.; or in locations X(k+(j-1)*N+(i-1)*N*M) when LTRANS=.FALSE. If INPL is .FALSE. X remains unchanged. COMPLEX*16 Y(L*M*N) [Output] On output: if INPL is .FALSE. then Y contains the three-dimensional transformed data. If LTRANS=.TRUE. then the (ijk)th data element is stored in Y(i+(j-1)*L+(k-1)*L*M); otherwise, the (ijk)th data element is stored in Y(k+(j- 1)*N+(i-1)*N*M). If INPL is .TRUE. then Y is not referenced. COMPLEX*16 COMM(L*M*N+3*(L+M+N)+300) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence dimensions. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Forward 3D FFT is performed unscaled, without final transpose C and out-of-place on data stored in array X and output to Y. C Manipulations are stored in vector Y which is then transformed C back, with scaling, into the first M rows of X. C COMPLEX *16 X(L*M*N), Y(L*M*N) SCALE = 1.0D0 INPL = .FALSE. LTRANS = .FALSE. CALL ZFFT3DX(0,SCALE,LTRANS,INPL,L,M,N,X,Y,COMM,INFO) CALL ZFFT3DX(-1,SCALE,LTRANS,INPL,L,M,N,X,Y,COMM,INFO) IY = 1 DO 20 I = 1, L DO 40 J = 1, M DO 10 K = 1, N Y(IY) = Y(IY)*EXP(-0.001D0*DBLE(I+J+K-3)) IY = IY + 1 10 CONTINUE 20 CONTINUE SCALE = 1.0D0/DBLE(L*M*N) CALL ZFFT3DX(1,SCALE,LTRANS,INPL,N,M,L,Y,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 57 CFFT3DX Routine Documentation CFFT3DX (MODE,SCALE,LTRANS,INPL,L,M,N,X,Y, [SUBROUTINE] COMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by CFFT3DX. On input: • MODE=0 : only initializations (speci?c to the values of L, M and N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward 3D transform is performed. Initializations are assumed to have been performed by a prior call to CFFT3DX. • MODE=1 : a backward (reverse) 3D transform is performed. Initializations are assumed to have been performed by a prior call to CFFT3DX. • MODE=-2 : (default) initializations and a forward 3D transform are performed. • MODE=2 : (default) initializations and a backward 3D transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the values of L, M and M) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the values of L, M and N. REAL SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL LTRANS [Input] On input: if LTRANS is .TRUE. then a normal ?nal transposition is performed internally to return transformed data using the same storage format as the input data. If LTRANS is .FALSE. then the ?nal transposition is not performed and transformed data is stored, in X or Y, in transposed form. LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER L [Input] On input: L is the ?rst dimension of the 3D transform. INTEGER M [Input] On input: M is the second dimension of the 3D transform. INTEGER N [Input] On input: N is the third dimension of the 3D transform.Chapter 5: Fast Fourier Transforms (FFTs) 58 COMPLEX X(L*M*N) [Input/Output] On input: X contains the L by M by N complex 3D data array to be transformed; the (ijk)th element is stored in X(i+(j-1)*L+(k-1)*L*M). On output: if INPL is .TRUE. then X contains the transformed data, either in the same locations as on input when LTRANS=.TRUE.; or in locations X(k+(j-1)*N+(i-1)*N*M) when LTRANS=.FALSE. If INPL is .FALSE. X remains unchanged. COMPLEX Y(L*M*N) [Output] On output: if INPL is .FALSE. then Y contains the three-dimensional transformed data. If LTRANS=.TRUE. then the (ijk)th data element is stored in Y(i+(j-1)*L+(k-1)*L*M); otherwise, the (ijk)th data element is stored in Y(k+(j- 1)*N+(k-1)*N*M). If INPL is .TRUE. then Y is not referenced. COMPLEX COMM(L*M*N+5*(L+M+N)+300) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence dimensions. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Forward 3D FFT is performed unscaled, without final transpose C and out-of-place on data stored in array X and output to Y. C Manipulations are stored in vector Y which is then transformed C back, with scaling, into the first M rows of X. C SCALE = 1.0 INPL = .FALSE. LTRANS = .FALSE. CALL CFFT3DX(0,SCALE,LTRANS,INPL,L,M,N,X,Y,COMM,INFO) CALL CFFT3DX(-1,SCALE,LTRANS,INPL,L,M,N,X,Y,COMM,INFO) IY = 1 DO 20 I = 1, L DO 40 J = 1, M DO 10 K = 1, N Y(IY) = Y(IY)*EXP(-0.001*REAL(I+J+K-3)) IY = IY + 1 10 CONTINUE 20 CONTINUE SCALE = 1.0/REAL(L*M*N) CALL CFFT3DX(1,SCALE,LTRANS,INPL,N,M,L,Y,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 59 ZFFT3DY Routine Documentation ZFFT3DY (MODE,SCALE,INPL,L,M,N,X, [SUBROUTINE] INCX1,INCX2,INCX3,Y,INCY1,INCY2,INCY3,COMM,LCOMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZFFT3DY. On input: • MODE=0 : only initializations (speci?c to the values of L, M and N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward 3D transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT3DY. • MODE=1 : a backward (reverse) 3D transform is performed. Initializations are assumed to have been performed by a prior call to ZFFT3DY. • MODE=-2 : (default) initializations and a forward 3D transform are performed. • MODE=2 : (default) initializations and a backward 3D transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the values of L, M and M) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the values of L, M and N. REAL SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER L [Input] On input: L is the ?rst dimension of the 3D transform. INTEGER M [Input] On input: M is the second dimension of the 3D transform. INTEGER N [Input] On input: N is the third dimension of the 3D transform. COMPLEX*16 X(*) [Input/Output] On input: X contains the L by M by N complex 3D data array to be transformed; the (ijk)th element is stored in X(1+(i-1)*INCX1+(j-1)*INCX2+(k- 1)*INCX3). On output: if INPL is .TRUE. then X contains the transformed data in the same locations as on input. If INPL is .FALSE. X remains unchanged.Chapter 5: Fast Fourier Transforms (FFTs) 60 INTEGER INCX1 [Input] On input: INCX1 is the step in index of X between successive data elements in the ?rst dimension of the 3D data. Usually INCX1=1 so that succesive elements in the ?rst dimension are stored contiguously. Constraint: INCX1 > 0. INTEGER INCX2 [Input] On input: INCX2 is the step in index of X between successive data elements in the second dimension of the 3D data. For completely contiguous data (no gaps in X) INCX2 should be set to L. Constraint: INCX2 > 0; INCX2 > (L-1)*INCX1 if max(M,N) > 1. INTEGER INCX3 [Input] On input: INCX3 is the step in index of X between successive data elements in the third dimension of the 3D data. For completely contiguous data (no gaps in X) INCX3 should be set to L*M. Constraint: INCX3 > 0; INCX3 > (L-1)*INCX1+(M-1)*INCX2 if N > 1. COMPLEX*16 Y(*) [Output] On output: if INPL is .FALSE. then Y contains the three-dimensional transformed data. If LTRANS=.TRUE. then the the (ijk)th element is stored in Y(1+(i-1)*INCY1+(j-1)*INCY2+(k-1)*INCY3). If INPL is .TRUE. then Y is not referenced. INTEGER INCY1 [Input] On input: if INPL is .FALSE. then INCY1 is the step in index of Y between successive data elements in the ?rst dimension of the 3D transformed data. Usually INCY1=1 so that succesive elements in the ?rst dimension are stored contiguously. If INPL is .TRUE. then INCY1 is not referenced. Constraint: If INPL is .FALSE. then INCY1 > 0. INTEGER INCY2 [Input] On input: if INPL is .FALSE. then INCY2 is the step in index of Y between successive data elements in the second dimension of the 3D transformed data. For completely contiguous data (no gaps in Y) INCY2 should be set to L. Constraint: INCY2 > 0 if INPL is .FALSE.; INCY2 > (L-1)*INCY1, if INPL is .FALSE. and max(M,N) > 1. INTEGER INCY3 [Input] On input: if INPL is .FALSE. then INCY3 is the step in index of Y between successive data elements in the third dimension of the 3D transformed data. For completely contiguous data (no gaps in Y) INCY3 should be set to L*M. Constraint: INCY3 > 0 if INPL is .FALSE.; INCY3 > (L-1)*INCY1+(M-1)*INCY2, if INPL is .FALSE. and N > 1.Chapter 5: Fast Fourier Transforms (FFTs) 61 COMPLEX*16 COMM(LCOMM) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence dimensions. The remainder is used as temporary store; if this is not su?cient for the requirements of the routine then temporary storage space will be dynamically allocated internally. INTEGER LCOMM [Input] On input: LCOMM is the length of the communication array COMM. The amount of internal dynamic allocation of temporary storage can be reduced signi?cantly by declaring COMM to be of length at least L*M*N + 4*(L+M+N) + 300. Constraint: LCOMM > 3*(L+M+N) + 150. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Forward 3D FFT is performed unscaled and in-place, on the leading C 10x10x10 submatrix of a larger 100x100x100 array of data. C The result is transformed back with scaling. C SCALE = 1.0D0 INPL = .TRUE. L = 10 M = 10 N = 10 LCOMM = 2000000 CALL ZFFT3DY(0,SCALE,INPL,L,M,N,X,1,100,10000,Y,1,1,1, * COMM,LCOMM,INFO) CALL ZFFT3DY(-1,SCALE,INPL,L,M,N,X,1,100,10000,Y,1,1,1, * COMM,LCOMM,INFO) IY = 1 DO 20 I = 1, L DO 40 J = 1, M DO 10 K = 1, N X(I,J,K) = X(I,J,K)*EXP(-1.0D-3*DBLE(I+J+K-3)) 10 CONTINUE 20 CONTINUE SCALE = 1.0/DBLE(L*M*N) CALL ZFFT3DY(1,SCALE,INPL,L,M,N,X,1,100,10000,Y,1,1,1, * COMM,LCOMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 62 CFFT3DY Routine Documentation CFFT3DY (MODE,SCALE,INPL,L,M,N,X, [SUBROUTINE] INCX1,INCX2,INCX3,Y,INCY1,INCY2,INCY3,COMM,LCOMM,INFO) INTEGER MODE [Input] The value of MODE on input determines the operation performed by CFFT3DY. On input: • MODE=0 : only initializations (speci?c to the values of L, M and N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=-1 : a forward 3D transform is performed. Initializations are assumed to have been performed by a prior call to CFFT3DY. • MODE=1 : a backward (reverse) 3D transform is performed. Initializations are assumed to have been performed by a prior call to CFFT3DY. • MODE=-2 : (default) initializations and a forward 3D transform are performed. • MODE=2 : (default) initializations and a backward 3D transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the values of L, M and M) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the values of L, M and N. REAL SCALE [Input] On input: SCALE is the scaling factor to apply to the output sequences LOGICAL INPL [Input] On input: if INPL is .TRUE. then X is overwritten by the output sequences; otherwise the output sequences are returned in Y. INTEGER L [Input] On input: L is the ?rst dimension of the 3D transform. INTEGER M [Input] On input: M is the second dimension of the 3D transform. INTEGER N [Input] On input: N is the third dimension of the 3D transform. COMPLEX X(*) [Input/Output] On input: X contains the L by M by N complex 3D data array to be transformed; the (ijk)th element is stored in X(1+(i-1)*INCX1+(j-1)*INCX2+(k- 1)*INCX3). On output: if INPL is .TRUE. then X contains the transformed data in the same locations as on input. If INPL is .FALSE. X remains unchanged.Chapter 5: Fast Fourier Transforms (FFTs) 63 INTEGER INCX1 [Input] On input: INCX1 is the step in index of X between successive data elements in the ?rst dimension of the 3D data. Usually INCX1=1 so that succesive elements in the ?rst dimension are stored contiguously. Constraint: INCX1 > 0. INTEGER INCX2 [Input] On input: INCX2 is the step in index of X between successive data elements in the second dimension of the 3D data. For completely contiguous data (no gaps in X) INCX2 should be set to L. Constraint: INCX2 > 0; INCX2 > (L-1)*INCX1 if max(M,N) > 1. INTEGER INCX3 [Input] On input: INCX3 is the step in index of X between successive data elements in the third dimension of the 3D data. For completely contiguous data (no gaps in X) INCX3 should be set to L*M. Constraint: INCX3 > 0; INCX3 > (L-1)*INCX1+(M-1)*INCX2 if N > 1. COMPLEX Y(*) [Output] On output: if INPL is .FALSE. then Y contains the three-dimensional transformed data. If LTRANS=.TRUE. then the the (ijk)th element is stored in Y(1+(i-1)*INCY1+(j-1)*INCY2+(k-1)*INCY3). If INPL is .TRUE. then Y is not referenced. INTEGER INCY1 [Input] On input: if INPL is .FALSE. then INCY1 is the step in index of Y between successive data elements in the ?rst dimension of the 3D transformed data. Usually INCY1=1 so that succesive elements in the ?rst dimension are stored contiguously. If INPL is .TRUE. then INCY1 is not referenced. Constraint: If INPL is .FALSE. then INCY1 > 0. INTEGER INCY2 [Input] On input: if INPL is .FALSE. then INCY2 is the step in index of Y between successive data elements in the second dimension of the 3D transformed data. For completely contiguous data (no gaps in Y) INCY2 should be set to L. Constraint: INCY2 > 0 if INPL is .FALSE.; INCY2 > (L-1)*INCY1, if INPL is .FALSE. and max(M,N) > 1. INTEGER INCY3 [Input] On input: if INPL is .FALSE. then INCY3 is the step in index of Y between successive data elements in the third dimension of the 3D transformed data. For completely contiguous data (no gaps in Y) INCY3 should be set to L*M. Constraint: INCY3 > 0 if INPL is .FALSE.; INCY3 > (L-1)*INCY1+(M-1)*INCY2, if INPL is .FALSE. and N > 1.Chapter 5: Fast Fourier Transforms (FFTs) 64 COMPLEX COMM(LCOMM) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence dimensions. The remainder is used as temporary store; if this is not su?cient for the requirements of the routine then temporary storage space will be dynamically allocated internally. INTEGER LCOMM [Input] On input: LCOMM is the length of the communication array COMM. The amount of internal dynamic allocation of temporary storage can be reduced signi?cantly by declaring COMM to be of length at least L*M*N + 4*(L+M+N) + 300.. Constraint: LCOMM > L*M*N + 2*(L+M+N) + 300. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Forward 3D FFT is performed unscaled and in-place, on the leading C 10x10x10 submatrix of a larger 100x100x100 array of data. C The result is transformed back with scaling. C SCALE = 1.0 INPL = .TRUE. L = 10 M = 10 N = 10 LCOMM = 2000000 CALL CFFT3DY(0,SCALE,INPL,L,M,N,X,1,100,10000,Y,1,1,1, * COMM,LCOMM,INFO) CALL CFFT3DY(-1,SCALE,INPL,L,M,N,X,1,100,10000,Y,1,1,1, * COMM,LCOMM,INFO) IY = 1 DO 20 I = 1, L DO 40 J = 1, M DO 10 K = 1, N X(I,J,K) = X(I,J,K)*EXP(-0.001*REAL(I+J+K-3)) 10 CONTINUE 20 CONTINUE SCALE = 1.0/REAL(L*M*N) CALL CFFT3DY(1,SCALE,INPL,L,M,N,X,1,100,10000,Y,1,1,1, * COMM,LCOMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 65 5.3 FFTs on real and Hermitian data sequences The routines documented here compute discrete Fourier transforms (DFTs) of sequences of real numbers or of Hermitian sequences in either single or double precision arithmetic. The DFTs are computed using a highly-e?cient FFT algorithm. Hermitian sequences are represented in a condensed form that is described in Section 5.1 [Introduction to FFTs], page 24. The DFT of a real sequence results in a Hermitian sequence; the DFT of a Hermitian sequence is a real sequence. Please note that prior to Release 2.0 of ACML the routine ZDFFT/CSFFT and ZDFFTM/CSFFTM returned results that were scaled by a factor 0.5 compared with the currently returned results.Chapter 5: Fast Fourier Transforms (FFTs) 66 5.3.1 FFT of single sequences of real data DZFFT Routine Documentation DZFFT (MODE,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by DZFFT. On input: • MODE=0 : only default initializations (speci?c to N) are performed; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=1 : a real transform is performed. Initializations are assumed to have been performed by a prior call to DZFFT. • MODE=2 : (default) initializations and a real transform are performed. • MODE=100 : similar to MODE=0; only initializations are performed, but ?rst a plan is generated. This plan is chosen based on the fastest FFT computation for a subset of all possible plans. INTEGER N [Input] On input: N is the length of the real sequence X DOUBLE PRECISION X(N) [Input/Output] On input: X contains the real sequence of length N to be transformed. On output: X contains the transformed Hermitian sequence. DOUBLE PRECISION COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL DZFFT(0,N,X,COMM,INFO) CALL DZFFT(1,N,X,COMM,INFO) DO 10 I = N/2+2, N X(I) = -X(I) 10 CONTINUE CALL ZDFFT(2,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 67 SCFFT Routine Documentation SCFFT (MODE,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by SCFFT. On input: • MODE=0 : only default initializations (speci?c to N) are performed; this is usually followed by calls to the same routine with MODE=-1 or 1. • MODE=1 : a real transform is performed. Initializations are assumed to have been performed by a prior call to SCFFT. • MODE=2 : (default) initializations and a real transform are performed. • MODE=100 : similar to MODE=0; only initializations are performed, but ?rst a plan is generated. This plan is chosen based on the fastest FFT computation for a subset of all possible plans. INTEGER N [Input] On input: N is the length of the real sequence X REAL X(N) [Input/Output] On input: X contains the real sequence of length N to be transformed. On output: X contains the transformed Hermitian sequence. REAL COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL SCFFT(0,N,X,COMM,INFO) CALL SCFFT(1,N,X,COMM,INFO) DO 10 I = N/2+2, N X(I) = -X(I) 10 CONTINUE CALL CSFFT(2,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 68 5.3.2 FFT of multiple sequences of real data DZFFTM Routine Documentation DZFFTM (M,N,X,COMM,INFO) [SUBROUTINE] INTEGER M [Input] On input: M is the number of sequences to be transformed. INTEGER N [Input] On input: N is the length of the real sequences in X DOUBLE PRECISION X(N*M) [Input/Output] On input: X contains the M real sequences of length N to be transformed. Element i of sequence j is stored in location i + (j - 1) * N of X. On output: X contains the transformed Hermitian sequences. DOUBLE PRECISION COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL DZFFTM(1,N,X,COMM,INFO) CALL DZFFTM(2,N,X,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*X(N-I+1,2) 10 CONTINUE CALL ZDFFTM(2,N,X(1,3),COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 69 SCFFTM Routine Documentation SCFFTM (M,N,X,COMM,INFO) [SUBROUTINE] INTEGER M [Input] On input: M is the number of sequences to be transformed. INTEGER N [Input] On input: N is the length of the real sequences in X REAL X(N*M) [Input/Output] On input: X contains the M real sequences of length N to be transformed. Element i of sequence j is stored in location i + (j - 1) * N of X. On output: X contains the transformed Hermitian sequences. REAL COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL SCFFTM(1,N,X,COMM,INFO) CALL SCFFTM(2,N,X,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*X(N-I+1,2) 10 CONTINUE CALL CSFFTM(1,N,X(1,3),COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 70 5.3.3 FFT of single Hermitian sequences ZDFFT Routine Documentation ZDFFT (MODE,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by ZDFFT. On input: • MODE=0 : only initializations (speci?c to the values of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=1. • MODE=1 : a real transform is performed. Initializations are assumed to have been performed by a prior call to ZDFFT. • MODE=2 : (default) initializations and a real transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. INTEGER N [Input] On input: N is length of the sequence in X DOUBLE PRECISION X(N) [Input/Output] On input: X contains the Hermitian sequence of length N to be transformed. On output: X contains the transformed real sequence. DOUBLE PRECISION COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL DZFFT(0,N,X,COMM,INFO) CALL DZFFT(1,N,X,COMM,INFO) DO 10 I = N/2+2, N X(I) = -X(I) 10 CONTINUE CALL ZDFFT(2,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 71 CSFFT Routine Documentation CSFFT (MODE,N,X,COMM,INFO) [SUBROUTINE] INTEGER MODE [Input] The value of MODE on input determines the operation performed by CSFFT. On input: • MODE=0 : only initializations (speci?c to the values of N) are performed using a default plan; this is usually followed by calls to the same routine with MODE=1. • MODE=1 : a real transform is performed. Initializations are assumed to have been performed by a prior call to CSFFT. • MODE=2 : (default) initializations and a real transform are performed. • MODE=100 : similar to MODE=0; only initializations (speci?c to the value of N) are performed, but these are based on a plan that is ?rst generated by timing a subset of all possible plans and choosing the quickest (i.e. the FFT computation was timed as fastest based on the chosen plan). The plan generation phase may take a signi?cant amount of time depending on the value of N. INTEGER N [Input] On input: N is the length of the sequence in X REAL X(N) [Input/Output] On input: X contains the Hermitian sequence of length N to be transformed. On output: X contains the transformed real sequence. REAL COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL SCFFT(0,N,X,COMM,INFO) CALL SCFFT(1,N,X,COMM,INFO) DO 10 I = N/2+2, N X(I) = -X(I) 10 CONTINUE CALL CSFFT(2,N,X,COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 72 5.3.4 FFT of multiple Hermitian sequences ZDFFTM Routine Documentation ZDFFTM (M,N,X,COMM,INFO) [SUBROUTINE] INTEGER M [Input] On input: M is the number of sequences to be transformed. INTEGER N [Input] On input: N is the length of the sequences in X DOUBLE PRECISION X(N*M) [Input/Output] On input: X contains the M Hermitian sequences of length N to be transformed. Element i of sequence j is stored in location i + (j - 1) * N of X. On output: X contains the transformed real sequences. DOUBLE PRECISION COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL DZFFTM(1,N,X,COMM,INFO) CALL DZFFTM(2,N,X,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*X(N-I+1,2) 10 CONTINUE CALL ZDFFTM(1,N,X(1,3),COMM,INFO) Chapter 5: Fast Fourier Transforms (FFTs) 73 CSFFTM Routine Documentation CSFFTM (M,N,X,COMM,INFO) [SUBROUTINE] INTEGER M [Input] On input: M is the number of sequences to be transformed. INTEGER N [Input] On input: N is the length of the sequences in X REAL X(N*M) [Input/Output] On input: X contains the M Hermitian sequences of length N to be transformed. Element i of sequence j is stored in location i + (j - 1) * N of X. On output: X contains the transformed real sequences. REAL COMM(3*N+100) [Input/Output] COMM is a communication array. Some portions of the array are used to store initializations for subsequent calls with the same sequence length N. The remainder is used as temporary store. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   CALL SCFFTM(1,N,X,COMM,INFO) CALL SCFFTM(2,N,X,COMM,INFO) DO 10 I = 1, N X(I,3) = X(I,1)*X(N-I+1,2) 10 CONTINUE CALL CSFFTM(1,N,X(1,3),COMM,INFO) Chapter 6: Random Number Generators 74 6 Random Number Generators Within the context of this document, a base random number generator (BRNG) is a mathematical algorithm that, given an initial state, produces a sequence (or stream) of variates (or values) uniformly distributed over the open interval (0,1). The period of the BRNG is de?ned as the maximum number of values that can be generated before the sequence starts to repeat. The initial state of a BRNG is often called the seed. A pseudo-random number generator (PRNG) is a BRNG that produces a stream of variates that are independent and statistically indistinguishable from a random sequence. A PRNG has several advantages over a true random number generator in that the generated sequence is repeatable, has known mathematical properties and is usually much quicker to generate. A quasi-random number generator (QRNG) is similar to a PRNG, however the variates generated are not statistically independent, rather they are designed to give a more even distribution in multidimensional space. Many books on statistics and computer science have good introductions to PRNGs and QRNGs, see for example Knuth [6] or Banks [7]. All of the BRNGs supplied in the ACML are PRNGs. In addition to standard PRNGs some applications require cryptologically secure generators. A PRNG is said to be cryptologically secure if there is no polynomial-time algorithm which, on input of the ?rst l bits of the output sequence can predict the (l + 1)st bit of the sequence with probability signi?cantly greater than 0.5. This is equivalent to saying there exists no polynomial-time algorithm that can correctly distinguish between an output sequence from the PRNG and a truly random sequence of the same length with probability signi?cantly greater than 0.5 [8]. A distribution generator is a routine that takes variates generated from a BRNG and transforms them into variates from a speci?ed distribution, for example the Gaussian (Normal) distribution. The ACML contains ?ve base generators, (Section 6.1 [Base Generators], page 74), and twenty-three distribution generators (Section 6.3 [Distribution Generators], page 95). In addition users can supply a custom built generator as the base generator for all of the distribution generators (Section 6.1.8 [User Supplied Generators], page 84). The base generators were tested using the Big Crush, Small Crush and Pseudo Diehard test suites from the TestU01 software library [15]. 6.1 Base Generators The ?ve base generators (BRNGs) supplied with the ACML are; the NAG basic generator [9], a series of Wichmann-Hill generators [10], the Mersenne Twister [11], L’Ecuyer’s combined recursive generator MRG32k3a [12] and the Blum-Blum-Shub generator [8]. If a single stream of variates is required it is recommended that the Mersenne Twister (Section 6.1.5 [Mersenne Twister], page 82) base generator is used. This generator combines speed with good statistical properties and an extremely long period. The NAG basic generator (Section 6.1.3 [Basic NAG Generator], page 81) is another quick generator suitable for generating a single stream. However it has a shorter period than the Mersenne Twister and being a linear congruential generator, its statistical properties are not as good. If 273 or fewer multiple streams, with a period of up to 2 80 are required then it is recommended that the Wichmann-Hill generators are used (Section 6.1.4 [Wichmann-Hill Generator], page 82). For more streams or multiple streams with a longer period it isChapter 6: Random Number Generators 75 recommended that the L’Ecuyer combined recursive generator (Section 6.1.6 [L’Ecuyer’s Combined Recursive Generator], page 83) is used in combination with the skip ahead routine (Section 6.2.3 [Skip Ahead], page 89). Generating multiple streams of variates by skipping ahead is generally quicker than generating the streams using the leap frog method. More details on multiple streams can be found in Section 6.2 [Multiple Streams], page 88. The Blum-Blum-Shub generator (Section 6.1.7 [Blum-Blum-Shub Generator], page 83) should only be used if a cryptologically secure generator is required. This generator is extremely slow and has poor statistical properties when used as a base generator for any of the distributional generators. 6.1.1 Initialization of the Base Generators A random number generator must be initialized before use. Three routines are supplied within the ACML for this purpose: DRANDINITIALIZE, DRANDINITIALIZEBBS and DRANDINITIALIZEUSER (see [DRANDINITIALIZE], page 76, [DRANDINITIALIZEBBS], page 79 and [DRANDINITIALIZEUSER], page 85, respectively). Of these, DRANDINITIALIZE is used to initialize all of the supplied base generators, DRANDINITIALIZEBBS supplies an alternative interface to DRANDINITIALIZE for the Blum-Blum-Shub generator, and DRANDINITIALIZEUSER allows the user to register and initialize their own base generator. Both double and single precision versions of all RNG routines are supplied. Double precision names are pre?xed by DRAND, and single precision by SRAND. Note that if a generator has been initialized using the relevant double precision routine, then the double precision versions of the distribution generators must also be used, and vice versa. This even applies to generators with no double or single precision parameters; for example, a call of DRANDDISCRETEUNIFORM must be preceded by a call to one of the double precision initializers (typically DRANDINITIALIZE). No utilities for saving, retrieving or copying the current state of a generator have been provided. All of the information on the current state of a generator (or stream, if multiple streams are being used) is stored in the integer array STATE and as such this array can be treated as any other integer array, allowing for easy copying, restoring etc. The statistical properties of a sequence of random numbers are only guaranteed within the sequence, and not between sequences provided by the same generator. Therefore it is likely that repeated initialization will render the numbers obtained less, rather than more, independent. In most cases there should only be a single call to one of the initialization routines, per application, and this call must be made before any variates are generated. One example of where multiple initialization may be required is brie?y touched upon in Section 6.2 [Multiple Streams], page 88. In order to initialize the Blum-Blum-Shub generator a number of additional parameters, as well as an initial state (seed), are required. Although this generator can be initialized through the DRANDINITIALIZE routine it is recommended that the DRANDINITIALIZEBBS routine is used instead.Chapter 6: Random Number Generators 76 DRANDINITIALIZE / SRANDINITIALIZE Initialize one of the ?ve supplied base generators; NAG basic generator, Wichmann-Hill generator, Mersenne Twister, L’Ecuyer’s combined recursive generator (MRG32k3a) or the Blum-Blum-Shub generator. (Note that SRANDINITIALIZE is the single precision version of DRANDINITIALIZE. The argument lists of both routines are identical except that any double precision arguments of DRANDINITIALIZE are replaced in SRANDINITIALIZE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDINITIALIZE (GENID,SUBID,SEED,LSEED,STATE, [SUBROUTINE] LSTATE,INFO) INTEGER GENID [Input] On input: a numerical code indicating which of the ?ve base generators to initialize. • 1 = NAG basic generator (Section 6.1.3 [Basic NAG Generator], page 81). • 2 = Wichmann-Hill generator (Section 6.1.4 [Wichmann-Hill Generator], page 82). • 3 = Mersenne Twister (Section 6.1.5 [Mersenne Twister], page 82). • 4 = L’Ecuyer’s Combined Recursive generator (Section 6.1.6 [L’Ecuyer’s Combined Recursive Generator], page 83). • 5 = Blum-Blum-Shub generator (Section 6.1.7 [Blum-Blum-Shub Generator], page 83). Constraint: 1= GENID = 5. INTEGER SUBID [Input] On input: if GENID = 2, then SUBID indicates which of the 273 WichmannHill generators to use. If GENID = 5 then SUBID indicates the number of bits to use (v) from each of iteration of the Blum-Blum-Shub generator. In all other cases SUBID is not referenced. Constraint: If GENID = 2 then 1= SUBID = 273 . INTEGER SEED(LSEED) [Input] On input: if GENID= 5 , then 6 SEED is a vector of initial values for the base generator. These values must be positive integers. The number of values required depends on the base generator being used. The NAG basic generator requires one initial value, the Wichmann-Hill generator requires four initial values, the L’Ecuyer combined recursive generator requires six initial values and the Mersenne Twister requires 624 initial values. If the number of seeds required by the chosen generator is > LSEED then SEED(1) is used to initialize the NAG basic generator. This is then used to generate all of the remaining seed values required. In general it is best not to set all the elements of SEED to anything too obvious, such as a single repeated value or a simple sequence. Using such a seed array may lead to several similar values being created in a row when the generator is subsequently called. This is particularly true for the Mersenne Twister generator.Chapter 6: Random Number Generators 77 In order to initialize the Blum-Blum-Shub generator two large prime values, p and q are required as well as an initial value s. As p, q and s can be of an arbitrary size, these values are expressed as a polynomial in B, where B = 2 24 . For example, p can be factored into a polynomial of order lp, with p = p1 + p2B + p3B2 + · · · + plpBlp-1 . The elements of SEED should then be set to the following: • SEED(1) = lp • SEED(2) to SEED(lp + 1) = p1 to plp • SEED(lp + 2) = lq • SEED(lp + 3) to SEED(lp + lq + 2) = q1 to qlq • SEED(lp + lq + 3) = ls • SEED(lp + lq + 4) to SEED(lp + lq + ls + 3) = s1 to sls Constraint: If GENID= 5 then 6 SEED(i) > 0, i = 1, 2, · · ·. If GENID = 5 then SEED must take the values described above. INTEGER LSEED [Input/Output] On input: either the length of the seed vector, SEED, or a value = 0 . On output: if LSEED= 0 on input, then LSEED is set to the number of initial values required by the selected generator, and the routine returns. Otherwise LSEED is left unchanged. INTEGER STATE(LSTATE) [Output] On output: the state vector required by all of the supplied distributional and base generators. INTEGER LSTATE [Input/Output] On input: either the length of the state vector, STATE, or a value = 0 . On output: if LSTATE= 0 on input, then LSTATE is set to the minimum length of the state vector STATE for the base generator chosen, and the routine returns. Otherwise LSTATE is left unchanged. Constraint: LSTATE= 0 or the minimum length for the chosen base generator, given by: • GENID = 1: LSTATE= 16, • GENID = 2: LSTATE= 20, • GENID = 3: LSTATE= 633, • GENID = 4: LSTATE= 61, • GENID = 5: LSTATE= lp + lq + ls + 6, where lp, lq and ls are the order of the polynomials used to express the parameters p, q and s respectively. INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then either, or both of LSEED and / or LSTATE have been set to the required length for vectors SEED and STATE respectively. Of the two variables LSEED and LSTATE, only those which had an input value = 0 will have been set. The STATE vector will not have been initialized. If INFO = 0 then the state vector, STATE, has been successfully initialized.Chapter 6: Random Number Generators 78 Example:   C Generate 100 values from the Beta distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A,B DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Beta distribution CALL DRANDBETA(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 79 DRANDINITIALIZEBBS / SRANDINITIALIZEBBS Alternative initialization routine for the Blum-Blum-Shub generator. Unlike the other base generators supplied with the ACML, the Blum-Blum-Shub generator requires two additional parameters, p and q as well as an initial state, s. The parameters p, q and s can be of an arbitrary size. In order to avoid over?ow these values are expressed as a polynomial in B, where B = 2 24 . For example, p can be factored into a polynomial of order lp, with p = p1 + p2B + p3B2 + · · · + plpBlp-1 , similarly q = q1 + q2B + q3B2 + · · · + qlqBlq-1 and s = s1 + s2B + s3B2 + · · · + slsBls-1 . (Note that SRANDINITIALIZEBBS is the single precision version of DRANDINITIALIZEBBS. The argument lists of both routines are identical except that any double precision arguments of DRANDINITIALIZEBBS are replaced in SRANDINITIALIZEBBS by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDINITIALIZEBBS (NBITS,LP,P,LQ,Q,LS,S,STATE,LSTATE, [SUBROUTINE] INFO) INTEGER NBITS [Input] On input: the number of bits, v, to use from each iteration of the BlumBlum-Shub generator. If NBITS < 1 then NBITS = 1. If NBITS > 15 then NBITS = 15. INTEGER LP [Input] On input: the order of the polynomial used to express p (lp). Constraint: 1 = LP = 25. INTEGER P(LP) [Input] On input: the coe?cients of the polynomial used to express p. P(i) = pi , i = 1 to lp. Constraint: 0 = P (i) < 2 24 INTEGER LQ [Input] On input: the order of the polynomial used to express q (lq). Constraint: 1 = LQ = 25. INTEGER Q(LQ) [Input] On input: the coe?cients of the polynomial used to express q. Q(i) = qi , i = 1 to lq. Constraint: 0 = Q (i) < 2 24 INTEGER LS [Input] On input: the order of the polynomial used to express s (ls). Constraint: 1 = LS = 25. INTEGER S(LS) [Input] On input: the coe?cients of the polynomial used to express s. S(i) = si , i = 1 to ls. Constraint: 0 = S (i) < 2 24 INTEGER STATE(*) [Output] On output: the initial state for the Blum-Blum-Shub generator with parameters P,Q,S and NBITS.Chapter 6: Random Number Generators 80 INTEGER LSTATE [Input/Output] On input: either the length of the state vector, STATE, or a value = 0 . On output: if LSTATE= 0 on input, then LSTATE is set to the minimum length of the state vector STATE for the parameters chosen, and the routine returns. Otherwise LSTATE is left unchanged. Constraint: LSTATE= 0 or LSTATE = lp + lq + ls + 6 INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LSTATE has been set to the required length for the STATE vector. If INFO = 0 then the state vector, STATE, has been successfully initialized. 6.1.2 Calling the Base Generators With the exception of the Blum-Blum-Shub generator, there are no interfaces for direct access to the base generators. All of the base generators return variates uniformly distributed over the open interval (0, 1). This functionality can be accessed using the uniform distributional generator DRANDUNIFORM, with parameter A = 0.0 and parameter B = 1.0 (see [DRANDUNIFORM], page 117). The base generator used is, as usual, selected during the initialization process (see Section 6.1.1 [Initialization of the Base Generators], page 75). To directly access the Blum-Blum-Shub generator, use the routine DRANDBLUMBLUMSHUB.Chapter 6: Random Number Generators 81 DRANDBLUMBLUMSHUB / SRANDBLUMBLUMSHUB Allows direct access to the bit stream generated by the Blum-Blum-Shub generator. (Note that SRANDBLUMBLUMSHUB is the single precision version of DRANDBLUMBLUMSHUB. The argument lists of both routines are identical except that any double precision arguments of DRANDBLUMBLUMSHUB are replaced in SRANDBLUMBLUMSHUB by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDBLUMBLUMSHUB (N,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. The total number of bits generated is 24N. Constraint: N= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDBLUMBLUMSHUB STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector holding the bit stream. The least signi?cant 24 bits of each of the X(i) contain the bit stream as generated by the Blum-Blum-Shub generator. The least signi?cant bit of X(1) is the ?rst bit generated, the second least signi?cant bit of X(1) is the second bit generated etc. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. 6.1.3 Basic NAG Generator The NAG basic generator is a linear congruential generator (LCG) and, like all LCGs, has the form: xi = a1xi-1 mod m1, ui = xi m1 , where the ui , i = 1, 2, · · · form the required sequence. The NAG basic generator takes a1 = 13 13 and m1 = 2 59 , which gives a period of approximately 2 57 . This generator has been part of the NAG numerical library [9] since Mark 6 and as such has been widely used. It su?ers from no known problems, other than those due to the lattice structure inherent in all LCGs, and, even though the period is relatively short compared to many of the newer generators, it is su?ciently large for many practical problems.Chapter 6: Random Number Generators 82 6.1.4 Wichmann-Hill Generator The Wichmann-Hill [10] base generator uses a combination of four linear congruential generators (LCGs) and has the form: wi = a1wi-1 mod m1 xi = a2xi-1 mod m2 yi = a3yi-1 mod m3 zi = a4zi-1 mod m4 ui = ( wi m1 + xi m2 + yi m3 + zi m4 ) mod 1, where the ui , i = 1, 2, · · · form the required sequence. There are 273 sets of parameters, {ai , mi : i = 1, 2, 3, 4}, to choose from. These values have been selected so that the resulting generators are independent and have a period of approximately 2 80 [10]. 6.1.5 Mersenne Twister The Mersenne Twister [11] is a twisted generalized feedback shift register generator. The algorithm is as follows: • Set some arbitrary initial values x1, x2, · · · , xr, each consisting of w bits. • Letting A =  0 Iw-1 aw aw-1 · · · a1  , where Iw-1 is the (w - 1) × (w - 1) identity matrix and each of the ai , i = 1 to w take a value of either 0 or 1 (i.e. they can be represented as bits). De?ne xi+r = (xi+s ? (x (w:(l+1)) i |x (l:1) i+1 )A), where x (w:(l+1)) i |x (l:1) i+1 indicates the concatenation of the most signi?cant (upper) w - l bits of xi and the least signi?cant (lower) l bits of xi+1. • Perform the following operations sequentially: z = xi+r ? (xi+r  t1) z = z ? ((z  t2) AND m1) z = z ? ((z  t3) AND m2) z = z ? (z  t4) ui+r = z/(2 w - 1), where t1, t2, t3 and t4 are integers and m1 and m2 are bit-masks and “ t” and “ t” represent a t bit shift right and left respectively, ? is bit-wise exclusively or (xor) operation and “AND” is a bit-wise and operation.Chapter 6: Random Number Generators 83 The ui+r : i = 1, 2, · · · then form a pseudo-random sequence, with ui ? (0, 1), for all i. This implementation of the Mersenne Twister uses the following values for the algorithmic constants: w = 32 a = 0x9908b0df l = 31 r = 624 s = 397 t1 = 11 t2 = 7 t3 = 15 t4 = 18 m1 = 0x9d2c5680 m2 = 0xefc60000 where the notation 0xDD · · · indicates the bit pattern of the integer whose hexadecimal representation is DD · · ·. This algorithm has a period length of approximately 2 19,937 - 1 and has been shown to be uniformly distributed in 623 dimensions. 6.1.6 L’Ecuyer’s Combined Recursive Generator The base generator referred to as L’Ecuyer’s combined recursive generator is referred to as MRG32k3a in [12] and combines two multiple recursive generators: xi = a11xi-1 + a12xi-2 + a13xi-3 mod m1 yi = a21yi-1 + a22yi-2 + a23yi-3 mod m2 zi = xi - yi mod m1 ui = zi m1 , where the ui , i = 1, 2, · · · form the required sequence and a11 = 0, a12 = 1403580, a13 = 810728, m1 = 2 32 - 209, a21 = 527612, a22 = 0, a23 = 1370589 and m2 = 2 32 - 22853. Combining the two multiple recursive generators (MRG) results in sequences with better statistical properties in high dimensions and longer periods compared with those generated from a single MRG. The combined generator described above has a period length of approximately 2 191 6.1.7 Blum-Blum-Shub Generator The Blum-Blum-Shub pseudo random number generator is cryptologically secure under the assumption that the quadratic residuosity problem is intractable [8]. The algorithm consists of the following: • Generate two large and distinct primes, p and q, each congruent to 3 mod 4. De?ne m = pq. • Select a seed s taking a value between 1 and m - 1, such that the greatest common divisor between s and m is 1.Chapter 6: Random Number Generators 84 • Let x0 = s 2 mod m. For i = 1, 2, · · · generate: xi = x 2 i-1 mod m zi = v least signi?cant bits of xi where v= 1 . • The bit-sequence z1, z2, z3, · · · is then the output sequence used. 6.1.8 User Supplied Generators All of the distributional generators described in Section 6.3 [Distribution Generators], page 95 require a base generator which returns a uniformly distributed value in the open interval (0, 1) and ACML includes several such generators (as detailed in Section 6.1 [Base Generators], page 74). However, for greater ?exibility, the ACML routines allow the user to register their own base generator function. This user-supplied generator then becomes the base generator for all of the distribution generators. A user supplied generator comes in the form of two routines, one to initialize the generator and one to generate a set of uniformly distributed values in the open interval (0, 1). These two routines can be named anything, but are referred to as UINI for the initialization routine and UGEN for the generation routine in the following documentation. In order to register a user supplied generator a call to DRANDINITIALIZEUSER must be made. Once registered the generator can be accessed and used in the same manner as the ACML supplied base generators. The speci?cations for DRANDINTIALIZEUSER, UINI and UGEN are given below. See the ACML example programs drandinitializeuser_example.f and drandinitializeuser_c_example.c (Section 2.9 [Examples], page 17) to understand how to use these routines in ACML.Chapter 6: Random Number Generators 85 DRANDINITIALIZEUSER / SRANDINITIALIZEUSER Registers a user supplied base generator so that it can be used with the ACML distributional generators. (Note that SRANDINITIALIZEUSER is the single precision version of DRANDINITIALIZEUSER. The argument lists of both routines are identical except that any double precision arguments of DRANDINITIALIZEUSER are replaced in SRANDINITIALIZEUSER by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDINITIALIZEUSER (UINI,UGEN,GENID,SUBID,SEED,LSEED, [SUBROUTINE] STATE,LSTATE,INFO) SUBROUTINE UINI [Input] On input: routine that will be used to initialize the user supplied generator, UGEN. SUBROUTINE UGEN [Input] On input: user supplied base generator. INTEGER GENID [Input] On input: parameter is passed directly to UINI. Its function therefore depends on that routine. INTEGER SUBID [Input] On input: parameter is passed directly to UINI. Its function therefore depends on that routine. INTEGER SEED(LSEED) [Input] On input: parameter is passed directly to UINI. Its function therefore depends on that routine. INTEGER LSEED [Input/Output] On input: length of the vector SEED. This parameter is passed directly to UINI and therefore its required value depends on that routine. On output: whether LSEED changes will depend on UINI. INTEGER STATE(LSTATE) [Output] On output: the state vector required by all of the supplied distributional generators. The value of STATE returned by UINI has some housekeeping elements appended to the end before being returned by DRANDINITIALIZEUSER. See Section 6.1.8 [User Supplied Generators], page 84 for details about the form of STATE. INTEGER LSTATE [Input/Output] On input: length of the vector STATE. This parameter is passed directly to UINI and therefore its required value depends on that routine. On output: whether LSTATE changes will depend on UINI. If LSTATE= 0 then it is assumed that a request for the required length of STATE has been made. The value of LSTATE returned from UINI is therefore adjusted to allow for housekeeping elements to be added to the end of the STATE vector. This results in the value of LSTATE returned by DRANDINITIALIZEUSER being 3 larger than that returned by UINI.Chapter 6: Random Number Generators 86 INTEGER INFO [Output] On output: INFO is an error indicator. DRANDINITIALIZEUSER will return a value of -6 if the value of LSTATE is between 1 and 3. Otherwise INFO is passed directly back from UINI. It is recommended that the value of INFO returned by UINI is kept consistent with the rest of the ACML, that is if INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then either, or both of LSEED and / or LSTATE have been set to the required length for vectors SEED and STATE respectively and the STATE vector has not have been initialized. If INFO = 0 then the state vector, STATE, has been successfully initialized. Example:   C Generate 100 values from the Uniform distribution using C a user supplied base generator INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,NSKIP,SEED(1),STATE(LSTATE) INTEGER X(N) DOUBLE PRECISION A,B C Set the seed SEED(1) = 1234 C Set the distributional parameters A = 0.0D0 B = 1.0D0 C Initialize the base generator. Here ACMLRNGNB0GND is a user C supplied generator and ACMLRNGNB0INI its initializer CALL DRANDINITIALIZEUSER(ACMLRNGNB0INI,ACMLRNGNB0GND,1,0,SEED, * LSEED,STATE,LSTATE,INFO) C Generate N variates from the Univariate distribution CALL DRANDUNIFORM(N,A,B,STATE,X,LDX,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 87 UINI Speci?cation for a user supplied initialization routine. UINI (GENID,SUBID,SEED,LSEED,STATE,LSTATE,INFO) [SUBROUTINE] INTEGER GENID [Input] On input: the ID associated with the generator. It may be used for anything you like. INTEGER SUBID [Input] On input: the sub-ID associated with the generator. It may be used for anything you like. INTEGER SEED(LSEED) [Input] On input: an array containing the initial seed for your generator. INTEGER LSEED [Input/Output] On input: either the size of the SEED array, or a value < 1. On output: if LSEED < 1 on entry, LSEED must be set to the required size of the SEED array. This allows a caller of UINI to query the required size. INTEGER STATE(LSTATE) [Output] On output: if LSTATE < 1 on entry, STATE should be unchanged. Otherwise, STATE is a state vector holding internal details required by your generator. On exit from UINI, the array STATE must hold the following information: STATE(1) = ESTATE, where ESTATE is your minimum allowed size of array STATE. STATE(2) = MAGIC, where MAGIC is a magic number of your own choice. This can be used by your routine UGEN as a check that UINI has previously been called. STATE(3) = GENID STATE(4) = SUBID STATE(5) ... STATE(ESTATE-1) = internal state values required by your generator routine UGEN; for example, the current value of your seed. STATE(ESTATE) = MAGIC, i.e. the same value as STATE(2). INTEGER LSTATE [Input/Output] On input: either the size of the STATE array, or a value < 1. On output: if LSTATE < 1 on entry, LSTATE should be set to the required size of the STATE array, i.e. the value ESTATE as described above. This allows the caller of UINI to query the required size. Constraint: either LSTATE < 1 or LSTATE= EST AT E . INTEGER INFO [Output] On output: an error code, to be used in whatever way you wish; for example to ?ag an incorrect argument to UINI. If no error is encountered, UINI must set INFO to 0.Chapter 6: Random Number Generators 88 UGEN Speci?cation for a user supplied base generator. UGEN (N,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: the number of random numbers to be generated. INTEGER STATE(*) [Input/Output] On input: the internal state of your generator. DOUBLE PRECISION X(N) [Output] On output: the array of N uniform distributed random numbers, each in the half-open interval [0.0, 1.0) - i.e. 0.0 is a legitimate return value, but 1.0 is not. INTEGER INFO [Output] On output: a ?ag which you can use to signal an error in the call of UGEN - for example, if UGEN is called without being initialized by UINI. 6.2 Multiple Streams It is often advantageous to be able to generate variates from multiple, independent, streams. For example when running a simulation in parallel on several processors. There are four ways of generating multiple streams using the routines available in the ACML: • (a) Using di?erent seeds • (b) Using di?erent sequences • (c) Block-splitting or skipping ahead • (d) Leap frogging The four methods are detailed in the following sections. Of the four, (a) should be avoided in most cases, (b) is only really of any practical use when using the Wichmann-Hill generator, and is then still limited to 273 streams. Both block-splitting and leap-frogging work using the sequence from a single generator, both guarantee that the di?erent sequences will not overlap and both can be scaled to an arbitrary number of streams. Leap-frogging requires no a-priori knowledge about the number of variates being generated, whereas block-splitting requires the user to know (approximately) the maximum number of variates required from each stream. Block-splitting requires no a-priori information on the number of streams required. In contrast leap-frogging requires the user to know the maximum number of streams required, prior to generating the ?rst value. It is known that, dependent on the number of streams required, leap-frogging can lead to sequences with poor statistical properties, especially when applied to linear congruential generators (see Section 6.2.4 [Leap Frogging], page 92 for a brief explanation). In addition, for more complicated generators like a L’Ecuyer’s multiple recursive generator leap-frogging can increase the time required to generate each variate compared to block-splitting. The additional time required by block-splitting occurs at the initialization stage, and not at the variate generation stage. Therefore in most instances block-splitting would be the preferred method for generating multiple sequences.Chapter 6: Random Number Generators 89 6.2.1 Using Di?erent Seeds A di?erent sequence of variates can be generated from the same base generator by initializing the generator using a di?erent set of seeds. Of the four methods for creating multiple streams described here, this is the least satisfactory. As mentioned in Section 6.1.1 [Initialization of the Base Generators], page 75, the statistical properties of the base generators are only guaranteed within sequences, not between sequences. For example, sequences generated from di?erent starting points may overlap if the initial values are not far enough apart. The potential for overlapping sequences is reduced if the period of the generator being used is large. Although there is no guarantee of the independence of the sequences, due to its extremely large period, using the Mersenne Twister with random starting values is unlikely to lead to problems, especially if the number of sequences required is small. This is the only way in which multiple sequences can be generated with the ACML using the Mersenne Twister as the base generator. If the statistical properties of di?erent sequences must be provable then one of the other methods should be adopted. 6.2.2 Using Di?erent Generators Independent sequences of variates can be generated using di?erent base generators for each sequence. For example, sequence 1 can be generated using the NAG basic generator, sequence 2 using the L’Ecuyer’s Combined Recursive generator, sequence 3 using the Mersenne Twister. The Wichmann-Hill generator implemented in the ACML is in fact a series of 273 independent generators. The particular sub-generator being used can be selected using the SUBID variable (see [DRANDINITIALIZE], page 76 for details). Therefore, in total, 277 independent streams can be generated with each using an independent generator (273 Wichmann-Hill generators, and 4 additional base generators). 6.2.3 Skip Ahead Independent sequences of variates can be generated from a single base generator through the use of block-splitting, or skipping-ahead. This method consists of splitting the sequence into k non-overlapping blocks, each of length n, where n is larger than the maximum number of variates required from any of the sequences. For example: x1, x2, · · · , xn, block 1 xn+1, xn+2, · · · , x2n, block 2 x2n+1, x2n+2, · · · , x3n, block 3 etc where x1, x2, · · · is the sequence produced by the generator of interest. Each of the k blocks provide an independent sequence. The block splitting algorithm therefore requires the sequence to be advanced a large number of places. Due to their form this can be done e?ciently for linear congruential generators and multiple congruential generators. The ACML provides block-splitting for the NAG Basic generator, the Wichmann-Hill generators and L’Ecuyer’s Combined Recursive generator.Chapter 6: Random Number Generators 90 DRANDSKIPAHEAD / SRANDSKIPAHEAD Advance a generator N places. (Note that SRANDSKIPAHEAD is the single precision version of DRANDSKIPAHEAD. The argument lists of both routines are identical except that any double precision arguments of DRANDSKIPAHEAD are replaced in SRANDSKIPAHEAD by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDSKIPAHEAD (N,STATE,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of places to skip ahead. Constraint: N= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDSKIPAHEAD STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: The STATE vector for a generator that has been advanced N places. Constraint: The STATE vector must be for either the NAG basic, WichmannHill or L’Ecuyer Combined Recursive base generators. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 91 Example:   C Generate 3 * 100 values from the Uniform distribution C Multiple streams generated using the Skip Ahead method INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,NSKIP INTEGER SEED(1),STATE1(LSTATE),STATE2(LSTATE),STATE3(LSTATE) INTEGER X1(N),X2(N),X3(N) DOUBLE PRECISION A,B C Set the seed SEED(1) = 1234 C Set the distributional parameters A = 0.0D0 B = 1.0D0 C Initialize the STATE1 vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE1,LSTATE,INFO) C Copy the STATE1 vector into other state vectors DO 20 I = 1,LSTATE STATE2(I) = STATE1(I) STATE3(I) = STATE1(I) 20 CONTINUE C Calculate how many places we want to skip, this C should be >> than the number of variates we C wish to generate from each stream NSKIP = N * N C Advance each stream, first does not need changing CALL DRANDSKIPAHEAD(NSKIP,STATE2,INFO) CALL DRANDSKIPAHEAD(2*NSKIP,STATE3,INFO) C Generate 3 sets of N variates from the Univariate distribution CALL DRANDUNIFORM(N,A,B,STATE1,X1,LDX,INFO) CALL DRANDUNIFORM(N,A,B,STATE2,X2,LDX,INFO) CALL DRANDUNIFORM(N,A,B,STATE3,X3,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) X1(I),X2(I),X3(I) 40 CONTINUE Chapter 6: Random Number Generators 92 6.2.4 Leap Frogging Independent sequences of variates can be generated from a single base generator through the use of leap-frogging. This method involves splitting the sequence from a single generator into k disjoint subsequences. For example: Subsequence 1 : x1, xk+1, x2k+1, · · · Subsequence 2 : x2, xk+2, x2k+2, · · · . . . Subsequence k : xk, x2k, x3k, · · · each subsequence is then provides an independent stream. The leap-frog algorithm therefore requires the generation of every kth variate of a sequence. Due to their form this can be done e?ciently for linear congruential generators and multiple congruential generators. The ACML provides leap-frogging for the NAG Basic generator, the Wichmann-Hill generators and L’Ecuyer’s Combined Recursive generator. As an illustrative example, a brief description of the algebra behind the implementation of the leap-frog algorithm (and block-splitting algorithm) for a linear congruential generator (LCG) will be given. A linear congruential generator has the form xi+1 = a1xi mod m1. The recursive nature of a LCG means that xi+v = a1xi+v-1 mod m1 = a1(a1xi+v-2 mod m1) mod m1 = a 2 1xi+v-2 mod m1 = a v 1xi mod m1 The sequence can be quickly advanced v places by multiplying the current state (xi) by a v 1 mod m1, hence allowing block-splitting. Leap-frogging is implemented by using a k 1 , where k is the number of streams required, in place of a1 in the standard LCG recursive formula. In a linear congruential generator the multiplier a1 is constructed so that the generator has good statistical properties in, for example, the spectral test. When using leap-frogging to construct multiple streams this multiplier is replaced with a k 1 , and there is no guarantee that this new multiplier will have suitable properties especially as the value of k depends on the number of streams required and so is likely to change depending on the application. This problem can be emphasised by the lattice structure of LCGs. Note that, due to rounding, a sequence generated using leap-frogging and a sequence constructed by taking every kth value from a set of variates generated without leap-frogging may di?er slightly. These di?erences should only a?ect the least signi?cant digit.Chapter 6: Random Number Generators 93 DRANDLEAPFROG / SRANDLEAPFROG Amend a generator so that it will generate every Kth value. (Note that SRANDLEAPFROG is the single precision version of DRANDLEAPFROG. The argument lists of both routines are identical except that any double precision arguments of DRANDLEAPFROG are replaced in SRANDLEAPFROG by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDLEAPFROG (N,K,STATE,INFO) [SUBROUTINE] INTEGER N [Input] On input: total number of streams being used. Constraint: N> 0. INTEGER K [Input] On input: number of the current stream Constraint: 0< K = N. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDLEAPFROG STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: The STATE vector for a generator that has been advanced K - 1 places and will return every Nth value. Constraint: The STATE array must be for either the NAG basic, WichmannHill or L’Ecuyer Combined Recursive base generators. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 94 Example:   C Generate 3 * 100 values from the Uniform distribution C Multiple streams generated using the Leap Frog method INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO INTEGER SEED(1),STATE1(LSTATE),STATE2(LSTATE),STATE3(LSTATE) INTEGER X1(N),X2(N),X3(N) DOUBLE PRECISION A,B C Set the seed SEED(1) = 1234 C Set the distributional parameters A = 0.0D0 B = 1.0D0 C Initialize the STATE1 vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE1,LSTATE,INFO) C Copy the STATE1 vector into other state vectors DO 20 I = 1,LSTATE STATE2(I) = STATE1(I) STATE3(I) = STATE1(I) 20 CONTINUE C Update each stream so they generate every 3rd value CALL DRANDLEAPFROG(3,1,STATE1,INFO) CALL DRANDLEAPFROG(3,2,STATE2,INFO) CALL DRANDLEAPFROG(3,3,STATE3,INFO) C Generate 3 sets of N variates from the Univariate distribution CALL DRANDUNIFORM(N,A,B,STATE1,X1,LDX,INFO) CALL DRANDUNIFORM(N,A,B,STATE2,X2,LDX,INFO) CALL DRANDUNIFORM(N,A,B,STATE3,X3,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) X1(I),X2(I),X3(I) 40 CONTINUE Chapter 6: Random Number Generators 95 6.3 Distribution Generators 6.3.1 Continuous Univariate Distributions DRANDBETA / SRANDBETA Generates a vector of random variates from a beta distribution with probability density function, f(X), where: f(X) = G(A + B) G(A)G(B) XA-1 (1 - X) B-1 if 0 = X = 1 and A, B > 0.0, otherwise f(X) = 0. (Note that SRANDBETA is the single precision version of DRANDBETA. The argument lists of both routines are identical except that any double precision arguments of DRANDBETA are replaced in SRANDBETA by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDBETA (N,A,B,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION A [Input] On input: ?rst parameter for the distribution. Constraint: A> 0. DOUBLE PRECISION B [Input] On input: second parameter for the distribution. Constraint: B> 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDBETA STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 96 Example:   C Generate 100 values from the Beta distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A,B DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Beta distribution CALL DRANDBETA(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 97 DRANDCAUCHY / SRANDCAUCHY Generates a vector of random variates from a Cauchy distribution with probability density function, f(X), where: f(X) = 1 pB(1 + ( X-A B ) 2 ) (Note that SRANDCAUCHY is the single precision version of DRANDCAUCHY. The argument lists of both routines are identical except that any double precision arguments of DRANDCAUCHY are replaced in SRANDCAUCHY by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDCAUCHY (N,A,B,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION A [Input] On input: median of the distribution. DOUBLE PRECISION B [Input] On input: semi-quartile range of the distribution. Constraint: B= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDCAUCHY STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 98 Example:   C Generate 100 values from the Cauchy distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A,B DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Cauchy distribution CALL DRANDCAUCHY(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 99 DRANDCHISQUARED / SRANDCHISQUARED Generates a vector of random variates from a ? 2 distribution with probability density function, f(X), where: f(X) = X ? 2 -1 e - X 2 2 ? 2 ( ? 2 - 1)! , if X > 0, otherwise f(X) = 0. Here ? is the degrees of freedom, DF. (Note that SRANDCHISQUARED is the single precision version of DRANDCHISQUARED. The argument lists of both routines are identical except that any double precision arguments of DRANDCHISQUARED are replaced in SRANDCHISQUARED by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDCHISQUARED (N,DF,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER DF [Input] On input: degrees of freedom of the distribution. Constraint: DF> 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDCHISQUARED STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 100 Example:   C Generate 100 values from the Chi-squared distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER DF DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) DF C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Chi-squared distribution CALL DRANDCHISQUARED(N,DF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 101 DRANDEXPONENTIAL / SRANDEXPONENTIAL Generates a vector of random variates from an exponential distribution with probability density function, f(X), where: f(X) = e - X A A if X > 0, otherwise f(X) = 0. (Note that SRANDEXPONENTIAL is the single precision version of DRANDEXPONENTIAL. The argument lists of both routines are identical except that any double precision arguments of DRANDEXPONENTIAL are replaced in SRANDEXPONENTIAL by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDEXPONENTIAL (N,A,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION A [Input] On input: exponential parameter. Constraint: A= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDEXPONENTIAL STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 102 Example:   C Generate 100 values from the Exponential distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Exponential distribution CALL DRANDEXPONENTIAL(N,A,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 103 DRANDF / SRANDF Generates a vector of random variates from an F distribution, also called the Fisher’s variance ratio distribution, with probability density function, f(X), where: f(X) = ( µ+?-2 2 )!X µ 2 -1 µ µ 2 ( µ 2 - 1)!( ? 2 - 1)!(1 + µX ? ) µ+? 2 ? µ 2 , if X > 0, otherwise f(X) = 0. Here µ is the ?rst degrees of freedom, (DF1) and ? is the second degrees of freedom, (DF2). (Note that SRANDF is the single precision version of DRANDF. The argument lists of both routines are identical except that any double precision arguments of DRANDF are replaced in SRANDF by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDF (N,DF1,DF2,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER DF1 [Input] On input: ?rst degrees of freedom. Constraint: DF1= 0. INTEGER DF2 [Input] On input: second degrees of freedom. Constraint: DF2= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDF STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 104 Example:   C Generate 100 values from the F distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER DF1,DF2 DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) DF1,DF2 C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the F distribution CALL DRANDF(N,DF1,DF2,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 105 DRANDGAMMA / SRANDGAMMA Generates a vector of random variates from a Gamma distribution with probability density function, f(X), where: f(X) = XA-1 e - X B BAG(A) , if X = 0 and A, B > 0.0, otherwise f(X) = 0. (Note that SRANDGAMMA is the single precision version of DRANDGAMMA. The argument lists of both routines are identical except that any double precision arguments of DRANDGAMMA are replaced in SRANDGAMMA by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDGAMMA (N,A,B,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION A [Input] On input: ?rst parameter of the distribution. Constraint: A> 0. DOUBLE PRECISION B [Input] On input: second parameter of the distribution. Constraint: B> 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDGAMMA STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 106 Example:   C Generate 100 values from the Gamma distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A,B DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Gamma distribution CALL DRANDGAMMA(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 107 DRANDGAUSSIAN / DRANDGAUSSIAN Generates a vector of random variates from a Gaussian distribution with probability density function, f(X), where: f(X) = e - (X-µ) 2 2s2 s v 2p . Here µ is the mean, (XMU) and s 2 the variance, (VAR) of the distribution. (Note that SRANDGAUSSIAN is the single precision version of DRANDGAUSSIAN. The argument lists of both routines are identical except that any double precision arguments of DRANDGAUSSIAN are replaced in SRANDGAUSSIAN by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDGAUSSIAN (N,XMU,VAR,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION XMU [Input] On input: mean of the distribution. DOUBLE PRECISION VAR [Input] On input: variance of the distribution. Constraint: VAR= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDGAUSSIAN STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 108 Example:   C Generate 100 values from the Gaussian distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION XMU,VAR DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) XMU,VAR C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Gaussian distribution CALL DRANDGAUSSIAN(N,XMU,VAR,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 109 DRANDLOGISTIC / SRANDLOGISTIC Generates a vector of random variates from a logistic distribution with probability density function, f(X), where: f(X) = e (X-A) B B(1 + e (X-A) B ) 2 . (Note that SRANDLOGISTIC is the single precision version of DRANDLOGISTIC. The argument lists of both routines are identical except that any double precision arguments of DRANDLOGISTIC are replaced in SRANDLOGISTIC by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDLOGISTIC (N,A,B,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION A [Input] On input: mean of the distribution. DOUBLE PRECISION B [Input] On input: spread of the distribution. B = v 3s/p where s is the standard deviation of the distribution. Constraint: B> 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDLOGISTIC STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 110 Example:   C Generate 100 values from the Logistic distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A,B DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Logistic distribution CALL DRANDLOGISTIC(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 111 DRANDLOGNORMAL / SRANDLOGNORMAL Generates a vector of random variates from a lognormal distribution with probability density function, f(X), where: f(X) = e - (log X-µ) 2 2s2 Xs v 2p , if X > 0, otherwise f(X) = 0. Here µ is the mean, (XMU) and s 2 the variance, (VAR) of the underlying Gaussian distribution. (Note that SRANDLOGNORMAL is the single precision version of DRANDLOGNORMAL. The argument lists of both routines are identical except that any double precision arguments of DRANDLOGNORMAL are replaced in SRANDLOGNORMAL by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDLOGNORMAL (N,XMU,VAR,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION XMU [Input] On input: mean of the underlying Gaussian distribution. DOUBLE PRECISION VAR [Input] On input: variance of the underlying Gaussian distribution. Constraint: VAR= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDLOGNORMAL STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 112 Example:   C Generate 100 values from the Lognormal distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION XMU,VAR DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) XMU,VAR C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Lognormal distribution CALL DRANDLOGNORMAL(N,XMU,VAR,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 113 DRANDSTUDENTST / SRANDSTUDENTST Generates a vector of random variates from a Students T distribution with probability density function, f(X), where: f(X) = (?-1) 2 ! ( ? 2 )! v p?(1 + X2 ? ) (?+1) 2 . Here ? is the degrees of freedom, DF. (Note that SRANDSTUDENTST is the single precision version of DRANDSTUDENTST. The argument lists of both routines are identical except that any double precision arguments of DRANDSTUDENTST are replaced in SRANDSTUDENTST by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDSTUDENTST (N,DF,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER DF [Input] On input: degrees of freedom. Constraint: DF> 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDSTUDENTST STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 114 Example:   C Generate 100 values from the Students T distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER DF DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) DF C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Students T distribution CALL DRANDSTUDENTST(N,DF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 115 DRANDTRIANGULAR / SRANDTRIANGULAR Generates a vector of random variates from a Triangular distribution with probability density function, f(X), where: f(X) = 2(X - XMIN) (XMAX - XMIN)(XMED - XMIN) , if XMIN < X = XMED, else f(X) = 2(XMAX - X) (XMAX - XMIN)(XMAX - XMED) , if XMED < X = XMAX, otherwise f(X) = 0. (Note that SRANDTRIANGULAR is the single precision version of DRANDTRIANGULAR. The argument lists of both routines are identical except that any double precision arguments of DRANDTRIANGULAR are replaced in SRANDTRIANGULAR by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDTRIANGULAR (N,XMIN,XMED,XMAX,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION XMIN [Input] On input: minimum value for the distribution. DOUBLE PRECISION XMED [Input] On input: median value for the distribution. Constraint: XMIN= XMED = XMAX. DOUBLE PRECISION XMAX [Input] On input: maximum value for the distribution. Constraint: XMAX= XMIN. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDTRIANGULAR STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 116 Example:   C Generate 100 values from the Triangular distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION XMIN,XMAX,XMED DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) XMIN,XMAX,XMED C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Triangular distribution CALL DRANDTRIANGULAR(N,XMIN,XMAX,XMED,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 117 DRANDUNIFORM / SRANDUNIFORM Generates a vector of random variates from a Uniform distribution with probability density function, f(X), where: f(X) = 1 B - A . (Note that SRANDUNIFORM is the single precision version of DRANDUNIFORM. The argument lists of both routines are identical except that any double precision arguments of DRANDUNIFORM are replaced in SRANDUNIFORM by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDUNIFORM (N,A,B,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION A [Input] On input: minimum value for the distribution. DOUBLE PRECISION B [Input] On input: maximum value for the distribution. Constraint: B= A. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDUNIFORM STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 118 Example:   C Generate 100 values from the Uniform distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A,B DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Uniform distribution CALL DRANDUNIFORM(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 119 DRANDVONMISES / SRANDVONMISES Generates a vector of random variates from a Von Mises distribution with probability density function, f(X), where: f(X) = e ? cos X 2pI0(?) where X is reduced modulo 2p so that it lies between ±p, and ? is the concentration parameter VK. (Note that SRANDVONMISES is the single precision version of DRANDVONMISES. The argument lists of both routines are identical except that any double precision arguments of DRANDVONMISES are replaced in SRANDVONMISES by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDVONMISES (N,VK,,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION VK [Input] On input: concentration parameter. Constraint: VK> 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDVONMISES STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 120 Example:   C Generate 100 values from the Von Mises distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION VK DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) VK C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Von Mises distribution CALL DRANDVONMISES(N,VK,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 121 DRANDWEIBULL / SRANDWEIBULL Generates a vector of random variates from a Weibull distribution with probability density function, f(X), where: f(X) = AXA-1 e - XA B B , if X > 0, otherwise f(X) = 0. (Note that SRANDWEIBULL is the single precision version of DRANDWEIBULL. The argument lists of both routines are identical except that any double precision arguments of DRANDWEIBULL are replaced in SRANDWEIBULL by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDWEIBULL (N,A,B,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION A [Input] On input: shape parameter for the distribution. Constraint: A> 0. DOUBLE PRECISION B [Input] On input: scale parameter for the distribution. Constraint: B> 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDWEIBULL STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 122 Example:   C Generate 100 values from the Weibull distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION A,B DOUBLE PRECISION X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Weibull distribution CALL DRANDWEIBULL(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 123 6.3.2 Discrete Univariate Distributions DRANDBINOMIAL / SRANDBINOMIAL Generates a vector of random variates from a Binomial distribution with probability, f(X), de?ned by: f(X) = M!P X (1 - P) (M-X) X!(M - 1)! , X = 0, 1, · · · , M (Note that SRANDBINOMIAL is the single precision version of DRANDBINOMIAL. The argument lists of both routines are identical except that any double precision arguments of DRANDBINOMIAL are replaced in SRANDBINOMIAL by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDBINOMIAL (N,M,P,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER M [Input] On input: number of trials. Constraint: M= 0. DOUBLE PRECISION P [Input] On input: probability of success. Constraint: 0= P < 1. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDBINOMIAL STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 124 Example:   C Generate 100 values from the Binomial distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER M DOUBLE PRECISION P INTEGER X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,P C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Binomial distribution CALL DRANDBINOMIAL(N,M,P,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 125 DRANDGEOMETRIC / SRANDGEOMETRIC Generates a vector of random variates from a Geometric distribution with probability, f(X), de?ned by: f(X) = P(1 - P) (X-1) , X = 1, 2, · · · (Note that SRANDGEOMETRIC is the single precision version of DRANDGEOMETRIC. The argument lists of both routines are identical except that any double precision arguments of DRANDGEOMETRIC are replaced in SRANDGEOMETRIC by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDGEOMETRIC (N,P,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION P [Input] On input: distribution parameter. Constraint: 0= P < 1. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDGEOMETRIC STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 126 Example:   C Generate 100 values from the Geometric distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION P INTEGER X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) P C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Geometric distribution CALL DRANDGEOMETRIC(N,P,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 127 DRANDHYPERGEOMETRIC / SRANDHYPERGEOMETRIC Generates a vector of random variates from a Hypergeometric distribution with probability, f(X), de?ned by: f(X) = s!m!(p - s)!(p - m)! X!(s - X)!(m - X)!(p - m - s + X)!p! , if X = max(0, m + s - p), · · · ,min(l, m), otherwise f(X) = 0. Here p is the size of the population, (NP), s is the size of the sample taken from the population, (NS) and m is the number of labeled, or speci?ed, items in the population, (M). (Note that SRANDHYPERGEOMETRIC is the single precision version of DRANDHYPERGEOMETRIC. The argument lists of both routines are identical except that any double precision arguments of DRANDHYPERGEOMETRIC are replaced in SRANDHYPERGEOMETRIC by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDHYPERGEOMETRIC (N,NP,NS,M,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER NP [Input] On input: size of population. Constraint: NP= 0. INTEGER NS [Input] On input: size of sample being taken from population. Constraint: 0= NS = NP. INTEGER M [Input] On input: number of speci?ed items in the population. Constraint: 0= M = NP. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDHYPERGEOMETRIC STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 128 Example:   C Generate 100 values from the Hypergeometric distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER NP,NS,M INTEGER X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) NP,NS,M C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Hypergeometric distribution CALL DRANDHYPERGEOMETRIC(N,NP,NS,M,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 129 DRANDNEGATIVEBINOMIAL / SRANDNEGATIVEBINOMIAL Generates a vector of random variates from a Negative Binomial distribution with probability f(X) de?ned by: f(X) = (M + X - 1)!P X (1 - P) M X!(M - 1)! , X = 0, 1, · · · (Note that SRANDNEGATIVEBINOMIAL is the single precision version of DRANDNEGATIVEBINOMIAL. The argument lists of both routines are identical except that any double precision arguments of DRANDNEGATIVEBINOMIAL are replaced in SRANDNEGATIVEBINOMIAL by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDNEGATIVEBINOMIAL (N,M,P,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER M [Input] On input: number of failures. Constraint: M= 0. DOUBLE PRECISION P [Input] On input: probability of success. Constraint: 0= P < 1. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDNEGATIVEBINOMIAL STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 130 Example:   C Generate 100 values from the Negative Binomial distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER M DOUBLE PRECISION P INTEGER X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,P C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Negative Binomial distribution CALL DRANDNEGATIVEBINOMIAL(N,M,P,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 131 DRANDPOISSON / SRANDPOISSON Generates a vector of random variates from a Poisson distribution with probability f(X) de?ned by: f(X) = ? X e -? X! , X = 0, 1, · · · , where ? is the mean of the distribution, LAMBDA. (Note that SRANDPOISSON is the single precision version of DRANDPOISSON. The argument lists of both routines are identical except that any double precision arguments of DRANDPOISSON are replaced in SRANDPOISSON by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDPOISSON (N,LAMBDA,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER M [Input] On input: number of failures. Constraint: M= 0. DOUBLE PRECISION LAMBDA [Input] On input: mean of the distribution. Constraint: LAMBDA= 0. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDPOISSON STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 132 Example:   C Generate 100 values from the Poisson distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION LAMBDA INTEGER X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) LAMBDA C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Poisson distribution CALL DRANDPOISSON(N,LAMBDA,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 133 DRANDDISCRETEUNIFORM / SRANDDISCRETEUNIFORM Generates a vector of random variates from a Uniform distribution with probability f(X) de?ned by: f(X) = 1 (B - A) , X = A, A + 1, · · · , B (Note that SRANDDISCRETEUNIFORM is the single precision version of DRANDDISCRETEUNIFORM. The argument lists of both routines are identical except that any double precision arguments of DRANDDISCRETEUNIFORM are replaced in SRANDDISCRETEUNIFORM by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDDISCRETEUNIFORM (N,A,B,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER A [Input] On input: minimum for the distribution. INTEGER B [Input] On input: maximum for the distribution. Constraint: B= A. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDDISCRETEUNIFORM STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 134 Example:   C Generate 100 values from the Uniform distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER A,B INTEGER X(N) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) A,B C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Uniform distribution CALL DRANDDISCRETEUNIFORM(N,A,B,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 135 DRANDGENERALDISCRETE / SRANDGENERALDISCRETE Takes a reference vector initialized via one of DRANDBINOMIALREFERENCE, DRANDGEOMETRIC REFERENCE, DRANDHYPERGEOMETRICREFERENCE, DRANDNEGATIVEBINOMIALREFERENCE, DRAND POISSONREFERENCE and generates a vector of random variates from it. (Note that SRANDGENERALDISCRETE is the single precision version of DRANDGENERALDISCRETE. The argument lists of both routines are identical except that any double precision arguments of DRANDGENERALDISCRETE are replaced in SRANDGENERALDISCRETE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDGENERALDISCRETE (N,REF,STATE,X,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION REF(*) [Input] On input: reference vector generated by one of the following: DRANDBINOMIALREFERENCE, DRANDGEOMETRICREFERENCE, DRANDHYPERGEOMETRICREFERENCE, DRANDNEGATIVEBINOMIALREFERENCE, DRANDPOISSONREFERENCE. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDGENERALDISCRETE STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(N) [Output] On output: vector of variates from the speci?ed distribution. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 136 Example:   C Generate 100 values from the Binomial distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER M DOUBLE PRECISION P INTEGER X(N) INTEGER LREF DOUBLE PRECISION REF(1000) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,P C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDBINOMIALREFERENCE(M,P,REF,LREF,INFO) C Generate N variates from the Binomial distribution CALL DRANDGENERALDISCRETE(N,REF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 137 DRANDBINOMIALREFERENCE / SRANDBINOMIALREFERENCE Initializes a reference vector for use with DRANDGENERALDISCRETE. Reference vector is for a Binomial distribution with probability, f(X), de?ned by: f(X) = M!P X (1 - P) (M-X) X!(M - 1)! , X = 0, 1, · · · , M (Note that SRANDBINOMIALREFERENCE is the single precision version of DRANDBINOMIALREFERENCE. The argument lists of both routines are identical except that any double precision arguments of DRANDBINOMIALREFERENCE are replaced in SRANDBINOMIALREFERENCE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDBINOMIALREFERENCE (M,P,REF,LREF,INFO) [SUBROUTINE] INTEGER M [Input] On input: number of trials. Constraint: M= 0. DOUBLE PRECISION P [Input] On input: probability of success. Constraint: 0= P < 1. DOUBLE PRECISION REF(LREF) [Output] On output: if INFO returns with a value of 0 then REF contains reference information required to generate values from a Binomial distribution using DRANDGENERALDISCRETE. INTEGER LREF [Input/Output] On input: either the length of the reference vector REF, or -1. On output: if LREF = -1 on input, then LREF is set to the recommended length of the reference vector and the routine returns. Otherwise LREF is left unchanged. INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LREF has been set to the recommended length for the reference vector REF. If INFO = 0 then the reference vector, REF, has been successfully initialized.Chapter 6: Random Number Generators 138 Example:   C Generate 100 values from the Binomial distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER M DOUBLE PRECISION P INTEGER X(N) INTEGER LREF DOUBLE PRECISION REF(1000) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,P C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDBINOMIALREFERENCE(M,P,REF,LREF,INFO) C Generate N variates from the Binomial distribution CALL DRANDGENERALDISCRETE(N,REF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 139 DRANDGEOMETRICREFERENCE / SRANDGEOMETRICREFERENCE Initializes a reference vector for use with DRANDGENERALDISCRETE. Reference vector is for a Geometric distribution with probability, f(X), de?ned by: f(X) = P(1 - P) (X-1) , X = 1, 2, · · · (Note that SRANDGEOMETRICREFERENCE is the single precision version of DRANDGEOMETRICREFERENCE. The argument lists of both routines are identical except that any double precision arguments of DRANDGEOMETRICREFERENCE are replaced in SRANDGEOMETRICREFERENCE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDGEOMETRICREFERENCE (P,REF,LREF,INFO) [SUBROUTINE] DOUBLE PRECISION P [Input] On input: distribution parameter. Constraint: 0= P < 1. DOUBLE PRECISION REF(LREF) [Output] On output: if INFO returns with a value of 0 then REF contains reference information required to generate values from a Geometric distribution using DRANDGENERALDISCRETE. INTEGER LREF [Input/Output] On input: either the length of the reference vector REF, or -1. On output: if LREF = -1 on input, then LREF is set to the recommended length of the reference vector and the routine returns. Otherwise LREF is left unchanged. INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LREF has been set to the recommended length for the reference vector REF. If INFO = 0 then the reference vector, REF, has been successfully initialized.Chapter 6: Random Number Generators 140 Example:   C Generate 100 values from the Geometric distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION P INTEGER X(N) INTEGER LREF DOUBLE PRECISION REF(1000) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) P C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDGEOMETRICREFERENCE(P,REF,LREF,INFO) C Generate N variates from the Geometric distribution CALL DRANDGENERALDISCRETE(N,REF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 141 DRANDHYPERGEOMETRICREFERENCE / SRANDHYPERGEOMETRICREFERENCE Initializes a reference vector for use with DRANDGENERALDISCRETE. Reference vector is for a Hypergeometric distribution with probability, f(X), de?ned by: f(X) = s!m!(p - s)!(p - m)! X!(s - X)!(m - X)!(p - m - s + X)!p! , if X = max(0, m + s - p), · · · ,min(l, m), otherwise f(X) = 0. Here p is the size of the population, (NP), s is the size of the sample taken from the population, (NS) and m is the number of labeled, or speci?ed, items in the population, (M). (Note that SRANDHYPERGEOMETRICREFERENCE is the single precision version of DRANDHYPERGEOMETRICREFERENCE. The argument lists of both routines are identical except that any double precision arguments of DRANDHYPERGEOMETRICREFERENCE are replaced in SRANDHYPERGEOMETRICREFERENCE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDHYPERGEOMETRICREFERENCE (NP,NS,M,REF,LREF,INFO) [SUBROUTINE] INTEGER NP [Input] On input: size of population. Constraint: NP= 0. INTEGER NS [Input] On input: size of sample being taken from population. Constraint: 0= NS = NP. INTEGER M [Input] On input: number of speci?ed items in the population. Constraint: 0= M = NP. DOUBLE PRECISION REF(LREF) [Output] On output: if INFO returns with a value of 0 then REF contains reference information required to generate values from a Hypergeometric distribution using DRANDGENERALDISCRETE. INTEGER LREF [Input/Output] On input: either the length of the reference vector REF, or -1. On output: if LREF = -1 on input, then LREF is set to the recommended length of the reference vector and the routine returns. Otherwise LREF is left unchanged. INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LREF has been set to the recommended length for the reference vector REF. If INFO = 0 then the reference vector, REF, has been successfully initialized.Chapter 6: Random Number Generators 142 Example:   C Generate 100 values from the Hypergeometric distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER NP, NS,M INTEGER X(N) INTEGER LREF DOUBLE PRECISION REF(1000) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) NP, NS,M C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDHYPERGEOMETRICREFERENCE(NP, NS,M,REF,LREF,INFO) C Generate N variates from the Hypergeometric distribution CALL DRANDGENERALDISCRETE(N,REF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 143 DRANDNEGATIVEBINOMIALREFERENCE / SRANDNEGATIVEBINOMIALREFERENCE Initializes a reference vector for use with DRANDGENERALDISCRETE. Reference vector is for a Negative Binomial distribution with probability f(X) de?ned by: f(X) = (M + X - 1)!P X (1 - P) M X!(M - 1)! , X = 0, 1, · · · (Note that SRANDNEGATIVEBINOMIALREFERENCE is the single precision version of DRANDNEGATIVEBINOMIALREFERENCE. The argument lists of both routines are identical except that any double precision arguments of DRANDNEGATIVEBINOMIALREFERENCE are replaced in SRANDNEGATIVEBINOMIALREFERENCE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDNEGATIVEBINOMIALREFERENCE (M,P,REF,LREF,INFO) [SUBROUTINE] INTEGER M [Input] On input: number of failures. Constraint: M= 0. DOUBLE PRECISION P [Input] On input: probability of success. Constraint: 0= P < 1. DOUBLE PRECISION REF(LREF) [Output] On output: if INFO returns with a value of 0 then REF contains reference information required to generate values from a Negative Binomial distribution using DRANDGENERALDISCRETE. INTEGER LREF [Input/Output] On input: either the length of the reference vector REF, or -1. On output: if LREF = -1 on input, then LREF is set to the recommended length of the reference vector and the routine returns. Otherwise LREF is left unchanged. INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LREF has been set to the recommended length for the reference vector REF. If INFO = 0 then the reference vector, REF, has been successfully initialized.Chapter 6: Random Number Generators 144 Example:   C Generate 100 values from the Negative Binomial distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) INTEGER M DOUBLE PRECISION P INTEGER X(N) INTEGER LREF DOUBLE PRECISION REF(1000) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,P C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDNEGATIVEBINOMIALREFERENCE(M,P,REF,LREF,INFO) C Generate N variates from the Negative Binomial distribution CALL DRANDGENERALDISCRETE(N,REF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 145 DRANDPOISSONREFERENCE / SRANDPOISSONREFERENCE Initializes a reference vector for use with DRANDGENERALDISCRETE. Reference vector is for a Poisson distribution with probability f(X) de?ned by: f(X) = ? X e -? X! , X = 0, 1, · · · , where ? is the mean of the distribution, LAMBDA. (Note that SRANDPOISSONREFERENCE is the single precision version of DRANDPOISSONREFERENCE. The argument lists of both routines are identical except that any double precision arguments of DRANDPOISSONREFERENCE are replaced in SRANDPOISSONREFERENCE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDPOISSONREFERENCE (LAMBDA,REF,LREF,INFO) [SUBROUTINE] INTEGER M [Input] On input: number of failures. Constraint: M= 0. DOUBLE PRECISION LAMBDA [Input] On input: mean of the distribution. Constraint: LAMBDA= 0. DOUBLE PRECISION REF(LREF) [Output] On output: if INFO returns with a value of 0 then REF contains reference information required to generate values from a Poisson distribution using DRANDGENERALDISCRETE. INTEGER LREF [Input/Output] On input: either the length of the reference vector REF, or -1. On output: if LREF = -1 on input, then LREF is set to the recommended length of the reference vector and the routine returns. Otherwise LREF is left unchanged. INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LREF has been set to the recommended length for the reference vector REF. If INFO = 0 then the reference vector, REF, has been successfully initialized.Chapter 6: Random Number Generators 146 Example:   C Generate 100 values from the Poisson distribution INTEGER LSTATE,N PARAMETER (LSTATE=16,N=100) INTEGER I,INFO,SEED(1),STATE(LSTATE) DOUBLE PRECISION LAMBDA INTEGER X(N) INTEGER LREF DOUBLE PRECISION REF(1000) C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) LAMBDA C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDPOISSONREFERENCE(LAMBDA,REF,LREF,INFO) C Generate N variates from the Poisson distribution CALL DRANDGENERALDISCRETE(N,REF,STATE,X,INFO) C Print the results WRITE(6,*) (X(I),I=1,N) Chapter 6: Random Number Generators 147 6.3.3 Continuous Multivariate Distributions DRANDMULTINORMAL / SRANDMULTINORMAL Generates an array of random variates from a Multivariate Normal distribution with probability density function, f(X), where: f(X) = s |C-1 | (2p)M e -(X-µ) T C -1 (X-µ) , where µ is the vector of means, XMU. (Note that SRANDMULTINORMAL is the single precision version of DRANDMULTINORMAL. The argument lists of both routines are identical except that any double precision arguments of DRANDMULTINORMAL are replaced in SRANDMULTINORMAL by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDMULTINORMAL (N,M,XMU,C,LDC,STATE,X,LDX,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER M [Input] On input: number of dimensions for the distribution. Constraint: M= 1. DOUBLE PRECISION XMU(M) [Input] On input: vector of means for the distribution. DOUBLE PRECISION C(LDC,M) [Input] On input: variance / covariance matrix for the distribution. INTEGER LDC [Input] On input: leading dimension of C in the calling routine. Constraint: LDC= N. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDMULTINORMAL STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(LDX,M) [Output] On output: matrix of variates from the speci?ed distribution. INTEGER LDX [Input] On input: leading dimension of X in the calling routine. Constraint: LDX= M. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 148 Example:   C Generate 100 values from the C Multivariate Normal distribution INTEGER LSTATE,N, MM PARAMETER (LSTATE=16,N=100,MM=10) INTEGER I,J,INFO,SEED(1),STATE(LSTATE) INTEGER LDC,LDX,M DOUBLE PRECISION X(N,MM),XMU(MM),C(MM,MM) C Set array sizes LDC = MM LDX = N C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M READ(5,*) (XMU(I),I=1,M) DO 20 I = 1,M READ(5,*) (C(I,J),J=1,M) 20 CONTINUE C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the C Multivariate Normal distribution CALL DRANDMULTINORMAL(N,M,XMU,C,LDC,STATE,X,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) (X(I,J),J=1,M) 40 CONTINUE Chapter 6: Random Number Generators 149 DRANDMULTISTUDENTST / SRANDMULTISTUDENTST Generates an array of random variates from a Multivariate Students T distribution with probability density function, f(X), where: f(X) = G  (?+M) 2  (p?) m 2 G( ? 2 )|C| 1 2 1 + (X - µ) T C -1 (X - µ) ? !- (?+M) 2 , where µ is the vector of means, XMU and ? is the degrees of freedom, DF. (Note that SRANDMULTISTUDENTST is the single precision version of DRANDMULTISTUDENTST. The argument lists of both routines are identical except that any double precision arguments of DRANDMULTISTUDENTST are replaced in SRANDMULTISTUDENTST by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDMULTISTUDENTST (N,M,DF,XMU,C,LDC,STATE,X,LDX,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER M [Input] On input: number of dimensions for the distribution. Constraint: M= 1. INTEGER DF [Input] On input: degrees of freedom. Constraint: DF> 2. DOUBLE PRECISION XMU(M) [Input] On input: vector of means for the distribution. DOUBLE PRECISION C(LDC,M) [Input] On input: matrix de?ning the variance / covariance for the distribution. The variance / covariance matrix is given by ? ?-2C, where ? are the degrees of freedom, DF. INTEGER LDC [Input] On input: leading dimension of C in the calling routine. Constraint: LDC= N. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDMULTISTUDENTST STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(LDX,M) [Output] On output: matrix of variates from the speci?ed distribution.Chapter 6: Random Number Generators 150 INTEGER LDX [Input] On input: leading dimension of X in the calling routine. Constraint: LDX= M. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value. Example:   C Generate 100 values from the C Multivariate Students T distribution INTEGER LSTATE,N, MM PARAMETER (LSTATE=16,N=100,MM=10) INTEGER I,J,INFO,SEED(1),STATE(LSTATE) INTEGER LDC,LDX,M,DF DOUBLE PRECISION X(N,MM),XMU(MM),C(MM,MM) C Set array sizes LDC = MM LDX = N C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,DF READ(5,*) (XMU(I),I=1,M) DO 20 I = 1,M READ(5,*) (C(I,J),J=1,M) 20 CONTINUE C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the C Multivariate Students T distribution CALL DRANDMULTISTUDENTST(N,M,DF,XMU,C,LDC,STATE,X,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) (X(I,J),J=1,M) 40 CONTINUE Chapter 6: Random Number Generators 151 DRANDMULTINORMALR / SRANDMULTINORMALR Generates an array of random variates from a Multivariate Normal distribution using a reference vector initialized by DRANDMULTINORMALREFERENCE. (Note that SRANDMULTINORMALR is the single precision version of DRANDMULTINORMALR. The argument lists of both routines are identical except that any double precision arguments of DRANDMULTINORMALR are replaced in SRANDMULTINORMALR by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDMULTINORMALR (N,REF,STATE,X,LDX,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION REF(*) [Input] On input: a reference vector generated by DRANDMULTINORMALREFERENCE. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDMULTINORMALR STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(LDX,M) [Output] On output: matrix of variates from the speci?ed distribution. INTEGER LDX [Input] On input: leading dimension of X in the calling routine. Constraint: LDX= M. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 152 Example:   C Generate 100 values from the C Multivariate Normal distribution INTEGER LSTATE,N, MM PARAMETER (LSTATE=16,N=100,MM=10) INTEGER I,J,INFO,SEED(1),STATE(LSTATE) INTEGER LDC,LDX,M DOUBLE PRECISION X(N,MM),XMU(MM),C(MM,MM) INTEGER LREF DOUBLE PRECISION REF(1000) C Set array sizes LDC = MM LDX = N C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M READ(5,*) (XMU(I),I=1,M) DO 20 I = 1,M READ(5,*) (C(I,J),J=1,M) 20 CONTINUE C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDMULTINORMALREFERENCE(M,XMU,C,LDC,REF,LREF,INFO) C Generate N variates from the C Multivariate Normal distribution CALL DRANDMULTINORMALR(N,REF,STATE,X,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) (X(I,J),J=1,M) 40 CONTINUE Chapter 6: Random Number Generators 153 DRANDMULTISTUDENTSTR / SRANDMULTISTUDENTSTR Generates an array of random variates from a Multivariate Students T distribution using a reference vector initialized by DRANDMULTISTUDENTSTREFERENCE. (Note that SRANDMULTISTUDENTSTR is the single precision version of DRANDMULTISTUDENTSTR. The argument lists of both routines are identical except that any double precision arguments of DRANDMULTISTUDENTSTR are replaced in SRANDMULTISTUDENTSTR by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDMULTISTUDENTSTR (N,REF,STATE,X,LDX,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. DOUBLE PRECISION REF(*) [Input] On input: a reference vector generated by DRANDMULTISTUDENTSTREFERENCE. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDMULTISTUDENTSTR STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. DOUBLE PRECISION X(LDX,M) [Output] On output: matrix of variates from the speci?ed distribution. INTEGER LDX [Input] On input: leading dimension of X in the calling routine. Constraint: LDX= M. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 154 Example:   C Generate 100 values from the C Multivariate Students T distribution INTEGER LSTATE,N, MM PARAMETER (LSTATE=16,N=100,MM=10) INTEGER I,J,INFO,SEED(1),STATE(LSTATE) INTEGER LDC,LDX,M,DF DOUBLE PRECISION X(N,MM),XMU(MM),C(MM,MM) INTEGER LREF DOUBLE PRECISION REF(1000) C Set array sizes LDC = MM LDX = N C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,DF READ(5,*) (XMU(I),I=1,M) DO 20 I = 1,M READ(5,*) (C(I,J),J=1,M) 20 CONTINUE C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDMULTISTUDENTSTREFERENCE(M,DF,XMU,C,LDC,REF,LREF,INFO) C Generate N variates from the C Multivariate Students T distribution CALL DRANDMULTISTUDENTSTR(N,REF,STATE,X,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) (X(I,J),J=1,M) 40 CONTINUE Chapter 6: Random Number Generators 155 DRANDMULTINORMALREFERENCE / SRANDMULTINORMALREFERENCE Initializes a reference vector for use with DRANDMULTINORMALR. Reference vector is for a Multivariate Normal distribution with probability density function, f(X), where: f(X) = s |C-1 | (2p)M e -(X-µ) T C -1 (X-µ) , where µ is the vector of means, XMU. (Note that SRANDMULTINORMALREFERENCE is the single precision version of DRANDMULTINORMALREFERENCE. The argument lists of both routines are identical except that any double precision arguments of DRANDMULTINORMALREFERENCE are replaced in SRANDMULTINORMALREFERENCE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDMULTINORMALREFERENCE (M,XMU,C,LDC,REF,LREF,INFO) [SUBROUTINE] INTEGER M [Input] On input: number of dimensions for the distribution. Constraint: M= 1. DOUBLE PRECISION XMU(M) [Input] On input: vector of means for the distribution. DOUBLE PRECISION C(LDC,M) [Input] On input: variance / covariance matrix for the distribution. INTEGER LDC [Input] On input: leading dimension of C in the calling routine. Constraint: LDC= N. DOUBLE PRECISION REF(LREF) [Output] On output: if INFO returns with a value of 0 then REF contains reference information required to generate values from a Multivariate Normal distribution using DRANDMULTINORMALR. INTEGER LREF [Input/Output] On input: either the length of the reference vector REF, or -1. On output: if LREF = -1 on input, then LREF is set to the recommended length of the reference vector and the routine returns. Otherwise LREF is left unchanged. INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LREF has been set to the recommended length for the reference vector REF. If INFO = 0 then the reference vector, REF, has been successfully initialized.Chapter 6: Random Number Generators 156 Example:   C Generate 100 values from the C Multivariate Normal distribution INTEGER LSTATE,N, MM PARAMETER (LSTATE=16,N=100,MM=10) INTEGER I,J,INFO,SEED(1),STATE(LSTATE) INTEGER LDC,LDX,M DOUBLE PRECISION X(N,MM),XMU(MM),C(MM,MM) INTEGER LREF DOUBLE PRECISION REF(1000) C Set array sizes LDC = MM LDX = N C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M READ(5,*) (XMU(I),I=1,M) DO 20 I = 1,M READ(5,*) (C(I,J),J=1,M) 20 CONTINUE C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDMULTINORMALREFERENCE(M,XMU,C,LDC,REF,LREF,INFO) C Generate N variates from the C Multivariate Normal distribution CALL DRANDMULTINORMALR(N,REF,STATE,X,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) (X(I,J),J=1,M) 40 CONTINUE Chapter 6: Random Number Generators 157 DRANDMULTISTUDENTSTREFERENCE / SRANDMULTISTUDENTSTREFERENCE Initializes a reference vector for use with DRANDMULTISTUDENTSTR. Reference vector is for a Multivariate Students T distribution with probability density function, f(X), where: f(X) = G  (?+M) 2  (p?) m 2 G( ? 2 )|C| 1 2 1 + (X - µ) T C -1 (X - µ) ? !- (?+M) 2 , where µ is the vector of means, XMU and ? is the degrees of freedom, DF. (Note that SRANDMULTISTUDENTSTREFERENCE is the single precision version of DRANDMULTISTUDENTSTREFERENCE. The argument lists of both routines are identical except that any double precision arguments of DRANDMULTISTUDENTSTREFERENCE are replaced in SRANDMULTISTUDENTSTREFERENCE by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDMULTISTUDENTSREFERENCE [SUBROUTINE] (M,DF,XMU,C,LDC,REF,LREF,INFO) INTEGER M [Input] On input: number of dimensions for the distribution. Constraint: M= 1. INTEGER DF [Input] On input: degrees of freedom. Constraint: DF> 2. DOUBLE PRECISION XMU(M) [Input] On input: vector of means for the distribution. DOUBLE PRECISION C(LDC,M) [Input] On input: matrix de?ning the variance / covariance for the distribution. The variance / covariance matrix is given by ? ?-2C, where ? are the degrees of freedom, DF. INTEGER LDC [Input] On input: leading dimension of C in the calling routine. Constraint: LDC= N. DOUBLE PRECISION REF(LREF) [Output] On output: if INFO returns with a value of 0 then REF contains reference information required to generate values from a Multivariate Students T distribution using DRANDMULTISTUDENTSTR. INTEGER LREF [Input/Output] On input: either the length of the reference vector REF, or -1. On output: if LREF = -1 on input, then LREF is set to the recommended length of the reference vector and the routine returns. Otherwise LREF is left unchanged.Chapter 6: Random Number Generators 158 INTEGER INFO [Output] On output: INFO is an error indicator. If INFO = -i on exit, the i-th argument had an illegal value. If INFO = 1 on exit, then LREF has been set to the recommended length for the reference vector REF. If INFO = 0 then the reference vector, REF, has been successfully initialized. Example:   C Generate 100 values from the C Multivariate Students T distribution INTEGER LSTATE,N, MM PARAMETER (LSTATE=16,N=100,MM=10) INTEGER I,J,INFO,SEED(1),STATE(LSTATE) INTEGER LDC,LDX,M,DF DOUBLE PRECISION X(N,MM),XMU(MM),C(MM,MM) INTEGER LREF DOUBLE PRECISION REF(1000) C Set array sizes LDC = MM LDX = N C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) M,DF READ(5,*) (XMU(I),I=1,M) DO 20 I = 1,M READ(5,*) (C(I,J),J=1,M) 20 CONTINUE C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Initialize the reference vector LREF = 1000 CALL DRANDMULTISTUDENTSTREFERENCE(M,DF,XMU,C,LDC,REF,LREF,INFO) C Generate N variates from the C Multivariate Students T distribution CALL DRANDMULTISTUDENTSTR(N,REF,STATE,X,LDX,INFO) C Print the results DO 40 I = 1,N WRITE(6,*) (X(I,J),J=1,M) 40 CONTINUE Chapter 6: Random Number Generators 159 6.3.4 Discrete Multivariate Distributions DRANDMULTINOMIAL / SRANDMULTINOMIAL Generates a matrix of random variates from a Multinomial distribution with probability, f(X), de?ned by: f(X) = M! QK i=1 Xi ! YK i=1 p Xi i , where X = {X1, X2, · · · , XK}, P = {P1, P2, · · · , PK}, PK i=1 Xi = 1 and PK i=1 Pi = 1. (Note that SRANDMULTINOMIAL is the single precision version of DRANDMULTINOMIAL. The argument lists of both routines are identical except that any double precision arguments of DRANDMULTINOMIAL are replaced in SRANDMULTINOMIAL by single precision arguments - type REAL in FORTRAN or type ?oat in C). DRANDMULTINOMIAL (N,M,P,K,STATE,X,LDX,INFO) [SUBROUTINE] INTEGER N [Input] On input: number of variates required. Constraint: N= 0. INTEGER M [Input] On input: number of trials. Constraint: M= 0. DOUBLE PRECISION P(K) [Input] On input: vector of probabilities for each of the K possible outcomes. Constraint: 0 = Pi = 1, i = 1, 2, · · · , K, PK i=1 Pi = 1. INTEGER K [Input] On input: number of possible outcomes. Constraint: K= 2. INTEGER STATE(*) [Input/Output] The STATE vector holds information on the state of the base generator being used and as such its minimum length varies. Prior to calling DRANDBINOMIAL STATE must have been initialized. See Section 6.1.1 [Initialization of the Base Generators], page 75 for information on initialization of the STATE variable. On input: the current state of the base generator. On output: the updated state of the base generator. INTEGER X(LDX,K) [Output] On output: matrix of variates from the speci?ed distribution. INTEGER LDX [Input] On input: leading dimension of X in the calling routine. Constraint: LDX= M. INTEGER INFO [Output] On output: INFO is an error indicator. On successful exit, INFO contains 0. If INFO = -i on exit, the i-th argument had an illegal value.Chapter 6: Random Number Generators 160 Example:   C Generate 100 values from the Multinomial distribution INTEGER LSTATE,N, MK PARAMETER (LSTATE=16,N=100,MK=10) INTEGER I,J,INFO,SEED(1),STATE(LSTATE) INTEGER LDC,LDX,K,M INTEGER X(N,MK) DOUBLE PRECISION P(MK) C Set array sizes LDX = N C Set the seed SEED(1) = 1234 C Read in the distributional parameters READ(5,*) K READ(5,*) (P(I),I=1,K) C Initialize the STATE vector CALL DRANDINITIALIZE(1,1,SEED,1,STATE,LSTATE,INFO) C Generate N variates from the Multinomial distribution CALL DRANDMULTINOMIAL(N,M,P,K,STATE,X,LDX,INFO) C Print the results DO 20 I = 1,N WRITE(6,*) (X(I,J),J=1,K) 20 CONTINUE Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 161 7 ACML MV: Fast Math and Fast Vector Math Library 7.1 Introduction to ACML MV ACML MV is a library which contains fast and/or vectorized versions of some familiar math library routines such as sin, cos and exp. The routines take advantage of the AMD64 architecture for performance, and so are currently only available with 64-bit versions of ACML. The routines in the library are very accurate over the range of acceptable input arguments. Some of the performance is gained by sacri?cing error handling or the acceptance of certain arguments. It is therefore the responsibility of the caller of these routines to ensure that their arguments are suitable. Furthermore, some of the routines are not callable from highlevel languages at all, but must be called via assembly language; see the documentation of individual routines for details. Hence, these routines are intended to be utilized by knowledgeable users only. 7.1.1 Terminology The individual documentation for a routine states what outputs will be returned for special arguments, and also gives an indication of performance of the routine. In general, special case arguments for any routine will cause a return value in accordance with the C99 language standard [13]. Special case arguments include NaNs and in?nities, as de?ned by the IEEE arithmetic standard [14]. In these documents, NaN means Not a Number, QNaN means Quiet NaN, and SNaN means Signalling NaN. A denormal number is a number which is very tiny (close to the machine arithmetic under- ?ow threshold) and is stored to less precision than a normal number. Due to their special nature, operations on such numbers are often very slow. While such numbers might not necessarily be regarded as special case arguments, for the sake of performance some of the ACML MV routines have been designed not to handle them. This has been noted in the documentation for each ACML MV routine. Performance of a routine is given in machine cycles, and is thus independent of processor speed. Accuracy of a routine is quoted in ulps, where ulp stands for Unit in the Last Place. Since ?oating-point numbers on a computer are limited precision approximations of mathematical numbers, not all real numbers can be represented by machine numbers, and the machine number must in general be rounded to available precision. An ulp is the distance between the two machine numbers that bracket a real number. In this document, the ulp is used as a measure of the error in a returned result when compared with the mathematically exact expected result. Because of the ?nite nature of machine arithmetic, a routine can never in general achieve accuracy of better than 0.5 ulps, and an accuracy of less than 1 ulp is good.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 162 7.1.2 Weak Aliases Some of the functions in ACML MV include a weak alias to an equivalent function in libm. For example, the fastcos function includes a weak alias to cos. If ACML MV is included in the link order before libm, then all calls to the aliased libm function name (e.g. cos) will use the equivalent ACML MV routine (e.g. fastcos). If ACML MV is included in the link order after libm, then all calls to libm functions will use the libm versions. ACML MV routines can always be accessed using their ACML MV names (e.g. fastcos), regardless of link order. 7.1.3 De?ned Types The following types are used to describe the functions contained in this chapter: m128d a pair of double precision values; m128 four single precision values.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 163 7.2 Fast Basic Math Functions This section documents the interfaces to a set of basic mathematical functions. fastcos: fast double precision Cosine double fastcos (double x) Weak alias: cos C Prototype: double fastcos (double x); Inputs: double x - the double precision input value. Outputs: Cosine of x. Fortran Function Interface: DOUBLE PRECISION FASTCOS(X) Inputs: DOUBLE PRECISION X - the double precision input value. Return Value: Cosine of X. Notes: fastcos computes the Cosine function of its argument x. This is a relaxed version of cos, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 2 ulp over most of the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 88 cycles for most valid inputs < 5e5.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 164 fastcosf: fast single precision Cosine float fastcosf (?oat x) Weak alias: cosf C Prototype: ?oat fastcosf (?oat x); Inputs: ?oat x - the single precision input value. Outputs: Single precision Cosine of x. Fortran Function Interface: REAL FASTCOSF(X) Inputs: REAL X - the single precision input value. Return Value: Cosine of X. Notes: fastcosf computes the single precision Cosine function of its argument x. This is a relaxed version of cosf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over most of the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 91 cycles for most valid inputs < 5e5.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 165 fastexp: fast double precision exponential function double fastexp (double x) Weak alias: exp C Prototype: double fastexp (double x); Inputs: double x - the double precision input value. Outputs: e raised to the power x (exponential of x). Fortran Function Interface: DOUBLE PRECISION FASTEXP(X) Inputs: DOUBLE PRECISION X - the double precision input value. Return Value: e raised to the power X (exponential of X). Notes: fastexp computes the double precision exponential function of the input argument x. This is a relaxed version of exp, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -708.5 0 > 709.8 +8 Performance: 75 cycles for most valid inputs.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 166 fastexpf: fast single precision exponential function float fastexpf (?oat x) Weak alias: expf C Prototype: ?oat fastexpf (?oat x); Inputs: ?oat x - the single precision input value. Outputs: e raised to the power x (exponential of x). Fortran Function Interface: REAL FASTEXPF(X) Inputs: REAL X - the single precision input value. Return Value: e raised to the power X (exponential of X). Notes: fastexpf computes the single precision exponential function of the input argument x. This is a relaxed version of expf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -87.5 0 > 88 +8 Performance: 75 cycles for most valid inputs.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 167 fastlog: fast double precision natural logarithm function double fastlog (double x) Weak alias: log C Prototype: double fastlog (double x); Inputs: double x - the double precision input value. Outputs: The natural logarithm (base e) of x. Fortran Function Interface: DOUBLE PRECISION FASTLOG(X) Inputs: DOUBLE PRECISION X - the double precision input value. Return Value: The natural logarithm (base e) of X. Notes: fastlog computes the double precision natural logarithm of its argument x. This is a relaxed version of log, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 97 cycles for most valid inputs. 86 cycles for .97 < x < 1.03Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 168 fastlogf: fast single precision natural logarithm function float fastlogf (?oat x) Weak alias: logf C Prototype: ?oat fastlogf (?oat x); Inputs: ?oat x - the single precision input value. Outputs: The natural logarithm (base e) of x. Fortran Function Interface: REAL FASTLOGF(X) Inputs: REAL X - the single precision input value. Return Value: The natural logarithm (base e) of X. Notes: fastlogf computes the single precision natural logarithm of its argument x. This is a relaxed version of logf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 94 cycles for most valid inputs. 85 cycles for .97 < x < 1.03Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 169 fastlog10: fast double precision base-10 logarithm function double fastlog10 (double x) Weak alias: log10 C Prototype: double fastlog10 (double x); Inputs: double x - the double precision input value. Outputs: The base-10 logarithm of x. Fortran Function Interface: DOUBLE PRECISION FASTLOG10(X) Inputs: DOUBLE PRECISION X - the double precision input value. Return Value: The base-10 logarithm of X. Notes: fastlog10 computes the double precision base-10 logarithm of its argument x. This is a relaxed version of log10, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 112 cycles for most valid inputs.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 170 fastlog10f: fast single precision base-10 logarithm function float fastlog10f (?oat x) Weak alias: log10f C Prototype: ?oat fastlog10f (?oat x); Inputs: ?oat x - the single precision input value. Outputs: The base-10 logarithm of x. Fortran Function Interface: REAL FASTLOG10F(X) Inputs: REAL X - the single precision input value. Return Value: The base-10 logarithm of X. Notes: fastlog10f computes the single precision base-10 logarithm of its argument x. This is a relaxed version of log10f, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 104 cycles for most valid inputs.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 171 fastlog2: fast double precision base-2 logarithm function double fastlog2 (double x) Weak alias: log2 C Prototype: double fastlog2 (double x); Inputs: double x - the double precision input value. Outputs: The base-2 logarithm of x. Fortran Function Interface: DOUBLE PRECISION FASTLOG2(X) Inputs: DOUBLE PRECISION X - the double precision input value. Return Value: The base-2 logarithm of X. Notes: fastlog2 computes the double precision base-2 logarithm of its argument x. This is a relaxed version of log2, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 112 cycles for most valid inputs.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 172 fastlog2f: fast single precision base-2 logarithm function float fastlog2f (?oat x) Weak alias: log2f C Prototype: ?oat fastlog2f (?oat x); Inputs: ?oat x - the single precision input value. Outputs: The base-2 logarithm of x. Fortran Function Interface: REAL FASTLOG2F(X) Inputs: REAL X - the single precision input value. Return Value: The base-2 logarithm of X. Notes: fastlog2f computes the single precision base-2 logarithm of its argument x. This is a relaxed version of log2f, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 107 cycles for most valid inputs.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 173 fastpow: fast double precision power function double fastpow (double x, double y) Weak alias: pow C Prototype: double fastpow (double x, double y); Inputs: double x - the double precision base input value. double y - the double precision exponent input value. Outputs: x raised to the power y. Fortran Function Interface: DOUBLE PRECISION FASTPOW(X,Y) Inputs: DOUBLE PRECISION X - the base value. DOUBLE PRECISION Y - the exponent value. Return Value: X raised to the power Y. Notes: fastpow computes the x raised to the power y in double precision. This is a relaxed version of pow, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs will produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input x Input y Output ±0 y < 0, odd integer ±8 ±0 y < 0, not odd integer +8 ±0 y > 0, odd integer ±0 ±0 y > 0, not odd integer +0 -1 +8 1 +1 y (incl. NaN) 1 x (incl. Nan) ±0 1 x < 0 y, not integer QNaN |x|<1 -8 +8 |x|>1 -8 +0 |x|<1 +8 +0 |x|>1 +8 +8 -8 y < 0, odd integer -0 -8 y < 0, not odd integer +0 -8 y > 0, odd integer -8 -8 y > 0, not odd integer +8Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 174 +8 y < 0, +0 +8 y > 0, +8 NaN y nonzero, NaN x<>1 NaN, NaN Performance: 200 cycles for most valid inputs.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 175 fastpowf: fast single precision power function float fastpowf (?oat x, ?oat y) Weak alias: powf C Prototype: ?oat fastpowf (?oat x, ?oat y); Inputs: ?oat x - the single precision base input value. ?oat y - the single precision exponent input value. Outputs: x raised to the power y. Fortran Function Interface: REAL FASTPOWF(X,Y) Inputs: REAL X - the single precision base value. REAL Y - the single precision exponent value. Return Value: X raised to the power Y. Notes: fastpowf computes the x raised to the power y in single precision. This is a relaxed version of powf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs will produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 0.5 ulp over the valid input range. Special case return values: Input x Input y Output ±0 y < 0, odd integer ±8 ±0 y < 0, not odd integer +8 ±0 y > 0, odd integer ±0 ±0 y > 0, not odd integer +0 -1 +8 1 +1 y (incl. NaN) 1 x (incl. Nan) ±0 1 x < 0 y, not integer QNaN |x|<1 -8 +8 |x|>1 -8 +0 |x|<1 +8 +0 |x|>1 +8 +8 -8 y < 0, odd integer -0 -8 y < 0, not odd integer +0 -8 y > 0, odd integer -8 -8 y > 0, not odd integer +8Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 176 +8 y < 0, +0 +8 y > 0, +8 NaN y nonzero, NaN x<>1 NaN, NaN Performance: 175 cycles for most valid inputs.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 177 fastsin: fast double precision Sine double fastsin (double x) Weak alias: sin C Prototype: double fastsin (double x); Inputs: double x - the double precision input value. Outputs: Sine of x. Fortran Function Interface: DOUBLE PRECISION FASTSIN(X) Inputs: DOUBLE PRECISION X - the double precision input value. Return Value: Sine of X. Notes: fastsin computes the Sine function of its argument x. This is a relaxed version of sin, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 88 cycles for most valid inputs < 5e5.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 178 fastsinf: fast single precision Sine float fastsinf (?oat x) Weak alias: sinf C Prototype: ?oat fastsinf (?oat x); Inputs: ?oat x - the single precision input value. Outputs: Single precision Sine of x. Fortran Function Interface: REAL PRECISION FASTSINF(X) Inputs: REAL PRECISION X - the single precision input value. Return Value: Sine of X. Notes: fastsinf computes the Sine function of its argument x. This is a relaxed version of sinf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 88 cycles for most valid inputs < 5e5.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 179 fastsincos: fast double precision Sine and Cosine void fastsincos (double x, double *s, double *c) Weak alias: sincos C Prototype: void fastsincos (double x, double *s, double *c); Inputs: double x - the double precision input value. Outputs: double *s - Sine of x. double *c - Cosine of x. Fortran Subroutine Interface: SUBROUTINE FASTSINCOS(X,S,C) Inputs: DOUBLE PRECISION X - the double precision input value. Outputs: DOUBLE PRECISION S - Sine of X. DOUBLE PRECISION C - Cosine of X. Notes: fastsincos computes the Sine and Cosine functions of its argument x. This function can provide a signi?cant performance advantage for applications that require both the sine and cosine of an angle, such as axis and matrix rotation. This is a relaxed version of sincos, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 2 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 99 cycles for most valid inputs < 5e5.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 180 fastsincosf: fast single precision Sine and Cosine void fastsincosf (?oat x, ?oat *s, ?oat *c) Weak alias: sincosf C Prototype: void fastsincosf (?oat x, ?oat *s, ?oat *c); Inputs: ?oat x - the single precision input value. Outputs: ?oat *s - Sine of x. ?oat *c - Cosine of x. Fortran Subroutine Interface: SUBROUTINE FASTSINCOSF(X,S,C) Inputs: REAL X - the single precision input value. Outputs: REAL S - Sine of X. REAL C - Cosine of X. Notes: fastsincosf computes the Sine and Cosine functions of its argument x. This function can provide a signi?cant performance advantage for applications that require both the sine and cosine of an angle, such as axis and matrix rotation. This is a relaxed version of sincosf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Error inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 91-102 cycles for most valid inputs < 5e5.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 181 7.3 Fast Vector Math Functions This section documents the interfaces to a set of vector mathematical functions. vrd2 cos: Two-valued double precision Cosine __m128d __vrd2_cos ( m128d x) C Prototype: m128d vrd2 cos( m128d x); Inputs: m128d x - the double precision input value pair. Outputs: m128d y - the double precision Cosine result pair, returned in xmm0. Notes: vrd2 cos computes the Cosine function of two input arguments. This routine accepts a pair of double precision input values passed as a m128d value. The result is the double precision Cosine of both values, returned as a m128d value. This is a relaxed version of cos, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 2 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 120 cycles for most valid inputs < 5e5 (60 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 182 vrd4 cos: Four-valued double precision Cosine __m128d,__m128d __vrd4_cos ( m128d x1, m128d x2) C Prototype: m128d vrd2 cos( m128d x); Note that this function uses a non-standard programming interface. The two m128d inputs, which contain four double precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128d is nonstandard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128d x1 - the ?rst double precision input value pair. m128d x2 - the second double precision input value pair. Outputs: m128d y1 - the ?rst double precision Cosine result pair, returned in xmm0. m128d y2 - second double precision Cosine result pair, returned in xmm1. Notes: vrd4 cos computes the Cosine function of four input arguments. This routine accepts four double precision input values passed as two m128d values. The result is the double precision Cosine of the four values, returned as two m128d values. This is a relaxed version of cos, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 2 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 172 cycles for most valid inputs < 5e5 (43 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 183 vrda cos: Array double precision Cosine void vrda_cos (int n, double *x, double *y) C Prototype: void vrda cos (int n, double *x, double *y) Inputs: int n - the number of values in both the input and output arrays. double *x - pointer to the array of input values. double *y - pointer to the array of output values. Outputs: Cosine for each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRDA COS(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. DOUBLE PRECISION X(N) - array of double precision input values. Outputs: DOUBLE PRECISION Y(N) - array of Cosines of input values. Notes: vrda cos computes the Cosine function for each element of an array of input arguments. This routine accepts an array of double precision input values, computes cos(x) for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of cos, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 2 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 172 cycles for most valid inputs < 5e5 (43 cycles per value), n = 24.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 184 vrs4 cosf: Four-valued single precision Cosine __m128 __vrs4_cosf ( m128 x) C Prototype: m128 vrs4 cosf( m128 x); Inputs: m128 x - the four single precision input values. Outputs: m128 y - the four single precision Cosine results , returned in xmm0. Notes: vrs4 cosf computes the Cosine function of four input arguments. This routine accepts four single precision input values passed as a m128 value. The result is the single precision Cosine of all four values, returned as a m128 value. This is a relaxed version of cosf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 176 cycles for most valid inputs < 5e5 (44 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 185 vrsa cosf: Array single precision Cosine void vrsa_cosf (int n, ?oat *x, ?oat *y) C Prototype: void vrsa cosf (int n, ?oat *x, ?oat *y) Inputs: int n - the number of values in both the input and output arrays. ?oat *x - pointer to the array of input values. ?oat *y - pointer to the array of output values. Outputs: Cosine for each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRSA COSF(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of single precision input values. Outputs: REAL Y(N) - array of Cosines of input values. Notes: vrsa cosf computes the Cosine function for each element of an array of input arguments. This routine accepts an array of single precision input values, computes cosf(x) for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of cosf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 43 cycles per value for most valid inputs < 5e5, n = 24.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 186 vrd2 exp: Two-valued double precision exponential function __m128d __vrd2_exp ( m128d x) C Prototype: m128d vrd2 exp( m128d x); Inputs: m128d x - the double precision input value pair. Outputs: e raised to the power x (exponential of x). m128d y - the double precision exponent result pair, returned in xmm0. Notes: vrd2 exp computes the exponential function of two input arguments. This routine accepts a pair of double precision input values passed as a m128d value. The result is the double precision exponent of both values, returned as a m128d value. This is a relaxed version of exp, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -708.5 0 > 709.8 +8 Performance: 80 cycles for most valid inputs (40 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 187 vrd4 exp: Four-valued double precision exponential function __m128d,__m128d __vrd4_exp ( m128d x1, m128d x2) Prototype: m128d, m128d vrd4 exp( m128d x1, m128d x2); Note that this function uses a non-standard programming interface. The two m128d inputs, which contain four double precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128d is nonstandard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128d x1 - the ?rst double precision input value pair. m128d x2 - the second double precision input value pair. Outputs: m128d y1 - the ?rst double precision exponent result pair, returned in xmm0. m128d y2 - the second double precision exponent result pair, returned in xmm1. Notes: vrd4 exp computes the double precision exponential function of four input arguments. This routine accepts four double precision input values passed as two m128d values. The result is the double precision exponent of the four values, returned as two m128d values. This is a relaxed version of exp, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -708.5 0 > 709.8 +8 Performance: 132 cycles for most valid inputs (33 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 188 vrda exp: Array double precision exponential function void vrda_exp (int n, double *x, double *y) C Prototype: void vrda exp (int n, double *x, double *y) Inputs: int n - the number of values in both the input and output arrays. double *x - pointer to the array of input values. double *y - pointer to the array of output values. Outputs: e raised to the power x (exponential of x) for each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRDA EXP(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. DOUBLE PRECISION X(N) - array of double precision input values. Outputs: DOUBLE PRECISION Y(N) - array of exponentials (e raised to the power x) of input values. Notes: vrda exp computes the double precision exponential function for each element of an array of input arguments. This routine accepts an array of double precision input values, computes the e x for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of exp, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -708.5 0 > 709.8 +8 Performance: 33 cycles per value for valid inputs, n = 24.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 189 vrs4 expf: Four-valued single precision exponential function __m128 __vrs4_expf ( m128 x) C Prototype: m128 vrs4 expf( m128 x); Inputs: m128 x - the four single precision input values. Outputs: e raised to the power x (exponential of x) for each input value x. m128 y - the four single precision exponent results, returned in xmm0. Notes: vrs4 expf computes the double precision exponential function of four input arguments. This routine accepts four single precision input values passed as a m128 value. The result is the single precision exponent of the four values, returned as a m128 value. This is a relaxed version of exp, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -87.5 0 > 88 +8 Performance: 91 cycles for most valid inputs (23 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 190 vrs8 expf: Eight-valued single precision exponential function __m128,__m128 __vrs8_expf ( m128 x1, m128 x2) Prototype: m128, m128 vrs8 expf( m128 x1, m128 x2); Note that this function uses a non-standard programming interface. The two m128 inputs, which contain eight single precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128 is non-standard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128 x1 - the ?rst single precision vector of four input values. m128 x2 - the second single precision vector of four input values. Outputs: m128 y1 - the ?rst four single precision exponent results, returned in xmm0. m128 y2 - the second four single precision exponent results, returned in xmm1. Notes: vrs8 expf computes the single precision exponential function of eight input arguments. This routine accepts eight single precision input values passed as two m128 values. The result is the single precision exponent of the eight values, returned as two m128 values. This is a relaxed version of exp, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -87.5 0 > 88 +8 Performance: 155 cycles for most valid inputs (19 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 191 vrsa expf: Array single precision exponential function void vrsa_expf (int n, ?oat *x, ?oat *y) C Prototype: void vrsa expf (int n, ?oat *x, ?oat *y) Inputs: int n - the number of single precision values in both the input and output arrays. ?oat *x - pointer to the array of input values. ?oat *y - pointer to the array of output values. Outputs: e raised to the power x (exponential of x) for each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRSA EXPF(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of single precision input values. Outputs: REAL Y(N) - array of exponentials (e raised to the power x) of input values. Notes: vrsa expf computes the single precision exponential function for each element of an array of input arguments. This routine accepts an array of single precision input values, computes the e x for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of exp, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN < -87.5 0 > 88 +8 Performance: 15 cycles per value for valid inputs, n = 24.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 192 vrd2 log: Two-valued double precision natural logarithm __m128d __vrd2_log ( m128d x) C Prototype: m128d vrd2 log( m128d x); Inputs: m128d x - the double precision input value pair. Outputs: The natural (base e) logarithm of x. m128d y - the double precision natural logarithm result pair, returned in xmm0. Notes: vrd2 log computes the natural logarithm for each of two input arguments. This routine accepts a pair of double precision input values passed as a m128d value. The result is the double precision natural logarithm of both values, returned as a m128d value. This is a relaxed version of log, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 130 cycles for most valid inputs (65 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 193 vrd4 log: Four-valued double precision natural logarithm __m128d,__m128d __vrd4_log ( m128d x1, m128d x2) Prototype: m128d, m128d vrd4 log( m128d x1, m128d x2); Note that this function uses a non-standard programming interface. The two m128d inputs, which contain four double precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128d is nonstandard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128d x1 - the ?rst double precision input value pair. m128d x2 - the second double precision input value pair. Outputs: The natural (base e) logarithm of x. m128d y1 - the ?rst double precision natural logarithm result pair, returned in xmm0. m128d y2 - the second double precision natural logarithm result pair, returned in xmm1. Notes: vrd4 log computes the natural logarithm for each of four input arguments. This routine accepts four double precision input values passed as two m128d values. The result is the double precision natural logarithm of the four values, returned as two m128d values. This is a relaxed version of log, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 196 cycles for most valid inputs (49 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 194 vrda log: Array double precision natural logarithm void vrda_log (int n, double *x, double *y) C Prototype: void vrda log (int n, double *x, double *y) Inputs: int n - the number of values in both the input and output arrays. double *x - pointer to the array of input values. double *y - pointer to the array of output values. Outputs: The natural (base e) logarithm of each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRDA LOG(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. DOUBLE PRECISION X(N) - array of double precision input values. Outputs: DOUBLE PRECISION Y(N) - array of natural (base e) logarithms of input values. Notes: vrda log computes the double precision natural logarithm for each element of an array of input arguments. This routine accepts an array of double precision input values, computes the natural log for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of log, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 51 cycles per value for valid inputs, n = 24.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 195 vrs4 logf: Four-valued single precision natural logarithm __m128 __vrs4_logf ( m128 x) C Prototype: m128 vrs4 logf( m128 x); Inputs: m128 x - the single precision input values. Outputs: The natural (base e) logarithm of x. m128 y - the single precision natural logarithm results, returned in xmm0. Notes: vrs4 logf computes the natural logarithm for each of four input arguments. This routine accepts four single precision input values passed as a m128 value. The result is the single precision natural logarithm of all four values, returned as a m128 value. This is a relaxed version of logf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 124 cycles for most valid inputs (31 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 196 vrs8 logf: Eight-valued single precision natural logarithm __m128,__m128 __vrs8_logf ( m128 x1, m128 x2) Prototype: m128, m128 vrs8 logf( m128 x1, m128 x2); Note that this function uses a non-standard programming interface. The two m128 inputs, which contain eight single precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128 is non-standard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128 x1 - the ?rst single precision input value pair. m128 x2 - the second single precision input value pair. Outputs: The natural (base e) logarithm of x. m128 y1 - the ?rst single precision natural logarithm result pair, returned in xmm0. m128 y2 - the second single precision natural logarithm result pair, returned in xmm1. Notes: vrs8 logf computes the natural logarithm for each of eight input arguments. This routine accepts eight single precision input values passed as two m128 values. The result is the single precision natural logarithm of the eight values, returned as two m128 values. This is a relaxed version of logf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 200 cycles for most valid inputs (25 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 197 vrsa logf: Array single precision natural logarithm void vrsa_logf (int n, ?oat *x, ?oat *y) C Prototype: void vrsa logf (int n, ?oat *x, ?oat *y) Inputs: int n - the number of values in both the input and output arrays. ?oat *x - pointer to the array of input values. ?oat *y - pointer to the array of output values. Outputs: The natural (base e) logarithm of each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRSA LOGF(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of single precision input values. Outputs: REAL Y(N) - array of natural (base e) logarithms of input values. Notes: vrsa logf computes the single precision natural logarithm for each element of an array of input arguments. This routine accepts an array of single precision input values, computes the natural log for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of logf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 26 cycles per value for valid inputs, n = 24.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 198 vrd2 log10: Two-valued double precision base-10 logarithm __m128d __vrd2_log10 ( m128d x) C Prototype: m128d vrd2 log10( m128d x); Inputs: m128d x - the double precision input value pair. Outputs: The base-10 logarithm of x. m128d y - the double precision base-10 logarithm result pair, returned in xmm0. Notes: vrd2 log10 computes the base-10 logarithm for each of two input arguments. This routine accepts a pair of double precision input values passed as a m128d value. The result is the double precision base-10 logarithm of both values, returned as a m128d value. This is a relaxed version of log10, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 142 cycles for most valid inputs (71 cycles per value), longer for input values very close to 1.0.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 199 vrd4 log10: Four-valued double precision base-10 logarithm __m128d,__m128d __vrd4_log10 ( m128d x1, m128d x2) Prototype: m128d, m128d vrd4 log10( m128d x1, m128d x2); Note that this function uses a non-standard programming interface. The two m128d inputs, which contain four double precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128d is nonstandard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128d x1 - the ?rst double precision input value pair. m128d x2 - the second double precision input value pair. Outputs: The base-10 logarithm of x. m128d y1 - the ?rst double precision base-10 logarithm result pair, returned in xmm0. m128d y2 - the second double precision base-10 logarithm result pair, returned in xmm1. Notes: vrd4 log10 computes the base-10 logarithm for each of four input arguments. This routine accepts four double precision input values passed as two m128d values. The result is the double precision base-10 logarithm of the four values, returned as two m128d values. This is a relaxed version of log10, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 235 cycles for most valid inputs (59 cycles per value), longer for input values very close to 1.0.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 200 vrda log10: Array double precision base-10 logarithm void vrda_log10 (int n, double *x, double *y) C Prototype: void vrda log10 (int n, double *x, double *y) Inputs: int n - the number of values in both the input and output arrays. double *x - pointer to the array of input values. double *y - pointer to the array of output values. Outputs: The base-10 logarithm of each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRDA LOG10(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. DOUBLE PRECISION X(N) - array of double precision input values. Outputs: DOUBLE PRECISION Y(N) - array of base-10 logarithms of input values. Notes: vrda log10 computes the double precision base-10 logarithm for each element of an array of input arguments. This routine accepts an array of double precision input values, computes the base- 10 log for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of log10, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 54 cycles per value for valid inputs, n = 24, longer for input values very close to 1.0.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 201 vrs4 log10f: Four-valued single precision base-10 logarithm __m128 __vrs4_log10f ( m128 x) Prototype: m128 vrs4 log10f( m128 x); Inputs: m128 x - the four single precision inputs. Outputs: The base-10 logarithm of x. m128 y - the four single precision base-10 logarithm results, returned in xmm0. Notes: vrs4 log10f computes the base-10 logarithm for each of four input arguments. This routine accepts four single precision input values passed as a m128 value. The result is the single precision base-10 logarithm of the four values, returned as a m128 value. This is a relaxed version of log10, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 141 cycles for most valid inputs (35 cycles per value), longer for input values very close to 1.0.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 202 vrs8 log10f: Eight-valued single precision base-10 logarithm __m128,__m128 __vrs8_log10f ( m128 x1, m128 x2) Prototype: m128, m128 vrs8 log10f( m128 x1, m128 x2); Note that this function uses a non-standard programming interface. The two m128 inputs, which contain eight single precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128 is non-standard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128 x1 - the ?rst set of four single precision input values. m128 x2 - the second set of four single precision input values. Outputs: The base-10 logarithm of x. m128 y1 - the ?rst set of four single precision base-10 logarithm results, returned in xmm0. m128 y2 - the second set of four single precision base-10 logarithm results, returned in xmm1. Notes: vrs8 log10f computes the base-10 logarithm for each of eight input arguments. This routine accepts eight single precision input values passed as two m128 values. The result is the single precision base-10 logarithm of the eight values, returned as two m128 values. This is a relaxed version of log10f, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 231 cycles for most valid inputs (29 cycles per value), longer for input values very close to 1.0.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 203 vrsa log10f: Array single precision base-10 logarithm void vrsa_log10f (int n, ?oat *x, ?oat *y) C Prototype: void vrsa log10f (int n, ?oat *x, ?oat *y) Inputs: int n - the number of values in both the input and output arrays. ?oat *x - pointer to the array of input values. ?oat *y - pointer to the array of output values. Outputs: The base-10 logarithm of each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRSA LOG10F(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of single precision input values. Outputs: REAL Y(N) - array of base-10 logarithms of input values. Notes: vrsa log10f computes the single precision base-10 logarithm for each element of an array of input arguments. This routine accepts an array of single precision input values, computes the base- 10 log for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of log10f, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 28 cycles per value for valid inputs, n = 24, longer for input values very close to 1.0.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 204 vrd2 log2: Two-valued double precision base-2 logarithm __m128d __vrd2_log2 ( m128d x) C Prototype: m128d vrd2 log2( m128d x); Inputs: m128d x - the double precision input value pair. Outputs: The base-2 logarithm of x. m128d y - the double precision base-2 logarithm result pair, returned in xmm0. Notes: vrd2 log2 computes the base-2 logarithm for each of two input arguments. This routine accepts a pair of double precision input values passed as a m128d value. The result is the double precision base-2 logarithm of both values, returned as a m128d value. This is a relaxed version of log2, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 142 cycles for most valid inputs (71 cycles per value), longer for input values very close to 1.0.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 205 vrd4 log2: Four-valued double precision base-2 logarithm __m128d,__m128d __vrd4_log2 ( m128d x1, m128d x2) Prototype: m128d, m128d vrd4 log2( m128d x1, m128d x2); Note that this function uses a non-standard programming interface. The two m128d inputs, which contain four double precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128d is nonstandard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128d x1 - the ?rst double precision input value pair. m128d x2 - the second double precision input value pair. Outputs: The base-2 logarithm of x. m128d y1 - the ?rst double precision base-2 logarithm result pair, returned in xmm0. m128d y2 - the second double precision base-2 logarithm result pair, returned in xmm1. Notes: vrd4 log2 computes the base-2 logarithm for each of four input arguments. This routine accepts four double precision input values passed as two m128d values. The result is the double precision base-2 logarithm of the four values, returned as two m128d values. This is a relaxed version of log2, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 235 cycles for most valid inputs (59 cycles per value), longer for input values very close to 1.0.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 206 vrda log2: Array double precision base-2 logarithm void vrda_log2 (int n, double *x, double *y) C Prototype: void vrda log2 (int n, double *x, double *y) Inputs: int n - the number of values in both the input and output arrays. double *x - pointer to the array of input values. double *y - pointer to the array of output values. Outputs: The base-2 logarithm of each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRDA LOG2(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. DOUBLE PRECISION X(N) - array of double precision input values. Outputs: DOUBLE PRECISION Y(N) - array of base-2 logarithms of input values. Notes: vrda log2 computes the double precision base-2 logarithm for each element of an array of input arguments. This routine accepts an array of double precision input values, computes the base- 2 log for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of log2, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 54 cycles per value for valid inputs, n = 24, longer for input values very close to 1.0.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 207 vrs4 log2f: Four-valued single precision base-2 logarithm __m128 __vrs4_log2f ( m128 x) Prototype: m128 vrs4 log2f( m128 x); Inputs: m128 x - the four single precision inputs. Outputs: The base-2 logarithm of x. m128 y - the four single precision base-2 logarithm results, returned in xmm0. Notes: vrs4 log2f computes the base-2 logarithm for each of four input arguments. This routine accepts four single precision input values passed as a m128 value. The result is the single precision base-2 logarithm of the four values, returned as a m128 value. This is a relaxed version of log2, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 141 cycles for most valid inputs (35 cycles per value), longer for input values very close to 1.0.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 208 vrs8 log2f: Eight-valued single precision base-2 logarithm __m128,__m128 __vrs8_log2f ( m128 x1, m128 x2) Prototype: m128, m128 vrs8 log2f( m128 x1, m128 x2); Note that this function uses a non-standard programming interface. The two m128 inputs, which contain eight single precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128 is non-standard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128 x1 - the ?rst set of four single precision input values. m128 x2 - the second set of four single precision input values. Outputs: The base-2 logarithm of x. m128 y1 - the ?rst set of four single precision base-2 logarithm results, returned in xmm0. m128 y2 - the second set of four single precision base-2 logarithm results, returned in xmm1. Notes: vrs8 log2f computes the base-2 logarithm for each of eight input arguments. This routine accepts eight single precision input values passed as two m128 values. The result is the single precision base-2 logarithm of the eight values, returned as two m128 values. This is a relaxed version of log2f, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 203 cycles for most valid inputs (25 cycles per value), longer for input values very close to 1.0.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 209 vrsa log2f: Array single precision base-2 logarithm void vrsa_log2f (int n, ?oat *x, ?oat *y) C Prototype: void vrsa log2f (int n, ?oat *x, ?oat *y) Inputs: int n - the number of values in both the input and output arrays. ?oat *x - pointer to the array of input values. ?oat *y - pointer to the array of output values. Outputs: The base-2 logarithm of each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRSA LOG2F(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of single precision input values. Outputs: REAL Y(N) - array of base-2 logarithms of input values. Notes: vrsa log2f computes the single precision base-2 logarithm for each element of an array of input arguments. This routine accepts an array of single precision input values, computes the base-2 log for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of log2f, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output ±0 -8 negative QNaN QNaN same QNaN SNaN same NaN converted to QNaN +8 +8 -8 QNaN Performance: 29 cycles per value for valid inputs, n = 24, longer for input values very close to 1.0.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 210 vrs4 powf: Four-valued single precision power function __m128 __vrs4_powf ( m128 x, m128 y) C Prototype: m128 vrs4 powf( m128 x, m128 y); Inputs: m128 x - the single precision input base values. m128 y - the single precision input exponent values. Outputs: m128 z - the single precision results of each x raised to the y power, returned in xmm0. Notes: vrs4 powf() computes the single precision x raised to the y power for four pairs of input arguments. This routine accepts four single precision input value pairs passed as m128 values. The result is the x raised to the y power for all four input pairs, returned as a m128 value. This is a relaxed version of powf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 0.5 ulp over the valid input range. Special case return values: Input x Input y Output ±0 y < 0, odd integer ±8 ±0 y < 0, not odd integer +8 ±0 y > 0, odd integer ±0 ±0 y > 0, not odd integer +0 -1 +8 1 +1 y (incl. NaN) 1 x (incl. Nan) ±0 1 x < 0 y, not integer QNaN |x|<1 -8 +8 |x|>1 -8 +0 |x|<1 +8 +0 |x|>1 +8 +8 -8 y < 0, odd integer -0 -8 y < 0, not odd integer +0 -8 y > 0, odd integer -8 -8 y > 0, not odd integer +8 +8 y < 0, +0 +8 y > 0, +8 NaN y nonzero, NaN x<>1 NaN, NaN Performance: 400 cycles for most valid inputs (100 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 211 vrsa powf: Array single precision power function void vrsa_powf (int n, ?oat *x, ?oat *y, ?oat *z) C Prototype: void vrsa powf(int n, ?oat *x, ?oat *y, ?oat *z); Inputs: ?oat *x - pointer to the array of single precision input x values. ?oat *y - pointer to the array of single precision input y values. ?oat *z - pointer to the array of single precision output values. int n - the number of single precision values in both the input and output arrays. Outputs: x raised to the y value for each array pair, ?lled into the z array. Fortran Subroutine Interface: VRSA POWF(INTEGER*4 N, REAL*4 X(), REAL*4 Y(), REAL*4 Z()) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of real x input values. REAL Y(N) - array of real y input values. Outputs: REAL Z(N) - array of real result values. Notes: vrsa powf() computes x to the y power in single precision for each pair of elements in the x and y input arrays. This routine accepts an array of single precision input x values and an arrayi of single precision input y values, computes x^y for each input value pair, and stores the result in the array pointed to by the z input. This is a relaxed version of powf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 0.5 ulp over the valid input range. Special case return values: Input x Input y Output ±0 y < 0, odd integer ±8 ±0 y < 0, not odd integer +8 ±0 y > 0, odd integer ±0 ±0 y > 0, not odd integer +0 -1 +8 1 +1 y (incl. NaN) 1 x (incl. Nan) ±0 1 x < 0 y, not integer QNaN |x|<1 -8 +8Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 212 |x|>1 -8 +0 |x|<1 +8 +0 |x|>1 +8 +8 -8 y < 0, odd integer -0 -8 y < 0, not odd integer +0 -8 y > 0, odd integer -8 -8 y > 0, not odd integer +8 +8 y < 0, +0 +8 y > 0, +8 NaN y nonzero, NaN x<>1 NaN, NaN Performance: 107 cycles per value for valid inputs, n = 24.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 213 vrs4 powxf: Four-valued single precision power function with constant y __m128 __vrs4_powxf ( m128 x,?oat y) C Prototype: m128 vrs4 powxf( m128 x,?oat y); Inputs: m128 x - the single precision input base values. ?oat y - the common single precision input exponent value. Outputs: m128 z - the single precision results of each x raised to the y power, returned in xmm0. Notes: vrs4 powxf() computes the single precision x raised to the y power for four input x arguments and a constant y input value. This routine accepts four single precision input values passed as an m128 value. The y value is passed as one single precision value. The result is the x raised to the y power for all four input values, returned as a m128 value. This is a relaxed version of powxf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 0.5 ulp over the valid input range. Special case return values: Input x Input y Output ±0 y < 0, odd integer ±8 ±0 y < 0, not odd integer +8 ±0 y > 0, odd integer ±0 ±0 y > 0, not odd integer +0 -1 +8 1 +1 y (incl. NaN) 1 x (incl. Nan) ±0 1 x < 0 y, not integer QNaN |x|<1 -8 +8 |x|>1 -8 +0 |x|<1 +8 +0 |x|>1 +8 +8 -8 y < 0, odd integer -0 -8 y < 0, not odd integer +0 -8 y > 0, odd integer -8 -8 y > 0, not odd integer +8 +8 y < 0, +0 +8 y > 0, +8 NaN y nonzero, NaN x<>1 NaN, NaNChapter 7: ACML MV: Fast Math and Fast Vector Math Library 214 Performance: 372 cycles for most valid inputs (93 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 215 vrsa powxf: Array single precision power function, constant y void vrsa_powf (int n, ?oat *x, ?oat y, ?oat *z) C Prototype: void vrsa powxf(int n, ?oat *x, ?oat y, ?oat *z); Inputs: int n - the number of single precision values in both the x input and output arrays. ?oat *x - pointer to the array of single precision input x values. ?oat *z - pointer to the array of single precision output values. ?oat y - the constant single precision input y value. Outputs: x raised to the y value for each x array value, ?lled into the z array Fortran Subroutine Interface: VRSA POWF(INTEGER*4 N, REAL*4 X(), REAL*4 Y, REAL*4 Z()) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of real x input values. REAL Y - the constant single precision input y value. Outputs: REAL Z(N) - array of real result values. Notes: vrsa powxf() computes x to the y power in single precision for each element in the x input arrays, using a constant y. This routine accepts an array of single precision input x values and one single precision input y value, computes x^y for each x input value, and stores the result in the array pointed to by the z pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of powf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 0.5 ulp over the valid input range. Special case return values: Input x Input y Output ±0 y < 0, odd integer ±8 ±0 y < 0, not odd integer +8 ±0 y > 0, odd integer ±0 ±0 y > 0, not odd integer +0 -1 +8 1Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 216 +1 y (incl. NaN) 1 x (incl. Nan) ±0 1 x < 0 y, not integer QNaN |x|<1 -8 +8 |x|>1 -8 +0 |x|<1 +8 +0 |x|>1 +8 +8 -8 y < 0, odd integer -0 -8 y < 0, not odd integer +0 -8 y > 0, odd integer -8 -8 y > 0, not odd integer +8 +8 y < 0, +0 +8 y > 0, +8 NaN y nonzero, NaN x<>1 NaN, NaN Performance: 115 cycles per value for valid inputs, n = 24.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 217 vrd2 sin: Two-valued double precision Sine __m128d __vrd2_sin ( m128d x) C Prototype: m128d vrd2 sin( m128d x); Inputs: m128d x - the double precision input value pair. Outputs: m128d y - the double precision Sine result pair, returned in xmm0. Notes: vrd2 sin computes the Sine function of two input arguments. This routine accepts a pair of double precision input values passed as a m128d value. The result is the double precision Sine of both values, returned as a m128d value. This is a relaxed version of sin, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 120 cycles for most valid inputs < 5e5 (60 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 218 vrd4 sin: Four-valued double precision Sine __m128d,__m128d __vrd4_sin ( m128d x1, m128d x2) C Prototype: m128d vrd4 sin( m128d x); Note that this function uses a non-standard programming interface. The two m128d inputs, which contain four double precision values, are passed by the AMD64 C ABI in registers xmm0, and xmm1. The corresponding results are returned in xmm0 and xmm1. The use of xmm1 to return a m128d is nonstandard, and this function can not be called directly from C. It can be called directly from assembly language. It is intended for internal use by vectorizing compilers, that may be able to take advantage of the non-standard calling interface. Inputs: m128d x1 - the ?rst double precision input value pair. m128d x2 - the second double precision input value pair. Outputs: m128d y1 - the ?rst double precision Sine result pair, returned in xmm0. m128d y2 - second double precision Sine result pair, returned in xmm1. Notes: vrd4 sin computes the Sine function of four input arguments. This routine accepts four double precision input values passed as two m128d values. The result is the double precision Sine of the four values, returned as two m128d values. This is a relaxed version of sin, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. This routine may return slightly worse than 1 ulp for very large values between 4e5 and 5e5. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 172 cycles for most valid inputs < 5e5 (43 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 219 vrda sin: Array double precision Sine void vrda_sin (int n, double *x, double *y) C Prototype: void vrda sin (int n, double *x, double *y) Inputs: int n - the number of values in both the input and output arrays. double *x - pointer to the array of input values. double *y - pointer to the array of output values. Outputs: Sine for each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRDA SIN(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. DOUBLE PRECISION X(N) - array of double precision input values. Outputs: DOUBLE PRECISION Y(N) - array of Sines of input values. Notes: vrda sin computes the Sine function for each element of an array of input arguments. This routine accepts an array of double precision input values, computes sin(x) for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of sin, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. This routine may return slightly worse than 1 ulp for very large values between 4e5 and 5e5. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 172 cycles for most valid inputs < 5e5 (43 cycles per value), n = 24.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 220 vrs4 sinf: Four-valued single precision Sine __m128 __vrs4_sinf ( m128 x) C Prototype: m128 vrs4 sinf( m128 x); Inputs: m128 x - the four single precision inputs. Outputs: m128 y - the four single precision Sine results, returned in xmm0. Notes: vrs4 sinf computes the Sine function of four input arguments. This routine accepts four single precision input values passed as a m128 value. The result is the single precision Sine of the four values, returned as a m128 value. This is a relaxed version of sinf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. This routine may return slightly worse than 1 ulp for very large values between 4e5 and 5e5. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 176 cycles for most valid inputs < 5e5 (44 cycles per value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 221 vrsa sinf: Array single precision Sine void vrsa_sinf (int n, ?oat *x, ?oat *y) C Prototype: void vrsa sinf (int n, ?oat *x, ?oat *y) Inputs: int n - the number of values in both the input and output arrays. ?oat *x - pointer to the array of input values. ?oat *y - pointer to the array of output values. Outputs: Sine for each x value, ?lled into the y array. Fortran Subroutine Interface: SUBROUTINE VRSA SINF(N,X,Y) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of single precision input values. Outputs: REAL Y(N) - array of Sines of input values. Notes: vrsa sinf computes the Sine function for each element of an array of input arguments. This routine accepts an array of single precision input values, computes sin(x) for each input value, and stores the result in the array pointed to by the y pointer input. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of sinf, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 43 cycles per value for most valid inputs < 5e5, n = 24.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 222 vrd2 sincos: Two-valued double precision Sine and Cosine void __vrd2_sincos ( m128d x, m128d* S, m128d* C) C Prototype: void vrd2 sincos( m128d x, m128d* S, m128d* C)); Inputs: m128d x - the double precision input value pair. Outputs: (Sine of x and Cosine of x.) m128d *S - Pointer to the double precision Sine result pair. m128d *C - Pointer to the double precision Cosine result pair. Notes: vrd2 sincos computes the Sine and Cosine functions of two input arguments. This routine accepts a pair of double precision input values passed as a m128d value. The result is the double precision Sin and Cosine of both values, returned as a m128d value. This is a relaxed version of sincos, suitable for use with fastmath compiler ?ags or application not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 2 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 154 cycles for most valid inputs < 5e5 (77 cycles per Sine and Cosine of a value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 223 vrda sincos: Array double precision Sine and Cosine void vrda_sincos (int n, double *x, double *ys, double *yc) C Prototype: void vrda sincos (int n, double *x, double *ys, double *yc) Inputs: int n - the number of values in both the input and output arrays. double *x - pointer to the array of input values. double *ys - pointer to the array of sin output values. double *yc - pointer to the array of cos output values. Outputs: Sine for each x value, ?lled into the ys array. Cosine for each x value, ?lled into the yc array. Fortran Subroutine Interface: SUBROUTINE VRDA SINCOS(N,X,YS,YC) Inputs: INTEGER N - the number of values in both the input and output arrays. DOUBLE PRECISION X(N) - array of double precision input values. Outputs: DOUBLE PRECISION YS(N) - array of Sines of input values. DOUBLE PRECISION YC(N) - array of Cosines of input values. Notes: vrda sincos computes the Sine and Cosine functions for each element of an array of input arguments. This routine accepts an array of double precision input values, computes sincos(x) for each input value, and stores the results in the arrays pointed to by the ys and yc pointer inputs. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of sincos, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 2 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance:Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 224 180 cycles for most valid inputs < 5e5 (43 cycles per Sin and Cos of a value), n = 24.Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 225 vrs4 sincosf: Four-valued single precision Sine and Cosine void __vrs4_sincosf ( m128 x, m128* S, m128* C) C Prototype: void vrs4 sincosf( m128 x, m128* S, m128* C)); Inputs: m128 x - the single precision input value pair. Outputs: (Sine of x and Cosine of x.) m128 *S - Pointer to the single precision Sine result pair. m128 *C - Pointer to the single precision Cosine result pair. Notes: vrs4 sincosf computes the Sine and Cosine functions of four input arguments. This routine accepts four single precision input values passed as a m128 value. The result is the single precision Sin and Cosine of all four values, returned as a m128 value. This is a relaxed version of sincosf, suitable for use with fastmath compiler ?ags or application not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 220 cycles for most valid inputs < 5e5 (55 cycles per Sine and Cosine of a value).Chapter 7: ACML MV: Fast Math and Fast Vector Math Library 226 vrsa sincosf: Array single precision Sine and Cosine void vrsa_sincosf (int n, ?oat *x, ?oat *ys, ?oat *yc) C Prototype: void vrsa sincosf (int n, ?oat *x, ?oat *ys, ?oat *yc) Inputs: int n - the number of values in both the input and output arrays. ?oat *x - pointer to the array of input values. ?oat *ys - pointer to the array of sin output values. ?oat *yc - pointer to the array of cos output values. Outputs: Sine for each x value, ?lled into the ys array. Cosine for each x value, ?lled into the yc array. Fortran Subroutine Interface: SUBROUTINE VRSA SINCOSF(N,X,YS,YC) Inputs: INTEGER N - the number of values in both the input and output arrays. REAL X(N) - array of single precision input values. Outputs: REAL YS(N) - array of Sines of input values. REAL YC(N) - array of Cosines of input values. Notes: vrsa sincosf computes the Sine and Cosine functions for each element of an array of input arguments. This routine accepts an array of single precision input values, computes sincos(x) for each input value, and stores the results in the arrays pointed to by the ys and yc pointer inputs. It is the responsibility of the calling program to allocate/deallocate enough storage for the output array. This is a relaxed version of sincos, suitable for use with fastmath compiler ?ags or applications not requiring full error handling. Denormal inputs may produce unpredictable results. Special case inputs produce C99 return values. The routine is accurate to better than 1 ulp over the valid input range. Special case return values: Input Output QNaN same QNaN SNaN same NaN converted to QNaN +8 QNaN -8 QNaN Performance: 53 cycles per value for most valid inputs < 5e5, n = 24.Chapter 8: References 227 8 References • [1] C.L. Lawson, R.J. Hanson, D. Kincaid, and F.T. Krogh, Basic linear algebra subprograms for Fortran usage, ACM Trans. Maths. Soft., 5 (1979), pp. 308–323. • [2] J.J. Dongarra, J. Du Croz, S. Hammarling, and R.J. Hanson, An extended set of FORTRAN basic linear algebra subroutines, ACM Trans. Math. Soft., 14 (1988), pp. 1–17. • [3] J.J. Dongarra, J. Du Croz, I.S. Du?, and S. Hammarling, A set of level 3 basic linear algebra subprograms, ACM Trans. Math. Soft., 16 (1990), pp. 1–17. • [4] David S. Dodson, Roger G. Grimes, John G. Lewis, Sparse Extensions to the FORTRAN Basic Linear Algebra Subprograms, ACM Trans. Math. Soft., 17 (1991), pp. 253–263. • [5] E. Anderson, Z. Bai, C. Bischof, S. Blackford, J. Demmel, J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, A. McKenney, and D. Sorensen, LAPACK User’s Guide, SIAM, Philidelphia, (1999). • [6] D. E. Knuth, The Art of Computer Programming Addison-Wesley, 1997. • [7] J. Banks, Handbook on Simulation, Wiley, 1998. • [8] A. Menezes, P. van Oorschot, and S. Vanstone, Handbook of Applied Cryptography, Chapter 5, CRC Press, 1996. • [9] Chapter Introduction G05 - Random Number Generators The NAG Fortran Library Manual, Mark 21 Numerical Algorithms Group, 2005. • [10] N. M. Maclaren, The generation of multiple independent sequences of pseudorandom numbers, Appl. Statist., 1989, 38, 351-359. • [11] M. Matsumoto and T. Nishimura, Mersenne twister: A 623-dimensionally equidistributed uniform pseudorandom number generator, ACM Transactions on Modelling and Computer Simulations, 1998. • [12] P. L’Ecuyer, Good parameter sets for combined multiple recursive random number generators, Operations Research, 1999, 47, 159-164. • [13] Programming languages - C - ISO/IEC 9899:1999 • [14] IEEE Standard for Binary Floating-Point Arithmetic (ANSI/IEEE Std 754-1985) • [15] P. L’Ecuyer and R. Simard, TestU01: A Software Library in ANSI C for Empirical Testing of Random Number Generators, Departement d’Informatique et de Recherche Operationnelle, Universite de Montreal, 2002. Software and user’s guide available at http://www.iro.umontreal.ca/~lecuyerSubject Index 228 Subject Index 2 2D FFT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3 3D FFT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 A accessing ACML (Compaq Visual Fortran under 32-bit Windows). . . . . . . . . . . . . . . . . . . . . . . . . 11 accessing ACML (GNU g77/gcc under 32-bit Windows). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 accessing ACML (GNU g77/gcc under Linux). . . 4 accessing ACML (GNU gfortran/gcc under Linux) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 accessing ACML (Intel Fortran/Microsoft C under 32-bit Windows). . . . . . . . . . . . . . . . . . . . . . . . . 10 accessing ACML (Intel Fortran/Microsoft C under 64-bit Windows). . . . . . . . . . . . . . . . . . . . . . . . . 12 accessing ACML (Intel ifort under Linux). . . . . . . 7 accessing ACML (Linux) . . . . . . . . . . . . . . . . . . . . . . 4 accessing ACML (NAGware f95 compiler under Linux). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 accessing ACML (other compilers under Linux). . 8 accessing ACML (PathScale pathf90/pathcc under Linux). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 accessing ACML (PGI pgf77/pgf90/Microsoft C under 32-bit Windows). . . . . . . . . . . . . . . . . . . . 9 accessing ACML (PGI pgf77/pgf90/pgcc or Microsoft C under 64-bit Windows). . . . . . . 11 accessing ACML (PGI pgf77/pgf90/pgcc under Linux). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 accessing ACML (Salford ftn95 under 32-bit Windows). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 accessing ACML (Solaris) . . . . . . . . . . . . . . . . . . . . 13 accessing ACML (Sun f95/cc under Solaris). . . . 13 accessing ACML under Windows . . . . . . . . . . . . . . . 8 accessing the base generators . . . . . . . . . . . . . . . . . 80 ACML C Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . 14 ACML FORTRAN interfaces . . . . . . . . . . . . . . . . . 14 ACML installation test. . . . . . . . . . . . . . . . . . . . . . . 17 ACML performance examples . . . . . . . . . . . . . . . . . 17 ACML version information . . . . . . . . . . . . . . . . . . . 16 ACML MV (ACML vector math functions). . . 161 ACML MV types . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 B base generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 base generator, basic NAG generator . . . . . . . . . . 81 base generator, blum-blum-shub . . . . . . . . . . . . . . 83 base generator, calling . . . . . . . . . . . . . . . . . . . . . . . 80 base generator, de?nition . . . . . . . . . . . . . . . . . . . . . 74 base generator, initialization . . . . . . . . . . . . . . . . . . 75 base generator, L’Ecuyer’s combined recursive generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 base generator, Mersenne twister . . . . . . . . . . . . . . 82 base generator, recommendation . . . . . . . . . . . . . . 74 base generator, user supplied . . . . . . . . . . . . . . . . . 84 base generator, Wichmann-Hill . . . . . . . . . . . . . . . 82 basic NAG base generator . . . . . . . . . . . . . . . . . . . . 81 beta distribution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 binomial distribution. . . . . . . . . . . . . . . . . . . . . . . . 123 binomial distribution, using reference vector . . 137 BLAS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 blum-blum-shub generator . . . . . . . . . . . . . . . . . . . . 83 BRNG, de?nition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 C C interfaces in ACML. . . . . . . . . . . . . . . . . . . . . . . . 14 calling the base generators . . . . . . . . . . . . . . . . . . . . 80 cauchy distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 97 chi-squared distribution . . . . . . . . . . . . . . . . . . . . . . 99 complex FFT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 continuous multivariate distribution, gaussian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 continuous multivariate distribution, gaussian, using reference vector . . . . . . . . . . . . . . . . . . . 151 continuous multivariate distribution, normal . . 147 continuous multivariate distribution, normal, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 151 continuous multivariate distribution, students t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 continuous multivariate distribution, students t, using reference vector . . . . . . . . . . . . . . . . . . . 153 continuous univariate distribution, beta . . . . . . . 95 continuous univariate distribution, cauchy . . . . . 97 continuous univariate distribution, chi-squared . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 continuous univariate distribution, exponential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 continuous univariate distribution, f . . . . . . . . . . 103 continuous univariate distribution, ?sher’s variance ratio distribution . . . . . . . . . . . . . . . . . . . . . . . 103 continuous univariate distribution, gamma . . . . 105 continuous univariate distribution, gaussian. . . 107 continuous univariate distribution, logistic . . . . 109 continuous univariate distribution, lognormal. . 111 continuous univariate distribution, normal . . . . 107 continuous univariate distribution, students t. . 113 continuous univariate distribution, t. . . . . . . . . . 113 continuous univariate distribution, triangular . . 115 continuous univariate distribution, uniform . . . 117 continuous univariate distribution, von mises . . 119 continuous univariate distribution, weibull . . . . 121 copying a generator . . . . . . . . . . . . . . . . . . . . . . . . . . 75 cryptologically secure, de?nition . . . . . . . . . . . . . . 74 cryptologically secure, generator . . . . . . . . . . . . . . 83Subject Index 229 D determining the best ACML version for your system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 discrete multivariate distribution, multinomial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 discrete univariate distribution, binomial . . . . . 123 discrete univariate distribution, binomial, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 137 discrete univariate distribution, geometric . . . . 125 discrete univariate distribution, geometric, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 139 discrete univariate distribution, hypergeometric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 discrete univariate distribution, hypergeometric, using reference vector . . . . . . . . . . . . . . . . . . . 141 discrete univariate distribution, negative binomial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 discrete univariate distribution, negative binomial, using reference vector . . . . . . . . . . . . . . . . . . . 143 discrete univariate distribution, poisson. . . . . . . 131 discrete univariate distribution, poisson, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 145 discrete univariate distribution, uniform . . . . . . 133 distribution generator, de?nition . . . . . . . . . . . . . . 74 E example programs . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 exponential distribution . . . . . . . . . . . . . . . . . . . . . 101 F f distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 fast basic math functions . . . . . . . . . . . . . . . . . . . . 163 Fast Fourier Transforms . . . . . . . . . . . . . . . . . . . . . . 24 feedback shift generator . . . . . . . . . . . . . . . . . . . . . . 82 FFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 FFT e?ciency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 FFT of multiple complex sequences . . . . . . . . . . . 34 FFT of multiple Hermitian sequences. . . . . . . . . . 72 FFT of multiple real sequences . . . . . . . . . . . . . . . 68 FFT of single complex sequence. . . . . . . . . . . . . . . 27 FFT of single Hermitian sequence . . . . . . . . . . . . . 70 FFT of single real sequence . . . . . . . . . . . . . . . . . . . 66 FFT plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 ?sher’s variance ratio distribution . . . . . . . . . . . . 103 FORTRAN interfaces in ACML. . . . . . . . . . . . . . . 14 G gamma distribution . . . . . . . . . . . . . . . . . . . . . . . . . 105 gaussian distribution (multivariate) . . . . . . . . . . 147 gaussian distribution (univariate) . . . . . . . . . . . . 107 gaussian distribution, multivariate, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 general information . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 generalized feedback shift generator . . . . . . . . . . . 82 generating discrete variates from a reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 geometric distribution. . . . . . . . . . . . . . . . . . . . . . . 125 geometric distribution, using reference vector . . 139 H Hermitian data sequences (FFT). . . . . . . . . . . . . . 65 hypergeometric distribution . . . . . . . . . . . . . . . . . 127 hypergeometric distribution, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 I IEEE exceptions and LAPACK . . . . . . . . . . . . . . . 23 initialization of a generator . . . . . . . . . . . . . . . . . . . 75 installation test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 INTEGER*8 arguments . . . . . . . . . . . . . . . . . . . . . . 15 introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 L L’Ecuyer’s combined recursive generator . . . . . . . 83 language interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 LAPACK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 LAPACK blocking factors . . . . . . . . . . . . . . . . . . . . 21 LAPACK reference sources . . . . . . . . . . . . . . . . . . . 20 libm names. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 library manual. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 library version information . . . . . . . . . . . . . . . . . . . 16 linear congruential generator, basic NAG generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 linear congruential generator, Wichmann-Hill . . 82 linking with ACML . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 linking with Linux ACML . . . . . . . . . . . . . . . . . . . . . 4 linking with Solaris ACML . . . . . . . . . . . . . . . . . . . 13 linking with Windows ACML . . . . . . . . . . . . . . . . . . 8 logistic distribution . . . . . . . . . . . . . . . . . . . . . . . . . 109 lognormal distribution . . . . . . . . . . . . . . . . . . . . . . 111 M Mersenne twister . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Mersenne twister, multiple streams . . . . . . . . . . . . 89 multinomial distribution . . . . . . . . . . . . . . . . . . . . 159 multiple recursive generator, L’Ecuyer’s combined recursive generator . . . . . . . . . . . . . . . . . . . . . . 83 multiple streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 multiple streams, block splitting . . . . . . . . . . . . . . 89 multiple streams, L’Ecuyer’s combined recursive generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89, 92 multiple streams, leap frogging. . . . . . . . . . . . . . . . 92 multiple streams, Mersenne twister . . . . . . . . . . . . 89 multiple streams, NAG basic generator . . . . . 89, 92 multiple streams, skip ahead. . . . . . . . . . . . . . . . . . 89 multiple streams, using di?erent generators . . . . 89 multiple streams, using di?erent seeds . . . . . . . . . 89 multiple streams, Wichmann-Hill generator . . . 89, 92 multivariate distribution, gaussian . . . . . . . . . . . 147 multivariate distribution, gaussian, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 multivariate distribution, multinomial . . . . . . . . 159Subject Index 230 multivariate distribution, normal. . . . . . . . . . . . . 147 multivariate distribution, normal, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 multivariate distribution, students t. . . . . . . . . . 149 multivariate distribution, students t, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 153 N negative binomial distribution . . . . . . . . . . . . . . . 129 negative binomial distribution, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 normal distribution (multivariate). . . . . . . . . . . . 147 normal distribution (univariate). . . . . . . . . . . . . . 107 normal distribution, multivariate, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 P performance example programs . . . . . . . . . . . . . . . 17 period of a random number generator, de?nition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 plan, default, FFTs . . . . . . . . . . . . . . . . . . . . . . . . . . 26 plan, generated, FFTs. . . . . . . . . . . . . . . . . . . . . . . . 26 poisson distribution . . . . . . . . . . . . . . . . . . . . . . . . . 131 poisson distribution, using reference vector . . . 145 PRNG, de?nition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 pseudo-random number, de?nition . . . . . . . . . . . . 74 Q QRNG, de?nition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 quasi-random number, de?nition . . . . . . . . . . . . . . 74 R random bit stream . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 real data sequences (FFT). . . . . . . . . . . . . . . . . . . . 65 real FFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 reference vector, binomial distribution . . . . . . . . 137 reference vector, gaussian (multivariate) . . . . . . 155 reference vector, generating discrete variates from . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 reference vector, geometric distribution . . . . . . . 139 reference vector, hypergeometric distribution. . 141 reference vector, negative binomial distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 reference vector, normal (multivariate) . . . . . . . 155 reference vector, poisson distribution . . . . . . . . . 145 reference vector, students t (multivariate). . . . . 157 retrieving the state of a generator . . . . . . . . . . . . . 75 S saving the state of a generator . . . . . . . . . . . . . . . . 75 seed, de?nition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 size of integer arguments . . . . . . . . . . . . . . . . . . . . . 15 sparse BLAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 students t distribution . . . . . . . . . . . . . . . . . . . . . . 113 students t distribution (multivariate). . . . . . . . . 149 students t distribution, multivariate, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 153 T t distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 triangular distribution . . . . . . . . . . . . . . . . . . . . . . 115 U uniform distribution (continuous) . . . . . . . . . . . . 117 uniform distribution (discrete). . . . . . . . . . . . . . . 133 univariate distribution, beta . . . . . . . . . . . . . . . . . . 95 univariate distribution, binomial . . . . . . . . . . . . . 123 univariate distribution, binomial, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 univariate distribution, cauchy . . . . . . . . . . . . . . . . 97 univariate distribution, chi-squared. . . . . . . . . . . . 99 univariate distribution, exponential . . . . . . . . . . 101 univariate distribution, f . . . . . . . . . . . . . . . . . . . . 103 univariate distribution, ?sher’s variance ratio. . 103 univariate distribution, gamma . . . . . . . . . . 105, 107 univariate distribution, geometric . . . . . . . . . . . . 125 univariate distribution, geometric, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 univariate distribution, hypergeometric . . . . . . . 127 univariate distribution, hypergeometric, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 141 univariate distribution, logistic. . . . . . . . . . . . . . . 109 univariate distribution, lognormal . . . . . . . . . . . . 111 univariate distribution, negative binomial. . . . . 129 univariate distribution, negative binomial, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . 143 univariate distribution, normal. . . . . . . . . . . . . . . 107 univariate distribution, poisson . . . . . . . . . . . . . . 131 univariate distribution, poisson, using reference vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 univariate distribution, students t. . . . . . . . . . . . 113 univariate distribution, t . . . . . . . . . . . . . . . . . . . . 113 univariate distribution, triangular . . . . . . . . . . . . 115 univariate distribution, uniform (continuous). . 117 univariate distribution, uniform (discrete) . . . . 133 univariate distribution, von mises . . . . . . . . . . . . 119 univariate distribution, weibull. . . . . . . . . . . . . . . 121 user supplied generators . . . . . . . . . . . . . . . . . . . . . . 84 V vector math functions . . . . . . . . . . . . . . . . . . . . . . . 181 von mises distribution. . . . . . . . . . . . . . . . . . . . . . . 119 W weak aliases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 weibull distribution . . . . . . . . . . . . . . . . . . . . . . . . . 121 Wichmann-Hill base generator . . . . . . . . . . . . . . . . 82 Wichmann-Hill, multiple streams . . . . . . . . . . . . . 89Routine Index 231 Routine Index __vrd2_cos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 __vrd2_exp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 __vrd2_log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 __vrd2_log10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 __vrd2_log2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 __vrd2_sin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 __vrd2_sincos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 __vrd4_cos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 __vrd4_exp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 __vrd4_log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 __vrd4_log10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 __vrd4_log2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 __vrd4_sin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 __vrs4_cosf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 __vrs4_expf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 __vrs4_log10f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 __vrs4_log2f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 __vrs4_logf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 __vrs4_powf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 __vrs4_powxf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 __vrs4_sincosf . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 __vrs4_sinf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 __vrs8_expf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 __vrs8_log10f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 __vrs8_log2f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 __vrs8_logf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 A acmlinfo. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 ACMLINFO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 acmlversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 ACMLVERSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 C CFFT1D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 CFFT1DX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 CFFT1M . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 CFFT1MX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 CFFT2D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 CFFT2DX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 CFFT3D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 CFFT3DX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 CFFT3DY. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 CSFFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 CSFFTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 D DRANDBETA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 DRANDBINOMIAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 DRANDBINOMIALREFERENCE . . . . . . . . . . . . . . . . . . . 137 DRANDBLUMBLUMSHUB . . . . . . . . . . . . . . . . . . . . . . . . . . 81 DRANDCAUCHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 DRANDCHISQUARED . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 DRANDDISCRETEUNIFORM. . . . . . . . . . . . . . . . . . . . . . 133 DRANDEXPONENTIAL . . . . . . . . . . . . . . . . . . . . . . . . . . 101 DRANDF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 DRANDGAMMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 DRANDGAUSSIAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 DRANDGENERALDISCRETE. . . . . . . . . . . . . . . . . . . . . . 135 DRANDGEOMETRIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 DRANDGEOMETRICREFERENCE . . . . . . . . . . . . . . . . . . 139 DRANDHYPERGEOMETRIC. . . . . . . . . . . . . . . . . . . . . . . 127 DRANDHYPERGEOMETRICREFERENCE . . . . . . . . . . . . . 141 DRANDINITIALIZE . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 DRANDINITIALIZEBBS . . . . . . . . . . . . . . . . . . . . . . . . . 79 DRANDINITIALIZEUSER. . . . . . . . . . . . . . . . . . . . . . . . 85 DRANDLEAPFROG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 DRANDLOGISTIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 DRANDLOGNORMAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 DRANDMULTINOMIAL . . . . . . . . . . . . . . . . . . . . . . . . . . 159 DRANDMULTINORMAL . . . . . . . . . . . . . . . . . . . . . . . . . . 147 DRANDMULTINORMALR . . . . . . . . . . . . . . . . . . . . . . . . . 151 DRANDMULTINORMALREFERENCE . . . . . . . . . . . . . . . . 155 DRANDMULTISTUDENTSREFERENCE . . . . . . . . . . . . . . 157 DRANDMULTISTUDENTST. . . . . . . . . . . . . . . . . . . . . . . 149 DRANDMULTISTUDENTSTR. . . . . . . . . . . . . . . . . . . . . . 153 DRANDNEGATIVEBINOMIAL . . . . . . . . . . . . . . . . . . . . 129 DRANDNEGATIVEBINOMIALREFERENCE . . . . . . . . . . . 143 DRANDPOISSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 DRANDPOISSONREFERENCE . . . . . . . . . . . . . . . . . . . . 145 DRANDSKIPAHEAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 DRANDSTUDENTST . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 DRANDTRIANGULAR . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 DRANDUNIFORM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 DRANDVONMISES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 DRANDWEIBULL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 DZFFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 DZFFTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 F fastcos. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 fastcosf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 fastexp. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 fastexpf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 fastlog. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 fastlog10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 fastlog10f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 fastlog2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 fastlog2f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 fastlogf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 fastpow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 fastpowf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 fastsin. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 fastsincos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 fastsincosf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 fastsinf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178Routine Index 232 I ILAENVSET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 S SCFFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 SCFFTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 SRANDBINOMIALREFERENCE . . . . . . . . . . . . . . . . . . . 137 SRANDCHISQUARED . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 SRANDGEOMETRICREFERENCE . . . . . . . . . . . . . . . . . . 139 SRANDBETA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 SRANDBINOMIAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 SRANDBLUMBLUMSHUB . . . . . . . . . . . . . . . . . . . . . . . . . . 81 SRANDCAUCHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 SRANDDISCRETEUNIFORM. . . . . . . . . . . . . . . . . . . . . . 133 SRANDEXPONENTIAL . . . . . . . . . . . . . . . . . . . . . . . . . . 101 SRANDF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 SRANDGAMMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 SRANDGAUSSIAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 SRANDGENERALDISCRETE. . . . . . . . . . . . . . . . . . . . . . 135 SRANDGEOMETRIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 SRANDHYPERGEOMETRIC. . . . . . . . . . . . . . . . . . . . . . . 127 SRANDHYPERGEOMETRICREFERENCE . . . . . . . . . . . . . 141 SRANDINITIALIZE . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 SRANDINITIALIZEBBS . . . . . . . . . . . . . . . . . . . . . . . . . 79 SRANDINITIALIZEUSER. . . . . . . . . . . . . . . . . . . . . . . . 85 SRANDLEAPFROG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 SRANDLOGISTIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 SRANDLOGNORMAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 SRANDMULTINOMIAL . . . . . . . . . . . . . . . . . . . . . . . . . . 159 SRANDMULTINORMAL . . . . . . . . . . . . . . . . . . . . . . . . . . 147 SRANDMULTINORMALR . . . . . . . . . . . . . . . . . . . . . . . . . 151 SRANDMULTINORMALREFERENCE . . . . . . . . . . . . . . . . 155 SRANDMULTISTUDENTST. . . . . . . . . . . . . . . . . . . . . . . 149 SRANDMULTISTUDENTSTR. . . . . . . . . . . . . . . . . . . . . . 153 SRANDMULTISTUDENTSTREFERENCE . . . . . . . . . . . . . 157 SRANDNEGATIVEBINOMIAL . . . . . . . . . . . . . . . . . . . . 129 SRANDNEGATIVEBINOMIALREFERENCE . . . . . . . . . . . 143 SRANDPOISSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 SRANDPOISSONREFERENCE . . . . . . . . . . . . . . . . . . . . 145 SRANDSKIPAHEAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 SRANDSTUDENTST . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 SRANDTRIANGULAR . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 SRANDUNIFORM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 SRANDVONMISES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 SRANDWEIBULL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 U UGEN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 UINI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 V vrda_cos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 vrda_exp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 vrda_log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 vrda_log10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 vrda_log2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 vrda_sin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 vrda_sincos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 vrsa_cosf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 vrsa_expf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 vrsa_log10f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 vrsa_log2f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 vrsa_logf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 vrsa_powf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211, 215 vrsa_sincosf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 vrsa_sinf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 Z ZDFFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 ZDFFTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 ZFFT1D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 ZFFT1DX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 ZFFT1M . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 ZFFT1MX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 ZFFT2D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 ZFFT2DX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 ZFFT3D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 ZFFT3DX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 ZFFT3DY. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 TM Cray Fortran Reference Manual S–3901–80© 1995, 1997-2011 Cray Inc. All Rights Reserved. This document or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Inc. The CF90 compiler includes United States software patents 5,257,696, 5,257,372, and 5,361,354. U.S. GOVERNMENT RESTRICTED RIGHTS NOTICE The Computer Software is delivered as "Commercial Computer Software" as defined in DFARS 48 CFR 252.227-7014. All Computer Software and Computer Software Documentation acquired by or for the U.S. Government is provided with Restricted Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7014, as applicable. Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7013, as applicable. Cray, LibSci, and PathScale are federally registered trademarks and Active Manager, Cray Apprentice2, Cray Apprentice2 Desktop, Cray C++ Compiling System, Cray CX, Cray CX1, Cray CX1-iWS, Cray CX1-LC, Cray CX1000, Cray CX1000-C, Cray CX1000-G, Cray CX1000-S, Cray CX1000-SC, Cray CX1000-SM, Cray CX1000-HN, Cray Fortran Compiler, Cray Linux Environment, Cray SHMEM, Cray X1, Cray X1E, Cray X2, Cray XD1, Cray XE, Cray XEm, Cray XE5, Cray XE5m, Cray XE6, Cray XE6m, Cray XK6, Cray XMT, Cray XR1, Cray XT, Cray XTm, Cray XT3, Cray XT4, Cray XT5, Cray XT5 h , Cray XT5m, Cray XT6, Cray XT6m, CrayDoc, CrayPort, CRInform, ECOphlex, Gemini, Libsci, NodeKARE, RapidArray, SeaStar, SeaStar2, SeaStar2+, Sonexion, The Way to Better Science, Threadstorm, uRiKA, and UNICOS/lc are trademarks of Cray Inc. AMD is a trademark of Advanced Micro Devices, Inc. GNU is a trademark of The Free Software Foundation. IBM is a trademark of International Business Machines Corporation. ISO is a trademark of International Organization for Standardization (Organisation Internationale de Normalisation). Linux is a trademark of Linus Torvalds. NVIDIA and OpenACC are trademarks of NVIDIA Corporation. Opteron is a trademark of Advanced Micro Devices, Inc. PGI is a trademark of The Portland Group Compiler Technology, STMicroelectronics, Inc. Platform is a trademark of Platform Computing Corporation. Quantum is a trademark of Quantum Corporation. Sun is a trademark of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. UNIX, the “X device,” X Window System, and X/Open are trademarks of The Open Group in the United States and other countries. All other trademarks are the property of their respective owners. RECORD OF REVISION S–3901–80 Published December 2011 Supports the Cray Compiling Environment 8.0 release running on Cray XE and Cray XK systems. S–3901–74 Published June 2011 Supports the Cray Compiling Environment 7.4 release running on Cray XE and Cray XT systems. S–3901–73 Published December 2010 Supports the Cray Compiling Environment 7.3 release running on Cray XE and Cray XT systems. 7.2 Published February 2010 Supports the Cray Compiling Environment 7.2 release running on Cray XT systems. 7.1 Published June 2009 Supports the Cray Compiling Environment 7.1 release running on Cray XT systems. 7.0 Published December 2008 Supports the Cray Compiling Environment 7.0 release running on Cray XT systems. 6.0 Published September 2007 Supports the Cray Fortran compiler 6.0 release running on Cray X1 series and Cray X2 systems. 5.6 Published March 2007 Supports the Cray Fortran compiler 5.6 release running on Cray X1 series systems.Changes to this Document Cray Fortran Reference Manual S–3901–80 S–3901–80 Added information • Add Support for 128-bit intrinsics. See 128-Bit Precision on page 177. • A new class of compiler directives provide support for accelerators on Cray XK systems. See Chapter 7, Using OpenACC on page 165 and OpenMP Accelerator Support on page 152. • New compiler command line options selectively control recognition of OpenMP accelerator directives and other OpenMP directives (!$omp sentinel). See -h [no]omp_acc on page 45 and -x dirlist on page 83. • New compiler command line options selectively control recognition of OpenACC directives (!$acc sentinel). See-h [no]acc on page 41. • The ISO_C_BINDING module provides interoperability between Fortran intrinsic types and C types. See ISO_C_BINDING on page 203. • This release adds support for the OpenMP 3.1 atomic construct. See OpenMP Standard Support on page 149. • A new command line option controls the aggressiveness of optimizations which may affect floating point and complex repeatability. See -h flex_mp=level on page 43. • By default, the compiler now uses a modified malloc implementation that offers better support for Cray XE memory needs than the native malloc provided by the OS. New compiler command line option to link in the native malloc routine instead of a modified version. See -h system_alloc on page 46. • New environment variables CRAY_ACC_DEBUG and PGAS_ERROR_FILE. See Run Time Environment Variables on page 91. • Support directive to prevent loop fission. See Do Not Fission Loop: NOFISSION on page 133. Revised information • The CCE 8.0 release does not support Cray XT systems. • An attribute cannot appear more than once in a given type declaration, which is consistent with the Fortran 2008 standard. Deleted statement to the opposite effect. • This compiler conforms to the Fortran 2003 standard and supports selected features from the Fortran 2008 standard. This release adds support for intrinsic functions, ERFC_SCALED, GAMMA, LOG_GAMMA, HYPOT, and IS_CONTIGUOUS. The RADIX argument for SELECTED_REAL_KIND and IEEE_SELECTED_REAL_KIND intrinsic functions is now supported. Allocatable components of parent (recursive) type are supported. The Fortran 2003 constraint on elemental features was removed (C1279 at [288:2-4] in 04-007) in Fortran 2008. See Fortran 2003 and 2008 Compatibility on page 26. • The -g debugging option is now supported with OpenMP directives. See OpenMP Debugging on page 152.S-3901-74 Added information • A new command line option initializes all memory allocated by Fortran ALLOCATE statements to zero. See the -ez command line option, -d disable and -e enable on page 31 and -h zero on page 47. • The -ef option turns .mod files into lowercase names for makefile flexibility. See -d disable and -e enable on page 31. • The -h [no]add_paren option automatically adds parenthesis to select associative operations (+,-,*) to encourage left to right evaluation of floating point and complex expressions. See -h [no]add_paren on page 41. • The -h [no]autoprefetch option controls whether the compiler performs automatic prefetching. See -h [no]autoprefetch on page 41. • Global atomic memory operation AMO_AFLUSH flushes local cache to the global address space for a particular address on a Cray XE system. See Section 9.12.1.1. • The CLONENEVER directive identifies functions that are never to be cloned and the CLONEALWAYS directive identifies functions that the compiler should always attempt to clone. See Specify Cloning for a Procedure: CLONEALWAYS and CLONENEVER on page 115. Revised information • The default column width for free-form source input increased from 132 to 255. • When -Ovector0 is specified in conjunction with -hfp0 or -hfp1, then array syntax containing associative floating point or complex operations will not be vectorized. • This compiler conforms to the Fortran 2003 standard and supports selected features from the Fortran 2008 standard. This release adds support for MERGE_BITS, IANY, IALL, IPARITY, NORM2, PARITY. See Fortran 2003 and 2008 Compatibility on page 26.S-3901-73 Added • On Cray XE systems, a new class of compiler directive supports PGAS programming models. The DEFER_SYNC directive defers the synchronization of PGAS data until the next synchronization. See PGAS Directive on page 129. • A new option enables traps for specified exceptions. See -K trap=opt[,opt] ... on page 48. • A new option instructs the compiler to treat all module variables as PUBLIC. See the -eA command line option, -d disable and -e enable on page 31. • A new option instructs the compiler to initialize undefined local stack variables to 0 (zero). See the -e0 command line option, -d disable and -e enable on page 31, and-h zero on page 47. • A new set of options cause the compiler backend (optimizer, inliner, codegenerator) to be invoked at application link time, allowing full whole program automatic inlining and future whole program interprocedural analysis (IPA) optimizations. See -h wp on page 47 and -h pl=program_library on page 45. • This compiler includes partial support for the proposed OpenMP 3.1 standard specification and conforms to the OpenMP Application Program Interface, Version 3.0 standard, with limitations. See OpenMP Standard Support on page 149 and Limitations on page 161. • Global atomic memory operations (AMOs) are supported. See Intrinsic Procedures on page 199. • On Cray XE systems, an enhancement to the address format of the pointer to shared type allows the number of PEs available to a single Fortran or UPC job to increase from 2**16-1 to 2**31-1. See -X npes on page 83. Revised • This compiler conforms to the Fortran 2003 standard and supports selected features from the Fortran 2008 standard. This release adds support for LOCK, UNLOCK, LOCK_TYPE, BGT, BGE, BLT, BLE, and CONTAINS. See Fortran 2003 and 2008 Compatibility on page 26. • This compiler introduces a new set of collective intrinsic subroutines, CO_BCAST, CO_SUM, CO_MIN, and CO_MAX, which are extensions of the Fortran 2008 standard, supported on Cray XE systems. For specific information about these routines, see the co_bcast(3i), co_max(3i), co_sum(3i) man pages.Contents Page Introduction [1] 23 1.1 The Cray Fortran Programming Environment . . . . . . . . . . . . . . . . 23 1.2 Cray Fortran Compiler Messages . . . . . . . . . . . . . . . . . . . 24 1.3 Document-specific Conventions . . . . . . . . . . . . . . . . . . . . 25 1.4 Fortran Standard Compatibility . . . . . . . . . . . . . . . . . . . . 25 1.4.1 Fortran 90 Compatibility . . . . . . . . . . . . . . . . . . . . 25 1.4.2 Fortran 95 Compatibility . . . . . . . . . . . . . . . . . . . . 26 1.4.3 Fortran 2003 and 2008 Compatibility . . . . . . . . . . . . . . . . . 26 1.5 Related Fortran Publications . . . . . . . . . . . . . . . . . . . . . 28 Invoking the Cray Fortran Compiler [2] 29 2.1 -A module_name[, module_name] ... . . . . . . . . . . . . . . . . 30 2.2 -b bin_obj_file . . . . . . . . . . . . . . . . . . . . . . . 30 2.3 -c . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.4 -d disable and -e enable . . . . . . . . . . . . . . . . . . . . 31 2.5 -D identifier[=value] . . . . . . . . . . . . . . . . . . . . . . 38 2.6 -f source_form . . . . . . . . . . . . . . . . . . . . . . . 39 2.7 -F . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.8 -g . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.9 -G debug_lvl . . . . . . . . . . . . . . . . . . . . . . . . 40 2.10 -h arg . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.10.1 -h [no]acc . . . . . . . . . . . . . . . . . . . . . . . 41 2.10.2 -h [no]add_paren . . . . . . . . . . . . . . . . . . . . 41 2.10.3 -h [no]autoprefetch . . . . . . . . . . . . . . . . . . 41 2.10.4 -h [no]autothread . . . . . . . . . . . . . . . . . . . 41 2.10.5 -h byteswapio . . . . . . . . . . . . . . . . . . . . . 41 2.10.6 -h cachen . . . . . . . . . . . . . . . . . . . . . . . 41 2.10.7 -h [no]caf . . . . . . . . . . . . . . . . . . . . . . 42 2.10.8 -h cpu=target_system . . . . . . . . . . . . . . . . . . . . 42 2.10.9 -h display_opt . . . . . . . . . . . . . . . . . . . . . 42 S–3901–80 7Cray Fortran Reference Manual Page 2.10.10 -h [no]dwarf . . . . . . . . . . . . . . . . . . . . . 43 2.10.11 -h flex_mp=level . . . . . . . . . . . . . . . . . . . . 43 2.10.12 -h [no]fp_trap . . . . . . . . . . . . . . . . . . . . 43 2.10.13 -h [no]func_trace . . . . . . . . . . . . . . . . . . . 43 2.10.14 -h keepfiles . . . . . . . . . . . . . . . . . . . . . 44 2.10.15 -h loop_trips=[tiny|small|medium|large|huge] . . . . . . . 44 2.10.16 -h mpin . . . . . . . . . . . . . . . . . . . . . . . 44 2.10.17 -h [no]msgs . . . . . . . . . . . . . . . . . . . . . . 44 2.10.18 -h [no]negmsgs . . . . . . . . . . . . . . . . . . . . 44 2.10.19 -h network=nic . . . . . . . . . . . . . . . . . . . . . 44 2.10.20 -h [no]omp . . . . . . . . . . . . . . . . . . . . . . 45 2.10.21 -h [no]omp_acc . . . . . . . . . . . . . . . . . . . . 45 2.10.22 -h [no]omp_trace . . . . . . . . . . . . . . . . . . . 45 2.10.23 -h page_align_allocate . . . . . . . . . . . . . . . . . 45 2.10.24 -h pic, -h PIC . . . . . . . . . . . . . . . . . . . . . 45 2.10.25 -h pl=program_library . . . . . . . . . . . . . . . . . . . 45 2.10.26 -h profile_generate . . . . . . . . . . . . . . . . . . 46 2.10.27 -h system_alloc . . . . . . . . . . . . . . . . . . . . 46 2.10.28 -h [no]second_underscore . . . . . . . . . . . . . . . . 46 2.10.29 -h threadn . . . . . . . . . . . . . . . . . . . . . . 47 2.10.30 -h wp . . . . . . . . . . . . . . . . . . . . . . . . 47 2.10.31 -h zero . . . . . . . . . . . . . . . . . . . . . . . 47 2.11 -I incldir . . . . . . . . . . . . . . . . . . . . . . . . . 47 2.12 -J dir_name . . . . . . . . . . . . . . . . . . . . . . . . 48 2.13 -K trap=opt[,opt] ... . . . . . . . . . . . . . . . . . . . . . 48 2.14 -l libname . . . . . . . . . . . . . . . . . . . . . . . . 49 2.15 -L ldir . . . . . . . . . . . . . . . . . . . . . . . . . . 50 2.16 -m msg_lvl . . . . . . . . . . . . . . . . . . . . . . . . 51 2.17 -M msgs . . . . . . . . . . . . . . . . . . . . . . . . . 51 2.18 -N col . . . . . . . . . . . . . . . . . . . . . . . . . . 52 2.19 -O opt [,opt] ... . . . . . . . . . . . . . . . . . . . . . . . 52 2.19.1 -O n . . . . . . . . . . . . . . . . . . . . . . . . . 55 2.19.2 -O [no]aggress . . . . . . . . . . . . . . . . . . . . . 55 2.19.3 -O [no]autoprefetch . . . . . . . . . . . . . . . . . . 56 2.19.4 -O cachen . . . . . . . . . . . . . . . . . . . . . . . 56 2.19.5 -O fpn . . . . . . . . . . . . . . . . . . . . . . . . 56 2.19.6 -O fusionn . . . . . . . . . . . . . . . . . . . . . . 58 8 S–3901–80Contents Page 2.19.7 -O inlinelib . . . . . . . . . . . . . . . . . . . . . 59 2.19.8 -O ipan . . . . . . . . . . . . . . . . . . . . . . . . 59 2.19.8.1 Inlining . . . . . . . . . . . . . . . . . . . . . . . 60 2.19.8.2 Cloning . . . . . . . . . . . . . . . . . . . . . . . 60 2.19.9 -O ipafrom=source[:source]... . . . . . . . . . . . . . . 62 2.19.10 -O [no]modinline . . . . . . . . . . . . . . . . . . . 63 2.19.11 -O [no]msgs . . . . . . . . . . . . . . . . . . . . . . 63 2.19.12 -O [no]negmsgs . . . . . . . . . . . . . . . . . . . . 64 2.19.13 -O nointerchange . . . . . . . . . . . . . . . . . . . 64 2.19.14 -O [no]omp . . . . . . . . . . . . . . . . . . . . . . 64 2.19.15 -O [no]overindex . . . . . . . . . . . . . . . . . . . 64 2.19.16 -O [no]pattern . . . . . . . . . . . . . . . . . . . . 65 2.19.17 -O scalarn . . . . . . . . . . . . . . . . . . . . . . 66 2.19.18 -O shortcircuitn . . . . . . . . . . . . . . . . . . . 66 2.19.19 -O threadn . . . . . . . . . . . . . . . . . . . . . . 68 2.19.20 -O unrolln . . . . . . . . . . . . . . . . . . . . . . 69 2.19.21 -O vectorn . . . . . . . . . . . . . . . . . . . . . . 69 2.19.22 -O [no]zeroinc . . . . . . . . . . . . . . . . . . . . 70 2.20 -o out_file . . . . . . . . . . . . . . . . . . . . . . . . 70 2.21 -p module_site [,module_site] . . . . . . . . . . . . . . . . . . . 70 2.22 -Q path . . . . . . . . . . . . . . . . . . . . . . . . . . 73 2.23 -r list_opt . . . . . . . . . . . . . . . . . . . . . . . . . 74 2.24 -R runchk . . . . . . . . . . . . . . . . . . . . . . . . . 76 2.25 -rpath ldir . . . . . . . . . . . . . . . . . . . . . . . . 78 2.26 -s size . . . . . . . . . . . . . . . . . . . . . . . . . . 78 2.26.1 Different Default Data Size Options on the Command Line . . . . . . . . . . 80 2.26.2 Pointer Scaling Factor . . . . . . . . . . . . . . . . . . . . . 80 2.27 -S asm_file . . . . . . . . . . . . . . . . . . . . . . . . 81 2.28 -T . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 2.29 -U identifier [,identifier] ... . . . . . . . . . . . . . . . . . . . 82 2.30 -v . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 2.31 -V . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 2.32 -Wa"assembler_opt" . . . . . . . . . . . . . . . . . . . . . . 82 2.33 -Wr"lister_opt" . . . . . . . . . . . . . . . . . . . . . . . 83 2.34 -x dirlist . . . . . . . . . . . . . . . . . . . . . . . . . 83 2.35 -X npes . . . . . . . . . . . . . . . . . . . . . . . . . . 83 2.36 -Y phase,dirname . . . . . . . . . . . . . . . . . . . . . . 84 S–3901–80 9Cray Fortran Reference Manual Page 2.37 -- . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 2.38 sourcefile [sourcefile.suffix ...] . . . . . . . . . . . . . . . . . . 85 Setting Environment Variables [3] 87 3.1 Compiler and Library Environment Variables . . . . . . . . . . . . . . . . 88 3.1.1 CRAY_FTN_OPTIONS . . . . . . . . . . . . . . . . . . . . 88 3.1.2 CRAY_PE_TARGET . . . . . . . . . . . . . . . . . . . . . 88 3.1.3 FORMAT_TYPE_CHECKING . . . . . . . . . . . . . . . . . . 88 3.1.4 FORTRAN_MODULE_PATH . . . . . . . . . . . . . . . . . . 89 3.1.5 LISTIO_PRECISION . . . . . . . . . . . . . . . . . . . . 89 3.1.6 NLSPATH . . . . . . . . . . . . . . . . . . . . . . . . 89 3.1.7 NPROC . . . . . . . . . . . . . . . . . . . . . . . . . 90 3.1.8 TMPDIR . . . . . . . . . . . . . . . . . . . . . . . . 90 3.1.9 ZERO_WIDTH_PRECISION . . . . . . . . . . . . . . . . . . 90 3.2 OpenMP Environment Variables . . . . . . . . . . . . . . . . . . . . 90 3.3 Run Time Environment Variables . . . . . . . . . . . . . . . . . . . 91 3.3.1 aprun Resource Limits . . . . . . . . . . . . . . . . . . . . 91 Using Cray Fortran Directives [4] 93 4.1 Using Directives . . . . . . . . . . . . . . . . . . . . . . . . 97 4.1.1 Directive Lines . . . . . . . . . . . . . . . . . . . . . . . 97 4.1.2 Range and Placement of Directives . . . . . . . . . . . . . . . . . . 98 4.1.3 Interaction of Directives with the -x Command Line Option . . . . . . . . . . . 100 4.1.4 Command Line Options and Directives . . . . . . . . . . . . . . . . 100 4.2 Vectorization Directives . . . . . . . . . . . . . . . . . . . . . . 101 4.2.1 Copy Arrays to Temporary Storage: COPY_ASSUMED_SHAPE . . . . . . . . . 102 4.2.2 Limit Optimizations: HAND_TUNED . . . . . . . . . . . . . . . . . 103 4.2.3 Ignore Vector Dependencies: IVDEP . . . . . . . . . . . . . . . . . 103 4.2.4 Specify Scalar Processing: NEXTSCALAR . . . . . . . . . . . . . . . 104 4.2.5 Request Pattern Matching: [NO]PATTERN . . . . . . . . . . . . . . . 104 4.2.6 Declare an Array with No Repeated Values: PERMUTATION . . . . . . . . . . 105 4.2.7 Designate Loop Nest for Vectorization: PREFERVECTOR . . . . . . . . . . . 106 4.2.8 Conditional Density: PROBABILITY . . . . . . . . . . . . . . . . 106 4.2.9 Allow Speculative Execution of Memory References within Loops: SAFE_ADDRESS . . . 107 4.2.10 Allow Speculative Execution of Memory References and Arithmetic Operations: SAFE_CONDITIONAL . . . . . . . . . . . . . . . . . . . . . . 108 4.2.11 Designate Loops with Low Trip Counts: SHORTLOOP, SHORTLOOP128 . . . . . . 109 4.2.12 Provide More Information for Loops: LOOP_INFO . . . . . . . . . . . . 109 4.2.13 Autothreading for Loops: LOOP_INFO PREFER_[NO]THREAD . . . . . . . 111 10 S–3901–80Contents Page 4.2.14 Unroll Loops: [NO]UNROLL . . . . . . . . . . . . . . . . . . 111 4.2.15 Enable and Disable Vectorization: [NO]VECTOR . . . . . . . . . . . . . 114 4.2.16 Enable or Disable, Temporarily, Soft Vector-pipelining: [NO]PIPELINE . . . . . . 114 4.3 Inlining and Cloning Directives . . . . . . . . . . . . . . . . . . . . 115 4.3.1 Disable or Enable Cloning for a Block of Code: [NO]CLONE and RESETCLONE . . . . 115 4.3.2 Specify Cloning for a Procedure: CLONEALWAYS and CLONENEVER . . . . . . . 115 4.3.3 Disable or Enable Inlining for a Block of Code: [NO]INLINE and RESETINLINE . . . 117 4.3.4 Specify Inlining for a Procedure: INLINEALWAYS and INLINENEVER . . . . . . 117 4.3.5 Create Inlinable Templates for Module Procedures: [NO]MODINLINE . . . . . . . 118 4.4 Scalar Optimization Directives . . . . . . . . . . . . . . . . . . . . 119 4.4.1 Control Loop Interchange: [NO]INTERCHANGE . . . . . . . . . . . . . 119 4.4.2 Control Loop Collapse: [NO]COLLAPSE . . . . . . . . . . . . . . . 121 4.4.3 Determine Register Storage: NOSIDEEFFECTS . . . . . . . . . . . . . 122 4.4.4 Suppress Scalar Optimization: SUPPRESS . . . . . . . . . . . . . . . 123 4.5 Local Use of Compiler Features . . . . . . . . . . . . . . . . . . . . 124 4.5.1 Check Array Bounds: [NO]BOUNDS . . . . . . . . . . . . . . . . 124 4.5.2 Specify Source Form: FREE and FIXED . . . . . . . . . . . . . . . . 126 4.6 Storage Directives . . . . . . . . . . . . . . . . . . . . . . . . 126 4.6.1 Permit Cache Blocking: BLOCKABLE Directive . . . . . . . . . . . . . . 126 4.6.2 Declare Cache Blocking: BLOCKINGSIZE and NOBLOCKING Directives . . . . . . 127 4.6.3 Request Stack Storage: STACK . . . . . . . . . . . . . . . . . . 128 4.7 PGAS Directive . . . . . . . . . . . . . . . . . . . . . . . . 129 4.7.1 DEFER_SYNC Directive . . . . . . . . . . . . . . . . . . . . 129 4.8 Miscellaneous Directives . . . . . . . . . . . . . . . . . . . . . . 130 4.8.1 Control Autothreading: [NO]AUTOTHREAD . . . . . . . . . . . . . . 130 4.8.2 Allocate Cache: CACHE . . . . . . . . . . . . . . . . . . . . 130 4.8.3 Non-temporal Reads and Writes: CACHE_NT . . . . . . . . . . . . . . 131 4.8.4 Specify Array Dependencies: CONCURRENT . . . . . . . . . . . . . . 132 4.8.5 Fuse Loops: [NO]FUSION . . . . . . . . . . . . . . . . . . . 132 4.8.6 Do Not Fission Loop: NOFISSION . . . . . . . . . . . . . . . . . 133 4.8.7 Create Identification String: ID . . . . . . . . . . . . . . . . . . 133 4.8.8 Disregard Dummy Argument Type, Kind, and Rank: IGNORE_TKR . . . . . . . . 134 4.8.9 External Name Mapping: NAME . . . . . . . . . . . . . . . . . . 135 4.8.10 Preprocess Include File: PREPROCESS . . . . . . . . . . . . . . . . 136 4.8.11 Specify Weak Procedure Reference: WEAK . . . . . . . . . . . . . . . 136 Source Preprocessing [5] 139 5.1 General Rules . . . . . . . . . . . . . . . . . . . . . . . . . 139 S–3901–80 11Cray Fortran Reference Manual Page 5.2 Directives . . . . . . . . . . . . . . . . . . . . . . . . . . 140 5.2.1 #include Directive . . . . . . . . . . . . . . . . . . . . . 140 5.2.2 #define Directive . . . . . . . . . . . . . . . . . . . . . 141 5.2.3 #undef Directive . . . . . . . . . . . . . . . . . . . . . . 143 5.2.4 # (Null) Directive . . . . . . . . . . . . . . . . . . . . . . 143 5.2.5 Conditional Directives . . . . . . . . . . . . . . . . . . . . . 143 5.2.5.1 #if Directive . . . . . . . . . . . . . . . . . . . . . . 144 5.2.5.2 #ifdef Directive . . . . . . . . . . . . . . . . . . . . . 144 5.2.5.3 #ifndef Directive . . . . . . . . . . . . . . . . . . . . 145 5.2.5.4 #elif Directive . . . . . . . . . . . . . . . . . . . . . 145 5.2.5.5 #else Directive . . . . . . . . . . . . . . . . . . . . . 145 5.2.5.6 #endif Directive . . . . . . . . . . . . . . . . . . . . . 145 5.3 Predefined Macros . . . . . . . . . . . . . . . . . . . . . . . 146 5.4 Command Line Options . . . . . . . . . . . . . . . . . . . . . . 148 Using the OpenMP Fortran API [6] 149 6.1 OpenMP Standard Support . . . . . . . . . . . . . . . . . . . . . 149 6.2 OpenMP Implementation-Dependent Behaviors . . . . . . . . . . . . . . . 150 6.3 OpenMP Debugging . . . . . . . . . . . . . . . . . . . . . . . 152 6.4 OpenMP Accelerator Support . . . . . . . . . . . . . . . . . . . . 152 6.4.1 OpenMP Accelerator Execution Model . . . . . . . . . . . . . . . . 152 6.4.2 OpenMP Accelerator Memory Model . . . . . . . . . . . . . . . . . 153 6.4.3 OpenMP Accelerator Directives . . . . . . . . . . . . . . . . . . 153 6.4.3.1 acc_region . . . . . . . . . . . . . . . . . . . . . 153 6.4.3.2 acc_data . . . . . . . . . . . . . . . . . . . . . . 155 6.4.3.3 acc_loop . . . . . . . . . . . . . . . . . . . . . . 157 6.4.3.4 acc_region_loop . . . . . . . . . . . . . . . . . . . 158 6.4.3.5 acc_update . . . . . . . . . . . . . . . . . . . . . 159 6.4.4 OpenMP Accelerator Runtime Functions . . . . . . . . . . . . . . . . 159 6.4.5 OpenMP Acclerator Environment Variables . . . . . . . . . . . . . . . 160 6.4.6 OpenMP Accelerator Module Support . . . . . . . . . . . . . . . . . 160 6.5 Optimizations . . . . . . . . . . . . . . . . . . . . . . . . . 160 6.6 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . 161 6.7 Compiler Options . . . . . . . . . . . . . . . . . . . . . . . . 162 6.8 aprun Options . . . . . . . . . . . . . . . . . . . . . . . . 162 Using OpenACC [7] 165 7.1 OpenACC Execution Model . . . . . . . . . . . . . . . . . . . . . 165 12 S–3901–80Contents Page 7.2 OpenACC Memory Model . . . . . . . . . . . . . . . . . . . . . 165 7.3 OpenACC Directives . . . . . . . . . . . . . . . . . . . . . . . 166 7.3.1 parallel . . . . . . . . . . . . . . . . . . . . . . . . 166 7.3.2 loop . . . . . . . . . . . . . . . . . . . . . . . . . 169 7.3.3 parallel_loop . . . . . . . . . . . . . . . . . . . . . 171 7.3.4 data . . . . . . . . . . . . . . . . . . . . . . . . . 171 7.3.5 host_data . . . . . . . . . . . . . . . . . . . . . . . 173 7.3.6 update . . . . . . . . . . . . . . . . . . . . . . . . . 173 7.4 OpenACC Runtime Routines . . . . . . . . . . . . . . . . . . . . . 174 7.5 OpenACC Environment Variables . . . . . . . . . . . . . . . . . . . 174 7.6 OpenACC Compiler Options . . . . . . . . . . . . . . . . . . . . 174 Cray Fortran Defined Externals [8] 175 8.1 Conformance Checks . . . . . . . . . . . . . . . . . . . . . . . 175 Cray Fortran Language Extensions [9] 177 9.1 128-Bit Precision . . . . . . . . . . . . . . . . . . . . . . . . 177 9.2 Characters, Lexical Tokens, and Source Form . . . . . . . . . . . . . . . . 178 9.2.1 Characters Allowed in Names . . . . . . . . . . . . . . . . . . . 178 9.2.2 Switching Source Forms . . . . . . . . . . . . . . . . . . . . 178 9.2.3 Continuation Line Limit . . . . . . . . . . . . . . . . . . . . . 178 9.2.4 D Lines in Fixed Source Form . . . . . . . . . . . . . . . . . . . 178 9.3 Types . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 9.3.1 Alternate Form of LOGICAL Constants . . . . . . . . . . . . . . . . 179 9.3.2 Cray Pointer Type . . . . . . . . . . . . . . . . . . . . . . 179 9.3.3 Cray Character Pointer Type . . . . . . . . . . . . . . . . . . . 183 9.3.4 Boolean Type . . . . . . . . . . . . . . . . . . . . . . . 184 9.3.5 Alternate Form of ENUM Statement . . . . . . . . . . . . . . . . . 184 9.3.6 TYPEALIAS Statement . . . . . . . . . . . . . . . . . . . . 184 9.4 Data Object Declarations and Specifications . . . . . . . . . . . . . . . . 185 9.4.1 Attribute Specification Statements . . . . . . . . . . . . . . . . . . 185 9.4.1.1 BOZ Constants in DATA Statements . . . . . . . . . . . . . . . . 185 9.4.1.2 AUTOMATIC Attribute and Statement . . . . . . . . . . . . . . . 186 9.4.2 IMPLICIT Statement . . . . . . . . . . . . . . . . . . . . . 187 9.4.2.1 IMPLICIT Extensions . . . . . . . . . . . . . . . . . . . 187 9.4.3 Storage Association of Data Objects . . . . . . . . . . . . . . . . . 187 9.4.3.1 EQUIVALENCE Statement Extensions . . . . . . . . . . . . . . . 187 9.4.3.2 COMMON Statement Extensions . . . . . . . . . . . . . . . . . 188 S–3901–80 13Cray Fortran Reference Manual Page 9.5 Expressions and Assignment . . . . . . . . . . . . . . . . . . . . . 188 9.5.1 Expressions . . . . . . . . . . . . . . . . . . . . . . . . 188 9.5.1.1 Rules for Forming Expressions . . . . . . . . . . . . . . . . . 188 9.5.1.2 Intrinsic and Defined Operations . . . . . . . . . . . . . . . . . 188 9.5.1.3 Intrinsic Operations . . . . . . . . . . . . . . . . . . . . 189 9.5.1.4 Bitwise Logical Expressions . . . . . . . . . . . . . . . . . . 190 9.5.2 Assignment . . . . . . . . . . . . . . . . . . . . . . . . 192 9.6 Execution Control . . . . . . . . . . . . . . . . . . . . . . . . 192 9.6.1 STOP Code Extension . . . . . . . . . . . . . . . . . . . . . 192 9.7 Input/Output Statements . . . . . . . . . . . . . . . . . . . . . . 192 9.7.1 File Connection . . . . . . . . . . . . . . . . . . . . . . . 193 9.7.1.1 OPEN Statement . . . . . . . . . . . . . . . . . . . . . 193 9.8 Error, End-of-record, and End-of-file Conditions . . . . . . . . . . . . . . . 193 9.8.1 End-of-file Condition and the END-specifier . . . . . . . . . . . . . . . 193 9.8.1.1 Multiple End-of-file Records . . . . . . . . . . . . . . . . . . 193 9.9 Input/Output Editing . . . . . . . . . . . . . . . . . . . . . . . 193 9.9.1 Data Edit Descriptors . . . . . . . . . . . . . . . . . . . . . 193 9.9.1.1 Integer Editing . . . . . . . . . . . . . . . . . . . . . . 193 9.9.1.2 Real Editing . . . . . . . . . . . . . . . . . . . . . . 194 9.9.1.3 Logical Editing . . . . . . . . . . . . . . . . . . . . . . 194 9.9.1.4 Character Editing . . . . . . . . . . . . . . . . . . . . . 194 9.9.2 Control Edit Descriptors . . . . . . . . . . . . . . . . . . . . . 194 9.9.2.1 Q Editing . . . . . . . . . . . . . . . . . . . . . . . 194 9.9.3 List-directed Formatting . . . . . . . . . . . . . . . . . . . . . 195 9.9.3.1 List-directed Input . . . . . . . . . . . . . . . . . . . . . 195 9.9.4 Namelist Formatting . . . . . . . . . . . . . . . . . . . . . . 195 9.9.4.1 Namelist Extensions . . . . . . . . . . . . . . . . . . . . 195 9.9.5 I/O Editing . . . . . . . . . . . . . . . . . . . . . . . . 196 9.10 Program Units . . . . . . . . . . . . . . . . . . . . . . . . 198 9.10.1 Main Program . . . . . . . . . . . . . . . . . . . . . . . 198 9.10.1.1 Program Statement Extension . . . . . . . . . . . . . . . . . 198 9.10.2 Block Data Program Units . . . . . . . . . . . . . . . . . . . . 198 9.10.2.1 Block Data Program Unit Extension . . . . . . . . . . . . . . . . 198 9.11 Procedures . . . . . . . . . . . . . . . . . . . . . . . . . 199 9.11.1 Procedure Interface . . . . . . . . . . . . . . . . . . . . . . 199 9.11.1.1 Interface Duplication . . . . . . . . . . . . . . . . . . . . 199 9.11.2 Procedure Definition . . . . . . . . . . . . . . . . . . . . . 199 14 S–3901–80Contents Page 9.11.2.1 Recursive Function Extension . . . . . . . . . . . . . . . . . 199 9.11.2.2 Empty CONTAINS Sections . . . . . . . . . . . . . . . . . 199 9.12 Intrinsic Procedures and Modules . . . . . . . . . . . . . . . . . . . 199 9.12.1 Standard Generic Intrinsic Procedures . . . . . . . . . . . . . . . . 199 9.12.1.1 Intrinsic Procedures . . . . . . . . . . . . . . . . . . . . 199 9.13 Exceptions and IEEE Arithmetic . . . . . . . . . . . . . . . . . . . 202 9.13.1 The Exceptions . . . . . . . . . . . . . . . . . . . . . . . 202 9.13.1.1 IEEE Intrinsic Module Extensions . . . . . . . . . . . . . . . . 202 9.14 Interoperability with C . . . . . . . . . . . . . . . . . . . . . . 203 9.14.1 Interoperability Between Fortran and C Entities . . . . . . . . . . . . . . 203 9.14.1.1 BIND(C) Syntax . . . . . . . . . . . . . . . . . . . . 203 9.14.2 ISO_C_BINDING . . . . . . . . . . . . . . . . . . . . . 203 9.15 Coarrays . . . . . . . . . . . . . . . . . . . . . . . . . . 203 9.16 Compiling and Executing Programs Containing Coarrays . . . . . . . . . . . . 204 9.16.1 ftn and aprun Options Affecting Coarrays . . . . . . . . . . . . . . 204 9.16.2 Interoperating with Other Message Passing and Data Passing Models . . . . . . . . 205 9.16.3 Optimizing Programs with Coarrays . . . . . . . . . . . . . . . . . 205 9.17 Submodules . . . . . . . . . . . . . . . . . . . . . . . . . 206 Obsolete Features [10] 207 10.1 IMPLICIT UNDEFINED . . . . . . . . . . . . . . . . . . . . 208 10.2 Type Statement with *n . . . . . . . . . . . . . . . . . . . . . . 208 10.3 BYTE Data Type . . . . . . . . . . . . . . . . . . . . . . . 208 10.4 DOUBLE COMPLEX Statement . . . . . . . . . . . . . . . . . . . 209 10.5 STATIC Attribute and Statement . . . . . . . . . . . . . . . . . . . 209 10.6 Slash Data Initialization . . . . . . . . . . . . . . . . . . . . . . 211 10.7 DATA Statement Features . . . . . . . . . . . . . . . . . . . . . 211 10.8 Hollerith Data . . . . . . . . . . . . . . . . . . . . . . . . 211 10.8.1 Hollerith Constants . . . . . . . . . . . . . . . . . . . . . . 212 10.8.2 Hollerith Values . . . . . . . . . . . . . . . . . . . . . . 213 10.8.3 Hollerith Relational Expressions . . . . . . . . . . . . . . . . . . 213 10.9 PAUSE Statement . . . . . . . . . . . . . . . . . . . . . . . 214 10.10 ASSIGN, Assigned GO TO Statements, and Assigned Format Specifiers . . . . . . . . 214 10.10.1 Form of the ASSIGN and Assigned GO TO Statements . . . . . . . . . . . 215 10.10.2 Assigned Format Specifiers . . . . . . . . . . . . . . . . . . . 216 10.11 Two-branch IF Statements . . . . . . . . . . . . . . . . . . . . 216 10.11.1 Two-branch Arithmetic IF . . . . . . . . . . . . . . . . . . . 217 10.11.2 Indirect Logical IF . . . . . . . . . . . . . . . . . . . . . 217 S–3901–80 15Cray Fortran Reference Manual Page 10.12 Real and Double Precision DO Variables . . . . . . . . . . . . . . . . . 217 10.13 Nested Loop Termination . . . . . . . . . . . . . . . . . . . . . 217 10.14 Branching into a Block . . . . . . . . . . . . . . . . . . . . . . 218 10.15 ENCODE and DECODE Statements . . . . . . . . . . . . . . . . . . 218 10.15.1 ENCODE Statement . . . . . . . . . . . . . . . . . . . . . 218 10.15.2 DECODE Statement . . . . . . . . . . . . . . . . . . . . . 219 10.16 BUFFER IN and BUFFER OUT Statements . . . . . . . . . . . . . . . 220 10.17 Asterisk Delimiters . . . . . . . . . . . . . . . . . . . . . . . 223 10.18 Negative-valued X Descriptor . . . . . . . . . . . . . . . . . . . . 223 10.19 A and R Descriptors for Noncharacter Types . . . . . . . . . . . . . . . . 223 10.20 H Edit Descriptor . . . . . . . . . . . . . . . . . . . . . . . 224 10.21 Obsolete Intrinsic Procedures . . . . . . . . . . . . . . . . . . . . 225 Cray Fortran Deferred Implementation and Optional Features [11] 233 11.1 ISO_10646 Character Set . . . . . . . . . . . . . . . . . . . . . 233 11.2 Restrictions on Unlimited Polymorphic Variables . . . . . . . . . . . . . . . 233 11.3 ENCODING= in I/O Statements . . . . . . . . . . . . . . . . . . . 233 11.4 Allocatable Assignment (Optionally Enabled) . . . . . . . . . . . . . . . . 233 Cray Fortran Implementation Specifics [12] 235 12.1 Companion Processor . . . . . . . . . . . . . . . . . . . . . . 235 12.2 INCLUDE Line . . . . . . . . . . . . . . . . . . . . . . . . 235 12.3 INTEGER Kinds and Values . . . . . . . . . . . . . . . . . . . . 235 12.4 REAL Kinds and Values . . . . . . . . . . . . . . . . . . . . . 235 12.5 DOUBLE PRECISION Kinds and Values . . . . . . . . . . . . . . . . 236 12.6 LOGICAL Kinds and Values . . . . . . . . . . . . . . . . . . . . 236 12.7 CHARACTER Kinds and Values . . . . . . . . . . . . . . . . . . . 236 12.8 Cray Pointers . . . . . . . . . . . . . . . . . . . . . . . . . 236 12.9 ENUM Kind . . . . . . . . . . . . . . . . . . . . . . . . . 236 12.10 Storage Issues . . . . . . . . . . . . . . . . . . . . . . . . 236 12.10.1 Storage Units and Sequences . . . . . . . . . . . . . . . . . . . 237 12.10.2 Static and Stack Storage . . . . . . . . . . . . . . . . . . . . 237 12.10.3 Dynamic Memory Allocation . . . . . . . . . . . . . . . . . . 238 12.11 Finalization . . . . . . . . . . . . . . . . . . . . . . . . . 238 12.12 ALLOCATE Error Status . . . . . . . . . . . . . . . . . . . . . 239 12.13 DEALLOCATE Error Status . . . . . . . . . . . . . . . . . . . . 239 12.14 ALLOCATABLE Module Variable Status . . . . . . . . . . . . . . . . 239 12.15 Kind of a Logical Expression . . . . . . . . . . . . . . . . . . . . 239 16 S–3901–80Contents Page 12.16 STOP Code Availability . . . . . . . . . . . . . . . . . . . . . 239 12.17 Stream File Record Structure and Position . . . . . . . . . . . . . . . . 239 12.18 File Unit Numbers . . . . . . . . . . . . . . . . . . . . . . . 240 12.19 OPEN Specifiers . . . . . . . . . . . . . . . . . . . . . . . 240 12.20 FLUSH Statement . . . . . . . . . . . . . . . . . . . . . . . 240 12.21 Asynchronous I/O . . . . . . . . . . . . . . . . . . . . . . . 241 12.22 REAL I/O of an IEEE NaN . . . . . . . . . . . . . . . . . . . . 241 12.22.1 Input of an IEEE NaN . . . . . . . . . . . . . . . . . . . . 241 12.22.2 Output of an IEEE NaN . . . . . . . . . . . . . . . . . . . . 241 12.23 List-directed and NAMELIST Output Default Formats . . . . . . . . . . . . . 242 12.24 Random Number Generator . . . . . . . . . . . . . . . . . . . . 243 12.25 Timing Intrinsics . . . . . . . . . . . . . . . . . . . . . . . 243 12.26 IEEE Intrinsic Modules . . . . . . . . . . . . . . . . . . . . . 244 Enhanced I/O: Using the Assign Environment [13] 245 13.1 Understanding the assign Environment . . . . . . . . . . . . . . . . . 245 13.1.1 Assign Objects and Open Processing . . . . . . . . . . . . . . . . . 246 13.1.2 assign Command Syntax . . . . . . . . . . . . . . . . . . . 247 13.1.3 Using the Library Routines . . . . . . . . . . . . . . . . . . . 249 13.2 Tuning File Connection Behavior . . . . . . . . . . . . . . . . . . . 250 13.2.1 Using Alternative File Names . . . . . . . . . . . . . . . . . . . 250 13.2.2 Specifying File Structure . . . . . . . . . . . . . . . . . . . . 252 13.2.2.1 Unblocked File Structure . . . . . . . . . . . . . . . . . . . 254 13.2.2.2 assign -s sbin File Processing . . . . . . . . . . . . . . . 255 13.2.2.3 assign -s bin File Processing . . . . . . . . . . . . . . . 255 13.2.2.4 assign -s u File Processing . . . . . . . . . . . . . . . . 255 13.2.2.5 text File Structure . . . . . . . . . . . . . . . . . . . . 255 13.2.2.6 cos or blocked File Structure . . . . . . . . . . . . . . . . 256 13.2.3 Specifying Buffer Behavior . . . . . . . . . . . . . . . . . . . 257 13.2.3.1 Default Buffer Sizes . . . . . . . . . . . . . . . . . . . . 259 13.2.3.2 Library Buffering . . . . . . . . . . . . . . . . . . . . . 259 13.2.3.3 System Cache . . . . . . . . . . . . . . . . . . . . . . 260 13.2.3.4 Unbuffered I/O . . . . . . . . . . . . . . . . . . . . . 260 13.2.4 Specifying Foreign File Formats . . . . . . . . . . . . . . . . . . 261 13.2.5 Specifying Memory Resident Files . . . . . . . . . . . . . . . . . 261 13.2.6 Using and Suppressing File Truncation . . . . . . . . . . . . . . . . 262 13.3 Defining the Assign Environment File . . . . . . . . . . . . . . . . . . 263 13.4 Using Local Assign Mode . . . . . . . . . . . . . . . . . . . . . 263 S–3901–80 17Cray Fortran Reference Manual Page Using Flexible File I/O (FFIO) [14] 265 14.1 Understanding FFIO . . . . . . . . . . . . . . . . . . . . . . . 265 14.2 Using FFIO Layers . . . . . . . . . . . . . . . . . . . . . . . 267 14.2.1 Available I/O Layers . . . . . . . . . . . . . . . . . . . . . 268 14.2.2 Specifying Layered I/O Options . . . . . . . . . . . . . . . . . . 269 14.3 Using FFIO with Common File Structures . . . . . . . . . . . . . . . . . 270 14.3.1 Reading and Writing Text Files . . . . . . . . . . . . . . . . . . 270 14.3.2 Reading and Writing Unblocked Files . . . . . . . . . . . . . . . . 271 14.3.3 Reading and Writing Fixed-length Records . . . . . . . . . . . . . . . 272 14.3.4 Reading and Writing Blocked Files . . . . . . . . . . . . . . . . . 272 14.4 Tips for Enhancing I/O Performance . . . . . . . . . . . . . . . . . . 272 14.4.1 Buffer Size Considerations . . . . . . . . . . . . . . . . . . . 272 14.4.2 Removing Blocking . . . . . . . . . . . . . . . . . . . . . 272 14.4.2.1 The syscall Layer . . . . . . . . . . . . . . . . . . . 273 14.4.2.2 The bufa and cachea Layers . . . . . . . . . . . . . . . . 273 14.4.2.3 The mr Layer . . . . . . . . . . . . . . . . . . . . . . 273 14.4.2.4 The global Layer . . . . . . . . . . . . . . . . . . . . 274 14.4.2.5 The cache Layer . . . . . . . . . . . . . . . . . . . . 274 14.5 Sample Programs . . . . . . . . . . . . . . . . . . . . . . . 275 FFIO Layer Reference [15] 279 15.1 Characteristics of Layers . . . . . . . . . . . . . . . . . . . . . 280 15.2 The bufa Layer . . . . . . . . . . . . . . . . . . . . . . . 281 15.3 The cache Layer . . . . . . . . . . . . . . . . . . . . . . . 282 15.4 The cachea Layer . . . . . . . . . . . . . . . . . . . . . . . 284 15.5 The cos Blocked Layer . . . . . . . . . . . . . . . . . . . . . 285 15.6 The event Layer . . . . . . . . . . . . . . . . . . . . . . . 286 15.7 The f77 Layer . . . . . . . . . . . . . . . . . . . . . . . . 288 15.8 The fd Layer . . . . . . . . . . . . . . . . . . . . . . . . 289 15.9 The global Layer . . . . . . . . . . . . . . . . . . . . . . . 290 15.10 The ibm Layer . . . . . . . . . . . . . . . . . . . . . . . . 291 15.11 The mr Layer . . . . . . . . . . . . . . . . . . . . . . . . 293 15.12 The null Layer . . . . . . . . . . . . . . . . . . . . . . . 296 15.13 The syscall Layer . . . . . . . . . . . . . . . . . . . . . . 296 15.14 The system Layer . . . . . . . . . . . . . . . . . . . . . . 297 15.15 The text Layer . . . . . . . . . . . . . . . . . . . . . . . 297 15.16 The user and site Layers . . . . . . . . . . . . . . . . . . . . 298 15.17 The vms Layer . . . . . . . . . . . . . . . . . . . . . . . 299 18 S–3901–80Contents Page Creating a user Layer [16] 303 16.1 Internal Functions . . . . . . . . . . . . . . . . . . . . . . . 303 16.1.1 The Operations Structure . . . . . . . . . . . . . . . . . . . . 304 16.1.2 FFIO and the stat Structure . . . . . . . . . . . . . . . . . . . 305 16.2 user Layer Example . . . . . . . . . . . . . . . . . . . . . . 306 Named Pipe Support [17] 321 17.1 Piped I/O Example without End-of-file Detection . . . . . . . . . . . . . . . 322 17.2 Detecting End-of-file on a Named Pipe . . . . . . . . . . . . . . . . . . 323 17.3 Piped I/O Example with End-of-file Detection . . . . . . . . . . . . . . . . 324 Examples Example 1. Unrolling outer loops . . . . . . . . . . . . . . . . . . . . 112 Example 2. Illegal unrolling of outer loops . . . . . . . . . . . . . . . . . 113 Example 3. Unrolling nearest neighbor pattern . . . . . . . . . . . . . . . . 113 Example 4. Using defer_sync . . . . . . . . . . . . . . . . . . . 129 Example 5. Local assign mode . . . . . . . . . . . . . . . . . . . . . 263 Example 6. Unformatted direct mr with unblocked file . . . . . . . . . . . . . . 276 Example 7. Unformatted sequential mr with blocked file . . . . . . . . . . . . . 277 Example 8. No EOF Detection: program writerd . . . . . . . . . . . . . . 323 Example 9. No EOF Detection: program readwt . . . . . . . . . . . . . . . 323 Example 10. EOF Detection: program writerd . . . . . . . . . . . . . . . 324 Example 11. EOF Detection: program readwt . . . . . . . . . . . . . . . . 325 Tables Table 1. Compiling Options . . . . . . . . . . . . . . . . . . . . . 32 Table 2. Floating-point Optimization Levels . . . . . . . . . . . . . . . . . 58 Table 3. IPA level . . . . . . . . . . . . . . . . . . . . . . . . 59 Table 4. File Types . . . . . . . . . . . . . . . . . . . . . . . . 62 Table 5. Scaling Factor in Pointer Arithmetic . . . . . . . . . . . . . . . . . 81 Table 6. -Y phase Definitions . . . . . . . . . . . . . . . . . . . . 84 Table 7. Directives . . . . . . . . . . . . . . . . . . . . . . . . 93 Table 8. Explanation of Ignored TKRs . . . . . . . . . . . . . . . . . . 135 Table 9. Operand Types and Results for Intrinsic Operations . . . . . . . . . . . . 189 Table 10. Cray Fortran Intrinsic Bitwise Operators and the Allowed Types of their Operands . . . . 190 Table 11. Data Types in Bitwise Logical Operations . . . . . . . . . . . . . . . 191 Table 12. Values for Keyword Specifier Variables in an OPEN Statement . . . . . . . . . 193 Table 13. Default Fractional and Exponent Digits . . . . . . . . . . . . . . . 194 Table 14. Summary of Control Edit Descriptors . . . . . . . . . . . . . . . . 196 S–3901–80 19Cray Fortran Reference Manual Page Table 15. Summary of Data Edit Descriptors . . . . . . . . . . . . . . . . . 196 Table 16. Default Compatibility Between I/O List Data Types and Data Edit Descriptors . . . . . 197 Table 17. RELAXED Compatibility Between Data Types and Data Edit Descriptors . . . . . . 197 Table 18. STRICT77 Compatibility Between Data Types and Data Edit Descriptors . . . . . 198 Table 19. STRICT90 and STRICT95 Compatibility Between Data Types and Data Edit Descriptors 198 Table 20. Cray Fortran IEEE Intrinsic Module Extensions . . . . . . . . . . . . . 202 Table 21. Obsolete Features and Preferred Alternatives . . . . . . . . . . . . . . 207 Table 22. Summary of String Edit Descriptors . . . . . . . . . . . . . . . . 225 Table 23. Obsolete Procedures and Alternatives . . . . . . . . . . . . . . . . 225 Table 24. List-directed and NAMELIST Assign Environment Options . . . . . . . . . . 242 Table 25. Assign Object Open Processing . . . . . . . . . . . . . . . . . . 246 Table 26. Fortran Access Methods and Options . . . . . . . . . . . . . . . . 253 Table 27. Default Buffer Sizes for Fortran I/O Library Routines . . . . . . . . . . . . 259 Table 28. FFIO Layers . . . . . . . . . . . . . . . . . . . . . . . 268 Table 29. Data Manipulation: bufa Layer . . . . . . . . . . . . . . . . . 282 Table 30. Supported Operations: bufa Layer . . . . . . . . . . . . . . . . 282 Table 31. Data Manipulation: cache Layer . . . . . . . . . . . . . . . . . 283 Table 32. Supported Operations: cache Layer . . . . . . . . . . . . . . . . 283 Table 33. Data Manipulation: cachea Layer . . . . . . . . . . . . . . . . 284 Table 34. Supported Operations: cachea Layer . . . . . . . . . . . . . . . 285 Table 35. Data Manipulation: cos Layer . . . . . . . . . . . . . . . . . . 286 Table 36. Supported Operations: cos Layer . . . . . . . . . . . . . . . . . 286 Table 37. Data Manipulation: f77 Layer . . . . . . . . . . . . . . . . . . 288 Table 38. Supported Operations: f77 Layer . . . . . . . . . . . . . . . . . 288 Table 39. Data Manipulation: global Layer . . . . . . . . . . . . . . . . 290 Table 40. Supported Operations: global Layer . . . . . . . . . . . . . . . 290 Table 41. Values for Maximum Record Size on ibm Layer . . . . . . . . . . . . . 292 Table 42. Values for Maximum Block Size in ibm Layer . . . . . . . . . . . . . 292 Table 43. Data Manipulation: ibm Layer . . . . . . . . . . . . . . . . . . 293 Table 44. Supported Operations: ibm Layer . . . . . . . . . . . . . . . . . 293 Table 45. Data Manipulation: mr Layer . . . . . . . . . . . . . . . . . . 295 Table 46. Supported Operations: mr Layer . . . . . . . . . . . . . . . . . 295 Table 47. Data Manipulation: syscall Layer . . . . . . . . . . . . . . . . 296 Table 48. Supported Operations: syscall Layer . . . . . . . . . . . . . . . 297 Table 49. Data Manipulation: text Layer . . . . . . . . . . . . . . . . . 298 Table 50. Supported Operations: text Layer . . . . . . . . . . . . . . . . 298 Table 51. Values for Record Size: vms Layer . . . . . . . . . . . . . . . . 299 20 S–3901–80Contents Page Table 52. Values for Maximum Block Size: vms Layer . . . . . . . . . . . . . . 300 Table 53. Data Manipulation: vms Layer . . . . . . . . . . . . . . . . . . 300 Table 54. Supported Operations: vms Layer . . . . . . . . . . . . . . . . . 301 Table 55. C Program Entry Points . . . . . . . . . . . . . . . . . . . . 304 Figures Figure 1. Optimization Values . . . . . . . . . . . . . . . . . . . . . 53 Figure 2. Memory Use . . . . . . . . . . . . . . . . . . . . . . . 238 Figure 3. Access Methods and Default Buffer Sizes . . . . . . . . . . . . . . . 262 Figure 4. Typical Data Flow . . . . . . . . . . . . . . . . . . . . . 265 S–3901–80 21Introduction [1] This manual describes the Cray Fortran compiler for the Cray Compiling Environment (CCE) 8.0 Release. This compiler supports Cray XK and Cray XE systems using the Cray Linux Environment (CLE) operating system. The Cray Fortran compiler supports ISO/IEC 1539-1:2004, the Fortran 2003 standard adopted by the International Organization for Standardization (ISO). This compiler also supports selected features from the Fortran 2008 standard. The Cray Fortran compiler is also documented in man pages, beginning with the crayftn(1) man page. Where the information in this manual differs from the man page, the information in the man page supersedes this manual. 1.1 The Cray Fortran Programming Environment The Cray Fortran Programming Environment consists of the tools and libraries used to develop Fortran applications. These are: • The ftn command, which invokes the Cray Fortran compiler. The ftn command is properly termed a compiler driver, as it is used both to compile source code into object code and to link object code files and libraries to create executable files. This compiling and linking can be performed either as separate processes or as one contiguous process, which has significant implications for file handling considerations. These implications are described later in this section. See the crayftn(1) man page for more information. • CrayLibs libraries, which provides library routines, intrinsic procedures, I/O routines, and data conversion routines. • The ftnlx command, which generates listings and checks for possible errors in Fortran programs. See the ftnlx(1) man page for more information. • The ftnsplit command, which splits named Fortran files into separate files with one unit per file. See the ftnsplit(1) man page for more information. • The ftnmgen command, which invokes the Fortran makefile generator. See the ftnmgen(1) man page for more information. S–3901–80 23Cray Fortran Reference Manual In addition, Fortran program development is supported by the following asynchronous products. • LibSci libraries, which provide scientific library routines. • MPT, the Cray Message Passing Toolkit, which supports MPI and SHMEM. • CrayPat, the optional Cray Performance Analysis toolkit. • A variety of optional debuggers, available from Cray and other vendors. The Cray Fortran compiler uses and creates several types of files during processing. • Source files in fixed source form (.f or .F files). • Source files in free source form (.ftn, .FTN, .f90, .F90, .f95, .F95, .f03, .F03, .f08, or .F08, files). • Files containing output from the source preprocessor (.i files). • Relocatable object code (.o files). During compilation, these relocatable object files are saved in the current directory automatically. • If specified, library files containing external references (.a files). • If specified, assembly language output (.s files). Files with .s extensions are assembled and written to the corresponding .o file. • During linking, object files are linked to form an executable file, which by default is named a.out. You can use ftn command line options to modify the default file handling behavior. For example, use the ftn -o option to specify an executable name other than a.out. Alternatively, if you use CrayPat to conduct performance analysis experiments, you must keep the object files created during compilation in order to preserve source-to-executable function mapping. To do so, use the ftn -h keepfiles option. For more information about command line options, see Chapter 2, Invoking the Cray Fortran Compiler on page 29. 1.2 Cray Fortran Compiler Messages The Cray Fortran compiler can produce many messages during compilation and linking. To expand on these messages, use the explain command. For more information, see the explain(1) man page. 24 S–3901–80Introduction [1] 1.3 Document-specific Conventions The following conventions are specific to this document: Convention Meaning Rnnn The Rnnn notation indicates that the feature is in the Fortran standard and can be located in the standard via the Rnnn syntax rule number. Cray pointer The term Cray pointer refers to the Cray pointer data type extension. 1.4 Fortran Standard Compatibility In the Fortran standard, the term processor means the combination of a Fortran compiler and the computing system that executes the code. A processor conforms to the standard if it compiles and executes programs that conform to the standard, provided that the Fortran program is not too large or complex for the computer system in question. You can direct the compiler to flag and generate messages when nonstandard usage of Fortran is encountered. For more information about this command line option (ftn -en), see -d disable and -e enable on page 31 or the ftn(1) man page. When the option is in effect, the compiler prints messages for extensions to the standard that are used in the program. As required by the standard, the compiler also flags the following items and provides the reason that the item is being flagged: • Obsolescent features • Deleted features • Kind type parameters not supported • Violations of any syntax rules and the accompanying constraints • Characters not permitted by the processor • Illegal source form • Violations of the scope rules for names, labels, operators, and assignment symbols The Cray Fortran compiler includes extensions to the Fortran standard. Because the compiler processes programs according to the standard, it is considered to be a standard-conforming processor. When the option to note deviations from the Fortran standard is in effect (-en), extensions to the standard are flagged with ANSI messages when detected at compile time. 1.4.1 Fortran 90 Compatibility No known issues. S–3901–80 25Cray Fortran Reference Manual 1.4.2 Fortran 95 Compatibility No known issues. 1.4.3 Fortran 2003 and 2008 Compatibility This Cray Fortran compiler release conforms to the Fortran 2003 standard. This release also supports the following features in the Fortran 2008 standard: • Support for intrinsic functions ERFC_SCALED, GAMMA, LOG_GAMMA, and HYPOT. • Support for allocatable components of parent (recursive) type • Support for the RADIX argument for SELECTED_REAL_KIND and IEEE_SELECTED_REAL_KIND intrinsics • Support for the IS_CONTIGUOUS intrinsic function • The Fortran 2003 constraint on elemental features was removed (C1279 at [288:2-4] in 04-007) in Fortran 2008. • Support for IANY, IALL, IPARITY intrinsic functions. • Support for the MERGE_BITS intrinsic function. • Support for the NORM2 intrinsic function. • Support for the PARITY intrinsic function. • Support for LOCK, UNLOCK, LOCK_TYPE • Support for BGT, BGE, BLT, BLE • Empty CONTAINS is extended to a section in a type definition (i.e., no following type-bound procedure declarations). • Full support for coarray syntax, declaration, and definition • Support for coarray image control statements – CRITICAL and END CRITICAL statements delimit a critical region. Instructions within a critical region are executed on one image at a time. – SYNC ALL, SYNC IMAGES, and SYNC MEMORY • CODIMENSION attribute • Allocatable coarrays • Coarray structures with allocatable and pointer components • Structures with coarray components • Coarray variable I/O 26 S–3901–80Introduction [1] • Run time support for coarrays • THIS_IMAGE, IMAGE_INDEX, NUM_IMAGES, LCOBOUND, and UCOBOUND intrinsics • Submodules • Separate module procedures • BLOCK construct • CONTIGUOUS attribute • Extended code on the STOP statement, including character constant expressions • Empty CONTAINS (contains with no following function or subroutine statement) • Omitted constructor value for ALLOCATABLE component • ALLOCATE with MOLD= specifier • Complex arguments to ACOS, ASIN, ATAN, COSH, SINH, TAN, and TANH intrinsics • ACOSH, ASINH, and ATANH intrinsics • Two-argument form of ATAN • SHIFTL, SHIFTR, SHIFTA, DSHIFTL, and DSHIFTR intrinsics • POPCNT, LEADZ, TRAILZ, and POPPAR intrinsics • ERF and ERFC intrinsics • BACK= argument for the MAXLOC and MINLOC intrinsics S–3901–80 27Cray Fortran Reference Manual 1.5 Related Fortran Publications For more information about the Fortran language and its history, consult the following commercially available reference books: • Fortran 2003 and 2008 standards can be downloaded from http://www.nag.co.uk/sc22wg5/. The Fortran 2008 standard is also available directly from the ISO. • Chapman, S. Fortran 95/2003 for Scientists & Engineers. McGraw Hill, 2007. ISBN 0073191574. • Metcalf, M., J. Reid, and M. Cohen. Fortran 95/2003 Explained. Oxford University Press, 2004. ISBN 0-19-852693-8. • Jeanne C. Adams, Walter S. Brainerd, Richard A. Hendrickson, Richard E. Maine, Jeanne T. Martin, and Brian T. Smith, The Fortran 2003 Handbook: The Complete Syntax, Features, and Procedures. Springer, 2009. ISBN 978-1-84628-378-9. 28 S–3901–80Invoking the Cray Fortran Compiler [2] The following files are produced by or accepted by the Cray Fortran compiler: File Type a.out Default name of the executable output file. See the -o out_file option for information about specifying a different name for the executable file. file.a Library files to be searched for external references or modules. file.cg and file.opt Files containing decompilation of the intermediate representation of the compiler. These listings resemble the format of the source code. These files are generated when the -rd option is specified. file.f or file.F Input Fortran source file in fixed source form. If file ends in .F, the source preprocessor is invoked. file.f90 file.F90 file.f95 file.F95 file.f03 file.F03 file.f08 file.F08 file.ftn file.FTN Input Fortran source file in free source form. If file ends in .F90, .F95, .F03, .F08, or .FTN, the source preprocessor is invoked. Note: The file suffix does not restrict the source file to a given standard. Regardless of the file suffix, the Cray Fortran compiler processes the file according to the full current Fortran standard. For example, a source file with the suffix .f90 may contain code using language features not implemented until the Fortran 2003 standard. S–3901–80 29Cray Fortran Reference Manual file.i File containing output from the source preprocessor. file.lst Listing file. file.o Relocatable object file. file.s Assembly language file. modulename.mod If the -em option is specified, the compiler writes a modulename.mod file for each module; modulename is created by taking the name of the module and, if necessary, converting it to uppercase. This file contains module information, including any contained module procedures. The syntax of the ftn command is as follows: ftn [-A module_name[, module_name ] ...] [-b bin_obj_file] [-c] [-d disable] [-D identifier[= value]] [-e enable] [-f source_form] [-F] [-g] [-G debug_lvl ] [-h arg], [-I incldir] [-J dir_name] [-K trap=opt[,opt] ...] [-l lib_file] [-L ldir] [-m msg_lvl] [-M msgs] [-N col] [-o out_file] [-O opt[,opt] ...] [-p module_site] [-Q path] [-r list_opt] [-R runchk][-rpath ldir] [-s size] [-S asm_file] [ -T] [-U identifier[,identifier] ...] [-v] [-V] [-Wphase,"opt..."] [-x dirlist] [-X npes] [-Yphase,dirname] [--] sourcefile [sourcefile ...] Note: Some default values shown for ftn command options may have been changed by your site. See your system support staff for details. 2.1 -A module_name[, module_name] ... The -A module_name [, module_name] ... option directs the compiler to behave as if you entered a USE module_name statement for each module_name in your Fortran source code. The USE statements are entered in every program unit and interface body in the source file being compiled. 2.2 -b bin_obj_file The -b bin_obj_file option disables the link step and saves the binary object file version of your program in bin_obj_file. Only one input file is allowed when the -b bin_obj_file option is specified. If you have more than one input file, use the -c option to disable the link step and save the binary files to their default file names. If only one input file is processed and neither the -b nor the -c option is specified, the binary version of your program is not saved after the link is completed. 30 S–3901–80Invoking the Cray Fortran Compiler [2] If both the -b bin_obj_file and -c options are specified on the ftn command line, the link step is disabled and the binary object file is written to the name specified as the argument to the -b bin_obj_file option. For more information about the -c option, see -c on page 31. By default, the binary file is saved in file.o, where file is the name of the source file and .o is the suffix used. 2.3 -c The -c option disables the link step and saves the binary object file version of your program in file.o, where file is the name of the source file and .o is the suffix used. If there is more than one input file, a file.o is created for each input file specified. By default, this option is off. If only one input file is processed and neither the -b bin_obj_file nor the -c options are specified, the binary version of your program is not saved after the link is completed. If both the -b bin_obj_file and -c options are specified on the ftn command line, the link step is disabled and the binary object file is written to the name specified as the argument to the -b bin_obj_file option. For more information about the -b bin_obj_file option, see -b bin_obj_file on page 30. If both the -o out_file and the -c option are specified on the ftn command line, the link step is disabled and the binary file is written to the out_file specified as an argument to -o. For more information about the -o out_file option, see -o out_file on page 70. 2.4 -d disable and -e enable The -d disable and -e enable options disable or enable compiling options. To specify more than one compiling option, enter the options without separators between them; for example, -eaf. Table 1 shows the arguments to use for disable or enable. S–3901–80 31Cray Fortran Reference Manual Table 1. Compiling Options args Action, if enabled 0 Initializes all undefined local numeric stack variables to 0. If a user variable is of type character, it is initialized to NUL. If a user variable is type logical, it is initialized to false. The variables are initialized upon each execution of each procedure. Enabling this option can help identify problems caused by using uninitialized numeric and logical variables. Default: disabled a Aborts compilation after encountering the first error. Default: disabled A Treat all module variables as PUBLIC. Does not override explicit PRIVATE statements or attributes. Disabling this option has the effect of including a PRIVATE statement in the specification part of the module. Default: enabled B Generates binary output. If disabled, inhibits all optimization and allows only syntactic and semantic checking. Default: enabled c Interface checking: use Cray's system modules to check library calls in a compilation. If you have a procedure with the same name as one in the library, you will get errors because the compiler does not skip user-specified procedures when it performs the checks. Default: disabled 32 S–3901–80Invoking the Cray Fortran Compiler [2] args Action, if enabled C Enable/disable some types of standard call site checking. The current Fortran standard requires that the number and types of arguments must agree between the caller and callee. These constraints are enforced in cases where the compiler can detect them, however, specifying -dC disables some of this error checking, which may be necessary in order to get some older Fortran codes to compile. Note: If error checking is disabled, unexpected compile-time or runtime results may occur. In addition, the compiler by default attempts to detect situations in which an interface block should be specified but is not. Specifying -dC disables this type of checking as well. Default: enabled d Controls a column-oriented debugging feature when using fixed source form. When the option is enabled, the compiler replaces the D or d characters in column 1 of your source with a blank and treats the entire line as a valid source line. This feature can be useful, for example, during debugging if you want to insert PRINT statements. When disabled, a D or d character in column 1 is treated as a comment character. Default: disabled D Turns on all debugging information. This option is equivalent to specifying these options: -O0, -g, -m2, -R bcdspi, and -rl. See also -ed. Default: disabled E The -eE option enables existing declarations to duplicate the declarations contained in a used module. Therefore, you do not have to modify the older code by removing the existing declarations. Because the declarations are not removed, the use associated objects will duplicate declarations already in the code, which is not standard. However, this option allows the compiler to accept these statements as long as the declarations match the declarations in the module. Existing declarations of a procedure must match the interface definitions in the module; otherwise an error is generated. Only existing declarations that declare the function name or generic name in an EXTERNAL or type statement are allowable under this option. S–3901–80 33Cray Fortran Reference Manual args Action, if enabled This example illustrates some of the acceptable types of existing declarations. Program older contains the older code, while module m contains the interfaces to check. module m interface subroutine one(r) real :: r F end subroutine function two() integer :: two end function end interface end module program older use m !Or use -Am on the compiler command line external one !Use associated objects integer :: two !in declarative statements call one(r) j = two() end program Default: disabled f Turns .mod files into lowercase names for makefile flexibility. Default: disabled F Controls preprocessor expansion of macros in Fortran source lines. Default: enabled g Allows branching into the code block for a DO or DO WHILE construct. Historically, codes used branches out of and into DO constructs. Fortran standards prohibit branching into a DO construct from outside of that construct. By default, the Cray Fortran compiler will issue an error for this situation. Cray does not recommend branching into a DO construct, but if you specify -eg, the code will compile. Default: disabled 34 S–3901–80Invoking the Cray Fortran Compiler [2] args Action, if enabled h Enables support for 8-bit and 16-bit INTEGER and LOGICAL types that use explicit kind or star values. By default (-eh), data objects declared as INTEGER(kind=1) or LOGICAL(kind=1) are 8 bits long, and objects declared as INTEGER(kind=2) or LOGICAL(kind=2) are 16 bits long. When this option is disabled (-dh), data objects declared as INTEGER(kind=1), INTEGER(kind=2), LOGICAL(kind=1), or LOGICAL(kind=2) are 32 bits long. Deferred Implementation: Vectorization of 8- and 16-bit objects is deferred. Default: enabled I Treats all variables as if an IMPLICIT NONE statement had been specified. Does not override any IMPLICIT statements or explicit type statements. All variables must be typed. Default: disabled j Executes DO loops at least once. Default: disabled m When this option is enabled, the compiler creates .mod files to hold module information for future compiles. When it is disabled, and a module is compiled, the compiler deletes any existing MODULENAME.mod files it finds in the output directory before creating new module information in the .o file. By default, module files are written to the current working directory. You can use the -J dir_name option to specify an alternate output directory for .mod files only. For more information about the -J dir_name option, see -J dir_name on page 48. Whether this option is enabled or disabled, the search order for satisfying modules references in USE statements is as follows: 1. The current working directory. 2. Any directories or files specified with the -p option. 3. Any directories specified with the -I option. 4. Any directories or files specified with the FTN_MODULE_PATH environment variable. S–3901–80 35Cray Fortran Reference Manual args Action, if enabled When searching within a directory, the compiler first checks all .mod files, then the .o files, and then the .a files. Note: The compiler creates modules through the MODULE statement. A module is referenced with the USE statement. All .mod files are named modulename.mod, where modulename is the name of the module specified in the MODULE or USE statement. Default: disabled n Generates messages to note all nonstandard Fortran usage. Default: disabled o Display to stderr the optimization options used by the compiler for this compilation. Default: disabled P Performs source preprocessing on Fortran source files, but does not compile (see sourcefile [sourcefile.suffix ...] on page 85 for valid file extensions). When specified, source code is included by #include directives but not by Fortran INCLUDE lines. Generates file.i, which contains the source code after the preprocessing has been performed and the effects have been applied to the source program. For more information about source preprocessing, see Chapter 5, Source Preprocessing on page 139. Default: disabled q Aborts compilation if 100 or more errors are generated. Default: enabled Q Controls whether the compiler accepts variable names that begin with a leading underscore (_) character. For example, when Q is enabled, the compiler accepts _ANT as a variable name. Enabling this option can cause collisions with system name space, such library entry point names. Default: disabled R Compiles all functions and subroutines as if they had been defined with the RECURSIVE attribute. Default: disabled 36 S–3901–80Invoking the Cray Fortran Compiler [2] args Action, if enabled s Scales the values of the count and count_rate arguments for the SYSTEM_CLOCK intrinsic function down by a factor of 2 **14 (16384) if the storage size of the value of each of the count and count-rate argumens is 32 bits. Otherwise, no scaling occurs. Default: enabled S Generates assembly language output and saves it in file.s. When the -eS option is specified on the command line with the -S asm_file option, the -S asm_file option overrides the -eS option. Default: disabled v Allocates variables to static storage. These variables are treated as if they had appeared in a SAVE statement. The following types of variables are not allocated to static storage: automatic variables (explicitly or implicitly stated), variables declared with the AUTOMATIC attribute, variables allocated in an ALLOCATE statement, and local variables in explicit recursive procedures. Variables with the ALLOCATABLE attribute remain allocated on procedure exit, unless explicitly deallocated, but they are not allocated in static memory. Variables in explicit recursive procedures consist of those in functions, in subroutines, and in internal procedures within functions and subroutines that have been defined with the RECURSIVE attribute. The STACK compiler directive overrides -ev; for more information about this compiler directive, see Request Stack Storage: STACK on page 128. Default: disabled w Enables support for automatic memory allocation for allocatable variables and arrays that are on the left hand side of intrinsic assignment statements. The option can potentially decrease run time performance, even when automatic memory allocation is not needed. It will affect optimizations for a code region that contains an assignment to allocatable variables or arrays. For example, it could easily prevent loop fusion for multiple array syntax assignment statements with the same shape. Default: disabled S–3901–80 37Cray Fortran Reference Manual args Action, if enabled z Initialize all memory allocated by Fortran ALLOCATE statements to zero. This option applies only for the current source file and should be specified for each source file compilation where this behavior is desired. Default: disabled Z Performs source preprocessing and compilation on Fortran source files (see sourcefile [sourcefile.suffix ...] on page 85 for valid file extensions). When specified, source code is included by #include directives and by Fortran INCLUDE lines. Generates file.i, which contains the source code after the preprocessing has been performed and the effects applied to the source program. For more information about source preprocessing, see Chapter 5, Source Preprocessing on page 139. Default: disabled 2.5 -D identifier[=value] The -D identifier[=value] option defines variables used for source preprocessing as if they had been defined by a #define source preprocessing directive. If a value is specified, there can be no spaces on either side of the equal sign (=). If no value is specified, the default value of 1 is used. The -U option undefines variables used for source preprocessing. If both -D and -U are used for the same identifier, in any order, the identifier is undefined. For more information about the -U option, see -U identifier [,identifier] ... on page 82. This option is ignored unless one of the following conditions is true: • The Fortran input source file is specified as either file.F, file.F90, file.F95, file.F03, file.F08, or file.FTN. • The -eP or -eZ options have been specified. For more information about source preprocessing, see Chapter 5, Source Preprocessing on page 139. 38 S–3901–80Invoking the Cray Fortran Compiler [2] 2.6 -f source_form The -f source_form option specifies whether the Fortran source file is written in fixed source form or free source form. For source_form, enter free or fixed. The source_form specified here overrides any source form implied by the source file suffix. A FIXED or FREE directive specified in the source code overrides this option (see Specify Source Form: FREE and FIXED on page 126). The default source form is fixed for input files that have the .f or .F suffix. The default source form is free for input files that have the .f90, .F90, .f95, .F95, .f03, .F03, .f08, .F08, .ftn, or .FTN suffix. Note that the Fortran standard has declared fixed source form to be obsolescent. If the file has a .F, .F90, .F95, .F03, .F08, or .FTN suffix, the source preprocessor is invoked. See Chapter 5, Source Preprocessing on page 139 about preprocessing. 2.7 -F The -F option is obsolete and is supported for compatibility with legacy make files. Macro expansion is now enabled by default. To disable macro expansion, use the -dF option. For more information about source preprocessing, see Chapter 5, Source Preprocessing on page 139. 2.8 -g The -g option provides debugging support identical to specifying the -G0 option. This level of debugging implies -O0 -homp (optimizations disabled but OpenMP directives are recognized) and -h fp0. Default: off S–3901–80 39Cray Fortran Reference Manual 2.9 -G debug_lvl The -G debug_lvl option controls the tradeoffs between ease of debugging and compiler optimizations. The compiler produces some level of internal debugger information (DWARF) at all times. This DWARF data provides function and source line information to debuggers for tracebacks and breakpoints, as well as type and location information about data variables. Note: The -g or -G options can be specified on a per-file basis, so that only part of an application pays the price for improved debugging. debug_lvl Support 0 Optimizations disabled: full DWARF information is available for debugging, but at the cost of a slower and larger executable. Breakpoints can be set at each line. This level of debugging implies -O0 -homp (optimizations disabled but OpenMP directives are recognized) and -h fp0. 1 Partial optimization: most DWARF and at least some optimizations make tracebacks and limited breakpoints available in the debugger. Some scalar optimizations and all loop nest restructuring is disabled, but the source code will be visible and most symbols will be available. This allows block-by-block debugging, with the exception of innermost loops. The executable will be faster than with -g or -G0. 2 Full optimization: with partial DWARF and most optimizations, tracebacks and very limited breakpoints are available in the debugger. The source code will be visible and some symbols will be available. This level allows post-mortem debugging, but local information such as the value of a loop index variable is not necessarily reliable at this level because such information often is carried in registers in optimized code. The executable will be faster and smaller than with -G1. fast Compile code for use with Cray fast-track debugging. This option is useful only if used in conjunction with a debugger that supports fast-track debugging. For more information, see the lgdb(1) man page. 2.10 -h arg The -h arg allows you to access various compiler functionality. For more information about what to specify for arg, see the following subsections. 40 S–3901–80Invoking the Cray Fortran Compiler [2] 2.10.1 -h [no]acc The -h [no]acc option controls recognition of OpenACC directives (!$acc sentinel). Default: -h acc 2.10.2 -h [no]add_paren The -h [no]add_paren option automatically adds parenthesis to select associative operations (+,-,*) to encourage left to right evaluation of floating point and complex expressions. Left to right evaluation is not required by the language standards, but some applications may expect it. Default: -h noadd_paren 2.10.3 -h [no]autoprefetch The -h [no]autoprefetch option controls autoprefetch optimization. It does not affect the loop_info [no]prefetch directive. This is identical to the -O autoprefetch option and is provided for command-line compatibility with the Cray C compiler. Default: -h autoprefetch 2.10.4 -h [no]autothread The -h [no]autothread option enables or disables autothreading. Default: -h noautothread 2.10.5 -h byteswapio The -h byteswapio option forces byte-swapping of all input and output files for direct and sequential unformatted I/O. 2.10.6 -h cachen The -h cachen option specifies the level of automatic cache management to be performed, where n is a value from 0 to 3 with 0 being no cache management and 3 being the most aggressive cache management. This is identical to the -O cachen option and is provided for command-line compatibility with the Cray C compiler. For more information, see -O cachen on page 56. Default: -h cache2 S–3901–80 41Cray Fortran Reference Manual 2.10.7 -h [no]caf The -h caf option enables the compiler to recognize coarray syntax. Coarrays are a Fortran 2008 feature that offer a method for performing data passing. Coarrays are discussed in more detail in Coarrays on page 203. Data passing is an effective method for programming single-program-multiple-data (SPMD) parallelism. Its chief advantages over MPI are lower latency and high bandwidth for data transfers, both of which lead to improved scalability for parallel applications. Compared to MPI and SHMEM, programs using coarrays are also more human-readable, and thus increase programmer productivity. As a language feature, the code can be conditionally analyzed and optimized by the compiler. Default: -h nocaf 2.10.8 -h cpu=target_system The -h cpu=target_system option specifies the Cray system on which the absolute binary file is to be executed, where target_system can be either x86-64, opteron, barcelona, shanghai, istanbul, mc8, mc12, or interlagos. The x86-64 and opteron options produce identical output, for use on singleand dual-core systems. If you are creating executables for use on a system with quad-core processors (either AMD Opteron barcelona or shanghai processors), you must also have the associated module (either xtpe-barcelona or xtpe-shanghai) loaded when compiling and linking your code. Likewise, if you are creating executables for use on a system with AMD Opteron six-core processors (istanbul), eight-core processors (mc8), twelve-core processors (mc12), 16-core processors (interlagos) you must have the xtpe-istanbul, xtpe-mc8, xtpe-mc12, or xtpe-interlagos module loaded when compiling and linking your code. If one of these modules is loaded, the default target_system changes to the corresponding cpu target. If the target_system is set during compilation of any source file, it must also be set to that same target during linking and loading. The target system may also be specified using the CRAY_PE_TARGET environment variable. For more information, see CRAY_PE_TARGET on page 88. Default: x86-64 2.10.9 -h display_opt The -h display_opt option displays the compiler optimization settings currently in force. This option is identical to the -eo option and is provided for command-line compatibility with the Cray C compiler. 42 S–3901–80Invoking the Cray Fortran Compiler [2] 2.10.10 -h [no]dwarf The -h [no]dwarf option controls whether DWARF debugging information is generated during compilation. Default: dwarf 2.10.11 -h flex_mp=level The -h flex_mp=level option controls the aggressiveness of optimizations which may affect floating point and complex repeatability when application requirements require identical results when varying the number of ranks or threads. -hflex_mp=intolerant has the highest probability of repeatable results, but also has the highest performance penalty. -hflex_mp=conservative uses more aggressive optimization and yields higher performance than -hflex_mp=intolerant, but results may not be sufficiently repeatable for some applications. -hflex_mp=tolerant uses most aggressive optimization and yields highest performance, but results may not be sufficiently repeatable for some applications. 2.10.12 -h [no]fp_trap Controls whether the compiler generates code that is compatible with floating-point traps. Default: fp_trap, if traps are enabled using the -K trap option, or if -Ofp[0,1] is in effect. Otherwise, the default is nofp_trap. 2.10.13 -h [no]func_trace The -h func_trace option is for use only with CrayPat (Cray performance analysis tool). If this option is specified, the compiler inserts CrayPat entry points into each function in the compiled source file. The names of the entry points are: __pat_tp_func_entry __pat_tp_func_return These are resolved by CrayPat when the program is instrumented using the pat_build command. When the instrumented program is executed and it encounters either of these entry points, CrayPat captures the address of the current function and its return address. Default: nofunc_trace S–3901–80 43Cray Fortran Reference Manual 2.10.14 -h keepfiles The -h keepfiles option prevents the removal of the object (.o) and temporary assembly (.s) files after an executable is created. Normally, the compiler automatically removes these files after linking them to create an executable. Since the original object files are required in order to instrument a program for performance analysis, if you plan to use CrayPat to conduct performance analysis experiments, you can use this option to preserve the object files. 2.10.15 -h loop_trips=[tiny|small|medium|large|huge] Specifies runtime loop trip counts for all loops in a compiled source file. This information is used to tune optimizations to the runtime characteristics of the application. Default: none 2.10.16 -h mpin Enables or disables optimization around a selected subset of MPI library calls. mpi0 disables this option. Default: mpi1 2.10.17 -h [no]msgs The -h [no]msgs option causes the compiler to write optimization messages to stderr. This option is identical to the -O [no]msgs option and is provided for command-line compatibility with the Cray C compiler. For more information, see -O [no]msgs on page 63. Default: -h nomsgs 2.10.18 -h [no]negmsgs The -h [no]negmsgs option causes the compiler to generate messages to stderr explaining why optimizations did not occur in a given instance. This option is identical to the -O [no]negmsgs option and is provided for command-line compatibility with the Cray C compiler. For more information, see -O [no]negmsgs on page 64. Default: -h nonegmsgs 2.10.19 -h network=nic The -h network=nic option is used to specify the target machine's interconnection attributes. gemini is supported. 44 S–3901–80Invoking the Cray Fortran Compiler [2] 2.10.20 -h [no]omp The -h [no]omp option enables or disables compiler recognition of OpenMP directives. Using the -h noomp option is similar to the -h thread0 option, in that it disables OpenMP, but unlike -h thread0 it does not affect autothreading. The -h [no]omp option is identical to the -O [no]omp option and is provided for command-line compatibility with the Cray C compiler. For more information, see -O [no]omp on page 64. Default: -h omp 2.10.21 -h [no]omp_acc The -h [no]omp_acc option enables or disables compiler recognition of OpenMP accelerator directives. See OpenMP Accelerator Directives on page 153. 2.10.22 -h [no]omp_trace The -h [no]omp_trace turns the insertion of CrayPat OpenMP tracing calls on or off. By default tracing is off. Default: -h noomp_trace 2.10.23 -h page_align_allocate The -h page_align_allocate option directs the compiler to force allocations of arrays larger than the memory page size to be aligned on a page boundary. This option affects only the ALLOCATE statements of the current source file; therefore it must be specified for each source file where this behavior is desired. Using this option can improve DIRECTIO performance. 2.10.24 -h pic, -h PIC Generate position independent code (PIC), which allows a virtual address change from one process to another, as is necessary in the case of shared, dynamically linked objects. The virtual addresses of the instructions and data in PIC code are not known until dynamic link time. For the Cray implementation, the pic and PIC options have the same effect and should be used to compile codes using more than 2GB of static memory, or for creating dynamically linked libraries. 2.10.25 -h pl=program_library Create and use a persistent repository of compiler information specified by program_library. When used with -hwp, this option provides application-wide, cross-file, automatic inlining. See -h wp on page 47. S–3901–80 45Cray Fortran Reference Manual The program_library repository is implemented as a directory and the information contained in program library is built up with each compiler invocation. Any compilation that does not have the -hpl option will not add information to this repository. Because of the persistence of program_library, it is the user's responsibility to manage it. For example, rm -r program_library might be added to the make clean target in an application makefile. Because program_library is a directory, use rm -r to remove it. If an application makefile works by creating files in multiple directories during a single build, the program_library should be an absolute path, otherwise multiple and incomplete program library repositories will be created. For example, avoid -hpl=./PL.1 and use -hpl=/fullpath/builddir/PL.1 instead. 2.10.26 -h profile_generate The -h profile_generate option lets you request that the source code be instrumented for profile information gathering with CrayPat (Cray performance analysis tool). The compiler inserts calls and data gathering instructions that enable CrayPat to gather information about the loops in a compilation unit. In order to get useful data out of this feature, the CrayPat pat_build command must then be run on the resulting executable in order to link in the CrayPat data gathering routines. If this is not done, the code will still execute, however, no data is recorded. For more information, see the intro_craypat(1) man page. 2.10.27 -h system_alloc The -h system_alloc option directs the compiler to link in the native malloc routine. By default, the compiler now uses a modified malloc implementation that offers better support for Cray XE memory needs than the native malloc provided by the OS. The user may choose to modify the default behavior, directing the compiler to link in the native malloc routine by using the -hsystem_alloc option during compilation. 2.10.28 -h [no]second_underscore The -h [no]second_underscore option controls the way in which external names are generated. By default, the compiler generates external names in lower case and will add one trailing underscore (_). This behavior matches the PGI Fortran compiler's external behavior. If -h second_underscore is specified, the compiler adds a second trailing underscore if the original external name has any underscores in it. This behavior matches the GNU compiler's external naming behavior. Default: -h nosecond_underscore 46 S–3901–80Invoking the Cray Fortran Compiler [2] 2.10.29 -h threadn The -h threadn option enables you to control the compilation and optimization of OpenMP and autothreading directives, where n is a value from 0 to 3 with 0 being off and 3 specifying the most aggressive optimization. This option is identical to the -O threadn option and is provided for command-line compatibility with the Cray C compiler. For more information, see -O threadn on page 68. Default: -h thread2 2.10.30 -h wp Enables the whole program mode. This option causes the compiler backend (IPA, optimizer, codegenerator) to be invoked at application link time, enabling whole program automatic inlining/cloning and future whole program interprocedural analysis (IPA) optimizations. Since the -hwp option provides automatic application-wide inlining, the -Oipafrom option is no longer needed for cross-file inlining. Requires that pl=program_library is specified. See -h pl=program_library on page 45. The options -hpl=program_library and -hwp should be specified on all compiler invocations and on the compiler link invocation. Since -hwp delays the compiler optimization step until link time, -c compiles will take less time and the link step will take longer. Normally, this is just a time shift from one build phase to another with roughly the same overall compile time. In some cases increased inlining may cause an increase in overall compile time. Using -hwp allows the compiler backend to be invoked in parallel during a build. Setting the environment variable NPROC controls the number of concurrent compiler backend invocations and this parallelism may reduce overall compile time. 2.10.31 -h zero Initializes all undefined local numeric stack variables to 0. If a user variable is of type character, it is initialized to NUL. If a user variable is type logical, it is initialized to false. The variables are initialized upon each execution of each procedure. Enabling this option can help identify problems caused by using uninitialized numeric and logical variables. Equivalent to specifying -e0. See -d disable and -e enable on page 31. 2.11 -I incldir The -I incldir option specifies a directory to be searched for files named in INCLUDE lines in the Fortran source file and for files named in #include source preprocessing directives. Additionally, all user-specified -I incldir directories are searched for MODULE USE resolution after all user-specified -p paths are searched. S–3901–80 47Cray Fortran Reference Manual You must specify an -I option for each directory you want searched. Directories can be specified in incldir as full path names or as path names relative to the working directory. By default, only the directory of the file referencing the included file and system directories are searched. None of the system-specified -I incldir directories are searched during MODULE USE resolution. The following example causes the compiler to search for files included within earth.f in the directories /usr/local/sun and ../moon: % ftn -I /usr/local/sun -I ../moon earth.f If the INCLUDE line or #include directive in the source file specifies an absolute name (that is, one that begins with a slash (/)), that name is used, and no other directory is searched. If a relative name is used (that is, one that does not begin with a slash (/)), the compiler searches for the file in the directory of the source file containing the INCLUDE line or #include directive. If this directory contains no file of that name, the compiler then searches the directories named by the -I options, as specified on the command line, from left to right. 2.12 -J dir_name The -J dir_name option specifies the directory to which file.mod files are written when the -e m option is specified on the command line. By default, module files are written to the current working directory. The compiler will automatically search the dir_name directory for modules to satisfy USE statements. An error is issued if the -em option is not specified when the -J dir_name is used. 2.13 -K trap=opt[,opt] ... Enable traps for the specified exceptions. By default, no exceptions are trapped. Enabling traps by using this option also has the effect of setting -h fp_trap. If the specified options contradict each other, the last option predominates. For example, -K trap=none,fp is equivalent to -Ktrap=fp. 48 S–3901–80Invoking the Cray Fortran Compiler [2] This option is processed only at link time and affects the entire program; it is not processed when compiling subprograms. Therefore, use this command line option to set traps only at the beginning of execution of the main program. The program may subsequently change these settings by calling intrinsic or library procedures. If you use this option, you may have to specify -hfp_trap when you compile other files of the application. opt Exceptions denorm Trap on denormalized operands. divz Trap on divide-by-zero. fp Trap on divz, inv, or ovf exceptions. inexact Trap on inexact result (i.e., rounded result). Enabling traps for inexact results is not recommended. inv Trap on invalid operation. none Disables all traps (default). ovf Trap on overflow (i.e., the result of an operation is too large to be represented). unf Trap on underflow (i.e., the result of an operation is too small to be represented). 2.14 -l libname The -l libname option directs the compiler driver to search for the specified object library file when linking an executable file. To request more than one library file, specify multiple -l options. When statically linking, the compiler driver searches for libraries by prepending ldir/lib to libname and appending .a, for each ldir that has been specified by using the -L option. It uses the first file it finds. When dynamically linking, the library search process is similar to the static case, with a few differences. The compiler driver searches for libraries by prepending ldir/lib on the front of libname and appending .so on the end of it, for each ldir that has been specified by using the -L option. If a matching .so is not found, the compiler driver replaces .so with .a and repeats the process from the beginning. It uses the first file it finds. There is no search order dependency for libraries. If you specify personal libraries by using the -l command line option, those libraries are added before the default CCE library list. S–3901–80 49Cray Fortran Reference Manual For example, when the following command line is issued, the linker looks for a library named libmylib.a (following the naming convention) and adds it to the top of the list of default libraries. % ftn -l mylib target.f For more information about library search rules, see -L ldir on page 50 and -rpath ldir on page 78. 2.15 -L ldir The -L ldir option changes the -l option search algorithm to look for library files in directory ldir during link time. To request more than one library directory, specify multiple -L options. Note: Multiple -L options are treated cumulatively as if all ldir arguments appeared on one -L option preceding all -l options. Therefore, do not attempt to link functions of the same name from different libraries through the use of alternating -L and -l options. The compiler driver searches for library files in directory ldir before searching the default directories: /opt/ctl/libs and /lib. For example, when statically linking, if -L ../mylib, -L /loclib, and -l m are specified, the compiler driver searches for the following files and uses the first one found: ../mylibs/libm.a /loclib/libm.a /opt/ctl/libs/libm.a /lib/libm.a For more information about search rules for dynamically linked libraries, see -rpath ldir on page 78. For information about specifying module locations, see -p module_site [,module_site] on page 70. 50 S–3901–80Invoking the Cray Fortran Compiler [2] 2.16 -m msg_lvl The -m msg_lvl option specifies the minimum compiler message levels to enable. The following list shows the integers to specify in order to enable each type of message and which messages are generated by default. msg_lvl Message types enabled 0 Error, warning, caution, note, and comment 1 Error, warning, caution, and note 2 Error, warning, and caution 3 Error and warning (default) 4 Error Caution and warning messages denote, respectively, possible and probable user errors. By default, messages are sent to the standard error file, stderr, and are displayed on your terminal. If the -r option is specified, messages are also sent to the listing file. To see more detailed explanations of messages, use the explain command. This command retrieves message explanations and displays them online. For example, to obtain documentation on message 500, enter the following command: % explain ftn-500 The default msg_lvl is 3, which suppresses messages at the comment, note, and caution level. It is not possible to suppress messages at the error level. To suppress specific comment, note, caution, and warning messages, see -M msgs on page 51. To obtain messages regarding nonstandard Fortran usage, specify -e n. For more information about this option, see -d disable and -e enable on page 31. 2.17 -M msgs The -M msgs option suppresses specific messages at the warning, caution, note, and comment levels and can change the default message severity to an error or a warning level. You cannot suppress or alter the severity of error-level messages with this option. To suppress messages, specify one or more integer numbers that correspond to the Cray Fortran compiler messages you want to suppress. To specify more than one message number, specify a comma (but no spaces) between the message numbers. For example, -M 110,300 suppresses messages 110 and 300. S–3901–80 51Cray Fortran Reference Manual To change a message's severity to an error level or a warning level, specify an E (for error) or a W (for warning) and then the number of the message. For example, consider the following option: -M 300,E600,W400. This specification results in the following messages: • Message 300 is disabled and is not issued, provided that it is not an error-level message by default. Error-level messages cannot be suppressed and cannot have their severity downgraded. • Message 600 is issued as an error-level message, regardless of its default severity. • Message 400 is issued as a warning-level message, provided that it is not an error-level message by default. 2.18 -N col The -N col option specifies the line width for fixed- and free-format source lines. The value used for col specifies the maximum number of columns per line. For free form sources, col can be set to 132 or 255. For fixed form sources, col can be set to 72, 80, 132, or 255. Characters in columns beyond the col specification are ignored. By default, lines are 72 characters wide for fixed-format sources and 255 characters wide for free-form sources. 2.19 -O opt [,opt] ... The -O opt option specifies optimization features. You can specify more than one -O option, with accompanying arguments, on the command line. If specifying more than one argument to -O, separate the individual arguments with commas and do not include intervening spaces. Note: The -eo option or the ftnlx command displays the optimization options the compiler uses at compile time. The -eo option is identical to the -h display_opt option which is provided for command-line compatibility with the Cray C compiler. The -O 0, -O 1, -O 2, and -O 3 options allow you to specify a general level of optimization that includes vectorization, scalar optimization, and inlining. Generally, as the optimization level increases, compilation time increases and execution time decreases. The -O 1, -O 2, and -O 3 specifications do not directly correspond to the numeric optimization levels for scalar optimization, vectorization, and inlining. For example, specifying -O 3 does not necessarily enable vector3. Cray reserves the right to alter the specific optimizations performed at these levels from release to release. 52 S–3901–80Invoking the Cray Fortran Compiler [2] The other optimization options, such as -O aggress and -O cachen, control pattern matching, cache management, zero incrementing, and several other optimization features. Some of these features can also be controlled through compiler directives. Figure 1 shows the relationships between some of the -O opt values. Figure 1. Optimization Values thread0 thread1 thread2 thread3 X X X X X X X X X X Low compile cost Moderate compile cost Potentially high compile cost Potential numerical differences from unoptimized execution (operator reassociation) Implies at least scalar1 Implies at least scalar2 Loop nest restructuring Vectorize array syntax statements OpenMP disabled X scalar0 scalar1 scalar2 scalar3 vector0 vector1 vector2 vector3 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X S–3901–80 53Cray Fortran Reference Manual The following -O options are also available with the -h option for command-line compatibility with the Cray C compiler: • [no]autothread • [no]autoprefetch • cache0-3 • fp0-3 • ipa0-5 • ipafrom • loop_trips • mpi0-3 • [no]msgs • [no]negmsgs • [no]omp • [no]pattern • thread0-3 • [no]zeroinc 54 S–3901–80Invoking the Cray Fortran Compiler [2] 2.19.1 -O n The -On option performs general optimization at these levels: 0 (none), 1 (conservative), 2 (moderate, default), and 3 (aggressive). • The -O 0 option inhibits optimization including inlining. This option's characteristics include low compile time, small compile size, and no global scalar optimization. Most array syntax statements are vectorized, but all other vectorizations are disabled. • The -O 1 option specifies conservative optimization. This option's characteristics include moderate compile time and size, global scalar optimizations, and loop nest restructuring. Results may differ from the results obtained when -O 0 is specified because of operator reassociation. No optimizations will be performed that might create false exceptions. Only array syntax statements and inner loops are vectorized and the system does not perform some vector reductions. User tasking is enabled, so !$OMP directives are recognized. • The -O 2 option specifies moderate optimization. This option's characteristics include moderate compile time and size, global scalar optimizations, pattern matching, and loop nest restructuring. Results may differ from results obtained when -O 1 is specified because of vector reductions. The -O 2 option enables automatic vectorization of array syntax and entire loop nests. This is the default level of optimization. • The -O 3 option specifies aggressive optimization. This option's characteristics include a potentially larger compile size, longer compile time, global scalar optimizations, possible loop nest restructuring, and pattern matching. The optimizations performed might create false exceptions in rare instances. Results may differ from results obtained when -O 1 is specified because of vector reductions. 2.19.2 -O [no]aggress The -O aggress option causes the compiler to treat a program unit (for example, a subroutine or a function) as a single optimization region. Doing so can improve the optimization of large program units by raising the limits for internal tables, which increases opportunities for optimization. This option increases compile time and size. Default: -O noaggress S–3901–80 55Cray Fortran Reference Manual 2.19.3 -O [no]autoprefetch The -O [no]autoprefetch option controls automatic software prefetch optimization. It does not affect the loop_info [no]prefetch directive. Default: -O autoprefetch 2.19.4 -O cachen The -O cachen option specifies the following levels of automatic cache management. • -O cache0 specifies no automatic cache management; all memory references are allocated to cache. Both automatic cache blocking and manual cache blocking (by use of the BLOCKABLE directive, as described in Permit Cache Blocking: BLOCKABLE Directive on page 126) are shut off. Characteristics include low compile time. The -O cache0 option is compatible with all scalar and vector optimization levels. • -O cache1 specifies conservative automatic cache management. Characteristics include moderate compile time. Data are placed in the cache when the possibility of cache reuse exists and the predicted cache footprint of the datum in isolation is small enough to experience the reuse. • -O cache2 specifies moderately aggressive automatic cache management. Characteristics include moderate compile time. Data are placed in the cache when the possibility of cache reuse exists and the predicted state of the cache model is such that the datum will experience the reuse. • -O cache3 specifies aggressive automatic cache management. Characteristics include potentially high compile time. Data are placed in the cache when the possibility of cache reuse exists and the allocation of the datum to the cache is predicted to increase the number of cache hits. Default: -O cache2 2.19.5 -O fpn The -O fp option allows you to control the level of floating-point optimizations. The n argument controls the level of allowable optimization; 0 gives the compiler minimum freedom to optimize floating-point operations, while 3 gives it maximum freedom. The higher the level, the less the floating-point operations conform to the IEEE standard. This option is useful for code that uses unstable algorithms, but which is optimizable. It is also useful for applications that want aggressive floating-point optimizations that go beyond what the Fortran standard allows. 56 S–3901–80Invoking the Cray Fortran Compiler [2] Generally, this is the behavior and usage for each -O fp level: • -O fp0 causes your program's executable code to conform more closely to the IEEE floating-point standard than the default mode (-O fp2). When this level is specified, many identity optimizations are disabled, executable code is slower than higher floating-point optimization levels, floating point reductions are disabled, and a scaled complex divide mechanism is enabled that increases the range of complex values that can be handled without producing an underflow. The -O fp0 option should only be used when your code pushes the limits of IEEE accuracy or requires strong IEEE standard conformance. • -O fp1 performs various, generally safe, IEEE non-conforming optimizations, such as folding a == a to true, where a is a floating point object. At this level, floating-point reassociation 1 is greatly limited, which may affect the performance of your code. The -O fp1 options should only be used when your code pushes the limits of IEEE accuracy, or requires substantial IEEE standard conformance. • -O fp2 includes optimizations of -O fp1. This is the default. • -O fp3 includes optimizations of -O fp1 and -O fp2. The -O fp3 option should be used when performance is more critical than the level of IEEE standard conformance provided by -O fp2. Table 2 compares the various optimization levels of the -O fp option (levels 2 and 3 are usually the same). The table lists some of the optimizations performed; the compiler may perform other optimizations not listed. 1 An example of reassociation is when a+b+c is rearranged to b+a+c, where a, b, and c are floating point variables. S–3901–80 57Cray Fortran Reference Manual Table 2. Floating-point Optimization Levels Optimization Type fp0 fp1 fp2 (default) fp3 Safety Maximum Moderate Moderate Low Complex divisions Accurate and slower Accurate and slower Fast 2 Fast 2 Exponentiation rewrite None None Maximum performance 3 Maximum performance 3, 4 Strength reduction Fast Fast Aggressive Aggressive Rewrite division as reciprocal equivalent 5 None None Yes Aggressive Floating point reductions Slow Fast Fast Fast Expression factoring None Yes Yes Yes Expression tree balancing None None Yes Yes Inline 32-bit operations 6 No No No Yes Fused multiply-add 7 No Yes Yes Yes 2.19.6 -O fusionn The -O fusionn option globally controls loop fusion and changes the assertiveness of the FUSION directive. Loop fusion can improve the performance of loops, though in rare cases it may degrade performance. 2 Algebraically correct but may lack precision in boundary cases. 3 Rewriting values raised to a constant power into an algebraically equivalent series of multiplications and/or square roots. 4 Rewriting exponentiations (a b ) not previously optimized into the algebraically equivalent form exp(b * ln(a)). 5 For example, x/y is transformed to x * 1.0/y. 6 32-bit division, square root, and reciprocal square root use very fast but less precise code sequences. 7 Uses fused multiply-add instructions on architectures that support it. 58 S–3901–80Invoking the Cray Fortran Compiler [2] The n argument allows you to turn loop fusion on or off and determine where fusion should occur. It also affects the assertiveness of the FUSION directive. Use one of these values for n: 0 No fusion (ignore all FUSION directives and do not attempt to fuse other loops) 1 Attempt to fuse loops that are marked by the FUSION directive. 2 (default) Attempt to fuse all loops (includes array syntax implied loops), except those marked with the NOFUSION directive. 2.19.7 -O inlinelib Deferred Implementation: Support for the -O inlinelib option has been deferred until a later release. The -O inlinelib option causes the compiler to attempt inlining of those Cray scientific library routines that are available for inlining. For a report of what was inlined or not, see the -O msgs,negmsgs option. This option is off by default. 2.19.8 -O ipan Control interprocedural analysis (IPA) optimization which includes control over the level of automatic inlining, and cloning. Table 3 explains what type of optimization is performed at each level. Table 3. IPA level IPA level Description 0 All inlining and cloning are disabled. All inlining and cloning compiler directives are ignored. 1 Directive inlining. Inlining is attempted for call sites and routines that are under the control of an inlining compiler directive. See Chapter 4, Using Cray Fortran Directives on page 93. Cloning disabled and cloning directives are ignored. S–3901–80 59Cray Fortran Reference Manual IPA level Description 2 Call nest inlining. Inline a call nest to an arbitrary depth as long as the nest does not exceed some compiler-determined threshold. A call nest can be a leaf routine. The expansion of the call nest must yield straight-line code (code containing no external calls) for any expansion to occur. The call site is said to "flatten" when there are no calls present in the expanded code. The call site must reside within the body of a loop for expansion to be attempted. Cloning disabled and cloning directives are ignored. 3 Constant actual argument inlining and tiny routine inlining. Default level for inlining. This includes levels 1 and 2, plus any call site that contains a constant actual argument. Additionally, any call nest (regardless of location) that is below some small compiler-determined threshold will be inlined provided that call nest completely flattens. Cloning disabled and cloning directives are ignored. 4 This includes levels 1, 2, and 3, plus routine cloning is attempted if inlining fails at a given call site. Cloning directives are enabled. 5 Aggressive interprocedural analysis (IPA). Includes levels 1, 2, 3, and 4. 2.19.8.1 Inlining Inlining is the process of replacing a user procedure call with the procedure definition itself. This saves subprogram call overhead and may allow better optimization of the inlined code. If all calls within a loop are inlined, the loop becomes a candidate for parallelization. The compiler supports the following inlining modes through the indicated options: • Automatic inlining allows the compiler to automatically select which functions to inline. This occurs with the -Oipa2 or greater option. • Explicit inlining allows you to explicitly indicate which procedures the compiler should attempt to inline and occurs with the -O ipafrom=source option. • Combined inlining allows you to specify potential targets for inline expansion, while applying the selected level of inlining heuristics. If -O ipan is used in conjunction with -O ipafrom=source, the candidates for expansion are those functions present in source. 2.19.8.2 Cloning Cloning replaces a call to a procedure with a call to a modified version of that same procedure (clone). 60 S–3901–80Invoking the Cray Fortran Compiler [2] Automatic cloning is enabled at -Oipa4 and higher. The compiler first attempts to inline a call site. If inlining the call site fails, the compiler may attempt to clone the procedure for the specific call site if a potential code performance improvement can be attained. Constant actual arguments is just one situation where the compiler will attempt cloning. When the clone is made, dummy arguments are replaced with associated constant values throughout the routine. The following example shows this situation: PROGRAM TEST CALL SAM(3, .TRUE.) ! Call site with constants END SUBROUTINE SAM(I, L) INTEGER I LOGICAL L IF (L) THEN PRINT *, I ENDIF END Compiling the previous program with the -O ipa4 option produces the following program: PROGRAM TEST CALL SAM@1(3, .TRUE.) ! This is a call to a clone of SAM. END ! Original Subroutine SUBROUTINE SAM(I, L) INTEGER I LOGICAL L IF (L) THEN PRINT *, I ENDIF END ! Cloned subroutine SUBROUTINE SAM@1(I, L) INTEGER I LOGICAL L IF (.TRUE.) THEN ! The optimizer will eliminate this PRINT *, 3 ENDIF END S–3901–80 61Cray Fortran Reference Manual 2.19.9 -O ipafrom=source[:source]... The -O ipafrom=source[:source] option allows you to explicitly indicate the procedures to consider for inline expansion or cloning. The source arguments identify each file or directory that contains the routines to consider for inlining or cloning. Note: Blank spaces are not allowed on either side of the equal sign. All inlining directives are recognized with explicit inlining, however cloning and cloning directives are only enabled when -O ipa4 or greater. For information about inlining directives, see Chapter 4, Using Cray Fortran Directives on page 93. Note that the routines in source are not actually linked with the final program. They are simply templates for the inliner. To have a routine contained in source linked with the program, you must include it in an input file to the compilation. Use one or more of the objects described in Table 4 in the source argument. Table 4. File Types Fortran source files The routines in Fortran source files are candidates for inline expansion and must contain error-free code. Source files that are acceptable for inlining are files that have one of the following extensions: • .f • .F • .f90 • .F90 • .f95 • .F95 • .f03 • .F03 • .f08 • .F08 • .ftn • .FTN 62 S–3901–80Invoking the Cray Fortran Compiler [2] Module files When compiling with -em and -Omodinline is in effect, the precompiled module information is written to modulename.mod. The compiler writes a modulename.mod file for each module; modulename is created by taking the name of the module and, if necessary, converting it to uppercase. dir A directory that contains any of the file types described in this table. Combined inlining is invoked by specifying the -O ipan and -O ipafrom= options on the command line. This inlining mode will look only in source for potential targets for expansion, while applying the selected level of inlining heuristics specified by the -O ipan option. 2.19.10 -O [no]modinline The -O modinline option prepares module procedures so they can be inlined by directing the compiler to create templates for module procedures encountered in a module. These templates are attached to file.o or modulename.mod. The files that contain these inlinable templates can be saved and used later to inline call sites within a program being compiled. When -e m is in effect, module information is stored in modname.mod. The compiler writes a modulename.mod file for each module; modulename is created by taking the name of the module and, if necessary, converting it to uppercase. The process of inlining module procedures requires only that file.o or modulename.mod be available during compilation through the typical module processing mechanism. The USE statement makes the templates available to the inliner. You do not need to specify the file.o or modulename.mod with the -O ipafrom option. When -O modinline is specified, the MODINLINE and NOMODINLINE directives are recognized. Using the -O modinline option increases the size of file.o. To ensure that file.o is not removed, specify this option in conjunction with the -c option. For information about the -c option, see -c on page 31. Default: -O modinline 2.19.11 -O [no]msgs The -O msgs option causes the compiler to write optimization messages to stderr. S–3901–80 63Cray Fortran Reference Manual Similar information in a more-readable format can be obtained by using the -rm option instead. Specifying the -rm option enables -O msgs. For more information, see -r list_opt on page 74. Default: -O nomsgs 2.19.12 -O [no]negmsgs The -O negmsgs option causes the compiler to generate messages to stderr that indicate why optimizations such as vectorization or inlining did not occur in a given instance. The -O negmsgs option enables the -O msgs option. The -rm option enables the -O negmsgs option. Default: -O nonegmsgs 2.19.13 -O nointerchange The -O nointerchange option inhibits the compiler's attempts to interchange loops. Interchanging loops by having the compiler replace an inner loop with an outer loop can increase performance. The compiler performs this optimization by default. Specifying the -O nointerchange option is equivalent to specifying a NOINTERCHANGE directive prior to every loop. To disable loop interchange on individual loops, use the NOINTERCHANGE directive. For more information about the NOINTERCHANGE directive, see Control Loop Interchange: [NO]INTERCHANGE on page 119. 2.19.14 -O [no]omp The -O [no]omp option enables or disables compiler recognition of OpenMP directives. Using the -O noomp option is similar to the -O thread0 option, in that it disables OpenMP, but unlike -O thread0 it does not affect autothreading. The -O [no]omp option is identical to the -h [no]omp option. Default: -O omp 2.19.15 -O [no]overindex The -O nooverindex option declares that there are no array subscripts which index a dimension of an array that are outside the declared bounds of that dimension. Short loop code generation occurs when the extent does not exceed the maximum vector length of the machine. 64 S–3901–80Invoking the Cray Fortran Compiler [2] Specifying -O overindex declares that the program contains code that makes array references with subscripts that exceed the defined extents. This prevents the compiler from performing the short loop optimizations described in the preceding paragraph. Overindexing is nonstandard, but it compiles correctly as long as data dependencies are not hidden from the compiler. This technique collapses loops; that is, it replaces a loop nest with a single loop. An example of this practice is as follows: DIMENSION A(20, 20) DO I = 1, N A(I, 1) = 0.0 END DO Assuming that N equals 400 in the previous example, the compiler might generate more efficient code than a doubly nested loop. However, incorrect results can occur in this case if -O nooverindex is in effect. You do not need to specify -O overindex if the overindexed array is a Cray pointee, has been equivalenced, or if the extent of the overindexed dimension is declared to be 1 or *. In addition, the -O overindex option is enabled automatically for the following extension code, where the number of subscripts in an array reference is less than the declared number: DIMENSION A(20, 20) DO I = 1, N A(I) = 0.0 ! 1-dimension reference; ! 2-dimension array END DO Note: The -O overindex option is used by the compiler for detection of short loops and subsequent code scheduling. This allows manual overindexing as described in this section, but it may have a negative performance effect because of fewer recognized short loops and more restrictive code scheduling. In addition, the compiler continues to assume, by default, a standard-conforming user program that does not overindex when doing dependency analysis for other loop nest optimizations. Default: -O nooverindex 2.19.16 -O [no]pattern The -O pattern option enables pattern matching for library substitution. The pattern matching feature searches your code for specific code patterns and replaces them with calls to highly optimized routines. The -O pattern option is enabled only for optimization levels -O 2, -O vector2 or higher; there is no way to force pattern matching for lower levels. S–3901–80 65Cray Fortran Reference Manual Specifying -O nopattern disables pattern matching and causes the compiler to ignore the PATTERN and NOPATTERN directives. For information about the PATTERN and NOPATTERN directives, see Request Pattern Matching: [NO]PATTERN on page 104. Default: -O pattern 2.19.17 -O scalarn The -O scalarn option specifies these levels of scalar optimization: • scalar0 disables scalar optimization. Characteristics include low compile time and size. The -O scalar0 option is compatible with -O vector0. • scalar1 specifies conservative scalar optimization. Characteristics include moderate compile time and size. Results can differ from the results obtained when -O scalar0 is specified because of operator reassociation. No optimizations are performed that could create false exceptions. The -O scalar1 option is compatible with -O vector0 or -O vector1. • scalar2 specifies moderate scalar optimization. Characteristics include moderate compile time and size. Results can differ slightly from the results obtained when -O scalar1 is specified because of possible changes in loop nest restructuring. Generally, no optimizations are done that could create false exceptions. The -O scalar2 option is compatible with all vectorization levels. This is the default scalar optimization level. • scalar3 specifies aggressive scalar optimization. Characteristics include potentially greater compile time and size. Results can differ from the results obtained when -O scalar1 is specified because of possible changes in loop nest restructuring. The optimization techniques used can create false exceptions in rare instances. Analysis that determines whether a variable is used before it is defined is enabled at this level. 2.19.18 -O shortcircuitn The -O shortcircuitn option specify various levels of short circuit evaluation. Short circuit evaluation is an optimization in which the compiler analyzes all or part of a logical expression based on the results of a preliminary analysis. When short circuiting is enabled, the compiler attempts short circuit evaluation of logical expressions that are used in IF statement scalar logical expressions. This evaluation is performed on the .AND. operator and the .OR. operator. 66 S–3901–80Invoking the Cray Fortran Compiler [2] Example 1: Assume the following logical expression: operand1 .AND. operand2 The operand2 need not be evaluated if operand1 is false because in that case, the entire expression evaluates to false. Likewise, if operand2 is false, operand1 need not be evaluated. Example 2: Assume the following logical expression: operand1 .OR. operand2 The operand2 need not be evaluated if operand1 is true because in that case, the entire expression evaluates to true. Likewise, if operand2 is true, operand1 need not be evaluated. The compiler performs short circuit evaluation in a variety of ways, based on the following command line options: • -O shortcircuit0 disables short circuiting of IF and ELSEIF statement logical conditions. • -O shortcircuit1 specifies short circuiting of IF and ELSEIF logical conditions only when a PRESENT, ALLOCATED, or ASSOCIATED intrinsic procedure is in the condition. The short circuiting is performed left to right. In other words, the left operand is evaluated first, and if it determines the value of the operation, the right operand is not evaluated. The following code segment shows how this option could be used: SUBROUTINE SUB(A) INTEGER,OPTIONAL::A IF (PRESENT(A) .AND. A==0) THEN ... The expression A==0 must not be evaluated if A is not PRESENT. The short circuiting performed when -O shortcircuit1 is in effect causes the evaluation of PRESENT(A) first. If that is false, A==0 is not evaluated. If -O shortcircuit1 is in effect, the preceding example is equivalent to the following example: SUBROUTINE SUB(A) INTEGER,OPTIONAL::A IF (PRESENT(A)) THEN IF (A==0) THEN ... • -O shortcircuit2 specifies short circuiting of IF and ELSEIF logical conditions, and it is done left to right. All .AND. and .OR. operators in these expressions are evaluated in this way. The left operand is evaluated, and if it determines the result of the operation, the right operand is not evaluated. S–3901–80 67Cray Fortran Reference Manual • -O shortcircuit3 specifies short circuiting of IF and ELSEIF logical conditions. It is an attempt to avoid making function calls. When this option is in effect, the left and right operands to .AND. and .OR. operators are examined to determine if one or the other contains function calls. If either operand has functions, short circuit evaluation is performed. The operand that has fewer calls is evaluated first, and if it determines the result of the operation, the remaining operand is not evaluated. If both operands have no calls, then no short circuiting is done. For the following example, the right operand of .OR. is evaluated first. If A==0 then ifunc() is not called: IF (ifunc() == 0 .OR. A==0) THEN ... -O shortcircuit2 is the default. 2.19.19 -O threadn The -O threadn option enables you to control the compilation and optimization of OpenMP directives and automatic threading, where n is a value from 0 to 3 with 0 being off and 3 specifying the most aggressive optimization. This option is identical to the -h threadn option. The valid values for n are: 0 No autothreading or OpenMP threading. The -O thread0 option is similar to -O noomp, but -O noomp disables OpenMP only and does not affect autothreading. 1 Specifies strict compliance with the OpenMP standard for directive compilation. Strict compliance is defined as no extra optimizations in or around OpenMP constructs. In other words, the compiler performs only the requested optimizations. 2 OpenMP parallel regions are subjected to some optimizations; that is, some parallel region expansion. Parallel region expansion is an optimization that merges two adjacent parallel regions in a compilation unit into a single parallel region. Limited loop restructuring is done on OpenMP partitioned loop. Legal scalar optimizations are performed across OpenMP constructs. 3 Full optimization: loop restructuring, including modifying iteration space for static schedules (breaking standard compliance). Reduction results may not be repeatable. Default: -O thread2 68 S–3901–80Invoking the Cray Fortran Compiler [2] 2.19.20 -O unrolln The -O unrolln option globally controls loop unrolling and changes the assertiveness of the UNROLL directive. By default, the compiler attempts to unroll all loops, unless the NOUNROLL directive is specified for a loop. Generally, unrolling loops increases single processor performance at the cost of increased compile time and code size. The n argument allows you to turn loop unrolling on or off and determine where unrolling should occur. It also affects the assertiveness of the UNROLL directive. Use one of these values for n: 0 No unrolling (ignore all UNROLL directives and do not attempt to unroll other loops) 1 Honor the UNROLL directive. Attempt to unroll loops if there is proof that the loop will benefit. 2 (default) Attempt to unroll all loops (includes array syntax implied loops), except those marked with the NOUNROLL directive, if a performance benefit is expected. 2.19.21 -O vectorn The -O vectorn option specifies the level of vectorization performed by the compiler. Vectorization results in dramatic performance improvements with a small increase in object code size. Vectorization directives are unaffected by this option. • -O vector0 specifies minimal automatic vectorization. Characteristics include low compile time and small compile size. The -O vector0 option is compatible with all scalar optimization levels. The compiler will still vectorize array syntax in order to allow full source level debugging with reasonable performance. When this option is specified in conjunction with -hfp0 or -hfp1, then array syntax containing associative floating point or complex operations will not be vectorized, • -O vector1 specifies conservative vectorization. The -O vector1 option is compatible with -O scalar1, -O scalar2, or -O scalar3. S–3901–80 69Cray Fortran Reference Manual • -O vector2 specifies moderate vectorization. Loop nests are restructured. The -O vector2 option is compatible with -O scalar2 or -O scalar3. This is the default vectorization level. • -O vector3 specifies aggressive vectorization. The -O vector3 option is compatible with -O scalar2 or -O scalar3. 2.19.22 -O [no]zeroinc The -O zeroinc option causes the compiler to assume that a constant increment variable (CIV) can be incremented by zero. A CIV is a variable that is incremented only by a loop invariant value. For example, in a loop with variable J, the statement J = J + K, where K can be equal to zero, J is a CIV. -O zeroinc can cause less strength reduction to occur in loops that have variable increments. Default: -O nozeroinc 2.20 -o out_file The -o out_file option overrides the default executable file name, a.out, with the name specified by the out_file argument. If the -o out_file option is specified on the command line along with the -c option, the link step is disabled and the binary file is written to the out_file specified as an argument to -o. For more information about the -c option, see -c on page 31. 2.21 -p module_site [,module_site] The -p module_site option tells the compiler where to look for Fortran modules to satisfy USE statements. The module_site argument specifies the name of a file or directory to search for modules. The module_site specified can be a .mod file, .o (object) file, .a (archive) file, or a directory. By default, module files are written to the current working directory. Alternatively, you can use the -J dir_name option during compilation to specify an alternate output directory for .mod files only. The compiler will search for modules stored in the directories you specified using the -J dir_name option for the current compilation automatically; you do not need to use the -p option explicitly to make the compiler do this. For more information about the -J dir_name option, see -J dir_name on page 48. 70 S–3901–80Invoking the Cray Fortran Compiler [2] The search order for satisfying module references in USE statements is as follows: 1. The current working directory (or -J dir_name directory, if specified). 2. Any directories or files specified with the -p option. 3. Any directories specified with the -I option. 4. Any directories or files specified with the FTN_MODULE_PATH environment variable. When searching within a directory, the compiler first searches the .mod files, then the .o files, then the .a files, and then the directories, in the order specified. File name substitution (such as *.o) is not allowed. If the path name begins with a slash (/), the name is assumed to be an absolute path name. Otherwise, it is assumed to be a path name relative to the working directory. You can specify multiple module_site locations with the -p option either by separating them with commas or by using a separate -p argument for each module_site. There is no limit on the number of -p options you can specify. Cray provides some modules as part of the Cray Fortran Compiler Programming Environment. These are referred to as system modules. The system files containing these modules are searched last. Example 1: Consider the following command line: % ftn -p steve.o -p mike.o joe.f Assume that steve.o contains a module called Rock and mike.o contains a module called Stone. A reference to use Rock in joe.f causes the compiler to use Rock from steve.o. A reference to Stone in joe.f causes the compiler to use Stone from mike.o. Example 2: The following example specifies binary file murphy.o and library file molly.a: % ftn -p murphy.o -p molly.a prog.f Example 3: In this example, assume that the following directory structure exists in your home directory: programs / | \ tests one.f two.f | use_it.f The following module is in file programs/one.f, and the compiled version of it is in programs/one.o: MODULE one INTEGER i END MODULE S–3901–80 71Cray Fortran Reference Manual The next module is in file programs/two.f, and the compiled version of it is in programs/two.o: MODULE two INTEGER j END MODULE The following program is in file programs/tests/use_it.f: PROGRAM demo USE one USE two . . . END PROGRAM To compile use_it.f, enter the following command from your home directory, which contains the subdirectory programs: % ftn -p programs programs/tests/use_it.f Example 4: In the next set of program units, a module is contained within the first program unit and accessed by more than one program unit. The first file, progone.f, contains the following code: MODULE split INTEGER k REAL a END MODULE PROGRAM demopr USE split INTEGER j j = 3 k = 1 a = 2.0 CALL suba(j) PRINT *, 'j=', j PRINT *, 'k=', k PRINT *, 'a=', a END 72 S–3901–80Invoking the Cray Fortran Compiler [2] The second file, progtwo.f, contains the following code: SUBROUTINE suba(l) USE split INTEGER l l = 4 k = 5 CALL subb(l) RETURN END SUBROUTINE subb(m) USE split INTEGER m m = 6 a = 7.0 RETURN END Use the following command line to compile the two files with one ftn command and a relative pathname: % ftn -p progone.o progone.f progtwo.f When the -e m option is in effect, you can use the -p module_site option to specify one or more directories that contain module files rather than specifying every individual module file name. 2.22 -Q path The -Q path option specifies the directory that will contain all saved nontemporary files from this compilation (for example, all .o and .mod files). Specific file types (like .o files) are saved to a different directory if the -b, -J, -o, or -S option is specified. The following examples use this directory structure: current_dir ---------------------- | | | | | | | | | bin_out mod_out all_out The following example saves all nontemporary files (x.o and any .mod files) in the current directory: % ftn -b x.o -em x.f90 The following example saves all nontemporary files in the all_out directory and x.o in the current directory. % ftn -Q all_out -em -b x.o x.f90 S–3901–80 73Cray Fortran Reference Manual The following example saves the x.o file to the bin_out and all .mod files to the all_out directory. % ftn -Q all_out -b bin_out/x.o -em x.f90 The following example saves the a.out file to the all_out and all .mod files to the mod_out directory. % ftn -Q all_out -J mod_out x.f90 2.23 -r list_opt The -r list_opt option generates a listing. The list_opt argument produces listings with commonly needed information. If one or more input files are specified on the compiler command line, the listing is placed in file.lst. The arguments for list_opt are shown below. Note: Options a, c, l, m, o, s, T, and x invoke the ftnlx command. list_opt Listing Type -r a Includes all reports in the listing (including source, cross references, lint, loopmarks, common block, and options used during compilation). -r c Listing includes a report of all COMMON blocks and all members of each common block. It also shows the program units that use the COMMON blocks. -r d Decompiles (translates) the intermediate representation of the compiler into listings that resemble the format of the source code. This is performed twice, resulting in two output files, at different points during the optimization process. You can use these files to examine the restructuring and optimization changes made by the compiler, which can lead to insights about changes you can make to your Fortran source to improve its performance. The compiler produces two decompilation listing files with these extensions per specified source file: .opt and .cg. The compiler generates the .opt file after applying most high level loop nest transformations to the code. The code structure of this listing most resembles your Fortran code and is readable by most users. In some cases, because of optimizations, the structure of the loops and conditionals will be significantly different than the structure in your source file. 74 S–3901–80Invoking the Cray Fortran Compiler [2] The .cg file contains a much lower level of decompilation. It is still displayed in a Fortran-like format, but is quite close to what will be produced as assembly output. This version displays the intermediate text after all vector translation and other optimizations have been performed. An intimate knowledge of the hardware architecture of the system is helpful to understanding this listing. The .opt and .cg files are intended as a tool for performance analysis, and are not valid Fortran source code. The format and contents of the files can be expected to change from release to release. The following examples show the listings generated when -rd is applied to this example: Note: The column of numbers in the left-hand side of the .opt and .cg files refer to the line number in the Fortran source file. !Source code, in file example.f: subroutine example( a, b, c ) real a(*), b(*), c(*) do i = 1,100 a(i) = b(i) * c(i) enddo end Enter the following command: % ftn -c -rd example.f This is the listing of the example.opt file after loop optimizations are performed: 1. subroutine example( a, b, c ) 3. $Induc01_N4 = 0 3. !dir$ ivdep 3. do 4. A(1 + $Induc01_N4) = C(1 + $Induc01_N4) * B(1 + 4. . $Induc01_N4) 5. $Induc01_N4 = 1 + $Induc01_N4 3. if ( $Induc01_N4 >= 100 ) exit 3. enddo 6. return 6. end -r e Expands included files in the source listing. This option is off by default. -r l Lists source code and includes lint style checking. The listing includes the COMMON block report (see the -r c option for more information about the COMMON block report). S–3901–80 75Cray Fortran Reference Manual -r m Produces a source listing with loopmark information. To provide a more complete report, this option automatically enables the -O negmsg option to show why loops were not optimized. If you do not require this information, use the -O nonegmsg option on the same command line. Loopmark information will not be displayed if the -d B option has been specified. -r o Show in the list file all options used by the compiler at compile time. -r s Lists source code and messages. Error and warning messages are interspersed with the source lines. Optimization messages appear after each program unit. Produces 80-column output by default. -r T Retains file.T after processing rather than deleting it. The file.T can be used to call ftnlx directly. For more information, see the ftnlx(1) man page. -r x Generates a cross-reference listing. 2.24 -R runchk The -R runchk option lets you specify any of a group of run time checks for your program. To specify more than one type of checking, specify consecutive runchk arguments, such as: -R bs. Note: Performance is degraded when run time checking is enabled. This capability, though useful for debugging, is not recommended for production runs. 76 S–3901–80Invoking the Cray Fortran Compiler [2] The run time checks available are as follows: runchk Checking performed b Enables checking of array bounds. If a problem is detected at run time, a message is issued but execution continues. The NOBOUNDS directive overrides this option. For more information about NOBOUNDS, see Check Array Bounds: [NO]BOUNDS on page 124. Note: Bounds checking behavior differs with the optimization level. At the default optimization level, -O 2, some run time checking is inhibited. Complete checking is guaranteed only when optimization is turned off by specifying -O 0 on the ftn command line. c Enables conformance checking of array operands in array expressions. Even without the -R option, such checking is performed during compilation when the dimensions of array operands can be determined. d Enables directive checking at run time. Errors detected at compile time are reported during compilation and so are not reported at run time. The collapse directive is checked, as are the loop_info clauses min_trips and max_trips. Violation of a run time check results in an immediate fatal error diagnostic. p Generates run time code to check the association or allocation status of referenced POINTER variables, ALLOCATABLE arrays, or assumed-shape arrays. A warning message is issued at run time for references to disassociated pointers, unallocated allocatable arrays, or assumed shape dummy arguments that are associated with a pointer or allocatable actual argument when the actual argument is not associated or allocated. s Enables checking of character substring bounds. This option behaves similarly to option -R b. Note: Bounds checking behavior differs with the optimization level. At the default optimization level, -O 2, some run time checking is inhibited. Complete checking is guaranteed only when optimization is turned off by specifying -O 0 on the ftn command line. S–3901–80 77Cray Fortran Reference Manual 2.25 -rpath ldir The -rpath ldir option changes the run time library search algorithm to look for files in directory ldir. To request more than one library directory, specify multiple -rpath options. Note that a library may be found at link time with an -L option, but may not be found at run time if a corresponding -rpath option was not supplied on the link line. Also note that the compiler driver does not pass the -rpath option to the linker. You must explicitly specify -Wl when using this option. At link time, all ldir arguments are added to the executable. The dynamic linker will search these paths first for shared dynamic libraries at run time, with one exception. The Linux environment variable LD_LIBRARY_PATH precedes all other search paths for shared dynamically linked libraries. The use of LD_LIBRARY_PATH is discouraged. ! Caution: Caution should be used when setting LD_LIBRARY_PATH. Doing so will change the shared dynamically linked library search paths for all executable files in your environment. 2.26 -s size The -s size option allows you to modify the sizes of variables, literal constants, and intrinsic function results declared as type REAL, INTEGER, LOGICAL, COMPLEX, DOUBLE COMPLEX, or DOUBLE PRECISION. Use one of these for size: size Action byte_pointer (Default) Applies a byte scaling factor to integers used in pointer arithmetic involving Cray pointers. That is, Cray pointers are moved on byte instead of word boundaries. Pointer arithmetic scaling is explained in Pointer Scaling Factor on page 80. default32 (Default) Adjusts the data size of default types as follows: • 32 bits: REAL, INTEGER, LOGICAL • 64 bits: COMPLEX, DOUBLE PRECISION • 128 bits: DOUBLE COMPLEX Note: The data sizes of integers and logicals that use explicit kind and star values are not affected by this option. However, they are affected by the -e h option. See -d disable and -e enable on page 31. 78 S–3901–80Invoking the Cray Fortran Compiler [2] default64 Adjust the data size of default types as follows: • 64 bits: REAL, INTEGER, LOGICAL • 64 bits: DOUBLE PRECISION (implied -dp) • 128 bits: COMPLEX • 128 bits: DOUBLE COMPLEX (implied -dp) If you used the -s default64 at compile time, you must also specify this option when invoking the ftn command. Note: The data sizes of integers and logicals that use explicit kind and star values are not affected by this option. However, they are affected by the -dh option. See -d disable and -e enable on page 31. integer32 (Default) Adjusts the default data size of default integers and logicals to 32 bits. integer64 Adjusts the default data size of default integers and logicals to 64 bits. real32 (Default) Adjusts the default data size of default real types as follows: • 32 bits: REAL • 64 bits: COMPLEX and DOUBLE PRECISION • 128 bits: DOUBLE COMPLEX real64 Adjusts the default data size of default real types as follows: • 64 bits: REAL • 64 bits: DOUBLE PRECISION (implied -dp) • 128 bits: COMPLEX • 128 bits: DOUBLE PRECISION (implied -dp) word_pointer Applies a word scaling factor to integers used in pointer arithmetic involving Cray pointers. That is, Cray pointers are moved on word instead of byte boundaries. Pointer arithmetic scaling is explained later in Pointer Scaling Factor on page 80. The default data size options (for example, -s default64) option does not affect the size of data that explicitly declare the size of the data (for example, REAL(KIND=4) R. Note: REAL(KIND=16) and COMPLEX(KIND=16) are not supported. S–3901–80 79Cray Fortran Reference Manual 2.26.1 Different Default Data Size Options on the Command Line You must be careful when mixing different default data size options on the same command line because equivalencing data of one default size with data of another default size can cause unexpected results. For example, assume that the following command line is used for a program: % ftn -s default64 -s integer32 ... The mixture of these default size options causes the program below to equivalence 32-bit integer data with 64-bit real data and to incompletely clear the real array. Program test IMPLICIT NONE real r integer i common /blk/ r(10), i(10) integer overlay(10) equivalence (overlay, r) call clear(overlay) call clear(i) contains subroutine clear(i) integer, dimension (10) :: i i = 0 end subroutine end program test The above program sets only the first 10 32-bit words of array r to zero. It should instead set 10 64-bit words to zero. 2.26.2 Pointer Scaling Factor You can specify that the compiler apply a scaling factor to integers used in pointer arithmetic involving Cray pointers so that the pointer is moved to the proper word or byte boundary. For example, the compiler views this code statement: Cray_ptr = Cray_ptr + integer_value as Cray_ptr = Cray_ptr + (integer_value * scaling_factor) 80 S–3901–80Invoking the Cray Fortran Compiler [2] The scaling factor is dependent on the size of the default integer and which scaling option (-s byte_pointer or -s word_pointer) is enabled. Table 5. Scaling Factor in Pointer Arithmetic Scaling Option Default Integer Size Scaling Factor -s byte_pointer 32 or 64 bits 1 -s word_pointer and -s default32 enabled 32 bits 4 -s word_pointer and -s default64 enabled 64 bits 8 Therefore, when the -s byte_pointer option is enabled, this example increments ptr by i bytes: pointer (ptr, ptee) !Cray pointer ptr = ptr + i When the -s word_pointer and -s default32 options are enabled, the same example is viewed by the compiler as: ptr = ptr + (4*i) When the -s word_pointer and -s default64 options are enabled, the same example is viewed by the compiler as: ptr = ptr + (8*i) 2.27 -S asm_file The -S asm_file option specifies the assembly language output file name. When -S asm_file is specified on the command line with either the -e S or -b bin_obj_file options, the -e S and -b bin_obj_file options are overridden. 2.28 -T The -T option disables the compiler but displays all options currently in effect. The Cray Fortran compiler generates information identical to that generated when the -v option is specified on the command line; when -T is specified, however, no processing is performed. When this option is specified, output is written to the standard error file (stderr). S–3901–80 81Cray Fortran Reference Manual 2.29 -U identifier [,identifier] ... The -U identifier [,identifier] ... option undefines variables used for source preprocessing. This option removes the initial definition of a predefined macro or sets a user predefined macro to an undefined state. The -D identifier [=value] option defines variables used for source preprocessing. If both -D and -U are used for the same identifier, in any order, the identifier is undefined. For more information about the -D option, see -D identifier[=value] on page 38. This option is ignored unless one of the following conditions is true: • The Fortran input source file is specified as either file.F, file.F90, file.F95, file.F03, file.F08, or file.FTN. • The -e P or -e Z options have been specified. For more information about source preprocessing, see Chapter 5, Source Preprocessing on page 139. 2.30 -v The -v option sends compilation information to the standard error file (stderr). The information generated indicates the compilation phases as they occur and all options and arguments being passed to each processing phase. 2.31 -V The -V option displays to the standard error file (stderr) the release version of the ftn command. Unlike all other command-line options, you can specify this option without specifying an input file name; that is, specifying ftn -V is valid. 2.32 -Wa"assembler_opt" The -Wa"assembler_opt" option passes assembler_opt directly to the assembler. For example, -Wa"-h" passes the -h option directly the as command, directing it to enable all pseudos, regardless of location field name. This option is meaningful to the system only when file.s is specified as an input file on the command line. For more information about assembler options, see the as(1) man page. 82 S–3901–80Invoking the Cray Fortran Compiler [2] 2.33 -Wr"lister_opt" The -Wr"lister_opt" option passes lister_opt directly to the ftnlx command. For example, specifying -Wr"-o cfile.o" passes the argument cfile.o directly to the ftnlx command's -o option; this directs ftnlx to override the default output listing and put the output file in cfile.o. If you specify the -Wr"lister_opt" option, you must specify the -r list_opt option. For more information about options, see the ftnlx man page. 2.34 -x dirlist The -x dirlist option disables specified directives or specified classes of directives. If specifying a multiword directive, either enclose the directive name in quotation marks or remove the spaces between the words in the directive's name. For dirlist, enter one of the following arguments: dirlist Item disabled all All compiler directives and OpenMP Fortran directives. For information about the OpenMP directives see Chapter 6, Using the OpenMP Fortran API on page 149. dir All compiler directives. directive One or more compiler directives or OpenMP Fortran directives. If specifying more than one, separate them with commas; for example: -x INLINEALWAYS,"NO SIDE EFFECTS",BOUNDS. omp All OpenMP Fortran directives. acc All OpenACC Fortran directives. conditional_omp All C$ and !$ conditional compilation lines. 2.35 -X npes The -X npes option specifies the number of processing elements (PEs) that will be specified at job launch. On Cray XE and Cray XK systems, the value for npes ranges from 1 through 2**31 - 1 inclusive. If -X is specified, the user must invoke aprun -n npes using the same value for npes. Otherwise, a run time error is generated. N$PES is a special symbol whose value is equal to the number of PEs available to your program. When the -X npes option is specified at compile time, the N$PES constant is replaced by integer value npes. S–3901–80 83Cray Fortran Reference Manual Use the N$PES constant when: • The -X npes option is specified on the command line, or • The value of the expression containing the N$PES constant is not known until run time (that is, it can only be used in run time expressions) One of the uses for the N$PES symbol is illustrated in the following example, which declares the size of an array within a subroutine to be dependent on the number of processors: SUBROUTINE WORK DIMENSION A(N$PES) Using the N$PES symbol in conjunction with the -X npes option enables the programmer to program the number of PEs into a program in places that do not accept run time values. Specifying the number of PEs at compile time can also enhance compiler optimization. The programmer is responsible for ensuring that all object files are compiled and linked with the same -X npes value, and for running the resulting executable on that number of PEs. If mixed -X values are used when compiling and linking different object files, or the number of PEs specified at run time differs from that specified when compiling and linking, a run time error is generated. 2.36 -Y phase,dirname The -Y phase,dirname option specifies a new directory (dirname) from which the designated phase should be executed. phase can be one or more of the values shown in Table 6. Table 6. -Y phase Definitions phase System Phase Command 0 Compiler ftn a Assembler as 2.37 -- The -- symbol signifies the end of options. After this symbol, you can specify files to be processed. This symbol is optional. It may be useful if your input file names begin with one or more dash (-) characters. 84 S–3901–80Invoking the Cray Fortran Compiler [2] 2.38 sourcefile [sourcefile.suffix ...] The sourcefile[sourcefile.suffix ...] option names the file or files to be processed. The file suffixes indicate the content of each file and determine whether the preprocessor, compiler, assembler, or linker will be invoked. Preprocessor Files having the F, F90, F95, F03, F08, or FTN suffix invoke the preprocessor. Compiler Fortran source files having the following suffixes invoke the compiler: • .f or .F, indicates a fixed source form file. • .f90, .F90, .f95, .F95, .f03, .F03, .f08, .F08, .ftn, .FTN, indicates a free source form file. Note: The source form specified on the -f source_form option overrides the source form implied by the file suffixes. Linker Files with a .o extension (object files) invoke the linker. If only one source file is specified on the command line, the .o file is created and deleted. To retain the .o file, use the -c option to disable the linker. You can specify object files produced by the Cray Fortran, C, C++, or assembler compilers. Object files are passed to the linker in the order in which they appear on the ftn command line. If the linker is disabled by the -b or -c option, no files are passed to the linker. S–3901–80 85Cray Fortran Reference Manual 86 S–3901–80Setting Environment Variables [3] Environment variables are predefined shell variables, taken from the execution environment, that determine some of your shell characteristics. Several environment variables pertain to the Cray Fortran compiler. The Cray Fortran compiler recognizes general and multiprocessing environment variables. The multiprocessing variables in the following sections affect the way your program will perform on multiple processors. Using environment variables lets you tune the system for parallel processing without rebuilding libraries or other system software. The variables allow you to control parallel processing at compile time and at run time. Compile time environment variables apply to all compilations in a session. The following examples show how to set an environment variable: • With the standard shell, enter: CRAY_FTN_OPTIONS=options export CRAY_FTN_OPTIONS • With the C shell, enter: setenv CRAY_FTN_OPTIONS options The following sections describe the environment variables recognized by the Cray Fortran compiler. Note: Many of the environment variables described in this chapter refer to the default system locations of Programming Environment components. If the Cray Fortran Compiler Programming Environment has been installed in a non-default location, see your system support staff for path information. S–3901–80 87Cray Fortran Reference Manual 3.1 Compiler and Library Environment Variables 3.1.1 CRAY_FTN_OPTIONS The CRAY_FTN_OPTIONS environment variable specifies additional options to attach to the command line. This option follows the options specified directly on the command line. File names cannot appear. These options are inserted at the rightmost portion of the command line before the input files and binary files are listed. This allows you to set the environment variable once and have the specified set of options used in all compilations. This is especially useful for adding options to compilations done with build tools. For example, assume that this environment variable was set as follows: setenv CRI_FTN_OPTIONS -G0 With the variable set, the following two command line specifications are equivalent: % ftn -c t.f % ftn -c -G0 t.f 3.1.2 CRAY_PE_TARGET The CRAY_PE_TARGET environment variable specifies the target_system for compilation. The command line option -h cpu=target_system takes precedence over the CRAY_PE_TARGET setting. The currently acceptable values for CRAY_PE_TARGET are x86-64, opteron, barcelona, shanghai, istanbul, mc8 , mc12 or interlagos. At this time the x86-64 and opteron options produce identical output. If you are creating executables for use on a barcelona or shanghai (quad-core), istanbul (six-core), Magny-Cours (8- or 12-core), or interlagos (16-core) system, you must also have the associated module, xtpe-barcelona, xtpe-shanghai, xtpe-istanbul, xtpe-mc8, xtpe-mc12, or xtpe-interlagos loaded when compiling and linking your code. If one of these modules is loaded, the default target_system changes to the corresponding cpu target. If the target_system is set during compilation of any source file, it must also be set to that same target during linking and loading. 3.1.3 FORMAT_TYPE_CHECKING The FORMAT_TYPE_CHECKING environment variable specifies various levels of conformance between the data type of each I/O list item and the formatted data edit descriptor. When set to RELAXED, the run time I/O library enforces limited conformance between the data type of each I/O list item and the formatted data edit descriptor. 88 S–3901–80Setting Environment Variables [3] When set to STRICT77, the run time I/O library enforces strict FORTRAN 77 conformance between the data type of each I/O list item and the formatted data edit descriptor. When set to STRICT90 or STRICT95, the run time I/O library enforces strict Fortran 90/95 conformance between the data type of each I/O list item and the formatted data edit descriptor. See the following tables: Table 16, Table 17, Table 18, and Table 19. 3.1.4 FORTRAN_MODULE_PATH Like the Cray Fortran compiler -p module_site command line option, this environment variable allows you to specify the files or the directory to search for the modules to use. The files can be archive files, build files (bld file), or binary files. The compiler appends the paths specified by the FORTRAN_MODULE_PATH environment variable to the path specified by the -p module_site command line option. Since the FORTRAN_MODULE_PATH environment variable can specify multiple files and directories, a colon separates each path as shown in the following example: % set FORTRAN_MODULE_PATH='path1 : path2 : path3' 3.1.5 LISTIO_PRECISION The LISTIO_PRECISION environment variable controls the number of digits of precision printed by list-directed output. The LISTIO_PRECISION environment variable can be set to FULL or PRECISION. • FULL prints full precision (default). • PRECISION prints x or x + 1 decimal digits, where x is value of the PRECISION intrinsic function for a given real value. This is a smaller number of digits, which usually ensures that the last decimal digit is accurate to within 1 unit. This number of digits is usually insufficient to assure that subsequent input will restore a bit-identical floating-point value. 3.1.6 NLSPATH The NLSPATH environment variable specifies the message system library catalog path. This environment variable affects compiler interactions with the message system. For more information about this environment variable, see the catopen(3) man page. S–3901–80 89Cray Fortran Reference Manual 3.1.7 NPROC The NPROC environment variable specifies the maximum number of processes to be run. Setting NPROC to a number other than 1 can speed up a compilation if machine resources permit. The effect of NPROC is seen at compilation time, not at execution time. NPROC requests a number of compilations to be done in parallel. It affects all the compilers and also the make command. For example, assume that NPROC is set as follows: setenv NPROC 2 The following command is entered: ftn -o t main.f sub.f In this example, the compilations from .f files to .o files for main.f and sub.f happen in parallel, and when both are done, the link step is performed. If NPROC is unset, or set to 1, main.f is compiled to main.o; sub.f is compiled to sub.o, and then the link step is performed. You can set NPROC to any value, but large values can overload the system. For debugging purposes, NPROC should be set to 1. By default, NPROC is 1. 3.1.8 TMPDIR The TMPDIR environment variable specifies the directory containing the compiler temporary files. The location of the directory is defined by your administrator and cannot be changed. 3.1.9 ZERO_WIDTH_PRECISION The ZERO_WIDTH_PRECISION environment variable controls the field width when field width w of Fw.d is zero on output. The ZERO_WIDTH_PRECISION environment variable can be set to PRECISION or HALF. • PRECISION specifies that full precision will be written. This is the default. • HALF specifies that half of the full precision will be written. 3.2 OpenMP Environment Variables For Cray-specific information about OpenMP environment variables, see Chapter 6, Using the OpenMP Fortran API on page 149. For documentation of standard OpenMP environment variables, see the OpenMP Application Program Interface Version 3.0 May 2008 standard (http://openmp.org/wp/openmp-specifications/). 90 S–3901–80Setting Environment Variables [3] 3.3 Run Time Environment Variables Run time environment variables allow you to adjust the following elements of your run time environment: • Stack and heap sizes, see the memory(7) man page for more information. • CRAY_ACC_DEBUG Write accelerator-related activity to stdout for debugging purposes. Valid output levels range from 0, which indicates no output, through 3, which indicates verbose. Default: 0 • CRAY_AUTO_APRUN_OPTIONS Default options for automatic aprun. See the aprun(1) man page. • CRAY_MALLOPT_OFF If set, then the system default mallopt parameters are used, instead of the compiler default parameters. For most programs, run time performance is improved by using the compiler defaults, but more memory may be used. • MALLOC_MMAP_MAX_ Specifies the maximum number of memory chunks to allocate with mmap. The compiler default value is 0. For most programs, run time performance is improved by using the compiler default, but more memory may be used. • MALLOC_TRIM_THRESHOLD_ Specifies the minimum size of the unused memory region at the top of the heap before the region is returned to the operating system. The compiler default value is 536870912 bytes. For most programs, run time performance is improved by using the compiler default, but more memory may be used. • PGAS_ERROR_FILE Specifies the location to which libpgas (the library which provides an interface to the internal system network) error messages are written. The default is stderr. If stdout is specified, errors will be written to standard output. 3.3.1 aprun Resource Limits The aprun command always forwards its own core and cpu resource limits (RLIMIT_CPU and RLIMIT_CORE) to the compute nodes where those limits are set for the application. If a -m value is specified, RLIMIT_RSS is also forwarded. S–3901–80 91Cray Fortran Reference Manual If the APRUN_XFER_LIMITS run time environment variable is set to a non-zero value, the following resource limits are also forwarded: • RLIMIT_FSIZE • RLIMIT_DATA • RLIMIT_STACK • RLIMIT_RSS • RLIMIT_NPROC • RLIMIT_NOFILE • RLIMIT_MEMLOCK • RLIMIT_AS • RLIMIT_LOCKS • RLIMIT_SIGPENDING • RLIMIT_MSGQUEUE • RLIMIT_NICE • RLIMIT_RTPRIO This forwarding is disabled by default. This forwarding of user resource limits can cause problems on systems where the login node's limits are more restrictive than the default compute node limits. 92 S–3901–80Using Cray Fortran Directives [4] Directives are lines inserted into source code that specify actions to be performed by the compiler. They are not Fortran statements. This chapter describes the Cray Fortran compiler directives. If you specify a directive while running on a system that does not support that particular directive, the compiler generates a message and continues with the compilation. Note: The Cray Fortran compiler also supports the OpenMP Fortran API directives. See Chapter 6, Using the OpenMP Fortran API on page 149 for more information. The compiler also supports the OpenACC API directives. See OpenACC Directives on page 166. Using Directives on page 97 describes how to use the directives and the effects they have on programs. Table 7 categorizes the Cray Fortran compiler directives according to purpose and directs you to the pages containing more details. Table 7. Directives Purpose and Name Description Vectorization: COPY_ASSUMED_SHAPE Copy arrays to temporary storage. For more information, see Copy Arrays to Temporary Storage: COPY_ASSUMED_SHAPE on page 102. HAND_TUNED Assert that the loop has been hand-tuned for maximum performance and restrict automatic compiler optimizations. For more information, see Limit Optimizations: HAND_TUNED on page 103. IVDEP Ignore loop vector-dependencies that a loop might have. For more information, see Ignore Vector Dependencies: IVDEP on page 103. NEXTSCALAR Disable loop vectorization. For more information, see Specify Scalar Processing: NEXTSCALAR on page 104. S–3901–80 93Cray Fortran Reference Manual Purpose and Name Description [NO]PATTERN Replace or do not replace recognized code patterns with optimized library routines. For more information, see Request Pattern Matching: [NO]PATTERN on page 104. PERMUTATION Declare that an integer array has no repeating values. For more information, see Declare an Array with No Repeated Values: PERMUTATION on page 105. [NO]PIPELINE Attempt to force or inhibit software-based vector pipelining. For more information, see Enable or Disable, Temporarily, Soft Vector-pipelining: [NO]PIPELINE on page 114. PREFERVECTOR Vectorize nested loops. For more information, see Designate Loop Nest for Vectorization: PREFERVECTOR on page 106. PROBABILITY Suggest the probability of a branch being executed. For more information, see Conditional Density: PROBABILITY on page 106. SAFE_ADDRESS Speculatively execute memory references within a loop. For more information, see Allow Speculative Execution of Memory References within Loops: SAFE_ADDRESS on page 107. SAFE_CONDITIONAL Speculatively execute memory references and arithmetic operations within a loop. For more information, see Allow Speculative Execution of Memory References and Arithmetic Operations: SAFE_CONDITIONAL on page 108. LOOP_INFO Provide loop count and cache allocation information to the optimizer to produce faster code sequences. This directive can be used to override CACHE or CACHE_NT. For more information, see Provide More Information for Loops: LOOP_INFO on page 109 and Autothreading for Loops: LOOP_INFO PREFER_[NO]THREAD on page 111. SHORTLOOP[128] The SHORTLOOP and SHORTLOOP128 directives are special cases of LOOP_INFO and are superseded by the general LOOP_INFO directive. For more information, see Designate Loops with Low Trip Counts: SHORTLOOP, SHORTLOOP128 on page 109. [NO]UNROLL Unroll or do not unroll loops to improve performance. For more information, see Unroll Loops: [NO]UNROLL on page 111. [NO]VECTOR Vectorize or do not vectorize loops and array statements. For more information, see Enable and Disable Vectorization: [NO]VECTOR on page 114. 94 S–3901–80Using Cray Fortran Directives [4] Purpose and Name Description Inlining and Cloning: [NO]CLONE, RESETCLONE Attempt cloning or do not attempt cloning at call sites, or reset cloning to the state specified on the command line. For more information, see Disable or Enable Cloning for a Block of Code: [NO]CLONE and RESETCLONE on page 115. CLONEALWAYS, CLONENEVER Always or never attempt to clone the specified functions. For more information, see Specify Cloning for a Procedure: CLONEALWAYS and CLONENEVER on page 115. [NO]INLINE, RESETINLINE Attempt to inline or do not attempt to inline call sites, or reset inlining to the state specified on the command line. For more information, see Disable or Enable Inlining for a Block of Code: [NO]INLINE and RESETINLINE on page 117. INLINEALWAYS,INLINENEVER Always or never inline the specified procedures. For more information, see Specify Inlining for a Procedure: INLINEALWAYS and INLINENEVER on page 117. [NO]MODINLINE Enable or disable inlineable templates for the designated procedures. For more information, see Create Inlinable Templates for Module Procedures: [NO]MODINLINE on page 118. Scalar optimization: [NO]INTERCHANGE Control whether or not to interchange the order of the two or more loops immediately following the directive. For more information, see Control Loop Interchange: [NO]INTERCHANGE on page 119. [NO]COLLAPSE Collapse or do not collapse the loop nest immediately following the directive. For more information, see Control Loop Collapse: [NO]COLLAPSE on page 121. NOSIDEEFFECTS Tell the compiler that the data in the registers will not change when calling the specified subprogram. For more information, see Determine Register Storage: NOSIDEEFFECTS on page 122. SUPPRESS Suppress scalar optimization of specified variables. For more information, see Suppress Scalar Optimization: SUPPRESS on page 123. Local use of compiler features: [NO]BOUNDS Check or do not check the bounds of array references. For more information, see Check Array Bounds: [NO]BOUNDS on page 124. FREE, FIXED Specify that the source uses a free or fixed format. For more information, see Specify Source Form: FREE and FIXED on page 126. S–3901–80 95Cray Fortran Reference Manual Purpose and Name Description Storage: BLOCKABLE Specify that it is legal to cache block subsequent loops. For more information, see Permit Cache Blocking: BLOCKABLE Directive on page 126. BLOCKINGSIZE, NOBLOCKING Assert that the loop following the directive is or is not involved in cache blocking. For more information, see Declare Cache Blocking: BLOCKINGSIZE and NOBLOCKING Directives on page 127. STACK Allocate variables on the stack. For more information, see Request Stack Storage: STACK on page 128. PGAS: DEFER_SYNC Defer the synchronization of PGAS data until the next fence instruction. For more information, see DEFER_SYNC Directive on page 129. Miscellaneous: [NO]AUTOTHREAD Turn autothreading on and off for the selected block of code. For more information, see Control Autothreading: [NO]AUTOTHREAD on page 130. CACHE Advisory directive to override automatic cache allocation and keep specified objects in cache. For more information, see Allocate Cache: CACHE on page 130. CACHE_NT Advisory directive to override automatic cache allocation and prevent specified objects from being cached. For more information, see Non-temporal Reads and Writes: CACHE_NT on page 131. CONCURRENT Convey user-known array dependencies to the compiler. For more information, see Specify Array Dependencies: CONCURRENT on page 132. [NO]FUSION Fine-tune the selection of the DO loops to be fused. For more information, see Fuse Loops: [NO]FUSION on page 132. ID Insert a character string into the file.o object file. For more information, see Create Identification String: ID on page 133. IGNORE_TKR Ignore the type, kind, and rank (TKR) of specified dummy arguments of a procedure interface. For more information, see Disregard Dummy Argument Type, Kind, and Rank: IGNORE_TKR on page 134. NAME Define a name that uses characters that are outside of the Fortran character set. See External Name Mapping: NAME on page 135. 96 S–3901–80Using Cray Fortran Directives [4] Purpose and Name Description NOFISSION Prevent the fission of specified loops. For more information, see Do Not Fission Loop: NOFISSION on page 133. PREPROCESS Allow an include file to be preprocessed when the compiler command line does not specify preprocessing. See Preprocess Include File: PREPROCESS on page 136. WEAK Define a procedure reference as weak. See Specify Weak Procedure Reference: WEAK on page 136. 4.1 Using Directives 4.1.1 Directive Lines A directive line begins with the characters CDIR$ or !DIR$, unless otherwise noted. These leading characters are sometimes referred to as a sentinel. OpenMP directives are indicated using the !OMP$ sentinel. OpenACC directives are indicated with the !ACC$ sentinel. How you specify directives depends on the source form you are using, as follows: • If using fixed source form, indicate a directive line by placing the characters CDIR$ or !DIR$ in columns 1 through 5. If the compiler encounters a nonblank character in column 6, the line is assumed to be a directive continuation line. Columns 7 and beyond can contain one or more directives. Characters in directives entered in columns beyond the default column width are ignored. • If using free source form, indicate a directive by the characters !DIR$, followed by a space, and then one or more directives. If the position following the !DIR$ contains a character other than a blank, tab, or newline character, the line is assumed to be a continuation line. The !DIR$ need not start in column 1, but it must be the first text on a line. In the following example, an asterisk (*) appears in column 6 to indicate that the second line is a continuation of the preceding line: !DIR$ NOSIDEEFFECTS !DIR$*ab The FIXED and FREE directives must appear alone on a directive line and cannot be continued. If you want to specify more than one directive on a line, separate each directive with a comma. Some directives require that you specify one or more arguments; when specifying a directive of this type, no other directive can appear on the line. S–3901–80 97Cray Fortran Reference Manual Spaces can precede, follow, or be embedded within a directive, regardless of source form. Code portability is maintained despite the use of directives. In the following example, the ! symbol in column 1 causes other compilers to treat the Cray Fortran compiler directive as a comment: A=10. !DIR$ NOVECTOR DO 10,I=1,10... Do not use source preprocessor (#) directives within multiline compiler directives (CDIR$ or !DIR$). 4.1.2 Range and Placement of Directives The range and placement of directives are as follows: • The FIXED and FREE directives can appear anywhere in your source code. All other directives must appear within a program unit. • These directives must reside in the declarative portion of a program unit and apply only to that program unit: – CACHE – CACHE_NT – COPY_ASSUMED_SHAPE – IGNORE_TKR – INLINEALWAYS, INLINENEVER, RESETINLINE – NAME – NOSIDEEFFECTS – STACK – PREPROCESS – WEAK • The following directives toggle a compiler feature on or off at the point at which the directive appears in the code. These directives are in effect until the opposite directive appears, until the directive is reset, or until the end of the program unit, at which time the command line settings become the default for the remainder of the compilation. – [NO]AUTOTHREAD – [NO]BOUNDS 98 S–3901–80Using Cray Fortran Directives [4] – [NO]CLONE, RESETCLONE – [NO]INLINE – [NO]INTERCHANGE – [NO]PATTERN – [NO]VECTOR • The SUPPRESS directive applies at the point at which it appears. • The ID directive does not apply to any particular range of code. It adds information to the file.o generated from the input program. • The following directives apply only to the next loop or block of code encountered lexically: – BLOCKABLE – BLOCKINGSIZE, NOBLOCKING – CONCURRENT – HAND_TUNED – [NO]INTERCHANGE – IVDEP – NEXTSCALAR – PERMUTATION – [NO]PIPELINE – PREFERVECTOR – PROBABILITY – SAFE_ADDRESS – SAFE_CONDITIONAL – SHORTLOOP[128] – LOOP_INFO – LOOP_INFO PREFER_[NO]THREAD – [NO]UNROLL • The MODINLINE and NOMODINLINE directives are in effect for the scope of the program unit in which they are specified, including all contained procedures. If one of these directives is specified in a contained procedure, the contained procedure's directive overrides the containing procedure's directive. S–3901–80 99Cray Fortran Reference Manual 4.1.3 Interaction of Directives with the -x Command Line Option The -x option on the ftn command accepts one or more directives as arguments. When your input is compiled, the compiler ignores directives named as arguments to the -x option. If you specify -x all, all directives are ignored. If you specify -x dir, all directives preceded by !DIR$ or CDIR$ are ignored. For more information about the -x option, see -x dirlist on page 83. 4.1.4 Command Line Options and Directives Some features activated by directives can also be specified on the ftn command line. A directive applies to parts of programs in which it appears, but a command line option applies to the entire compilation. Vectorization, scalar optimization, and tasking can be controlled through both command line options and directives. If a compiler optimization feature is disabled by default or is disabled by an argument to the -O option to the ftn command, the associated !prefix$ directives are ignored. The following list shows Cray Fortran compiler optimization features, related command line options, and related directives: • Specifying the -O 0 option on the command line disables all optimization. All scalar optimization and vectorization directives are ignored. • Specifying the -O ipa0 option on the command line disables inlining and causes the compiler to ignore all inlining directives. • Specifying the -O scalar0 option disables scalar optimization and causes the compiler to ignore all scalar optimization and all vectorization directives. • Specifying the -O noomp option disables OpenMP and causes the compiler to ignore OpenMP directives. • Specifying the -O thread0 option disables both OpenMP and autothreading and causes the compiler to ignore OpenMP and autothreading directives. • Specifying the -O vector0 option causes the compiler to ignore all vectorization directives. Specifying the NOVECTOR directive in a program unit causes the compiler to ignore subsequent directives in that program unit that may specify vectorization. The following sections describe directive syntax and the effects of directives on Cray Fortran compiler programs. 100 S–3901–80Using Cray Fortran Directives [4] 4.2 Vectorization Directives This section describes the following directives used to control vectorization: • COPY_ASSUMED_SHAPE • HAND_TUNED • IVDEP • NEXTSCALAR • [NO]PATTERN • PERMUTATION • PREFERVECTOR • PROBABILITY • SAFE_ADDRESS • SAFE_CONDITIONAL • SHORTLOOP[128] • LOOP_INFO • LOOP_INFO PREFER_[NO]THREAD • [NO]UNROLL • [NO]VECTOR • [NO]PIPELINE The -O 0, -O scalar0, -O task0, and -O vector0 options on the ftn command override these directives. S–3901–80 101Cray Fortran Reference Manual 4.2.1 Copy Arrays to Temporary Storage: COPY_ASSUMED_SHAPE The COPY_ASSUMED_SHAPE directive copies assumed-shape dummy array arguments into contiguous local temporary storage upon entry to the procedure in which the directive appears. During execution, it is the temporary storage that is used when the assumed-shape dummy array argument is referenced or defined. The format of this directive is as follows: !DIR$ COPY_ASSUMED_SHAPE [ array [, array ] ...] array The name of an array to be copied to temporary storage. If no array names are specified, all assumed-shape dummy arrays are copied to temporary contiguous storage upon entry to the procedure. When the procedure is exited, the arrays in temporary storage are copied back to the dummy argument arrays. If one or more arrays are specified, only those arrays specified are copied. The arrays specified must not have the TARGET attribute. All arrays specified, or all assumed-shape dummy arrays (if specified without array arguments), on a single COPY_ASSUMED_SHAPE directive must be shape conformant with each other. Incorrect code may be generated if the arrays are not. You can use the -R c command line option to verify whether the arrays are shape conformant. The COPY_ASSUMED_SHAPE directive applies only to the program unit in which it appears. Assumed-shape dummy array arguments cannot be assumed to be stored in contiguous storage. In the case of multidimensional arrays, the elements cannot be assumed to be stored with uniform stride between each element of the array. These conditions can arise, for example, when an actual array argument associated with an assumed-shape dummy array is a non-unit strided array slice or section. If the compiler cannot determine whether an assumed-shape dummy array is stored contiguously or with a uniform stride between each element, some optimizations are inhibited in order to ensure that correct code is generated. If an assumed-shape dummy array is passed to a procedure and becomes associated with an explicit-shape dummy array argument, additional copy-in and copy-out operations may occur at the call site. For multidimensional assumed-shape arrays, some classes of loop optimizations cannot be performed when an assumed-shape dummy array is referenced or defined in a loop or an array assignment statement. The lost optimizations and the additional copy operations performed can significantly reduce the performance of a procedure that uses assumed-shape dummy arrays when compared to an equivalent procedure that uses explicit-shape array dummy arguments. 102 S–3901–80Using Cray Fortran Directives [4] The COPY_ASSUMED_SHAPE directive causes a single copy to occur upon entry and again on exit. The compiler generates a test at run time to determine whether the array is contiguous. If the array is contiguous, the array is not copied. This directive allows the compiler to perform all the optimizations it would otherwise perform if explicit-shape dummy arrays were used. If there is sufficient work in the procedure using assumed-shape dummy arrays, the performance improvements gained by the compiler outweigh the cost of the copy operations upon entry and exit of the procedure. 4.2.2 Limit Optimizations: HAND_TUNED This directive asserts that the code in the loop that follows the directive has been arranged by hand for maximum performance and the compiler should restrict some of the more aggressive automatic expression rewrites. The compiler will still fully optimize and vectorize the loop within the constraints of the directive. The syntax of this directive is as follows: !DIR$ HAND_TUNED Warning: Exercise caution when using this directive and evaluate code performance before and after using it. The use of this directive may severely impair performance. 4.2.3 Ignore Vector Dependencies: IVDEP When the IVDEP directive appears before a loop, the compiler ignores vector dependencies, including explicit dependencies, in any attempt to vectorize the loop. IVDEP applies to the first DO loop or DO WHILE loop that follows the directive. The directive applies to only the first loop that appears after the directive within the same program unit. For array operations, Fortran requires that the complete right-hand side (RHS) expression be evaluated before the assignment to the array or array section on the left-hand side (LHS). If possible dependencies exist between the RHS expression and the LHS assignment target, the compiler creates temporary storage to hold the RHS expression result. If an IVDEP directive appears before an array syntax statement, the compiler ignores potential dependencies and suppresses the creation and use of array temporaries for that statement. Using array syntax statements allows you to reference referencing arrays in a compact manner. Array syntax allows you to use either the array name, or the array name with a section subscript, to specify actions on all the elements of an array, or array section, without using DO loops. S–3901–80 103Cray Fortran Reference Manual Whether or not IVDEP is used, conditions other than vector dependencies can inhibit vectorization. The format of this directive is as follows: !DIR$ IVDEP [ SAFEVL=vlen | INFINITEVL] vlen Specifies a vector length in which no dependency will occur. vlen must be an integer between 1 and 1024 inclusive. INFINITEVL Specifies an infinite safe vector length. That is, no dependency will occur at any vector length. If no vector length is specified, the vector length used is infinity. If a loop with an IVDEP directive is enclosed within another loop with an IVDEP directive, the IVDEP directive on the outer loop is ignored. When the Cray Fortran compiler vectorizes a loop, it may reorder the statements in the source code to remove vector dependencies. When IVDEP is specified, the statements in the loop or array syntax statement are assumed to contain no dependencies as written, and the Cray Fortran compiler does not reorder loop statements. 4.2.4 Specify Scalar Processing: NEXTSCALAR The NEXTSCALAR directive disables vectorization for the first DO loop or DO WHILE loop that follows the directive. The directive applies to only one loop, the first loop that appears after the directive within the same program unit. NEXTSCALAR is ignored if vectorization has been disabled. The format of this directive is as follows: !DIR$ NEXTSCALAR If the NEXTSCALAR directive appears prior to any array syntax statement, it disables vectorization for the array syntax statement. 4.2.5 Request Pattern Matching: [NO]PATTERN By default, the compiler detects coding patterns in source code sequences and replaces these sequences with calls to optimized library routines. In most cases, this replacement improves performance. There are cases, however, in which this substitution degrades performance. This can occur, for example, in loops with very low trip counts. In such a case, you can use the NOPATTERN directive to disable pattern matching and cause the compiler to generate inline code. The formats of these directives are as follows: !DIR$ PATTERN !DIR$ NOPATTERN 104 S–3901–80Using Cray Fortran Directives [4] When !DIR$ NOPATTERN has been encountered, pattern matching is suspended for the remainder of the program unit or until a !DIR$ PATTERN directive is encountered. When the -O nopattern command line option (default) is in effect, the PATTERN and NOPATTERN compiler directives are ignored. For more information about -O nopattern, see -O [no]pattern on page 65. The PATTERN and NOPATTERN directives should be specified before the beginning of a pattern. Example: By default, the compiler would detect that the following loop is a matrix multiply and replace it with a call to a matrix multiply library routine. By preceding the loop with a !DIR$ NOPATTERN directive, however, pattern matching is inhibited and no replacement is done. !DIR$ NOPATTERN DO k= 1,n DO i= 1,n DO j= 1,m A(i,j) = A(i,j) + B(i,k) * C(k,j) END DO END DO END DO 4.2.6 Declare an Array with No Repeated Values: PERMUTATION The !DIR$ PERMUTATION directive declares that an integer array has no repeated values. This directive is useful when the integer array is used as a subscript for another array (vector-valued subscript). When this directive precedes a loop to be vectorized, it may cause more efficient code to be generated. The format for this directive is as follows: !DIR$ PERMUTATION (ia [, ia ] ...) ia Integer array that has no repeated values for the entire routine. When an array with a vector-valued subscript appears on the left side of the equal sign in a loop, many-to-one assignment is possible. Many-to-one assignment occurs if any repeated elements exist in the subscripting array. If it is known that the integer array is used merely to permute the elements of the subscripted array, it can often be determined that many-to-one assignment does not exist with that array reference. Sometimes a vector-valued subscript is used as a means of indirect addressing because the elements of interest in an array are sparsely distributed; in this case, an integer array is used to select only the desired elements, and no repeated elements exist in the integer array, as in the following example: !DIR$ PERMUTATION(IPNT) ! IPNT has no repeated values ... DO I = 1, N A(IPNT(I)) = B(I) + C(I) END DO S–3901–80 105Cray Fortran Reference Manual 4.2.7 Designate Loop Nest for Vectorization: PREFERVECTOR For cases in which the compiler could vectorize more than one loop, the PREFERVECTOR directive indicates that the loop following the directive should be vectorized. This directive can be used if there is more than one loop in the nest that could be vectorized. The format of this directive is as follows: !DIR$ PREFERVECTOR In the following example, both loops can be vectorized, but the compiler generates vector code for the outer DO I loop. Note that the DO I loop is vectorized even though the inner DO J loop was specified with an IVDEP directive: !DIR$ PREFERVECTOR DO I = 1, N !DIR$ IVDEP DO J = 1, M A(I) = A(I) + B(J,I) END DO END DO 4.2.8 Conditional Density: PROBABILITY This directive is used to guide inlining decisions, branch elimination optimizations, branch hint marking, and the choice of the optimal algorithmic approach to the vectorization of conditional code. The information specified by this directive is used by interprocedural analysis and the optimizer to produce faster code sequences. This directive can appear anywhere executable code is legal, and the syntax of this directive takes one of three forms. !DIR$ PROBABILITY const !DIR$ PROBABILITY_ALMOST_ALWAYS !DIR$ PROBABILITY_ALMOST_NEVER Where const is an expression between 0.0 (never) and 1.0 (always) that evaluates to a floating point constant at compilation time. The specified probability is a hint, rather than a statement of fact. The directive applies to the block of code where it appears. It is important to realize that the directive should not be applied to a conditional test directly; rather, it should be used to indicate the relative probability of a THEN or ELSE branch being executed. For example: IF ( A(I) > B(I) ) THEN !DIR$ PROBABILITY 0.3 A(I) = B(I) ENDIF 106 S–3901–80Using Cray Fortran Directives [4] This example states that the probability of entering the block of code with the assignment statement is 0.3, or 30%. In turn, this means that a(i) is expected to be greater than b(i) 30% of the time as well. For vector IF code, a probability of very low (< 0.1) or probability_almost_never will cause the compiler to use the vector gather/scatter methods used for sparse IF vector code instead of the vector merge methods used for denser IF code. For example: DO I = 1,N IF ( A(I) > 0.0 ) THEN !DIR$ PROBABILITY_ALMOST_NEVER B(I) = B(I)/A(I) + A(I)/B(I) ! EVALUATE USING SPARSE METHODS ENDIF ENDDO Note that the PROBABILITY directive appears within the conditional, rather than before the condition. This removes some of the ambiguity of tying the directive directly to the conditional test. 4.2.9 Allow Speculative Execution of Memory References within Loops: SAFE_ADDRESS Deferred Implementation: Support for the SAFE_ADDRESS directive has been deferred to a later release. The SAFE_ADDRESS directive allows you to tell the compiler that it is safe to speculatively execute memory references within all conditional branches of a loop. In other words, you know that these memory references can be safely executed in each iteration of the loop. For most code, the SAFE_ADDRESS directive can improve performance significantly by preloading vector expressions. However, most loops do not require this directive to have preloading performed. The directive is only required when the safety of the operation cannot be determined or index expressions are very complicated. The SAFE_ADDRESS directive is an advisory directive. That is, the compiler may override the directive if it determines the directive is not beneficial. If you do not use the directive on a loop and the compiler determines that it would benefit from the directive, it issues a message indicating such. The message is similar to this: DO I = 1,N FTN-6375 FTN_DRIVER.EXE: VECTOR X7, FILE = 10928.F, LINE = 110 A LOOP STARTING AT LINE 110 WOULD BENEFIT FROM "!DIR$ SAFE_ADDRESS". If you use the directive on a loop and the compiler determines that it does not benefit from the directive, it issues a message that states the directive is superfluous and can be removed. To see the messages you must use the -O msgs option. S–3901–80 107Cray Fortran Reference Manual Incorrect use of the directive can result in segmentation faults, bus errors, or excessive page faulting. However, it should not result in incorrect answers. Incorrect usage can result in very severe performance degradations or program aborts. This is the syntax of the SAFE_ADDRESS directive: !DIR$ SAFE_ADDRESS In the example below, the compiler will not preload vector expressions, because the value of j is unknown. However, if you know that references to b(i,j) are safe to evaluate for all iterations of the loop, regardless of the condition, we can use the SAFE_ADDRESS directive for this loop as shown below: SUBROUTINE X3( A, B, N, M, J ) REAL A(N), B(N,M) !DIR$ SAFE_ADDRESS DO I = 1,64 ! VECTORIZED LOOP IF ( A(I).NE.0.0 ) THEN B(I,J) = 0.0 ! VALUE OF 'J' IS UNKNOWN ENDIF ENDDO END With the directive, the compiler can load b(i,j) with a full vector mask, merge 0.0 where the condition is true, and store the resulting vector using a full mask. 4.2.10 Allow Speculative Execution of Memory References and Arithmetic Operations: SAFE_CONDITIONAL The SAFE_CONDITIONAL directive expands upon the SAFE_ADDRESS directive. It implies SAFE_ADDRESS and further specifies that arithmetic operations are safe, as well as memory operations. This directive applies to scalar and vector loop nests. It can improve performance by allowing the hoisting of invariant expressions from conditional code and allowing prefetching of memory references. The SAFE_CONDITIONAL directive is an advisory directive. The compiler may override the directive if it determines that the directive is not beneficial. ! Caution: Incorrect use of the directive may result in segmentation faults, bus errors, excessive page faulting, or arithmetic aborts. However, it should not result in incorrect answers. Incorrect usage may result in severe performance degradation or program aborts. The syntax of this directive is as follows: !DIR$ SAFE_CONDITIONAL 108 S–3901–80Using Cray Fortran Directives [4] In the example below, the compiler cannot precompute the invariant expression s1*s2 because these values are unknown and may cause an arithmetic trap if executed unconditionally. However, if you know that the condition is true at least once, then it is safe to use the SAFE_CONDITIONAL directive and execute s1*s2 speculatively. SUBROUTINE SAFE_COND( A, N, S1, S2 ) REAL A(N), S1, S2 !DIR$ SAFE_CONDITIONAL DO I = 1,N IF ( A(I) /= 0.0 ) THEN A(I) = A(I) + S1*S2 ENDIF ENDDO END With the directive, the compiler evaluates s1*s2 outside of the loop, rather than under control of the conditional code. In addition, all control flow is removed from the body of the vector loop as s1*s2 no longer poses a safety risk. 4.2.11 Designate Loops with Low Trip Counts: SHORTLOOP, SHORTLOOP128 The SHORTLOOP and SHORTLOOP128 directives are special cases of LOOP_INFO that have been superseded by the generalized LOOP_INFO directive. The SHORTLOOP and SHORTLOOP128 directives are equivalent, respectively, to: ! DIR$ LOOP_INFO MIN_TRIPS(1) MAX_TRIPS(64) ! DIR$ LOOP_INFO MIN_TRIPS(1) MAX_TRIPS(128) The meaning of SHORTLOOP and SHORTLOOP128 can be modified by using the -eL option. If enabled, this option changes the lower bound to allow zero-trip loops. For more information about the -eL option, see -d disable and -e enable on page 31. 4.2.12 Provide More Information for Loops: LOOP_INFO The LOOP_INFO directive causes additional information to be reported about the behavior of a loop, including run time trip count and hints on cache allocation strategy. With respect to the trip count information, the LOOP_INFO directive is similar to the SHORTLOOP or SHORTLOOP128 directive but provides more information to the optimizer and can produce faster code sequences. LOOP_INFO is used immediately before a DO or WHILE loop with a low or known trip count, to indicate the minimum, maximum, or estimated trip count. The compiler will diagnose misuse at compile time (when able) or under option -Rd at run time. S–3901–80 109Cray Fortran Reference Manual For cache allocation hints, use the LOOP_INFO directive to override default settings, CACHE, or CACHE_NT directives, or to override automatic cache management decisions. The cache hints are local and apply only to the specified loop nest. The syntax of the LOOP_INFO directive is as follows: !DIR$ LOOP_INFO [min_trips(c)] [est_trips(c)] [max_trips(c)] [cache( symbol [, symbol ...] )] [cache_nt( symbol [, symbol ...] )] [prefetch ][noprefetch ] Where: c An expression that evaluates to an integer constant at compilation time. min_trips Specifies the guaranteed minimum number of trips. est_trips Specifies the estimated or average number of trips. max_trips Specifies the guaranteed maximum number of trips. cache Specifies that symbol is to be allocated in cache. This is the default if no hint is specified and the cache_nt directive is not specified. cache_nt Specifies that symbol is to use non-temporal reads and writes. symbol The base name of the object that should (cache) or should not (cache_nt) be placed into cache. This can be the base name of any object such as an array or scalar structure without member references. If you specify a pointer in the list, only the references, not the pointer itself, are subject to the cache or cache_nt instruction. prefetch Specifies a preference that prefetches be performed for the following loop. noprefetch Specifies a preference that no prefetches be performed for the following loop. The prefetch option is deferred. 110 S–3901–80Using Cray Fortran Directives [4] (Deferred implementation) The prefetch clause instructs the compiler to preload scalar data into the first-level cache to improve the frequency of cache hits and to lower latency. They are generated in situations where the compiler expects them to improve performance. Strategic use of prefetch instructions can hide latency for scalar loads that feed vector instructions or scalar loads in purely scalar loops. Prefetch instructions are generated at default and higher levels of optimization. Thus, they are turned off at -O0 or -O1. Prefetch can be turned off at the loop level via the following directive: !DIR$ LOOP_INFO NOPREFETCH DO I = 1, N Note: The noprefetch option is not deferred and can be used to prevent compiler-generated prefetching for specified loops. 4.2.13 Autothreading for Loops: LOOP_INFO PREFER_[NO]THREAD The PREFER_THREAD and PREFER_NOTHREAD directives are special cases of the LOOP_INFO advisory directive. Use these directives to indicate a preference for turning threading on or off for the subsequent loop. Use the LOOP_INFO PREFER_THREAD directive to indicate your preference that the loop following the directive be threaded. Use the LOOP_INFO PREFER_NOTHREAD directive to indicate that the loop should not be threaded. The format of these directives is: !DIR$ LOOP_INFO PREFER_THREAD DO I = 1, N !DIR$ LOOP_INFO PREFER_NOTHREAD DO J = 1, N 4.2.14 Unroll Loops: [NO]UNROLL Loop unrolling can improve program performance by revealing cross-iteration memory optimization opportunities such as read-after-write and read-after-read. The effects of loop unrolling also include: • Improved loop scheduling by increasing basic block size • Reduced loop overhead • Improved chances for cache hits S–3901–80 111Cray Fortran Reference Manual The formats of these directives are as follows: !DIR$ UNROLL [ n ] !DIR$ NOUNROLL n Specifies the total number of loop body copies to be generated. n is an integer value from 0 through 1024. If you specify a value for n, the compiler unrolls the loop by that amount. If you do not specify n, the compiler determines if it is appropriate to unroll the loop, and if so, the unroll amount. The subsequent DO loop is not unrolled if you specify UNROLL0, UNROLL1, or NOUNROLL. These directives are equivalent. The UNROLL directive should be placed immediately before the DO statement of the loop that should be unrolled. Note: The compiler cannot always safely unroll non-innermost loops due to data dependencies. In these cases, the directive is ignored (see Example 1). The UNROLL directive can be used only on loops whose iteration counts can be calculated before entering the loop. If UNROLL is specified on a loop that is not the innermost loop in a loop nest, the inner loops must be nested perfectly. That is, at each nest level, there is only one loop and only the innermost loop contains work. The NOUNROLL directive inhibits loop unrolling. Note: Loop unrolling occurs for both vector and scalar loops automatically. It is usually not necessary to use the unrolling directives. The UNROLL directive should be limited to non-inner loops such as Example 1 in which unroll-and-jam conditions can occur. Such loop unrolling is associated with compiler message 6005. Using the UNROLL directive for inner loops may be detrimental to performance and is not recommended. Typically, loop unrolling occurs in both vector and scalar loops without need of the UNROLL directive. Example 1. Unrolling outer loops Assume that the outer loop of the following nest will be unrolled by two: !DIR$ UNROLL 2 DO I = 1, 10 DO J = 1,100 A(J,I) = B(J,I) + 1 END DO END DO 112 S–3901–80Using Cray Fortran Directives [4] With outer loop unrolling, the compiler produces the following nest, in which the two bodies of the inner loop are adjacent to each other: DO I = 1, 10, 2 DO J = 1,100 A(J,I) = B(J,I) + 1 END DO DO J = 1,100 A(J,I+1) = B(J,I+1) + 1 END DO END DO The compiler jams, or fuses, the inner two loop bodies together, producing the following nest: DO I = 1, 10, 2 DO J = 1,100 A(J,I) = B(J,I) + 1 A(J,I+1) = B(J,I+1) + 1 END DO END DO Example 2. Illegal unrolling of outer loops Outer loop unrolling is not always legal because the transformation can change the semantics of the original program. For example, unrolling the following loop nest on the outer loop would change the program semantics because of the dependency between A(...,I) and A(...,I+1): !DIR$ UNROLL 2 DO I = 1, 10 DO J = 1,100 A(J,I) = A(J-1,I+1) + 1 END DO END DO Example 3. Unrolling nearest neighbor pattern The following example shows unrolling with nearest neighbor pattern. This allows register reuse and reduces memory references from 2 per trip to 1.5 per trip. !DIR$ UNROLL 2 DO J = 1,N DO I = 1,N ! VECTORIZE A(I,J) = B(I,J) + B(I,J+1) ENDDO ENDDO The preceding code fragment is converted to the following code: DO J = 1,N,2 ! UNROLLED FOR REUSE OF B(I,J+1) DO I = 1,N ! VECTORIZED A(I,J) = B(I,J) + B(I,J+1) A(I,J+1) = B(I,J+1) + B(I,J+2) END DO END DO S–3901–80 113Cray Fortran Reference Manual 4.2.15 Enable and Disable Vectorization: [NO]VECTOR The NOVECTOR directive suppresses compiler attempts to vectorize loops and array syntax statements. NOVECTOR takes effect at the beginning of the next loop and applies to the rest of the program unit unless it is superseded by a VECTOR directive. These directives are ignored if vectorization or scalar optimization have been disabled. The formats of these directives are as follows: !DIR$ VECTOR !DIR$ NOVECTOR When !DIR$ NOVECTOR has been used within the same program unit, !DIR$ VECTOR causes the compiler to resume its attempts to vectorize loops and array syntax statements. After a VECTOR directive is specified, automatic vectorization is enabled for all loop nests. The VECTOR directive affects subsequent loops. The NOVECTOR directive also affects subsequent loops, but if it is specified within the body of a loop, it affects the loop in which it is contained and all subsequent loops. 4.2.16 Enable or Disable, Temporarily, Soft Vector-pipelining: [NO]PIPELINE Software-based vector pipelining (software vector pipelining) provides additional optimization beyond the normal hardware-based vector pipelining. In software vector pipelining, the compiler analyzes all vector loops and will automatically attempt to pipeline a loop if doing so can be expected to produce a significant performance gain. This optimization also performs any necessary loop unrolling. In some cases the compiler will either not pipeline a loop that could be pipelined, or pipeline a loop without producing performance gains. In these cases, you can use the PIPELINE or NOPIPELINE directives to advise the compiler to pipeline or not pipeline the loop immediately following the directive. The format of the pipelining directives is as follows: !DIR$ PIPELINE !DIR$ NOPIPELINE Software vector pipelining is valid only for the innermost loop of a loop nest. The PIPELINE and NOPIPELINE directives are advisory only. While you can use the NOPIPELINE directive to inhibit automatic pipelining, and you can use the PIPELINE directive to attempt to override the compiler's decision not to pipeline a loop, you cannot force the compiler to pipeline a loop that cannot be pipelined. Vector loops that have been pipelined generate compile-time messages to that effect, if optimization messaging is enabled (-O msgs). 114 S–3901–80Using Cray Fortran Directives [4] 4.3 Inlining and Cloning Directives These directives allow you to specify whether the compiler should attempt to inline or clone certain subprograms or procedures: • [NO]CLONE, RESETCLONE • CLONEALWAYS, CLONENEVER • [NO]INLINE, RESETINLINE • INLINEALWAYS, INLINENEVER • [NO]MODINLINE These directives work in conjunction with the following command line options: • -O ipan and -O ipafrom, described in -O ipan on page 59 and -O ipafrom=source[:source]... on page 62. • -O modinline and -O nomodinline, described in -O [no]modinline on page 63. 4.3.1 Disable or Enable Cloning for a Block of Code: [NO]CLONE and RESETCLONE The CLONE and NOCLONE directives control whether cloning is attempted over a range of code. If !DIR$ CLONE is in effect, cloning is attempted at call sites. If !DIR$ NOCLONE is in effect, cloning is not attempted at call sites. The RESETCLONE resets cloning to the state specified on the compiler command line. The formats of these directives are as follows: !DIR$ CLONE !DIR$ NOCLONE !DIR$ RESETCLONE One of these directives remains in effect until the opposite directive is encountered, until the end of the program unit, or until the RESETCLONE directive is encountered. These directives are recognized when cloning is enabled on the command line (-O ipa4). 4.3.2 Specify Cloning for a Procedure: CLONEALWAYS and CLONENEVER The CLONEALWAYS directive forces attempted cloning of specified procedures. The CLONENEVER directive suppresses cloning of specified procedures. The formats of these directives are as follows: !DIR$ CLONEALWAYS name [, name ] ... !DIR$ CLONENEVER name [, name ] ... where name is the name of a procedure. S–3901–80 115Cray Fortran Reference Manual The following rules determine the scope of these directives: • A !DIR$ CLONENEVER name directive suppresses cloning for name. If !DIR$ CLONENEVER b appears in routine b, no call to b, within the entire program, is cloned. If !DIR$ CLONENEVER b appears in a routine other than b, no call to b from within that routine is cloned. • A !DIR$ CLONEALWAYS directive specifies that inlining should always be attempted for name. That is, if !DIR$ CLONEALWAYS c appears in routine c, cloning is attempted for all calls to c, throughout the entire program. If !DIR$ CLONEALWAYS c appears in a routine other than c, cloning is attempted for all calls to c from within that routine. An error message is issued if CLONENEVER and CLONEALWAYS are specified for the same procedure in the same program unit. Example: The following file is compiled with -O ipa4: SUBROUTINE S() !DIR$ CLONEALWAYS S ! THIS SAYS ATTEMPT ! CLONING OF S AT ALL CALLS. ... END SUBROUTINE SUBROUTINE T !DIR$ CLONENEVER S ! DO NOT CLONE ANY CALLS TO S ! IN SUBROUTINE T. CALL S() ... END SUBROUTINE SUBROUTINE V !DIR$ NOCLONE ! HAS HIGHER PRECEDENCE THAN CLONEALWAYS. CALL S() ! DO NOT CLONE THIS CALL TO S. !DIR$ CLONE CALL S() ! ATTEMPT CLONING OF THIS CALL TO S. ... END SUBROUTINE SUBROUTINE W CALL S() ! ATTEMPT CLONING OF THIS CALL TO S. ... END SUBROUTINE 116 S–3901–80Using Cray Fortran Directives [4] 4.3.3 Disable or Enable Inlining for a Block of Code: [NO]INLINE and RESETINLINE The INLINE, NOINLINE, and RESETINLINE directives control whether inlining is attempted over a range of code. If !DIR$ INLINE is in effect, inlining is attempted at call sites. If !DIR$ NOINLINE is in effect, inlining is not attempted at call sites. After either directive is used, !DIR$ RESETINLINE can be used to return inlining to the state specified on the compiler command line. These are the formats of these directives: !DIR$ INLINE !DIR$ NOINLINE !DIR$ RESETINLINE The INLINE and NOINLINE directives remain in effect until the opposite directive is encountered, until the RESETINLINE directive is encountered, or until the end of the program unit. These directives are ignored if -O ipa0 is in effect. 4.3.4 Specify Inlining for a Procedure: INLINEALWAYS and INLINENEVER The INLINEALWAYS directive forces attempted inlining of specified procedures. The INLINENEVER directive suppresses inlining of specified procedures. The formats of these directives are as follows: !DIR$ INLINEALWAYS name [, name ] ... !DIR$ INLINENEVER name [, name ] ... where name is the name of a procedure. The following rules determine the scope of these directives: • A !DIR$ INLINENEVER directive suppresses inlining for name. That is, if !DIR$ INLINENEVER b appears in routine b, no call to b, within the entire program, is inlined. If !DIR$ INLINENEVER b appears in a routine other than b, no call to b from within that routine is inlined. • A !DIR$ INLINEALWAYS directive specifies that inlining should always be attempted for name. That is, if !DIR$ INLINEALWAYS c appears in routine c, inlining is attempted for all calls to c, throughout the entire program. If !DIR$ INLINEALWAYS c appears in a routine other than c, inlining is attempted for all calls to c from within that routine. An error message is issued if INLINENEVER and INLINEALWAYS are specified for the same procedure in the same program unit. S–3901–80 117Cray Fortran Reference Manual Example: The following file is compiled with -O ipa1: SUBROUTINE S() !DIR$ INLINEALWAYS S ! THIS SAYS ATTEMPT ! INLINING OF S AT ALL CALLS. ... END SUBROUTINE SUBROUTINE T !DIR$ INLINENEVER S ! DO NOT INLINE ANY CALLS TO S ! IN SUBROUTINE T. CALL S() ... END SUBROUTINE SUBROUTINE V !DIR$ NOINLINE ! HAS HIGHER PRECEDENCE THAN INLINEALWAYS. CALL S() ! DO NOT INLINE THIS CALL TO S. !DIR$ INLINE CALL S() ! ATTEMPT INLINING OF THIS CALL TO S. ... END SUBROUTINE SUBROUTINE W CALL S() ! ATTEMPT INLINING OF THIS CALL TO S. ... END SUBROUTINE 4.3.5 Create Inlinable Templates for Module Procedures: [NO]MODINLINE The MODINLINE and NOMODINLINE directives enable and disable the creation of inlinable templates for specific module procedures. The formats of these directives are as follows: !DIR$ MODINLINE !DIR$ NOMODINLINE Note: The MODINLINE and NOMODINLINE directives are ignored if -O nomodinline is specified on the ftn command line. These directives are in effect for the scope of the program unit in which they are specified, including all contained procedures. If one of these directives is specified in a contained procedure, the contained procedure's directive overrides the containing procedure's directive. The compiler generates a message if these directives are specified outside of a module and ignores the directive. 118 S–3901–80Using Cray Fortran Directives [4] Example: MODULE BEGIN ... CONTAINS SUBROUTINE S() ! Uses SUBROUTINE S's !DIR$ !DIR$ NOMODINLINE ... CONTAINS SUBROUTINE INSIDE_S() ! Uses SUBROUTINE S's !DIR$ ... END SUBROUTINE INSIDE_S END SUBROUTINE S SUBROUTINE T() ! Uses MODULE BEGIN's !DIR$ ... CONTAINS SUBROUTINE INSIDE_T() ! Uses MODULE BEGIN's !DIR$ ... END SUBROUTINE INSIDE_T SUBROUTINE MORE_INSIDE_T !DIR$ NOMODINLINE ... END SUBROUTINE MORE_INSIDE_T END SUBROUTINE T END MODULE BEGIN In the preceding example, the subroutines are affected as follows: • Inlining templates are not produced for S, INSIDE_S, or MORE_INSIDE_T. • Inlining templates are produced for T and INSIDE_T. 4.4 Scalar Optimization Directives The following directives control aspects of scalar optimization: • [NO]INTERCHANGE • [NO]COLLAPSE • NOSIDEEFFECTS • SUPPRESS The following subsections describe these directives. 4.4.1 Control Loop Interchange: [NO]INTERCHANGE The loop interchange control directives specify whether or not the order of the following two or more loops should be interchanged. These directives apply to the loops that they immediately precede. S–3901–80 119Cray Fortran Reference Manual The formats of these directives are as follows: !DIR$ INTERCHANGE (do_variable1,do_variable2 [,do_variable3]...) !DIR$ NOINTERCHANGE do_variable Specifies two or more do_variable names. The do_variable names can be specified in any order, and the compiler reorders the loops. The loops must be perfectly nested. If the loops are not perfectly nested, you may receive unexpected results. The compiler reorders the loops such that the loop with do_variable1 is outermost, then loop do_variable2, then loop do_variable3. The NOINTERCHANGE directive inhibits loop interchange on the loop that immediately follows the directive. Example: The following code has an INTERCHANGE directive: !DIR$ INTERCHANGE (I,J,K) DO K = 1,NSIZE1 DO J = 1,NSIZE1 DO I = 1,NSIZE1 X(I,J) = X(I,J) + Y(I,K) * Z(K,J) ENDDO ENDDO ENDDO The following code results when the INTERCHANGE directive is used on the preceding code: DO I = 1,NSIZE1 DO J = 1,NSIZE1 DO K = 1,NSIZE1 X(I,J) = X(I,J) + Y(I,K) * Z(K,J) ENDDO ENDDO ENDDO 120 S–3901–80Using Cray Fortran Directives [4] 4.4.2 Control Loop Collapse: [NO]COLLAPSE The loop collapse directives control collapse of the immediately following loop nest or elemental array syntax statement. When the COLLAPSE directive is applied to a loop nest, the participating loops must be listed in order of increasing access stride. NOCOLLAPSE disqualifies the immediately following loop from collapsing with any other loop; before an elemental array syntax statement, it inhibits all collapse in said statement. SUBROUTINE S(A, N, N1, N2) REAL A(N, *) !DIR$ COLLAPSE (I, J) DO I = 1, N1 DO J = 1, N2 A(I,J) = A(I,J) + 42.0 ENDDO ENDDO END The above yields code equivalent to the following, which should not be coded directly because as program source, it violates the Fortran language standard. SUBROUTINE S(A, N, N1, N2) REAL A(N, *) DO IJ = 1, N1*N2 A(IJ, 1) = A(IJ, 1) + 42.0 ENDDO END With array syntax, the collapse directive appears as follows: SUBROUTINE S( A, B ) REAL, DIMENSION(:,:) :: A, B !DIR$ COLLAPSE A = B ! USER PROMISES UNIFORM ACCESS STRIDE. END In each of the above examples, the directive enables the compiler to assume appropriate conformity between trip counts and array extends. The compiler will diagnose misuse at compile time (when able); or, under option -Rd, at run time. NOCOLLAPSE prevents the compiler from collapsing a given loop with others or from performing any loop collapse within a specified array syntax statement. Collapse is almost always desirable, so this directive should be used sparingly. SUBROUTINE S(A, N) DIMENSION A(N,N) !DIR$ NOCOLLAPSE DO I = 1, N ! DISALLOW COLLAPSE INVOLVING I-LOOP. DO J = 1, N A(I,J) = 1.2 ENDDO ENDDO END S–3901–80 121Cray Fortran Reference Manual Loop collapse is a special form of loop coalesce. Any perfect loop nest may be coalesced into a single loop, with explicit rediscovery of the intermediate values of original loop control variables. The rediscovery cost, which generally involves integer division, is quite high. Hence, coalesce is rarely suitable for vectorization. It may be beneficial for multithreading. By definition, loop collapse occurs when loop coalesce may be done without the rediscovery overhead. To meet this requirement, all memory accesses must have uniform stride. This typically occurs when a computation can flow from one column of a multidimensional array into the next, viewing the array as a flat sequence. Hence, array sections such as A(:,3:7) are generally suitable for collapse, while a section like A(1:n-1,:) lacks the needed access uniformity. Care must taken when applying the collapse directive to assumed shape dummy arguments and Fortran pointers because the underlying storage need not be contiguous. 4.4.3 Determine Register Storage: NOSIDEEFFECTS The NOSIDEEFFECTS directive allows the compiler to keep information in registers across a single call to a subprogram without reloading the information from memory after returning from the subprogram. The directive is not needed for intrinsic functions. NOSIDEEFFECTS declares that a called subprogram does not redefine any variables that meet the following conditions: • Local to the calling program • Passed as arguments to the subprogram • Accessible to the calling subprogram through host association • Declared in a common block or module • Accessible through USE association The format of this directive is as follows: !DIR$ NOSIDEEFFECTS f [, f ] ... f Symbolic name of a subprogram that the user is sure has no side effects. f must not be the name of a dummy procedure, module procedure, or internal procedure. A procedure declared NOSIDEEFFECTS should not define variables in a common block or module shared by a program unit in the calling chain. All arguments should have the INTENT(IN) attribute; that is, the procedure must not modify its arguments. If these conditions are not met, results are unpredictable. The NOSIDEEFFECTS directive must appear in the specification part of a program unit and must appear before the first executable statement. 122 S–3901–80Using Cray Fortran Directives [4] The compiler may move invocations of a NOSIDEEFFECTS subprogram from the body of a DO loop to the loop preamble if the arguments to that function are invariant in the loop. This may affect the results of the program, particularly if the NOSIDEEFFECTS subprogram calls functions such as the random number generator or the real-time clock. The effects of the NOSIDEEFFECTS directive are similar to those that can be obtained by specifying the PURE prefix on a function or a subroutine declaration. For more information about the PURE prefix, refer to the Fortran Standard. 4.4.4 Suppress Scalar Optimization: SUPPRESS The SUPPRESS directive suppresses scalar optimization for all variables or only for those specified at the point where the directive appears. This often prevents or adversely affects vectorization of any loop that contains SUPPRESS. The format of this directive is as follows: !DIR$ SUPPRESS [ var [, var ] ... ] var Variable that is to be stored to memory. If no variables are listed, all variables in the program unit are stored. If more than one variable is specified, use a comma to separate vars. At the point at which !DIR$ SUPPRESS appears in the source code, variables in registers are stored to memory (to be read out at their next reference), and expressions containing any of the affected variables are recomputed at their next reference after !DIR$ SUPPRESS. The effect on optimization is equivalent to that of an external subroutine call with an argument list that includes the variables specified by !DIR$ SUPPRESS (or, if no variable list is included, all variables in the program unit). SUPPRESS takes effect only if it is on an execution path. Optimization proceeds normally if the directive path is not executed because of a GOTO or IF. Example: SUBROUTINE SUB (L) LOGICAL L A = 1.0 ! A is local IF (L) THEN !DIR$ SUPPRESS ! Has no effect if L is false CALL ROUTINE() ELSE PRINT *, A END IF END S–3901–80 123Cray Fortran Reference Manual In this example, optimization replaces the reference to A in the PRINT statement with the constant 1.0, even though !DIR$ SUPPRESS appears between A=1.0 and the PRINT statement. The IF statement can cause the execution path to bypass !DIR$ SUPPRESS. If SUPPRESS appears before the IF statement, A in PRINT * is not replaced by the constant 1.0. 4.5 Local Use of Compiler Features The following directives provide local control over specific compiler features. • [NO]BOUNDS • FREE and FIXED The -f and -R command line options apply to an entire compilation, but these directives override any command line specifications for source form or bounds checking. The following subsections describe these directives. 4.5.1 Check Array Bounds: [NO]BOUNDS Array bounds checking provides a check of most array references at both compile time and run time to ensure that each subscript is within the array's declared size. Note: Bounds checking behavior differs with the optimization level. Complete checking is guaranteed only when optimization is turned off by specifying -O 0 on the ftn command line. The -R command line option controls bounds checking for a whole compilation. The BOUNDS and NOBOUNDS directives toggle the feature on and off within a program unit. Either directive can specify particular arrays or can apply to all arrays. The formats of these directives are as follows: !DIR$ BOUNDS [ array [, array ] ... ] !DIR$ NOBOUNDS [ array [, array ] ... ] array The name of an array. The name cannot be a subobject of a derived type. When no array name is specified, the directive applies to all arrays. BOUNDS remains in effect for a given array until the appearance of a NOBOUNDS directive that applies to that array, or until the end of the program unit. Bounds checking can be enabled and disabled many times in a single program unit. Note: To be effective, these directives must follow the declarations for all affected arrays. It is suggested that they be placed at the end of a program unit's specification statements unless they are meant to control particular ranges of code. 124 S–3901–80Using Cray Fortran Directives [4] The bounds checking feature detects any reference to an array element whose subscript exceeds the array's declared size. For example: REAL A(10) C DETECTED AT COMPILE TIME: A(11) = X C DETECTED AT RUN TIME IF IFUN(M) EXCEEDS 10: A(IFUN(M)) = W The compiler generates an error message when it detects an out-of-bounds subscript. If the compiler cannot detect the out-of-bounds subscript (for example, if the subscript includes a function reference), a message is issued for out-of-bound subscripts when your program runs, but the program is allowed to complete execution. Bounds checking does not inhibit vectorization but typically increases program run time. If an array's last dimension declarator is *, checking is not performed on the last dimension's upper bound. Arrays in formatted WRITE and READ statements are not checked. Note: Array bounds checking does not prevent operand range errors that result when operand prefetching attempts to access an invalid address outside an array. Bounds checking is needed when very large values are used to calculate addresses for memory references. If bounds checking detects an out-of-bounds array reference, a message is issued for only the first out-of-bounds array reference in the loop. For example: DIMENSION A(10) MAX = 20 A(MAX) = 2 DO 10 I = 1, MAX A(I) = I 10 CONTINUE CALL TWO(MAX,A) END SUBROUTINE TWO(MAX,A) REAL A(*) ! NO UPPER BOUNDS CHECKING DONE END The following messages are issued for the preceding program: lib-1961 a.out: WARNING Subscript 20 is out of range for dimension 1 for array 'A' at line 3 in file 't.f' with bounds 1:10. lib-1962 a.out: WARNING Subscript 1:20:1 is out of range for dimension 1 for array 'A' at line 5 in file 't.f' with bounds 1:10. S–3901–80 125Cray Fortran Reference Manual 4.5.2 Specify Source Form: FREE and FIXED The FREE and FIXED directives specify whether the source code in the program unit is written in free source form or fixed source form. The FREE and FIXED directives override the -f option, if specified, on the command line. The formats of these directives are as follows: !DIR$ FREE !DIR$ FIXED These directives apply to the source file in which they appear, and they allow you to switch source forms within a source file. You can change source form within an INCLUDE file. After the INCLUDE file has been processed, the source form reverts back to the source form that was being used prior to processing of the INCLUDE file. 4.6 Storage Directives The following directives specify aspects of storing common blocks, variables, or arrays: • BLOCKABLE • BLOCKINGSIZE and NOBLOCKING • STACK The following sections describe these directives. 4.6.1 Permit Cache Blocking: BLOCKABLE Directive The BLOCKABLE directive specifies that it is legal to cache block the subsequent loops. The format of this directive is as follows: !DIR$ BLOCKABLE (do_variable,do_variable [,do_variable]...) where do_variable specifies the do_variable names of two or more loops. The loops identified by the do_variable names must be adjacent and nested within each other, although they need not be perfectly nested. This directive tells the compiler that these loops can be involved in a blocking situation with each other, even if the compiler would consider such a transformation illegal. The loops must also be interchangeable and unrollable. This directive does not instruct the compiler on which of these transformations to apply. 126 S–3901–80Using Cray Fortran Directives [4] 4.6.2 Declare Cache Blocking: BLOCKINGSIZE and NOBLOCKING Directives The BLOCKINGSIZE and NOBLOCKING directives assert that the loop following the directive either is (or is not) involved in a cache blocking for the primary or secondary cache. The formats of these directives are as follows: !DIR$ BLOCKINGSIZE(n1[,n2]) !DIR$ NOBLOCKING n1,n2 An integer number that indicates the block size. If the loop is involved in a blocking, it will have a block size of n1 for the primary cache and n2 for the secondary cache. The compiler attempts to include this loop within such a block, but it cannot guarantee this. For n1, specify a value such that n1 .GE. 0. For n2, specify a value such that n2 .LE. 2 30 . If n1 or n2 are 0, the loop is not blocked, but the entire loop is inside the block. Example: SUBROUTINE AMAT(X,Y,Z,N,M,MM) REAL(KIND=8) X(100,100), Y(100,100), Z(100,100) DO K = 1, N !DIR$ BLOCKABLE(J,I) !DIR$ BLOCKING SIZE (20) DO J = 1, M !DIR$ BLOCKING SIZE (20) DO I = 1, MM Z(I,K) = Z(I,K) + X(I,J)*Y(J,K) END DO END DO END DO END S–3901–80 127Cray Fortran Reference Manual For the preceding code, the compiler makes 20 x 20 blocks when blocking, but it could block the loop nest such that loop K is not included in the tile. If it did not, add a BLOCKINGSIZE(0) directive just before loop K to specify that the compiler should generate a loop such as the following: SUBROUTINE AMAT(X,Y,Z,N,M,MM) REAL(KIND=8) X(100,100), Y(100,100), Z(100,100) DO JJ = 1, M, 20 DO II = 1, MM, 20 DO K = 1, N DO J = JJ, MIN(M, JJ+19) DO I = II, MIN(MM, II+19) Z(I,K) = Z(I,K) + X(I,J)*Y(J,K) END DO END DO END DO END DO END DO END Note that an INTERCHANGE directive can be applied to the same loop nest as a BLOCKINGSIZE directive. The BLOCKINGSIZE directive applies to the loop it directly precedes; it moves with that loop when an interchange is applied. The NOBLOCKING directive prevents the compiler from involving the subsequent loop in a cache blocking situation. 4.6.3 Request Stack Storage: STACK The STACK directive causes storage to be allocated to the stack in the program unit that contains the directive. This directive overrides the -ev command line option in specific program units of a compilation unit. For more information about the -ev command line option, see -d disable and -e enable on page 31. The format of this directive is as follows: !DIR$ STACK Data specified in the specification part of a module or in a DATA statement is always allocated to static storage. This directive has no effect on this static storage allocation. All SAVE statements are honored in program units that also contain a STACK directive. This directive does not override the SAVE statement. If the compiler finds a STACK directive and a SAVE statement without any objects specified in the same program unit, a warning message is issued. 128 S–3901–80Using Cray Fortran Directives [4] The following rules apply when using this directive: • It must be specified within the scope of a program unit. • If it is specified in the specification part of a module, a message is issued. The STACK directive is allowed in the scope of a module procedure. • If it is specified within the scope of an interface body, a message is issued. 4.7 PGAS Directive Only Cray XE systems support the following PGAS directive. 4.7.1 DEFER_SYNC Directive The DEFER_SYNC directive defers the synchronization of PGAS data until the next fence instruction. Normally the compiler synchronizes the references in a statement as late as possible, without violating program semantics. The purpose of the DEFER_SYNC directive is to synchronize the references even later, beyond where the compiler can determine it is safe. PGAS data references made by the single statement immediately following the pgas defer_sync directive are not synchronized until the next fence instruction. Use this directive to force all references in the next statement to be non-blocking. For example, if there is a remote-memory access (RMA) put near the end of a subroutine, the compiler must guard against the put value being read back immediately after the subroutine returns, so the put is synchronized just before returning. The programmer, however, may know that the value is not read back and can insert a pgas defer_sync directive. The format is as follows: !dir$ pgas defer_sync Example 4. Using defer_sync subroutine my_put( x, image, value ) integer :: x[*], image, value !dir$ pgas defer_sync x[image] = value end subroutine See the defer_sync(7) intro_pgas(7) man pages. S–3901–80 129Cray Fortran Reference Manual 4.8 Miscellaneous Directives The following directives allow you to use several different compiler features: • [NO]AUTOTHREAD • CACHE • CACHE_NT • CONCURRENT • [NO]FUSION • ID • IGNORE_TKR • NAME • NOFISSION • PREPROCESS • WEAK 4.8.1 Control Autothreading: [NO]AUTOTHREAD The AUTOTHREAD and NOAUTOTHREAD directives turn autothreading on and off for selected blocks of code. These directives are ignored if the -h thread0 or -O thread0 options are used. The formats of these directives are as follows: !DIR$ AUTOTHREAD !DIR$ NOAUTOTHREAD The PREFER_THREAD and PREFER_NOTHREAD advisory directives can be used to indicate a preference for threading in the loop immediately following the advisory directive. The NOAUTOTHREAD directive takes precedence over PREFER_THREAD. For more information, see Autothreading for Loops: LOOP_INFO PREFER_[NO]THREAD on page 111. 4.8.2 Allocate Cache: CACHE The CACHE directive is an advisory directive that asserts that all memory operations with the specified symbols as the base are to be allocated in cache. Use this directive to identify objects that should be placed in cache. Advisory directives are directives the compiler honors if conditions permit. When this directive is used, code performance may be improved because objects with high cache reuse rates are retained in cache. 130 S–3901–80Using Cray Fortran Directives [4] To use the CACHE directive, place it only in the specification part, before any executable statement. The format of the CACHE directive is: !DIR$ CACHE base_name[, base_name] Where base_name is the object that should be placed into cache. This can be the base name of any object such as an array, scalar structure, and so on, without member references. If you specify a pointer in the list, only the references and not the pointer itself are cached. The CACHE directive overrides the automatic cache management level that was specified using the -O cachen option on the compiler command line. This directive may be overridden locally by use of the LOOP_INFO directive. 4.8.3 Non-temporal Reads and Writes: CACHE_NT The CACHE_NT directive is an advisory directive that specifies objects that should use non-temporal reads and writes. Use this directive to identify objects that should not be placed in cache. Advisory directives are directives the compiler honors if conditions permit. When this directive is used, code performance may be improved because objects with low cache reuse rates are kept out of cache, thus making room for objects with higher cache reuse rates. To use the CACHE_NT directive, place it only in the specification part, before any executable statement. The format of the CACHE_NT directive is: !DIR$ CACHE_NT base_name[, base_name] Where base_name is the object that should use non-temporal reads and writes. This can be the base name of any object such as an array, scalar structure, and so on, without member references. If you specify a pointer in the list, only the references and not the pointer itself have the cache non-temporal property. The CACHE_NT directive overrides the automatic cache management level that was specified using the -O cachen option on the compiler command line. This directive may be overridden locally by use of the LOOP_INFO directive. S–3901–80 131Cray Fortran Reference Manual 4.8.4 Specify Array Dependencies: CONCURRENT The CONCURRENT directive conveys array dependency information to the compiler. This directive affects the loop that immediately follows it. The CONCURRENT directive is useful when vectorization is specified by the command line. The format of this directive is as follows: !DIR$ CONCURRENT [ SAFE_DISTANCE=n] n An integer number that represents the number of additional consecutive loop iterations that can be executed in parallel without danger of data conflict. n must be an integeral constant > 0. If SAFE_DISTANCE=n is not specified, the distance is assumed to be infinite, and the compiler ignores all cross-iteration data dependencies. The CONCURRENT directive is ignored if the SAFE_DISTANCE argument is used and vectorization is requested on the command line. Example. Consider the following code: !DIR$ CONCURRENT SAFE_DISTANCE=3 DO I = K+1, N X(I) = A(I) + X(I-K) ENDDO The CONCURRENT directive in this example informs the optimizer that the relationship K > 3 is true. This allows the compiler to load all of the following array references safely during the Ith loop iteration: X(I-K) X(I-K+1) X(I-K+2) X(I-K+3) 4.8.5 Fuse Loops: [NO]FUSION The FUSION and NOFUSION directives allow you to fine-tune the selection of which DO loops the compiler should attempt to fuse. If there are only a few loops out of many that you want to fuse, then use the FUSION directive with the -O fusion1 option to confine loop fusion to these few loops. If there are only a few loops out of many that you do not want to fuse, use the NOFUSION directive with the -O fusion2 option to specify no fusion for these loops. These are the formats of the directives: !DIR$ FUSION !DIR NOFUSION The FUSION directive should be placed immediately before the DO statement of the loop that should be fused. 132 S–3901–80Using Cray Fortran Directives [4] 4.8.6 Do Not Fission Loop: NOFISSION The NOFISSION directive allows you to prevent the compiler from splitting the statements in a given loop into distinct loops. Fission is prevented only for the specific loop; loops nested within the indicated loop remain fission candidates unless likewise annotated. The format of the directive is as follows: !DIR NOFISSION The NOFISSION directive should be placed immediately before the DO statement of the loop that should not be fissioned. 4.8.7 Create Identification String: ID The ID directive inserts a character string into the file.o produced for a Fortran source file. The format of this directive is as follows: !DIR$ ID "character_string" character_ string The character string to be inserted into file.o. The syntax box shows quotation marks as the character_string delimiter, but you can use either apostrophes (' ') or quotation marks (" "). The character_string can be obtained from file.o in one of the following ways: • Method 1 — Using the what command. To use the what command to retrieve the character string, begin the character string with the characters @(#). For example, assume that id.f contains the following source code: !DIR$ ID '@(#)file.f 03 February 1999' PRINT *, 'Hello, world' END The next step is to use file id.o as the argument to the what command, as follows: % what id.o % id.o: % file.f 03 February 1999 Note that what does not include the special sentinel characters in the output. In the following example, character_string does not begin with the characters @(#). The output shows that what does not recognize the string. Input file id2.o contains the following: !DIR$ ID 'file.f 03 February 1999' PRINT *, 'Hello, world' END S–3901–80 133Cray Fortran Reference Manual The what command generates the following output: % what id2.o % id2.o: • Method 2 — Using strings or od. The following example shows how to obtain output using the strings command. Input file id.f contains the following: !DIR$ ID "File: id.f Date: 03 February 1999" PRINT *, 'Hello, world' END The strings command generates the following output: % strings id.o 02/03/9913:55:52f90 3.3cn $MAIN @CODE @DATA @WHAT $MAIN $STKOFEN f$init _FWF $END *?$F(6( Hello, world $MAIN File: id.f Date: 03 February 1999 % od -tc id.o ... portion of dump deleted 0000000001600 \0 \0 \0 \0 \0 \0 \0 \n F i l e : i d 0000000001620 . f D a t e : 0 3 F e b 0000000001640 r u a r y 1 9 9 9 \0 \0 \0 \0 \0 \0 ... portion of dump deleted 4.8.8 Disregard Dummy Argument Type, Kind, and Rank: IGNORE_TKR The IGNORE_TKR directive directs the compiler to ignore the type, kind, and/or rank (TKR) of specified dummy arguments in a procedure interface. The format for this directive is as follows: !DIR$ IGNORE_TKR [ [(letter) dummy_arg] ... ] letter The letter can be T, K, or R, or any combination of these letters (for example, TK or KR). The letter applies only to the dummy argument it precedes. If letter appears, dummy_arg must appear. 134 S–3901–80Using Cray Fortran Directives [4] dummy_arg If specified, it indicates the dummy arguments for which TKR rules should be ignored. If not specified, TKR rules are ignored for all dummy arguments in the procedure that contains the directive. The directive causes the compiler to ignore the type, kind, and/or rank of the specified dummy arguments when resolving a generic call to a specific call. The compiler also ignores the type, kind, and/or rank on the specified dummy arguments when checking all the specifics in a generic call for ambiguities. Example: The following directive instructs the compiler to ignore type, kind, and/or rank rules for the dummy arguments of the following subroutine fragment: subroutine example(A,B,C,D) !DIR$ IGNORE_TKR A, (R) B, (TK) C, (K) D Table 8 indicates what is ignored for each dummy argument. Table 8. Explanation of Ignored TKRs Dummy Argument Ignored A Type, kind and rank is ignored B Only rank is ignored C Type and kind is ignored D Only kind is ignored 4.8.9 External Name Mapping: NAME The NAME directive allows you to specify a case-sensitive external name, or a name that contains characters outside of the Fortran character set, in a Fortran program. The case-sensitive external name is specified on the NAME directive, in the following format: !DIR$ NAME (fortran_name="external_name" [, fortran_name="external_name" ] ... ) fortran_name The name used for the object throughout the Fortran program. external_name The external form of the name. Rules for Fortran naming do not apply to the external_name string; any character sequence is valid. You can use this directive, for example, when writing calls to C routines. S–3901–80 135Cray Fortran Reference Manual Example: PROGRAM MAIN !DIR$ NAME (FOO="XyZ") CALL FOO ! XyZ is really being called END PROGRAM Note: The Fortran standard BIND feature provides some of the capability of the NAME directive. 4.8.10 Preprocess Include File: PREPROCESS The PREPROCESS directive allows an include file to be preprocessed when the compilation does not specify the preprocessing command line option. This directive does not cause preprocessing of included files, unless they too use the directive. If the preprocessing command line option is used, preprocessing occurs normally for all files. To use the directive, it must be the first line in the include file and in each included file that needs to be preprocessing. This is the format of the PREPROCESS directive: !DIR$ PREPROCESS [expand_macros] The optional expand_macros clause allows the compiler to expand all macros within the include files. Without this clause, macro expansion occurs only within preprocessing directives. 4.8.11 Specify Weak Procedure Reference: WEAK Sometimes, the code path of a program never executes at run time because of some condition. If this code path references a procedure that is external to the program (for example, a library procedure), the linker will add the binary for the procedure to the compiled program, resulting in a larger program. When statically linking, the weak directive specifies an external identifier that may remain unresolved throughout the compilation. This directive has no effect when dynamically linking. The WEAK directive can prevent the compiler driver from adding the binary to your program, resulting in a smaller program and less use of memory. The WEAK directive is used with procedures and variables to declare weak objects. The use of a weak object is referred to as a weak reference. The existence of a weak reference does not cause the compiler driver to add the appropriate binaries into a compiled program, so executing a weak reference will cause the program to fail. The compiler support for determining if the binary of a weak object is linked is deferred. To cause the compiler driver to add the binaries so the weak reference will work, you must have a strong reference (a normal reference) somewhere in the program. 136 S–3901–80Using Cray Fortran Directives [4] The following example illustrates the reason the WEAK directive is used. The startup code, which is compiled into every Fortran program, calls the SHMEM initialization routine, which causes the linker to add the binary of the initialization routine to every compiled program if a strong reference to the routine is used. This binary is unnecessary if a program does not use SHMEM. To avoid linking unnecessary code, the startup code uses the WEAK directive for the initialization routine. In this manner, if the program does not use SHMEM, the linker does not add the binary of the initialization routine even though the startup code calls it. However, if the program calls the SHMEM routines using strong references, the linker adds the necessary binaries, including the initialization binary into the compiled program. The WEAK directive has two forms: !DIR$ WEAK procedure_name [, procedure_name] ... !DIR$ WEAK procedure_name = stub_name[, procedure_name1 = stub_name1] ... The first form allows you to specify one or more weak objects. This form requires you to implement code that senses that the procedure_name procedure is linked before calling it. The second form allows you to point a weak reference (procedure_name) to a stub procedure that exists in your code. This allows you to call the stub if a strong reference to procedure_name does not exist. If a strong reference to procedure_name exists, it is called instead of the stub. The stub_name procedure must have the same name and dummy argument list as procedure_name. Note: The linker does not issue an unresolved reference error message for weak procedure references. S–3901–80 137Cray Fortran Reference Manual 138 S–3901–80Source Preprocessing [5] Source preprocessing can help you port a program from one platform to another by allowing you to specify source text that is platform specific. For a source file to be preprocessed automatically, it must have an uppercase extension, either .F (for a file in fixed source form), or .F90 or .FTN (for a file in free source form). To specify preprocessing of source files with other extensions, including lowercase ones, use the -eP or -eZ options described in Command Line Options on page 148. 5.1 General Rules You can alter the source code through source preprocessing directives. These directives are fully explained in Directives on page 140. The directives must be used according to the following rules: • Do not use source preprocessor (#) directives within multiline compiler directives (CDIR$, !DIR$, CSD$, !CSD$, C$OMP, or !$OMP). • You cannot include a source file that contains an #if directive without a balancing #endif directive within the same file. The #if directive includes the #ifdef and #ifndef directives. • If a directive is too long for one source line, the backslash character (\) is used to continue the directive on successive lines. Successive lines of the directive can begin in any column. The backslash character (\) can appear in any location within a directive in which white space can occur. A backslash character (\) in a comment is treated as a comment character. It is not recognized as signaling continuation. • Every directive begins with the pound character (#), and the pound character (#) must be in column 1. • Blank and tab (HT) characters can appear between the pound character (#) and the directive keyword. • You cannot write form feed (FF) or vertical tab (VT) characters to separate tokens on a directive line. That is, a source preprocessing line must be continued, by using a backslash character (\), if it spans source lines. S–3901–80 139Cray Fortran Reference Manual • Blanks are significant, so the use of spaces within a source preprocessing directive is independent of the source form of the file. The fields of a source preprocessing directive must be separated by blank or tab (HT) characters. • Any user-specified identifier that is used in a directive must follow Fortran rules for identifier formation. The exceptions to this rule are as follows: – The first character in a source preprocessing name (a macro name) can be an underscore character (_). – Source preprocessing names are significant in their first 132 characters whereas a typical Fortran identifier is significant only in its first 63 characters. • Source preprocessing identifier names are case sensitive. • Numeric literal constants must be integer literal constants or real literal constants, as defined for Fortran. • Comments written in the style of the C language, beginning with /* and ending with */, can appear anywhere within a source preprocessing directive in which blanks or tabs can appear. The comment, however, must begin and end on a single source line. • Directive syntax allows an identifier to contain the ! character. Therefore, placing the ! character to start a Fortran comment on the same line as the directive should be avoided. 5.2 Directives The blanks shown in the syntax descriptions of the source preprocessing directives are significant. The tab character (HT) can be used in place of a blank. Multiple blanks can appear wherever a single blank appears in a syntax description. 5.2.1 #include Directive The #include directive directs the system to use the content of a file. Just as with the INCLUDE line path processing defined by the Fortran standard, an #include directive effectively replaces that directive line by the content of filename. This directive has the following formats: #include "filename" #include 140 S–3901–80Source Preprocessing [5] filename A file or directory to be used. In the first form, if filename does not begin with a slash (/) character, the system searches for the named file, first in the directory of the file containing the #include directive, then in the sequence of directories specified by the -I option(s) on the ftn command line, and then the standard (default) sequence. If filename begins with a slash (/) character, it is used as is and is assumed to be the full path to the file. The second form directs the search to begin in the sequence of directories specified by the -I option(s) on the ftn command line and then search the standard (default) sequence. The Fortran standard prohibits recursion in INCLUDE files, so recursion is also prohibited in the #include form. The #include directives can be nested. When the compiler is invoked to do only source preprocessing, not compilation, text will be included by #include directives but not by Fortran INCLUDE lines. For information about the source preprocessing command line options, see Command Line Options on page 148. 5.2.2 #define Directive The #define directive lets you declare a variable and assign a value to the variable. It also allows you to define a function-like macro. This directive has the following format: #define identifier value #define identifier(dummy_arg_list) value The first format defines an object-like macro (also called a source preprocessing variable), and the second defines a function-like macro. In the second format, the left parenthesis that begins the dummy_arg_list must immediately follow the identifier, with no intervening white space. identifier The name of the variable or macro being defined. Rules for Fortran variable names apply; that is, the name cannot have a leading underscore character (_). For example, ORIG is a valid name, but _ORIG is invalid. S–3901–80 141Cray Fortran Reference Manual dummy_arg_list A list of dummy argument identifiers. value The value is a sequence of tokens. The value can be continued onto more than one line using backslash (\) characters. If a preprocessor identifier appears in a subsequent #define directive without being the subject of an intervening #undef directive, and the value in the second #define directive is different from the value in the first #define directive, then the preprocessor issues a warning message about the redefinition. The second directive's value is used. For more information about the #undef directive, see #undef Directive on page 143. When an object-like macro's identifier is encountered as a token in the source file, it is replaced with the value specified in the macro's definition. This is referred to as an invocation of the macro. The invocation of a function-like macro is more complicated. It consists of the macro's identifier, immediately followed by a left parenthesis with no intervening white space, then a list of actual arguments separated by commas, and finally a terminating right parenthesis. There must be the same number of actual arguments in the invocation as there are dummy arguments in the #define directive. Each actual argument must be balanced in terms of any internal parentheses. The invocation is replaced with the value given in the macro's definition, with each occurrence of any dummy argument in the definition replaced with the corresponding actual argument in the invocation. For example, the following program prints Hello, world. when compiled and run: PROGRAM P #define GREETING 'Hello, world.' PRINT *, GREETING END PROGRAM P The following program prints Hello, Hello, world. when compiled and run: PROGRAM P #define GREETING(str1, str2) str1, str1, str2 PRINT *, GREETING('Hello, ', 'world.') END PROGRAM P 142 S–3901–80Source Preprocessing [5] 5.2.3 #undef Directive The #undef directive sets the definition state of identifier to an undefined value. If identifier is not currently defined, the #undef directive has no effect. This directive has the following format: #undef identifier identifier The name of the variable or macro being undefined. 5.2.4 # (Null) Directive The null directive simply consists of the pound character (#) in column 1 with no significant characters following it. That is, the remainder of the line is typically blank or is a source preprocessing comment. This directive is generally used for spacing out other directive lines. 5.2.5 Conditional Directives Conditional directives cause lines of code to either be produced by the source preprocessor or to be skipped. The conditional directives within a source file form if-groups. An if-group begins with an #if, #ifdef, or #ifndef directive, followed by lines of source code that you may or may not want skipped. Several similarities exist between the Fortran IF construct and if-groups: • The #elif directive corresponds to the ELSE IF statement. • The #else directive corresponds to the ELSE statement. • Just as an IF construct must be terminated with an END IF statement, an if-group must be terminated with an #endif directive. • Just as with an IF construct, any of the blocks of source statements in an if-group can be empty. For example, you can write the following directives: #if MIN_VALUE == 1 #else ... #endif Determining which group of source lines (if any) to compile in an if-group is essentially the same as the Fortran determination of which block of an IF construct should be executed. S–3901–80 143Cray Fortran Reference Manual 5.2.5.1 #if Directive The #if directive has the following format: #if expression expression An expression. The values in expression must be integer literal constants or previously defined preprocessor variables. The expression is an integer constant expression as defined by the C language standard. All the operators in the expression are C operators, not Fortran operators. The expression is evaluated according to C language rules, not Fortran expression evaluation rules. Note that unlike the Fortran IF construct and IF statement logical expressions, expression in an #if directive need not be enclosed in parentheses. The #if expression can also contain the unary defined operator, which can be used in either of the following formats: defined identifier defined(identifier) When the defined subexpression is evaluated, the value is 1 if identifieris currently defined, and 0 if it is not. All currently defined source preprocessing variables in expression, except those that are operands of defined unary operators, are replaced with their values. During this evaluation, all source preprocessing variables that are undefined evaluate to 0. Note that the following two directive forms are not equivalent: • #if X • #if defined(X) In the first case, the condition is true if X has a nonzero value. In the second case, the condition is true only if X has been defined (has been given a value that could be 0). 5.2.5.2 #ifdef Directive The #ifdef directive is used to determine if identifier is predefined by the source preprocessor, has been named in a #define directive, or has been named in a ftn -D command line option. For more information about the -D option, see Command Line Options on page 148. This directive has the following format: #ifdef identifier 144 S–3901–80Source Preprocessing [5] The #ifdef directive is equivalent to either of the following two directives: • #if defined identifier • #if defined(identifier) 5.2.5.3 #ifndef Directive The #ifndef directive tests for the presence of an identifier that is not defined. This directive has the following format: #ifndef identifier This directive is equivalent to either of the following two directives: • #if ! defined identifier • #if ! defined(identifier) 5.2.5.4 #elif Directive The #elif directive serves the same purpose in an if-group as does the ELSE IF statement of a Fortran IF construct. This directive has the following format: #elif expression expression The expression follows all the rules of the integer constant expression in an #if directive. 5.2.5.5 #else Directive The #else directive serves the same purpose in an if-group as does the ELSE statement of a Fortran IF construct. This directive has the following format: #else 5.2.5.6 #endif Directive The #endif directive serves the same purpose in an if-group as does the END IF statement of a Fortran IF construct. This directive has the following format: #endif S–3901–80 145Cray Fortran Reference Manual 5.3 Predefined Macros The Cray Fortran compiler source preprocessing supports a number of predefined macros. They are divided into groups as follows: • Macros based on the host machine • Macros based on CLE system targets • Macros based on the Cray Fortran compiler • Macros based on the source file The following predefined macros are based on the host system (the system upon which the compilation is being done): unix, __unix, __unix__ Always defined. (The leading characters in the second form consist of 2 consecutive underscores; the third form consists of 2 leading and 2 trailing underscores.) The following predefined macros are based on CLE systems as targets: _ADDR64 Defined for CLE systems as targets. The target system must have 64-bit address registers. _OPENMP Defined as the publication date of the OpenMP standard supported, as a string of the form yyyymm. _MAXVL_8 Defined as 16, the number of 8-bit elements that fit in an XMM register ("vector length"). On targets that support AVX, defined as 32, the number of 8-bit elements that fit in a YMM register ("vector length"). _MAXVL_16 Defined as 8. On targets that support AVX, defined as 16. _MAXVL_32 Defined as 4. On targets that support AVX, defined as 8. 146 S–3901–80Source Preprocessing [5] _MAXVL_64 Defined as 2. On targets that support AVX, defined as 4. _MAXVL_128 Defined as 0. On targets that support AVX, defined as 2. The following macro is based on the Cray Fortran compiler: _CRAYFTN Defined as 1. The following predefined macros are based on the source file: __line__, __LINE__ Defined to be the line number of the current source line in the source file. __file__, __FILE__ Defined to be the name of the current source file. __date__, __DATE__ Defined to be the current date in the form mm/dd/yy. __time__, __TIME__ Defined to be the current in the form hh:mm:ss. S–3901–80 147Cray Fortran Reference Manual 5.4 Command Line Options The following ftn command line options affect source preprocessing. • The -D identifier[=value] option, which defines variables used for source preprocessing. For more information about this option, see -D identifier[=value] on page 38. • The -dF option, which controls macro expansion in Fortran source statements. For more information about this option, see -d disable and -e enable on page 31. • The -eP option, which performs source preprocessing on file.f[90], file.F[90], file.ftn, or file.FTN but does not compile. The -eP option produces file.i. For more information about this option, see -d disable and -e enable on page 31. • The -eZ option, which performs source preprocessing and compilation on file.f[90], file.F[90], file.ftn, or file.FTN. The -eZ option produces file.i. For more information about this option, see -d disable and -e enable on page 31. • The -U identifier [, identifier] ... option, which undefines variables used for source preprocessing. For more information about this option, see -U identifier [,identifier] ... on page 82. The -D identifier [=value] and -U identifier [, identifier] ... options are ignored unless one of the following is true: • The Fortran input source file is specified as either file.F, file.F90, or file.FTN. • The -eP or -eZ options have been specified. 148 S–3901–80Using the OpenMP Fortran API [6] OpenMP is a parallel programming model that is portable across shared memory architectures from Cray and other vendors. The Cray Fortran compiler supports the OpenMP Application Program Interface, Version 3.0 standard. All OpenMP library procedures and directives, except for limitations in a few directive clauses, are supported. All OpenMP directives and library procedures are documented by the OpenMP Fortran specification which is accessible at http://openmp.org/wp/openmp-specifications/. Starting with CCE 7.3, this compiler also includes partial support for the OpenMP 3.1 specification. As a result, applications using OpenMP API constructs cannot combine object files compiled with CCE 7.3 (or later) and object files compiled with previously released CCE compilers. This restriction also applies to the OpenMP runtime library released with CCE 7.3; it is not possible to link CCE 7.2 (or earlier) object files with the CCE 7.3 (or later) OpenMP runtime library. 6.1 OpenMP Standard Support This release supports OpenMP Application Program Interface, Version 3.0. All OpenMP library procedures and directives, except for limitations in a few directive clauses, are supported. The CCE 8.0 release includes partial support for the OpenMP 3.1 specification. See http://www.openmp.org/mp-documents/omp3.1-2011.0203a.pdf . The following extensions, described in the OpenMP 3.1 specification, are supported: • atomic construct and extensions. • taskyield construct • firstprivate clause now accepts intend(in) and constant objects This release also adds support for proposed extensions to the OpenMP 3.1 specification to support accelerators. S–3901–80 149Cray Fortran Reference Manual 6.2 OpenMP Implementation-Dependent Behaviors The following are Cray-specific behaviors in areas that are defined as implementation-dependent by the OpenMP specification. • Parallel region constructs: – If a parallel region is encountered while dynamic adjustment of the number of threads is disabled, and the number of threads specified for the parallel region exceeds the number that the run time system can supply, the program terminates. – The number of physical processors actually hosting the threads at any given time is fixed at program startup and is specified by the aprun -d depth option. • DO and PARALLEL DO directives: – SCHEDULE(GUIDED,chunk)—The size of the initial chunk for the master thread and other team members is approximately equal to the trip count divided by the number of threads. – SCHEDULE(RUNTIME)—The schedule type and chunk size can be chosen at run time by setting the OMP_SCHEDULE environment variable. If this environment variable is not set, the schedule type and chunk size default to STATIC and 0, respectively. – Default schedule—In the absence of the SCHEDULE clause, the default schedule is STATIC and the default chunk size is roughly the number of iterations divided by the number of threads. • THREADPRIVATE directives: if the dynamic threads mechanism is enabled, the definition and association status of a thread's copy of the variable is undefined and the allocation status of an allocatable array is undefined. • PRIVATE clause: if a variable is declared as PRIVATE and the variable is referenced in the definition of a statement function, and the statement function is used within the lexical extent of the directive construct, then the statement function references the PRIVATE version of the variable. • ATOMIC directives: the ATOMIC directive is replaced with a critical section that encloses the statement. • OpenMP library functions: – OMP_SET_NUM_THREADS—If dynamic adjustment of the number of threads is disabled, the number_of_threads argument sets the number of threads for all subsequent parallel regions until this procedure is called again with a different value. – OMP_SET_DYNAMIC—The default for dynamic thread adjustment is on. 150 S–3901–80Using the OpenMP Fortran API [6] – OMP_NESTED—The default for nested parallelism is false. – OMP_SET_MAX_ACTIVE_LEVELS—The Cray implementation of OpenMP supports the OpenMP 3.0 omp_set_max_active_levels option to limit the depth of nested parallelism. The number specified controls the maximum number of nested parallel levels with more than one thread. The default value is 1 (nesting disabled). – OMP_GET_MAX_ACTIVE_LEVELS—The Cray implementation of OpenMP supports the OpenMP 3.0 omp_get_max_active_levels function to return the maximum number of nested parallel levels currently allowed. • OpenMP environment variables: – OMP_DYNAMIC—The default value is .TRUE. – OMP_NESTED—The default value is .FALSE. – OMP_NUM_THREADS—If this environment variable is not set and the user does not use the omp_set_num_threads call to set the number of OpenMP threads, the default is 1 thread. The maximum number of threads per compute node is 4 times the number of allocated processors. If the requested value of OMP_NUM_THREADS is more than the number of threads an implementation can support, the behavior of the program depends on the value of the OMP_DYNAMIC environment variable. If OMP_DYNAMIC is .FALSE., the program terminates. If OMP_DYNAMIC is .TRUE., it uses up to 4 times the number of allocated processors. For example, on a quad-core system, this means the program can use up to 16 threads per compute node. – OMP_SCHEDULE—The default values for this environment variable are STATIC for schedule and 0 for chunk size. – OMP_MAX_ACTIVE_LEVELS—The default value is 1. – OMP_STACKSIZE—The default value is 128MB. – OMP_THREAD_LIMIT—Sets the number of OpenMP threads to use for the whole OpenMP program by setting the thread-limit-var ICV. The Cray implementation defaults to four times the number of available processors. – OMP_WAIT_POLICY—Provides a hint to an OpenMP implementation about the desired behavior of waiting threads by setting the wait-policy-var ICV. A compliant OpenMP implementation may or may not abide by the setting of the environment variable. The default value is active. • OpenMP library routines with generic interfaces: if an OMP run time library routine interface is defined to be generic by an implementation, use of arguments of kind other than those specified by the OMP_*_KIND constants is undefined. S–3901–80 151Cray Fortran Reference Manual These OpenMP features have Cray-specific behaviors in areas not defined as implementation-dependent by the OpenMP specification: • If the omp_lib module is not used and the kind of the actual argument does not match the kind of the dummy argument, the behavior of the procedure is undefined. • The omp_get_wtime and omp_get_wtick procedures return REAL(KIND=8) values instead of DOUBLE PRECISION values. 6.3 OpenMP Debugging The -g option provides debugging support for OpenMP directives. The -g option provides debugging support identical to specifying the -G0 option. This level of debugging implies -O0 -homp (optimizations disabled but OpenMP directives are recognized) and -h fp0. If you want to debug without OpenMP, use -g -xomp or -g -hnoomp, which will disable OpenMP and turn on debugging. 6.4 OpenMP Accelerator Support To support accelerators, the Cray implementation of OpenMP extends the standard OpenMP programming model adding the concept of execution engines (accelerators) which are not members of the parallel team in the standard sense. Accelerators are managed by the runtime environment, but only available to the programmer through directives or calls to accelerator specific functions. The intro_openmp_acc(7) man page contains the most current information regarding OpenMP accelerator support. 6.4.1 OpenMP Accelerator Execution Model When a thread encounters an accelerator region, an explicit accelerator task is generated and execution of the task is assigned to an accelerator attached to the system. The runtime environment controls which accelerator to use. Completion of an accelerator task is guaranteed before the next barrier completes or when an accelerator task synchronization construct is encountered. All accelerator tasks are guaranteed to complete before the program exits. The compiler supports accelerator regions that contain a single level of accelerator loop. If there are loops within the accelerator loop, the compiler will attempt to automatically create a second level of parallelism using one of more of those loops. If a second level is found, the outer loop is partitioned across the thread blocks and the inner loop is partitioned across the threads within a single block. If a second level of parallelism is not found, the outer loops is partitioned across both the thread blocks and threads within the block. 152 S–3901–80Using the OpenMP Fortran API [6] 6.4.2 OpenMP Accelerator Memory Model The OpenMP 3.1 specification describes a relaxed-consistency, shared-memory model in which hreads have access to both shared and threadprivate memory. Directives which specify data-sharing attributes determine the type of access to variables used in the associated structured block: shared and private. In this description, memory located on the accelerator will be referred to as resident memory and memory located on the host CPU is referred to as host memory. An accelerator will have access to both private and shared accelerator resident objects. Host memory accesses may be accomplished via direct access or via data copying. If the host modifies or reads a memory location that is duplicated on the accelerator while the accelerator region is active, the result is unspecified, unless the threads accessing the location are explicitly synchronized. How data is moved between the host and the accelerator is unspecified however data motion directives will be provided as hints to the compiler. The result of global memory modifications made by either the host or the accelerator may not be visible until a flush is executed on both the host and the accelerator. Accelerators do not have access to memory equivalent to OpenMP threadprivate memory; all thread private data shall be marked as firstprivate. Accelerator resident shared objects may be updated and or copied back to the host whenever the accelerator is not modifying the data. 6.4.3 OpenMP Accelerator Directives Note that these accelerator directives are subject to change in future releases. See the intro_openmp_acc(7) man page for the most current information regarding these directives. Clauses described as "Ignored" have no effect. Clauses described as "Not Supported" will produce an error. 6.4.3.1 acc_region When a thread encounters an acc_region directive, an accelerated task is created. This task can either be assigned to the default accelerator, or it can be assigned to the threads in a parallel team. If an acc_region is not placed on an accelerator because of lack of resources, and the construct is not already inside of a parallel region then a new parallel team will be constructed. If the acc_region is inside of a data environment, it shall be assigned to the same accelerator the acc_data is bound to. The compiler currently does not support function calls within an accelerator region. The compiler will attempt to inline any routines called within a region. If the calls cannot be inlined, the compiler will issue an error. The binding thread set for an acc_region is the encountering thread. The binding accelerator is the accelerator that is assigned the region by the runtime or the team of threads created to accelerate the region. S–3901–80 153Cray Fortran Reference Manual Restrictions: A program should not branch into or out of the construct. A program should not depend on any ordering of the evaluations of the clauses. A program should not depend on any side effects of the evaluations of the clauses. !$omp acc_region [clause [[,] clause]...] Structured-block !$omp end acc_region Clauses: async(handle[, handle]) (Deferred implementation) async_handle(handle[, handle]) (Deferred implementation) device(integer-expression[, integer-expression]) (Deferred implementation) The device clause causes the construct to be tied to the accelerator determined by the constant positive integer expression. The meaning of multiple device integer expressions is still undefined. if(scalar-expression | scalar-logical-expression) (Deferred implementation) If the clause evaluates to true, the region is accelerated. If the clause evaluates to false the region is run by the encountering thread. num_pes(depth:num[, depth:num ]) Controls the number of processing elements that are applied to a given level. The depth and num are integer expressions. If the expressions are constant the compiler may take advantage of the information and generate improved code. Two depths are supported. The first depth represents the number of threads per block and it must be a multiple of 32. If there is only a single layer of parallelism, the first depth number is treated as the total number of threads and the actual number of blocks is calculated accordingly. If a depth is given that is less than the current depth, it will be ignored. If a depth is given that is greater than the current depth. it will be preallocated or reserve resources for the depth when it is encountered. acc_shared(list) Supported. See acc_data on page 155. 154 S–3901–80Using the OpenMP Fortran API [6] acc_copy(list) Supported. See acc_data on page 155. acc_copyin(list) Supported. See acc_data on page 155. acc_copyout(list) Supported. See acc_data on page 155. host_shared(list) Treated as an acc_copy. firstprivate(list) Causes all of the private versions of the list objects to be initialized to the state of the associated host shared object. private(list) Causes a unique copy of the objects in the list to be provided to each task on the accelerator. present(list|*) Supported. See acc_data on page 155. default(acc_shared|acc_copy|firstprivate|private|none|ignore) Specifies the default behavior for all objects that are used but not explicitly placed in a given memory type. If default( none ) is specified then all objects used inside the construct must be present on a data-sharing attribute clause list. If no default is provided the default is acc_copy. If default ( ignore ) is specified no default scoping will occur. Any objects that are not explicitly scoped must be in a present clause. 6.4.3.2 acc_data The acc_data directive starts an accelerator data region and defines a data scope that is reused across multiple accelerated constructs. When a thread encounters an acc_data construct, an accelerated task data environment is created. This task data environment is assigned to the default accelerator. S–3901–80 155Cray Fortran Reference Manual The binding thread set for the region is the encountering thread. The binding accelerator is the accelerator that is assigned the region by the runtime or the team of threads created to accelerate the region. A program that branches into or out of the construct is non-conforming. A program shall not depend on any ordering of the evaluations of the clauses. A program shall not depend on any side effects of the evaluations of the clauses. !$omp acc_data [clause [[,] clause]...] Structured-block !$omp end acc_data Clauses: device(integer-expression[, integer-expression]) (Deferred implementation) Ignored. Only a single device is supported. acc_copy(list) Causes the accelerator shared objects to be initialized to the host's memory state when the region starts and then causes the host's memory state to be updated with the accelerator memory state when the region ends. This combines the behavior of the acc_copyin and acc_copyout clauses. An object that appears in an acc_copy clause shall not appear in any other data sharing clauses. acc_copyin(list) Causes the accelerator shared objects to be initialized to the host's memory state when the region starts. Objects in this clause may also appear in the acc_copyout clause. acc_copyout(list) Causes the host's memory to be updated with the accelerators shared objects state when the accelerator region ends. Objects in this clause may also appear in the acc_copyin clause. host_shared(list) Treated as an acc_copy clause. acc_shared(list) Causes the objects in the list to be shared by all tasks executing on the associated accelerator. 156 S–3901–80Using the OpenMP Fortran API [6] present(list|*) Supported with restrictions. Causes the system to look for the list items on the accelerator. If the list item also appears in an acc_shared, acc_copy, acc_copyin or acc_copyout clause, then if the object is not found, the copy clause takes effect. If the list item is not in another data placement clause it is an error to not find the object on the accelerator. The special * list is the same as listing all objects that appear in the other acc_shared, acc_copy, acc_copyin or acc_copyout clauses. Restrictions: Array sections of assumed size or allocatable arrays are not supported. The array will not be found in the present table. default(acc_shared|host_shared|acc_copy|none|ignore) Specifies the default behavior for all objects that are used but not explicitly placed in a given memory type. If default( none ) is specified then all objects used inside the construct shall be present on a data-sharing attribute clause list. If no default is provided the default is copy. If default( ignore ) is specified no default scoping will occur. Any objects that are not explicitly scoped must be in a present clause. 6.4.3.3 acc_loop The accelerator loop construct specifies that the iterations of one or more associated loops will be executed in an accelerated manner. More than one level of acc_loop is allowed within an accelerator region. The associated loops must be structured blocks. The binding set for an acc_loop is the encountering accelerator. !$omp acc_loop [clause [[,] clause]...] Do-loop-nest !$omp end acc_loop Clauses: cache(integer-expression[, integer-expression)] Deprecated. collapse(integer-expression) Causes the first integer-expression loops in the associated loop nest to be turned into a single loop incorporating the iteration space of the associated loops. The argument integer-expression is a positive constant integer expression and less than the depth of the loop-nest. All loops associated with the collapse clause shall be perfectly nested. S–3901–80 157Cray Fortran Reference Manual hetero(list) Ignored. host(list) Ignored. kernel or innermost Only kernel is supported. The kernel clause causes all contained loops to be run sequentially. level(dimension) Ignored. max_par_level(expr) Ignored. num_pes(depth:num[, depth:num)] Ignored. reduction(operator:list) Specifies an operator and one or more list items. For each list item, a private copy is created in each thread, and is initialized appropriately for the operator. After the end of the region, the original list item is updated with the values of the private copies that are combined using the specified operator. See section 2.9.3.6 of the OpenMP 3.1 specification for clarification. schedule Ignored. seq Ignored. stripmine Ignored. vector Ignored. 6.4.3.4 acc_region_loop The acc_region_loop is a combined construct consisting of an accelerator region and an accelerator loop which contains a single loop nest. The binding set for acc_region_loop is the encountering accelerator. This directive uses the combined set of clauses available with the acc_region and acc_loop directives. !$omp acc_region_loop [clause [[,] clause]...] Do-loop-nest !$omp end acc_region_loop 158 S–3901–80Using the OpenMP Fortran API [6] 6.4.3.5 acc_update The acc_update directive is used within an explicit or implicit data region to update all or part of a host memory array with values from the corresponding array in accelerator memory, or to update all or part of an accelerator memory object with values from the corresponding object in host memory. This directive may not appear inside of an accelerator region. The list items on the accelerator shall be visible in accelerator shared memory. This directive is executable. It shall not appear in place of the statement following an if, while, do, switch, or label in C, or in place of the statement following a logical if in Fortran. !$omp acc_update [clause [[,] clause]...] Clauses: host( obj1: [obj2][[,] obj1[: obj2]] ) This directive allows both the host and the accelerator to work independently and then to update each other with any visible changes before continuing. Causes the obj1 objects to be copied from the host memory to the accelerator memory, if the memories are distinct. If the optional obj2 object is provided then obj1 is the source object and obj2 is the destination object. This allows the programmer to move data between two objects that have different identifiers. See data clause restrictions. acc(obj1: [obj2][[,] obj1[: obj2]]) Causes the obj1 objects to be copied from the host memory to the accelerator memory, if the memories are distinct. If the optional obj2 object is provided then obj1 is the source object and obj2 is the destination object. See data clause restrictions. 6.4.4 OpenMP Accelerator Runtime Functions omp_acc_async_wait(handle) (Deferred implementation) Wait until the accelerator region associated with the handle to complete. omp_acc_async_test(handle) (Deferred implementation) Check to see if the accelerator region and data motion associated the handle have completed. If complete return true, else return false. S–3901–80 159Cray Fortran Reference Manual 6.4.5 OpenMP Acclerator Environment Variables CRAY_ACC_DEBUG Valid output levels range from 0, which indicates no output, through 3, which indicates verbose. The compiler support library will print information about data being transferred. For parallel codes, every accelerator action on every core may generate output. If STDERR is piped to file, this file can be very large. Default: 0 6.4.6 OpenMP Accelerator Module Support The craype-accel-nvidia20 module sets the necessary compiler options and targets. To compile, ensure that PrgEnv-cray module is loaded and that it includes CCE 8.0. Also, load the craype-accel-nvidia20 module. Use either the ftn or cc command to compile. 6.5 Optimizations A certain amount of overhead is associated with multiprocessing a loop. If the work occurring in the loop is small, the loop can actually run slower by multiprocessing than by single processing. To avoid this, make the amount of work inside the multiprocessed region as large as possible, as is shown in the following examples. Consider the following code: DO K = 1, N DO I = 1, N DO J = 1, N A(I,J) = A(I,J) + B(I,K) * C(K,J) END DO END DO END DO For the preceding code fragment, you can parallelize the J loop or the I loop. You cannot parallelize the K loop because different iterations of the K loop read and write the same values of A(I,J). Try to parallelize the outermost DO loop if possible, because it encloses the most work. In this example, that is the I loop. For this example, use the technique called loop interchange. Although the parallelizable loops are not the outermost ones, you can reorder the loops to make one of them outermost. 160 S–3901–80Using the OpenMP Fortran API [6] Thus, loop interchange would produce the following code fragment: !$OMP PARALLEL DO PRIVATE(I, J, K) DO I = 1, N DO K = 1, N DO J = 1, N A(I,J) = A(I,J) + B(I,K) * C(K,J) END DO END DO END DO Now the parallelizable loop encloses more work and shows better performance. In practice, relatively few loops can be reordered in this way. However, it does occasionally happen that several loops in a nest of loops are candidates for parallelization. In such a case, it is usually best to parallelize the outermost one. Occasionally, the only loop available to be parallelized has a fairly small amount of work. It may be worthwhile to force certain loops to run without parallelism or to select between a parallel version and a serial version, on the basis of the length of the loop. The loop is worth parallelizing if N is sufficiently large. To overcome the parallel loop overhead, N needs to be around 1000, depending on the specific hardware and the context of the program. The optimized version would use an IF clause on the PARALLEL DO directive: !$OMP PARALLEL DO IF (N .GE. 1000), PRIVATE(I) DO I = 1, N A(I) = A(I) + X*B(I) END DO 6.6 Limitations The following known limitations affect OpenMP on Cray systems. • Orphaned task constructs may have an implicit taskwait directive added to the end of the routine. This is not required by the specification but is currently required by the Cray implementation. This limits the amount of parallelism that may be seen. This limitation will be removed in a future release. • Task switching is not implemented. The thread that starts executing a task will be the thread that finishes the task. Task switching will be implemented in a future release. • The workshare constructs are only partially optimized. The current implementation workshares parallel work it discovers inside the workshare construct. However, there may be more synchronization than strictly required at this time. This limitation will be addressed in a future release. S–3901–80 161Cray Fortran Reference Manual 6.7 Compiler Options These Cray Fortran compiler options affect OpenMP directives and usage. -h [no]omp Enables or disables compiler recognition of OpenMP directives. By default, OpenMP is enabled. This option is identical to the -O [no]omp option and is provided for command-line compatibility with the Cray C/C++ compiler. For more information, see -h [no]omp on page 45. -h [no]omp_acc Enables or disables compiler recognition of OpenMP accelerator directives. See OpenMP Accelerator Directives on page 153. -h [no]omp_trace Enables or disables the insertion of CrayPat OpenMP tracing calls. By default tracing is off. For more information, see -h [no]omp_trace on page 45. -O [no]omp This option is identical to -h [no]omp. -O [no]omp_acc This option is identical to -h [no]omp_acc. -h threadn This option controls both OpenMP and autothreading. If n is 0, both OpenMP and autothreading are disabled. For n 1 through 3, other behaviors are specified. This option is identical to -O threadn and is provided for command-line compatibility with the Cray C/C++ compiler. For more information, see -O threadn on page 68. -O threadn This option is identical to -h threadn. -x dirlist This option can be used to disable specified directives or classes of directives, including OpenMP directives, and OpenACC directives. For more information, see -x dirlist on page 83. 6.8 aprun Options The -d depth option of the aprun command is required to reserve more than one physical processor for an OpenMP process. For best performance, depth should be the same as the maximum number of threads the program uses. The maximum number of threads per compute node is 4 times the number of allocated processors. This example shows how to reserve the physical processors: aprun -d depth ompProgram 162 S–3901–80Using the OpenMP Fortran API [6] If neither the OMP_NUM_THREADS environment variable nor the omp_set_num_threads() call is used to set the number of OpenMP threads, the system defaults to 1 thread. The aprun options -n processes and -N processes_per_node are compatible with OpenMP but do not directly affect the execution of OpenMP programs. S–3901–80 163Cray Fortran Reference Manual 164 S–3901–80Using OpenACC [7] OpenACC is a parallel programming model which facilitates the use of an accelerator device attached to a host CPU. The OpenACC API allows the programmer to supplement information available to the compilers in order to offload code from a host CPU to an attached accelerator device. This release partially supports the OpenACC Application Programming Interface, Version 1.0 standard developed by PGI, Cray Inc., and NVIDIA, with support from CAPS entreprise. Refer to the OpenACC home page at http://www.openacc-standard.org/. Under the Downloads link, select the OpenACC 1.0 Specification. Note that the Cray implementation of this API is subject to change in future releases. See the intro_openacc(7) man page for the most current information regarding this implementation of OpenACC. 7.1 OpenACC Execution Model The host offloads compute intensive regions to the accelerator device. The accelerator executes parallel regions. The host manages execution on the accelerator by allocating memory on the accelerator, initiating data transfer, sending code, passing arguments to the region, waiting for completion, transferring results back to the host and releasing memory. The accelerator on the Cray XK supports three levels of parallelism. 7.2 OpenACC Memory Model The memory on the accelerator is separate from host memory. Accelerator device memory is not mapped onto the host's virtual memory space. All data movement between host and accelerator memory is initiated by the host through the library functions that move data. Also, it is not assumed that the accelerator can access host memory, though it is supported by some devices. In this model, data movement between memories is managed by the compiler according to OpenACC directives. The programmer needs to be aware of device memory size, as well as memory bandwidth between host and device in order to effectively accelerate a region of code. S–3901–80 165Cray Fortran Reference Manual Current GPUs implement a weak memory model; they do not support memory coherence between operations executed by different execution units. Execution unit describes an entity that executes a gang or worker. If an operation updates a memory location and another reads from the same location, or two operations store a value to the same location, the hardware may not guarantee repeatable results. Some potential errors of this type are prevented by the compiler, but it is possible to write an accelerator parallel region that produces inconsistent results. Memory coherence is guaranteed when memory operations are separated by an explicit barrier. 7.3 OpenACC Directives By default, OpenACC directives (!$acc sentinel) are recognized by the compiler. Recognition of these directives can be controlled from the command line. See -h [no]acc on page 41 and -x dirlist on page 83. 7.3.1 parallel When the compiler encounters a parallel directive, it creates a parallel region to be executed on an attached accelerator. Gangs of workers are created to execute the parallel region on the accelerator. Once the gangs are created, the number of gangs and workers in each gang remain constant for the duration of the parallel region. One worker in each gang begins executing the code in the structured block of the construct. Each gang may execute a different path of statements. There is an implicit barrier at the end of the accelerator parallel region; the program running on the host will wait for gangs to complete execution before continuing. Restrictions: Parallel regions may not contain other parallel regions. A program should not branch into or out of the construct. A program should not depend on any ordering of the evaluations of the clauses. A program should not depend on any side effects of the evaluations of the clauses. At most, one if clause may appear. !$acc parallel [clause [[,] clause]...] Structured-block !$acc end parallel Clauses: if(condition) (Deferred implementation) If the clause evaluates to .true., the region runs on the accelerator. If the clause evaluates to .false. , the region is run on the host. async[ (scalar-integer-expression ) ] (Deferred implementation) 166 S–3901–80Using OpenACC [7] num_gangs(scalar-integer-expression) Defines the number of parallel gangs that will execute the region. num_workers( scalar-integer-expression ) Defines the number of workers within each gang that will execute the region and scalar-integer-expression must be 1, 2, 4, 8, 16 or 32. If the user specifies num_workers, vector_length can only be 32. If the user does not specify num_workers, then vector_length can be 1, 32, 64, 128, 256, 512 or 1024. vector_length( scalar-integer-expression ) integer-expression defines the vector length to use for vector or SIMD operations within each worker of the gang. vector_length is used for loops annotated with the vector clause on a loop directive, and for loop automatically vectorized by the compiler within the parallel region. If the user specifies num_workers, then vector_length can only be 32. If the user does not specify num_workers, then vector_length can be 1, 32, 64, 128, 256, 512 or 1024. reduction(operator:list) The reduction clause specifies a reduction operator and one or more scalar variables. For each variable, a private copy is created for each parallel gang and initialized for that operator. At the end of the region, the values for each gang are combined using the reduction operator, and the result combined with the value of the original variable and stored in the original variable. The reduction result is available after the region. The following table lists the operators that are valid and the initialization values; in each case, the initialization value will be cast into the variable type. For max and min reductions, the initialization values are the least representable value and the largest representable value for the variable’s data type, respectively. Supported data types are the numerical data types in C and C++ (int, float, double, complex) and Fortran (integer, real, double precision, complex). C, C++ Fortran operator init value operator init value + 0 + 0 * 1 * 1 max least max least S–3901–80 167Cray Fortran Reference Manual C, C++ Fortran operator init value operator init value min largest min largest & ~0 iand all bits on | 0 ior 0 ^ 0 ieor 0 && 1 .and. .true. || 0 .or. .false. .eqv .true. .neqv .false. copy(list) See data clauses available with the data directive. See data on page 171. copyin(list) See data clauses available with the data directive. See data on page 171. copyout(list) See data clauses available with the data directive. See data on page 171. create(list) See data clauses available with the data directive. See data on page 171. present(list) See data clauses available with the data directive. See data on page 171. present_or_copy(list) See data clauses available with the data directive. See data on page 171. present_or_copyin(list) See data clauses available with the data directive. See data on page 171. 168 S–3901–80Using OpenACC [7] present_or_copyout(list) See data clauses available with the data directive. See data on page 171. present_or_create(list) See data clauses available with the data directive. See data on page 171. deviceptr(list) See data clauses available with the data directive. See data on page 171. private(list) A copy of each item on the list will be created for each parallel gang. firstprivate(list) The private versions of the items on the list will be initialized to the state of the associated item on the host. 7.3.2 loop The loop directive applies to the loop that immediately follows this directive. The directive can describe what type of parallelism to use to execute the loop and declare loop-private variables and arrays and reduction operations. !$acc loop [clause [[,] clause]...] Do-loop-nest !$acc end loop S–3901–80 169Cray Fortran Reference Manual Clauses: collapse(scalar-integer-expression) Specifies that scalar-integer-expression nested loops are associated with the loop construct. scalar-integer-expression must be a constant positive integer expression. If more than one loop is associated with the loop construct, the iterations of all the associated loops are all scheduled according to the rest of the clauses. The trip count for all loops associated with the collapse clause must be computable and invariant in all the loops. A gang, worker, or vector clause on the directive may be applied to each loop or to the linearized iteration space as determined by the compiler. If no collapse clause is present, only the immediately following loop is associated with the loop directive. gang Specifies that the iterations of the associated loop or loops are to be executed in parallel by distributing the iterations among the gangs. The loop iterations must be data independent, except for variables specified in a reduction clause. worker Specifies that the iterations of the associated loop or loops are to be executed in parallel by distributing the iterations among the multiple workers within a single gang. The loop iterations must be data independent, except for variables specified in a reduction clause. It is implementation-defined whether a loop with the worker clause may contain a loop containing the gang clause. seq Specifies that the associated loop or loops are to be executed sequentially by the accelerator. Overrides any automatic compiler parallelization or vectorization. Default in context of a parallel region. vector Specifies that the iterations of the associated loop/s are to be executed in vector or SIMD mode. The operations will execute using vectors of the length specified or chosen for the parallel region. A loop with the vector clause may not contain a loop containing the gang or worker clause. private(list) A copy of each item on the list will be created for each iteration of the associated loop(s). reduction(operator:list) See parallel on page 166. 170 S–3901–80Using OpenACC [7] 7.3.3 parallel_loop The parallel_loop construct is a combined construct consisting of an parallel region and an accelerator loop which contains a single loop nest. This is identical to explicitly specifying a parallel directive containing a loop directive. It is currently supported in Fortran only. !$acc region_loop [clause [[,] clause]...] Do-loop-nest !$acc end region_loop Uses the combined set of clauses available with the parallel and loop directives. 7.3.4 data Starts an accelerator data region and defines scalars, arrays and subarrays to be allocated in the accelerator memory for the duration of the region. The compiler allocates a copy of the variable in accelerator memory. A program that uses data on the host and assigns the same data on the accelerator within a data region must manage the coherence of the two copies with update directives. Clauses determine whether data is copied from the host to the accelerator memory upon region entry, and copied from the accelerator memory to the host memory upon region exit. The list argument to each data clause is a comma-separated collection of variable names, array names, or subarray specifications. Restrictions: A program should not branch into or out of the construct. A program should not depend on any ordering of the evaluations of the clauses. A program should not depend on any side effects of the evaluations of the clauses. At most, one if clause may appear. Only symbols can appear in data clauses. If the symbol is an array, the entire array is moved. If a non-contiguous array section is specified, the smallest contiguous bounding region is moved. Data clauses may also appear on the parallel construct. !$acc data [clause [[,] clause]...] Structured-block !$acc end data Clauses: if(condition) (Deferred implementation) If the clause evaluates to .true., the program allocates memory on, and moves data to and/or from the accelerator. If the clause evaluates to .false., no memory is allocated and no data is moved. async[ (scalar-integer-expression ) ] (Deferred implementation) S–3901–80 171Cray Fortran Reference Manual copy(list) Combines the behavior of the copyin and copyout clauses. An object that appears in an copy clause shall not appear in any other data sharing clauses. copyin(list) Causes the accelerator shared objects to be initialized to the hosts memory state when the region starts. Objects in this clause may also appear in the copyout clause. copyout(list) Causes the host's memory to be updated with the accelerator's shared objects state when the accelerator region ends. Objects in this clause may also appear in the copyin clause. create(list) Declares that the variables, arrays or subarrays in the list need to be allocated in the device memory, but the values in the host memory are not used on the accelerator, and any values computed and assigned on the accelerator are not used on the host. Data in this clause is not copied between the host and device memories. present(list) Causes the system to look for the list items on the accelerator. It is an error to not find the object on the accelerator. Array sections of assumed size or allocatable arrays are not supported. The array will not be found in the present table. present_or_copy(list) Causes the system to look for the list items on the accelerator. If the object is not found, the copy clause takes effect. Array sections of assumed size or allocatable arrays are not supported. The array will not be found in the present table. present_or_copyin(list) Causes the system to look for the list items on the accelerator. If the object is not found, the copyin clause takes effect. Array sections of assumed size or allocatable arrays are not supported. The array will not be found in the present table. 172 S–3901–80Using OpenACC [7] present_or_copyout(list) Causes the system to look for the list items on the accelerator. If not found on the accelerator, the data is allocated in the accelerator memory and copied from the accelerator back to the host when the region exits. Array sections of assumed size or allocatable arrays are not supported. The array will not be found in the present table. present_or_create(list) Causes the system to look for the list items on the accelerator. If not found, the data is allocated in the accelerator. Array sections of assumed size or allocatable arrays are not supported. The array will not be found in the present table. deviceptr(list) Used to declare that the pointers in the list are device pointers, so the data need not be allocated or moved between the host and device for this pointer. The variable in list must be dummy arguments (arrays or scalars), and may not have the Fortran pointer, allocatable or value attributes. 7.3.5 host_data Makes the device address of data available in host code. !$acc host_data [clause [[,] clause]...] Structured-block !$acc end host_data Clause: use_device(list) Use the device address of any variable or array in the list in code within the construct. May be used to pass the device address of variables or arrays to optimized procedures written in a lower-level API. The variables or arrays in list must be present in the accelerator memory due to data regions that contain this construct. 7.3.6 update The update directive is used within an explicit or implicit data region to update all or part of a host memory array with values from the corresponding array in accelerator memory, or to update all or part of an accelerator memory object with values from the corresponding object in host memory. S–3901–80 173Cray Fortran Reference Manual The host process will wait until the updates are complete before executing any of the code that follows the update directive. This directive may not appear inside of a parallel region. The list items on the accelerator shall be visible in accelerator shared memory. This directive is executable. It shall not appear in place of the statement following an if, while, do, switch, or label in C, or in place of the statement following a logical if in Fortran. !$acc update [clause [[,] clause]...] Clauses: host( list ) Specifies that the variables, arrays or subarrays in the list are to be copied from the accelerator device memory to the host memory. device( list ) Specifies that the variables, arrays or subarrays in the list are to be copied from the host memory to the accelerator device memory. if(condition) (Deferred implementation) If the condition evaluates to .true., the program moves data to/from the accelerator. If the clause evaluates to .false., no data is moved. async[ ( scalar-integer-expression ) ] (Deferred implementation) 7.4 OpenACC Runtime Routines The current release does not support OpenACC runtime routines. 7.5 OpenACC Environment Variables The current release does not support OpenACC environment variables. 7.6 OpenACC Compiler Options These Fortran compiler options affect OpenACC directives. -h [no]acc Enables or disables compiler recognition of OpenACC accelerator directives. By default, OpenACC is enabled. -x acc Ignore OpenACC accelerator directives. By default, OpenACC directives are recognized. 174 S–3901–80Cray Fortran Defined Externals [8] 8.1 Conformance Checks The amount of error-checking of edit descriptors with input/output (I/O) list items during formatted READ and WRITE statements can be selected through a compiler driver option or through an environment variable. By default, the compiler provides only limited error-checking. Use the compiler driver options to choose the table to be used for the conformance check. The table is then part of the executable and no environment variable is required. The compiler driver options allow a choice of checking or no checking with a particular version of the Fortran standard for formatted READ and WRITE. See the following tables: Table 16, Table 17, Table 18, and Table 19. The environment variable FORMAT_TYPE_CHECKING is evaluated during execution. The environment variable overrides a table chosen through the compiler driver option. The environment variable provides an intermediate type of checking that is not provided by the compiler driver option. The environment variable FORMAT_TYPE_CHECKING is described in Interaction of Directives with the -x Command Line Option on page 100. To select the least amount of checking, use one or more of the following ftn command line options. • On Cray Linux Environment (CLE) systems with formatted READ, use: ftn -W1,--defsym,_RCHK=_RNOCHK *.f(note the double dashes that precede defsym) • On CLE systems with formatted WRITE, use: ftn -W1,--defsym,_WCHK=_WNOCHK *.f • On CLE systems with both formatted READ and WRITE, use: ftn -W1,--defsym,_WCHK=_WNOCHK -W1,--defsym,_RCHK=_RNOCHK *.f S–3901–80 175Cray Fortran Reference Manual To select strict amount of checking for either FORTRAN 77 or Fortran 90, use one or more of the following ftn command line options. • On CLE systems with formatted READ, use: ftn -W1,--defsym,_RCHK=_RCHK77 *.f ftn -W1,--defsym,_RCHK=_RCHK90 *.f • On CLE systems with formatted WRITE, use: ftn -W1,--defsym,_WCHK=_WCHK77 *.f ftn -W1,--defsym,_WCHK=_WCHK90 *.f • On CLE systems with both formatted READ and WRITE, use: ftn -W1,--defsym,_WCHK=_WCHK77 -W1,--defsym,_RCHK=_RCHK77 *.f ftn -W1,--defsym,_WCHK=_WCHK90 -W1,--defsym,_RCHK=_RCHK90 *.f 176 S–3901–80Cray Fortran Language Extensions [9] The Cray Fortran Compiler supports extended features beyond those specified by the current standard. Some of these extensions are widely implemented in other compilers and likely to become standard features in the future, while others are unique and specific to Cray systems. The implementation of any extension may change in order to conform to future language standards. For information about obsolete features, see Obsolete Features (Chapter 10, Obsolete Features on page 207). The listings provided by the compiler identify language extensions when the -e n command line option is specified. 9.1 128-Bit Precision The Fortran Compiler supports 128-bit floating point and 256-bit complex predefined types using the X86-64 ABI definitions for type names and data layout. These types are sometimes referred to as "quad-precision". In Fortran, use real(kind=16) and complex(kind=16) to declare variables of these types. In C and C++, use __float128, and __float128 complex. Fortran and C forms of intrinsic math functions (for example, QSIN, QCOS, QTAN, QSQRT, sinq, cosq, tanq) offer full support for quad-precision types. See the intro_quad_precision(3i) man page for a complete list of intrinsic functions that support quad-precision. The base type itself uses 128 bits of storage with a guaranteed minimum alignment on a 128-bit boundary, little endian, has a 15-bit exponent, a 113-bit mantissa, and an exponent bias of 16383, and is compatible with the gcc implementation. S–3901–80 177Cray Fortran Reference Manual 9.2 Characters, Lexical Tokens, and Source Form 9.2.1 Characters Allowed in Names Variables, named constants, program units, common blocks, procedures, arguments, constructs, derived types (types for structures), namelist groups, structure components, dummy arguments, and function results are among the elements in a program that have a name. As extensions, the Cray Fortran compiler permits the following characters in names: alphanumeric_character is currency_symbol currency_symbol is $ A name must begin with a letter and can consist of letters, digits, and underscores. The Cray Fortran compiler permits you to use the dollar sign ($) in a name, but it cannot be the first character of a name. Cray does not recommend using $ in user names because it can cause conflicts with the names of internal variables or library routines. 9.2.2 Switching Source Forms The Cray Fortran compiler allows you to switch between fixed and free source forms within a source or include file by using the FIXED and FREE compiler directives. 9.2.3 Continuation Line Limit The Cray Fortran compiler allows a statement to have an unlimited number of continuation lines. The Fortran standard allows only 255 continuation lines. 9.2.4 D Lines in Fixed Source Form The Cray Fortran compiler allows a D or d character to occur in column one in fixed source form. Typically, the compiler treats a line with a D or d character in column one as a comment line. When the -e d command line option is in effect, however, the compiler replaces the D or d character with a blank and treats the rest of the line as a source statement. This can be used, for example, for debugging purposes if the rest of the line contains a PRINT statement. This functionality is controlled through the -e d and -d d options on the compiler command line. For more information about these options, see the ftn(1) man page. 178 S–3901–80Cray Fortran Language Extensions [9] 9.3 Types The Cray Fortran compiler supports the following additional data types. This preserves compatibility with other vendor's systems. • Cray pointer • Cray character pointer • Boolean (or typeless) The Cray Fortran compiler also supports the TYPEALIAS statement as a means of creating alternate names for existing types and supports an expanded form of the ENUM statement. 9.3.1 Alternate Form of LOGICAL Constants The Cray Fortran compiler accepts .T. and .F. as alternate forms of .true. and .false., respectively. 9.3.2 Cray Pointer Type The Cray POINTER statement declares one variable to be a Cray pointer (that is, to have the Cray pointer data type) and another variable to be its pointee. The value of the Cray pointer is the address of the pointee. This POINTER statement has the following format: POINTER (pointer_name, pointee_name [ (array_spec) ]) [, (pointer_name, pointee_name [ (array_spec) ]) ] ... pointer_name Pointer to the corresponding pointee_name. pointer_name contains the address of pointee_name. Only a scalar variable can be declared type Cray pointer; constants, arrays, statement functions, and external functions cannot. pointee_name Pointee of corresponding pointer_name. Must be a variable name, array declarator, or array name. The value of pointer_name is used as the address for any reference to pointee_name; therefore, pointee_name is not assigned storage. If pointee_name is an array declarator, it can be explicit-shape (with either constant or nonconstant bounds) or assumed-size. array_spec If present, this must be either an explicit_shape_spec_list, with either constant or nonconstant bounds) or an assumed_size_spec. Fortran pointers are declared as follows: POINTER :: [ object-name-list ] S–3901–80 179Cray Fortran Reference Manual Cray Fortran pointers and Fortran standard pointers cannot be mixed. Example: POINTER(P,B),(Q,C) This statement declares Cray pointer P and its pointee B, and Cray pointer Q and pointee C; the pointer's current value is used as the address of the pointee whenever the pointee is referenced. An array that is named as a pointee in a Cray POINTER statement is a pointee array. Its array declarator can appear in a separate type or DIMENSION statement or in the pointer list itself. In a subprogram, the dimension declarator can contain references to variables in a common block or to dummy arguments. As with nonconstant bound array arguments to subprograms, the size of each dimension is evaluated on entrance to the subprogram, not when the pointee is referenced. For example: POINTER(IX, X(N,0:M)) In addition, pointees must not be deferred-shape or assumed-shape arrays. An assumed-size pointee array is not allowed in a main program unit. You can use pointers to access user-managed storage by dynamically associating variables and arrays to particular locations in a block of storage. Cray pointers do not provide convenient manipulation of linked lists because, for optimization purposes, it is assumed that no two pointers have the same value. Cray pointers also allow the accessing of absolute memory locations. The range of a Cray pointer or Cray character pointer depends on the size of memory for the machine in use. Restrictions on Cray pointers are as follows: • A Cray pointer variable should only be used to alias memory locations by using the LOC intrinsic. • A Cray pointer cannot be pointed to by another Cray or Fortran pointer; that is, a Cray pointer cannot also be a pointee or a target. • A Cray pointer cannot appear in a PARAMETER statement or in a type declaration statement that includes the PARAMETER attribute. • A Cray pointer variable cannot be declared to be of any other data type. • A Cray character pointer cannot appear in a DATA statement. • An array of Cray pointers is not allowed. • A Cray pointer cannot be a component of a structure. 180 S–3901–80Cray Fortran Language Extensions [9] Restrictions on Cray pointees are as follows: • A Cray pointee cannot appear in a SAVE, STATIC, DATA, EQUIVALENCE, COMMON, AUTOMATIC, or PARAMETER statement or Fortran pointer statement. • A Cray pointee cannot be a dummy argument; that is, it cannot appear in a FUNCTION, SUBROUTINE, or ENTRY statement. • A function value cannot be a Cray pointee. • A Cray pointee cannot be a structure component. • An equivalence object cannot be a Cray pointee. Note: Cray pointees can be of type character, but their Cray pointers are different from other Cray pointers; the two kinds cannot be mixed in the same expression. The Cray pointer is a variable of type Cray pointer and can appear in a COMMON list or be a dummy argument in a subprogram. The Cray pointee does not have an address until the value of the Cray pointer is defined; the pointee is stored starting at the location specified by the pointer. Any change in the value of a Cray pointer causes subsequent references to the corresponding pointee to refer to the new location. Cray pointers can be assigned values in the following ways: • A Cray pointer can be set as an absolute address. For example: Q = 0 • Cray pointers can have integer expressions added to or subtracted from them and can be assigned to or from integer variables. For example: P = Q + 100 However, Cray pointers are not integers. For example, assigning a Cray pointer to a real variable is not allowed. The (nonstandard) LOC intrinsic function generates the address of a variable and can be used to define a Cray pointer, as follows: P = LOC(X) S–3901–80 181Cray Fortran Reference Manual The following example uses Cray pointers in the ways just described: SUBROUTINE SUB(N) INTEGER WORDS COMMON POOL(100000), WORDS(1000) INTEGER BLK(128), WORD64 REAL A(1000), B(N), C(100000-N-1000) POINTER(PBLK,BLK), (IA,A), (IB,B), & (IC,C), (ADDRESS,WORD64) ADDRESS = LOC(WORDS) + 64*KIND(WORDS) PBLK = LOC(WORDS) IA = LOC(POOL) IB = IA + 1000*KIND(POOL) IC = IB + N*KIND(POOL) BLK is an array that is another name for the first 128 words of array WORDS. A is an array of length 1000; it is another name for the first 1000 elements of POOL. B follows A and is of length N. C follows B. A, B, and C are associated with POOL. WORD64 is the same as BLK(65) because BLK(1) is at the initial address of WORDS. If a pointee is of a noncharacter data type that is one machine word or longer, the address stored in a pointer is a word address. If the pointee is of type character or of a data type that is less than one word, the address is a byte address. The following example also uses Cray pointers: PROGRAM TEST REAL X(*), Y(*), Z(*), A(10) POINTER (P_X,X) POINTER (P_Y,Y) POINTER (P_Z,Z) INTEGER*8 I,J !USE LOC INTRINSIC TO SET POINTER MEMORY LOCATIONS !*** RECOMMENDED USAGE, AS PORTABLE CRAY POINTERS *** P_X = LOC(A(1)) P_Y = LOC(A(2)) !USE POINTER ARITHMETIC TO DEMONSTRATE COMPILER AND COMPILER !FLAG DIFFERENCES !*** USAGE NOT RECOMMENDED, HIGHLY NON-PORTABLE *** P_Z = P_X + 1 I = P_Y J = P_Z IF ( I .EQ. J ) THEN PRINT *, 'NOT A BYTE-ADDRESSABLE MACHINE' ELSE PRINT *, 'BYTE-ADDRESSABLE MACHINE' ENDIF END 182 S–3901–80Cray Fortran Language Extensions [9] On Cray systems, this prints the following: Byte-addressable machine Note: Cray does not recommend the use of pointer arithmetic because it is not portable. For purposes of optimization, the compiler assumes that the storage of a pointee is never overlaid on the storage of another variable; that is, it assumes that a pointee is not associated with another variable or array. This kind of association occurs when a Cray pointer has two pointees, or when two Cray pointers are given the same value. Although these practices are sometimes used deliberately (such as for equivalencing arrays), results can differ depending on whether optimization is turned on or off. You are responsible for preventing such association. For example: POINTER(P,B), (P,C) REAL X, B, C P = LOC(X) B = 1.0 C = 2.0 PRINT *, B Because B and C have the same pointer, the assignment of 2.0 to C gives the same value to B; therefore, B will print as 2.0 even though it was assigned 1.0. As with a variable in common storage, a pointee, pointer, or argument to a LOC intrinsic function is stored in memory before a call to an external procedure and is read out of memory at its next reference. The variable is also stored before a RETURN or END statement of a subprogram. 9.3.3 Cray Character Pointer Type If a pointee is declared as a character type, its Cray pointer is a Cray character pointer. Restrictions for Cray pointers also apply to Cray character pointers. In addition, the following restrictions apply: • When included in an I/O statement iolist, a Cray character pointer is treated as an integer. • If the length of the pointee is explicitly declared (that is, not of an assumed length), any reference to that pointee uses the explicitly declared length. • If a pointee is declared with an assumed length (that is, as CHARACTER(*)), the length of the pointee comes from the associated Cray character pointer. • A Cray character pointer can be used in a relational operation only with another Cray character pointer. Such an operation applies only to the character address and bit offset; the length field is not used. S–3901–80 183Cray Fortran Reference Manual 9.3.4 Boolean Type A Boolean constant represents the literal constant of a single storage unit. There are no Boolean variables or arrays, and there is no Boolean type statement. Binary, octal, and hexadecimal constants are used to represent Boolean values. For more information about Boolean expressions, see Expressions on page 188. 9.3.5 Alternate Form of ENUM Statement An enumeration defines the name of a group of related values and the name of each value within the group. The Cray Fortran compiler allows the following additional form for enum_def (enumerations): enum_def_stmt is ENUM, [,BIND(C)] [[::] type_alias_name] or ENUM [ kind_selector ] [[ :: ] type_alias_name] • kind_selector. If it is not specified, the compiler uses the default integer kind. • type_alias_name is the name you assign to the group. This name is treated as a type alias name. 9.3.6 TYPEALIAS Statement A TYPEALIAS statement allows you to define another name for an intrinsic data type or user-defined data type. Thus, the type alias and the type specification it aliases are interchangeable. Type aliases do not define a new type. This is the form for type aliases: type_alias_stmt is TYPEALIAS :: type_alias_list type_alias is type_alias_name => type_spec This example shows how a type alias can define another name for an intrinsic type, a user-defined type, and another type alias: TYPEALIAS :: INTEGER_64 => INTEGER(KIND = 8), & TYPE_ALIAS => TYPE(USER_DERIVED_TYPE), & ALIAS_OF_TYPE_ALIAS => TYPE(TYPE_ALIAS) INTEGER(KIND = 8) :: I TYPE(INTEGER_64) :: X, Y TYPE(TYPE_ALIAS) :: S TYPE(ALIAS_OF_TYPE_ALIAS) :: T 184 S–3901–80Cray Fortran Language Extensions [9] You can use a type alias or the data type it aliases interchangeably. That is, explicit or implicit declarations that use a type alias have the same effect as if the data type being aliased was used. For example, the above declarations of I, X, and Y are the same. Also, S and T are the same. If the type being aliased is a derived type, the type alias name can be used to declare a structure constructor for the type. The following are allowed as the type_spec in a TYPEALIAS statement: • Any intrinsic type defined by the Cray Fortran compiler. • Any type alias in the same scoping unit. • Any derived type in the same scoping unit. 9.4 Data Object Declarations and Specifications The Cray Fortran compiler accepts the following extensions to declarations. 9.4.1 Attribute Specification Statements 9.4.1.1 BOZ Constants in DATA Statements The Cray Fortran compiler permits a default real object to be initialized with a BOZ, typeless, or character (used as Hollerith) constant in a DATA statement. BOZ constants are formatted in binary, octal, or hexadecimal. No conversion of the BOZ value, typeless value, or character constant takes place. The Cray Fortran compiler permits an integer object to be initialized with a BOZ, typeless, or character (used as Hollerith) constant in a type declaration statement. The Cray Fortran compiler also allows an integer object to be initialized with a typeless or character (used as Hollerith) constant in a DATA statement. If the last item in the data_object_list is an array name, the value list can contain fewer values than the number of elements in the array. Any element that is not assigned a value is undefined. S–3901–80 185Cray Fortran Reference Manual The following alternate forms of BOZ constants are supported. literal-constant is typeless-constant typeless-constant is octal-typeless-constant octal-typeless-constant is digit [ digit... ] B or " digit [ digit... ] "O or ' digit [ digit... ] 'O hexadecimal-typeless-constant is X' hex-digit [ hex-digit... ]' or X" hex-digit [ hex-digit... ] " or ' hex-digit [ hex-digit... ] 'X or " hex-digit [ hex-digit... ] "X 9.4.1.2 AUTOMATIC Attribute and Statement The Cray Fortran AUTOMATIC attribute specifies stack-based storage for a variable or array. Such variables and arrays are undefined upon entering and exiting the procedure. The following is the format for the AUTOMATIC specification: type, AUTOMATIC [ , attribute-list ] :: entity-list automatic-stmt is AUTOMATIC [[::] ]entity-list entity-list For entity-list, specify a variable name or an array declarator. If an entity-list item is an array, it must be declared with an explicit-shape-spec with constant bounds. If an entity-list item is a pointer, it must be declared with a deferred-shape-spec. If an entity-list item has the same name as the function in which it is declared, the entity-list item must be scalar and of type integer, real, logical, complex, or double precision. If the entity-list item is a pointer, the AUTOMATIC attribute applies to the pointer itself and not to any target that may become associated with the pointer. 186 S–3901–80Cray Fortran Language Extensions [9] Subject to the rules governing combinations of attributes, attribute-list can contain the following: DIMENSION TARGET POINTER VOLATILE The following entities cannot have the AUTOMATIC attribute: • Pointers or arrays used as function results • Dummy arguments • Statement functions • Automatic array or character data objects An entity-list item cannot have the following characteristics: • It cannot be defined in the scoping unit of a module. • It cannot be a common block item. • It cannot be specified more than once within the same scoping unit. • It cannot be initialized with a DATA statement or with a type declaration statement. • It cannot also have the SAVE or STATIC attribute. • It cannot be specified as a Cray pointee. 9.4.2 IMPLICIT Statement 9.4.2.1 IMPLICIT Extensions The Cray Fortran compiler accepts the IMPLICIT AUTOMATIC or IMPLICIT STATIC syntax. It is recommended that none of the IMPLICIT extensions be used in new code. 9.4.3 Storage Association of Data Objects 9.4.3.1 EQUIVALENCE Statement Extensions The Cray Fortran compiler allows equivalencing of character data with noncharacter data. The Fortran standard does not address this. It is recommended that you do not perform equivalencing in this manner, however, because alignment and padding differs across platforms, thus rendering your code less portable. S–3901–80 187Cray Fortran Reference Manual 9.4.3.2 COMMON Statement Extensions The Cray Fortran compiler treats named common blocks and blank common blocks identically, as follows: • Variables in blank common and variables in named common blocks can be initialized. • Named common blocks and blank common are always saved. • Named common blocks of the same name and blank common can be of different sizes in different scoping units. 9.5 Expressions and Assignment 9.5.1 Expressions In Fortran, calculations are specified by writing expressions. Expressions look much like algebraic formulas in mathematics, particularly when the expressions involve calculations on numerical values. Expressions often involve nonnumeric values, such as character strings, logical values, or structures; these also can be considered to be formulas that involve nonnumeric quantities rather than numeric ones. 9.5.1.1 Rules for Forming Expressions The Cray Fortran compiler supports exclusive disjunct expressions of the form: exclusive-disjunct-expr is [ exclusive-disjunct-expr .XOR. ] inclusive-disjunct-expr 9.5.1.2 Intrinsic and Defined Operations Cray supports the following intrinsic operators as extensions: less_greater_op is .LG. or <> not_op is .N. and_op is .A. or_op is .O. exclusive_disjunct_op is .XOR. or .X. 188 S–3901–80Cray Fortran Language Extensions [9] The Cray Fortran less than or greater than intrinsic operation is represented by the <> operator and the .LG. keyword. This operation is suggested by the IEEE standard for floating-point arithmetic, and the Cray Fortran compiler supports this operator. Only values of type real can appear on either side of the <> or .LG. operators. If the operands are not of the same kind type value, the compiler converts them to equivalent kind types. The <> and .LG. operators perform a less-than-or-greater-than operation as specified in the IEEE standard for floating-point arithmetic. The Cray Fortran compiler allows abbreviations for the logical and masking operators. The abbreviations .A., .O., .N., and .X. are synonyms for .AND., .OR., .NOT., and .XOR., respectively. The masking of Boolean operators and their abbreviations, which are extensions to Fortran, can be redefined as defined operators. If you redefine a masking operator, your definition overrides the intrinsic masking operator definition. See Table 10, for a list of the operators. 9.5.1.3 Intrinsic Operations In the following table, the symbols I, R, Z, C, L, B, and P stand for the types integer, real, complex, character, logical, Boolean, and Cray pointer, respectively. Where more than one type for x 2 is given, the type of the result of the operation is given in the same relative position in the next column. Boolean and Cray pointer types are extensions of the Fortran standard. Table 9. Operand Types and Results for Intrinsic Operations Intrinsic operator Type of x 1 Type of x 2 Type of result Unary +, - I, R, Z, B, P I, R, Z, I, P Binary +, -, *, /, ** I I, R, Z, B, P I, R, Z, I, P R I, R, Z, B R, R, Z, R Z I, R, Z Z, Z, Z B I, R, B, P I, R, B, P P I , B, P P, P, P (For Cray pointer, only + and - are allowed.) // C C C .EQ., ==, .NE., /= I I, R, Z, B, P L, L, L, L, L R I, R, Z, B, P L, L, L, L, L Z I, R, Z, B, P L, L, L, L, L S–3901–80 189Cray Fortran Reference Manual Intrinsic operator Type of x 1 Type of x 2 Type of result B I, R, Z, B, P L, L, L, L, L P I, R, Z, B, P L, L, L, L, L C C L .GT., >, .GE., >=, .LT., <, .LE., <= I I, R, B, P L, L, L, L R I, R, B L, L, L C C L P I, P L, L .LG., <> R R L .NOT. L L I, R, B B .AND., .OR., .EQV., .NEQV., .XOR. L L L I, R, B I, R, B B The operators .NOT., .AND., .OR., .EQV., and .XOR. can also be used in the Cray Fortran compiler's bitwise masking expressions; these are extensions to the Fortran standard. The result is Boolean (or typeless) and has no kind type parameters. 9.5.1.4 Bitwise Logical Expressions A bitwise logical expression (also called a masking expression) is an expression in which a logical operator operates on individual bits within integer, real, Cray pointer, or Boolean operands, giving a result of type Boolean. Each operand is treated as a single storage unit. The result is a single storage unit, which is either 32 or 64 bits depending on the -s option specified during compilation. Boolean values and bitwise logical expressions use the same operators but are different from logical values and expressions. Table 10. Cray Fortran Intrinsic Bitwise Operators and the Allowed Types of their Operands Operator category Intrinsic operator Operand types Bitwise masking (Boolean) expressions .NOT., .AND., .OR., .XOR., .EQV., .NEQV. Integer, real, typeless, or Cray pointer. Bitwise logical operators can also be written as functions; for example A .AND. B can be written as IAND(A,B) and .NOT. A can be written as NOT(A). Table 11 shows which data types can be used together in bitwise logical operations. 190 S–3901–80Cray Fortran Language Extensions [9] Table 11. Data Types in Bitwise Logical Operations x 1 x 2 1 Integer Real Boolean Pointer Logical Character Integer Masking operation, Boolean result. Masking operation, Boolean result. Masking operation, Boolean result. Masking operation, Boolean result. Not valid Not valid 2 Real Masking operation, Boolean result. Masking operation, Boolean result. Masking operation, Boolean result. Masking operation, Boolean result. Not valid Not valid 2 Boolean Masking operation, Boolean result. Masking operation, Boolean result. Masking operation, Boolean result. Masking operation, Boolean result. Not valid Not valid 2 Pointer Masking operation, Boolean result. Masking operation, Boolean result. Masking operation, Boolean result. Masking operation, Boolean result. Not valid Not valid 2 Logical Not valid 2 Not valid 2 Not valid 2 Not valid 2 Logical operation logical result Not valid 2 Character Not valid 2 Not valid 2 Not valid 2 Not valid 2 Not valid Not valid 2 Bitwise logical expressions can be combined with expressions of Boolean or other types by using arithmetic, relational, and logical operators. Evaluation of an arithmetic or relational operator processes a bitwise logical expression with no type conversion. Boolean data is never automatically converted to another type. A bitwise logical expression performs the indicated logical operation separately on each bit. The interpretation of individual bits in bitwise multiplication-exprs, summation-exprs, and general expressions is the same as for logical expressions. The results of binary 1 and 0 correspond to the logical results TRUE and FALSE, respectively, in each of the bit positions. These values are summarized as follows: .NOT. 1100 1100 1100 1100 1100 =0011 .AND. 1010 .OR. 1010 .XOR. 1010 .EQV. 1010 ---- ---- ---- ---- 1000 1110 0110 1001 1 x 1 and x 2 represent operands for a logical or bitwise expression, using operators .NOT., .AND., .OR., .XOR., .NEQV., and .EQV.. 2 Indicates that if the operand is a character operand of 32 or fewer characters, the operand is treated as a Hollerith constant and is allowed. S–3901–80 191Cray Fortran Reference Manual 9.5.2 Assignment The Cray Fortran compiler supports Boolean and Cray pointer intrinsic assignments. The Cray Fortran compiler supports type Boolean or BOZ constants in assignment statements in which the variable is of type integer or real. The bits specified by the constant are moved into the variable with no type conversion. 9.6 Execution Control 9.6.1 STOP Code Extension The STOP statement terminates the program whenever and wherever it is executed. The STOP statement is defined as follows: stop-stmt is STOP [stop_code] stop-code is scalar_char_constant or digit ... The character constant or list of digits identifying the STOP statement is optional and is called a stop-code. When the stop-code is a string of digits, leading zeros are not significant; 10 and 010 are the same stop-code. The Cray Fortran compiler accepts 1 to 80 digits; the standard accepts up to 5 digits. The stop code is accessible following program termination. The Cray Fortran compiler sends it to the standard error file (stderr). The following are examples of STOP statements: STOP STOP 'Error #823' STOP 20 9.7 Input/Output Statements The Fortran standard does not specifically describe the implementation of I/O processing. This section provides information about processor-dependent areas and the implementation of the support for I/O. 192 S–3901–80Cray Fortran Language Extensions [9] 9.7.1 File Connection 9.7.1.1 OPEN Statement The OPEN statement specifies the connection properties between the file and the unit, using keyword specifiers, which are described in this section. Table 12 indicates the Cray Fortran compiler extension in an OPEN statement. Table 12. Values for Keyword Specifier Variables in an OPEN Statement Specifier Possible Values Default Value FORM= SYSTEM Unformatted with no record marks The FORM= specifier has the following format: FORM= scalar-char-expr A file opened with SYSTEM is unformatted and has no record marks. 9.8 Error, End-of-record, and End-of-file Conditions 9.8.1 End-of-file Condition and the END-specifier 9.8.1.1 Multiple End-of-file Records The file position prior to data transfer depends on the method of access: sequential or direct. Although the Fortran standard does not allow files that contain an end-of-file to be positioned after the end-of-file prior to data transfer, the Cray Fortran compiler permits more than one end-of-file for some file structures. 9.9 Input/Output Editing 9.9.1 Data Edit Descriptors 9.9.1.1 Integer Editing The Cray Fortran compiler allows w to be zero for the G edit descriptor, and it permits w to be omitted for the I, B, O, Z, or G edit descriptors. The Cray Fortran compiler allows signed binary, octal, or hexadecimal values as input. If the minimum digits (m) field is specified, the default field width is increased, if necessary, to allow for that minimum width. S–3901–80 193Cray Fortran Reference Manual 9.9.1.2 Real Editing The Cray Fortran compiler allows the use of B, O, and Z edit descriptors of REAL data items. The Cray Fortran compiler accepts the D[w.dEe] edit descriptor. The Cray Fortran compiler accepts the ZERO_WIDTH_PRECISION environment variable, which can be used to modify the default size of the width w field. This environment variable is examined only upon program startup. Changing the value of the environment variable during program execution has no effect. For more information about the ZERO_WIDTH_PRECISION environment, see ZERO_WIDTH_PRECISION on page 90. The Cray Fortran compiler allows w to be zero or omitted for the D, E, EN, ES, or G edit descriptors. The Cray Fortran compiler does not restrict the use of Ew.d and Dw.d to an exponent less than or equal to 999. The Ew.dEe form must be used. Table 13. Default Fractional and Exponent Digits Data Size and Representation w d e 4-byte (32-bit) IEEE 17 9 2 8-byte (64-bit) IEEE 26 17 3 9.9.1.3 Logical Editing The Cray Fortran compiler allows w to be zero or omitted on the L or G edit descriptors. 9.9.1.4 Character Editing The Cray Fortran compiler allows w to be zero or omitted on the G edit descriptor. 9.9.2 Control Edit Descriptors 9.9.2.1 Q Editing The Cray Fortran supports the Q edit descriptor. The Q edit descriptor is used to determine the number of characters remaining in the input record. It has the following format: Q When a Q edit descriptor is encountered during execution of an input statement, the corresponding input list item must be of type integer. Interpretation of the Q edit descriptor causes the input list item to be defined with a value that represents the number of characters remaining to be read in the formatted record. 194 S–3901–80Cray Fortran Language Extensions [9] For example, if c is the character position within the current record of the next character to be read, and the record consists of n characters, then the item is defined with the following value MAX(n-c+1,0). If no characters have yet been read, then the item is defined as n (the length of the record). If all the characters of the record have been read (c>n), then the item is defined as zero. The Q edit descriptor must not be encountered during the execution of an output statement. The following example code uses Q on input: INTEGER N CHARACTER LINE * 80 READ (*, FMT='(Q,A)') N, LINE(1:N) 9.9.3 List-directed Formatting 9.9.3.1 List-directed Input Input values are generally accepted as list-directed input if they are the same as those required for explicit formatting with an edit descriptor. The exceptions are as follows: • When the data list item is of type integer, the constant must be of a form suitable for the I edit descriptor. The Cray Fortran compiler permits binary, octal, and hexadecimal based values in a list-directed input record to correspond to I edit descriptors. 9.9.4 Namelist Formatting 9.9.4.1 Namelist Extensions The Cray Fortran compiler has extended the namelist feature. The following additional rules govern namelist processing: • An ampersand (&) or dollar sign ($) can precede the namelist group name or terminate namelist group input. If an ampersand precedes the namelist group name, either the slash (/) or the ampersand must terminate the namelist group input. If the dollar sign precedes the namelist group name, either the slash or the dollar sign must terminate the namelist group input. • Octal and hexadecimal constants are allowed as input to integer and single-precision real namelist group items. An error is generated if octal and hexadecimal constants are specified as input to character, complex, or double-precision real namelist group items. S–3901–80 195Cray Fortran Reference Manual Octal constants must be of the following form: – O"123" – O'123' – o"123" – o'123' Hexadecimal constants must be of the following form: – Z"1a3" – Z'1a3' – z"1a3" – z'1a3' 9.9.5 I/O Editing Usually, data is stored in memory as the values of variables in some binary form. On the other hand, formatted data records in a file consist of characters. Thus, when data is read from a formatted record, it must be converted from characters to the internal representation. When data is written to a formatted record, it must be converted from the internal representation into a string of characters. Table 14 and Table 15, list the control and data edit descriptor extensions supported by the Cray Fortran compiler and provide a brief description of each. Table 14. Summary of Control Edit Descriptors Descriptor Description $ or \ Suppress carriage control Table 15. Summary of Data Edit Descriptors Descriptor Description Q Return number of characters left in record For more information about the Q edit descriptor, see Q Editing on page 194. 196 S–3901–80Cray Fortran Language Extensions [9] The following tables show the use of the Cray Fortran compiler's edit descriptors with all intrinsic data types. In these tables: • NA indicates invalid usage that is not allowed. • I,O indicates that usage is allowed for both input and output. • I indicates legal usage for input only. Table 16. Default Compatibility Between I/O List Data Types and Data Edit Descriptors Data types Q Z R O L I G F ES EN E D B A Integer I I,O I,O I,O NA I,O I,O NA NA NA NA NA I,O I,O Real NA I,O I,O I,O NA NA I,O I,O I,O I,O I,O I,O I,O I,O Complex NA I,O I,O I,O NA NA I,O I,O I,O I,O I,O I,O I,O I,O Logical NA I,O I,O I,O I,O NA I,O NA NA NA NA NA I,O I,O Character NA NA NA NA NA NA I,O NA NA NA NA NA NA I,O Table 17 shows the restrictions for the various data types that are allowed when you set the FORMAT_TYPE_CHECKING environment variable to RELAXED. Not all data edit descriptors support all data sizes; for example, you cannot read/write a 16–byte real variable with an I edit descriptor. Table 17. RELAXED Compatibility Between Data Types and Data Edit Descriptors Data types Q Z R O L I G F ES EN E D B A Integer I I,O I,O I,O I,O I,O I,O I,O I,O I,O I,O NA I,O I,O Real NA I,O I,O I,O I,O I,O I,O I,O I,O I,O I,O I,O I,O I,O Complex NA I,O I,O I,O NA NA I,O I,O I,O I,O I,O I,O I,O I,O Logical NA I,O I,O I,O I,O I,O I,O I,O I,O I,O I,O NA I,O I,O Character NA NA NA NA NA NA I,O NA NA NA NA NA NA I,O Table 18 shows the restrictions for the various data types that are allowed when you set the FORMAT_TYPE_CHECKING environment variable to STRICT77. S–3901–80 197Cray Fortran Reference Manual Table 18. STRICT77 Compatibility Between Data Types and Data Edit Descriptors Data types Q Z R O L I G F ES EN E D B A Integer NA I,O NA I,O NA I,O NA NA NA NA NA NA I,O NA Real NA NA NA NA NA NA I,O I,O NA NA I,O I,O NA NA Complex NA NA NA NA NA NA I,O I,O NA NA I,O I,O NA NA Logical NA NA NA NA I,O NA NA NA NA NA NA NA NA NA Character NA NA NA NA NA NA NA NA NA NA NA NA NA I,O Table 19 shows the restrictions for the various data types that are allowed when you set the FORMAT_TYPE CHECKING environment variable to STRICT90 or STRICT95. Table 19. STRICT90 and STRICT95 Compatibility Between Data Types and Data Edit Descriptors Data types Q Z R O L I G F ES EN E D B A Integer NA I,O NA I,O NA I,O I,O NA NA NA NA NA I,O NA Real NA NA NA NA NA NA I,O I,O I,O I,O I,O I,O NA NA Complex NA NA NA NA NA NA I,O I,O I,O I,O I,O I,O NA NA Logical NA NA NA NA I,O NA I,O NA NA NA NA NA NA NA Character NA NA NA NA NA NA I,O NA NA NA NA NA NA I,O 9.10 Program Units 9.10.1 Main Program 9.10.1.1 Program Statement Extension The Cray Fortran compiler supports the use of a parenthesized list of args at the end of a program statement. The compiler ignores any args specified after program-name. 9.10.2 Block Data Program Units 9.10.2.1 Block Data Program Unit Extension The Cray Fortran compiler permits named common blocks to appear in more than one block data program unit. 198 S–3901–80Cray Fortran Language Extensions [9] 9.11 Procedures 9.11.1 Procedure Interface 9.11.1.1 Interface Duplication The Cray Fortran compiler allows you to specify an interface body for the program unit being compiled if the interface body matches the program unit definition. 9.11.2 Procedure Definition 9.11.2.1 Recursive Function Extension The Cray Fortran compiler allows direct recursion for functions that do not specify a RESULT clause on the FUNCTION statement. 9.11.2.2 Empty CONTAINS Sections The Cray Fortran compiler allows a CONTAINS statement with no internal or module procedure following. This is proposed for the 2008 Fortran standard. 9.12 Intrinsic Procedures and Modules 9.12.1 Standard Generic Intrinsic Procedures 9.12.1.1 Intrinsic Procedures The Cray Fortran compiler has implemented intrinsic procedures in addition to the ones required by the standard. These procedures have the status of intrinsic procedures, but programs that use them may not be portable. It is recommended that such procedures be declared INTRINSIC to allow other processors to diagnose whether or not they are intrinsic for those processors. The nonstandard intrinsic procedures supported by the Cray Fortran compiler are summarized in the following list. For more information about a particular procedure, see its man page. ACOSD Arccosine, value in degrees AMO_AADD Atomic memory add AMO_AADDF Atomic memory add, return new AMO_AFADD Atomic memory add, return old AMO_AAX Atomic memory AND and XOR S–3901–80 199Cray Fortran Reference Manual AMO_AFAX Atomic memory AND and XOR, return old AMO_AANDF Atomic memory AND, return new AMO_AFAND Atomic memory AND, return old AMO_ANANDF Atomic memory NAND, return new AMO_AFNAND Atomic memory NAND, return old AMO_AORF Atomic memory OR, return new AMO_AFOR Atomic memory OR, return old AMO_AXORF Atomic memory XOR, return new AMO_AFXOR Atomic memory XOR, return old AMO_ACSWAP Atomic memory swap, return old AMO_ASWAP Atomic memory swap, return new AMO_AFLUSH Atomic memory flush, flush local cache to the global address space for a particular address on a Cray XE or XK system. ASIND Arcsine, value in degrees ATAND Arctangent, value in degrees ATAND2 Arctangent, value in degrees CO_BCAST Broadcast a coarray to all images in an application. CO_SUM Sum of corresponding elements on all images in a coarray application CO_MIN, CO_MAX Maximum or minimum value of corresponding elements on all images in a coarray application COSD Cosine, argument in degrees COT Cotangent DSHIFTL Double word left shift (Proposed Fortran 2008 function) DSHIFTR Double word right shift (Proposed Fortran 2008 function) EXIT Program termination FREE Free Cray pointee memory GET_BORROW_S@ Get scalar borrow bit 200 S–3901–80Cray Fortran Language Extensions [9] GSYNC Complete outstanding memory references IBCHNG Reverse bit within a word ILEN Length in bits of an integer INT_MULT_UPPER Upper bits of integer product LEADZ Number of leading 0 bits (Proposed Fortran 2008 function) LOC Address of argument MALLOC Allocate Cray pointee memory MASK Creates a bit mask in a word NUM_IMAGES Number of executing images (Proposed Fortran 2008 function) POPCNT Number of 1 bits in a word (Proposed Fortran 2008 function) POPPAR XOR reduction of bits in a word (Proposed Fortran 2008 function) SET_BORROW_S@ Set scalar borrow bits SET_CARRY_S@ Set scalar carry bits SHIFTA Arithmetic right shift (Proposed Fortran 2008 function) SHIFTL Left shift, zero fill (Proposed Fortran 2008 function) SHIFTR Right shift, zero fill (Proposed Fortran 2008 function) SIND Sin, argument in degrees SIZEOF Size of argument in bytes SUB_BORROW_S@ Subtract scalar with borrow SYNC_IMAGES Synchronize indicated images TAND Tangent, argument in degrees THIS_IMAGE Image number of executing image (Proposed Fortran 2008 function) TRAILZ Number of trailing 0 bits (Proposed Fortran 2008 function) S–3901–80 201Cray Fortran Reference Manual CO_BCAST, CO_SUM, CO_MIN, and CO_MAX are collective intrinsic subroutines, which are extensions of the Fortran 2008 standard. Support for teams is deferred. For specific information about these routines, see the co_bcast(3i), co_max(3i), co_sum(3i) man pages. Many intrinsic procedures have both a vector and a scalar version. If a vector version of an intrinsic procedure exists, and the intrinsic is called within a vectorizable loop, the compiler uses the vector version of the intrinsic. For information about which intrinsic procedures vectorize, see the intro_intrin(3i) man page. For more information about the atomic memory intrinsic procedures see the amo(3i) man page. 9.13 Exceptions and IEEE Arithmetic 9.13.1 The Exceptions 9.13.1.1 IEEE Intrinsic Module Extensions The intrinsic module IEEE_EXCEPTIONS supplied with the Cray Fortran compiler contains three named constants in addition to those specified by the standard. These are of type IEEE_STATUS_TYPE and can be used as arguments to the IEEE_SET_STATUS subroutine. Their definitions correspond to common combinations of settings and allow for simple and fast changes to the IEEE mode settings. The constants are: Table 20. Cray Fortran IEEE Intrinsic Module Extensions Name Effect of CALL IEEE_SET_STATUS (Name) ieee_cri_nostop_mode • Clears all currently set exception flags • Disables halting for all exceptions • Enables setting of all exception flags • Sets rounding mode to round_to_nearest ieee_cri_default_mode • Clears all currently set exception flags • Enables halting for overflow, divide_by_zero, and invalid • Disables halting for underflow and inexact • Enables setting of all exception flags • Sets rounding mode to round_to_nearest 202 S–3901–80Cray Fortran Language Extensions [9] 9.14 Interoperability with C 9.14.1 Interoperability Between Fortran and C Entities 9.14.1.1 BIND(C) Syntax The proc-language-binding-spec specification allows Fortran programs to interoperate with C objects. The optional commas in SUBROUTINE name(), BIND(C) and FUNCTION name(), BIND(C) are Cray extensions to the Fortran standard. 9.14.2 ISO_C_BINDING The ISO_C_BINDING module provides interoperability between Fortran intrinsic types and C types. The ISO_C_BINDING module provides named constants which can be used as KIND type parameters, compatible with C types. In addition to the named constants required by the Fortran 2003 standard, Cray compiler provides, as an extension, definitions for 128-bit floating, and complex types. C_FLOAT128 and C_FLOAT128_COMPLEX correspond to C types __float128 and __float128 complex. 9.15 Coarrays The Cray Fortran compiler implements coarrays as a mechanism for data exchange in parallel programs. Data passing has proven itself to be an effective method for programming single-program-multiple-data (SPMD) parallel computation. Its chief advantage over message passing is lower latency for data transfers, which leads to better scalability of parallel applications. coarrays are a syntactic extension to the Fortran Language that offers a method for programming data passing. Data passing can also be accomplished by using the shared memory (SHMEM) library routines. Using SHMEM, the program transfers data from an object on one processing element to an object on another via subroutine calls. This technique is often referred to as one-sided communication. Coarrays provide an alternative syntax for specifying these transfers. With coarrays, the concept of a processing element is replaced by the concept of an image. When data objects are declared as coarrays, the corresponding coarrays on different images can be referenced or defined in a fashion similar to the way in which arrays are referenced or defined in Fortran. This is done by adding additional dimensions, or codimensions, within brackets ([ ]) to an object's declarations and references. These extra dimensions express the image upon which the object resides. S–3901–80 203Cray Fortran Reference Manual Coarrays offer the following advantages over SHMEM: • Coarrays are syntax-based, so programs that use them can be analyzed and optimized by the compiler. This offers greater opportunity for hiding data transfer latency. • Coarray syntax can eliminate the need to create and copy data to local temporary arrays. • Coarrays express data transfer naturally through the syntax of the language, making the code more readable and maintainable. • The unique bracket syntax allows you to scan for and to identify communication in a program easily. Consider the following SHMEM code fragment from a finite differencing algorithm: CALL SHMEM_REAL_GET(T1, U, NROW, LEFT) CALL SHMEM_REAL_GET(T2, U, NROW, RIGHT) NU(1:NROW) = NU(1:NROW) + T1(1:NROW) + T2(1:NROW) Coarrays can be used to express this fragment simply as: NU(1:NROW) = NU(1:NROW) + U(1:NROW)[LEFT] + U(1:NROW)[RIGHT] Notice that the resulting code is more concise and easier to read. The compiler copies to local temporary objects automatically. Coarrays can interoperate with the other message passing and data passing models. This interoperability allows you to introduce coarrays gradually into codes that presently use the Message Passing Interface (MPI) or SHMEM. For more information about using coarrays, see ISO/IEC JTC1/SC22/W65 N1747, "Coarrays in the Next Fortran Standard," by John Reid. This document can be accessed at the following location: ftp://ftp.nag.co.uk/sc22wg5/N1801-N1850/N1824.pdf. 9.16 Compiling and Executing Programs Containing Coarrays There are various commands, tools, and products available in the programming environment to use for compiling and executing programs containing coarrays. 9.16.1 ftn and aprun Options Affecting Coarrays The -h caf compiler option on the ftn command line must be specified in order for coarray syntax to be recognized and translated. Otherwise, the coarray syntax generates ERROR messages. 204 S–3901–80Cray Fortran Language Extensions [9] Upon execution of an a.out file that has been compiled and linked with the -h caf option, an image is created and executed on every processing element assigned to the job. Images 1 through NUM_IMAGES are assigned to processing elements 0 through N$PES-1, consecutively. You can set the number of processing elements assigned to a job at compile time by specifying the -X option on the ftn command. The number of processing elements can also be set at run time by executing the a.out file by using the aprun command with the -n option specified. If mixed -X values are used when compiling and linking different object files, or the number of PEs specified at run time differs from that specified when compiling and linking, you will receive a run time error. Bounds checking is performed by specifying the -Rb option on the ftn command line. This feature is not implemented for codimensions of coarrays. For more information about the ftn and aprun commands, see the ftn(1) and aprun(1) man pages. 9.16.2 Interoperating with Other Message Passing and Data Passing Models Coarrays can interoperate with all other message and data passing models: MPI and SHMEM. This allows you to introduce coarrays into existing application codes incrementally. These models are implemented through procedure calls, so the language interaction between coarrays and these models is well defined. ! Caution: MPI and SHMEM generally use processing element numbers, which start at zero, but the coarray model generally deals with image numbers, which start at one. For information about the mapping between processing elements and image numbers, see ftn and aprun Options Affecting Coarrays on page 204. Coarrays are symmetric for the purposes of SHMEM programming. Pointers in coarrays of derived type, however, may not necessarily point to symmetric data. For more information about the other message passing and data passing models, see the following man pages. • intro_mpi(3) • intro_shmem(3) 9.16.3 Optimizing Programs with Coarrays Programs containing coarrays benefit from all the usual steps you can take to improve run time performance of code that runs on a single image. S–3901–80 205Cray Fortran Reference Manual 9.17 Submodules The Cray Fortran Compiler fully supports submodules, which extend specifications and definitions to other program units by use association and stand in a tree-like relationship to other Fortran modules and submodules. There are no known differences between the Cray implementation and the Fortran 2008 standard. 206 S–3901–80Obsolete Features [10] The Cray Fortran compiler supports legacy features to allow the continued use of existing codes. In general, these features should not be used in new codes. The obsolete features are divided into two groups. The first is the set of features identified in Annex B of the Fortran standard as deleted. These were part of the Fortran language but their usage is explicitly discouraged in new codes. The second group is the set of legacy extensions supported in the Cray compiler for which preferred alternatives now exist. The obsolete features and their preferred alternatives are listed in Table 21. Table 21. Obsolete Features and Preferred Alternatives Obsolete Feature Preferred Alternative IMPLICIT UNDEFINED IMPLICIT NONE Type statements with *n Type statements with KIND= parameters BYTE data type INTEGER(KIND=1) DOUBLE COMPLEX statement COMPLEX statement with KIND parameter STATIC attribute and statement SAVE attribute and statement Slash data initialization Standard initialization syntax DATA statement features Standard conforming DATA statements Hollerith data Character data PAUSE statement READ statement ASSIGN, assigned GOTO statements and assigned format specifiers Standard branching constructs Two-branch IF statements IF construct or statement Real and double precision DO variables Integer DO variables Nested loop termination Separate END DO statements Branching into a block Restructure code ENCODE and DECODE statements WRITE and READ with internal file BUFFER IN and BUFFER OUT statements Asynchronous I/O statements Asterisk character constant delimiters Use standard character delimiters Negative-values X descriptor TL descriptor S–3901–80 207Cray Fortran Reference Manual Obsolete Feature Preferred Alternative A descriptor used for noncharacter conventional data and R descriptor Character type and other conventional matchings of data and descriptors H edit descriptor Character constants Obsolete intrinsic procedures For list and replacements, see Obsolete Intrinsic Procedures on page 225 Initialization using long strings Replace the numeric target with a character item. Replace a Hollerith constant with a character constant 10.1 IMPLICIT UNDEFINED The Cray Fortran compiler accepts the IMPLICIT UNDEFINED statement. It is equivalent to the IMPLICIT NONE statement. 10.2 Type Statement with *n The Cray Fortran compiler defines the following additional forms of type_declaration_stmt: type_spec is INTEGER* length_value or REAL* length_value or DOUBLE PRECISION* length_value or COMPLEX* length_value or LOGICAL* length_value • length-value is the size of the data object in bytes. Data type declarations that include the data length are outmoded. The Cray Fortran compiler recognizes this usage in type statements, IMPLICIT statements, and FUNCTION statements, mapping these numbers onto kind values appropriate for the target machine. 10.3 BYTE Data Type The BYTE statement and data type declares a 1–byte value. This data type is equivalent to the INTEGER(KIND=1) and INTEGER*1 declarations. 208 S–3901–80Obsolete Features [10] 10.4 DOUBLE COMPLEX Statement The DOUBLE COMPLEX statement is used to declare an item to be of type double complex. The format for the DOUBLE COMPLEX statement is as follows: DOUBLE COMPLEX [ , attribute-list :: ] entity-list Items declared as DOUBLE COMPLEX contain two double precision entities. When the -dp option is in effect, double complex entities are affected as follows: • The nonstandard DOUBLE COMPLEX declaration is treated as a single-precision complex type. • Double precision intrinsic procedures are changed to the corresponding single-precision intrinsic procedures. The -ep or -dp specification is used for all source files compiled with a single invocation of the Cray Fortran compiler command. If a module is compiled separately from a program unit that uses the module, they both shall be compiled with the same -ep or -dp specification. 10.5 STATIC Attribute and Statement The STATIC attribute and statement provides the same effect as the SAVE attribute and statement. Variables with the Cray Fortran STATIC attribute retain their value and their definition, association, and allocation status after the subprogram in which they are declared completes execution. Variables without this attribute cannot be depended on to retain its value and status, although the Cray Fortran compiler treats named common blocks as if they had this attribute. This attribute should always be specified for an object or the object's common named block, if it is necessary for the object to retain its value and status. In Cray's implementation, the system retains the value of an object that is in a module whether or not the STATIC specifier is used. Objects declared in recursive subprograms can be given the attribute. Such objects are shared by all instances of the subprogram. Any object that is data initialized (in a DATA statement or a type declaration statement) has the STATIC attribute by default. The following is a format for a type declaration statement with the attribute: type, STATIC [, attribute-list ] :: entity-decl-list static-stmt is STATIC [ [ :: ] static-entity-list ] static-entity is data-object-name or / common-block-name / S–3901–80 209Cray Fortran Reference Manual A statement without an entity list is treated as though it contained the names of all items that could be saved in the scoping unit. The Cray Fortran compiler allows you to insert multiple statements without entity lists in a scoping unit. If STATIC appears in a main program as an attribute or a statement, it has no effect. The following objects must not be saved: • A procedure • A function result • A dummy argument • A named constant • An automatic data object • An object in a common block • A namelist group A variable in a common block cannot be saved individually; the entire named common block must be saved if you want any variables in it to be saved. A named common block saved in one scoping unit of a program is saved throughout the program. If a named common block is specified in a main program, it is available to any scoping unit of the program that specifies the named common block; it does not need to be saved. The statement also confers the attribute. It is subject to the same rules and restrictions as the attribute. The following example shows an entity-oriented declaration: CHARACTER(LEN = 12), SAVE :: NAME CHARACTER(LEN = 12), STATIC :: NAME The following example shows an attribute-oriented declaration: CHARACTER*12 NAME STATIC NAME !Use SAVE OR STATIC, but not both on the same name The following example shows saving objects and named common blocks: STATIC A, B, /BLOCKA/, C, /BLOCKB/ 210 S–3901–80Obsolete Features [10] 10.6 Slash Data Initialization The Fortran type declaration statements provide a means for data initialization. For example, the following two methods are standard means for initializing integer data: • Method 1: INTEGER :: I=3 • Method 2: INTEGER I DATA I /3/ The Cray Fortran compiler supports an additional method for each data type. The following example shows the additional, nonstandard method, used to define integer data: • Method 3: INTEGER [::] I /3/ 10.7 DATA Statement Features The DATA statement has the following outmoded features: • A constant need not exist for each element of a whole array named in a data-stmt-object-list if the array is the last item in the list. • A Hollerith or character constant can initialize more than one element of an integer or single-precision real array if the array is specified without subscripts. Example 1: If the -s default32 compiler option is used (default), an array is declared by INTEGER A(2), the following DATA statements have the same effect: DATA A /'12345678'/ DATA A /'1234','5678'/ Example 2: If the -s default64 compiler option is specified, an array is declared by INTEGER A(2), the following DATA statements have the same effect: DATA A /'1234567890123456'/ DATA A /'12345678','90123456'/ An integer or single-precision real array can be defined in the same way in a DATA implied-DO statement. 10.8 Hollerith Data Before the character data type was added to the Fortran 77 standard, Hollerith data provided a method of supplying character data. S–3901–80 211Cray Fortran Reference Manual 10.8.1 Hollerith Constants A Hollerith constant is expressed in one of three forms. The first of these is specified as a nonzero integer constant followed by the letter H, L, or R and as many characters as equal the value of the integer constant. The second form of Hollerith constant specification delimits the character sequence between a pair of apostrophes followed by the letter H, L, or R. The third form is like the second, except that quotation marks replace apostrophes. For example: Character sequence: ABC 12 Form 1: 6HABC 12 Form 2: 'ABC 12'H Form 3: "ABC 12"H Two adjacent apostrophes or quotation marks appearing between delimiting apostrophes or quotation marks are interpreted and counted by the compiler as a single apostrophe or quotation mark within the sequence. Thus, the sequence DON'T USE "*" would be specified with apostrophe delimiters as 'DON''T USE "*"'H, and with quotation mark delimiters as "DON'T USE ""*"""H. Each character of a Hollerith constant is represented internally by an 8-bit code, with up to 32 such codes allowed. This limit corresponds to the size of the largest numeric type, COMPLEX(KIND = 16). The ultimate size and makeup of the Hollerith data depends on the context. If the Hollerith constant is larger than the size of the type implied by context, the constant is truncated to the appropriate size. If the Hollerith constant is smaller than the size of the type implied by context, the constant is padded with a character dependent on the Hollerith indicator. When an H Hollerith indicator is used, the truncation and padding is done on the right end of the constant. The pad character is the blank character code (20). Null codes can be produced in place of blank codes by substituting the letter L for the letter H in the Hollerith forms described above. The truncation and padding is also done on the right end of the constant, with the null character code (00) as the pad character. Using the letter R instead of the letter H as the Hollerith indicator means truncation and padding is done on the left end of the constant with the null character code (00) used as the pad character. All of the following Hollerith constants yield the same Hollerith constant and differ only in specifying the content and placement of the unused portion of the single 64-bit entity containing the constant: 212 S–3901–80Obsolete Features [10] Hollerith Internal byte, beginning on bit: Constant 0 8 16 24 32 40 48 56 6HABCDEF A B C D E F 20 16 20 16 'ABCDEF'H A B C D E F 20 16 20 16 "ABCDEF" H A B C D E F 20 16 20 16 6LABCDEF A B C D E F 00 00 'ABCDEF'L A B C D E F 00 00 "ABCDEF"L A B C D E F 00 00 6RABCDEF 00 00 A B C D E F 'ABCDEF'R 00 00 A B C D E F "ABCDEF"R 00 00 A B C D E F A Hollerith constant is limited to 32 characters except when specified in a CALL statement, a function argument list, or a DATA statement. An all-zero computer word follows the last word containing a Hollerith constant specified as an actual argument in an argument list. A character constant of 32 or fewer characters is treated as if it were a Hollerith constant in situations where a character constant is not allowed by the standard but a Hollerith constant is allowed by the Cray Fortran compiler. If the character constant appears in a DATA statement value list, it can be longer than 32 characters. 10.8.2 Hollerith Values A Hollerith value is a Hollerith constant or a variable that contains Hollerith data. A Hollerith value is limited to 32 characters. A Hollerith value can be used in any operation in which a numeric constant can be used. It can also appear on the right-hand side of an assignment statement in which a numeric constant can be used. It is truncated or padded to be the correct size for the type implied by the context. 10.8.3 Hollerith Relational Expressions Used with a relational operator, the Hollerith value e 1 is less than e 2 if its value precedes the value of e 2 in the collating sequence and is greater if its value follows the value of e 2 in the collating sequence. S–3901–80 213Cray Fortran Reference Manual The following examples are evaluated as true if the integer variable LOCK contains the Hollerith characters K, E, and Y in that order and left-justified with five trailing blank character codes: 3HKEY.EQ.LOCK 'KEY'.EQ.LOCK LOCK.EQ.LOCK 'KEY1'.GT.LOCK 'KEY0'H.GT.LOCK 10.9 PAUSE Statement Execution of a PAUSE statement requires operator or system-specific intervention to resume execution. In most cases, the same functionality can be achieved as effectively and in a more portable way with the use of an appropriate READ statement that awaits some input data. The execution of the PAUSE statement suspends the execution of a program. This is now redundant, because a WRITE statement can be used to send a message to any device, and a READ statement can be used to wait for and receive a message from the same device. The PAUSE statement is defined as follows: pause-stmt is PAUSE [ stop-code ] The character constant or list of digits identifying the PAUSE statement is called the stop-code because it follows the same rules as those for the STOP statement's stop code. The stop code is accessible following program suspension. The Cray Fortran compiler sends the stop-code to the standard error file (stderr). The following are examples of PAUSE statements: PAUSE PAUSE 'Wait #823' PAUSE 100 10.10 ASSIGN, Assigned GO TO Statements, and Assigned Format Specifiers The ASSIGN statement assigns a statement label to an integer variable. During program execution, the variable can be assigned labels of branch target statements, providing a dynamic branching capability in a program. The unsatisfactory property of these statements is that the integer variable name can be used to hold both a label and an ordinary integer value, leading to errors that can be hard to discover and programs that can be difficult to read. 214 S–3901–80Obsolete Features [10] A frequent use of the ASSIGN statement and assigned GO TO statement is to simulate internal procedures, using the ASSIGN statement to record the return point after a reusable block of code has completed. The internal procedure mechanism of Fortran now provides this capability. A second use of the ASSIGN statement is to simulate dynamic format specifications by assigning labels corresponding to different format statements to an integer variable and using this variable in I/O statements as a format specifier. This use can be accomplished in a clearer way by using character strings as format specifications. Thus, it is no longer necessary to use either the ASSIGN statement or the assigned GO TO statement. Execution of an ASSIGN statement causes the variable in the statement to become defined with a statement label value. When a numeric storage unit becomes defined, all associated numeric storage units of the same type become defined. Variables associated with the variable in an ASSIGN statement, however, become undefined as integers when the ASSIGN statement is executed. When an entity of double precision real type becomes defined, all totally associated entities of double precision real type become defined. Execution of an ASSIGN statement causes the variable in the statement to become undefined as an integer. Variables that are associated with the variable also become undefined. 10.10.1 Form of the ASSIGN and Assigned GO TO Statements Execution of an ASSIGN statement assigns a label to an integer variable. Subsequently, this value can be used by an assigned GO TO statement or by an I/O statement to reference a FORMAT statement. The ASSIGN statement is defined as follows: assign-stmt is ASSIGN label TO scalar-int-variable The term default integer type in this section means that the integer variable shall occupy a full word in order to be able to hold the address of the statement label. Programs that contain an ASSIGN statement and are compiled with -s default32 shall ensure that the scalar-int-variable is declared as INTEGER(KIND=8). This ensures that it occupies a full word. The variable shall be a named variable of default integer type. It shall not be an array element, an integer component of a structure, or an object of nondefault integer type. The label shall be the label of a branch target statement or the label of a FORMAT statement in the same scoping unit as the ASSIGN statement. When defined with an integer value, the integer variable cannot be used as a label. S–3901–80 215Cray Fortran Reference Manual When assigned a label, the integer variable cannot be used as anything other than a label. When the integer variable is used in an assigned GO TO statement, it shall be assigned a label. As the following example shows, the variable can be redefined during program execution with either another label or an integer value: ASSIGN 100 TO K Execution of the assigned GO TO statement causes a transfer of control to the branch target statement with the label that had previously been assigned to the integer variable. The assigned GO TO statement is defined as follows: assigned-goto-stmt is GO TO scalar-int-variable [ [ , ] (label-list) ] The variable shall be a named variable of default integer type. That is, it shall not be an array element, a component of a structure, or an object of nondefault integer type. The variable shall be assigned the label of a branch target statement in the same scoping unit as the assigned GO TO statement. If a label list appears, such as in the following examples, the variable shall have been assigned a label value that is in the list: GO TO K GO TO K (10, 20, 100) The ASSIGN statement also allows the label of a FORMAT statement to be dynamically assigned to an integer variable, which can later be used as a format specifier in READ, WRITE, or PRINT statements. This hinders readability, permits inconsistent usage of the integer variable, and can be an obscure source of error. This functionality is available through character variables, arrays, and constants. 10.10.2 Assigned Format Specifiers When an I/O statement containing the integer variable as a format specifier is executed, the integer variable can be defined with the label of a FORMAT specifier. 10.11 Two-branch IF Statements Outmoded IF statements are the two-branch arithmetic IF and the indirect logical IF. 216 S–3901–80Obsolete Features [10] 10.11.1 Two-branch Arithmetic IF A two-branch arithmetic IF statement transfers control to statement s 1 if expression e is evaluated as nonzero or to statement s 2 if e is zero. The arithmetic expression should be replaced with a relational expression, and the statement should be changed to an IF statement or an IF construct. This format is as follows: IF ( e ) s 1 , s 2 e Integer, real, or double precision expression s Label of an executable statement in the same program unit Example: IF (I+J*K) 100,101 10.11.2 Indirect Logical IF An indirect logical IF statement transfers control to statement s t if logical expression le is true and to statement s f if le is false. An IF construct or an IF statement should be used in place of this outmoded statement. This format is as follows: IF ( le ) s t , s f le Logical expression s t , s f Labels of executable statements in the same program unit Example: IF(X.GE.Y)148,9999 10.12 Real and Double Precision DO Variables The Cray Fortran compiler allows real variables and values as the DO variable and limits in DO statements. The preferred alternative is to use integer values and compute the desired real value. 10.13 Nested Loop Termination Older Cray Fortran compilers allowed nested DO loops to terminate on a single END DO statement if the END DO statement had a statement label. The END DO statement is included in the Fortran standard. The Fortran standard specifies that a separate END DO statement shall be used to terminate each DO loop, so allowing nested DO loops to end on a single, labeled END DO statement is an outmoded feature. S–3901–80 217Cray Fortran Reference Manual 10.14 Branching into a Block Although the standard does not permit branching into the code block for a DO construct from outside of that construct, the Cray Fortran compiler permits branching into the code block for a DO or DO WHILE construct. By default, the Cray Fortran compiler issues an error for this situation. Cray does not recommend branching into a DO construct, but if you specify the ftn -eg command, the code will compile. 10.15 ENCODE and DECODE Statements A formatted I/O operation defines entities by transferring data between I/O list items and records of a file. The file can be on an external media or in internal storage. The Fortran standard provides READ and WRITE statements for both formatted external and internal file I/O. This is the preferred method for formatted internal file I/O. It is the only method for list-directed internal file I/O. The ENCODE and DECODE statements are an alternative to standard Fortran READ and WRITE statements for formatted internal file I/O. An internal file in standard Fortran I/O shall be declared as character, while the internal file in ENCODE and DECODE statements can be any data type. A record in an internal file in standard Fortran I/O is either a scalar character variable or an array element of a character array. The record size in an internal file in an ENCODE or DECODE statement is independent of the storage size of the variable used as the internal file. If the internal file is a character array in standard Fortran I/O, multiple records can be read or written with internal file I/O. The alternative form does not provide the multiple record capability. 10.15.1 ENCODE Statement The ENCODE statement provides a method of converting or encoding the internal representation of the entities in the output list to a character representation. The format of the ENCODE statement is as follows: ENCODE ( n, f, dest ) [ elist ] n Number of characters to be processed. Nonzero integer expression not to exceed the maximum record length for formatted records. This is the record size for the internal file. f Format identifier. It cannot be an asterisk. dest Name of internal file. It can be a variable or array of any data type. It cannot be an array section, a zero-sized array, or a zero-sized character variable. elist Output list to be converted to character during the ENCODE statement. 218 S–3901–80Obsolete Features [10] The output list items are converted using format f to produce a sequence of n characters that are stored in the internal file dest. The n characters are packed 8 characters per word. An ENCODE statement transfers one record of length n to the internal file dest. If format f attempts to write a second record, ENCODE processing repositions the current record position to the beginning of the internal file and begins writing at that position. An error is issued when the ENCODE statement attempts to write more than n characters to the record of the internal file. If dest is a noncharacter entity and n is not a multiple of 8, the last word of the record is padded with blanks to a word boundary. If dest is a character entity, the last word of the record is not padded with blanks to a word boundary. Example 1: The following example assumes a machine word length of 64 bits and uses the underscore character (_) as a blank: INTEGER ZD(5), ZE(3) ZD(1)='THIS____' ZD(2)='MUST____' ZD(3)='HAVE____' ZD(4)='FOUR____' ZD(5)='CHAR____' 1 FORMAT(5A4) ENCODE(20,1,ZE)ZD DO 10 I=1,3 PRINT 2,'ZE(',I,')="',ZE(I),'"' 10 CONTINUE 2 FORMAT(A,I2,A,A8,A) END The output is as follows: >ZE( 1)="THISMUST" >ZE( 2)="HAVEFOUR" >ZE( 3)="CHAR____" 10.15.2 DECODE Statement The DECODE statement provides a method of converting or decoding from a character representation to the internal representation of the entities in the input list. The format of the DECODE statement is as follows: DECODE ( n, f, source ) [ dlist ] n Number of characters to be processed. Nonzero integer expression not to exceed the maximum record length for formatted records. This is the record size for the internal file. f Format identifier. It cannot be an asterisk. S–3901–80 219Cray Fortran Reference Manual source Name of internal file. It can be a variable or array of any data type. It cannot be an array section or a zero-sized array or a zero-sized character variable. dlist Input list to be converted from character during the DECODE statement. The input list items are converted using format f from a sequence of n characters in the internal file source to an internal representation and stored in the input list entities. If the internal file source is noncharacter, the internal file is assumed to be a multiple of 8 characters. Example 1: An example of a DECODE statement is as follows: INTEGER ZD(4), ZE(3) ZE(1)='WHILETHI' ZE(2)='S HAS F' ZE(3)='IVE ' 3 FORMAT(4A5) DECODE(20,3,ZE)ZD DO 10 I=1,4 PRINT 2,'ZD(',I,')="',ZD(I),'"' 10 CONTINUE 2 FORMAT(A,I2,A,A8,A) END The output is as follows: >ZD( 1)="WHILE " >ZD( 2)="THIS " >ZD( 3)="HAS " >ZD( 4)="FIVE " 10.16 BUFFER IN and BUFFER OUT Statements You can use the BUFFER IN and BUFFER OUT statements to transfer data. Data can be transferred while allowing the subsequent execution sequence to proceed concurrently. This is called asynchronous I/O. Asynchronous I/O may require the use of nondefault file formats or FFIO layers, as discussed in Chapter 14, Using Flexible File I/O (FFIO) on page 265. BUFFER IN and BUFFER OUT operations may proceed concurrently on several units or files. If they do not proceed asynchronously, they will use synchronous I/O. BUFFER IN is for reading, and BUFFER OUT is for writing. A BUFFER IN or BUFFER OUT operation includes only data from a single array or a single common block. 220 S–3901–80Obsolete Features [10] Either statement initiates a data transfer between a specified file or unit (at the current record) and memory. If the unit or file is completing an operation initiated by any earlier BUFFER IN or BUFFER OUT statement, the current BUFFER IN or BUFFER OUT statement suspends the execution sequence until the earlier operation is complete. When the unit's preceding operation terminates, execution of the BUFFER IN or BUFFER OUT statement completes as if no delay had occurred. You can use the UNIT or LENGTH intrinsic procedures to delay the execution sequence until the BUFFER IN or BUFFER OUT operation is complete. These functions can also return information about the I/O operation at its termination. The general format of the BUFFER IN and BUFFER OUT statements follows: buffer_in_stmt is BUFFER IN (id, mode) (start_loc, end_loc) buffer_out_stmt is BUFFER OUT (id, mode) (start_loc, end_loc) io_unit is external_file_unit or file_name_expr mode is scalar_integer_expr start_loc is variable end_loc is variable In the preceding definition, the variable specified for start_loc and end_loc cannot be of a derived type if you are performing implicit data conversion. The data items between start_loc and end_loc must be of the same type. The BUFFER IN and BUFFER OUT statements are defined as follows. BUFFER IN (io_unit, mode) (start_loc, end_loc) BUFFER OUT (io_unit, mode) (start_loc, end_loc) io_unit An identifier that specifies a unit. The I/O unit is a scalar integer expression with a nonnegative value, an asterisk (*), or a character literal constant (external name). The I/O unit forms indicate that the unit is a formatted sequential access external unit. mode Mode identifier. This integer expression controls the record position following the data transfer. The mode identifier is ignored on files that do not contain records; only full record processing is available. S–3901–80 221Cray Fortran Reference Manual start_loc, end_loc Symbolic names of the variables, arrays, or array elements that mark the beginning and ending locations of the BUFFER IN or BUFFER OUT operation. These names must be either elements of a single array (or equivalenced to an array) or members of the same common block. If start_loc or end_loc is of type character, then both must be of type character. If start_loc and end_loc are noncharacter, then the item length of each must be equal. For example, if the internal length of the data type of start_loc is 64 bits, the internal length of the data type of end_loc must be 64 bits. To ensure that the size of start_loc and end_loc are the same, use the same data type for both. The mode identifier, mode, controls the position of the record at unit io_unit after the data transfer is complete. The values of mode have the following effects: • Specifying mode = 0 causes full record processing. File and record positioning works as with conventional I/O. The record position following such a transfer is always between the current record (the record with which the transfer occurred) and the next record. Specifying BUFFER OUT with mode = 0 ends a series of partial-record transfers. • Specifying mode < 0 causes partial record processing. In BUFFER IN, the record is positioned to transfer its (n +1)th word if the nth word was the last transferred. In BUFFER OUT, the record is left positioned to receive additional words. The amount of data to be transferred is specified in words without regard to types or formats. However, the data type of end_loc affects the exact ending location of a transfer. If end_loc is of a multiple-word data type, the location of the last word in its multiple-word form of representation marks the ending location of the data transfer. BUFFER OUT with start_loc = end_loc + 1 and mode = 0 causes a zero-word transfer and concludes the record being created. Except for terminating a partial record, start_loc following end_loc in a storage sequence causes a run time error. Example: PROGRAM XFR DIMENSION A(1000), B(2,10,100), C(500) ... BUFFER IN(32,0) (A(1),A(1000)) ... DO 9 J=1,100 B(1,1,J) = B(1,1,J) + B(2,1,J) 9 CONTINUE BUFFER IN(32,0) (C(1),C(500)) BUFFER OUT(22,0) (A(1),A(1000)) ... END 222 S–3901–80Obsolete Features [10] The first BUFFER IN statement in this example initiates a transfer of 1000 words from unit 32. If asynchronous I/O is available, processing unrelated to that transfer proceeds. When this is complete, a second BUFFER IN is encountered, which causes a delay in the execution sequence until the last of the 1000 words is received. A transfer of another 500 words is initiated from unit 32 as the execution sequence continues. BUFFER OUT begins a transfer of the first 1000 words to unit 22. In all cases mode = 0, indicating full record processing. 10.17 Asterisk Delimiters The asterisk was allowed to delimit a literal character constant. It has been replaced by the apostrophe and quotation mark. *h 1 h 2 ... h n * * Delimiter for a literal character string h Any ASCII character indicated by a C that is capable of internal representation Example: *AN ASTERISK EDIT DESCRIPTOR* 10.18 Negative-valued X Descriptor A negative value could be used with the X descriptor to indicate a move to the left. This has been replaced by the TL descriptor. [-b]X b Any nonzero, unsigned integer constant X Indicates a move of as many positions as indicated by b Example: -55X ! Moves current position 55 spaces left 10.19 A and R Descriptors for Noncharacter Types The Rw descriptor and the use of the Aw descriptor for noncharacter data are available primarily for programs that were written before a true character type was available. Other uses include adding labels to binary files and the transfer of data whose type is not known in advance. List items can be of type real, integer, complex, or logical. For character use, the binary form of the data is converted to or from ASCII codes. The numeric list item is assumed to contain ASCII characters when used with these edit descriptors. S–3901–80 223Cray Fortran Reference Manual Complex items use two storage units and require two A descriptors, for the first and second storage units respectively. The Aw descriptor works with noncharacter list items containing character data in essentially the same way as described in the Fortran standard. The Rw descriptor works like Aw with the following exceptions: • Characters in an incompletely filled input list item are right-justified with the remainder of that list item containing binary zeros. • Partial output of an output list item is from its rightmost character positions. The following example shows the Aw and Rw edit descriptors for noncharacter data types: INTEGER IA LOGICAL LA REAL RA DOUBLE PRECISION DA COMPLEX CA CHARACTER*52 CHC CHC='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz' READ(CHC,3) IA, LA, RA, DA, CA 3 FORMAT(A4,A8,A10,A17,A7,A6) PRINT 4, IA, LA, RA, DA, CA 4 FORMAT(1x,3(A8,'-'),A16,'-',2A8) READ(CHC,5) IA, LA, RA 5 FORMAT(R2,R8,R9) PRINT 4, IA, LA, RA END The output of this program would be as follows: > ABCD -EFGHIJKL-OPQRSTUV-XYZabcdefghijklm-nopqrst uvwxyz > ooooooAB-CDEFGHIJ-LMNOPQRSThe arrow (>) indicates leading blanks in the use of the A edit descriptor. The lowercase letter o is used to indicate where binary zeros have been written with the R edit descriptor. The binary zeros are not printable characters, so the printed output simply contains the characters without the binary zeros. 10.20 H Edit Descriptor This edit descriptor can be a source of error because the number of characters following the descriptor can be miscounted easily. The same functionality is available using the character constant edit descriptor, for which no count is required. 224 S–3901–80Obsolete Features [10] The following information pertains to the H edit descriptor: Table 22. Summary of String Edit Descriptors Descriptor Description H Transfer of text character to output record 'text' Transfer of a character literal constant to output record "text" Transfer of a character literal constant to output record 10.21 Obsolete Intrinsic Procedures The Cray Fortran compiler supports many intrinsic procedures that have been used in legacy codes, but that are now obsolete. The following table indicates the obsolete procedures and the preferred alternatives. For more information about a particular procedure, see its man page. Table 23. Obsolete Procedures and Alternatives Obsolete Intrinsic Procedure Preferred Alternative AND IAND BITEST BTEST BJTEST BTEST BKTEST BTEST CDABS ABS CDCOS COS CDEXP EXP CDLOG LOG CDSIN SIN CDSQRT SQRT CLOC LOC or C_LOC COMPL NOT COTAN COT CQABS ABS CQDEXP EXP CQSIN SIN CQSQRT SQRT CSMG MERGE S–3901–80 225Cray Fortran Reference Manual Obsolete Intrinsic Procedure Preferred Alternative CVMGM MERGE CVMGN MERGE CVMGP MERGE CVMGZ MERGE CVMGT MERGE DACOSD ACOSD DASIND ASIND DATAN2D ATAN2D DATAND ATAND DCMPLX CMPLX DCONJG CONJG DCOSD COSD DCOT COT DCOTAN COT DFLOAT REAL DFLOATI REAL DFLOATJ REAL DFLOATK REAL DIMAG AIMAG DREAL REAL DSIND SIND DTAND TAND EQV NOT, IEOR FCD (none) FLOATI REAL FLOATJ REAL FLOATK REAL FP_CLASS IEEE_CLASS IDATE DATE_AND_TIME IEEE_REAL REAL IIABS ABS IIAND IAND IIBCHNG IBCHNG 226 S–3901–80Obsolete Features [10] Obsolete Intrinsic Procedure Preferred Alternative IIBCLR IBCLR IIBITS IBITS IIBSET IBSET IIEOR IEOR IIDIM DIM IIDINT INT IIFIX INT IINT INT IIOR IOR IIQINT INT IISHA SHIFTA IISHC ISHFT IISHFT ISHFTC IISHFTC ISHFTC IISHL ISHFT IISIGN SIGN IMAG AIMAG IMOD MOD ININT NINT INT2 INT INT4 INT INT8 INT INOT NOT IQNINT NINT IRTC SYSTEM_CLOCK ISHA SHIFTA ISHC ISHFTC ISHL IEEE_IS_NAN JDATE DATE_AND_TIME JFIX INT JIABS ABS JIAND IAND JIBCHNG IBCHNG S–3901–80 227Cray Fortran Reference Manual Obsolete Intrinsic Procedure Preferred Alternative JIBCLR IBCLR JIBITS IBITS JIBSET IBSET JIEOR IEOR JIDIM DIM JIDINT INT JIFIX INT JINT INT JIOR IOR JIQINT INT JISHA SHIFTA JISHC ISHFTC JISHFT ISHFT JISHFTC ISHFTC JISHL ISHFT JISIGN SIGN JMOD MOD JNINT NINT JNOT NOT KIABS ABS KIAND IAND KIBCHNG IBCHNG KIBCLR IBCLR KIBITS IBITS KIBSET IBSET KIEOR IEOR KIDIM DIM KIDINT INT KINT INT KIOR IOR KIQINT INT KISHA SHIFTA KISHC ISHFTC 228 S–3901–80Obsolete Features [10] Obsolete Intrinsic Procedure Preferred Alternative KISHFT ISHFT KISHFTC ISHFTC KISHL ISHFT KISIGN SIGN KMOD MOD KNINT NINT KNOT NOT LENGTH (none) LONG INT LSHIFT ISHFT or SHIFTL MY_PE THIS_IMAGE MEMORY_BARRIER SYNC MEMORY NEQV IEOR OR IOR QABS ABS QACOS ACOS QACOSD ACOSD QASIN ASIN QASIND ASIND QATAN ATAN QATAN2 ATAN2 DATAN2D ATAN2D QATAND ATAND QCMPLX CMPLX QCONJG CONJG QCOS COS QCOSD COSD QCOSH COSH QCOT COT QCOTAN COT QDIM DIM QEXP EXP QEXT REAL S–3901–80 229Cray Fortran Reference Manual Obsolete Intrinsic Procedure Preferred Alternative QFLOAT REAL QFLOATI REAL QFLOATJ REAL QFLOATJ REAL QFLOATK REAL QIMAG AIMAG QINT AINT QLOG LOG QLOG10 LOG10 QMAX1 MAX QMIN1 MIN QMOD MOD QNINT ANINT QREAL REAL QSIGN SIGN QSIN SIN QSIND SIND QSINH SINH QSQRT SQRT QTAN TAN QTAND TAND QTANH TANH RAN RANDOM_NUMBER RANF RANDOM_NUMBER RANGET RANDOM_SEED RANSET RANDOM_SEED REMOTE_WRITE_BARRIER SYNC MEMORY RSHIFT ISHFT or SHIFTR RTC SYSTEM_CLOCK SECNDS CPU_TIME SHIFT ISHFTC SHORT INT SNGLQ REAL 230 S–3901–80Obsolete Features [10] Obsolete Intrinsic Procedure Preferred Alternative TIME DATE_AND_TIME UNIT WAIT WRITE_MEMORY_BARRIER SYNC MEMORY XOR IEOR S–3901–80 231Cray Fortran Reference Manual 232 S–3901–80Cray Fortran Deferred Implementation and Optional Features [11] This release of the Cray Fortran compiler supports the Fortran 2003 standard, with the following exceptions. 11.1 ISO_10646 Character Set The Fortran 2003 features related to supporting the ISO_10646 character set are not supported. This includes declarations, constants, and operations on variables of character(kind=4) and I/O operations. Support for this feature is optional in Fortran 2003. 11.2 Restrictions on Unlimited Polymorphic Variables Unlimited polymorphic variables whose dynamic types are integer(1), integer(2), logical(1), or logical(2) are not supported, unless the -d h is specified to disable packed storage for short integers and logicals. 11.3 ENCODING= in I/O Statements The ENCODING= specifier in I/O statements is accepted by the compiler but has no effect. 11.4 Allocatable Assignment (Optionally Enabled) The Fortran 2003 standard allows an allocatable variable in an intrinsic assignment statement (variable = expression) to have a shape different from the expression. If the shapes are different, the variable is automatically deallocated and reallocated with the shape of the expression. This feature is available in the CCE Cray Fortran compiler but is not enabled by default because of potential adverse effects on performance. The new behavior is enabled by the -e w command line option. S–3901–80 233Cray Fortran Reference Manual 234 S–3901–80Cray Fortran Implementation Specifics [12] The Fortran standard specifies the rules for writing a standard conforming Fortran program. Many of the details of how such a program is compiled and executed are intentionally not specified or are explicitly specified as being processor-dependent. This chapter describes the implementation used by the Cray Fortran compiler. Included are descriptions of the internal representations used for data objects and the values of processor-dependent language parameters. 12.1 Companion Processor For the purpose of C interoperability, the Fortran standard refers to a companion processor. The companion processor for the Cray Fortran compiler is the Cray C compiler. 12.2 INCLUDE Line There is no limit to the nesting level for INCLUDE lines. The character literal constant in an INCLUDE line is interpreted as the name of the file to be included. This case-sensitive name may be prefixed with additional characters based on the -I compiler command line option. 12.3 INTEGER Kinds and Values INTEGER kind type parameters of 1, 2, 4, and 8 are supported. The default kind type parameter is 4 unless the -s default64 or -s integer64 command line option is specified, in which case the default kind type parameter is 8. The interpretation of kinds 1 and 2 depend on whether the -d h command line option is specified. Integer values are represented as two's complement binary values. 12.4 REAL Kinds and Values REAL kind type parameters of 4 and 8 are supported. The default kind type parameter is 4 unless the -s default64 or -s real64 command lines option is specified, in which case, the default kind type parameter is 8. Real values are represented in the format specified by the IEEE 754 standard, with kinds 4 and 8 corresponding to the 32 and 64 bit IEEE representations. S–3901–80 235Cray Fortran Reference Manual 12.5 DOUBLE PRECISION Kinds and Values The DOUBLE PRECISION type is an alternate specification of a REAL type. The kind type parameter of that REAL type is twice the value of the kind type parameter for default REAL unless the -sdefault64 or -sreal64 command line options are specified, in which case, the kind type parameter for DOUBLE PRECISION and default REAL are the same, and REAL constants with a D exponent are treated as if the D were an E. Note that if the -sdefault64 or -sreal64 options are specified, the compiler is not standard conforming. 12.6 LOGICAL Kinds and Values LOGICAL kind type parameters of 1, 2, 4, and 8 are supported. The default kind type parameter is 4 unless the -s default64 or -s integer64 command line option is specified, in which case, the default kind type parameter is 8. The interpretation of kinds 1 and 2 depend on whether the -d h command line option is specified. Logical values are represented by a bit sequence in which the low order bit is set to 1 for the value .true. and to 0 for .false., and the other bits in the representation are set to 0. 12.7 CHARACTER Kinds and Values The CHARACTER kind type parameter of 1 is supported. The default kind type parameter is 1. Character values are represented using the 8-bit ASCII character encoding. 12.8 Cray Pointers Cray pointers are 64-bit objects. 12.9 ENUM Kind An enumerator that specifies the BIND(C) attribute creates values with a kind type parameter of 4. 12.10 Storage Issues This section describes how the Cray Fortran compiler uses storage, including how this compiler accommodates programs that use overindexing of blank common. 236 S–3901–80Cray Fortran Implementation Specifics [12] 12.10.1 Storage Units and Sequences The size of the numeric storage units is 32 bits, unless the -s default64 option is specified, in which case the numeric storage unit is 64 bits. If the -s real64 or -s integer64 option is specified alone, or the -dp is specified in addition to -s default64 or -s real64, the relative sizes of the storage assigned for default intrinsic types do not conform to the standard. In this case, storage sequence associations involving variables declared with default intrinsic noncharacter types may be invalid and should be avoided. 12.10.2 Static and Stack Storage The Cray Fortran compiler allocates variables to storage according to the following criteria: • Variables in common blocks are always allocated in the order in which they appear in COMMON statements. • Data in modules are statically allocated. • User variables that are defined or referenced in a program unit, and that also appear in SAVE or DATA statements, are allocated to static storage, but not necessarily in the order shown in your source program. • Other referenced user variables are assigned to the stack. If -ev is specified on the Cray Fortran compiler command line, referenced variables are allocated to static storage. This allocation does not necessarily depend on the order in which the variables appear in your source program. • Compiler-generated variables are assigned to a register or to memory (to the stack or heap), depending on how the variable is used. Compiler-generated variables include DO-loop trip counts, dummy argument addresses, temporaries used in expression evaluation, argument lists, and variables storing adjustable dimension bounds at entries. • Automatic objects may be allocated to either the stack or to the heap, depending on how much stack space is available when the objects are allocated. • Heap or stack allocation can be used for TASK COMMON variables and some compiler-generated temporary data such as automatic arrays and array temporaries. • Unsaved variables may be assigned to a register by optimization and not allocated storage. • Unreferenced user variables not appearing in COMMON statements are not allocated storage. S–3901–80 237Cray Fortran Reference Manual 12.10.3 Dynamic Memory Allocation Many FORTRAN 77 programs contain a memory allocation scheme that expands an array in a common block located in central memory at the end of the program. This practice of expanding a blank common block or expanding a dynamic common block (sometimes referred to as overindexing) causes conflicts between user management of memory and the dynamic memory requirements of CLE libraries. It is recommended that you modify programs rather than expand blank common blocks, particularly when migrating from other environments. Figure 2 shows the structure of a program under the CLE operating systems in relation to expanding a blank common block. In both figures, the user area includes code, data, and common blocks. Figure 2. Memory Use Heap User area Without an expandable common block: Heap User area With an expandable common block: Dynamic area Address 0 12.11 Finalization A finalizable object in a module is not finalized in the event that there is no longer any active procedure referencing the module. A finalizable object that is allocated via pointer allocation is not finalized in the event that it later becomes unreachable due to all pointers to that object having their pointer association status changed. 238 S–3901–80Cray Fortran Implementation Specifics [12] 12.12 ALLOCATE Error Status If an error occurs during the execution of an ALLOCATE statement with a stat= specifier, subsequent items in the allocation list are not allocated. 12.13 DEALLOCATE Error Status If an error occurs during the execution of an DEALLOCATE statement with a stat= specifier, subsequent items in the deallocation list are not deallocated. 12.14 ALLOCATABLE Module Variable Status An unsaved allocatable module variable remains allocated if it is allocated when the execution of an END or RETURN statement results in no active program unit having access to the module. 12.15 Kind of a Logical Expression For an expression such as x1 op x2 where op is a logical intrinsic binary operator and the operands are of type logical with different kind type parameters, the kind type parameter of the result is the larger kind type parameter of the operands. 12.16 STOP Code Availability If a STOP code is specified in a STOP statement, its value is output to stderr when the STOP statement is executed. 12.17 Stream File Record Structure and Position A formatted file written with stream access may be later read as a record file. In that case, embedded newline characters (char(10)) indicate the end of a record and the terminating newline character is not considered part of the record. The file storage unit for a formatted stream file is a byte. The position is the ordinal byte number in the file; the first byte is position 1. Positions corresponding to newline characters (char(10)) that were inserted by the I/O library as part of record output do not correspond to positions of user-written data. S–3901–80 239Cray Fortran Reference Manual 12.18 File Unit Numbers The values of INPUT_UNIT, OUTPUT_UNIT, and ERROR_UNIT defined in the ISO_Fortran_env module are 100, 101, and 102, respectively. These three unit numbers are reserved and may not be used for other purposes. The files connected to these units are the same files used by the companion C processor for standard input (stdin), output (stdout), and error (stderr). An asterisk (*) specified as the unit for a READ statement specifies unit 100. An asterisk specified as the unit for a WRITE statement, and the unit for PRINT statements is unit 101. All positive default integer values are available for use as unit numbers. 12.19 OPEN Specifiers If the ACTION= specifier is omitted from an OPEN statement, the default value is determined by the protections associated with the file. If both reading and writing are permitted, the default value is READWRITE. If the ENCODING= specifier is omitted or specified as DEFAULT in an OPEN statement for a formatted file, the encoding used is ASCII. The case of the name specified in a FILE= specifier in an OPEN statement is significant. If the FILE= specifier is omitted, fort. is prepended to the unit number. If the RECL= specifier is omitted from an OPEN statement for a sequential access file, the default value for the maximum record length is 1024. If the file is connected for unformatted I/O, the length is measured in 8-bit bytes. The FORM= specifier may also be SYSTEM for unformatted files. If the ROUND= specifier is omitted from an OPEN statement, the default value is NEAREST. Specifying a value of PROCESSOR_DEFINED is equivalent to specifying NEAREST. If the STATUS= specifier is omitted or specified as UNKNOWN in an OPEN statement, the specification is equivalent to OLD if the file exists, otherwise, it is equivalent to NEW. 12.20 FLUSH Statement Execution of a FLUSH statement causes memory resident buffers to be flushed to the physical file. Output to the unit specified by ERROR_UNIT in the ISO_Fortran_env module is never buffered; execution of FLUSH on that unit has no effect. 240 S–3901–80Cray Fortran Implementation Specifics [12] 12.21 Asynchronous I/O The ASYNCHRONOUS= specifier may be set to YES to allow asynchronous I/O for a unit or file. Asynchronous I/O is used if the FFIO layer attached to the file provides asynchronous access. 12.22 REAL I/O of an IEEE NaN An IEEE NaN may be used as an I/O value for the F, E, D, or G edit descriptor or for list-directed or namelist I/O. 12.22.1 Input of an IEEE NaN The form of NaN is an optional sign followed by the string 'NAN' optionally followed by a hexadecimal digit string enclosed in parentheses. The input is case insensitive. Some examples are: NaN - quiet NaN nAN() - quiet NaN -nan(ffffffff) - quiet NaN NAn(7f800001) - signalling NaN NaN(ffc00001) - quiet NaN NaN(ff800001) - signalling NaN The internal value for the NaN becomes a quiet NaN if the hexadecimal string is not present or is not a valid NaN. A '+' or '-' preceding the NaN on input is used as the high order bit of the corresponding READ input list item. An explicit sign overrides the sign bit from the hexadecimal string. The internal value becomes the hexadecimal string if it represents an IEEE NaN in the internal data type. Otherwise, the form of the internal value is undefined. 12.22.2 Output of an IEEE NaN The form of an IEEE NaN for the F, E, D, or G edit descriptor or for list-directed or namelist output is: 1. If the field width w is absent, zero, or greater than (5 + 1/4 of the size of the internal value in bits), the output consists of the string 'NaN' followed by the hexadecimal representation of the internal value within a set of parentheses. An example of the output field is: NaN(7fc00000) S–3901–80 241Cray Fortran Reference Manual 2. If the field width w is at least 3 but less than (5 + 1/4 of the size of the internal value in bits), the string 'NaN' will be right-justified in the field with blank fill on the left. 3. If the field width w is 1 or 2, the field is filled with asterisks. The output field has no '+' or '-'; the sign is contained in the hexadecimal string. To get the same internal value for a NaN, write it with a list-directed write statement and read it with a list-directed read statement. To write and then read the same NaN, the field width w in D, E, F, or G must be at least the number of hexadecimal digits of the internal datum plus 5. REAL(4): w >= 13 REAL(8): w >= 21 REAL(16): w >= 37 12.23 List-directed and NAMELIST Output Default Formats The length of the output value in NAMELIST and list-directed output depends on the value being written. Blanks and unnecessary trailing zeroes are removed unless the -w option to the assign command is specified, which turns off this compression. By default, full-precision printing is assumed unless a precision is specified by the LISTIO_PRECISION environment variable (for more information about the LISTIO_PRECISION environment variable, see LISTIO_PRECISION on page 89). The form of list-directed and NAMELIST output can be changed by using the assign command with one of the following options. Table 24. List-directed and NAMELIST Assign Environment Options assign Option Effect -S Suppress comma-delimited output; use blank spaces instead -W Disable compression of floating-point values -y Disable the repeat-count form; write as many copies of the value as needed -U Set all three of the above 242 S–3901–80Cray Fortran Implementation Specifics [12] For example, consider this code: integer(4), dimension(5) :: ia real(4), dimension(5) :: ra ia = 102 ra = 200.10 NAMELIST/TNAMEL/ia,ra write(6,TNAMEL) print *, ' ia=',ia print *, ' ra=',ra print *, iarray, rarray end When compiled and executed with the default settings, it produces the following output: &TNAMEL RA = 2*200.100006, IA = 2*102 ia = 2*102 ra = 2*200.100006 2*102, 2*200.100006 However, if you set the FILENV environment variable to a file and use the assign -U command to change the output behavior, as shown below: % setenv FILENV ASGTMP % assign -U on g:sf The same code now produces the following output: &TNAMEL RA = 200.1000 200.1000 IA = 102 102 / ia = 102 102 ra = 200.1000 200.1000 102 102 200.1000 200.1000 For more information about the assign command and Assign Environment, see Chapter 13, Enhanced I/O: Using the Assign Environment on page 245. 12.24 Random Number Generator A multiplicative congruential generator with period 2**46 is used to produce the output of the RANDOM_NUMBER intrinsic subroutine. The seed array contains one 64-bit integer value. 12.25 Timing Intrinsics A call to the SYSTEM_CLOCK intrinsic subroutine with the COUNT argument present translates into the inline instructions that directly access the hardware clock register. See the description of the -e s and -d s command line options for information about the values returned for the count and count rate. For fine-grained timing, Cray recommends using a 64-bit COUNT argument. S–3901–80 243Cray Fortran Reference Manual The CPU_TIME subroutine obtains the value of its argument from the getrusage system call. Its execution time is significantly longer than for the SYSTEM_CLOCK routine, but the values returned are closer to those used by system accounting utilities. 12.26 IEEE Intrinsic Modules The IEEE intrinsics modules IEEE_EXCEPTIONS, IEEE_ARITHMETIC, and IEEE_FEATURES are supplied. Denormal numbers are not supported on Cray hardware. The IEEE_SUPPORT_DENORMAL inquiry function returns .false. for all kinds of arguments. At the start of program execution, all floating point exception traps are disabled. 244 S–3901–80Enhanced I/O: Using the Assign Environment [13] Fortran programs often need the ability to alter details of a file connection, such as device residency, an alternative file name, a file space allocation scheme or structure, or data conversion properties. These file connection details taken together comprise the assign environment, and they can be modified by using the assign command and assign library interface. The assign environment can also be accessed from C/C++ by using the ffassign library interface. For more information, see the assign(1), assign(3f), and ffassign(3c) man pages. 13.1 Understanding the assign Environment The assign command information is stored in the assign environment file, .assign, or in a shell environment variable. To begin using the assign environment to control a program's I/O behavior, follow these steps. 1. Set the FILENV environment variable to the desired path. set FILENV environment-file 2. Run the assign command to define the current assign environment. assign arguments assign-object For example: assign -F cachea g:su 3. Run your program. 4. If you are not satisfied with the I/O performance observed during program execution, return to step 2, use the assign command to adjust the assign environment, and try again. The assign command passes information to Fortran open statements and to the ffopen routine to identify the following elements: • A list of unit numbers • File names • File name patterns that have attributes associated with them S–3901–80 245Cray Fortran Reference Manual The assign object is the file name, file name pattern, unit number, or type of I/O open request to which the assign environment applies. When the unit or file is opened from Fortran, the environment defined by the assign command is used to establish the properties of the connection. 13.1.1 Assign Objects and Open Processing The I/O library routines apply options to a file connection for all related assign objects. If the assign object is a unit, the application of options to the unit occurs whenever that unit is connected. If the assign object is a file name or pattern, the application of options to the file connection occurs whenever a matching file name is opened from a Fortran program. When any of the library I/O routines opens a file, it uses the specified assign environment options for any assign objects that apply to the open request. Any of the following assign objects or categories can apply to a given open request. Table 25. Assign Object Open Processing Assign-object Applies To g:all All open requests g:su Open sequential unformatted g:du Open direct unformatted g:sf Open sequential formatted g:df Open direct formatted g_ff ffopen u:unit-number Open unit-number p:pattern When a file whose name matches pattern is opened. The assign environment can contain only one p:assign-object that matches the current open file. The exception is that the p:%pattern (which uses the % wildcard character) is silently ignored if a more specific pattern also matches the current file name being opened. f:filename Whenever file filename is opened. Options from the assign objects in these categories are collected to create the complete set of options used for any particular open. The options are collected in the listed order, with options collected later in the list of assign objects overriding those collected earlier. 246 S–3901–80Enhanced I/O: Using the Assign Environment [13] 13.1.2 assign Command Syntax Here is the syntax for the assign command: assign [-I] [-O] [-a actualfile] [-b bs] [-f fortstd] [-m setting] [-s ft] [-t] [-u bufcnt] [-y setting] [-B setting] [-C charcon] [-D fildes] [-F spec[,specs]] [-N numcon] [-R] [-S setting] [-T setting] [-U setting] [-V] [-W setting] [-Y setting] [-Z setting] assign-object The following specifications cannot be used with any other options: assign -R [assign-object] assign -V [assign-object] A summary of the command options follows. For details, see the assign(1) and intro_ffio(3f) man pages. Control options: -I Specifies an incremental use of assign. All attributes are added to the attributes already assigned to the current assign-object. This option and the -O option are mutually exclusive. -O Specifies a replacement use of assign. This is the default control option. All currently existing assign attributes for the current assign-object are replaced. This option and the -I option are mutually exclusive. -R Removes all assign attributes for assign-object. If assign-object is not specified, all currently assigned attributes for all assign-objects are removed. -V Views attributes for assign-object. If assign-object is not specified, all currently assigned attributes for all assign-objects are printed. Attribute options: -a actualfile The file= specifier or the actual file name. -b bs Library buffer size in 4096-byte (512-word) blocks. -f fortstd Specifies compatibility with a Fortran standard, where fortstd is either 2003 for the current Cray Fortran or 95 for Cray Fortran 95. If the value 95 is set, the list-directed and namelist output of a floating point will remain 0.E+0. S–3901–80 247Cray Fortran Reference Manual -m setting Special handling of a direct access file that will be accessed concurrently by several processes or tasks. Special handling includes skipping the check that only one Fortran unit be connected to a unit, suppressing file truncation to true size by the I/O buffering routines, and ensuring that the file is not truncated by the I/O buffering routines. Enter either on or off for setting. -s ft File type. Enter text, cos, blocked, unblocked, u, sbin, or bin for ft. The default is text. -t Temporary file. -u bufcnt Buffer count. Specifies the number of buffers to be allocated for a file. -y setting Suppresses repeat counts in list-directed output. setting can be either on or off. The default setting is off. -B setting Activates or suppresses the passing of the O_DIRECT flag to the open(2) system call. Enter either on or off for setting. This is an important feature for I/O optimization; if this is on, it enables reads and writes directly to and from the user program buffer. -C charcon Character set conversion information. Enter ascii, or ebcdic for charcon. If you specify the -C option, you must also specify the -F option. -D fildes Specifies a connection to a standard file. Enter stdin, stdout, or stderr for fildes. -F spec [,specs] Flexible file I/O (FFIO) specification. See the assign(1) man page for details about allowed values for spec and for details about hardware platform support. See the intro_ffio(3f) man page for details about specifying the FFIO layers. -N numcon Foreign numeric conversion specification. See the assign(1) man page for details about allowed values for numcon and for details about hardware platform support. -S setting Suppresses use of a comma as a separator in list-directed output. Enter either on or off for setting. The default setting is off. -T setting Activates or suppresses truncation after write for sequential Fortran files. Enter either on or off for setting. -U setting Produces a non-UNICOS form of list-directed output. This is a global setting that sets the value for the -y, -S, and -W options. Enter either on or off for setting. The default setting is off. 248 S–3901–80Enhanced I/O: Using the Assign Environment [13] -W setting Suppresses compressed width in list-directed output. Enter either on or off for setting. The default setting is off. -Y setting Skips unmatched namelist groups in a namelist input record. Enter either on or off for setting. The default setting is on. -Z setting Recognizes –0.0 for IEEE floating-point systems and writes the minus sign for edit-directed, list-directed, and namelist output. Enter either on or off for setting. The default setting is on. assign-object Specify either a file name or a unit number for assign-object. The assign command associates the attributes with the file or unit specified. These attributes are used during the processing of Fortran open statements or during implicit file opens. Use one of the following formats for assign-object: • f:filename • g:io-type, where io-type can be su, sf, du, df, or ff (for example, g:ff for ffopen(3C) • p:pattern (for example, p:file%) • u:unit-number (for example, u:9) • filename When the p:pattern form is used, the % and _ wildcard characters can be used. The % matches any string of 0 or more characters. The _ matches any single character. The % performs like the * when doing file name matching in shells. However, the % character also matches strings of characters containing the / character. 13.1.3 Using the Library Routines The assign, asnunit, asnfile, and asnrm routines can be called from a Fortran program to access and update the assign environment. The assign routine provides an easy interface to assign processing from a Fortran program. The asnunit and asnfile routines assign attributes to units and files, respectively. The asnrm routine removes all entries currently in the assign environment. The calling sequences for library routines are as follows: call assign (cmd, ier) call asnunit (iunit,astring,ier) call asnfile (fname,astring,ier) call asnrm (ier) S–3901–80 249Cray Fortran Reference Manual Where: cmd Fortran character variable containing a complete assign command in the format acceptable to the pxfsystem routine. ier Integer variable that is assigned the exit status on return from the library interface routine. iunit Integer variable or constant that contains the unit number to which attributes are assigned. astring Fortran character variable that contains any attribute options and option values from the assign command. Control options -I, -O, and -R can also be passed. fname Character variable or constant that contains the file name to which attributes are assigned. A status of 0 indicates normal return. A status of greater than 0 indicates a specific error status. Use the explain command to determine the meaning of the error status. The following calls are equivalent to the assign -s u f:file command: call assign('assign -s u f:file',ier) call asnfile('file','-s u',ier) The following call is equivalent to executing the assign -I -n 2 u:99 command: iun = 99 call asnunit(iun,'-i -n 2',ier) The following call is equivalent to executing the assign -R command: call asnrm(ier) 13.2 Tuning File Connection Behavior 13.2.1 Using Alternative File Names The -a option specifies the actual file name to which a connection is made. This option allows files to be created in different directories without changing the FILE= specifier on an OPEN statement. For example, consider the following assign command issued to open unit 1: assign -a /tmp/mydir/tmpfile u:1 250 S–3901–80Enhanced I/O: Using the Assign Environment [13] The program then opens unit 1 with any of the following statements: WRITE(1) variable ! implicit open OPEN(1) ! unnamed open OPEN(1,FORM='FORMATTED') ! unnamed open Unit 1 is connected to file /tmp/mydir/tmpfile. Without the -a attribute, unit 1 would be connected to file fort.1. When the -a attribute is associated with a file, any Fortran open that is set to connect to the file causes a connection to the actual file name. An assign command of the following form causes a connection to file $FILENV/joe: assign -a $FILENV/joe ftfile This is true when the following statement is executed in a program: OPEN(IUN,FILE='ftfile') If the following assign command is issued and in effect, any Fortran INQUIRE statement whose FILE= specification is foo refers to the file named actual instead of the file named foo for purposes of the EXISTS=, OPENED=, or UNIT= specifiers: assign -a actual f:foo If the following assign command is issued and in effect, the -a attribute does not affect INQUIRE statements with a UNIT= specifier: assign -a actual ftfile When the following OPEN statement is executed, INQUIRE(UNIT=n,NAME=fname) returns a value of ftfile in fname, as if no assign had occurred: OPEN(n,file='ftfile') The I/O library routines use only the actual file (-a) attributes from the assign environment when processing an INQUIRE statement. During an INQUIRE statement that contains a FILE= specifier, the I/O library searches the assign environment for a reference to the file name that the FILE= specifier supplies. If an assign-by-filename exists for the file name, the I/O library determines whether an actual name from the -a option is associated with the file name. If the assign-by-filename supplied an actual name, the I/O library uses that name to return values for the EXIST=, OPENED=, and UNIT= specifiers; otherwise, it uses the file name. The name returned for the NAME= specifier is the file name supplied in the FILE= specifier. The actual file name is not returned. S–3901–80 251Cray Fortran Reference Manual 13.2.2 Specifying File Structure A file structure defines the way records are delimited and how the end-of-file is represented. The assign command supports two mutually exclusive file structure options: • To select a structure using an FFIO layer, use assign -F • To select a structure explicitly, use assign -s Using FFIO layers is more flexible than selecting structures explicitly. FFIO allows nested file structures, buffer size specifications, and support for file structures not available through the -s option. You will also realize better I/O performance by using the -F option and FFIO layers. For more information about the -F option and FFIO layers, see Chapter 14, Using Flexible File I/O (FFIO) on page 265. The remainder of this section covers the -s option. Fortran sequential unformatted I/O uses four different file structures: f77 blocked structure, text structure, unblocked structure, and COS blocked structure. By default, the f77 blocked structure is used unless a file structure is selected at open time. If an alternative file structure is needed, the user can select a file structure by using the -s or -F option on the assign command. The -s and -F options are mutually exclusive. The following examples show how to use different assign command options to select different file structures. Structure assign Command F77 blocked assign -F f77 text assign -F text assign -s text unblocked assign -F system assign -s unblocked COS blocked assign -F cos assign -s cos 252 S–3901–80Enhanced I/O: Using the Assign Environment [13] The following examples show how to adjust blocking. • To select an unblocked file structure for a sequential unformatted file: IUN = 1 CALL ASNUNIT(IUN,'-s unblocked',IER) OPEN(IUN,FORM='UNFORMATTED',ACCESS='SEQUENTIAL') You can also use the assign -s u command to specify the unblocked file structure for a sequential unformatted file. When this option is selected, I/O is unbuffered. Each Fortran READ or WRITE statement results in a read or write system call such as the following: CALL ASNFILE('fort.1','-s u',IER) OPEN(1,FORM='UNFORMATTED',ACCESS='SEQUENTIAL') • To assign unit 10 a COS blocked structure: assign -s cos u:10 The full set of options allowed with the assign -s command are as follows: • bin (not recommended) • blocked • cos • sbin • text • unblocked Table 26. Fortran Access Methods and Options Access and Form assign -s ft Defaults assign -s ft Options Sequential unformatted, BUFFER IN and BUFFER OUT blocked / cos / f77 bin sbin u unblocked Direct unformatted unblocked bin sbin u unblocked Sequential formatted text blocked cos sbin/text Direct formatted text sbin/text S–3901–80 253Cray Fortran Reference Manual 13.2.2.1 Unblocked File Structure A file with an unblocked file structure contains undelimited records. Because it does not contain any record control words, it does not have record boundaries. The unblocked file structure can be specified for a file opened with either unformatted sequential access or unformatted direct access. It is the default file structure for a file opened as an unformatted direct-access file. Do not attempt to use a BACKSPACE statement to reposition a file with an unblocked file structure. Since record boundaries do not exist, you cannot reposition the file to a previous record. BUFFER IN and BUFFER OUT statements can specify a file having an unbuffered and unblocked file structure. If the file is specified with assign -s u, BUFFER IN and BUFFER OUT statements can perform asynchronous unformatted I/O. There are several ways to use the assign command to specify unblocked file structure. All ways result in a similar file structure but with different library buffering styles, use of truncation on a file, alignment of data, and recognition of an end-of-file record in the file. The following unblocked data file structure specifications are available: Specification Structure assign -s unblocked Library-buffered assign -F system No library buffering assign -s sbin Buffering that is compatible with standard I/O; for example, both library and system buffering The type of file processing for an unblocked data file structure depends on the assign -s ft option that is declared or assumed for a Fortran file. For more information about buffering, see Specifying Buffer Behavior on page 257. An I/O request for a file specified using the assign -s unblocked command does not need to be a multiple of a specific number of bytes. Such a file is truncated after the last record is written to the file. Padding occurs for files specified with the assign -s bin command and the assign -s unblocked command. Padding usually occurs when noncharacter variables follow character variables in an unformatted direct-access file. No padding is done in an unformatted sequential access file. An unformatted direct-access file created by a Fortran program on CLE systems contains records that are the same length. The end-of-file record is recognized in sequential-access files. 254 S–3901–80Enhanced I/O: Using the Assign Environment [13] 13.2.2.2 assign -s sbin File Processing You can use an assign -s sbin specification for a Fortran file opened with either unformatted direct access or unformatted sequential access. The file does not contain record delimiters. The file created for assign -s sbin in this instance has an unblocked data file structure and uses unblocked file processing. The assign -s sbin option can be specified for a Fortran file that is declared as formatted sequential access. Because the file contains records that are delimited with the new-line character, it is not an unblocked data file structure. It is the same as a text file structure. The assign -s sbin option is compatible with the standard C I/O functions. Note: Cray discourages the use of assign -s sbin because it typically yields poor I/O performance. If you cannot use an FFIO layer, using assign -s text for formatted files and assign -s unblocked for unformatted files usually produces better I/O performance than using assign -s sbin. 13.2.2.3 assign -s bin File Processing An I/O request for a file that is specified with assign -s bin does not need to be a multiple of a specific number of bytes. Padding occurs when noncharacter variables follow character variables in an unformatted record. The I/O library uses an internal buffer for the records. If opened for sequential access, a file is not truncated after each record is written to the file. 13.2.2.4 assign -s u File Processing The assign -s u command specifies undefined or unknown file processing. An assign -s u specification can be specified for a Fortran file declared as unformatted sequential or direct access. Because the file does not contain record delimiters, it has an unblocked data file structure. Both synchronous and asynchronous BUFFER IN and BUFFER OUT processing can be used with u file processing. Fortran sequential files declared by using assign -s u are not truncated after the last word written. The user must execute an explicit ENDFILE statement on the file. 13.2.2.5 text File Structure The text file structure consists of a stream of 8-bit ASCII characters. Every record in a text file is terminated by a newline character (\n, ASCII 012). Some utilities may omit the newline character on the last record, but the Fortran library treats such an occurrence as a malformed record. This file structure may be specified for a file that is declared as either formatted sequential access or formatted direct access. It is the default file structure for formatted sequential access and formatted direct access files. S–3901–80 255Cray Fortran Reference Manual The assign -s text command specifies the library-buffered text file structure. Both library and system buffering are done for all text file structures. An I/O request for a file using assign -s text does not need to be a multiple of a specific number of bytes. You cannot use BUFFER IN and BUFFER OUT statements with this structure. You can use a BACKSPACE statement to reposition a file with this structure. 13.2.2.6 cos or blocked File Structure The cos or blocked file structure uses control words to mark the beginning of each sector and to delimit each record. You can specify this file structure for a file that is declared as unformatted sequential access. Synchronous BUFFER IN and BUFFER OUT statements can create and access files with this file structure. You can specify this file structure with one of the following assign commands: assign -s cos assign -s blocked assign -F cos assign -F blocked These four assign commands result in the same file structure. An I/O request on a blocked file is library buffered. In a cos file structure, one or more ENDFILE records are allowed. BACKSPACE statements can be used to reposition a file with this structure. A blocked file is a stream of words that contains control words called Block Control Word (BCW) and Record Control Words (RCW) to delimit records. Each record is terminated by an EOR (end-of-record) RCW. At the beginning of the stream, and every 512 words thereafter (including any RCWs), a BCW is inserted. An end-of-file (EOF) control word marks a special record that is always empty. Fortran considers this empty record to be an endfile record. The end-of-data (EOD) control word is always the last control word in any blocked file. The EOD is always immediately preceded by either an EOR, or by an EOF and a BCW. Each control word contains a count of the number of data words to be found between it and the next control word. In the case of the EOD, this count is 0. Because there is a BCW every 512 words, these counts never point forward more than 511 words. A record always begins at a word boundary. If a record ends in the middle of a word, the rest of that word is zero filled; the ubc field of the closing RCW contains the number of unused bits in the last word. 256 S–3901–80Enhanced I/O: Using the Assign Environment [13] The following illustration and table is a representation of the structure of a BCW. m unused bdf unused bn fwi (4) (7) (1) (19) (24) (9) Field Bits Description m 0–3 Type of control word; 0 for BCW bdf 11 Bad Data flag (1-bit, 1=bad data) bn 31–54 Block number (modulo 2 24 ) fwi 55–63 Forward index; the number of words to the next control word The following illustration and table is a representation of the structure of an RCW. m ubc tran bdf srs unused pfi pri fwi (4) (6) (1) (1) (1) (7) (20) (15) (9) Field Bits Description m 0–3 Type of control word; 10 8 for EOR, 16 8 for EOF, and 17 8 for EOD ubc 4–9 Unused bit count; number of unused low-order bits in last word of previous record tran 10 Transparent record field (unused) bdf 11 Bad data flag (unused) srs 12 Skip remainder of sector (unused) pfi 20–39 Previous file index; offset modulo 2 20 to the block where the current file starts (as defined by the last EOF) pri 40–54 Previous record index; offset modulo 2 15 to the block where the current record starts fwi 55–63 Forward index; the number of words to the next control word 13.2.3 Specifying Buffer Behavior A buffer is a temporary storage location for data while the data is being transferred. Buffers are often used for the following purposes: • Small I/O requests can be collected into a buffer, and the overhead of making many relatively expensive system calls can be greatly reduced. S–3901–80 257Cray Fortran Reference Manual • Many data file structures such as cos contain control words. During the write process, a buffer can be used as a work area where control words can be inserted into the data stream (a process called blocking). The blocked data is then written to the device. During the read process, the same buffer work area can be used to remove the control words before passing the data on to the user (called deblocking). • When data access is random, the same data may be requested many times. A cache is a buffer that keeps old requests in the buffer in case these requests are needed again. A cache that is sufficiently large or efficient can avoid a large part of the physical I/O by having the data ready in a buffer. When the data is often found in the cache buffer, it is referred to as having a high hit rate. For example, if the entire file fits in the cache and the file is present in the cache, no more physical requests are required to perform the I/O. In this case, the hit rate is 100%. • Running the I/O devices and the processors in parallel often improves performance; therefore, it is useful to keep processors busy while data is being moved. To do this when writing, data can be transferred to the buffer at memory-to-memory copy speed. Use an asynchronous I/O request. The control is then immediately returned to the program, which continues to execute as if the I/O were complete (a process called write-behind). A similar process called read-ahead can be used while reading; in this process, data is read into a buffer before the actual request is issued for it. When it is needed, it is already in the buffer and can be transferred to the user at very high speed. • When direct I/O is enabled (assign -B on), data is staged in the system buffer cache. While this can yield improved performance, it also means that performance is affected by program competition for system buffer cache. To minimize this effect, avoid public caches when possible. • In many cases, the best asynchronous I/O performance can be realized by using the FFIO cachea layer (assign -F cachea). This layer supports read-ahead, write-behind, and improved cache reuse. The size of the buffer used for a Fortran file can have a substantial effect on I/O performance. A larger buffer size usually decreases the system time needed to process sequential files. However, large buffers increase a program's memory usage; therefore, optimizing the buffer size for each file accessed in a program on a case-by-case basis can help increase I/O performance and minimize memory usage. The -b option on the assign command specifies a buffer size, in blocks, for the unit. The -b option can be used with the -s option, but it cannot be used with the -F option. Use the -F option to provide I/O path specifications that include buffer sizes; the -b, and -u options do not apply when -F is specified. For more information about the selection of buffer sizes, see the assign(1) man page. 258 S–3901–80Enhanced I/O: Using the Assign Environment [13] The following examples of buffer size specification illustrate using the assign -b and assign -F options: • If unit 1 is a large sequential file for which many Fortran READ or WRITE statements are issued, you can increase the buffer size to a large value, using the following assign command: assign -b buffer-size u:buffer-count • If the file foo is a small file or is accessed infrequently, you can minimize the buffer size using the following assign command: assign -b 1 f:foo 13.2.3.1 Default Buffer Sizes The Fortran I/O library automatically selects default buffer sizes according to file access type as shown in Table 27. You can override the defaults by using the assign command. The following subsections describe the default buffer sizes on various systems. Note: One block is 4,096 bytes on CLE systems. Table 27. Default Buffer Sizes for Fortran I/O Library Routines Access Type Default Buffer Size Sequential formatted 16 blocks (65,536 bytes) Sequential unformatted 128 blocks (524,288 bytes) Direct formatted The smaller of: • The record length in bytes + 1 • 16 blocks (65,536 bytes) Direct unformatted The larger of: • The record length • 16 blocks (65,536 bytes) Four buffers of default size are allocated. For more information, see the description of the cachea layer in the intro_ffio(3F) man page. 13.2.3.2 Library Buffering The term library buffering refers to a buffer that the I/O library associates with a file. When a file is opened, the I/O library checks the access, form, and any attributes declared on the assign command to determine the type of processing that should be used on the file. Buffers are an integral part of the processing. S–3901–80 259Cray Fortran Reference Manual If the file is assigned with one of the following assign options, library buffering is used: -s blocked -F spec (buffering as defined by spec) -s cos -s bin -s unblocked The -F option specifies flexible file I/O (FFIO), which uses library buffering if the specifications selected include a need for buffering. In some cases, more than one set of buffers might be used in processing a file. For example, the -F bufa,cos option specifies two library buffers for a read of a blank compressed COS blocked file. One buffer handles the blocking and deblocking associated with the COS blocked control words, and the second buffer is used as a work area to process blank compression. In other cases (for example, -F system), no library buffering occurs. 13.2.3.3 System Cache The operating system uses a set of buffers in kernel memory for I/O operations. These are collectively called the system cache. The I/O library uses system calls to move data between the user memory space and the system buffer. The system cache ensures that the actual I/O to the logical device is well formed, and it tries to remember recent data in order to reduce physical I/O requests. The following assign command options can be expected to use system cache: -s sbin -F spec (FFIO, depends on spec) For the assign -F cachea command, a library buffer ensures that the actual system calls are well formed and the system buffer cache is bypassed. This is not true for the assign -s u option. If you plan to use assign -s u to bypass the system cache, all requests must be well formed. 13.2.3.4 Unbuffered I/O The simplest form of buffering is none at all; this unbuffered I/O is known as direct I/O. For sufficiently large, well-formed requests, buffering is not necessary and can add unnecessary overhead and delay. The following assign command specifies unbuffered I/O: assign -s u ... Use the assign command to bypass both library buffering and the system cache for all well-formed requests. The data is transferred directly between the user data area and the logical device. Requests that are not well formed will result in I/O errors. 260 S–3901–80Enhanced I/O: Using the Assign Environment [13] 13.2.4 Specifying Foreign File Formats The Fortran I/O library can read and write files with record blocking and data formats native to operating systems from other vendors. The assign -F command specifies a foreign record blocking; the assign -C command specifies the type of character conversion; the -N option specifies the type of numeric data conversion. When -N or -C is specified, the data is converted automatically during the processing of Fortran READ and WRITE statements. For example, assume that a record in file fgnfile contains the following character and integer data: character*4 ch integer int open(iun,FILE='fgnfile',FORM='UNFORMATTED') read(iun) ch, int Use the following assign command to specify foreign record blocking and foreign data formats for character and integer data: assign -F ibm.vbs -N ibm -C ebcdic fgnfile One of the most common uses of the assign command is to swap big-endian for little-endian files. To access big-endian unformatted files on a little-endian system such as the Cray XE, use the following command: assign -N swap_endian fgnfile This assumes the file is a normal f77 unformatted file with 32-bit record control images with a byte count. The library routines swap both the control images and the data when reading or writing the file. If all unformatted sequential files are the opposite endianness, use the following command: assign -N swap_endian g:su 13.2.5 Specifying Memory Resident Files The assign -F mr command specifies that a file will be memory resident. Because the mr flexible file I/O layer does not define a record-based file structure, it must be nested beneath a file structure layer when record blocking is needed. For example, if unit 2 is a sequential unformatted file that is to be memory resident, the following Fortran statements connect the unit: CALL ASNUNIT (2,'-F cos,mr',IER) OPEN(2,FORM='UNFORMATTED') The -F cos,mr specification selects COS blocked structure with memory residency. S–3901–80 261Cray Fortran Reference Manual 13.2.6 Using and Suppressing File Truncation The assign -T option activates or suppresses truncation after the writing of a sequential Fortran file. The -T on option specifies truncation; this behavior is consistent with the Fortran standard and is the default setting for most assign -s fs specifications. The assign(1) man page lists the default setting of the -T option for each -s fs specification. It also indicates if suppression or truncation is allowed for each of these specifications. FFIO layers that are specified by using the -F option vary in their support for suppression of truncation with -T off. Figure 3 summarizes the available access methods and the default buffer sizes. Figure 3. Access Methods and Default Buffer Sizes Blocked Unblocked Access method assign option Blocked -s cos Text -s text Undef -s u Binary -s bin Unblocked -s unblocked Buffer size for default Formatted sequential I/O WRITE(9,20) PRINT Valid Default 16 Formatted direct I/O WRITE(9,20,REC=) Unformatted sequential I/O WRITE(9) Unformatted direct I/O WRITE(9,REC=) Buffer in/buffer out Control words Yes NEWLINE No Library buffering System cached BACKSPACE Record size Default library buffer size* 48 16 16 Any Varies Valid Valid Valid Valid Default Valid Valid Valid Valid Valid Valid Valid Valid Valid Valid Default 16 128 No No Yes Yes Yes Yes No min(recl+1, 8) bytes max(16, recl) blocks Any Any Any Yes No Yes Yes No No† No†† 8*n No No Valid * † †† Cached if not well-formed No guarantee when physical size not 512 words * In units of 4096 bytes, unless otherwise specified 16 None Blocked -F f77 Yes 16 Any Yes Yes Yes Valid Default Valid Default 262 S–3901–80Enhanced I/O: Using the Assign Environment [13] 13.3 Defining the Assign Environment File The assign command information is stored in the assign environment file. The location of the active assign environment file must be provided by setting the FILENV environment variable to the desired path and file name. 13.4 Using Local Assign Mode The assign environment information is usually stored in the .assign environment file. Programs that do not require the use of the global .assign environment file can activate local assign mode. If you select local assign mode, the assign environment will be stored in memory. Thus, other processes can not adversely affect the assign environment used by the program. The ASNCTL routine selects local assign mode when it is called by using one of the following command lines: CALL ASNCTL('LOCAL',1,IER) CALL ASNCTL('NEWLOCAL',1,IER) Example 5. Local assign mode In the following example, a Fortran program activates local assign mode and then specifies an unblocked data file structure for a unit before opening it. The -I option is passed to ASNUNIT to ensure that any assign attributes continue to have an effect at the time of file connection. C Switch to local assign environment CALL ASNCTL('LOCAL',1,IER) IUN = 11 C Assign the unblocked file structure CALL ASNUNIT(IUN,'-I -s unblocked',IER) C Open unit 11 OPEN(IUN,FORM='UNFORMATTED') If a program contains all necessary assign statements as calls to ASSIGN, ASNUNIT, and ASNFILE, or if a program requires total shielding from any assign commands, use the second form of a call to ASNCTL, as follows: C New (empty) local assign environment CALL ASNCTL('NEWLOCAL',1,IER) IUN = 11 C Assign a large buffer size CALL ASNUNIT(IUN,'-b 336',IER) C Open unit 11 OPEN(IUN,FORM='UNFORMATTED') S–3901–80 263Cray Fortran Reference Manual 264 S–3901–80Using Flexible File I/O (FFIO) [14] 14.1 Understanding FFIO The flexible file I/O (FFIO) system is based on the concept that for all I/O, a series of processing steps must be performed in order to transfer the user data between the user's memory and the desired I/O device. I/O can be the slowest part of a computational process and the speed of I/O access methods varies depending on computational processes, but by using FFIO, it is often possible to enhance a program's I/O performance without modifying or recompiling source code. Figure 4 shows the typical flow of data from the user's variables to and from the I/O device. Figure 4. Typical Data Flow Kernel job User’s System call Think of each box as a stopover point for the data, and each transition between stopovers as a processing step. The actual I/O path can skip one or more steps in this process, depending on the I/O features being used at a given point in a given program. Each transition has benefits and costs, and different applications may use the I/O system in different ways. For example, if I/O requests are large, the library buffer is probably unnecessary, because the main use of the library buffer is to reduce the number of system calls by consolidating smaller requests. To achieve better I/O throughput with large I/O requests, do not use library buffering. On the other hand, if I/O requests are small, then using the library buffer improves performance by eliminating the overhead associated with making a system call for each I/O request. S–3901–80 265Cray Fortran Reference Manual The assign environment and FFIO enable you to modify the I/O process for existing programs without changing or recompiling source code. The difference is that the assign command lets you modify the total I/O path, by establishing an overall I/O environment, while the FFIO system lets you specify I/O behavior at each stopover point along the path. To specify FFIO layers, use the assign -F command with a comma-delimited list of FFIO specifications. For example: assign -F spec1,spec2,spec3... Each spec in the list is a processing step that requests one I/O layer, or logical grouping of layers. The layer specifies the operations performed on the data as it is passed between the user and the I/O device. A layer refers to the specific type of processing being done. In some cases, the name corresponds directly to the name of one layer. In other cases, however, specifying one layer invokes the routines used to pass the data through multiple layers. See the intro_ffio(3f) man page for details about using the assign command -F option. Processing steps are ordered as if the -F side (the left side) is the user and the system/device is the right side, as in the following example: assign -F user,bufa,system With this specification, a WRITE operation first performs the user operation on the data, then performs the bufa operation, and then sends the data to the system. In a READ operation, the process is performed from right to left. The data moves from the system to the user. The layers closest to the user are higher-level layers; those closer to the system are lower-level layers. The FFIO system has an internal model of the world of data, which it maps to any given actual logical file type. The following four concepts are essential to understanding the inner workings of the layers. Concept Definition Data Data is a stream of bits. Record marks End-of-record (EOR) marks are boundaries between logical records. File marks End-of-file (EOF) marks are special types of record marks that exist in some file formats. End-of-data (EOD) An end-of-data (EOD) is a point immediately beyond the last data bit, EOR, or EOF in the file. All files are streams of 0 or more bits that may contain record and/or file marks. 266 S–3901–80Using Flexible File I/O (FFIO) [14] Individual layers have varying rules about which of these things can appear and in which order they can appear in a file. Both Fortran programmers and C programmers can use FFIO. Fortran users can use the assign command to specify FFIO options, while C users use FFIO layers by calling the FFIO routines directly. (See the ffopen(3c), ffread(3c), and ffwrite(3c) man pages.) You can use FFIO with the Fortran I/O forms listed in the following table. For each form, the equivalent assign command is shown. Fortran I/O Form Equivalent assign Command Buffer I/O assign -F f77 Unformatted sequential assign -F f77 Unformatted direct access assign -F cache Formatted sequential assign -F text Namelist assign -F text List-directed assign -F text 14.2 Using FFIO Layers The assign -F command specification list defines all the processing steps the I/O system performs. If assign -F is specified, any default processing is overridden. For example, unformatted sequential I/O is assigned a default structure of f77, which is the same as is used if the -F f77 option is specified. The FFIO system provides detailed control over I/O processing requests. However, to effectively use the f77 option (or any FFIO option), you must understand the I/O processing details. For example, suppose you are making large I/O requests and do not require buffering or blocking. You can specify: assign -F system The system layer is a generic system interface that chooses an appropriate layer for your file. If the file is on a disk, it chooses the syscall layer, which maps each user I/O request directly to the corresponding system call. A Fortran READ statement is mapped to one or more read system calls and a Fortran WRITE statement to one or more write system calls. If you want your file to be F77 blocked (the default blocking for Fortran unformatted I/O), specify: assign -F f77 S–3901–80 267Cray Fortran Reference Manual If you want your file to be COS blocked, specify: assign -F cos Note: In all assign -F specifications, the system layer is the implied last layer. The above example is functionally identical to assign -F cos,system. These two specifications request that each WRITE request first be blocked (blocking adds control words to the data in the file to delimit records), and then the f77 layer sends the blocked data to the system layer. The system layer passes the data to the device. The process is reversed for READ requests. For these requests, the system layer first retrieves blocked data from the file, and then the blocked data is passed to the next higher layer (the f77 layer), where it is deblocked. The deblocked data is then presented to the user. 14.2.1 Available I/O Layers Several different layers are available for the spec argument. Each layer invokes one or more layers, which then handle the data they are given in the appropriate manner. For example, the syscall layer essentially passes each request to an appropriate system call. The mr layer tries to hold an entire file in a buffer that can change size as the size of the file changes; it also limits actual I/O to lower layers so that I/O occurs only at open, close, and overflow. Table 28 defines the classes you can specify for the spec argument to the assign -F option. For detailed information about each layer, see Chapter 15, FFIO Layer Reference on page 279. Table 28. FFIO Layers Layer Function bufa Asynchronous buffering layer cache Memory-cached I/O cachea Asynchronous memory-cached I/O cos or blocked COS blocking; this is the default for Fortran sequential unformatted I/O on UNICOS and UNICOS/mk systems event I/O monitoring layer f77 FORTRAN record blocking; this is the default for Fortran sequential unformatted I/O on CLE systems and the common blocking format used by most FORTRAN compilers 268 S–3901–80Using Flexible File I/O (FFIO) [14] Layer Function fd File descriptor open global Distributed cache layer for MPI, SHMEM, OpenMP, and Coarray Fortran ibm IBM file formats mr Memory-resident file handlers null Syntactic convenience for users (does nothing) site User-defined site-specific layer syscall System call I/O system Generic system interface text Newline separated record formats user User-defined layer vms VAX/VMS file formats 14.2.2 Specifying Layered I/O Options You can modify the behavior of each I/O layer. The following spec format shows how to specify a class and one or more opt and num fields: class.opt1.opt2:num1:num2:num3 For class, you can specify one of the layers listed in Table 28. Each layer has a different set of options and numeric parameters, because each layer performs different duties. The following rules apply to the spec argument: • The class and opt fields are case-insensitive. For example, the following two specs are identical: Ibm.VBs:100:200 IBM.vbS:100:200 • The opt and num fields are usually optional, but sufficient separators must be specified as placeholders to eliminate ambiguity. For example, the following specs are identical: cos..::40, cos.::40 cos::40 In this example, opt1, opt2, num1, and num2 can assume default values. • To specify more than one spec, use commas between specs. Within each spec you can specify more than one opt and num. Use periods between opt fields, and colons between num fields. S–3901–80 269Cray Fortran Reference Manual The following options all have the same effect, specifying the vms layer and setting the initial allocation to 100 blocks: -F vms:100 -F vms.:100 -F vms..:100 The following option contains one spec for an vms layer that has an opt field of scr (which requests scratch file behavior): -F vms.scr The following option requests two classes with no opts: -F f77,vms The following option contains two specs and requests two layers: cos and vms. The cos layer has no options; the vms layer has options scr and ovfl, which specify that the file is a scratch file that is allowed to overflow and that the maximum allocation is 1000 sectors: -F cos,vms.scr.ovfl::1000 When possible, the default settings of the layers are set so that optional fields are seldom needed. 14.3 Using FFIO with Common File Structures 14.3.1 Reading and Writing Text Files Use the fdcp command to copy files while converting record blocking. Most human-readable files are in text format; this format contains records comprised of ASCII characters with each record terminated by an ASCII line-feed character, which is the newline character in UNIX. The FFIO specification that selects this file structure is assign -F text. The FFIO package is seldom required to handle text files. In the following types of cases, however, using FFIO may be necessary: • Optimizing text file access to reduce I/O wait time • Handling multiple EOF records in text files • Converting data files to and from other formats I/O speed is important when optimizing text file access. Using assign -F text is expensive in terms of processor time but enables you to use memory-resident files, which may reduce or eliminate I/O wait time. 270 S–3901–80Using Flexible File I/O (FFIO) [14] The FFIO system can also process text files with embedded EOF records. The ~e string alone in a text record is used as an EOF record. Editors such as sed and other standard utilities can process these files, but this processing is sometimes easier with FFIO. The text layer is useful in conjunction with the fdcp command. The text layer provides a standard output format. Many forms of data that are not considered foreign are sometimes encountered in a heterogeneous computing environment: if a record format can be described with an FFIO specification, it usually can be converted to text format by using a script similar to the following example: OTHERSPEC=$1 INFILE=$2 OUTFILE=$3 assign -F ${OTHERSPEC} ${INFILE} assign -F text ${OUTFILE} fdcp ${INFILE} ${OUTFILE} For example, if your script is named to.text, you would invoke it as follows: % to.text cos data_cos data_text 14.3.2 Reading and Writing Unblocked Files The simplest data file format is the binary stream or unblocked data. It contains no record marks, file marks, or control words. This is usually the fastest way to move large amounts of data because it involves a minimal amount of processor and system overhead. The FFIO package provides several layers designed specifically to handle a binary stream of data. These layers are syscall, mr, bufa, cache, cachea, and global. These layers behave the same from the user's perspective, but use different system resources. The unblocked binary stream is usually used for unformatted data transfer; it is not usually useful for text files or for when record boundaries or backspace operations are required. The complete burden is placed on the application to know the format of the file and the structure and type of the data it contains. This lack of structure allows flexibility. For example, a file declared with one of these layers can be manipulated as a direct-access file with any record length. In this context fdcp can be called to do the equivalent of the cp command, but only if the input file is a binary stream, or used to remove blocking information, but only if the output file is a binary stream. S–3901–80 271Cray Fortran Reference Manual 14.3.3 Reading and Writing Fixed-length Records The most common use for fixed-length record files is for Fortran direct access. Both unformatted and formatted direct-access files use a form of fixed-length records. The simplest way to handle these files with the FFIO system is with binary stream layers, such as system, syscall, cache, cachea, global, and mr. These layers allow any requested pattern of access and also work with direct-access files. The syscall and system layers, however, are unbuffered and do not give optimal performance for small records. The FFIO system also directly supports some fixed-length record formats. 14.3.4 Reading and Writing Blocked Files The f77 blocking format is the default file structure for all Fortran sequential unformatted files. The f77 layer is provided to handle these files. The f77 layer is the default file structure on Cray systems. If you specify another layer, such as mr, you may have to specify a f77 layer to get f77 blocking. 14.4 Tips for Enhancing I/O Performance FFIO can be used to enhance performance in a program without changing or recompiling the source code. 14.4.1 Buffer Size Considerations In the FFIO system, buffering is the responsibility of the individual layers; therefore, you must understand the individual layers in order to control the use and size of buffers. The cos layer has high payoff potential to the user who wants to extract top performance by manipulating buffer sizes. As the following example shows, the cos layer accepts a buffer size as the first numeric parameter: assign -F cos:42 u:1 If the buffer is sufficiently large, the cos layer also lets you keep an entire file in the buffer and avoid almost all I/O operations. 14.4.2 Removing Blocking I/O optimization usually consists of reducing overhead. One part of the overhead in doing I/O is the processor time spent in record blocking. For many files in many programs, this blocking is unnecessary. If this is the case, the FFIO system can be used to deselect record blocking and thus obtain performance advantages. 272 S–3901–80Using Flexible File I/O (FFIO) [14] The following layers offer unblocked data transfer: Layer Definition syscall System call I/O bufa Buffering layer cachea Asynchronous cache layer cache Memory-resident buffer cache global SHMEM and MPI cache layer mr Memory-resident (MR) I/O You can use any of these layers alone for any file that does not require the existence of record boundaries. This includes applications written in C that require a byte stream file. 14.4.2.1 The syscall Layer The syscall layer offers a simple, direct system interface with a minimum of system and library overhead. If requests are larger than approximately 64 K, this method can be appropriate. 14.4.2.2 The bufa and cachea Layers The bufa and cachea layers permit efficient file processing. Both layers provide asynchronous buffering managed by the library, and the cachea layer allows recently accessed parts of a file to be cached in memory. The number of buffers and the size of each buffer are tunable. In the bufa:bs:nbufs or cachea:bs:nbufs FFIO specifications, the bs argument specifies the size in 4096-byte blocks of each buffer. The default depends on the st_blksize field returned from a stat system call of the file; if this return value is 0, the default is 8 for all files. The nbufs argument specifies the number of buffers to use. bufa defaults to 2 buffers, while cachea defaults to 512 buffers. 14.4.2.3 The mr Layer The mr layer lets you use main memory as an I/O device for many files. When used in combination with the other layers, this permits cos blocked files, text files, and direct-access files to reside in memory without recoding. This can result in improved performance for the file, or part of a file, that resides in memory. The mr layer features both scr and save mode and directs overflow to the next lower layer automatically. S–3901–80 273Cray Fortran Reference Manual The assign -F command specifies the entire set of processing steps that are performed when I/O is requested. If a file is blocked, you must specify the appropriate layer for the handling of block and record control words as in the following examples: assign -F f77,mr u:1 assign -F cos,mr fort.1 Sample Programs on page 275 contains several mr program examples. 14.4.2.4 The global Layer Deferred Implementation: Support for the global layer has been deferred until a later release. The global layer is a caching layer that distributes data across all multiple SHMEM or MPI processes. Open and close operations require participation by all processes that access the file; all other operations are performed independently by one or more processes. File positions can be private to a process or global to all processes. You can specify both the cache size and the number of cache pages to use. Since this layer is used by parallel processes, the actual number of cache pages used is the number specified times the number of processes. 14.4.2.5 The cache Layer The cache layer permits efficient file processing for repeated access to one or more regions of a file. It is a library-managed buffer cache that contains a tunable number of pages of tunable size. To specify the cache layer, use the following option: assign -F cache[:[bs][:[nbufs]]] The bs argument specifies the size in 4096-byte blocks of each cache page; the default is 16. The nbufs argument specifies the number of cache pages to use; the default is 4. You can achieve improved I/O performance by using one or more of the following strategies: • Use a cache page size that is a multiple of the user's record size. This ensures that no user record straddles two cache pages. If this is not possible or desirable, it is best to allocate a few additional cache pages (nbufs). • Use a number of cache pages that is greater than or equal to the number of file regions the code accesses at one time. 274 S–3901–80Using Flexible File I/O (FFIO) [14] If the number of regions accessed within a file is known, the number of cache pages can be chosen first. To determine the cache page size, divide the amount of memory to be used by the number of cache pages. For example, suppose a program uses direct access to read 10 vectors from a file and then writes the sum to a different file: integer VECTSIZE, NUMCHUNKS, CHUNKSIZE parameter(VECTSIZE=1000*512) parameter(NUMCHUNKS=100) parameter(CHUNKSIZE=VECTSIZE/NUMCHUNKS) read a(CHUNKSIZE), sum(CHUNKSIZE) open(11,access='direct',recl=CHUNKSIZE*8) call asnunit (2,'-s unblocked',ier) open (2,form='unformatted') do i = 1,NUMCHUNKS sum = 0.0 do j = 1,10 read(11,rec=(j-1)*NUMCHUNKS+i)a sum=sum+a enddo write(2) sum enddo end If 4 MB of memory are allocated for buffers for unit 11, 10 cache pages should be used, each of the following size: 4MB/10 = 400000 bytes = 97 blocks Make the buffer size an even multiple of the record length of 409600 bytes by rounding it up to 100 blocks (= 409600 bytes), then use the following assign command: assign -F cache:100:10 u:11 14.5 Sample Programs The following examples illustrate the use of the mr layers. S–3901–80 275Cray Fortran Reference Manual Example 6. Unformatted direct mr with unblocked file In the following example, batch job ex8 contains a program that uses unformatted direct-access I/O with an mr layer: #QSUB -r ex8 -lT 10 -lQ 500000 #QSUB -eo -o ex8.out date set -x cd $TMPDIR cat > ex8.f <= num1 + 8 vb 9 32,760 32,760 Must be >= num1 + 8 vbs 9 32,760 32,760 292 S–3901–80FFIO Layer Reference [15] Table 43. Data Manipulation: ibm Layer Granularity Data Model Truncate on Write Implementation Strategy 8 bits Record No for f and fb records. Yes for v, vb, and vbs records. f records for f and fb. v records for u, v, vb, and vbs. Table 44. Supported Operations: ibm Layer Supported Operations Required of next lower level? Operation Supported Comments Used Comments ffopen Yes Yes ffread Yes Yes ffreadc Yes No ffwrite Yes Yes ffwritec Yes No ffclose Yes Yes ffflush Yes No ffweof Passed through Yes ffweod Yes Yes ffseek Yes seek(fd,0,0) only (equals rewind) Yes seek(fd,0,0) only ffbksp No No 15.11 The mr Layer The memory-resident (mr) layer lets users declare that all or part of a file will reside in memory. This can improve performance for relatively small files that are heavily accessed or for larger files where the first part of the file is heavily accessed (for example, a file which contains a frequently updated directory at the beginning.) The mr layer tries to allocate a buffer large enough to hold the entire file. Note: It is generally more advantageous to configure the layer preceding the mr layer to make the file buffer-resident, assuming that layer can support buffers of sufficient size. The options are as follows: mr[.type[.subtype]]:num1:num2:num3 The keyword syntax is as follows: mr[.type[.subtype]][.start_size=num1][.max_size=num2][.inc_size=num3] S–3901–80 293Cray Fortran Reference Manual The type field specifies whether the file in memory is intended to be saved or is considered a scratch file. This argument accepts the following values: Value Definition save Default. The file is loaded into memory when opened and written back to the next lower layer when closed. The save option also modifies the behavior of overflow processing. scr Scratch file. The file is not read into memory when opened and not written when closed. The subtype field specifies the action to take when the data can no longer fit in the allowable memory space. It accepts the following values: Value Definition ovfl Default. Data which does not fit (overflows) the maximum specified memory allocation is written to the next lower layer, which is typically a disk file. An informative message is written to stderr on the first overflow. ovflnomsg Identical to ovfl, except that no message is issued when the data overflows the memory-resident buffer. novfl If data does not fit in memory, subsequent write(1) operations fail. The num1, num2, and num3 fields are nonnegative integer values that state the number of 4096-byte blocks to use in the following circumstances: Field Definition num1 The initial size of the memory allocation, specified in 4,096-byte blocks. The default is 0. num2 The maximum size of the memory allocation, specified in 4,096-byte blocks. The default is either num1 or 256 blocks (1 MB), whichever is larger. num3 Increment the size of the memory allocation, in 4,096-byte blocks. This value is used when allocating additional memory space. The default is 256 blocks (1 MB) or (num2-num1), whichever is smaller. The num1 and num3 fields represent best-effort values. They are intended for tuning purposes only and usually do not cause failure if not satisfied precisely as specified. For example, if the available memory space is 100 blocks and the specified num3 value is 200 blocks, growth is allowed up to the 100 available blocks rather than failing to grow. 294 S–3901–80FFIO Layer Reference [15] ! Caution: When using the mr layer, you must ensure that the size of the memory-resident portions of the files are limited to reasonable values. Unrestrained and unmanaged growth of such file portions can cause heap fragmentation, exhaustion of all available memory, and program abort. If this growth has consumed all available memory, the program may not abort gracefully, making such a condition difficult to diagnose. Large memory-resident files may reduce I/O performance for sites that provide memory scheduling that favors small processes over large processes. Check with your system administrator if I/O performance is diminished. Increment sizes which are too small can also contribute to heap fragmentation. Memory allocation is done by using the malloc and realloc library routines. The file space in memory is always allocated contiguously. When allocating new chunks of memory space, the num3 argument is used in conjunction with realloc as a minimum first try for reallocation. Table 45. Data Manipulation: mr Layer Primary Function Granularity Data Model Truncate on Write Keep the file resident in memory and avoid I/O if possible. 8 bit Stream No Table 46. Supported Operations: mr Layer Supported Operations Required of next lower level? Operation Supported Comments Used Comments ffopen Yes Yes Sometimes delayed until overflow ffread Yes Yes Only on open ffreadc Yes No ffwrite Yes Yes Only on close, overflow ffwritec Yes No ffclose Yes Yes ffflush Yes No-op No ffweof No No representation No No representation S–3901–80 295Cray Fortran Reference Manual Supported Operations Required of next lower level? Operation Supported Comments Used Comments ffweod Yes Yes ffseek Yes Full support (absolute, relative, and from end) Yes Used in open and close processing ffbksp No No records No 15.12 The null Layer The null layer is a syntactic convenience for users; it has no effect. This layer is commonly used to simplify the writing of a shell script when a shell variable is used to specify an FFIO layer specification. For example, the following line is from a shell script with a file using the assign command and with overlying blocking expected (as specified by BLKTYP): assign -F $BLKTYP,cos fort.1 If BLKTYP is undefined, the illegal specification list ,cos results. The existence of the null layer lets the programmer set BLKTYP to null as a default, and simplify the script, as in: assign -F null,cos fort.1 This is identical to the following command: assign -F cos fort.1 When used as the last layer above the system or syscall layer, the null layer supports the assign -B option to enable or disable direct I/O. 15.13 The syscall Layer The syscall layer directly maps each request to an appropriate system call. The layer does not accept any options. Table 47. Data Manipulation: syscall Layer Granularity Data Model Truncate on Write 8 bits (1 byte) Stream No 296 S–3901–80FFIO Layer Reference [15] Table 48. Supported Operations: syscall Layer Operation Supported Comments ffopen Yes open ffread Yes read ffreadc Yes read plus code ffwrite Yes write ffwritec Yes write plus code ffclose Yes close ffflush Yes None ffweof No None ffweod Yes trunc(2) ffseek Yes lseek(2) ffbksp No Lower-level layers are not allowed. 15.14 The system Layer The system layer is implicitly appended to all specification lists, if not explicitly added by the user (unless the syscall or fd layer is specified). It maps requests to appropriate system calls. For a description of options, see the syscall layer. Lower-level layers are not allowed. 15.15 The text Layer The text layer performs text blocking by terminating each record with a newline character. It can also recognize and represent the EOF mark. The text layer is used with character files and does not work with binary data. The general specification follows: text[.type]:[num1]:[num2] The keyword syntax is as follows: text[.type][.newline=num1][.bufsize=num2] S–3901–80 297Cray Fortran Reference Manual The type field can have either of the following values: Value Definition nl Newline-separated records. eof Newline-separated records with a special string such as ~e. More than one EOF in a file is allowed. The num1 field is the decimal value of a single character that represents the newline character. The default value is 10 (octal 012, ASCII line feed). The num2 field specifies the working buffer size (in decimal bytes). If any lower-level layers are record oriented, the num2 value also specifies the block size. Table 49. Data Manipulation: text Layer Granularity Data Model Truncate on Write 8 bits Record No Table 50. Supported Operations: text Layer Supported Operations Required of next lower level? Operation Supported Comments Used Comments ffopen Yes Yes ffread Yes Yes ffreadc Yes No ffwrite Yes Yes ffwritec Yes No ffclose Yes Yes ffflush Yes No ffweof Passed through Yes Only if explicitly requested ffweod Yes Yes ffseek Yes Yes ffbksp No No 15.16 The user and site Layers The user and site layers let users and site administrators build user-defined or site-specific layers to meet special needs. The syntax follows: user[num1]:[num2] site:[num1]:[num2] 298 S–3901–80FFIO Layer Reference [15] The open processing passes the num1 and num2 arguments to the layer and these arguments are interpreted by the layers. See Chapter 16, Creating a user Layer on page 303 for an example of how to create a user FFIO layer. 15.17 The vms Layer The vms layer handles record blocking for three common record types on VAX/VMS operating systems. The general format of the specification follows: vms.[type.subtype]:[num1]:[num2] The following is the alternate keyword syntax for this layer: vms.[type.subtype][.recsize=num1][.mbs=num2] The following type values are supported: Value Definition f VAX/VMS fixed-length records v VAX/VMS variable-length records s VAX/VMS variable-length segmented records In addition to the record type, you must specify a record subtype, which has one of the following values: Value Definition bb Format used for binary blocked transfers disk Same as binary blocked tr Transparent format, for files transferred as a bit stream to and from the VAX/VMS system tape VAX/VMS labeled tape The num1 field is the maximum record size that may be read or written. It is ignored by the s record type. Table 51. Values for Record Size: vms Layer Field Minimum Maximum Default Comments v.bb 1 32,767 32,767 v.tape 1 9995 2043 v.tr 1 32,767 2044 S–3901–80 299Cray Fortran Reference Manual Field Minimum Maximum Default Comments s.bb 1 None None No maximum record size s.tape 1 None None No maximum record size s.tr 1 None None No maximum record size The num2 field is the maximum segment or block size that is allowed on input and is produced on output. For vms.f.tr and vms.f.bb, num2 should be equal to the record size (num1). Because vms.f.tape places one or more records in each block, vms.f.tape num2 must be greater than or equal to num1. Table 52. Values for Maximum Block Size: vms Layer Field Minimum Maximum Default Comments v.bb 1 32,767 32,767 v.tape 6 32,767 2,048 v.tr 3 32,767 32,767 N/A s.bb 5 32,767 2,046 s.tape 7 32,767 2,048 s.tr 5 32,767 2,046 N/A For vms.v.bb and vms.v.disk records, num2 is a limit on the maximum record size. For vms.v.tape records, it is the maximum size of a block on tape; more specifically, it is the maximum size of a record that will be written to the next lower layer. If that layer is tape, num2 is the tape block size. If it is cos, it will be a COS record that represents a tape block. One or more records are placed in each block. For segmented records, num2 is a limit on the block size that will be produced. No limit on record size exists. For vms.s.tr and vms.s.bb, the block size is an upper limit on the size of a segment. For vms.s.tape, one or more segments are placed in a tape block. It functions as an upper limit on the size of a segment and as a preferred tape block size. Table 53. Data Manipulation: vms Layer Granularity Data Model Truncate on Write Implementation Strategy 8 bits Record No for f records. Yes for v and s records. f records for f formats. v records for v formats. 300 S–3901–80FFIO Layer Reference [15] Table 54. Supported Operations: vms Layer Supported Operations Required of next lower level? Operation Supported Comments Used Comments ffopen Yes Yes ffread Yes Yes ffreadc Yes No ffwrite Yes Yes ffwritec Yes No ffclose Yes Yes ffflush Yes No ffweof Yes and passed through Yes for s records; passed through for others Yes Only if explicitly requested ffweod Yes Yes ffseek Yes seek(fd,0,0) only (equals rewind) Yes seek(fd,0,0) only ffbksp No No S–3901–80 301Cray Fortran Reference Manual 302 S–3901–80Creating a user Layer [16] This chapter explains some of the internals of the FFIO system and explains the ways in which you can put together a user or site layer. 16.1 Internal Functions The FFIO system has an internal model of data that maps to any given actual logical file type based on the following concepts: • Data is a stream of bits. Layers must declare their granularity by using the fffcntl call. • Record marks are boundaries between logical records. • End-of-file (EOF) marks are a special type of record that exists in some file structures. • End-of-data (EOD) is a point immediately beyond the last data bit, EOR, or EOF in the file. You cannot read past or write after an EOD. In a case when a file is positioned after an EOD, a write operation (if valid) immediately moves the EOD to a point after the last data bit, end-of-record (EOR), or EOF produced by the write. All files are streams that contain zero or more data bits that may contain record or file marks. No inherent hierarchy or ordering is imposed on the file structures. Any number of data bits or EOR and EOF marks may appear in any order. The EOD, if present, is by definition last. Given the EOR, EOF, and EOD return statuses from read operations, only EOR may be returned along with data. When data bits are immediately followed by EOF, the record is terminated implicitly. Individual layers can impose restrictions for specific file structures that are more restrictive than the preceding rules. For instance, in COS blocked files, an EOR always immediately precedes an EOF. Successful mappings were used for all logical file types supported, except formats that have more than one type of partitioning for files (such as end-of-group or more than one level of EOF). For example, some file formats have level numbers in the partitions. FFIO maps level 017 to an EOF. No other handling is provided for these level numbers. S–3901–80 303Cray Fortran Reference Manual Internally, there are two main protocol components: the operations and the stat structure. 16.1.1 The Operations Structure Many of the operations try to mimic the CLE system calls. As described in the ffread(3c), ffwrite(3c), and other man pages, the calls can be made without the optional parameters and appear like the system calls. Internally, all parameters are required. Table 55 provides a brief synopsis of the interface routines that are supported at the user level. Each of these ff entry points checks the parameters and issues the corresponding internal call. Each interface routine provides defaults and dummy arguments for those optional arguments the user does not provide. Each layer must have an internal entry point for all of these operations, although in some cases the entry point may simply issue an error or do nothing. For example, the syscall layer uses _ff_noop for the ffflush entry point because it has no buffer to flush, and it uses _ff_err2 for the ffweof entry point because it has no representation for EOF. No optional parameters for calls to the internal entry points exist. All arguments are required. Table 55. C Program Entry Points Variable Definition fd The FFIO pointer (struct fdinfo *)fd. file A char* file. flags File status flag for open, such as O_RDONLY. buf Bit pointer to the user data. nb Number of bytes. ret The status returned; >=0 is valid, <0 is error. stat A pointer to the status structure. fulp The value FULL or PARTIAL defined in ffio.h for full or partial-record mode. &ubc A pointer to the unused bit count; this ranges from 0 to 7 and represents the bits not used in the last byte of the operation. It is used for both input and output. pos A byte position in the file. opos The old position of the file, just like the system call. whence The same as the system. cmd The command request to the fffcntl call. 304 S–3901–80Creating a user Layer [16] Variable Definition arg A generic pointer to the fffcntl argument. mode Bit pattern denoting file's access permissions. argp A pointer to the input or output data. len The length of the space available at argp. It is used primarily on output to avoid overwriting the available memory. 16.1.2 FFIO and the stat Structure The stat structure contains four fields in the current implementation. They mimic the iows structure of the CLE ASYNC syscalls to the extent possible. All operations are expected to update the stat structure on each call. The SETSTAT and ERETURN macros are provided in the ffio.h file for this purpose. The fields in the stat structure are as follows: Status field Description stat.sw_flag 0 indicates outstanding; 1 indicates I/O complete. stat.sw_error 0 indicates no error; otherwise, the error number. stat.sw_count Number of bytes transferred in this request. This number is rounded up to the next integral value if a partial byte is transferred. stat.sw_stat This indicates the status of the I/O operation. The FFSTAT(stat) macro accesses this field. The following values are valid: FFBOD: At beginning-of-data (BOD). FFCNT: Request terminated by count (either the count of bytes before EOF or EOD in the file or the count of the request). FFEOR: Request termination by EOR, or a full record mode read was processed. FFEOF: EOF encountered. FFEOD: EOD encountered. FFERR: Error encountered. S–3901–80 305Cray Fortran Reference Manual If count is satisfied simultaneously with EOR, the FFEOR is returned. The EOF and EOD status values must never be returned with data. This means that if a byte-stream file is being traversed and the file contains 100 bytes followed by an EOD, a read of 500 bytes returns a stat value of FFCNT and a return byte count of 100. The next read operation returns FFEOD and a count of 0. A FFEOF or FFEOD status is always returned with a 0-byte transfer count. 16.2 user Layer Example This section gives a complete and working user layer. It traces I/O at a given level. All operations are passed through to the next lower-level layer, and a trace record is sent to the trace file. The first step in generating a user layer is to create a table that contains the addresses for the routines that will fulfill the required functions described in The Operations Structure on page 304 and FFIO and the stat Structure on page 305. The format of the table can be found in struct xtr_s, which is found in the ffio.h file. No restriction is placed on the names of the routines, but the table must be called _usr_ffvect for it to be recognized as a user layer. In the example, the declaration of the table can be found with the code in the _usr_open routine. To use this layer, you must take advantage of the weak external files in the library. The following script fragment is suggested for CLE systems: # -D_LIB_INTERNAL is required to obtain the # declaration of struct fdinfo in # cc -c -D_LIB_INTERNAL -hcalchars usr*.c cat usr*.o > user.o # # Note that the -F option is selected that loads # and links the entries despite not having any # hard references. cc -o user.o myprog.o assign -F user,others... fort.1 ./abs static char USMID[] = "@(#)code/usrbksp.c 1.0 "; /* COPYRIGHT CRAY INC. * UNPUBLISHED -- ALL RIGHTS RESERVED UNDER * THE COPYRIGHT LAWS OF THE UNITED STATES. */ #include #include "usrio.h" /* * trace backspace requests */ int _usr_bksp(struct fdinfo *fio, struct ffsw *stat) { struct fdinfo *llfio; 306 S–3901–80Creating a user Layer [16] int ret; llfio = fio->fioptr; _usr_enter(fio, TRC_BKSP); _usr_pr_2p(fio, stat); ret = XRCALL(llfio, backrtn) llfio, stat); _usr_exit(fio, ret, stat); return(0); } static char USMID[] = "@(#)code.usrclose.c 1.0 "; /* COPYRIGHT CRAY INC. * UNPUBLISHED -- ALL RIGHTS RESERVED UNDER * THE COPYRIGHT LAWS OF THE UNITED STATES. */ #include #include #include #include "usrio.h" /* * trace close requests */ int _usr_close(struct fdinfo *fio, struct ffsw *stat) { struct fdinfo *llfio; struct trace_f *pinfo; int ret; llfio = fio->fioptr; /* * lyr_info is a place in the fdinfo block that holds * a pointer to the layer's private information. */ pinfo = (struct trace_f *)fio->lyr_info; _usr_enter(fio, TRC_CLOSE); _usr_pr_2p(fio, stat); /* * close file */ ret = XRCALL(llfio, closertn) llfio, stat); /* * It is the layer's responsibility to clean up its mess. */ free(pinfo->name); pinfo->name = NULL; free(pinfo); _usr_exit(fio, ret, stat); (void) close(pinfo->usrfd); return(0); } static char USMID[] = "@(#)code/usrfcntl.c 1.0 "; /* COPYRIGHT CRAY INC. * UNPUBLISHED -- ALL RIGHTS RESERVED UNDER * THE COPYRIGHT LAWS OF THE UNITED STATES. */ #include #include "usrio.h" S–3901–80 307Cray Fortran Reference Manual /* * trace fcntl requests * * Parameters: * fd - fdinfo pointer * cmd - command code * arg - command specific parameter * stat - pointer to status return word * * This fcntl routine passes the request down to the next lower * layer, so it provides nothing of its own. * * When writing a user layer, the fcntl routine must be provided, * and must provide correct responses to one essential function and * two desirable functions. * * FC_GETINFO: (essential) * If the 'cmd' argument is FC_GETINFO, the fields of the 'arg' is * considered a pointer to an ffc_info_s structure, and the fields * must be filled. The most important of these is the ffc_flags * field, whose bits are defined in .(Look for FFC_STRM * through FFC_NOTRN) * FC_STAT: (desirable) * FC_RECALL: (desirable) */ int _usr_fcntl(struct fdinfo *fio, int cmd, void *arg, struct ffsw *stat) { struct fdinfo *llfio; struct trace_f *pinfo; int ret; llfio = fio->fioptr; pinfo = (struct trace_f *)fio->lyr_info; _usr_enter(fio, TRC_FCNTL); _usr_info(fio, "cmd=%d ", cmd); ret = XRCALL(llfio, fcntlrtn) llfio, cmd, arg, stat); _usr_exit(fio, ret, stat); return(ret); } static char USMID[] = "@(#)code/usropen.c 1.0 "; /* COPYRIGHT CRAY INC. * UNPUBLISHED -- ALL RIGHTS RESERVED UNDER * THE COPYRIGHT LAWS OF THE UNITED STATES. */ #include #include #include #include #include "usrio.h" #define SUFFIX ".trc" /* * trace open requests; * The following routines compose the user layer. They are declared * in "usrio.h" 308 S–3901–80Creating a user Layer [16] */ /* * Create the _usr_ffvect structure. Note the _ff_err inclusion to * account for the listiortn, which is not supported by this user * layer */ struct xtr_s _usr_ffvect = { _usr_open, _usr_read, _usr_reada, _usr_readc, _usr_write, _usr_writea, _usr_writec, _usr_close, _usr_flush, _usr_weof, _usr_weod, _usr_seek, _usr_bksp, _usr_pos, _usr_err, _usr_fcntl }; _ffopen_t _usr_open( const char *name, int flags, mode_t mode, struct fdinfo * fio, union spec_u *spec, struct ffsw *stat, long cbits, int cblks, struct gl_o_inf *oinf) { union spec_u *nspec; struct fdinfo *llfio; struct trace_f *pinfo; char *ptr = NULL; int namlen, usrfd; _ffopen_t nextfio; char buf[256]; namlen = strlen(name); ptr = malloc(namlen + strlen(SUFFIX) + 1); if (ptr == NULL) goto badopen; pinfo = (struct trace_f *)malloc(sizeof(struct trace_f)); if (pinfo == NULL) goto badopen; fio->lyr_info = (char *)pinfo; /* * Now, build the name of the trace info file, and open it. */ strcpy(ptr, name); strcat(ptr, SUFFIX); usrfd = open(ptr, O_WRONLY | O_APPEND | O_CREAT, 0666); /* * Put the file info into the private data area. */ pinfo->name = ptr; pinfo->usrfd = usrfd; ptr[namlen] = '\0'; /* * Log the open call */ _usr_enter(fio, TRC_OPEN); S–3901–80 309Cray Fortran Reference Manual sprintf(buf,"(\"%s\", %o, %o...);\n", name, flags, mode); _usr_info(fio, buf, 0); /* * Now, open the lower layers */ nspec = spec; NEXT_SPEC(nspec); nextfio = _ffopen(name, flags, mode, nspec, stat, cbits, cblks, NULL, oinf); _usr_exit_ff(fio, nextfio, stat); if (nextfio != _FFOPEN_ERR) { DUMP_IOB(fio); /* debugging only */ return(nextfio); } /* * End up here only on an error * */ badopen: if(ptr != NULL) free(ptr); if (fio->lyr_info != NULL) free(fio->lyr_info); _SETERROR(stat, FDC_ERR_NOMEM, 0); return(_FFOPEN_ERR); } _usr_err(struct fdinfo *fio) { _usr_info(fio,"ERROR: not expecting this routine\n",0); return(0); } static char USMID[] = "@(#)code/usrpos.c 1.1 "; /* COPYRIGHT CRAY INC. * UNPUBLISHED -- ALL RIGHTS RESERVED UNDER * THE COPYRIGHT LAWS OF THE UNITED STATES. */ #include #include "usrio.h" /* * trace positioning requests */ _ffseek_t _usr_pos(struct fdinfo *fio, int cmd, void *arg, int len, struct ffsw *stat) { struct fdinfo *llfio; struct trace_f *usr_info; _ffseek_t ret; llfio = fio->fioptr; usr_info = (struct trace_f *)fio->lyr_info; _usr_enter(fio,TRC_POS); _usr_info(fio, " ", 0); 310 S–3901–80Creating a user Layer [16] ret = XRCALL(llfio, posrtn) llfio, cmd, arg, len, stat); _usr_exit_sk(fio, ret, stat); return(ret); } static char USMID[] = "@(#)code/usrprint.c 1.1 "; /* COPYRIGHT CRAY INC. * UNPUBLISHED -- ALL RIGHTS RESERVED UNDER * THE COPYRIGHT LAWS OF THE UNITED STATES. */ #include #include #include "usrio.h" static char *name_tab[] = { "???", "ffopen", "ffread", "ffreadc", "ffwrite", "ffwritec", "ffclose", "ffflush", "ffweof", "ffweod", "ffseek", "ffbksp", "fflistio", "fffcntl", }; /* * trace printing stuff */ int _usr_enter(struct fdinfo *fio, int opcd) { char buf[256], *op; struct trace_f *usr_info; op = name_tab[opcd]; usr_info = (struct trace_f *)fio->lyr_info; sprintf(buf, "TRCE: %s ",op); write(usr_info->usrfd, buf, strlen(buf)); return(0); } void _usr_info(struct fdinfo *fio, char *str, int arg1) { char buf[256]; struct trace_f *usr_info; usr_info = (struct trace_f *)fio->lyr_info; sprintf(buf, str, arg1); write(usr_info->usrfd, buf, strlen(buf)); S–3901–80 311Cray Fortran Reference Manual } void _usr_exit(struct fdinfo *fio, int ret, struct ffsw *stat) { char buf[256]; struct trace_f *usr_info; usr_info = (struct trace_f *)fio->lyr_info; fio->ateof = fio->fioptr->ateof; fio->ateod = fio->fioptr->ateod; sprintf(buf, "TRCX: ret=%d, stat=%d, err=%d\n", ret, stat->sw_stat, stat->sw_error); write(usr_info->usrfd, buf, strlen(buf)); } void _usr_exit_ss(struct fdinfo *fio, ssize_t ret, struct ffsw *stat) { char buf[256]; struct trace_f *usr_info; usr_info = (struct trace_f *)fio->lyr_info; fio->ateof = fio->fioptr->ateof; fio->ateod = fio->fioptr->ateod; sprintf(buf, "TRCX: ret=%ld, stat=%d, err=%d\n", ret, stat->sw_stat, stat->sw_error); write(usr_info->usrfd, buf, strlen(buf)); } void _usr_exit_ff(struct fdinfo *fio, _ffopen_t ret, struct ffsw *stat) { char buf[256]; struct trace_f *usr_info; usr_info = (struct trace_f *)fio->lyr_info; sprintf(buf, "TRCX: ret=%d, stat=%d, err=%d\n", ret, stat->sw_stat, stat->sw_error); write(usr_info->usrfd, buf, strlen(buf)); } void _usr_exit_sk(struct fdinfo *fio, _ffseek_t ret, struct ffsw *stat) { char buf[256]; struct trace_f *usr_info; usr_info = (struct trace_f *)fio->lyr_info; fio->ateof = fio->fioptr->ateof; fio->ateod = fio->fioptr->ateod; sprintf(buf, "TRCX: ret=%ld, stat=%d, err=%d\n", ret, stat->sw_stat, stat->sw_error); #endif write(usr_info->usrfd, buf, strlen(buf)); } void _usr_pr_rwc( struct fdinfo *fio, bitptr bufptr, 312 S–3901–80Creating a user Layer [16] size_t nbytes, struct ffsw *stat, int fulp) { char buf[256]; struct trace_f *usr_info; usr_info = (struct trace_f *)fio->lyr_info; sprintf(buf,"(fd / %lx */, &memc[%lx], %ld, &statw[%lx], ", fio, BPTR2CP(bufptr), nbytes, stat); write(usr_info->usrfd, buf, strlen(buf)); if (fulp == FULL) sprintf(buf,"FULL"); else sprintf(buf,"PARTIAL"); write(usr_info->usrfd, buf, strlen(buf)); } void _usr_pr_rww( struct fdinfo *fio, bitptr bufptr, size_t nbytes, struct ffsw *stat, int fulp, int *ubc) { char buf[256]; struct trace_f *usr_info; usr_info = (struct trace_f *)fio->lyr_info; sprintf(buf,"(fd / %lx */, &memc[%lx], %ld, &statw[%lx], ", fio, BPTR2CP(bufptr), nbytes, stat); write(usr_info->usrfd, buf, strlen(buf)); if (fulp == FULL) sprintf(buf,"FULL"); else sprintf(buf,"PARTIAL"); write(usr_info->usrfd, buf, strlen(buf)); sprintf(buf,", &conubc[%d]; ", *ubc); write(usr_info->usrfd, buf, strlen(buf)); } void _usr_pr_2p(struct fdinfo *fio, struct ffsw *stat) { char buf[256]; struct trace_f *usr_info; usr_info = (struct trace_f *)fio->lyr_info; sprintf(buf,"(fd / %lx */, &statw[%lx], ", fio, stat); write(usr_info->usrfd, buf, strlen(buf)); } static char USMID[] = "@(#)code/usrread.c 1.0 "; /* COPYRIGHT CRAY INC. * UNPUBLISHED -- ALL RIGHTS RESERVED UNDER * THE COPYRIGHT LAWS OF THE UNITED STATES. */ S–3901–80 313Cray Fortran Reference Manual #include #include "usrio.h" /* * trace read requests * * Parameters: * fio - Pointer to fdinfo block * bufptr - bit pointer to where data is to go. * nbytes - Number of bytes to be read * stat - pointer to status return word * fulp - full or partial read mode flag * ubc - pointer to unused bit count */ ssize_t _usr_read( struct fdinfo *fio, bitptr bufptr, size_t nbytes, struct ffsw *stat, int fulp, int *ubc) { struct fdinfo *llfio; char *str; ssize_t ret; llfio = fio->fioptr; _usr_enter(fio, TRC_READ); _usr_pr_rww(fio, bufptr, nbytes, stat, fulp, ubc); ret = XRCALL(llfio, readrtn) llfio, bufptr, nbytes, stat, fulp, ubc); _usr_exit_ss(fio, ret, stat); return(ret); } /* * trace reada (asynchronous read) requests * * Parameters: * fio - Pointer to fdinfo block * bufptr - bit pointer to where data is to go. * nbytes - Number of bytes to be read * stat - pointer to status return word * fulp - full or partial read mode flag * ubc - pointer to unused bit count */ ssize_t _usr_reada( struct fdinfo *fio, bitptr bufptr, size_t nbytes, struct ffsw *stat, int fulp, int *ubc) { struct fdinfo *llfio; char *str; 314 S–3901–80Creating a user Layer [16] ssize_t ret; llfio = fio->fioptr; _usr_enter(fio, TRC_READA); _usr_pr_rww(fio, bufptr, nbytes, stat, fulp, ubc); ret = XRCALL(llfio,readartn)llfio,bufptr,nbytes,stat,fulp,ubc); _usr_exit_ss(fio, ret, stat); return(ret); } /* * trace readc requests * * Parameters: * fio - Pointer to fdinfo block * bufptr - bit pointer to where data is to go. * nbytes - Number of bytes to be read * stat - pointer to status return word * fulp - full or partial read mode flag */ ssize_t _usr_readc( struct fdinfo *fio, bitptr bufptr, size_t nbytes, struct ffsw *stat, int fulp) { struct fdinfo *llfio; char *str; ssize_t ret; llfio = fio->fioptr; _usr_enter(fio, TRC_READC); _usr_pr_rwc(fio, bufptr, nbytes, stat, fulp); ret = XRCALL(llfio, readcrtn)llfio, bufptr, nbytes, stat, fulp); _usr_exit_ss(fio, ret, stat); return(ret); } /* * _usr_seek() * * The user seek call should mimic the lseek system call as * much as possible. */ _ffseek_t _usr_seek( struct fdinfo *fio, off_t pos, int whence, struct ffsw *stat) { struct fdinfo *llfio; _ffseek_t ret; char buf[256]; llfio = fio->fioptr; _usr_enter(fio, TRC_SEEK); S–3901–80 315Cray Fortran Reference Manual sprintf(buf,"pos %ld, whence %d\n", pos, whence); _usr_info(fio, buf, 0); ret = XRCALL(llfio, seekrtn) llfio, pos, whence, stat); _usr_exit_sk(fio, ret, stat); return(ret); } static char USMID[] = "@(#)code/usrwrite.c 1.0 "; /* COPYRIGHT CRAY INC. * UNPUBLISHED -- ALL RIGHTS RESERVED UNDER * THE COPYRIGHT LAWS OF THE UNITED STATES. */ #include #include "usrio.h" /* * trace write requests * * Parameters: * fio - Pointer to fdinfo block * bufptr - bit pointer to where data is to go. * nbytes - Number of bytes to be written * stat - pointer to status return word * fulp - full or partial write mode flag * ubc - pointer to unused bit count (not used for IBM) */ ssize_t _usr_write( struct fdinfo *fio, bitptr bufptr, size_t nbytes, struct ffsw *stat, int fulp, int *ubc) { struct fdinfo *llfio; ssize_t ret; llfio = fio->fioptr; _usr_enter(fio, TRC_WRITE); _usr_pr_rww(fio, bufptr, nbytes, stat, fulp, ubc); ret = XRCALL(llfio, writertn) llfio, bufptr, nbytes, stat, fulp,ubc); _usr_exit_ss(fio, ret, stat); return(ret); } /* * trace writea requests * * Parameters: * fio - Pointer to fdinfo block * bufptr - bit pointer to where data is to go. * nbytes - Number of bytes to be written * stat - pointer to status return word * fulp - full or partial write mode flag * ubc - pointer to unused bit count (not used for IBM) 316 S–3901–80Creating a user Layer [16] */ ssize_t _usr_writea( struct fdinfo *fio, bitptr bufptr, size_t nbytes, struct ffsw *stat, int fulp, int *ubc) { struct fdinfo *llfio; ssize_t ret; llfio = fio->fioptr; _usr_enter(fio, TRC_WRITEA); _usr_pr_rww(fio, bufptr, nbytes, stat, fulp, ubc); ret = XRCALL(llfio, writeartn) llfio, bufptr, nbytes, stat, fulp,ubc); _usr_exit_ss(fio, ret, stat); return(ret); } /* * trace writec requests * * Parameters: * fio - Pointer to fdinfo block * bufptr - bit pointer to where data is to go. * nbytes - Number of bytes to be written * stat - pointer to status return word * fulp - full or partial write mode flag */ ssize_t _usr_writec( struct fdinfo *fio, bitptr bufptr, size_t nbytes, struct ffsw *stat, int fulp) { struct fdinfo *llfio; ssize_t ret; llfio = fio->fioptr; _usr_enter(fio, TRC_WRITEC); _usr_pr_rwc(fio, bufptr, nbytes, stat, fulp); ret = XRCALL(llfio, writecrtn)llfio,bufptr, nbytes, stat, fulp); _usr_exit_ss(fio, ret, stat); return(ret); } /* * Flush the buffer and clean up * This routine should return 0, or -1 on error. */ int _usr_flush(struct fdinfo *fio, struct ffsw *stat) S–3901–80 317Cray Fortran Reference Manual { struct fdinfo *llfio; int ret; llfio = fio->fioptr; _usr_enter(fio, TRC_FLUSH); _usr_info(fio, "\n",0); ret = XRCALL(llfio, flushrtn) llfio, stat); _usr_exit(fio, ret, stat); return(ret); } /* * trace WEOF calls * * The EOF is a very specific concept. Don't confuse it with the * EOF, or the truncate(2) system call. */ int _usr_weof(struct fdinfo *fio, struct ffsw *stat) { struct fdinfo *llfio; int ret; llfio = fio->fioptr; _usr_enter(fio, TRC_WEOF); _usr_info(fio, "\n",0); ret = XRCALL(llfio, weofrtn) llfio, stat); _usr_exit(fio, ret, stat); return(ret); } /* * trace WEOD calls * * The EOD is a specific concept. Don't confuse it with the * EOF. It is usually mapped to the truncate(2) system call. */ int _usr_weod(struct fdinfo *fio, struct ffsw *stat) { struct fdinfo *llfio; int ret; llfio = fio->fioptr; _usr_enter(fio, TRC_WEOD); _usr_info(fio, "\n",0); ret = XRCALL(llfio, weodrtn) llfio, stat); _usr_exit(fio, ret, stat); return(ret); } /* USMID @(#)code/usrio.h 1.1 */ /* COPYRIGHT CRAY INC. * UNPUBLISHED -- ALL RIGHTS RESERVED UNDER * THE COPYRIGHT LAWS OF THE UNITED STATES. */ 318 S–3901–80Creating a user Layer [16] #define TRC_OPEN 1 #define TRC_READ 2 #define TRC_READA 3 #define TRC_READC 4 #define TRC_WRITE 5 #define TRC_WRITEA 6 #define TRC_WRITEC 7 #define TRC_CLOSE 8 #define TRC_FLUSH 9 #define TRC_WEOF 10 #define TRC_WEOD 11 #define TRC_SEEK 12 #define TRC_BKSP 13 #define TRC_POS 14 #define TRC_UNUSED 15 #define TRC_FCNTL 16 struct trace_f { char *name; /* name of the file */ int usrfd; /* file descriptor of trace file */ }; /* * Prototypes */ extern int _usr_bksp(struct fdinfo *fio, struct ffsw *stat); extern int _usr_close(struct fdinfo *fio, struct ffsw *stat); extern int _usr_fcntl(struct fdinfo *fio, int cmd, void *arg, struct ffsw *stat); extern _ffopen_t _usr_open(const char *name, int flags, mode_t mode, struct fdinfo * fio, union spec_u *spec, struct ffsw *stat, long cbits, int cblks, struct gl_o_inf *oinf); extern int _usr_flush(struct fdinfo *fio, struct ffsw *stat); extern _ffseek_t _usr_pos(struct fdinfo *fio, int cmd, void *arg, int len, struct ffsw *stat); extern ssize_t _usr_read(struct fdinfo *fio, bitptr bufptr, size_t nbytes, struct ffsw *stat, int fulp, int *ubc); extern ssize_t _usr_reada(struct fdinfo *fio, bitptr bufptr, size_t nbytes, struct ffsw *stat, int fulp, int *ubc); extern ssize_t _usr_readc(struct fdinfo *fio, bitptr bufptr, size_t nbytes, struct ffsw *stat, int fulp); extern _ffseek_t _usr_seek(struct fdinfo *fio, off_t pos, int whence, struct ffsw *stat); extern ssize_t _usr_write(struct fdinfo *fio, bitptr bufptr, size_t nbytes, struct ffsw *stat, int fulp, int *ubc); extern ssize_t _usr_writea(struct fdinfo *fio, bitptr bufptr, size_t nbytes, struct ffsw *stat, int fulp, int *ubc); extern ssize_t _usr_writec(struct fdinfo *fio, bitptr bufptr, size_t nbytes, struct ffsw *stat, int fulp); extern int _usr_weod(struct fdinfo *fio, struct ffsw *stat); extern int _usr_weof(struct fdinfo *fio, struct ffsw *stat); extern int _usr_err(); /* * Prototypes for routines that are used by the user layer. */ extern int _usr_enter(struct fdinfo *fio, int opcd); S–3901–80 319Cray Fortran Reference Manual extern void _usr_info(struct fdinfo *fio, char *str, int arg1); extern void _usr_exit(struct fdinfo *fio, int ret, struct ffsw *stat); extern void _usr_exit_ss(struct fdinfo *fio, ssize_t ret, struct ffsw *stat); extern void _usr_exit_ff(struct fdinfo *fio, _ffopen_t ret, struct ffsw *stat); extern void _usr_exit_sk(struct fdinfo *fio, _ffseek_t ret, struct ffsw *stat); extern void _usr_pr_rww(struct fdinfo *fio, bitptr bufptr, size_t nbytes, struct ffsw *stat, int fulp, int *ubc); extern void _usr_pr_2p(struct fdinfo *fio, struct ffsw *stat); 320 S–3901–80Named Pipe Support [17] Named pipes, or UNIX FIFO special files for I/O requests, are created with the mknod system call; these special files allow any two processes to exchange information. The system call creates an inode for the named pipe and establishes it as a named pipe that can be read to or written from. It can then be used by standard Fortran I/O or C I/O. Piped I/O is faster than normal I/O and requires less memory than memory-resident files. Fortran programs can communicate with each other using named pipes. After a named pipe is created, Fortran programs can access that pipe almost as if it were a normal file. The unique aspects of process communication using named pipes are discussed in the following list; the examples show how a Fortran program can use standard Fortran I/O on pipes: • A named pipe must be created before a Fortran program opens it. The following syntax for the command creates a named pipe called fort.13. The p argument makes it a pipe. /bin/mknod fort.13 p A named pipe can be created from within a Fortran program by using the pxfsystem function. The following example creates a named pipe: INTEGER ILEN,IERROR ILEN=0 CALL PXFSYSTEM ('/bin/mknod fort.13 p',ILEN,IERROR) • Fortran programs can use two named pipes: one to read and one to write. A Fortran program can read from or write to any named pipe, but it cannot do both at the same time. This is a Fortran restriction on pipes, not a system restriction. It occurs because Fortran does not allow read and write access at the same time. • I/O transfers through named pipes use memory for buffering. A separate buffer is created for each named pipe. The PIPE_BUF parameter defines the kernel buffer size in the /sys/param.h parameter file. The default value of PIPE_BUF is 8 blocks (8 * 512 words), but the full size may not be needed or used. I/O to named pipes does not transfer to or from a disk. However, if I/O transfers fill the buffer, the writing process waits for the receiving process to read the data before refilling the buffer. If the size of the PIPE_BUF parameter is increased, buffer contention may cause a decrease in I/O performance. If memory has already been allocated for buffers, more space will not be allocated. S–3901–80 321Cray Fortran Reference Manual • Binary data transferred between two processes through a named pipe must use the correct file structure. The sending process should specify an undefined file structure (assign -s u) for a pipe. The receiving process should specify an unblocked structure (assign -s unblocked) for a pipe. You can also select a file specification of system (assign -F system) for the sending process. The file structure of the receiving or read process can be set to either an undefined or an unblocked file structure. However, if the sending process writes a request that is larger than PIPE_BUF, it is essential for the receiving process to read the data from a pipe set to an unblocked file structure. A read of a transfer larger than PIPE_BUF on an undefined file structure yields only the amount of data specified by PIPE_BUF. The receiving process does not wait to see whether the sending process is refilling the buffer. The pipe may be less than the value of PIPE_BUF. For example, the following assign commands specify that the file structure of the named pipe (unit 13, file name pipe) for the sending process should be undefined (-s u). The named pipe (unit 15, file name pipe) is type unblocked (-s unblocked) for the read process. assign -s u -a pipe u:13 assign -s unblocked -a pipe u:15 • A read from a pipe that is closed by the sender causes an end-of-file (EOF). To detect EOF on a named pipe, the pipe must be opened as read-only by the receiving process. The remainder of this chapter presents more information about detecting EOF. 17.1 Piped I/O Example without End-of-file Detection In this example, two Fortran programs communicate without end-of-file (EOF) detection. Program writerd generates an array, which contains the elements 1 to 3, and writes the array to named pipe pipe1. Program readwt reads the three elements from named pipe pipe1, prints out the values, adds 1 to each value, and writes the new elements to named pipe pipe2. Program writerd reads the new values from named pipe pipe2 and prints them. The -a option of the assign command allows the two processes to access the same file with different assign characteristics. 322 S–3901–80Named Pipe Support [17] Example 8. No EOF Detection: program writerd program writerd parameter(n=3) dimension ia(n) do 10 i=1,n ia(i)=i 10 continue write (10) ia read (11) ia do 20 i=1,n print*,'ia(',i,') is ',ia(i),' in writerd' 20 continue end Example 9. No EOF Detection: program readwt program readwt parameter(n=3) dimension ia(n) read (15) ia do 10 i=1,n print*,'ia(',i,') is ',ia(i),' in readwt' ia(i)=ia(i)+1 10 continue write (16) ia end The following command sequence executes the programs: ftn -o readwt readwt.f ftn -o writerd writerd.f /bin/mknod pipe1 p /bin/mknod pipe2 p assign -s u -a pipe1 u:10 assign -s unblocked -a pipe2 u:11 assign -s unblocked -a pipe1 u:15 assign -s u -a pipe2 u:16 readwt & writerd The output of the two programs is: ia(1) is 1 in readwt ia(2) is 2 in readwt ia(3) is 3 in readwt ia(1) is 2 in writerd ia(2) is 3 in writerd ia(3) is 4 in writerd 17.2 Detecting End-of-file on a Named Pipe The following conditions must be met to detect end-of-file on a read from a named pipe within a Fortran program: S–3901–80 323Cray Fortran Reference Manual • The program that sends data must open the pipe in a specific way, and the program that receives the data must open the pipe as read-only. • The program that sends or writes the data must open the named pipe as read-and-write or write-only. Read-and-write is the default because the /bin/mknod command creates a named pipe with read-and-write permission. • The program that receives or reads the data must open the pipe as read-only. A read from a named pipe that is opened as read-and-write waits indefinitely for the data being sent. 17.3 Piped I/O Example with End-of-file Detection This example uses named pipes for communication between two Fortran programs with end-of-file detection. The programs in this example are similar to the programs used in the preceding section. This example shows that program readwt can detect the EOF. Program writerd generates array ia and writes the data to the named pipe pipe1. Program readwt reads the data from the named pipe pipe1, prints the values, adds one to each value, and writes the new elements to named pipe pipe2. Program writerd reads the new values from pipe2 and prints them. Finally, program writerd closes pipe1 and causes program readwt to detect the EOF. This command sequence executes these programs: ftn -o readwt readwt.f ftn -o writerd writerd.f assign -s u -a pipe1 u:10 assign -s unblocked -a pipe2 u:11 assign -s unblocked -a pipe1 u:15 assign -s u -a pipe2 u:16 /bin/mknod pipe1 p /bin/mknod pipe2 p readwt & writerd Example 10. EOF Detection: program writerd program writerd parameter(n=3) dimension ia(n) do 10 i=1,n ia(i)=i 10 continue write (10) ia read (11) ia do 20 i=1,n print*,'ia(',i,') is',ia(i),' in writerd' 20 continue close (10) end 324 S–3901–80Named Pipe Support [17] Example 11. EOF Detection: program readwt program readwt parameter(n=3) dimension ia(n) C open the pipe as read-only open(15,form='unformatted', action='read') read (15,end = 101) ia do 10 i=1,n print*,'ia(',i,') is ',ia(i),' in readwt' ia(i)=ia(i)+1 10 continue write (16) ia read (15,end = 101) ia goto 102 101 print *,'End of file detected' 102 continue end This is the output of the two programs: ia(1) is 1 in readwt ia(2) is 2 in readwt ia(3) is 3 in readwt ia(1) is 2 in writerd ia(2) is 3 in writerd ia(3) is 4 in writerd End of file detected S–3901–80 325 TM Cray C and C++ Reference Manual S–2179–80© 1996-2000, 2002-2011 Cray Inc. All Rights Reserved. This document or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Inc. U.S. GOVERNMENT RESTRICTED RIGHTS NOTICE The Computer Software is delivered as "Commercial Computer Software" as defined in DFARS 48 CFR 252.227-7014. All Computer Software and Computer Software Documentation acquired by or for the U.S. Government is provided with Restricted Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7014, as applicable. Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7013, as applicable. Cray, LibSci, and PathScale are federally registered trademarks and Active Manager, Cray Apprentice2, Cray Apprentice2 Desktop, Cray C++ Compiling System, Cray CX, Cray CX1, Cray CX1-iWS, Cray CX1-LC, Cray CX1000, Cray CX1000-C, Cray CX1000-G, Cray CX1000-S, Cray CX1000-SC, Cray CX1000-SM, Cray CX1000-HN, Cray Fortran Compiler, Cray Linux Environment, Cray SHMEM, Cray X1, Cray X1E, Cray X2, Cray XD1, Cray XE, Cray XEm, Cray XE5, Cray XE5m, Cray XE6, Cray XE6m, Cray XK6, Cray XMT, Cray XR1, Cray XT, Cray XTm, Cray XT3, Cray XT4, Cray XT5, Cray XT5 h , Cray XT5m, Cray XT6, Cray XT6m, CrayDoc, CrayPort, CRInform, ECOphlex, Gemini, Libsci, NodeKARE, RapidArray, SeaStar, SeaStar2, SeaStar2+, Sonexion, The Way to Better Science, Threadstorm, uRiKA, and UNICOS/lc are trademarks of Cray Inc. AMD and AMD Opteron are trademarks of Advanced Micro Devices, Inc. GNU is a trademark of The Free Software Foundation. ISO is a trademark of International Organization for Standardization (Organisation Internationale de Normalisation). Linux is a trademark of Linus Torvalds. Lustre is a trademark of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. O2 is a trademark of Silicon Graphics, Inc. NVIDIA and OpenACC are trademarks of NVIDIA Corporation. PGI is a trademark of The Portland Group Compiler Technology, STMicroelectronics, Inc. Platform is a trademark of Platform Computing Corporation. TotalView is a trademark of Rogue Wave Software, Inc. UNIX, the “X device,” X Window System, and X/Open are trademarks of The Open Group in the United States and other countries. All other trademarks are the property of their respective owners. Portions of this document were copied by permission of OpenMP Architecture Review Board from OpenMP C and C++ Application Program Interface, Version 2.0, March 2002, Copyright © 1997-2002, OpenMP Architecture Review Board. RECORD OF REVISION S–2179–80 Published December 2011 Supports the Cray C and C++ compilers running on Cray XE and Cray XK compute nodes. S–2179–74 Published June 2011 Supports the Cray C and C++ compilers running on Cray XT and Cray XE compute nodes. S–2179–73 Published December 2010 Supports the Cray C and C++ compilers running on Cray XT and Cray XE compute nodes. 7.2 Published February 2010 Supports the Cray C and C++ compilers running on Cray XT compute nodes. 7.1 Published June 2009 Supports the Cray C and C++ compilers running on Cray XT compute nodes. 7.0 Published December 2008 Supports the Cray C and (Deferred implementation) C++ compilers running on Cray XT compute nodes.6.0 Published September 2007 Supports the Cray C and Cray C++ 6.0 release running on Cray X1 series and Cray X2 systems. 5.6 Published March 2007 Supports Cray C++ 5.6 and Cray C 8.6 releases running on Cray X1 series systems. 5.5 Published December 2005 Supports Cray C++ 5.5 and Cray C 8.5 releases running on UNICOS/mp 3.0 or later operating systems. 5.4 Published March 2005 Supports Cray C++ 5.4 and Cray C 8.4 releases running on UNICOS/mp 3.0 or later operating systems. 5.3 Published November 2004 Supports Cray C++ 5.3 and Cray C 8.3 releases running on UNICOS/mp 2.5 or later operating systems. 5.2 Published April 2004 Supports Cray C++ 5.2 and Cray C 8.2 releases running on UNICOS/mp 2.3 or later operating systems. 5.1 Published October 2003 Supports Cray C++ 5.1 and Cray C 8.1 releases running on UNICOS/mp 2.2 or later operating systems. 5.0 Published June 2003 Supports Cray C++ 5.0 and Cray C 8.0 releases running on UNICOS/mp 2.1 or later operating systems. 4.3 Published March 31, 2003 Draft version to support Cray C 7.3 and Cray C++ 4.3 releases running on UNICOS/mp operating systems. 4.2 Published December 20, 2002 Draft version to support Cray C 7.2 and Cray C++ 4.2 releases running on UNICOS/mp operating systems. 4.1 Published August 20, 2002 Draft version to support Cray C 7.1 and Cray C++ 4.1 releases running on UNICOS/mp operating systems. 3.6 Published June 2002 This para supports the Cray Standard C 6.6 and Cray Standard C++ 3.6 releases running on UNICOS and UNICOS/mk operating systems. 3.4 Published October 2000 This para supports the Cray C 6.4 and Cray C++ 3.4 releases running on UNICOS and UNICOS/mk operating systems. This para supports a new inlining level, inline4. 3.4 Published August 2000 This para supports the Cray C 6.4 and Cray C++ 3.4 releases running on UNICOS and UNICOS/mk operating systems. It includes updates to para 3.3. 3.3 Published July 1999 This para supports the C and C++ compilers contained in the Cray C++ Programming Environment release 3.3, which is supported on the Cray SV1, Cray C90, Cray J90, and Cray T90 systems running UNICOS 10.0.0.5 and later, and Cray T3E systems running UNICOS/mk 2.0.4 and later. On all supported Cray systems, the C++ compiler is Cray C++ 3.3 and the C compiler is Cray C 6.3. 3.2 Published January 1999 This para supports the C and C++ compilers contained in the Cray C++ Programming Environment release 3.2, which is supported on all systems except the Cray T3D system. On all supported Cray systems, the C++ compiler is Cray C++ 3.2 and the C compiler is Cray C 6.2. 3.1 Published August 1998 This para supports the C and C++ compilers contained in the Cray C++ Programming Environment release 3.1, which is supported on all systems except the Cray T3D system. On all supported Cray systems, the C++ compiler is Cray C++ 3.1 and the C compiler is Cray C 6.1. 3.0.2 Published March 1998 This para supports the C and C++ compilers contained in the Cray C++ Programming Environment release 3.0.2, which is supported on all systems except the Cray T3D system. On all supported Cray systems, the C++ compiler is Cray C++ 3.0.2 and the C compiler is Cray C 6.0.2. 3.0 Published May 1997 This rewrite supports the C and C++ compilers contained in the Cray C++ Programming Environment release 3.0, which is supported on all systems except the Cray T3D system. On all supported Cray systems, the C++ compiler is Cray C++ 3.0 and the C compiler is Cray C 6.0. 2.0 Published January 1996 Original Printing. This manual supports the C and C++ compilers contained in the Cray C++ Programming Environment release 2.0. On all Cray systems, the C++ compiler is Cray C++ 2.0. On Cray systems with IEEE floating-point hardware, the C compiler is Cray Standard C 5.0. On Cray systems without IEEE floating-point hardware, the C compiler is Cray Standard C 4.0.Changes to this Document Cray C and C++ Reference Manual S–2179–80 S–2179–80 Added • Add Support for 128-bit floating point and 256-bit complex predefined types and quad-precision intrinsic functions. See 128-Bit Floating Point and 256-Bit Complex Predefined Types on page 160. • A new class of compiler directives provide support for accelerators on Cray XK systems. See Chapter 5, Using OpenACC on page 109 and OpenMP Accelerator Support on page 99. • New compiler command line options selectively control recognition of OpenMP accelerator directives and other OpenMP directives (#pragma omp). See -h [no]omp_acc on page 59. • New compiler command line options selectively control recognition of OpenACC directives (#pragma acc). See -h [no]acc on page 57. • The ISO_C_BINDING module provides interoperability between Fortran intrinsic types and C types. See Standard Fortran/C Interoperability on page 155. • By default, the compiler now uses a modified malloc implementation that offers better support for Cray XE memory needs. A new command line option directs the compiler to link in the native malloc provided by the OS instead of the modified implementation. See -h [system|default]_alloc on page 55. • A new command line option controls the aggressiveness of optimizations which may affect floating point and complex repeatability when application requirements require identical results when varying number of ranks or threads. Using the -hflex_mp option improves the probability of reproducible results, but does not guarantee them. See -h flex_mp=level on page 31. • The new nofission directive allows you to prevent the compiler from splitting statements in a specified loop into distinct loops. See nofission Directive on page 90. • New environment variables CRAY_ACC_DEBUG and PGAS_ERROR_FILE. See Run Time Environment Variables on page 64 and OpenMP Accelerator Environment Variables on page 107. Revised • The CCE 8.0 release does not support Cray XT systems. • The -g debugging option is now supported with OpenMP directives. See -G level and -g on page 46.S–2179–74 Added • The -h [no]autoprefetch option controls automatic prefetch optimization. See -h [no]autoprefetch on page 30. • The -h [no]add_paren option automatically adds parenthesis to select associative operations (+,-,*) to encourage left to right evaluation of floating point and complex expressions. The default is -hnoadd_paren. See -h [no]add_paren on page 30. • The _amo_aflush and _amo_aflush_upc functions force *ptr to be written to memory. See Global Atomic Memory Operations on page 190. • The clone_always and clone_never directives affect cloning behavior. See Interprocedural Analysis (IPA) Optimization Options on page 39, and Inlining and Cloning Directives on page 90. • Class name injection is disabled. Statement of support deleted from Supported C++ Language Features on page 167. • Wide characters (wchar_t) are not supported. Statement of support deleted from Supported C++ Language Features on page 167. • New UPC extensions defined in cray_upc.h. See Privatizability (Castability) Functions on page 122, Shared String Handling Functions, Non-blocking Versions on page 123, and Node Affinity on page 122, and the intro_pgas(7) man page. S–2179–73 Added • A new command line option instructs the compiler to initialize undefined local stack variables to 0 (zero). See -h zero on page 47. • A new set of options cause the compiler backend (optimizer, inliner, codegenerator) to be invoked at application link time, allowing full whole program automatic inlining and future whole program interprocedural analysis (IPA) optimizations. See -h wp on page 36 and -h pl=program_library on page 35. • A new option enables traps for specified exceptions. See -K trap=opt[,opt] ... on page 60 and -h [no]fp_trap on page 57. • Global and local atomic memory operations are supported. See Atomic Memory Operations on page 188. • On Cray XE systems, an enhancement to the address format of the pointer to shared type allows the number of PEs available to a single Fortran or UPC job to increase from 2**16 to 2**31-1. See -X npes on page 61. • On Cray XE systems, a new class of compiler directive supports PGAS programming models. See PGAS Directive on page 93. Revised • This version of the CCE release supports the ISO 1998 Standard Template Library headers while support for pre-standard template headers has been removed. See C++ Language Conformance on page 167. • This compiler includes partial support for the proposed OpenMP 3.1 standard specification and supports the OpenMP Application Program Interface, Version 3.0 standard, with limitations. See OpenMP 3.1 Standard Support on page 99.Contents Page Introduction [1] 17 1.1 General Compiler Description . . . . . . . . . . . . . . . . . . . . 17 1.1.1 Cray C Compiler . . . . . . . . . . . . . . . . . . . . . . . 17 1.1.2 Cray C++ Compiler . . . . . . . . . . . . . . . . . . . . . . 17 1.2 Related Publications . . . . . . . . . . . . . . . . . . . . . . . 18 Invoking the C and C++ Compilers [2] 19 2.1 CC Command . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.2 cc Command . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.3 Command Line Options . . . . . . . . . . . . . . . . . . . . . . 21 2.4 Standard Language Conformance Options . . . . . . . . . . . . . . . . . 22 2.4.1 -h [no]c99 (cc) . . . . . . . . . . . . . . . . . . . . . 22 2.4.2 -h [no]conform (CC, cc), -h [no]stdc (cc) . . . . . . . . . . . . 22 2.4.3 -h cfront (CC) . . . . . . . . . . . . . . . . . . . . . . 23 2.4.4 -h [no]parse_templates (CC) . . . . . . . . . . . . . . . . 23 2.4.5 -h [no]dep_name (CC) . . . . . . . . . . . . . . . . . . . 23 2.4.6 -h [no]exceptions (CC) . . . . . . . . . . . . . . . . . . 23 2.4.7 -h [no]anachronisms (CC) . . . . . . . . . . . . . . . . . 23 2.4.8 -h [no]new_for_init (CC) . . . . . . . . . . . . . . . . . 24 2.4.9 -h [no]tolerant (cc) . . . . . . . . . . . . . . . . . . . 24 2.4.10 -h [no]const_string_literals (CC) . . . . . . . . . . . . 24 2.4.11 -h [no]gnu . . . . . . . . . . . . . . . . . . . . . . . 25 2.5 Virtual Function Options . . . . . . . . . . . . . . . . . . . . . . 27 2.5.1 -h forcevtbl (CC) . . . . . . . . . . . . . . . . . . . . 27 2.5.2 -h suppressvtbl (CC) . . . . . . . . . . . . . . . . . . . 27 2.6 General Language Options . . . . . . . . . . . . . . . . . . . . . 27 2.6.1 -h keep=file (CC) . . . . . . . . . . . . . . . . . . . . . 28 2.6.2 -h restrict=args . . . . . . . . . . . . . . . . . . . . . 28 2.6.3 -h [no]calchars . . . . . . . . . . . . . . . . . . . . . 29 2.6.4 -h [no]signedshifts . . . . . . . . . . . . . . . . . . . 29 S–2179–80 7Cray C and C++ Reference Manual Page 2.7 General Optimization Options . . . . . . . . . . . . . . . . . . . . 30 2.7.1 -h [no]add_paren . . . . . . . . . . . . . . . . . . . . 30 2.7.2 -h [no]aggress . . . . . . . . . . . . . . . . . . . . . 30 2.7.3 -h [no]autoprefetch . . . . . . . . . . . . . . . . . . . 30 2.7.4 -h [no]autothread . . . . . . . . . . . . . . . . . . . . 30 2.7.5 -h display_opt . . . . . . . . . . . . . . . . . . . . . 30 2.7.6 -h [no]dwarf . . . . . . . . . . . . . . . . . . . . . . 31 2.7.7 -h flex_mp=level . . . . . . . . . . . . . . . . . . . . . 31 2.7.8 -h fusionn . . . . . . . . . . . . . . . . . . . . . . . 31 2.7.9 -h [no]intrinsics . . . . . . . . . . . . . . . . . . . . 32 2.7.10 -h list . . . . . . . . . . . . . . . . . . . . . . . . 32 2.7.11 -h [no]msgs . . . . . . . . . . . . . . . . . . . . . . 33 2.7.12 -h [no]negmsgs . . . . . . . . . . . . . . . . . . . . . 33 2.7.13 -h [no]omp_trace . . . . . . . . . . . . . . . . . . . . 34 2.7.14 -h [no]func_trace . . . . . . . . . . . . . . . . . . . 34 2.7.15 -h [no]overindex . . . . . . . . . . . . . . . . . . . . 34 2.7.16 -h [no]pattern . . . . . . . . . . . . . . . . . . . . . 34 2.7.17 -h pl=program_library . . . . . . . . . . . . . . . . . . . 35 2.7.18 -h profile_generate . . . . . . . . . . . . . . . . . . 35 2.7.19 -h threadn . . . . . . . . . . . . . . . . . . . . . . 35 2.7.20 -h unrolln . . . . . . . . . . . . . . . . . . . . . . 36 2.7.21 -h wp . . . . . . . . . . . . . . . . . . . . . . . . . 36 2.7.22 -O level . . . . . . . . . . . . . . . . . . . . . . . . 37 2.8 Automatic Cache Management Options . . . . . . . . . . . . . . . . . . 37 2.8.1 -h cachen . . . . . . . . . . . . . . . . . . . . . . . 37 2.9 Vector Optimization Options . . . . . . . . . . . . . . . . . . . . . 38 2.9.1 -h vectorn . . . . . . . . . . . . . . . . . . . . . . . 38 2.10 Interprocedural Analysis (IPA) Optimization Options . . . . . . . . . . . . . . 39 2.10.1 Inlining . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.10.2 Cloning . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.10.3 -h ipan . . . . . . . . . . . . . . . . . . . . . . . . 40 2.10.4 -h ipafrom=source[:source] ... . . . . . . . . . . . . . . . 41 2.11 Scalar Optimization Options . . . . . . . . . . . . . . . . . . . . 42 2.11.1 -h [no]interchange . . . . . . . . . . . . . . . . . . . 42 2.11.2 -h scalarn . . . . . . . . . . . . . . . . . . . . . . 42 2.11.3 -h [no]zeroinc . . . . . . . . . . . . . . . . . . . . . 43 2.12 Math Options . . . . . . . . . . . . . . . . . . . . . . . . 43 8 S–2179–80Contents Page 2.12.1 -h fpn . . . . . . . . . . . . . . . . . . . . . . . . 43 2.12.2 -h matherror . . . . . . . . . . . . . . . . . . . . . 46 2.13 Debugging Options . . . . . . . . . . . . . . . . . . . . . . . 46 2.13.1 -G level and -g . . . . . . . . . . . . . . . . . . . . . . 46 2.13.2 -h [no]bounds (cc) . . . . . . . . . . . . . . . . . . . . 47 2.13.3 -h dir_check . . . . . . . . . . . . . . . . . . . . . 47 2.13.4 -h zero . . . . . . . . . . . . . . . . . . . . . . . . 47 2.14 Compiler Message Options . . . . . . . . . . . . . . . . . . . . . 48 2.14.1 -h msglevel_n . . . . . . . . . . . . . . . . . . . . . 48 2.14.2 -h [no]message=n[:n...] . . . . . . . . . . . . . . . . . 48 2.14.3 -h report=args . . . . . . . . . . . . . . . . . . . . . 49 2.14.4 -h [no]abort . . . . . . . . . . . . . . . . . . . . . . 49 2.14.5 -h errorlimit . . . . . . . . . . . . . . . . . . . . . 49 2.15 Compilation Phase Options . . . . . . . . . . . . . . . . . . . . . 49 2.15.1 -E . . . . . . . . . . . . . . . . . . . . . . . . . . 50 2.15.2 -P . . . . . . . . . . . . . . . . . . . . . . . . . . 50 2.15.3 -h feonly . . . . . . . . . . . . . . . . . . . . . . . 50 2.15.4 -S . . . . . . . . . . . . . . . . . . . . . . . . . . 50 2.15.5 -c . . . . . . . . . . . . . . . . . . . . . . . . . . 50 2.15.6 -#, -##, and -### . . . . . . . . . . . . . . . . . . . . . 50 2.15.7 -W phase,"opt ..." . . . . . . . . . . . . . . . . . . . . . 51 2.15.8 -Y phase,dirname . . . . . . . . . . . . . . . . . . . . . 51 2.16 Preprocessing Options . . . . . . . . . . . . . . . . . . . . . . 52 2.16.1 -C . . . . . . . . . . . . . . . . . . . . . . . . . . 52 2.16.2 -D macro[=def] . . . . . . . . . . . . . . . . . . . . . 52 2.16.3 -h [no]pragma=name[:name ...] . . . . . . . . . . . . . . 52 2.16.4 -I incldir . . . . . . . . . . . . . . . . . . . . . . . . 53 2.16.5 -M . . . . . . . . . . . . . . . . . . . . . . . . . . 54 2.16.6 -nostdinc . . . . . . . . . . . . . . . . . . . . . . . 54 2.16.7 -U . . . . . . . . . . . . . . . . . . . . . . . . . . 55 2.17 Linker Options . . . . . . . . . . . . . . . . . . . . . . . . 55 2.17.1 -h [system|default]_alloc . . . . . . . . . . . . . . . . 55 2.17.2 -l libname . . . . . . . . . . . . . . . . . . . . . . . 55 2.17.3 -L ldir . . . . . . . . . . . . . . . . . . . . . . . . 56 2.17.4 -rpath ldir . . . . . . . . . . . . . . . . . . . . . . . 56 2.17.5 -o outfile . . . . . . . . . . . . . . . . . . . . . . . . 56 2.18 Miscellaneous Options . . . . . . . . . . . . . . . . . . . . . . 57 S–2179–80 9Cray C and C++ Reference Manual Page 2.18.1 -h [no]acc . . . . . . . . . . . . . . . . . . . . . . 57 2.18.2 -h cpu=target_system . . . . . . . . . . . . . . . . . . . . 57 2.18.3 -h [no]fp_trap . . . . . . . . . . . . . . . . . . . . . 57 2.18.4 -h ident=name . . . . . . . . . . . . . . . . . . . . . 58 2.18.5 -h keepfiles . . . . . . . . . . . . . . . . . . . . . 58 2.18.6 -h loop_trips=[tiny | small | medium | large | huge] . . . . . . . 58 2.18.7 -h mpin . . . . . . . . . . . . . . . . . . . . . . . . 58 2.18.8 -h network=nic . . . . . . . . . . . . . . . . . . . . . 58 2.18.9 -h [no]omp . . . . . . . . . . . . . . . . . . . . . . 58 2.18.10 -h [no]omp_acc . . . . . . . . . . . . . . . . . . . . 59 2.18.11 -h pic, -h PIC . . . . . . . . . . . . . . . . . . . . . 59 2.18.12 -h prototype_intrinsics . . . . . . . . . . . . . . . . 59 2.18.13 -h [no]threadsafe . . . . . . . . . . . . . . . . . . . 59 2.18.14 -h upc (cc) . . . . . . . . . . . . . . . . . . . . . . 59 2.18.15 -K trap=opt[,opt] ... . . . . . . . . . . . . . . . . . . . . 60 2.18.16 -V . . . . . . . . . . . . . . . . . . . . . . . . . . 60 2.18.17 -X npes . . . . . . . . . . . . . . . . . . . . . . . . 61 2.19 Command Line Examples . . . . . . . . . . . . . . . . . . . . . 61 2.20 Compile Time Environment Variables . . . . . . . . . . . . . . . . . . 62 2.21 Run Time Environment Variables . . . . . . . . . . . . . . . . . . . 64 2.22 OpenMP Environment Variables . . . . . . . . . . . . . . . . . . . 64 Using #pragma Directives [3] 65 3.1 Protecting Directives . . . . . . . . . . . . . . . . . . . . . . . 66 3.2 Directives in Cray C++ . . . . . . . . . . . . . . . . . . . . . . 66 3.3 Loop Directives . . . . . . . . . . . . . . . . . . . . . . . . 66 3.4 Alternative Directive Form: _Pragma . . . . . . . . . . . . . . . . . . 67 3.5 General Directives . . . . . . . . . . . . . . . . . . . . . . . 67 3.5.1 [no]bounds Directive . . . . . . . . . . . . . . . . . . . . 68 3.5.2 duplicate Directive . . . . . . . . . . . . . . . . . . . . 68 3.5.3 message Directive . . . . . . . . . . . . . . . . . . . . . 70 3.5.4 cache Directive . . . . . . . . . . . . . . . . . . . . . . 71 3.5.5 cache_nt Directive . . . . . . . . . . . . . . . . . . . . . 71 3.5.6 ident Directive . . . . . . . . . . . . . . . . . . . . . . 71 3.5.7 [no]opt Directive . . . . . . . . . . . . . . . . . . . . . 72 3.5.8 autothread, noautothread Directives . . . . . . . . . . . . . . 72 3.5.9 Probability Directives . . . . . . . . . . . . . . . . . . . . . 73 3.5.10 weak Directive . . . . . . . . . . . . . . . . . . . . . . 74 10 S–2179–80Contents Page 3.6 Vectorization Directives . . . . . . . . . . . . . . . . . . . . . . 75 3.6.1 hand_tuned Directive . . . . . . . . . . . . . . . . . . . . 75 3.6.2 loop_info Directive . . . . . . . . . . . . . . . . . . . . 76 3.6.3 loop_info prefer_thread, prefer_nothread Directives . . . . . . . 78 3.6.4 nopattern Directive . . . . . . . . . . . . . . . . . . . . 78 3.6.5 novector Directive . . . . . . . . . . . . . . . . . . . . . 79 3.6.6 permutation Directive . . . . . . . . . . . . . . . . . . . . 79 3.6.7 [no]pipeline Directive . . . . . . . . . . . . . . . . . . . 80 3.6.8 prefervector Directive . . . . . . . . . . . . . . . . . . . 80 3.6.9 pgo loop_info Directive . . . . . . . . . . . . . . . . . . . 81 3.6.10 safe_address Directive . . . . . . . . . . . . . . . . . . . 81 3.6.11 safe_conditional Directive . . . . . . . . . . . . . . . . . 82 3.6.12 shortloop and shortloop128 Directives . . . . . . . . . . . . . 83 3.7 Scalar Directives . . . . . . . . . . . . . . . . . . . . . . . . 84 3.7.1 blockable Directive . . . . . . . . . . . . . . . . . . . . 84 3.7.2 blockingsize Directive . . . . . . . . . . . . . . . . . . . 84 3.7.3 noblocking Directive . . . . . . . . . . . . . . . . . . . . 85 3.7.4 collapse and nocollapse Directives . . . . . . . . . . . . . . . 85 3.7.5 concurrent Directive . . . . . . . . . . . . . . . . . . . . 86 3.7.6 [no]interchange Directive . . . . . . . . . . . . . . . . . . 87 3.7.7 noreduction Directive . . . . . . . . . . . . . . . . . . . . 87 3.7.8 suppress Directive . . . . . . . . . . . . . . . . . . . . . 88 3.7.9 [no]unroll Directive . . . . . . . . . . . . . . . . . . . . 88 3.7.10 nofission Directive . . . . . . . . . . . . . . . . . . . . 90 3.7.11 [no]fusion Directive . . . . . . . . . . . . . . . . . . . . 90 3.8 Inlining and Cloning Directives . . . . . . . . . . . . . . . . . . . . 90 3.8.1 inline_enable, inline_disable, and inline_reset Directives . . . . . 91 3.8.2 inline_always and inline_never Directives . . . . . . . . . . . . 92 3.8.3 clone_enable, clone_disable, clone_reset Directives . . . . . . . 92 3.8.4 clone_always and clone_never Directives . . . . . . . . . . . . . 93 3.9 PGAS Directive . . . . . . . . . . . . . . . . . . . . . . . . 93 3.9.1 defer_sync Directive . . . . . . . . . . . . . . . . . . . . 93 Using OpenMP [4] 95 4.1 Deferred OpenMP Features . . . . . . . . . . . . . . . . . . . . . 95 4.2 Cray Implementation Differences . . . . . . . . . . . . . . . . . . . 95 4.2.1 Pragmas . . . . . . . . . . . . . . . . . . . . . . . . . 95 4.2.1.1 atomic Construct . . . . . . . . . . . . . . . . . . . . 95 S–2179–80 11Cray C and C++ Reference Manual Page 4.2.1.2 for Construct . . . . . . . . . . . . . . . . . . . . . . 95 4.2.1.3 parallel Construct . . . . . . . . . . . . . . . . . . . 96 4.2.1.4 private Clause . . . . . . . . . . . . . . . . . . . . . 96 4.2.1.5 threadprivate Construct . . . . . . . . . . . . . . . . . 96 4.2.2 OpenMP Library Routines . . . . . . . . . . . . . . . . . . . . 96 4.2.2.1 omp_get_max_active_levels() . . . . . . . . . . . . . . . 96 4.2.2.2 omp_set_dynamic() . . . . . . . . . . . . . . . . . . . 97 4.2.2.3 omp_set_schedule() . . . . . . . . . . . . . . . . . . 97 4.2.2.4 omp_set_max_active_levels() . . . . . . . . . . . . . . . 97 4.2.2.5 omp_set_nested() . . . . . . . . . . . . . . . . . . . 97 4.2.2.6 omp_set_num_threads() . . . . . . . . . . . . . . . . . 97 4.2.3 OpenMP Environment Variables . . . . . . . . . . . . . . . . . . 97 4.2.3.1 OMP_DYNAMIC . . . . . . . . . . . . . . . . . . . . . 97 4.2.3.2 OMP_MAX_ACTIVE_LEVELS . . . . . . . . . . . . . . . . 97 4.2.3.3 OMP_NESTED . . . . . . . . . . . . . . . . . . . . . 97 4.2.3.4 OMP_NUM_THREADS . . . . . . . . . . . . . . . . . . . 98 4.2.3.5 OMP_SCHEDULE . . . . . . . . . . . . . . . . . . . . 98 4.2.3.6 OMP_STACKSIZE . . . . . . . . . . . . . . . . . . . . 98 4.2.3.7 OMP_THREAD_LIMIT . . . . . . . . . . . . . . . . . . . 98 4.2.3.8 OMP_WAIT_POLICY . . . . . . . . . . . . . . . . . . . 98 4.3 Compiler Options Affecting OpenMP . . . . . . . . . . . . . . . . . . 98 4.4 OpenMP Program Execution . . . . . . . . . . . . . . . . . . . . . 99 4.5 OpenMP 3.1 Standard Support . . . . . . . . . . . . . . . . . . . . 99 4.6 OpenMP Accelerator Support . . . . . . . . . . . . . . . . . . . . 99 4.6.1 OpenMP Accelerator Execution Model . . . . . . . . . . . . . . . . 99 4.6.2 OpenMP Accelerator Memory Model . . . . . . . . . . . . . . . . . 100 4.6.3 OpenMP Accelerator Directives . . . . . . . . . . . . . . . . . . 100 4.6.3.1 acc_region . . . . . . . . . . . . . . . . . . . . . 101 4.6.3.2 acc_data . . . . . . . . . . . . . . . . . . . . . . 103 4.6.3.3 acc_loop . . . . . . . . . . . . . . . . . . . . . . 105 4.6.3.4 acc_region_loop . . . . . . . . . . . . . . . . . . . 106 4.6.3.5 acc_update . . . . . . . . . . . . . . . . . . . . . 106 4.6.4 OpenMP Accelerator Runtime Functions . . . . . . . . . . . . . . . . 107 4.6.5 OpenMP Accelerator Environment Variables . . . . . . . . . . . . . . . 107 4.6.6 OpenMP Accelerator Module Support . . . . . . . . . . . . . . . . . 108 Using OpenACC [5] 109 5.1 OpenACC Execution Model . . . . . . . . . . . . . . . . . . . . . 109 12 S–2179–80Contents Page 5.2 OpenACC Memory Model . . . . . . . . . . . . . . . . . . . . . 109 5.3 OpenACC Directives . . . . . . . . . . . . . . . . . . . . . . . 110 5.3.1 parallel . . . . . . . . . . . . . . . . . . . . . . . . 110 5.3.2 loop . . . . . . . . . . . . . . . . . . . . . . . . . 113 5.3.3 parallel_loop . . . . . . . . . . . . . . . . . . . . . 114 5.3.4 data . . . . . . . . . . . . . . . . . . . . . . . . . 114 5.3.5 host_data . . . . . . . . . . . . . . . . . . . . . . . 117 5.3.6 update . . . . . . . . . . . . . . . . . . . . . . . . . 117 5.4 OpenACC Runtime Routines . . . . . . . . . . . . . . . . . . . . . 118 5.5 OpenACC Environment Variables . . . . . . . . . . . . . . . . . . . 118 5.6 Compiler Options . . . . . . . . . . . . . . . . . . . . . . . . 118 Using Cray Unified Parallel C (UPC) [6] 119 6.1 Cray Implementation Differences . . . . . . . . . . . . . . . . . . . 120 6.2 Compiling and Linking UPC Code . . . . . . . . . . . . . . . . . . . 120 6.3 Launching a UPC Application . . . . . . . . . . . . . . . . . . . . 121 6.4 Cray Extensions Defined in cray_upc.h . . . . . . . . . . . . . . . . 121 6.4.1 Collective Deallocation . . . . . . . . . . . . . . . . . . . . . 122 6.4.1.1 upc_all_free() . . . . . . . . . . . . . . . . . . . . 122 6.4.1.2 upc_all_lock_free() . . . . . . . . . . . . . . . . . . 122 6.4.2 Node Affinity . . . . . . . . . . . . . . . . . . . . . . . 122 6.4.2.1 upc_nodeof() . . . . . . . . . . . . . . . . . . . . . 122 6.4.2.2 NODES . . . . . . . . . . . . . . . . . . . . . . . . 122 6.4.2.3 MYNODE . . . . . . . . . . . . . . . . . . . . . . . 122 6.4.3 Privatizability (Castability) Functions . . . . . . . . . . . . . . . . . 122 6.4.3.1 upc_cast() . . . . . . . . . . . . . . . . . . . . . . 122 6.4.3.2 upc_castable() . . . . . . . . . . . . . . . . . . . . 122 6.4.3.3 upc_thread_castable() . . . . . . . . . . . . . . . . . 123 6.4.4 Shared String Handling Functions, Non-blocking Versions . . . . . . . . . . . 123 6.4.4.1 upc_memcpy_nb(), upc_memcpy_nbi() . . . . . . . . . . . . . 123 6.4.4.2 upc_memput_nb(), upc_memput_nbi() . . . . . . . . . . . . . 123 6.4.4.3 upc_memget_nb(), upc_memget_nbi() . . . . . . . . . . . . . 123 6.4.4.4 upc_sync_nb(), upc_test_nb() . . . . . . . . . . . . . . . 123 6.4.5 Timing . . . . . . . . . . . . . . . . . . . . . . . . . 123 Using Cray C++ Libraries [7] 125 7.1 Unsupported Standard C++ Library Features . . . . . . . . . . . . . . . . 125 S–2179–80 13Cray C and C++ Reference Manual Page Using Cray C Extensions [8] 127 8.1 Complex Data Extensions . . . . . . . . . . . . . . . . . . . . . 127 8.2 fortran Keyword . . . . . . . . . . . . . . . . . . . . . . . 127 8.3 Hexadecimal Floating-point Constants . . . . . . . . . . . . . . . . . . 128 Using Predefined Macros [9] 131 9.1 Macros Required by the C and C++ Standards . . . . . . . . . . . . . . . . 132 9.2 Macros Based on the Host Machine . . . . . . . . . . . . . . . . . . . 132 9.3 Macros Based on the Target Machine . . . . . . . . . . . . . . . . . . 133 9.4 Macros Based on the Compiler . . . . . . . . . . . . . . . . . . . . 133 9.5 UPC Predefined Macros . . . . . . . . . . . . . . . . . . . . . . 134 Running C and C++ Applications [10] 135 Debugging Cray C and C++ Code [11] 137 11.1 TotalView Debugger . . . . . . . . . . . . . . . . . . . . . . . 138 11.2 Compiler Debugging Options . . . . . . . . . . . . . . . . . . . . 138 Using Interlanguage Communication [12] 141 12.1 Calls Between C and C++ Functions . . . . . . . . . . . . . . . . . . 141 12.2 Calling Fortran Functions and Subroutines from C or C++ . . . . . . . . . . . . 143 12.2.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . 143 12.2.2 Argument Passing . . . . . . . . . . . . . . . . . . . . . . 143 12.2.3 Array Storage . . . . . . . . . . . . . . . . . . . . . . . 144 12.2.4 Logical and Character Data . . . . . . . . . . . . . . . . . . . 145 12.2.5 Accessing Named Common from C and C++ . . . . . . . . . . . . . . 145 12.2.6 Accessing Blank Common from C or C++ . . . . . . . . . . . . . . . 147 12.2.7 Cray C and Fortran Example . . . . . . . . . . . . . . . . . . . 149 12.2.8 Calling a Fortran Program from Cray C++ . . . . . . . . . . . . . . . 151 12.3 Calling a C or C++ Function from Fortran . . . . . . . . . . . . . . . . . 151 12.3.1 Portable Interoperability Mechanism . . . . . . . . . . . . . . . . . 152 12.3.2 Standard Fortran/C Interoperability . . . . . . . . . . . . . . . . . 155 Implementation-defined Behavior [13] 157 13.1 Messages . . . . . . . . . . . . . . . . . . . . . . . . . . 157 13.2 Environment . . . . . . . . . . . . . . . . . . . . . . . . . 157 13.2.1 Identifiers . . . . . . . . . . . . . . . . . . . . . . . . 158 13.2.2 Types . . . . . . . . . . . . . . . . . . . . . . . . . 158 13.2.3 Characters . . . . . . . . . . . . . . . . . . . . . . . . 159 13.2.4 Wide Characters . . . . . . . . . . . . . . . . . . . . . . 160 14 S–2179–80Contents Page 13.2.5 Integers . . . . . . . . . . . . . . . . . . . . . . . . . 160 13.2.6 128-Bit Floating Point and 256-Bit Complex Predefined Types . . . . . . . . . . 160 13.2.7 Arrays and Pointers . . . . . . . . . . . . . . . . . . . . . 161 13.2.8 Registers . . . . . . . . . . . . . . . . . . . . . . . . 161 13.2.9 Classes, Structures, Unions, Enumerations, and Bit Fields . . . . . . . . . . . 161 13.2.10 Qualifiers . . . . . . . . . . . . . . . . . . . . . . . . 162 13.2.11 Declarators . . . . . . . . . . . . . . . . . . . . . . . 162 13.2.12 Statements . . . . . . . . . . . . . . . . . . . . . . . . 162 13.2.13 Exceptions . . . . . . . . . . . . . . . . . . . . . . . . 162 13.2.14 System Function Calls . . . . . . . . . . . . . . . . . . . . 163 13.3 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . 163 Appendix A Using Libraries and the Linker 165 A.1 Cray C and C++ Libraries . . . . . . . . . . . . . . . . . . . . . 165 A.2 Linker . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Appendix B Using Cray C and C++ Dialects 167 B.1 C++ Language Conformance . . . . . . . . . . . . . . . . . . . . 167 B.1.1 Supported C++ Language Features . . . . . . . . . . . . . . . . . . 167 B.2 C++ Anachronisms Accepted . . . . . . . . . . . . . . . . . . . . 171 B.3 Extensions Accepted in Normal C++ Mode . . . . . . . . . . . . . . . . . 172 B.4 Extensions Accepted in C or C++ Mode . . . . . . . . . . . . . . . . . 173 B.5 C++ Extensions Accepted in cfront Compatibility Mode . . . . . . . . . . . . 175 Appendix C Using the Compiler Message System 181 C.1 Expanding Messages with the explain Command . . . . . . . . . . . . . . 181 C.2 Controlling the Use of Messages . . . . . . . . . . . . . . . . . . . 181 C.2.1 Command Line Options . . . . . . . . . . . . . . . . . . . . . 182 C.2.2 Environment Options for Messages . . . . . . . . . . . . . . . . . 182 C.2.3 ORIG_CMD_NAME Environment Variable . . . . . . . . . . . . . . . 183 C.3 Message Severity . . . . . . . . . . . . . . . . . . . . . . . . 183 C.4 Common System Messages . . . . . . . . . . . . . . . . . . . . . 184 Appendix D Using Intrinsic Functions 187 D.1 Atomic Memory Operations . . . . . . . . . . . . . . . . . . . . . 188 D.1.1 Local Atomic Memory Operations . . . . . . . . . . . . . . . . . . 189 D.1.2 Global Atomic Memory Operations . . . . . . . . . . . . . . . . . 190 D.2 Bit Operations . . . . . . . . . . . . . . . . . . . . . . . . 192 D.3 Mask Operations . . . . . . . . . . . . . . . . . . . . . . . . 192 S–2179–80 15Cray C and C++ Reference Manual Page D.4 Miscellaneous Operations . . . . . . . . . . . . . . . . . . . . . 193 Tables Table 1. GCC C Language Extensions . . . . . . . . . . . . . . . . . . 25 Table 2. GCC C++ Language Extensions . . . . . . . . . . . . . . . . . . 27 Table 3. -h Option Descriptions . . . . . . . . . . . . . . . . . . . . 37 Table 4. Cache Levels . . . . . . . . . . . . . . . . . . . . . . . 38 Table 5. IPA Level . . . . . . . . . . . . . . . . . . . . . . . . 40 Table 6. File Types . . . . . . . . . . . . . . . . . . . . . . . . 42 Table 7. Floating-point Optimization Levels . . . . . . . . . . . . . . . . . 45 Table 8. -G level Definitions . . . . . . . . . . . . . . . . . . . . . 46 Table 9. -W phase Definitions . . . . . . . . . . . . . . . . . . . . . 51 Table 10. -Y phase Definitions . . . . . . . . . . . . . . . . . . . . 51 Table 11. -h pragma Directive Processing . . . . . . . . . . . . . . . . 53 Table 12. Data Type Mapping . . . . . . . . . . . . . . . . . . . . . 158 Table 13. Packed Characters . . . . . . . . . . . . . . . . . . . . . 159 Table 14. Unrecognizable Escape Sequences . . . . . . . . . . . . . . . . . 159 Examples Example 1. CC -X8 -h myprog.C . . . . . . . . . . . . . . . . . . 61 Example 2. CC -h conform myprog.C . . . . . . . . . . . . . . . . 61 Example 3. cc -c -h ipa1 myprog.c subprog.c . . . . . . . . . . . 61 Example 4. cc -I. disc.c vend.c . . . . . . . . . . . . . . . . . 61 Example 5. cc -P -D DEBUG newprog.c . . . . . . . . . . . . . . . 62 Example 6. cc -c -h report=s mydata1.c . . . . . . . . . . . . . 62 Example 7. CC -h ipa5,report=if myfile.C . . . . . . . . . . . . . 62 Example 8. Trip counts . . . . . . . . . . . . . . . . . . . . . . . 77 Example 9. Unrolling outer loops . . . . . . . . . . . . . . . . . . . . 89 Example 10. Illegal unrolling of outer loops . . . . . . . . . . . . . . . . . 90 Example 11. Using the inline_enable, inline_disable, and inline_reset directives 91 Example 12. Using inline_reset . . . . . . . . . . . . . . . . . . 92 Example 13. Using defer_sync . . . . . . . . . . . . . . . . . . . 94 Example 14. Calling a C function from Fortran . . . . . . . . . . . . . . . . 153 16 S–2179–80Introduction [1] The Cray Compiling Environment (CCE) contains both the Cray C and C++ compilers. The Cray C compiler conforms to the International Organization of Standards (ISO) standard ISO/IEC 9899:1999 (C99). The Cray C++ compiler conforms to the ISO/IEC 14882:2003 standard, with some exceptions. The exceptions are noted in Appendix B, Using Cray C and C++ Dialects on page 167. Log in either to a login node or a standalone application development system and use the Cray XE or Cray XK Environment and related products to create an application which executes on compute nodes. For further information about login nodes and the user environment, see the Cray Application Developer's Environment User's Guide. For further information about the standalone application development platform, see the Cray Application Developer's Environment Installation Guide. Throughout this manual, the differences between the Cray C and C++ compilers are noted when appropriate. When there is no difference, the phrase the compiler refers to both compilers. All compiler command options apply to Cray C and C++ unless noted. 1.1 General Compiler Description Both the Cray C and C++ compilers are contained within the Cray Compiling Environment (CCE). If compiling code written in C, use the cc command to compile source files. If you are compiling code written in C++, use the CC command. 1.1.1 Cray C Compiler The Cray C compiler consists of a preprocessor, a language parser, an optimizer, and a code generator. Invoke the Cray C compiler with the cc compiler driver command. The cc command is described in cc Command on page 20. This command and its options are also described in the craycc(1) man page. See Command Line Examples on page 61. 1.1.2 Cray C++ Compiler The Cray C++ compiler consists of a preprocessor, a language parser, an optimizer, and a code generator. Invoke the Cray C++ compiler with the CC compiler driver command. The CC command is described in CC Command on page 20 and the crayCC(1) man page. See Command Line Examples on page 61. S–2179–80 17Cray C and C++ Reference Manual 1.2 Related Publications The following documents contain additional information that may be helpful: • cc(1) compiler driver man page for all Cray XE and Cray XK C compilers • craycc(1) man page for the Cray C compiler • CC(1) compiler driver man page for all Cray XE and Cray XK C++ compilers • crayCC(1) man page for the Cray C++ compiler • intro_pragmas(1) man page • Cray Fortran Reference Manual • ftn(1) compiler driver man page for all Cray XE and Cray XK Fortran compilers • crayftn(1) man page for the Cray Fortran compiler • Cray Application Developer's Environment User's Guide • aprun(1) man page • Using Cray Performance Analysis Tools • Cray Application Developer's Environment Installation Guide 18 S–2179–80Invoking the C and C++ Compilers [2] This chapter describes the compiler driver commands that used to launch the Cray C and C++ compilers. The following commands invoke the compilers: • CC, which invokes the Cray C++ compiler. • cc, which invokes the Cray C compiler. • cpp, the C language preprocessor, is not part of the Cray Compilation Environment (CCE). The cpp command resolves to the GNU cpp command and does not predefine any Cray compiler-specific macros (Chapter 9, Using Predefined Macros on page 131). If the predefinition of the Cray compiler-specific macros is required, then use the cc or CC command to do the source preprocessing using the -E or -P option. A successful compilation creates an executable, named a.out by default, that reflects the contents of the source code and any referenced library functions. Use the aprun command to run the executable on the compute nodes. For example, the following command sequence compiles file mysource.c and launches the resulting executable program on 64 compute nodes: % cc mysource.c % aprun -n 64 ./a.out With the use of appropriate options, it is possible to direct the compiler to generate intermediate translations, including relocatable object files (-c option), assembly source expansions (-S option), or the output of the preprocessor phase of the compiler (-P or -E option). In general, it is possible to save the intermediate files and reference them later on a CC or cc command, with other files or libraries included as necessary. By default, the CC and cc commands automatically call the linker, which creates an executable file. If only one source file is specified, the object file (*.o) is deleted. If more than one source file is specified, the object files are retained. For example, the following command creates and retains object files file1.o, file2.o, and file3.o, and creates the executable file a.out: % cc file1.c file2.c file3.c The following command creates file.o and a.out; file.o is not retained. % cc file.c S–2179–80 19Cray C and C++ Reference Manual 2.1 CC Command The CC command invokes the Cray C++ compiler. The CC command accepts C++ source files with the following suffixes: .c .C .i .c++ .C++ .cc .cxx .Cxx .CXX .CC The .i files are created when the preprocessing compiler command option (-P) is used. The CC command also accepts object files with the .o suffix, library files with the .a suffix, and assembler source files with the .s suffix. The CC command format is as follows: CC [-c] [-C] [-D macro[=def]] [-E] [-g] [-G level] [-h arg] [-I incldir] [-K trap=opt[,opt]...] [-l libfile] [-L ldir] [-M] [-nostdinc] [-o outfile] [-O level] [-P] [ -rpath ldir ] [-S] [-U macro] [-V] [-Wphase,"opt ..."] [-X npes] [-Yphase,dirname] [-#] [-##] [-###] files ... For an explanation of the command line options, see Command Line Options on page 21. 2.2 cc Command The cc command invokes the Cray C compiler. The cc command accepts C source files that have the .c and .i suffixes; object files with the .o suffix; library files with the .a suffix; and assembler source files with the .s suffix. The cc command format is as follows: cc [-c] [-C] [-D macro[=def]] [-E] [-g] [-G level] [-h arg] [-I incldir] [-K trap=opt[,opt]...] [-l libfile] [-L ldir] [-M] [-nostdinc] [-o outfile] [-O level] [-P] [ -rpath ldir ] [-S] [-U macro] [-V] [-W phase,"opt..."] [-X npes] [-Y phase,dirname] [-#] [-##] [-###] files ... For an explanation of the command line options, see Command Line Options on page 21. 20 S–2179–80Invoking the C and C++ Compilers [2] 2.3 Command Line Options The following subsections describe options for the CC and cc commands. These options are grouped according to the following functions: • Standard conformance options (Standard Language Conformance Options on page 22) • Virtual function options (Virtual Function Options on page 27) • General language options (General Language Options on page 27) • General optimization options (General Optimization Options on page 30) • Automatic cache management options (-h cachen on page 37) • Vector optimization options (Vector Optimization Options on page 38) • Inlining options (Interprocedural Analysis (IPA) Optimization Options on page 39) • Scalar optimization options (Scalar Optimization Options on page 42) • Math options (Math Options on page 43) • Debugging options (Debugging Options on page 46) • Compiler message options (Compiler Message Options on page 48) • Compilation phase options (Compilation Phase Options on page 49) • Preprocessing options (Preprocessing Options on page 52) • Linker options (Linker Options on page 55) • Miscellaneous options (Miscellaneous Options on page 57) • Command line examples (Command Line Examples on page 61) Options other than those described in this manual are passed to the linker. There are many options that start with -h. Specify multiple -h options using commas to separate the arguments. For example, the -h parse_templates and -h fp0 command line options can be specified as -h parse_templates,fp0. If you specify conflicting options, the option specified last on the command line overrides the previously specified option. Exceptions to this rule are noted in the individual descriptions of the options. S–2179–80 21Cray C and C++ Reference Manual The following examples illustrate the use of conflicting options: • In this example, -h fp0 overrides -h fp1: % cc -h fp1,fp0 myfile.c • In this example, -h vector2 overrides the earlier vector optimization level 3 implied by the -O3 option: % CC -O3 -h vector2 myfile.C Most #pragma directives override corresponding command line options. Exceptions to this rule are noted in descriptions of options or #pragma directives. 2.4 Standard Language Conformance Options This section describes standard conformance language options. Each subsection heading shows in parentheses the compiler with which the option can be used. 2.4.1 -h [no]c99 (cc) Default: -h c99 This option enables or disables language features new to the C99 standard and Cray C compiler, while providing support for features that were previously defined as Cray extensions. If the previous implementation of the Cray extension differed from the C99 standard, both implementations will be available when the -h c99 option is enabled. The -h c99 option is also required for C99 features not previously supported as extensions. When -h noc99 is used, C99 language features such as variable-length arrays (VLAs) and restricted pointers that were available as extensions previously to adoption of the C99 standard remain available to you. 2.4.2 -h [no]conform (CC, cc), -h [no]stdc (cc) Default: -h noconform, -h nostdc The -h conform and -h stdc options specify strict conformance to the ISO C standard or the ISO C++ standard. The -h noconform and -h nostdc options specify partial conformance to the standard. The -h exceptions, -h dep_name, -h parse_templates, and -h const_string_literals options are enabled by the -h conform option in Cray C++. 22 S–2179–80Invoking the C and C++ Compilers [2] 2.4.3 -h cfront (CC) The -h cfront option causes the Cray C++ compiler to accept or reject constructs that were accepted by previous cfront-based compilers (such as Cray C++ 1.0) but which are not accepted in the C++ standard. The -h anachronisms option is implied when -h cfront is specified. 2.4.4 -h [no]parse_templates (CC) Default: -h noparse_templates This option allows existing code that defines templates using previous versions of the Cray Standard Template Library (STL) (before Programming Environment 3.6) to compile successfully with the -h conform option. Consequently, this allows you to compile existing code without having to use the Cray C++ STL. To do this, use the noparse_templates option. Also, the compiler defaults to this mode when the -h dep_name option is used. To have the compiler verify that your code uses the Cray C++ STL properly, use the parse_templates option. 2.4.5 -h [no]dep_name (CC) Default: -h nodep_name This option enables or disables dependent name processing (that is, the separate lookup of names in templates when the template is parsed and when it is instantiated). The -h dep_name option cannot be used with the -h noparse_templates option. 2.4.6 -h [no]exceptions (CC) Default: The default is -h exceptions; however, if the CRAYOLDCPPLIB environment variable is set to a nonzero value, the default is -h noexceptions. The -h exceptions option enables support for exception handling. The -h noexceptions option issues an error whenever an exception construct, a try block, a throw expression, or a throw specification on a function declaration is encountered. The -h exceptions option is enabled by -h conform. 2.4.7 -h [no]anachronisms (CC) Default: -h noanachronisms The -h [no]anachronisms option disables or enables anachronisms in Cray C++. This option is overridden by -h conform. S–2179–80 23Cray C and C++ Reference Manual 2.4.8 -h [no]new_for_init (CC) Default: -h new_for_init The -h new_for_init option enables the new scoping rules for a declaration in a for-init-statement. This means that the new standard-conforming rules are in effect; the entire for statement is wrapped in its own implicitly generated scope. The -h new_for_init option is implied by the -h conform option. This is the result of the scoping rule: { . . . for (int i = 0; i < n; i++) { . . . } // scope of i ends here for -h new_for_init . . . } // scope of i ends here for -h nonew_for_init 2.4.9 -h [no]tolerant (cc) Default: -h notolerant The -h tolerant option allows older, less standard C constructs, thereby making it easier to port code written for previous C compilers. Errors involving comparisons or assignments of pointers and integers become warnings. The compiler generates casts so that the types agree. With -h notolerant, the compiler is intolerant of the older constructs. The -h tolerant option causes the compiler to tolerate accessing an object with one type through a pointer to an entirely different type. For example, a pointer to a long might be used to access an object declared with type double. Such references violate the C standard and should be eliminated if possible. They can reduce the effectiveness of alias analysis and inhibit optimization. 2.4.10 -h [no]const_string_literals (CC) Default: -h noconst_string_literals The -h [no]const_string_literals option controls whether string literals are const (as required by the standard) or non-const (as was true in earlier versions of the C++ language). 24 S–2179–80Invoking the C and C++ Compilers [2] 2.4.11 -h [no]gnu Default: -h nognu The -h gnu option enables the compiler to recognize the subset of the GCC version 4.4.4 extensions to C listed in Table 1. Table 2 lists the extensions that apply only to C++. For detailed descriptions of the GCC C and C++ language extensions, see http://gcc.gnu.org/onlinedocs/. Table 1. GCC C Language Extensions GCC C Language Extension Description Typeof typeof: referring to the type of an expression Lvalues Using ?:, and casts in lvalues Conditionals Omitting the middle operand of a ?: expression Long Long Double-word integers – long long int Complex Data types for complex numbers Statement Exprs Putting statements and declarations inside expressions Zero Length Zero-length arrays Variable Length Arrays whose length is computed at run time Empty Structures Structures with no members; applies to C but not C++ Variadic Macros Macros with a variable number of arguments Escaped Newlines Slightly looser rules for escaped newlines Multiline strings String literals with embedded newlines Initializers Non-constant initializers Compound Literals Compound literals give structures, unions or arrays as values Designated Inits Labeling elements of initializers Cast to Union Casting to union type from any member of the union Case Ranges 'case 1 ... 9' and such Mixed Declarations Mixing declarations and code Attribute Syntax Formal syntax for attributes Function Prototypes Prototype declarations and old-style definitions; applies to C but not C++ C++ Comments C++ comments are recognized Dollar Signs Dollar sign is allowed in identifiers Character Escapes \e stands for the character Alignment Inquiring about the alignment of a type or variable S–2179–80 25Cray C and C++ Reference Manual GCC C Language Extension Description Inline Defining inline functions (as fast as macros) Alternate Keywords __const__, __asm__, and so on, for header files Incomplete Enums enum foo;, with details to follow Function Names Printable strings which are the name of the current function Return Address Getting the return or frame address of a function Unnamed Fields Unnamed struct/union fields within structs/unions Function Attributes: nothrow; format, format_arg; deprecated; used; unused; alias; weak Declaring that functions have no side effects, or that they can never return Variable Attributes: alias; deprecated; unused; used; transparent_union; weak; Specifying attributes of variables Type Attributes: deprecated; unused; used; transparent_union Specifying attributes of types Asm Labels Specifying the assembler name to use for a C symbol Other Builtins: __builtin_types_compatible_p, __builtin_choose_expr, __builtin_constant_p, __builtin_huge_val, __builtin_huge_valf, __builtin_huge_vall, __builtin_inf, __builtin_inff, __builtin_infl, __builtin_nan, __builtin_nanf, __builtin_nanl, __builtin_nans, __builtin_nansf, __builtin_nansl Other built-in functions Special files such as /dev/null may be used as source files. The supported subset of the GCC version 4.4.4 extensions to C++ are listed in Table 2. 26 S–2179–80Invoking the C and C++ Compilers [2] Table 2. GCC C++ Language Extensions GCC C++ Extensions Description Min and Max C++ minimum and maximum operators Restricted Pointers C99 restricted pointers and references Backwards Compatibility Compatibilities with earlier definitions of C++ Strong Using A using directive with __attribute ((strong)) Explicit template specializations Attributes may be used on explicit template specializations 2.5 Virtual Function Options This section describes general language options. 2.5.1 -h forcevtbl (CC) The -h forcevtbl option forces the definition of virtual function tables in cases where the heuristic methods used by the compiler to decide on definition of virtual function tables provide no guidance. The virtual function table for a class is defined in a compilation if the compilation contains a definition of the first non-inline, non-pure virtual function of the class. For classes that contain no such function, the default behavior is to define the virtual function table (but to define it as a local static entity). The -h forcevtbl option differs from the default behavior in that it does not force the definition to be local. 2.5.2 -h suppressvtbl (CC) The -h suppressvtbl option suppresses the definition of virtual function tables in cases where the heuristic methods used by the compiler to decide on definition of virtual function tables provide no guidance. 2.6 General Language Options This section describes general language options. Each subsection heading shows in parentheses the compiler with which the option can be used. S–2179–80 27Cray C and C++ Reference Manual 2.6.1 -h keep=file (CC) When the -h keep=file option is specified, the static constructor/destructor object (.o) file is retained as file. This option is useful when linking .o files on a system that does not have a C++ compiler. The use of this option requires that the main function must be compiled by C++ and the static constructor/destructor function must be included in the link. With these precautions, mixed object files (files with .o suffixes) from C and C++ compilations can be linked into executables by using the linker command instead of the CC command. 2.6.2 -h restrict=args The -h restrict=args option globally tells the compiler to treat certain classes of pointers as restricted pointers. Use this option to enhance optimizations (this includes vectorization). Classes of affected pointers are determined by the value contained in args, as follows: args Description a All pointers to object and incomplete types are considered restricted pointers, regardless of where they appear in the source code. This includes pointers in class, struct, and union declarations, type casts, function prototypes, and so on. ! Caution: Do not specify restrict=a if, during execution of any function, an object is modified and that object is referenced through either two different pointers or through the declared name of the object and a pointer. Undefined behavior may result. f All function parameters that are pointers to objects or incomplete types can be treated as restricted pointers. ! Caution: Do not specify restrict=f if, during execution of any function, an object is modified and that object is referenced through either two different pointer function parameters or through the declared name of the object and a pointer function parameter. Undefined behavior may result. t All parameters that are this pointers can be treated as restricted pointers (Cray C++ only). ! Caution: Do not specify restrict=t if, during execution of any function, an object is modified and that object is referenced through the declared name of the object and a this pointer. Undefined behavior may result. 28 S–2179–80Invoking the C and C++ Compilers [2] The args arguments tell the compiler to assume that, in the current compilation unit, each pointer (=a), each pointer that is a function parameter (=f), or each this pointer (=t) points to a unique object. This assumption eliminates those pointers as sources of potential aliasing, and may allow additional vectorization or other optimizations. These options cause only data dependencies from pointer aliasing to be ignored, rather than all data dependencies. ! Caution: The arguments make assertions about your program that, if incorrect, can introduce undefined behavior. Do not use -h restrict=a if, during the execution of any function, an object is modified and that object is referenced through either of the following: • Two different pointers • The declared name of the object and a pointer The -h restrict=f and -h restrict=t options are subject to the analogous restriction, with "function parameter pointer" replacing "pointer." 2.6.3 -h [no]calchars Default: -h nocalchars The -h calchars option allows the use of the $ character in identifier names. This option is useful for porting code containing identifiers that include this character. With -h nocalchars, this character is not allowed in identifier names. ! Caution: Use this option with extreme care, because identifiers with this character are within CNL name space and are included in many library identifiers, internal compiler labels, objects, and functions. Prevent conflicts between identifiers within CNL name space and your code; any such conflict is an error. 2.6.4 -h [no]signedshifts Default: -h nosignedshifts The -h [no]signedshifts option affects the result of the right shift operator. For the expression e1 >> e2, where e1 has a signed type, when -h signedshifts is in effect, the vacated bits are filled with the sign bit of e1. When -h nosignedshifts is in effect, the vacated bits are filled with zeros, identical to the behavior when e1 has an unsigned type. Also, see Integers on page 160 about the effects of this option when shifting integers. S–2179–80 29Cray C and C++ Reference Manual 2.7 General Optimization Options 2.7.1 -h [no]add_paren Default: -h noadd_paren The -h [no]add_paren option automatically adds parenthesis to select associative operations (+,-,*) to encourage left to right evaluation of floating point and complex expressions. The default is -hnoadd_paren. For more information, see the crayftn(1) man page. Left to right evaluation is not required by the language standards, but some applications may expect it. 2.7.2 -h [no]aggress Default: -h noaggress The -h aggress option provides greater opportunity to optimize loops that would otherwise by inhibited from optimization due to an internal compiler size limitation. -h noaggress leaves this size limitation in effect. With -h aggress, internal compiler tables are expanded to accommodate larger loop bodies. This option can increase the compilation's time and memory size. 2.7.3 -h [no]autoprefetch Default: -h autoprefetch The -h [no]autoprefetch option controls automatic prefetch optimization. Does not affect the loop_info [no]prefetch directive. 2.7.4 -h [no]autothread Default: -h noautothread The -h [no]autothread option enables or disables automatic threading. 2.7.5 -h display_opt The -h display_opt option displays the current optimization settings for this compilation. 30 S–2179–80Invoking the C and C++ Compilers [2] 2.7.6 -h [no]dwarf The -h [no]dwarf option controls whether DWARF debugging information is generated during compilation. Default: -h dwarf 2.7.7 -h flex_mp=level The -h flex_mp=level option controls the aggressiveness of optimizations which may affect floating point and complex repeatability when application requirements require identical results when varying the number of ranks or threads. -hflex_mp=intolerant has the highest probability of repeatable results, but also has the highest performance penalty. -hflex_mp=conservative uses more aggressive optimization and yields higher performance than -hflex_mp=intolerant, but results may not be sufficiently repeatable for some applications. -hflex_mp=tolerant uses most aggressive optimization and yields highest performance, but results may not be sufficiently repeatable for some applications. 2.7.8 -h fusionn Default: -h fusion2 The –h fusion n option controls loop fusion and changes the assertiveness of the fusion pragma. Loop fusion can improve the performance of loops, although in rare cases it may degrade performance. The n argument allows you to turn loop fusion on or off and determine where fusion should occur. Note: Loop fusion is disabled when the scalar level is set to 0. Default: -h fusion2 The values for n are: 0 No fusion (ignore all fusion pragmas and do not attempt to fuse other loops) 1 Attempt to fuse loops that are marked by the fusion pragma. 2 Attempt to fuse all loops (includes array syntax implied loops), except those marked with the nofusion pragma. S–2179–80 31Cray C and C++ Reference Manual 2.7.9 -h [no]intrinsics Default: -h intrinsics The -h intrinsics option allows the use of intrinsic hardware functions, which allow direct access to some hardware instructions or generate inline code for some functions. This option has no effect on specially handled library functions. Intrinsic functions are described in Appendix D, Using Intrinsic Functions on page 187. 2.7.10 -h list The -h list=opt option allows you to create listings and control their formats. The listings are written to source_file_name_without_suffix.lst. The values for opt are: a Use all list options; source_file_name_without_suffix.lstincludes a summary report, an options report, and the source listing. d Decompiles (translates) the intermediate representation of the compiler into listings that resemble the format of the source code. This is performed twice, resulting in two output files, at different points during the optimization process. You can use these files to examine the restructuring and optimization changes made by the compiler, which can lead to insights about changes you can make to your source code to improve its performance. The compiler produces two decompilation listing files with these extensions per specified source file: .opt and .cg. The compiler generates the .opt file after applying most high-level loop nest transformations to the code. The code structure of this listing most resembles your source code and is readable by most users. In some cases, because of optimizations, the structure of the loops and conditionals will be significantly different than the structure in your source file. 32 S–2179–80Invoking the C and C++ Compilers [2] The .cg file contains a much lower level of decompilation. It is quite close to what will be produced as assembly output. This version displays the intermediate text after all vector translation and other optimizations have been performed. An intimate knowledge of the hardware architecture of the system is helpful to understanding this listing. The .opt and .cg files are intended as a tool for performance analysis and are not valid source code. The format and contents of the files can be expected to change from release to release. e Expand include files. Note: Using this option may result in a very large listing file. All system include files are also expanded. i Intersperse optimization messages within the source listing rather than at the end. m Create loopmark listing; source_file_name_without_suffix.lst includes summary report and source listing. s Create a complete source listing (include files not expanded). Using -h list=m creates a loopmark listing. The e, i, s, and w options provide additional listing features. Using -h list=a combines all options. 2.7.11 -h [no]msgs Default: -h nomsgs The -h msgs option causes the compiler to write optimization messages to stderr. When the -h msgs option is in effect, you may request that a listing be produced so that you can see the optimization messages in the listing. For information about obtaining listings, see -h list on page 32. 2.7.12 -h [no]negmsgs Default: -h nonegmsgs The -h negmsgs option causes the compiler to generate messages to stderr that indicate why optimizations such as vectorization, inlining, or cloning did not occur in a given instance. The -h negmsgs option enables the -h msgs option. The -h list=a option enables the -h negmsgs option. S–2179–80 33Cray C and C++ Reference Manual 2.7.13 -h [no]omp_trace Default: -h noomp_trace (tracing is off) The -h [no]omp_trace option turns the insertion of the CrayPat OpenMP tracing calls on or off. 2.7.14 -h [no]func_trace The -h func_trace option is for use only with CrayPat. If this option is specified, the compiler inserts CrayPat trace entry points into each function in the compiled source file. The names of the trace entry points are: • __pat_tp_func_entry • __pat_tp_func_return These are resolved by CrayPat when the program is instrumented using the pat_build command. When the instrumented program is executed and it encounters either of these trace entry points, CrayPat captures the address of the current function and its return address. 2.7.15 -h [no]overindex Default: -h nooverindex The -h overindex option declares that there are array subscripts that index a dimension of an array that is outside the declared bounds of that array. The -h nooverindex option declares that there are no array subscripts that index a dimension of an array that is outside the declared bounds of that array. 2.7.16 -h [no]pattern Default: -h pattern The -h [no]pattern option globally enables or disables pattern matching. When the compiler recognizes certain patterns in the source code, it replaces the construct with a call to an optimized library routine. A loop or statement that has been pattern matched and replaced with a call to a library routine is indicated with an A in the loopmark listing. Note: Pattern matching is not always worthwhile. If there is a small amount of work in the pattern-matched construct, the call overhead may outweigh the time saved by using the optimized library routine. When compiling using the default optimization settings, the compiler attempts to determine whether each given candidate for pattern matching will in fact yield improved performance. 34 S–2179–80Invoking the C and C++ Compilers [2] 2.7.17 -h pl=program_library Create and use a persistent repository of compiler information specified by program_library. When used with -hwp, this option provides application-wide, cross-file, automatic inlining. See -h wp on page 36. The program_library repository is implemented as a directory and the information contained in program library is built up with each compiler invocation. Any compilation that does not have the -hpl option will not add information to this repository. Because of the persistence of program_library, it is the user's responsibility to manage it. For example, rm -r program_library might be added to the make clean target in an application makefile. Because program_library is a directory, use rm -r to remove it. If an application makefile works by creating files in multiple directories during a single build, the program_library should be an absolute path, otherwise multiple and incomplete program library repositories will be created. For example, avoid -hpl=./PL.1 and use -hpl=/fullpath/builddir/PL.1 instead. 2.7.18 -h profile_generate The -h profile_generate option directs that the source code be instrumented for gathering profile information. The compiler inserts calls and data-gathering instructions to allow CrayPat to gather information about the loops in a compilation unit. If you use this option, you must run CrayPat on the resulting executable so the CrayPat data-gathering routines are linked in. For information about CrayPat and profile information, see the Using Cray Performance Analysis Tools guide. 2.7.19 -h threadn Default: –h thread2 The -h threadn options control the optimization of both OpenMP and automatic threading. The values of n are: 0 No autothreading or OMP (OpenMP) threading. 1 No parallel region expansion, no loop restructuring for OMP loops, no optimization across OMP constructs. 2 Parallel region expansion, limited loop restructuring, optimization across OMP constructs. 3 Reduction results may not be repeatable. Loop restructuring, including modifying iteration space for static schedules (breaking standard compliance). S–2179–80 35Cray C and C++ Reference Manual 2.7.20 -h unrolln Default: –h unroll2 The -h unrolln option globally controls loop unrolling and changes the assertiveness of the unroll pragma. By default, the compiler attempts to unroll all loops, unless the nounroll pragma is specified for a loop. Generally, unrolling loops increases single processor performance at the cost of increased compile time and code size. The n argument allows you to turn loop unrolling on or off and specify where unrolling should occur. It also affects the assertiveness of the unroll pragma. The values for n are: 0 No unrolling (ignore all unroll pragmas and do not attempt to unroll other loops). 1 Attempt to unroll loops that are marked by the unroll pragma. 2 Unroll loops when performance is expected to improve. Loops marked with the unroll or nounroll pragma override automatic unrolling. Note: Loop unrolling is disabled when the scalar level is set to 0. 2.7.21 -h wp Enables the whole program mode. This option causes the compiler backend (IPA, optimizer, codegenerator) to be invoked at application link time, enabling whole program automatic inlining/cloning and future whole program interprocedural analysis (IPA) optimizations. Since the -hwp option provides automatic application-wide inlining, the -Oipafrom option is no longer needed for cross-file inlining. Requires that pl=program_library is also specified. See -h pl=program_library on page 35. The options -hpl=program_library and -hwp should be specified on all compiler invocations and on the compiler link invocation. Since -hwp delays the compiler optimization step until link time, -c compiles will take less time and the link step will take longer. Normally, this is just a time shift from one build phase to another with roughly the same overall compile time. In some cases increased inlining may cause an increase in overall compile time. Using -hwp allows the compiler backend to be invoked in parallel during a build. Setting the environment variable NPROC controls the number of concurrent compiler backend invocations and this parallelism may reduce overall compile time. 36 S–2179–80Invoking the C and C++ Compilers [2] 2.7.22 -O level Default: Equivalent to the appropriate -h option except that -O3 sets -h cache2, rather than -h cache3. The -O level option specifies the optimization level for a group of compiler features. Specifying -O with no argument is the same as not specifying the -O option; this syntax is supported for compatibility with other vendors. A value of 0, 1, 2, or 3 sets that level of optimization for each of the -h scalarn and -h vectorn options. The -O values of 0, 1, 2, or 3 set that level of optimization for -h cachen options, except that -O3 sets -h cache2, rather than -h cache3. The -O2 option is equivalent to ipa2, scalar2, vector2, cache2, and thread2. The -O3 option is equivalent to ipa4. The -O0 option also implies -fp0. Optimization features specified by -O are equivalent to the -h options listed in Table 3. Table 3. -h Option Descriptions -h Option Description Location -h cachen -h cachen on page 37 -h vectorn -h vectorn on page 38 -h scalarn -h scalarn on page 42 Table 4 shows the equivalent level of automatic cache optimization for the -h option. 2.8 Automatic Cache Management Options This section describes the automatic cache management options. Automatic cache management can be overridden by the use of the cache directives (cache, cache_nt, and loop_info). 2.8.1 -h cachen Default: -h cache2 The -h cachen option specifies the levels of automatic cache management to perform. The default is -h cache2. S–2179–80 37Cray C and C++ Reference Manual The values for n are: 0 Cache blocking (including directive-based blocking) is turned off. This level is compatible with all scalar and vector optimization levels. 1 Conservative automatic cache management. Characteristics include moderate compile time. Symbols are placed in the cache when the possibility of cache reuse exists and the predicted cache footprint of the symbol in isolation is small enough to experience the reuse. 2 Moderately aggressive automatic cache management. Characteristics include moderate compile time. Symbols are placed in the cache when the possibility of cache reuse exists and the predicted state of the cache model is such that the symbol will experience the reuse. 3 Aggressive automatic cache management. Characteristics include potentially high compile time. Symbols are placed in the cache when the possibility of cache reuse exists and the allocation of the symbol to the cache is predicted to increase the number of cache hits. Table 4. Cache Levels -O Option Cache Level -O0 -h cache0 -O1 -h cache1 -O2 -h cache2 -O3 -h cache2 2.9 Vector Optimization Options This section describes vector optimization options. Each subsection heading shows in parentheses the compiler command with which the option can be used. 2.9.1 -h vectorn Default: -h vector2 The -h vectorn option specifies the level of automatic vectorizing to be performed. Vectorization results in significant performance improvements with a small increase in object code size. Vectorization directives are unaffected by this option. 38 S–2179–80Invoking the C and C++ Compilers [2] The values of n are: n Description 0 No automatic vectorization. Characteristics include low compile time and small compile size. This option is compatible with all scalar optimization levels. 1 Specifies conservative vectorization. Characteristics include moderate compile time and size. No loop nests are restructured; only inner loops are vectorized. No vectorizations that might create false exceptions are performed. Results may differ slightly from results obtained when -h vector0 is specified because of vector reductions. The -h vector1 option is compatible with -h scalar1, -h scalar2, and -h scalar3. 2 Specifies moderate vectorization. Characteristics include moderate compile time and size. Loop nests are restructured. The -h vector2 option is compatible with -h scalar2 and -h scalar3. 3 Specifies aggressive vectorization. Characteristics include potentially high compile time and size. Loop nests are restructured. Vectorizations that might create false exceptions in rare cases may be performed. For further information, see Vectorization Directives on page 75. 2.10 Interprocedural Analysis (IPA) Optimization Options Inlining and cloning transform code in ways that increase the opportunity for interprocedural (IPA) optimizations. The user controls inlining and cloning through the use of command line options alone or command line options in combination with directives placed within the code. By default, the compiler will attempt inline optimizations where appropriate, but not cloning. Inlining and cloning may increase object code size. Also see Inlining and Cloning Directives on page 90. 2.10.1 Inlining Inlining is the process of replacing a user procedure call with the procedure definition itself. This can improve performance by saving the expense of the call overhead. It also increases the possibility of additional code optimization of the inlined code. If all calls within a loop are inlined, the loop becomes a candidate for parallelization. S–2179–80 39Cray C and C++ Reference Manual The compiler supports the following inlining modes through the indicated options: • Automatic inlining allows the compiler to automatically select which functions to inline. This occurs with the -h ipa2 or greater option. When -h ipan is used alone, the candidates for expansion are all those functions that are present in the input file to the compile step. See -h ipan on page 40. • Explicit inlining allows you to explicitly indicate which procedures the compiler should attempt to inline and occurs with the -h ipafrom=source [:source] option alone as described in -h ipafrom=source[:source] ... on page 41. • Combined inlining allows you to specify potential targets for inline expansion, while applying the selected level of inlining heuristics. If -h ipan is used in conjunction with -h ipafrom=source[:source], the candidates for expansion are those functions present in source and -h ipan selects level of heuristics. 2.10.2 Cloning Cloning replaces a call to a procedure with a call to a modified version of that same procedure (clone) in which the dummy arguments in the original procedure are replaced with the constant actual arguments present at the call site. Automatic cloning is enabled at -Oipa4 and higher. The compiler first attempts to inline a call site. If inlining the call site fails, the compiler attempts to clone the procedure for the specific call site. 2.10.3 -h ipan Default: -h ipa3 The -h ipan option controls the level of automatic inlining and cloning.Table 5 explains what type of IPA optimization is performed at each level. Table 5. IPA Level IPA Level Description 0 All inlining and cloning are disabled. All inlining and cloning compiler directives are ignored. 1 Directive inlining. Inlining is attempted for call sites and routines that are under the control of an inlining pragma directive. Cloning is disabled and cloning directives are ignored. See Inlining and Cloning Directives on page 90 for more information about inlining directives. 40 S–2179–80Invoking the C and C++ Compilers [2] IPA Level Description 2 Call nest inlining. Inline a call nest to an arbitrary depth as long as the nest does not exceed some compiler-determined threshold. A call nest can be a leaf routine. The expansion of the call nest must yield straight-line code (code containing no external calls) for any expansion to occur. The call site is said to "flatten" when there are no calls present in the expanded code. The call site must reside within the body of a loop for expansion to be attempted. Cloning is disabled and cloning directives are ignored. 3 Default level for inlining. Constant actual argument inlining and tiny routine inlining. This includes levels 1, and 2, plus any call site that contains a constant actual argument. Additionally, any call nest (regardless of location) that is below some small compiler-determined threshold will be inlined provided that call nest completely flattens. Cloning is disabled and cloning directives are ignored. 4 This includes levels 1, 2, and 3, plus routine cloning is attempted if inlining fails at a given call site. Cloning and cloning directives are enabled. 5 Aggressive interprocedural analysis (IPA). Includes levels 1, 2, 3, and 4. 2.10.4 -h ipafrom=source[:source] ... The -h ipafrom=source [:source] option allows you to explicitly indicate the procedures to consider for inline expansion or cloning. The source arguments identify each file or directory that contains the routines to consider for inlining or cloning. Note: Spaces are not allowed on either side of the equal sign. All inlining directives are recognized with explicit inlining, however cloning and cloning directives are only enabled with -O ipa4 or greater. For information about inlining directives, see Inlining and Cloning Directives on page 90. Note: The routines in source are not actually linked with the final program. They are simply templates for the inliner. To have a routine contained in source linked with the program, you must include it in an input file to the compilation. Use one or more of the following file types in the source argument. S–2179–80 41Cray C and C++ Reference Manual Table 6. File Types C or C++ source files The routines in C or C++ source files are candidates for expansion and must contain error-free code. Source files that are acceptable are files that have one of the following extensions: .C, .c++, .C++, .cc, .cxx, .Cxx, .CXX, or .CC. dir A directory that contains any of the file types described in this table. 2.11 Scalar Optimization Options This section describes scalar optimization options. Each subsection heading shows in parentheses the compiler command with which the option can be used. 2.11.1 -h [no]interchange Default: -h interchange The -h interchange option allows the compiler to attempt to interchange all loops, a technique that is used to gain performance by having the compiler swap an inner loop with an outer loop. The compiler attempts the interchange only if the interchange will increase performance. Loop interchange is performed only at scalar optimization level 2 or higher. The -h nointerchange option prevents the compiler from attempting to interchange any loops. To disable interchange of loops individually, use the #pragma _CRI nointerchange directive. 2.11.2 -h scalarn Default: -h scalar2 The -h scalarn option specifies the level of automatic scalar optimization to be performed. Scalar optimization directives are unaffected by this option (see Scalar Directives on page 84). 42 S–2179–80Invoking the C and C++ Compilers [2] The values for n are: 0 Minimal automatic scalar optimization. The -h matherror=errno and the -h zeroinc options are implied by -h scalar0. 1 Conservative automatic scalar optimization. This level implies -h matherror=abort and -h nozeroinc. 2 Aggressive automatic scalar optimization. The scalar optimizations that provide the best application performance are used, with some limitations imposed to allow for faster compilation times. 3 Very aggressive optimization; compilation times may increase significantly. 2.11.3 -h [no]zeroinc Default: -h nozeroinc The -h nozeroinc option improves run time performance by causing the compiler to assume that constant increment variables (CIVs) in loops are not incremented by expressions with a value of 0. The -h zeroinc option causes the compiler to assume that some constant increment variables (CIVs) in loops might be incremented by 0 for each pass through the loop, preventing generation of optimized code. For example, in a loop with index i, the expression expr in the statement i +=expr can evaluate to 0. This rarely happens in actual code. -h zeroinc is the safer and slower option. This option is affected by the -h scalarn option (see -h scalarn on page 42). 2.12 Math Options This section describes compiler options pertaining to math functions. Each subsection heading shows in parentheses the compiler command with which the option can be used. 2.12.1 -h fpn Default: -h fp2 The -h fp option allows you to control the level of floating-point optimizations. The n argument controls the level of allowable optimization; 0 gives the compiler minimum freedom to optimize floating-point operations, while 3 gives it maximum freedom. The higher the level, the lesser the floating-point operations conform to the IEEE standard. S–2179–80 43Cray C and C++ Reference Manual This option is useful for code using algorithms that are unstable but optimizable. Generally, this is the behavior and usage for each -h fp level: • The -h fp0 option causes your program's executable code to conform more closely to the IEEE floating-point standard than the default mode (-h fp2). When you specify this level, many identity optimizations are disabled, vectorization of floating-point reductions are disabled, executable code is slower than higher floating-point optimization levels, and a scaled complex divide mechanism is enabled that increases the range of complex values that can be handled without producing an underflow. Note: Use the -h fp0 option only when your code pushes the limits of IEEE accuracy or requires strong IEEE standard conformance. • The -h fp1 option performs various, generally safe, non-conforming IEEE optimizations, such as folding a == a to true, where a is a floating point object. At this level, floating-point reassociation 1 is greatly limited, which may affect the performance of your code. You should never use the -h fp1 option except when your code pushes the limits of IEEE accuracy or requires strong IEEE standard conformance. • -h fp2 — includes optimizations of -h fp1. • -h fp3 — includes optimizations of -h fp2. You should use the -h fp3 option when performance is more critical than the level of IEEE standard conformance provided by -h fp2. Table 7 compares the various optimization levels of the -h fp option (levels 2 and 3 are usually the same). The table lists some of the optimizations performed; the compiler may perform other optimizations not listed. 1 For example, a+b+c is rearranged to b+a+c, where a, b, and c are floating point variables. 44 S–2179–80Invoking the C and C++ Compilers [2] Table 7. Floating-point Optimization Levels Optimization Type fp0 fp1 fp2 (default) fp3 Safety Maximum Moderate Moderate Low Complex divisions Accurate and slower Accurate and slower Fast 2 Fast 2 Exponentiation rewrite None None Maximum performance 3 Maximum performance 3, 4 Strength reduction Fast Fast Aggressive Aggressive Rewrite division as reciprocal equivalent 5 None None Yes Aggressive Floating point reductions Slow Fast Fast Fast Expression factoring None Yes Yes Yes Expression tree balancing None None Yes Yes Inline 32-bit operations 6 No No No Yes Fused multiply-add 7 No Yes Yes Yes 2 Algebraically correct but may lack precision in boundary cases. 3 Rewriting values raised to a constant power into an algebraically equivalent series of multiplications and/or square roots. 4 Rewriting exponentiations (a b ) not previously optimized into the algebraically equivalent form exp(b * ln(a)). 5 For example, x/y is transformed to x * 1.0/y. 6 32-bit division, square root, and reciprocal square root use very fast but less precise code sequences. 7 Uses fused multiply-add instructions on architectures that support it. S–2179–80 45Cray C and C++ Reference Manual 2.12.2 -h matherror Default: -h matherror=abort The -h matherror=method option specifies the method of error processing used if a standard math function encounters an error. The method argument can have one of the following values: method Description abort If an error is detected, errno is not set. Instead, a message is issued and the program aborts. An exception may be raised. errno If an error is detected, errno is set, and the math function returns to the caller. This method is implied by the -h conform, -h scalar0, -O0, -Gn, and -g options. Certain optimized library functions do not set errno, therefore full support for this option is limited to codes which do not use the following functions: cos, exp, log, log2, log10, pow, sin, cosf, expf, logf, log2f, log10f, powf, or sinf. 2.13 Debugging Options This section describes compiler options used for debugging. Each subsection heading shows in parentheses the compiler command with which the option can be used. 2.13.1 -G level and -g The -G level and -g options enable the generation of debugging information used by symbolic debuggers. These options allow debugging with breakpoints. Table 8 describes the values for the -G option. Table 8. -G level Definitions level Optimization Breakpoints Allowed On Debugging Execution Speed -Gf Full Function entry and exit Limited Best -Gp Partial Block boundaries Better Better -Gn None Every executable statement Best Limited -Gfast Full Every executable statement Best: requires fast-track debugger Best 46 S–2179–80Invoking the C and C++ Compilers [2] Better debugging information comes at the cost of inhibiting certain optimization techniques, so choose the option that best fits the debugging needs of any particular source file in an application. The -g option is equivalent to -Gn. The -g option is included for compatibility with earlier versions of the compiler and many other UNIX systems; the -G option is the preferred specification. The -Gn and -g options recognize OpenMP directives and disable all optimizations. They imply -O0 -homp -hfp0. The debugging options take precedence over any conflicting options that appear on the command line. If more than one debugging option appears, the last one specified overrides the others. Debugging is described in more detail in Chapter 11, Debugging Cray C and C++ Code on page 137. 2.13.2 -h [no]bounds (cc) Default: -h nobounds The -h bounds option provides checking of pointer and array references to ensure that they are within acceptable boundaries. The -h nobounds option disables these checks. The pointer check verifies that the pointer is greater than 0 and less than the machine memory limit. The array check verifies that the subscript is greater than or equal to 0 and is less than the array size, if declared. 2.13.3 -h dir_check The -h dir_check option enables directive checking at run time. Errors detected at compile time are reported during compilation and so are not reported at run time. The following directives are checked: shortloop, shortloop128, collapse, and the loop_info clauses min_trips and max_trips. Violation of a run time check results in an immediate fatal error diagnostic. Warning: Optimization of enclosing and adjacent loops is degraded when run time directive checking is enabled. This capability, though useful for debugging, is not recommended for production runs. 2.13.4 -h zero The -h zero option causes stack-allocated memory to be initialized to all zeros. S–2179–80 47Cray C and C++ Reference Manual 2.14 Compiler Message Options This section describes compiler options that affect messages. Each subsection heading shows in parentheses the compiler command with which the option can be used. 2.14.1 -h msglevel_n Default: -h msglevel_3 The -h msglevel_n option specifies the lowest level of severity of messages to be issued. Messages at the specified level and above are issued. Values for n are: 0 Comment 1 Note 2 Caution 3 Warning 4 Error 2.14.2 -h [no]message=n[:n...] Default: Determined by -h msglevel_n The -h [no]message=n[:n...] option enables or disables specified compiler messages, where n is the number of a message to be enabled or disabled. You can specify more than one message number; multiple numbers must be separated by a colon with no intervening spaces. For example, to disable messages CC-174 and CC-9, specify: -h nomessage=174:9 The -h [no]message=n option overrides -h msglevel_n for the specified messages. If n is not a valid message number, it is ignored. Any compiler message except ERROR, INTERNAL, and LIMIT messages can be disabled; attempts to disable these messages by using the -h nomessage=n option are ignored. 48 S–2179–80Invoking the C and C++ Compilers [2] 2.14.3 -h report=args The -h report=args option generates report messages specified in args and lets you direct the specified messages to a file. The args field can be any combination of the following options: f Writes specified messages to file.V, where file is the source file specified on the command line. If the f option is not specified, messages are written to stderr. i Generates inlining optimization messages. s Generates scalar optimization messages. v Generates vector optimization messages. No spaces are allowed around the equal sign (=) or any of the args codes. For example, the following example prints inlining and scalar optimization messages for myfile.c: % cc -h report=is myfile.c The -h msgs option also provides optimization messages. 2.14.4 -h [no]abort Default: -h noabort The -h [no]abort option controls whether a compilation aborts if an error is detected. 2.14.5 -h errorlimit Default: -h errorlimit=100 The -h errorlimit[=n] option specifies the maximum number of error messages the compiler prints before it exits, where n is a positive integer. Specifying -h errorlimit=0 disables exiting on the basis of the number of errors. Specifying -h errorlimit with no qualifier is the same as setting n to 1. 2.15 Compilation Phase Options This section describes compiler options that affect compilation phases. Each subsection heading shows in parentheses the compiler command with which the option can be used. S–2179–80 49Cray C and C++ Reference Manual 2.15.1 -E The -E option directs the compiler to execute only the preprocessor phase of the compiler. The -E and -P options are equivalent, except that -E directs output to stdout and inserts appropriate #line linenumber preprocessing directives. The -E option takes precedence over the -h feonly, -S, and -c options. When both the -E and -P options are specified, the last one specified takes precedence. 2.15.2 -P The -P option directs the compiler to execute only the preprocessor phase of the compiler for each source file specified. The preprocessed output for each source file is written to a file with a name that corresponds to the name of the source file and has a .i suffix substituted for the suffix of the source file. The -P option is similar to the -E option, except that #line linenumber directives are suppressed, and the preprocessed source does not go to stdout. This option takes precedence over -h feonly, -S, and -c. When both the -P and -E options are specified, the last one specified takes precedence. 2.15.3 -h feonly The -h feonly option limits the compiler to syntax checking. The optimizer and code generator are not executed. This option takes precedence over -S and -c. 2.15.4 -S The -S option compiles the named source files and leaves their assembly language output in the corresponding files suffixed with a .s. If this option is used with -G or -g, debugging information is not generated. This option takes precedence over -c. 2.15.5 -c The -c option creates a relocatable object file for each named source file but does not link the object files. The relocatable object file name corresponds to the name of the source file. The .o suffix is substituted for the suffix of the source file. 2.15.6 -#, -##, and -### The -# option produces output indicating each phase of the compilation as it is executed. Each succeeding output line overwrites the previous line. The -## option produces output indicating each phase of the compilation as it is executed. 50 S–2179–80Invoking the C and C++ Compilers [2] The -### option is the same as -##, except the compilation phases are not executed. 2.15.7 -W phase,"opt ..." The -W phase option passes arguments directly to a phase of the compiling system. Table 9 shows the system phases that phase can indicate. Table 9. -W phase Definitions phase System Phase Command 0 (zero) Compiler CC and cc a Assembler as l Linker ld Arguments to be passed to system phases can be entered in either of two styles. If spaces appear within a string to be passed, the string is enclosed in double quotes. When double quotes are not used, spaces cannot appear in the string. Commas can appear wherever spaces normally appear; an option and its argument can be either separated by a comma or not separated. If a comma is part of an argument, it must be preceded by the \ character. For example, any of the following command lines would send -e name and -s to the linker: % cc -Wl,"-e name -s" file.c % cc -Wl,-e,name,-s file.c % cc -Wl,"-ename",-s file.c Because the preprocessor is built into the compiler, -Wp and -W0 are equivalent. 2.15.8 -Y phase,dirname The -Y phase,dirname option specifies a new directory (dirname) from which the designated phase should be executed. The values of phase are Table 10. Table 10. -Y phase Definitions phase System Phase Command 0 (zero) Compiler CC,cc a Assembler as l Linker ld Because there is no separate preprocessor, -Yp and -Y0 are equivalent. S–2179–80 51Cray C and C++ Reference Manual 2.16 Preprocessing Options This section describes compiler options that affect preprocessing. Each subsection heading shows in parentheses the compiler command with which the option can be used. 2.16.1 -C The -C option retains all comments in the preprocessed source code, except those on preprocessor directive lines. By default, the preprocessor phase strips comments from the source code. This option is useful in combination with the -P or -E option. 2.16.2 -D macro[=def] The -D macro[=def] option defines macro as if it were defined by a #define directive. If no =def argument is specified, macro is defined as 1. Predefined macros also exist; these are described in Chapter 9, Using Predefined Macros on page 131. Any predefined macro except those required by the standard (see Macros Required by the C and C++ Standards on page 132) can be redefined by the -D option. The -U option overrides the -D option when the same macro name is specified, regardless of the order of options on the command line. 2.16.3 -h [no]pragma=name[:name ...] Default: -h pragma (no pragmas diabled) The [no]pragma=name[:name...] option enables or disables the processing of specified directives in the source code, where name can be the name of a directive or a word shown in Table 11 to specify a group of directives. More than one name can be specified. Multiple names must be separated by a colon and have no intervening spaces. 52 S–2179–80Invoking the C and C++ Compilers [2] Table 11. -h pragma Directive Processing name Group Directives Affected all All All directives allinline Inlining inline_enable, inline_disable, inline_reset, inline_always, inline_never allscalar Scalar optimization blockable, blockingsize, noblocking, concurrent, nointerchange, noreduction, suppress, unroll/nounroll allvector Vectorization novector, loop_info, hand_tuned, nopattern, novector, permutation, pipeline/nopipeline, prefervector, safe_address, safe_conditional, shortloop, shortloop128 omp OpenMP All OpenMP directives acc OpenACC All OpenACC directives When using this option to enable or disable individual directives, note that some directives must occur in pairs. For these directives, you must disable both directives if you want to disable either; otherwise, the disabling of one of the directives may cause errors when the other directive is (or is not) present in the compilation unit. 2.16.4 -I incldir The -I incldir option specifies a directory for files named in #include directives when the #include file names do not have a specified path. Each directory specified must be specified by a separate -I option. The order in which directories are searched for files named on #include directives is determined by enclosing the file name in either quotation marks ("") or angle brackets (< and >). S–2179–80 53Cray C and C++ Reference Manual Directories for #include "file" are searched in the following order: 1. Directory of the input file. 2. Directories named in -I options, in command-line order. 3. Site-specific and compiler release-specific include files directories. 4. Directory /usr/include. Directories for #include are searched in the following order: 1. Directories named in -I options, in command-line order. 2. Site-specific and compiler release-specific include files directories. 3. Directory /usr/include. If the -I option specifies a directory name that does not begin with a slash (/), the directory is interpreted as relative to the current working directory and not relative to the directory of the input file (if different from the current working directory). For example: % cc -I. -I yourdir mydir/b.c The preceding command line produces the following search order: 1. mydir (#include "file" only). 2. Current working directory, specified by -I. 3. yourdir (relative to the current working directory), specified by -I yourdir. 4. Site-specific and compiler release-specific include files directories. 5. Directory /usr/include. 2.16.5 -M The -M option provides information about recompilation dependencies that the source file invokes on #include files and other source files. This information is printed in the form expected by make. Such dependencies are introduced by the #include directive. The output is directed to stdout. 2.16.6 -nostdinc The -nostdinc option stops the preprocessor from searching for include files in the standard directories (/usr/include and for Cray C++ also /usr/include/c++). 54 S–2179–80Invoking the C and C++ Compilers [2] 2.16.7 -U The -U option removes any initial definition of macro. Any predefined macro except those required by the standard (see Macros Required by the C and C++ Standards on page 132) can be undefined by the -U option. The -U option overrides the -D option when the same macro name is specified, regardless of the order of options on the command line. Predefined macros are described in Chapter 9, Using Predefined Macros on page 131. Macros defined in the system headers are not predefined macros and are not affected by the -U option. 2.17 Linker Options This section describes compiler options that affect linker tasks. 2.17.1 -h [system|default]_alloc Default: -h default_alloc By default, the compiler uses a modified malloc implementation that offers better support for Cray XE memory needs. The -hsystem_alloc option directs the compiler to link in the native malloc provided by the OS instead of the modified implementation. 2.17.2 -l libname The -l libname option directs the compiler driver to search for the specified object library file when linking an executable file. To request more than one library file, specify multiple -l options. When statically linking, the compiler driver searches for libraries by prepending ldir/lib on the front of libname and appending .a on the end of it, for each ldir that has been specified by using the -L option. It uses the first file it finds. See also the -L option (-L ldir on page 56). When dynamically linking, the library search process is similar to the static case, with a few differences. The compiler driver searches for libraries by prepending ldir/lib on the front of libname and appending .so on the end of it, for each ldir that has been specified by using the -L option. If a matching .so is not found, the compiler driver replaces .so with .a and repeats the process from the beginning. It uses the first file it finds. See also the -L option (-L ldir on page 56). There is no search order dependency for libraries. S–2179–80 55Cray C and C++ Reference Manual If you specify personal libraries by using the -l command line option, as in the following example, those libraries are added before the default CCE library list. (The -l option is passed to the linker.) cc -l mylib target.c When the previous command line is issued, the linker looks for a library named libmylib.a (following the naming convention) and adds it to the top of the list of default libraries. 2.17.3 -L ldir The -L ldir option changes the -l option search algorithm to look for library files in directory ldir during link time. To request more than one library directory, specify multiple -L options. The linker searches for library files in the compiler release-specific directories. Note: Multiple -L options are treated cumulatively as if all ldir arguments appeared on one -L option preceding all -l options. Therefore, do not attempt to link functions of the same name from different libraries through the use of alternating -L and -l options. 2.17.4 -rpath ldir The -rpath ldir option changes the runtime library search algorithm to look for files in directory ldir. To request more than one library directory, specify multiple -rpath options. Note that a library may be found at link time with a -L option, but may not be found at run time if a corresponding -rpath option was not supplied. Also note that the compiler driver does not pass the -rpath option to the linker. You must explicitly specify -Wl when using this option. See also the -W option (-W phase,"opt ..." on page 51). At link time, all ldir arguments are added to the executable. The dynamic linker will search these paths first for shared dynamic libraries at runtime, with one exception. The Linux environment variable LD_LIBRARY_PATH precedes all other search paths for shared dynamically linked libraries. The use of LD_LIBRARY_PATH is discouraged. ! Caution: Caution should be used when setting LD_LIBRARY_PATH. Doing so will change the shared dynamically linked library search paths for all executable files in your environment. 2.17.5 -o outfile The -o outfile option produces an absolute binary file named outfile. A file named a.out is produced by default. When this option is used in conjunction with the -c option and a single source file, a relocatable object file named outfile is produced. 56 S–2179–80Invoking the C and C++ Compilers [2] 2.18 Miscellaneous Options This section describes compiler options that affect general tasks. Each subsection heading shows in parentheses the compiler command with which the option can be used. 2.18.1 -h [no]acc Default: -h acc The -h [no]acc option enables or disables compiler recognition of OpenACC pragmas. 2.18.2 -h cpu=target_system The -h cpu=target_system option specifies the Cray system on which the absolute binary file is to be executed, where target_system can be either x86-64, opteron, barcelona, shanghai, istanbul, mc8, mc12, or interlagos. The default is x86-64. The x86-64 and opteron options produce identical output, for use on singleand dual-core systems. If you are creating executables for use on a system with quad-core processors (either AMD Opteron barcelona or shanghai processors), you must also have the associated module (either xtpe-barcelona or xtpe-shanghai) loaded when compiling and linking your code. Likewise, if you are creating executables for use on a system with AMD Opteron six-core processors (istanbul), eight-core processors (mc8), twelve-core processors (mc12), or 16-core processors (interlagos), you must have the xtpe-istanbul, xtpe-mc8, xtpe-mc12, or xtpe-interlagos module loaded when compiling and linking your code. If one of these modules is loaded, the default target_system changes to the corresponding cpu target. If the target_system is set during compilation of any source file, it must also be set to that same target during linking and loading. You can use the CRAY_PE_TARGET environment variable to set the target system; see Compile Time Environment Variables on page 62. For more information, see the Cray Application Developer's Environment User's Guide. 2.18.3 -h [no]fp_trap Controls whether the compiler generates code that is compatible with floating-point traps. Default: fp_trap, if traps are enabled using the -K trap option, or if -Ofp[0,1] is in effect. Otherwise, the default is nofp_trap. See -K trap=opt[,opt] ... on page 60. S–2179–80 57Cray C and C++ Reference Manual 2.18.4 -h ident=name Default: File name specified on the command line The -h ident=name option changes the ident name to name. This name is used as the module name in the object file (.o suffix) and assembler file (.s suffix). Regardless of whether the name is specified or the default name is used, the following transformations are performed on name: • All . characters in the ident name are changed to $. • If the ident name starts with a number, a $ is added to the beginning of the ident name. 2.18.5 -h keepfiles The -h keepfiles option prevents the removal of the object (.o) and temporary assembly (.s) files after an executable is created. Normally, the compiler automatically removes these files after linking them to create an executable. Since the original object files are required to instrument a program for performance analysis, if you plan to use CrayPat to conduct performance analysis experiments, you can use this option to preserve the object files. 2.18.6 -h loop_trips=[tiny | small | medium | large | huge] Specifies runtime loop trip counts for all loops in a compiled source file. This information is used to optimize the runtime characteristics of the application. 2.18.7 -h mpin Enables or disables optimization around a selected subset of MPI library calls. -h mpi0 disables this option. The default is -h mpi0. 2.18.8 -h network=nic The -h network=nic option specifies the target machine's system interconnection network. Currently, supported value for nic is gemini. Default: gemini 2.18.9 -h [no]omp Default: -h omp The -h [no]omp option enables or disables compiler recognition of OpenMP pragmas. For details, see Chapter 4, Using OpenMP on page 95. 58 S–2179–80Invoking the C and C++ Compilers [2] 2.18.10 -h [no]omp_acc Default: -h omp_acc The -h [no]omp_acc option enables or disables compiler recognition of OpenMP acc pragmas. For details, see OpenMP Accelerator Directives on page 100. 2.18.11 -h pic, -h PIC Generate position independent code (PIC), which allows a virtual address change from one process to another, as is necessary in the case of shared, dynamically linked objects. The virtual addresses of the instructions and data in PIC code are not known until dynamic link time. For the Cray XE and XK implementation, the pic and PIC options have the same effect and should be used to compile codes using more than 2GB of static memory, or for creating dynamically linked libraries. 2.18.12 -h prototype_intrinsics The -h prototype_intrinsics option simulates the effect of including intrinsics.h at the beginning of a compilation. Use this option if the source code does not include the intrinsics.h statement and you cannot modify the code. This option is off by default. For details, see Appendix D, Using Intrinsic Functions on page 187. 2.18.13 -h [no]threadsafe Default: -h threadsafe The -h [no]threadsafe option enables or disables the generation of threadsafe code. Code that is threadsafe can be used with pthreads and OpenMP. This option is not binary-compatible with code generated by Cray C 8.1 or Cray C++ 5.1 and earlier compilers. Users who need binary compatibility with previously compiled code can use -h nothreadsafe, which causes the compiler to be compatible with Cray C 8.1 or Cray C++ 5.1 and earlier compilers at the expense of not being threadsafe. C or C++ code compiled with -h threadsafe (the default) cannot be linked with C or C++ code compiled with -h nothreadsafe or with code compiled with a Cray C 8.1, Cray C++ 5.1, or earlier compiler. 2.18.14 -h upc (cc) The -h upc option enables compilation of Unified Parallel C (UPC) code. UPC is a C language extension for parallel program development that allows you to explicitly specify parallel programming through language syntax rather than through library functions such as are used in MPI or SHMEM. S–2179–80 59Cray C and C++ Reference Manual The Cray implementation of UPC is discussed in Chapter 6, Using Cray Unified Parallel C (UPC) on page 119. 2.18.15 -K trap=opt[,opt] ... Enable traps for the specified exceptions. By default, no exceptions are trapped. Enabling traps by using this option also has the effect of setting -h fp_trap. If the specified options contradict each other, the last option predominates. For example, -K trap=none,fp is equivalent to -Ktrap=fp. This option is processed only at link time and affects the entire program; it is not processed when compiling subprograms. Therefore, use this command line option to set traps only at the beginning of execution of the main program. The program may subsequently change these settings by calling intrinsic or library procedures. If you use this option, you may have to specify -hfp_trap when you compile other files of the application. See -h [no]fp_trap on page 57. opt Exceptions denorm Trap on denormalized operands. divz Trap on divide-by-zero. fp Trap on divz, inv, or ovf exceptions. inexact Trap on inexact result (i.e., rounded result). Enabling traps for inexact results is not recommended. inv Trap on invalid operation. none Disables all traps (default). ovf Trap on overflow (i.e., the result of an operation is too large to be represented). unf Trap on underflow (i.e., the result of an operation is too small to be represented). 2.18.16 -V The -V option displays compiler version information. If the command line specifies no source file, no compilation occurs. Version information consists of the product name, the version number, and the current date and time, as shown in the following example: % CC -V /opt/cray/xt-asyncpe/2.5/bin/CC: INFO: native target is being used Cray C++ : Version 7.1.0.129 Thu May 21, 2009 12:59:44 60 S–2179–80Invoking the C and C++ Compilers [2] 2.18.17 -X npes The -X npes option specifies the number of processing elements (PEs) that will be specified through aprun at job launch. On Cray XE and Cray XK systems, the value for npes ranges from 1 through 2**31 - 1 inclusive. Ensure that you compile all object files with the same -X npes value and run the resulting executable with that number of PEs. If you use mixed -X npes values or if the number of PEs provided at run time differs from the -X npes value, you will receive a run time error. You cannot change the number of PEs to use at link or run time. You must recompile the program with a different value for npes to change the number of PEs. For further information about running applications, see the Cray Application Developer's Environment User's Guide or the aprun(1) man page. 2.19 Command Line Examples The following examples illustrate a variety of command lines for the C and C++ compiler commands: Example 1. CC -X8 -h myprog.C This example compiles myprog.C and fixes the number of processing elements to 8. % CC -X8 -h myprog.C Example 2. CC -h conform myprog.C This example compiles myprog.C. The -h conform option specifies strict conformance to the ISO C++ standard. % CC -h conform myprog.C Example 3. cc -c -h ipa1 myprog.c subprog.c This example compiles input files myprog.c and subprog.c. The -c option tells the compiler to create object files myprog.o and subprog.o but not call the linker. Option -h ipa1 tells the compiler to inline function calls marked with the inline_always pragma. % cc -c -h ipa1 myprog.c subprog.c Example 4. cc -I. disc.c vend.c This example specifies that the compiler search the current working directory, represented by a period (.), for #include files before searching the default #include file locations. % cc -I. disc.c vend.c S–2179–80 61Cray C and C++ Reference Manual Example 5. cc -P -D DEBUG newprog.c This example specifies that source file newprog.c be preprocessed only. Compilation and linking are suppressed. In addition, the macro DEBUG is defined. % cc -P -D DEBUG newprog.c Example 6. cc -c -h report=s mydata1.c This example compiles mydata1.c, creates object file mydata1.o, and produces a scalar optimization report to stdout. % cc -c -h report=s mydata1.c Example 7. CC -h ipa5,report=if myfile.C This example compiles myfile.C and tells the compiler to attempt to aggressively inline calls to functions defined within myfile.C. An inlining report is directed to myfile.V. % CC -h ipa5,report=if myfile.C 2.20 Compile Time Environment Variables The following environment variables are used during compilation. Variable Description CRAYOLDCPPLIB When set to a nonzero value, enables C++ code to use the following nonstandard Cray C++ headers files: • common.h • complex.h • fstream.h • generic.h • iomanip.h • iostream.h • stdiostream.h • stream.h • strstream.h • vector.h 62 S–2179–80Invoking the C and C++ Compilers [2] If you want to use the standard header files, your code may require modification to compile successfully. For more information, see Appendix B, Using Cray C and C++ Dialects on page 167. Note: Setting the CRAYOLDCPPLIB environment variable disables exception handling, unless you compile with the -h exceptions option. CRAY_PE_TARGET Specifies the target system to be used as the default in all compilations. Valid options are x86-64, opteron, barcelona, shanghai, istanbul, mc8,mc12, and interlagos. It is overridden by the -hcpu=target option (see -h cpu=target_system on page 57). If unset, the default target is x86-64. CRI_CC_OPTIONS CRI_cc_OPTIONS Specifies command line options that are applied to all compilations. Options specified by this environment variable are added following the options specified directly on the command line. This is especially useful for adding options to compilations done with build tools. Identifies your requirements for native language, local customs, and coded character set with regard to compiler messages. Controls the format in which you receive compiler messages. Specifies the message system catalogs that should be used. Specifies the number of processes used for simultaneous compilations The default is 1. When more than one source file is specified on the command line, compilations may be multiprocessed by setting the environment variable NPROC to a value greater than 1. You can set NPROC to any value; however, large values can overload the system. S–2179–80 63Cray C and C++ Reference Manual 2.21 Run Time Environment Variables CRAY_MALLOPT_OFF If set, then the system default mallopt parameters are used, instead of the compiler default parameters. For most programs, run time performance is improved by using the compiler defaults, but more memory may be used. MALLOC_MMAP_MAX_ Specifies the maximum number of memory chunks to allocate with mmap. The compiler default value is 0. For most programs, run time performance is improved by using the compiler default, but more memory may be used. MALLOC_TRIM_THRESHOLD_ Specifies the minimum size of the unused memory region at the top of the heap before the region is returned to the operating system. The compiler default value is 536870912 bytes. For most programs, run time performance is improved by using the compiler default, but more memory may be used. PGAS_ERROR_FILE Specifies the location to which libpgas (the library which provides an interface to the internal system network) error messages are written. The default is stderr. If stdout is specified, errors will be written to standard output. 2.22 OpenMP Environment Variables For Cray-specific information about OpenMP environment variables, see Chapter 4, Using OpenMP on page 95. For documentation of standard OpenMP environment variables, see the OpenMP Application Program Interface Version 3.0 May 2008 standard (http://openmp.org/wp/openmp-specifications/). 64 S–2179–80Using #pragma Directives [3] The #pragma directives are used within the source program to request certain kinds of special processing. The directives are part of the C and C++ languages, but the meaning of any #pragma directive is defined by the implementation. #pragma directives are expressed in the following form: #pragma [_CRI] identifier [arguments] The _CRI specification is optional; it ensures that the compiler will issue a message concerning any directives that it does not recognize. Diagnostics are not generated for directives that do not contain the _CRI specification. These directives are classified according to the following types: • General (General Directives on page 67) • Vectorization (Vectorization Directives on page 75) • Scalar (Scalar Directives on page 84) • Inlining (Inlining and Cloning Directives on page 90) Macro expansion occurs on the directive line after the directive name. That is, macro expansion is applied only to arguments. Note: OpenMP #pragma directives are described in Chapter 4, Using OpenMP on page 95. At the beginning of each section that describes a directive, information is included about the compilers that allow the use of the directive and the scope of the directive. Unless otherwise noted, the following default information applies to each directive: Compiler: Cray C and Cray C++ Scope: Local and global The scoping list may also indicate that a directive has a lexical block scope. A lexical block is the scope within which a directive is on or off and is bounded by the opening curly brace just before the directive was declared and the corresponding closing curly brace. Only applicable executable statements within the lexical block are affected as indicated by the directive. The lexical block does not include the statements contained within a procedure that is called from the lexical block. S–2179–80 65Cray C and C++ Reference Manual This example code fragment shows the lexical block for the upc strict and upc relaxed directives: void Example(void) { #pragma _CRI upc strict // UPC strict state is on ... { ... // UPC strict state is still on #pragma _CRI upc relaxed // UPC strict state is now off ... } // UPC strict state is back on ... } 3.1 Protecting Directives To ensure that your directives are interpreted only by the Cray C and C++ compilers, use the following coding technique in which directive is the name of the directive: #if _CRAYC #pragma _CRI directive #endif This ensures that other compilers used to compile this code will not interpret the directive. Some compilers diagnose any directives that they do not recognize. The Cray C and C++ compilers diagnose directives that are not recognized only if the _CRI specification is used. 3.2 Directives in Cray C++ C++ prohibits referencing undeclared objects or functions. Objects and functions must be declared prior to using them in a #pragma directive. This is not always the case with C. Some #pragma directives take function names as arguments (for example: #pragma _CRI weak, #pragma _CRI suppress, and #pragma _CRI inline_always name [,name ...]). Member functions and qualified names are allowed for these directives. 3.3 Loop Directives Many directives apply to groups. Unless otherwise noted, these directives must appear before a for, while, or do while loop. These directives may also appear before a label for if...goto loops. If a loop directive appears before a label that is not the top of an if...goto loop, it is ignored. 66 S–2179–80Using #pragma Directives [3] 3.4 Alternative Directive Form: _Pragma Compiler directives can also be specified in the following form, which has the advantage in that it can appear inside macro definitions: _Pragma("_CRI identifier") This form has the same effect as using the #pragma form, except that everything that appeared on the line following the #pragma must now appear inside the double quotation marks and parentheses. The expression inside the parentheses must be a single string literal; it cannot be a macro that expands into a string literal. _Pragma is an extension to the C and C++ standards. The following is an example using the #pragma form: #pragma _CRI concurrent The following is the same example using the alternative form: _Pragma("_CRI concurrent") In the following example, the loop automatically vectorizes wherever the macro is used: #define _str( _X ) # _X #define COPY( _A, _B, _N { int i; _Pragma( "_CRI concurrent" ) _Pragma( _str( _CRI loop_info cache_nt( _B ) ) ) for ( i = 0; i < _N; i++ ) { _A[i] = _B[i]; } } void copy_data( int *a, int *b, int n ) { COPY( a, b, n ); } Macros are expanded in the string literal argument for _Pragma in an identical fashion to the general specification of a #pragma directive. 3.5 General Directives General directives specify compiler actions that are specific to the directive and have no similarities to the other types of directives. The following sections describe general directives. S–2179–80 67Cray C and C++ Reference Manual 3.5.1 [no]bounds Directive The bounds directive specifies that pointer and array references are to be checked. The nobounds directive specifies that this checking is to be disabled. When bounds checking is in effect, pointer references are checked to ensure that they are neither 0 nor greater than the machine memory limit. Array references are checked to ensure that the array subscript is not less than 0 or greater than or equal to the declared size of the array. Both directives may be used only within function bodies. They apply until the end of the function body or until another bounds/nobounds directive appears. They ignore block boundaries. These directives have the following format: #pragma _CRI bounds #pragma _CRI nobounds The following example illustrates the use of the bounds directive: int a[30]; #pragma _CRI bounds void f(void) { int x; x = a[30]; . . . } 3.5.2 duplicate Directive Scope: Global The duplicate directive lets you provide additional, externally visible names for specified functions. You can specify duplicate names for functions by using a directive with one of the following forms: #pragma _CRI duplicate actual as dupname... #pragma _CRI duplicate actual as (dupname...) The actual argument is the name of the actual function to which duplicate names will be assigned. The dupname list contains the duplicate names that will be assigned to the actual function. The dupname list may be optionally parenthesized. The word as must appear as shown between the actual argument and the comma-separated list of dupname arguments. The duplicate directive can appear anywhere in the source file and it must appear in global scope. The actual name specified on the directive line must be defined somewhere in the source as an externally accessible function; the actual function cannot have a static storage class. 68 S–2179–80Using #pragma Directives [3] The following example illustrates the use of the duplicate directive: #include extern void maxhits(void); #pragma _CRI duplicate maxhits as count, quantity /* OK */ void maxhits(void) { #pragma _CRI duplicate maxhits as tempcount /* Error: #pragma _CRI duplicate can't appear in local scope */ } double _Complex minhits; #pragma _CRI duplicate minhits as lower_limit /* Error: minhits is not declared as a function */ extern void derivspeed(void); #pragma _CRI duplicate derivspeed as accel /* Error: derivspeed is not defined */ static void endtime(void) { } #pragma _CRI duplicate endtime as limit /* Error: endtime is defined as a static function */ Because duplicate names are simply additional names for functions and are not functions themselves, they cannot be declared or defined anywhere in the compilation unit. To avoid aliasing problems, duplicate names may not be referenced anywhere within the source file, including appearances on other directives. In other words, duplicate names may only be referenced from outside the compilation unit in which they are defined. The following example references duplicate names: void converter(void) { structured(void); } #pragma _CRI duplicate converter as factor, multiplier /* OK */ void remainder(void) { } #pragma _CRI duplicate remainder as factor, structured /* Error: factor and structured are referenced in this file */ S–2179–80 69Cray C and C++ Reference Manual Duplicate names can be used to provide alternate external names for functions, as shown in the following examples. main.c: extern void fctn(void), FCTN(void); main() { fctn(); FCTN(); } fctn.c: #include void fctn(void) { printf("Hello world\n"); } #pragma _CRI duplicate fctn as FCTN Files main.c and fctn.c are compiled and linked using the following command line: % cc main.c fctn.c When the executable file a.out is run, the program generates the following output: Hello world Hello world 3.5.3 message Directive The message directive directs the compiler to write the message defined by text to stderr as a warning message. Unlike the error directive, the compiler continues after processing a message directive. The format of this directive is as follows: #pragma _CRI message "text" The following example illustrates the use of the message compiler directive: #define FLAG 1 #ifdef FLAG #pragma _CRI message "FLAG is Set" #else #pragma _CRI message "FLAG is NOT Set" #endif 70 S–2179–80Using #pragma Directives [3] 3.5.4 cache Directive The cache directive asserts that all memory operations with the specified symbols as the base are to be allocated in cache. This is an advisory directive. The cache directive is meaningful for stores in that it allows the user to override a decision made by the automatic cache management. This directive may be locally overridden by the use of a #pragma loop_info directive. This directive overrides automatic cache management decisions (see -h cachen). To use the directive, you must place it only in the specification part, before any executable statement. The format of the cache directive is: #pragma _CRI cache base_name [,base_name ...] base_name The base name of the object that should be placed into the cache. This can be the base name of any object such as an array, scalar structure, and so on, without member references like C[10]. If you specify a pointer in the list, only the references, not the pointer itself, are cached. 3.5.5 cache_nt Directive The cache_nt directive is an advisory directive that specifies objects that should use non-temporal reads and writes. Use this directive to identify objects that should not be placed in cache. The format of the cache_nt directive is: #pragma _CRI cache_nt base_name [,base_name ...] base_name The base name of the object that should use non-temporal reads and writes. This can be the base name of any object such as an array, scalar structure, and so on, without member references like C[10]. If you specify a pointer in the list, only the references, not the pointer itself, have the cache non-temporal property. This directive overrides the automatic cache management level that was specified using the -h cachen option on the compiler command line. This directive may be overridden locally by use of a loop_info directive. 3.5.6 ident Directive The ident pragma directs the compiler to store the string indicated by text into the object (.o) file. This can be used to place a source identification string into an object file. The format of this directive is as follows: #pragma _CRI ident text S–2179–80 71Cray C and C++ Reference Manual 3.5.7 [no]opt Directive Scope: Global The noopt directive disables all automatic optimizations and causes optimization directives to be ignored in the source code that follows the directive. Disabling optimization removes various sources of potential confusion in debugging. The opt directive restores the state specified on the command line for automatic optimization and directive recognition. These directives have global scope and override related command line options. The format of these directives is as follows: #pragma _CRI opt #pragma _CRI noopt The following example illustrates the use of the opt and noopt compiler directives: #include void sub1(void) { printf("In sub1, default optimization\n"); } #pragma _CRI noopt void sub2(void) { printf("In sub2, optimization disabled\n"); } #pragma _CRI opt void sub3(void) { printf("In sub3, optimization enabled\n"); } main() { printf("Start main\n"); sub1(); sub2(); sub3(); } 3.5.8 autothread, noautothread Directives Scope: Local The autothread and noautothread directives turn autothreading on and off for selected blocks of code. The format of these directives is as follows: #pragma _CRI autothread #pragma _CRI noautothread 72 S–2179–80Using #pragma Directives [3] 3.5.9 Probability Directives The probability, probability_almost_always, and probability_almost_never directives specify information used by interprocedure analysis (IPA) and the optimizer to produce faster code sequences. The specified probability is a hint, rather than a statement of fact. You can also specify almost_never and almost_always by using the values 0.0 and 1.0, respectively. These directives have the following format: #pragma probability const #pragma probability_almost_always #pragma probability_almost_never const is an expression that evaluates to a floating point constant at compilation time. (0.0 <= const <= 1.0.) These directives can appear anywhere executable code is legal. Each directive applies to the block of code where it appears. It is important to realize that the directive should not be applied to a conditional test directly; rather, it should be used to indicate the relative probability of a then or else branch being executed. Example: if ( a[i] > b[i] ) { #pragma probability 0.3 a[i] = b[i]; } This example states that the probability of entering the block of code with the assignment statement is 0.3 or 30%. This also means that a[i] is expected to be greater than b[i] 30% of the time. Note that the probability directive appears within the conditional block of code, rather than before it. This removes some of the ambiguity that has plagued other implementations that tie the directive directly to the conditional code. This information is used to guide inlining decisions, branch elimination optimizations, branch hint marking, and the choice of the optimal algorithmic approach to the vectorization of conditional code. The following GCC-style intrinsic is also accepted when it appears in a conditional test: __builtin_expect( expr, const ) S–2179–80 73Cray C and C++ Reference Manual The following example: if ( __builtin_expect( a[i] > b[i], 0 ) ) { a[i] = b[i]; } is roughly equivalent to: if ( a[i] > b[i] ) { #pragma _CRI probability_almost_never a[i] = b[i]; } 3.5.10 weak Directive Scope: Global When statically linking, the weak directive specifies an external identifier that may remain unresolved throughout the compilation. This directive has no effect when dynamically linking. A weak external reference can be a reference to a function or to a data object. A weak external does not increase the total memory requirements of your program. Declaring an object as a weak external directs the linker to do one of these tasks: • Link the object only if it is already linked (that is, if a strong reference exists); otherwise, leave it is as an unsatisfied external. The linker does not display an unsatisfied external message if weak references are not resolved. • If a strong reference is specified in the weak directive, resolve all weak references to it. Note: The linker treats weak externals as unsatisfied externals, so they remain silently unresolved if no strong reference occurs during compilation. Thus, it is your responsibility to ensure that run time references to weak external names do not occur unless the linker (using some "strong” reference elsewhere) has actually linked the entry point in question. These are the forms of the weak directive: #pragma _CRI weak var #pragma _CRI weak sym1 = sym2 var The name of an external sym1 Defines an externally visible weak symbol sym2 Defines an externally visible strong symbol defined in the current compilation. The first form allows you to declare one or more weak references on one line. The second form allows you to assign a strong reference to a weak reference. The weak directive must appear at global scope. 74 S–2179–80Using #pragma Directives [3] The attributes that weak externals must have depend on the form of the weak directive that you use: • First form, weak externals must be declared, but not defined or initialized, in the source file. • Second form, weak externals may be declared, but not defined or initialized, in the source file. • Either form, weak externals cannot be declared with a static storage class. The following example illustrates these restrictions: extern long x; #pragma _CRI weak x /* x is a weak external data object */ extern void f(void); #pragma _CRI weak f /* f is a weak external function */ extern void g(void); #pragma _CRI weak g=fun; /* g is a weak external function with a strong reference to fun */ long y = 4; #pragma _CRI weak y /* ERROR - y is actually defined */ static long z; #pragma _CRI weak z /* ERROR - z is declared static */ void fctn(void) { #pragma _CRI weak a /* ERROR - directive must be at global scope */ } 3.6 Vectorization Directives Because vector operations cannot be expressed directly in Cray C and C++, the compilers must be capable of vectorization, which means transforming scalar operations into equivalent vector operations. The candidates for vectorization are operations in loops and assignments of structures. The subsections that follow describe the compiler directives used to control vectorization. 3.6.1 hand_tuned Directive The format of this directive is: #pragma _CRI hand_tuned This directive asserts that the code in the loop nest has been arranged by hand for maximum performance, and the compiler should restrict some of the more aggressive automatic expression rewrites. The compiler should still fully optimize and vectorize the loop within the constraints of the directive. S–2179–80 75Cray C and C++ Reference Manual The hand_tuned directive applies to the next loop in the same manner as the concurrent and safe_address directives. Warning: Use of this directive may severely impede performance. Use carefully and evaluate before and after performance. 3.6.2 loop_info Directive Scope: Local The loop_info directive allows additional information to be specified about the behavior of a loop, including run time trip count and hints on cache allocation strategy. In regard to trip count information, the loop_info directive is similar to the shortloop or shortloop128 directive but provides more information to the optimizer and can produce faster code sequences. loop_info is used immediately before a for loop to indicate minimum, maximum, or estimated trip count. The compiler will diagnose misuse at compile time (when able) or when option -h dir_check is specified at run time. For cache allocation hints, the loop_info directive can be used to override default settings, cache or cache_nt directives, or override automatic cache management decisions. The cache hints are local and apply only to the specified loop nest. The format of this directive is: #pragma _CRI loop_info [min_trips(c)] [est_trips(c)] [max_trips(c)][cache( symbol[,symbol ...] )][cache_nt(symbol[,symbol ...] ) ][prefetch] [noprefetch] 76 S–2179–80Using #pragma Directives [3] The prefetch and noprefetch options are deferred. c An expression that evaluates to an integer constant at compilation time. min_trips Specifies guaranteed minimum number of trips. est_trips Specifies estimated or average number of trips. max_trips Specifies guaranteed maximum number of trips. cache Specifies that symbol is to be allocated in cache; this is the default if no hint is specified and the cache_nt directive is not specified. cache_nt Specifies that symbol is to use non-temporal reads and writes. prefetch Specifies a preference that prefetches be performed for the following loop. noprefetch Specifies a preference that no prefetches be performed for the following loop. symbol The base name of the object that should not be placed into the cache. This can be the base name of any object (such as an array or scalar structure) without member references like C[10]. If you specify a pointer in the list, only the references, not the pointer itself, have the no cache allocate property. Example 8. Trip counts In the following example, the minimum trip count is 1 and the maximum trip count is 1000: void loop_info( double *restrict a, double *restrict b, double s1, int n ) { int i; #pragma _CRI loop_info min_trips(1) max_trips(1000), cache_nt(b) for (i = 0; i< n; i++) { if(a[i] != 0.0) { a[i] = a[i] + b[i]*s1; } } } S–2179–80 77Cray C and C++ Reference Manual 3.6.3 loop_info prefer_thread, prefer_nothread Directives Scope: Local Use these directives to indicate a preference for turning threading on or off for selected loops. Use the loop_info prefer_thread directive to indicate your preference that the loop following the directive be threaded. The loop_info prefer_nothread indicates your preference that the loop following the directive should not be threaded. The format of these directives is: #pragma _CRI loop_info prefer_thread #pragma _CRI loop_info prefer_nothread 3.6.4 nopattern Directive Scope: Local The nopattern directive disables pattern matching for the loop immediately following the directive. The format of this directive is as follows: #pragma _CRI nopattern By default, the compiler detects coding patterns in source code sequences and replaces these sequences with calls to optimized library functions. In most cases, this replacement improves performance. There are cases, however, in which this substitution degrades performance. This can occur, for example, in loops with very low trip counts. In such a case, you can use the nopattern directive to disable pattern matching and cause the compiler to generate inline code. In the following example, placing the nopattern directive in front of the outer loop of a nested loop turns off pattern matching for the matrix multiply that takes place inside the inner loop: double a[100][100], b[100][100], c[100][100]; void nopat(int n) { int i, j, k; #pragma _CRI nopattern for (i=0; i < n; ++i) { for (j = 0; j < n; ++j) { for (k = 0; k < n; ++k) { c[i][j] += a[i][k] * b[k][j]; } } } } 78 S–2179–80Using #pragma Directives [3] 3.6.5 novector Directive Scope: Local The novector directive directs the compiler to not vectorize the loop immediately following the directive. It overrides any other vectorization-related directives, as well as the -h vector command line option. The format of this directive is as follows: #pragma _CRI novector The following example illustrates the use of the novector compiler directive: #pragma _CRI novector for (i = 0; i < h; i++) { /* Loop not vectorized */ a[i] = b[i] + c[i]; } 3.6.6 permutation Directive The permutation directive specifies that an integer array has no repeated values. This directive is useful when the integer array is used as a subscript for another array (vector-valued subscript). This directive may improve code performance. This directive has the following format: #pragma _CRI permutation symbol [, symbol ] ... In a sequence of array accesses that read array element values from the specified symbols with no intervening accesses that modify the array element values, each of the accessed elements will have a distinct value. When an array with a vector-valued subscript appears on the left side of the equal sign in a loop, many-to-one assignment is possible. Many-to-one assignment occurs if any repeated elements exist in the subscripting array. If it is known that the integer array is used merely to permute the elements of the subscripted array, it can often be determined that many-to-one assignment does not exist with that array reference. Sometimes a vector-valued subscript is used as a means of indirect addressing because the elements of interest in an array are sparsely distributed; in this case, an integer array is used to select only the desired elements, and no repeated elements exist in the integer array, as in the following example: int *ipnt; #pragma permutation ipnt ... for ( i = 0; i < N; i++ ) { a[ipnt[i]] = b[i] + c[i]; } The permutation directive does not apply to the array a. Rather, it applies to the pointer used to index into it, ipnt. By knowing that ipnt is a permutation, the compiler can safely generate an unordered scatter for the write to a. S–2179–80 79Cray C and C++ Reference Manual 3.6.7 [no]pipeline Directive Software-based vector pipelining (software vector pipelining) provides additional optimization beyond the normal hardware-based vector pipelining. In software vector pipelining, the compiler analyzes all vector loops and automatically attempts to pipeline a loop if doing so can be expected to produce a significant performance gain. This optimization also performs any necessary loop unrolling. In some cases the compiler either does not pipeline a loop that could be pipelined or pipelines a loop without producing performance gains. In these situations, you can use the pipeline or nopipeline directive to advise the compiler to pipeline or not pipeline the loop immediately following the directive. Software vector pipelining is valid only for the innermost loop of a loop nest. The pipeline and nopipeline directives are advisory only. While you can use the nopipeline directive to inhibit automatic pipelining, and you can use the pipeline directive to attempt to override the compiler's decision not to pipeline a loop, you cannot force the compiler to pipeline a loop that cannot be pipelined. Loops that have been pipelined are so noted in loopmark listing messages. The formats of the pipelining directives are as follows: #pragma _CRI pipeline #pragma _CRI nopipeline 3.6.8 prefervector Directive Scope: Local The prefervector pragma directs the compiler to vectorize the loop immediately following the directive if the loop contains more than one loop in the nest that can be vectorized. The directive states a vectorization preference and does not guarantee that the loop has no memory-dependence hazard. The format of this directive is: #pragma _CRI prefervector 80 S–2179–80Using #pragma Directives [3] The following example illustrates the use of the prefervector directive: float a[1000], b[100][1000]; void f(int m, int n) { int i, j; #pragma _CRI prefervector for (i = 0; i < n; i++) { for (j = 0; j < m; j++) { a[i] += b[j][i]; } } } In this example, both loops can be vectorized, but the directive directs the compiler to vectorize the outer for loop. Without the directive and without any knowledge of n and m, the compiler would vectorize the inner loop. 3.6.9 pgo loop_info Directive Scope: Local The format of this directive is as follows: #pragma _CRI pgo loop_info The pgo loop_info directive enables profile-guided optimizations by tagging loopmark information as having come from profiling. For information about CrayPat and profile information, see the Using Cray Performance Analysis Tools guide. 3.6.10 safe_address Directive Scope: Local The format of this directive is as follows: #pragma _CRI safe_address The safe_address directive specifies that it is safe to speculatively execute memory references within all conditional branches of a loop. In other words, you know that these memory references can be safely executed in each iteration of the loop. For most code, the safe_address directive can improve performance significantly by preloading vector expressions. However, most loops do not require this directive to have preloading performed. The directive is required only when the safety of the operation cannot be determined or index expressions are very complicated. The safe_address directive is an advisory directive. That is, the compiler may override the directive if it determines the directive is not beneficial. S–2179–80 81Cray C and C++ Reference Manual If you do not use the directive on a loop and the compiler determines that it would benefit from the directive, it issues a message indicating such. The message is similar to this: CC-6375 cc: VECTOR File = ctest.c, Line = 6 A loop would benefit from "#pragma safe_address". If you use the directive on a loop and the compiler determines that it does not benefit from the directive, it issues a message that states the directive is superfluous and can be removed. To see the messages, you must use the -h report=v or -h msgs option. ! Caution: Incorrect use of the directive can result in segmentation faults, bus errors, or excessive page faulting. However, it should not result in incorrect answers. Incorrect usage can result in very severe performance degradations or program aborts. In the example below, the compiler will not preload vector expressions, because the value of j is unknown. However, if you know that references to b[i][j] is safe to evaluate for all iterations of the loop, regardless of the condition, you can use the safe_address directive for this loop as shown below: void x3( double a[restrict 1000], int j ) { int i; #pragma _CRI safe_address for ( i = 0; i < 1000; i++ ) { if ( a[i] != 0.0 ) { b[j][i] = 0.0; } } } With the directive, the compiler can safely load b[i][j] as a vector, merge 0.0 where the condition is true, and store the resulting vector safely. 3.6.11 safe_conditional Directive The safe_conditional directive specifies that it is safe to execute all references and operations within all conditional branches of a loop. In other words, you know that these memory references can be safely executed in each iteration of the loop. This directive specifies that memory and arithmetic operations are safe. This directive applies to scalar and vector loop nests. It can improve performance by allowing the hoisting of invariant expressions from conditional code and by allowing prefetching of memory references. The safe_conditional directive is an advisory directive. That is, the compiler may override the directive if it determines the directive is not beneficial. 82 S–2179–80Using #pragma Directives [3] ! Caution: Incorrect use of the directive can result in segmentation faults, bus errors, excessive page faulting, or arithmetic aborts. However, it should not result in incorrect answers. Incorrect usage can result in severe performance degradations or program aborts. The safe_conditional directive has the following format: #pragma _CRI safe_conditional In the following example, without the safe_conditional directive, the compiler cannot precompute the invariant expression s1*s2 because their values are unknown and may cause an arithmetic trap if executed unconditionally. However, if you know that the condition is true at least once, then s1*s2 is safe to speculatively execute. The safe_conditional compiler directive can be used to imply the safety of the operation. With the directive, the compiler evaluates s1*s2 outside of the loop, rather than under control of the conditional code. In addition, all control flow is removed from the body of the vector loop, because s1*s2 no longer poses a safety risk. void safe_cond( double a[restrict 1000], double s1, double s2 ) { int i; #pragma _CRI safe_conditional for (i = 0; i< 1000; i++) { if( a[i] != 0.0) { a[i] = a[i] + s1*s2; } } } 3.6.12 shortloop and shortloop128 Directives Scope: Local The shortloop compiler directive identifies loops that execute with a maximum iteration count of 64 and a minimum iteration count of 1. The shortloop128 compiler directive identifies loops that execute with a maximum iteration count of 128 and a minimum iteration count of 1. If the iteration count is outside the range for the directive, results are unpredictable. The compiler will diagnose misuse at compile time (when able) or at run time if option -h dir_check is specified. The syntax of these directives are as follows: #pragma _CRI shortloop #pragma _CRI shortloop128 The shortloop and shortloop128 directives are exactly equivalent to #pragma _CRI loop_info min_trips(1) max_trips(64) and #pragma _CRI loop_info min_trips(1) max_trips(128), respectively. The loop_info pragma is the preferred form. S–2179–80 83Cray C and C++ Reference Manual The following examples illustrate the use of the shortloop and shortloop128 directives: #pragma _CRI shortloop for (i = 0; i < n; i++) { /* 0 < = n < = 63 */ a[i] = b[i] + c[i]; } #pragma _CRI shortloop128 for (i = 0; i < n; i++) { /* 0 < = n < = 127 */ a[i] = b[i] + c[i]; } 3.7 Scalar Directives This section describes the scalar optimization directives, which control aspects of code generation, register storage, and other scalar operations. 3.7.1 blockable Directive The blockable directive specifies that it is legal to cache block the subsequent loops. The format of this directive is as follows: #pragma _CRI blockable (nest-depth) where nest-depth specifies the depth of the loop nest to be blocked. This directive instructs the compiler to perform a cache blocking rewrite in which the following nest-depth loops participate. The nest to be transformed must be a fully-permutable, perfect, rectangular nest. A fully-permutable nest is a nest that may be legally interchanged in any order. To be perfect, a nest must have no statements between the participating loops. Rectangular means that the lower and upper bounds (and stride) of each loop must be independent of the value of the loop control variables of all enclosing member loops in the rewrite set. Though the compiler will catch some misuse of this directive, it will not detect all misuses. The blockingsize directive may be used to control the size and shape of the cache block, otherwise known as a tile. 3.7.2 blockingsize Directive When a loop is annotated with a blockingsize directive, and this loop is to be cache blocked automatically or by blockable directive, the indicated blocking factors are employed. In the absence of a blockingsize directive, the compiler will select blocking factors. The compiler attempts to include this loop within cache, but it cannot guarantee this. 84 S–2179–80Using #pragma Directives [3] The formats of this directive is as follows: #pragma _CRI blockingsize (n1,[,n2]) where n1 and n2 are integer constants that indicate the block size, with 0 <= n1 <= n2 <= 2**30. Note: The Cray compiler only blocks one level of cache – the secondary cache, which is specified by n2. If a single is block size is specified, it is interpreted as the blocking factor for the secondary cache. A loop with blockingsize n > 1 is strip mined to length n as shown: do ii = 1, trips, n do i = ii, min( ii + n - 1, trips ) To fully understand the interaction of blockingsizes, it is helpful to examine the two steps employed by the compiler: 1. Strip mine blockable nest members according to blockingsize • if n > 1, strip mine by n, creating loop_outer and loop_inner. • if n = 0, do not strip mine, treat loop as loop_inner. Full tripcount. The entire loop is inside the block. • if n = 1, do not strip mine, treat loop as loop_outer. 2. Interchange the loops resulting from previous step so all "outer" loops are outside the "inner" loops and the relative order within each subset is preserved. 3.7.3 noblocking Directive The noblocking directive asserts that the loop following the directive should not be cache blocked for the primary or secondary cache. It is an error to place a noblocking directive before a loop that is part of a blockable collection. The format of this directive is as follows: #pragma _CRI noblocking 3.7.4 collapse and nocollapse Directives Scope: Local The loop collapse directives control collapse of the immediately following loop nest. The formats of these directives are as follows: #pragma _CRI collapse loop-number1,loop-number2[,loop-number3]... #pragma _CRI nocollapse S–2179–80 85Cray C and C++ Reference Manual When the collapse directive is applied to a loop nest, the loop numbers of the participating loops must be listed in order of increasing access stride. Loop numbers range from 1 to the nesting level of the most deeply nested loop. The directive enables the compiler to assume appropriate conformity between trip counts. The compiler diagnoses misuse at compile time (when able); or, if -h dir_check is specified, at run time. The nocollapse directive disqualifies the immediately following loop from collapsing with any other loop. Collapse is almost always desirable, so use this directive sparingly. Loop collapse is a special form of loop coalesce. Any perfect loop nest may be coalesced into a single loop, with explicit rediscovery of the intermediate values of original loop control variables. The rediscovery cost, which generally involves integer division, is quite high. Therefore, coalesce is rarely suitable for vectorization. It may be beneficial for multithreading. By definition, loop collapse occurs when loop coalesce may be done without the rediscovery overhead. To meet this requirement, all memory accesses must have uniform stride. 3.7.5 concurrent Directive Scope: Local The concurrent directive indicates that no data dependence exists between array references in different iterations of the loop that follows the directive. This can be useful for vectorization optimizations. The format of the concurrent directive is as follows: #pragma _CRI concurrent [safe_distance=n] n An integer that represents the number of additional consecutive loop iterations that can be executed in parallel without danger of data conflict. n must be an integral constant > 0. The concurrent directive is ignored if the safe_distance clause is used and vectorization is requested on the command line. In the following example, the concurrent directive indicates that the relationship k>3 is true. The compiler will safely load all the array references x[i-k], x[i-k+1], x[i-k+2], and x[i-k+3] during loop iteration i. #pragma _CRI concurrent safe_distance=3 for (i = k + 1; i < n;i++) { x[i] = a[i] + x[i-k]; } 86 S–2179–80Using #pragma Directives [3] 3.7.6 [no]interchange Directive Scope: Local The loop interchange control directives specify whether or not the order of the following two or more loops should be interchanged. These directives apply to the loops that they immediately precede. The formats of these directives are as follows: #pragma _CRI interchange(loop_number1, loop_number2[, loop_number3] ...) #pragma _CRI nointerchange The first format specifies two or more loop numbers. Loop numbers range from 1 to nesting depth of the most deeply nested loop. They can be specified in any order, and the compiler reorders the loops. The loops must be perfectly nested. If the loops are not perfectly nested, you may receive unexpected results. The compiler reorders the loops such that the loop with loop-number1 is outermost, then loop-number2, then loop-number3. The second format inhibits loop interchange on the loop that immediately follows the directive. In the following example, the interchange directive reorders the loops; the k loop becomes the outermost and the i loop the innermost: #define N 100 A[N][N][N]; void f(int n) { int i, j, k; #pragma _CRI interchange( 2, 3, 1 ) for (i=0; i < n; i++) { for (k=0; k < n; k++) { for (j = 0; j < n; j++) { A[k][j][i] = 1.0; } } } } 3.7.7 noreduction Directive Note: This directive is no longer recognized. Use the #pragma _CRI novector directive instead. S–2179–80 87Cray C and C++ Reference Manual 3.7.8 suppress Directive The suppress directive suppresses optimization in two ways, determined by its use with either global or local scope. The global scope suppress directive specifies that all associated local variables are to be written to memory before a call to the specified function. This ensures that the value of the variables will always be current. The global suppress directive takes the following form: #pragma _CRI suppress func... The local scope suppress directive stores current values of the specified variables in memory. If the directive lists no variables, all variables are stored to memory. This directive causes the values of these variables to be reloaded from memory at the first reference following the directive. The local suppress directive has the following format: #pragma _CRI suppress [var] ... The net effect of the local suppress directive is similar to declaring the affected variables to be volatile except that the volatile qualifier affects the entire program, whereas the local suppress directive affects only the block of code in which it resides. 3.7.9 [no]unroll Directive Scope: Local The unroll directive allows the user to control unrolling for individual loops or to specify no unrolling of a loop. Loop unrolling can improve program performance by revealing cross-iteration memory optimization opportunities such as read-after-write and read-after-read. The effects of loop unrolling also include: • Improved loop scheduling by increasing basic block size • Reduced loop overhead • Improved chances for cache hits The formats for these compiler directives are: #pragma _CRI unroll n #pragma _CRI nounroll The nounroll directive disables loop unrolling for the next loop and does not accept the integer argument n. The nounroll directive is equivalent to the unroll 0 and unroll 1 directives. The n argument applies only to the unroll directive and specifies no loop unrolling (n = 0 or 1) or the total number of loop body copies to be generated (2 = n = 63). 88 S–2179–80Using #pragma Directives [3] If you do not specify a value for n, the compiler will determine the number of copies to generate based on the number of statements in the loop nest. Note: The compiler cannot always safely unroll non-innermost loops due to data dependencies. In these cases, the directive is ignored (see Example 10). The unroll directive can be used only on loops with iteration counts that can be calculated before entering the loop. If unroll is specified on a loop that is not the innermost loop in a loop nest, the inner loops must be nested perfectly. That is, all loops in the nest can contain only one loop, and the innermost loop can contain work. Example 9. Unrolling outer loops In the following example, assume that the outer loop of the following nest will be unrolled by 2: #pragma _CRI unroll 2 for (i = 0; i < 10; i++) { for (j = 0; j < 100; j++) { a[i][j] = b[i][j] + 1; } } With outer loop unrolling, the compiler produces the following nest, in which the two bodies of the inner loop are adjacent: for (i = 0; i < 10; i += 2) { for (j = 0; j < 100; j++) { a[i][j] = b[i][j] + 1; } for (j = 0; j < 100; j++) { a[i+1][j] = b[i+1][j] + 1; } } The compiler then jams, or fuses, the inner two loop bodies, producing the following nest: for (i = 0; i < 10; i += 2) { for (j = 0; j < 100; j++) { a[i][j] = b[i][j] + 1; a[i+1][j] = b[i+1][j] + 1; } } S–2179–80 89Cray C and C++ Reference Manual Example 10. Illegal unrolling of outer loops Outer loop unrolling is not always legal because the transformation can change the semantics of the original program. For example, unrolling the following loop nest on the outer loop would change the program semantics because of the dependency between a[i][...] and a[i+1][...]: /* directive will cause incorrect code due to dependencies! */ #pragma _CRI unroll 2 for (i = 0; i < 10; i++) { for (j = 1; j < 100; j++) { a[i][j] = a[i+1][j-1] + 1; } } 3.7.10 nofission Directive Scope: Local The nofission directive instructs the compiler not to split statements in a given loop into distinct loops. Fission is prevented only for the loop specified; loops nested within the indicated loop remain fission candidates unless likewise annotated. 3.7.11 [no]fusion Directive Scope: Local The nofusion directive instructs the compiler to not attempt loop fusion on the following loop even when the -h fusion option was specified on the compiler command line. The fusion directive instructs the compiler to attempt loop fusion on the following loop unless -h nofusion was specified on the compiler command line. The formats for these compiler directives are: #pragma _CRI fusion #pragma _CRI nofusion 3.8 Inlining and Cloning Directives Inlining and cloning directives can only appear in local scope — inside a function definition. Inlining directives always take precedence over the command line settings with the exception of -h ipa0, which instructs the compiler to ignore inlining directives. Cloning directives are enabled with -h ipa4, or higher. 90 S–2179–80Using #pragma Directives [3] 3.8.1 inline_enable, inline_disable, and inline_reset Directives The inline_enable pragma directs the compiler to attempt to inline functions at call sites. It has the following format: #pragma _CRI inline_enable The inline_disable directive tells the compiler to not inline functions at call sites. It has the following format: #pragma _CRI inline_disable The inline_reset directive returns the inlining state to the state specified on the command line (-h ipan). It has the following format: #pragma _CRI inline_reset The following example illustrates the use of these directives. Example 11. Using the inline_enable, inline_disable, and inline_reset directives The following code fragment shows how the inline_enable, inline_disable, and inline_reset directives would affect code compiled with the -h ipa4 option: void qux(int x) { void bar(void); int a = 1; x = a+a+a+a+a+a+a+a+a+a+a+a; bar(); } void foo(void) { int j = 1; /* enable inlining at all call sites here forward */ #pragma _CRI inline_enable qux(j); qux(j); /* disable inlining at all call sites here forward */ #pragma _CRI inline_disable qux(j); /* reset control to the command line -hipa4 */ #pragma _CRI inline_reset qux(j); } S–2179–80 91Cray C and C++ Reference Manual Example 12. Using inline_reset The following code fragment shows how the #pragma _CRI inline_reset directive would affect code compiled with the -h ipa3 option: void f1() { #pragma _CRI inline_disable /* No inlining will be done in f1; ... } void f2() { /* turn off all inlining to the end of the routine or another directive is encountered. */ #pragma _CRI inline_disable ... /* The inlining state is -h ipa3 for the remainder of f2 */ #pragma _CRI inline_reset ... } 3.8.2 inline_always and inline_never Directives The inline_always directive specifies functions that the compiler should always attempt to inline. If the directive is placed in the definition of the function, inlining is attempted at every call site to name in the entire input file being compiled. If the directive is placed in a function other than the definition, inlining is attempted at every call site to name within the specific function containing the directive. The format of the inline_always directive is as follows: #pragma _CRI inline_always name [,name ] ... The inline_never directive specifies functions that are never to be inlined. If the directive is placed in the definition of the function, inlining is never attempted at any call site to name in the entire input file being compiled. If the directive is placed in a function other than the definition, inlining is never attempted at any call site to name within the specific function containing the directive. The format of the inline_never directive is as follows: #pragma _CRI inline_never name [,name ] ... The name argument is the name of a function. 3.8.3 clone_enable, clone_disable, clone_reset Directives The clone_enable and clone_disable directives control whether cloning is attempted over a range of code. 92 S–2179–80Using #pragma Directives [3] If clone_enable is in effect, cloning is attempted at call sites. If clone_disable is in effect, cloning is not attempted at call sites. The clone_reset directive resets cloning to the state specified on the compiler command line. These directives have the following formats: #pragma _CRI clone_enable #pragma _CRI clone_disable #pragma _CRI clone_reset One of these directives remains in effect until the opposite directive is encountered, until the end of the program unit, or until the clone_reset directive is encountered. 3.8.4 clone_always and clone_never Directives The clone_always and clone_never directives specify functions or procedures that the compiler should always/never attempt to clone. If the directive is placed in the definition of the function, cloning is always/never attempted at every call site to name in the entire input file being compiled. If the directive is placed in a function other the definition, cloning is always/never attempted at every call to name within the specific function containing the directive. The clone_always, clone_never directives control cloning of a procedure for the compilation of the whole input file. Use compile options -hipaN or -OipaN where N is equal to or greater than 4 to enable cloning directives. These directives have the following formats: #pragma _CRI clone_always name [,name ...] #pragma _CRI clone_never name [,name ...] !DIR$ CLONEALWAYS name [,name ...] !DIR$ CLONENEVER name [,name ...] The name argument is the name of a function. 3.9 PGAS Directive Cray XE systems support the following PGAS directive. 3.9.1 defer_sync Directive The defer_sync directive defers the synchronization of PGAS data until the next synchronization. S–2179–80 93Cray C and C++ Reference Manual Normally the compiler synchronizes the references in a statement as late as possible without violating program semantics. The purpose of the defer_sync directive is to synchronize the references even later, beyond where the compiler can determine it is safe. PGAS data references made by the single statement immediately following the pgas defer_sync directive are not synchronized until the next fence instruction. Use this directive to force all references in the next statement to be non-blocking. This helps for cases where the compiler cannot prove that it is safe. For example, if there is a remote-memory access (RMA) put near the end of a subroutine, the compiler must guard against the put value being read back immediately after the subroutine returns, so the put is synchronized just before returning. The programmer, however, may know that the value is not read back and can insert a pgas defer_sync directive. The format is as follows: #pragma pgas defer_sync Example 13. Using defer_sync void my_put( shared int* x, int thread, int value ) { #pragma pgas defer_sync x[thread] = value; } 94 S–2179–80Using OpenMP [4] This compiler supports the OpenMP 3.0 standard (OpenMP Application Program Interface Version 3.0 May 2008 Copyright © 1997-2008 OpenMP Architecture Review Board). 4.1 Deferred OpenMP Features The following OpenMP features are not currently supported by the Cray C and Cray C++ compilers. These limitations will be removed in future releases. • Orphaned task constructs may have an implicit taskwait directive added to the end of the routine. This is not required by the specification but is currently required by the Cray implementation. This limits the amount of parallelism that may be seen. • Task switching is not implemented. The thread that starts executing a task is the thread that finishes the task. • Support for OpenMP Random Access Iterators (RAIs) in the C++ Standard Template Library (STL) is deferred to a future release. 4.2 Cray Implementation Differences The OpenMP C and C++ Application Program Interface specification defines areas of implementation that have vendor-specific behaviors. The following sections describe Cray-specific implementations. 4.2.1 Pragmas 4.2.1.1 atomic Construct The atomic directive is replaced with a critical section that encloses the statement. 4.2.1.2 for Construct For the schedule(guided,chunk) clause, the size of the initial chunk for the master thread and other team members is approximately equal to the trip count divided by the number of threads. S–2179–80 95Cray C and C++ Reference Manual For the schedule(runtime) clause, the schedule type and chunk size can be chosen at run time by setting the OMP_SCHEDULE environment variable. If this environment variable is not set, the schedule type and chunk size default to static and 0, respectively. In the absence of the schedule clause, the default schedule is static and the default chunk size is approximately the number of iterations divided by the number of threads. 4.2.1.3 parallel Construct If a parallel region is encountered while dynamic adjustment of the number of threads is disabled, and the number of threads specified for the parallel region exceeds the number that the runtime system can supply, the program terminates. The number of physical processors actually hosting the threads at any given time is fixed at program startup and is specified by the aprun -d depth option. The OMP_NESTED environment variable and the omp_set_nested() call control nested parallelism. To enable nesting, set OMP_NESTED to true or use the omp_set_nested() call. Nesting is disabled by default. 4.2.1.4 private Clause If a variable is declared as private, the variable is referenced in the definition of a statement function, and the statement function is used within the lexical extent of the directive construct, then the statement function references the private version of the variable. 4.2.1.5 threadprivate Construct The threadprivate directive specifies that variables are replicated, with each thread having its own copy. If the dynamic threads mechanism is enabled, the definition and association status of a thread's copy of the variable is undefined, and the allocation status of an allocatable array is undefined. 4.2.2 OpenMP Library Routines 4.2.2.1 omp_get_max_active_levels() The Cray implementation of OpenMP supports the proposed OpenMP 3.0 omp_get_max_active_levels() routine to return the maximum number of nested parallel levels currently allowed. 96 S–2179–80Using OpenMP [4] 4.2.2.2 omp_set_dynamic() The omp_set_dynamic() routine enables or disables dynamic adjustment of the number of threads available for the execution of subsequent parallel regions by setting the value of the dyn-var ICV. The default is on. 4.2.2.3 omp_set_schedule() The omp_set_schedule() routine affects the schedule that is applied when runtime is used as schedule kind, by setting the value of the run-sched-var ICV. The default is on. 4.2.2.4 omp_set_max_active_levels() The Cray implementation of OpenMP supports the proposed OpenMP 3.0 omp_set_max_active_levels() routine to limit the depth of nested parallelism. The number specified controls the maximum number of nested parallel levels with more than one thread. The default value is 1 (nesting disabled). 4.2.2.5 omp_set_nested() The omp_set_nested() routine enables or disables nested parallelism, by setting the nest-var ICV. The default is false. 4.2.2.6 omp_set_num_threads() If dynamic adjustment of the number of threads is disabled, the number_of_threads_expr argument sets the number of threads for all subsequent parallel regions until this procedure is called again with a different value. 4.2.3 OpenMP Environment Variables 4.2.3.1 OMP_DYNAMIC The default value is true. 4.2.3.2 OMP_MAX_ACTIVE_LEVELS The default value is 1. 4.2.3.3 OMP_NESTED The default value is false. S–2179–80 97Cray C and C++ Reference Manual 4.2.3.4 OMP_NUM_THREADS If this environment variable is not set and you do not use the omp_set_num_threads() routine to set the number of OpenMP threads, the default is 1 thread. The maximum number of threads per compute node is 4 times the number of allocated processors. If the requested value of OMP_NUM_THREADS is more than the number of threads an implementation can support, the behavior of the program depends on the value of the OMP_DYNAMIC environment variable. If OMP_DYNAMIC is false, the program terminates. If OMP_DYNAMIC is true, it uses up to 4 times the number of allocated processors. For example, on a 8-core Cray XE system, this means the program can use up to 32 threads per compute node. 4.2.3.5 OMP_SCHEDULE The default values for this environment variable are static for type and 0 for chunk. 4.2.3.6 OMP_STACKSIZE The default value for this environment variable is 128 MB. 4.2.3.7 OMP_THREAD_LIMIT Sets the number of OpenMP threads to use for the entire OpenMP program by setting the thread-limit-var ICV. The Cray implementation defaults to 4 times the number of available processors. 4.2.3.8 OMP_WAIT_POLICY Provides a hint to an OpenMP implementation about the desired behavior of waiting threads by setting the wait-policy-var ICV. A compliant OpenMP implementation may or may not abide by the setting of the environment variable. The default value for this environment variable is active. 4.3 Compiler Options Affecting OpenMP These Cray C and C++ Compiler options affect OpenMP applications: • -h [no]omp (-h [no]omp on page 58) • -h threadn (-h threadn on page 35) 98 S–2179–80Using OpenMP [4] 4.4 OpenMP Program Execution The aprun -d option can be used to define the number of processors available to the application. If neither the OMP_NUM_THREADS environment variable nor the omp_set_num_threads() call is used to set the number of OpenMP threads, the system defaults to 1 thread. The maximum number of threads per compute node is 4 times the number of allocated processors. For further information, including example OpenMP programs, see the Cray Application Developer's Environment User's Guide. 4.5 OpenMP 3.1 Standard Support The following extensions, described in the OpenMP 3.1 standard specification, are supported: • Atomic construct extensions • taskyield construct • firstprivate clause now accepts intend(in) and constant objects. 4.6 OpenMP Accelerator Support To support accelerators, the Cray implementation of OpenMP extends the standard OpenMP programming model adding the concept of execution engines (accelerators) which are not members of the parallel team in the standard sense. Accelerators are managed by the runtime environment, but only available to the programmer through directives or calls to accelerator specific functions. The intro_openmp_acc(7) man page contains the most current information regarding OpenMP accelerator support. 4.6.1 OpenMP Accelerator Execution Model When a thread encounters an accelerator region, an explicit accelerator task is generated and execution of the task is assigned to an accelerator attached to the system. The runtime environment controls which accelerator to use. Completion of an accelerator task is guaranteed before the next barrier completes or when an accelerator task synchronization construct is encountered. All accelerator tasks are guaranteed to complete before the program exits. S–2179–80 99Cray C and C++ Reference Manual The compiler supports accelerator regions that contain a single level of accelerator loop. If there are loops within the accelerator loop, the compiler will attempt to automatically create a second level of parallelism using one of more of those loops. If a second level is found, the outer loop is partitioned across the thread blocks and the inner loop is partitioned across the threads within a single block. If a second level of parallelism is not found, the outer loops is partitioned across both the thread blocks and threads within the block. 4.6.2 OpenMP Accelerator Memory Model The OpenMP 3.1 specification describes a relaxed-consistency, shared-memory model in which threads have access to both shared and threadprivate memory. Directives which specify data-sharing attributes determine the type of access to variables used in the associated structured block: shared and private. In this description, memory located on the accelerator will be referred to as resident memory and memory located on the host CPU is referred to as host memory. An accelerator will have access to both private and shared accelerator resident objects. Host memory accesses may be accomplished via direct access or via data copying. If the host modifies or reads a memory location that is duplicated on the accelerator while the accelerator region is active, the result is unspecified, unless the threads accessing the location are explicitly synchronized. How data is moved between the host and the accelerator is unspecified however data motion directives will be provided as hints to the compiler. The result of global memory modifications made by either the host or the accelerator may not be visible until a flush is executed on both the host and the accelerator. Accelerators do not have access to memory equivalent to OpenMP threadprivate memory; all thread private data shall be marked as firstprivate. Accelerator resident shared objects may be updated and or copied back to the host whenever the accelerator is not modifying the data. 4.6.3 OpenMP Accelerator Directives Note that these accelerator directives are subject to change in future releases. See the intro_openmp_acc(7) man page for the most current information regarding this implementation. The following accelerator directives are supported in the current release : acc_region on page 101, acc_data on page 103, acc_loop on page 105, acc_region_loop on page 106, and acc_update on page 106. Clauses described as "Ignored" have no effect. Clauses described as "Not supported" will produce an error. 100 S–2179–80Using OpenMP [4] 4.6.3.1 acc_region When a thread encounters an acc_region directive, an accelerated task is created. This task can either be assigned to the default accelerator, or it can be assigned to the threads in a parallel team. If an acc_region is not placed on an accelerator because of lack of resources, and the construct is not already inside of a parallel region then a new parallel team will be constructed. If the acc_region is inside of a data environment, it shall be assigned to the same accelerator the acc_data is bound to. The compiler currently does not support function calls within an accelerator region. The compiler will attempt to inline any routines called within a region. If the calls cannot be inlined, the compiler will issue an error. The binding thread set for an acc_region is the encountering thread. The binding accelerator is the accelerator that is assigned the region by the runtime or the team of threads created to accelerate the region. Restrictions: A program should not branch into or out of the construct. A program should not depend on any ordering of the evaluations of the clauses. A program should not depend on any side effects of the evaluations of the clauses. #pragma omp acc_region [clause [[,] clause]...] Structured-block Clauses: async(handle[, handle]) (Deferred implementation) Causes the encountering thread to initiate execution of the region on the accelerator and then return to its own work. The handle provides a mechanism for the user to query the runtime about the state of the accelerated region. See OpenMP Accelerator Runtime Functions on page 107. Two or more constructs that use the same async_handle handle will be considered dependant on each other and will be executed in the order they are encountered. device(integer-expression[, integer-expression]) (Deferred implementation) The device clause causes the construct to be tied to the accelerator determined by the constant positive integer expression. The meaning of multiple device integer expressions is still undefined. if(scalar-expression | scalar-logical-expression) (Deferred implementation) If the clause evaluates to true, the region is accelerated. If the clause evaluates to false the region is run by the encountering thread. S–2179–80 101Cray C and C++ Reference Manual num_pes(depth:num[, depth:num ]) Controls the number of processing elements that are applied to a given level. The depth and num are integer expressions. If the expressions are constant the compiler may take advantage of the information and generate improved code. Two depths are supported. The first depth represents the number of threads per block and it must be a multiple of 32. The second depth represents the number of threads per block and must be a multiple of 32. If there is only a single layer of parallelism, the first depth number is treated as the total number of threads and the actual number of blocks is calculated accordingly. If a depth is given that is less than the current depth, it will be ignored. If a depth is given that is greater than the current depth. it will be preallocated or reserve resources for the depth when it is encountered. acc_shared(list) Supported. See clause definition in acc_data on page 103. acc_copy(list) Supported. See clause definition in acc_data on page 103. acc_copyin(list) Supported. See clause definition in acc_data on page 103. acc_copyout(list) Supported. See clause definition in acc_data on page 103. host_shared(list) Treated as an acc_copy. firstprivate(list) Causes all of the private versions of the list objects to be initialized to the state of the associated host shared object. private(list) Causes a unique copy of the objects in the list to be provided to each task on the accelerator. present(list|*) Supported. See clause definition in acc_data on page 103. 102 S–2179–80Using OpenMP [4] default(acc_shared|acc_copy|firstprivate|private|none|ignore) Specifies the default behavior for all objects that are used but not explicitly placed in a given memory type. If default( none ) is specified then all objects used inside the construct must be present on a data-sharing attribute clause list. If no default is provided the default is acc_copy. If default ( ignore ) is specified no default scoping will occur. Any objects that are not explicitly scoped must be in a present clause. 4.6.3.2 acc_data Starts an accelerator data region and defines a data scope that is reused across multiple accelerated constructs. When a thread encounters an acc_data construct, an accelerated task data environment is created. This task data environment is assigned to the default accelerator. The binding thread set for the region is the encountering thread. The binding accelerator is the accelerator that is assigned the region by the runtime or the team of threads created to accelerate the region. A program that branches into or out of the construct is non-conforming. A program should not depend on any ordering of the evaluations of the clauses. A program should not depend on any side effects of the evaluations of the clauses. Data Clause Restrictions: Only symbols can appear in data clauses. If the symbol is an array, the entire array is moved. If a non-contiguous array section is specified, the smallest contiguous bounding region is moved. Support for arrays and pointers in C and C++ has been implemented for single dimensional arrays an single level pointers. Non-contiguous array sections are not supported at this time. #pragma omp acc_data [clause [[,] clause]...] Structured-block Clauses: device(integer-expression[, integer-expression]) (Deferred implementation) Ignored. Only a single device is supported. S–2179–80 103Cray C and C++ Reference Manual acc_copy(list) The acc_copy clause causes the accelerator shared objects to be initialized to the host's memory state when the region starts and then causes the host's memory state to be updated with the accelerator memory state when the region ends. This combines the behavior of the acc_copyin and acc_copyout clauses. An object that appears in an acc_copy clause shall not appear in any other data sharing clauses. acc_copyin(list) Causes the accelerator shared objects to be initialized to the host's memory state when the region starts. Objects in this clause may also appear in the acc_copyout clause. acc_copyout(list) Causes the host's memory to be updated with the accelerators shared objects state when the accelerator region ends. Objects in this clause may also appear in the acc_copyin clause. host_shared(list) Treated as an acc_copy clause. acc_shared(list) Causes the objects in the list to be shared by all tasks executing on the associated accelerator. present(list|*) Supported with restrictions. Causes the system to look for the list items on the accelerator. If the list item also appears in an acc_shared, acc_copy, acc_copyin or acc_copyout clause, then if the object is not found, the copy clause takes effect. If the list item is not in another data placement clause it is an error to not find the object on the accelerator. The special * list is the same as listing all objects that appear in the other acc_shared, acc_copy, acc_copyin or acc_copyout clauses. Restrictions: Array sections of assumed size or allocatable arrays are not supported. The array will not be found in the present table. The C and C++ front-end reject legal conditional present clause combinations. This will be fixed in a future release. 104 S–2179–80Using OpenMP [4] default(acc_shared|host_shared|acc_copy|none|ignore) Specifies the default behavior for all objects that are used but not explicitly placed in a given memory type. If default( none ) is specified then all objects used inside the construct shall be present on a data-sharing attribute clause list. If no default is provided the default is copy. If default( ignore ) is specified no default scoping will occur. Any objects that are not explicitly scoped must be in a present clause. 4.6.3.3 acc_loop The accelerator loop construct specifies that the iterations of one or more associated loops will be executed in an accelerated manner. More than one level of acc_loop is allowed within an accelerator region. The associated loops must be structured blocks. The binding set for an acc_loop is the encountering accelerator. #pragma omp acc_loop [clause [[,] clause]...] For-loop-nest Clauses: cache(integer-expression[, integer-expression)] Deprecated. collapse(integer-expression) Causes the first integer-expression loops in the associated loop nest to be turned into a single loop incorporating the iteration space of the associated loops. The argument integer-expression is a positive constant integer expression and less than the depth of the loop-nest. All loops associated with the collapse clause shall be perfectly nested. hetero(list) Ignored. host(list) Ignored. kernel or innermost Only kernel is supported. The kernel clause causes all contained loops to be run sequentially. level(dimension) Ignored. S–2179–80 105Cray C and C++ Reference Manual max_par_level(expr) Ignored. num_pes(depth:num[, depth:num)] Ignored. reduction(operator:list) Specifies an operator and one or more list items. For each list item, a private copy is created in each thread, and is initialized appropriately for the operator. After the end of the region, the original list item is updated with the values of the private copies that are combined using the specified operator. See section 2.9.3.6 of the OpenMP 3.1 specification for clarification. schedule Ignored. seq Ignored. stripmine Ignored. vector Ignored. 4.6.3.4 acc_region_loop Uses the combined set of clauses available with the acc_region and acc_loop directives. #pragma omp acc_region_loop [clause [[,] clause]...] For-loop-nest 4.6.3.5 acc_update The acc_update directive is used within an explicit or implicit data region to update all or part of a host memory array with values from the corresponding array in accelerator memory, or to update all or part of an accelerator memory object with values from the corresponding object in host memory. 106 S–2179–80Using OpenMP [4] This directive may not appear inside of an accelerator region. The list items on the accelerator shall be visible in accelerator shared memory. This directive is executable. It shall not appear in place of the statement following an if, while, do, switch, or label in C, or in place of the statement following a logical if in Fortran. #pragma omp acc_update [clause [[,] clause]...] Clauses: host( obj1: [obj2][[,] obj1[: obj2]] ) This directive allows both the host and the accelerator to work independently and then to update each other with any visible changes before continuing. Causes the obj1 objects to be copied from the host memory to the accelerator memory, if the memories are distinct. If the optional obj2 object is provided then obj1 is the source object and obj2 is the destination object. This allows the programmer to move data between two objects that have different identifiers. See data clause restrictions. acc(obj1: [obj2][[,] obj1[: obj2]]) Causes the obj1 objects to be copied from the host memory to the accelerator memory, if the memories are distinct. If the optional obj2 object is provided then obj1 is the source object and obj2 is the destination object. See data clause restrictions. 4.6.4 OpenMP Accelerator Runtime Functions omp_acc_async_wait(handle) (Deferred implementation) Wait until the accelerator region associated with the handle to complete. omp_acc_async_test(handle) (Deferred implementation) Check to see if the accelerator region and data motion associated the handle have completed. If complete return true, else return false. 4.6.5 OpenMP Accelerator Environment Variables CRAY_ACC_DEBUG Write accelerator-related activity to stdout for debugging purposes. Valid output levels range from 0, which indicates no output, through 3, which indicates verbose. For parallel codes, every accelerator action on every core may generate output. If STDERR is piped to file, this file can be very large. Default: 0 S–2179–80 107Cray C and C++ Reference Manual 4.6.6 OpenMP Accelerator Module Support The craype-accel-nvidia20 module sets the necessary compiler options and targets. To compile, ensure that PrgEnv-cray module is loaded and that it includes CCE 8.0. Also, load the craype-accel-nvidia20 module. Use either the ftn or cc command to compile. 108 S–2179–80Using OpenACC [5] OpenACC is a parallel programming model which facilitates the use of an accelerator device attached to a host CPU. The OpenACC API allows the programmer to supplement information available to the compilers in order to offload code from a host CPU to an attached accelerator device. This release partially supports the OpenACC Application Programming Interface, Version 1.0 standard developed by PGI, Cray Inc., NVIDIA, with support from CAPS entreprise. Refer to the OpenACC home page at http://www.openacc-standard.org. Under the Downloads link, select the OpenACC 1.0 Specification. Note that the Cray Implementation of this API is subject to change in future releases. See the intro_openacc(7) man page for the most current information regarding this implementation of OpenACC. 5.1 OpenACC Execution Model The host offloads compute intensive regions to the accelerator device. The accelerator executes parallel regions. The host manages execution on the accelerator by allocating memory on the accelerator, initiating data transfer, sending code, passing arguments to the region, waiting for completion, transferring results back to the host and releasing memory. The accelerator on the Cray XK supports three levels of parallelism. 5.2 OpenACC Memory Model The memory on the accelerator is separate from host memory. Accelerator device memory is not mapped onto the host's virtual memory space. All data movement between host and accelerator memory is initiated by the host through the library functions that move data. Also, it is not assumed that the accelerator can access host memory, though it is supported by some devices. In this model, data movement between memories is managed by the compiler according to OpenACC directives. The programmer needs to be aware of device memory size, as well as memory bandwidth between host and device in order to effectively accelerate a region of code. S–2179–80 109Cray C and C++ Reference Manual Current GPUs implement a weak memory model; they do not support memory coherence between operations executed by different execution units. Execution unit describes an entity that executes a gang or worker. If an operation updates a memory location and another reads from the same location, or two operations store a value to the same location, the hardware may not guarantee repeatable results. Some potential errors of this type are prevented by the compiler, but it is possible to write an accelerator parallel region that produces inconsistent results. Memory coherence is guaranteed when memory operations are separated by an explicit barrier. 5.3 OpenACC Directives 5.3.1 parallel When the compiler encounters a parallel directive, it creates a parallel region to be executed on an attached accelerator. Gangs of workers are created to execute the parallel region on the accelerator. Once the gangs are created, the number of gangs and workers in each gang remain constant for the duration of the parallel region. One worker in each gang begins executing the code in the structured block of the construct. Each gang may execute a different path of statements. There is an implicit barrier at the end of the accelerator parallel region; the program running on the host will wait for gangs to complete execution before continuing. Restrictions: Parallel regions may not contain other parallel regions. A program should not branch into or out of the construct. A program should not depend on any ordering of the evaluations of the clauses. A program should not depend on any side effects of the evaluations of the clauses. At most, one if clause may appear. #pragma acc parallel [clause [[,] clause]...] Structured-block Clauses: if(condition) (Deferred implementation) If the clause evaluates to nonzero, the region runs on the accelerator. If the clause evaluates to zero, the region runs on the host. async[ (scalar-integer-expression) ] (Deferred implementation) num_gangs(scalar-integer-expression) integer-expression defines the number of parallel gangs that will execute the region. If not specified the default will be determined by the compiler. 110 S–2179–80Using OpenACC [5] num_workers(scalar-integer-expression) Defines the number of workers within each gang that will execute the region and scalar-integer-expression must be 1, 2, 4, 8, 16 or 32. If the user specifies num_workers, vector_length can only be 32. If the user does not specify num_workers, then vector_length can be 1, 32, 64, 128, 256, 512 or 1024. vector_length(scalar-integer-expression) integer-expression defines the vector length to use for vector or SIMD operations within each worker of the gang. vector_length is used for loops annotated with the vector clause on a loop directive, and for loop automatically vectorized by the compiler within the parallel region. If the user specifies num_workers, then vector_length can only be 32. If the user does not specify num_workers, then vector_length can be 1, 32, 64, 128, 256, 512 or 1024. reduction(operator:list) The reduction clause specifies a reduction operator and one or more scalar variables. For each variable, a private copy is created for each parallel gang and initialized for that operator. At the end of the region, the values for each gang are combined using the reduction operator, and the result combined with the value of the original variable and stored in the original variable. The reduction result is available after the region. The following table lists the operators that are valid and the initialization values; in each case, the initialization value will be cast into the variable type. For max and min reductions, the initialization values are the least representable value and the largest representable value for the variable’s data type, respectively. Supported data types are the numerical data types in C and C++ (int, float, double, complex) and Fortran (integer, real, double precision, complex). C, C++ Fortran operator init value operator init value + 0 + 0 * 1 * 1 max least max least min largest min largest & ~0 iand all bits on S–2179–80 111Cray C and C++ Reference Manual C, C++ Fortran operator init value operator init value | 0 ior 0 ^ 0 ieor 0 && 1 .and. .true. || 0 .or. .false. .eqv .true. .neqv .false. copy(list) See data clauses data on page 114. copyin(list) See data clauses data on page 114. copyout(list) See data clauses data on page 114. create(list) See data clauses data on page 114. present(list) See data clauses data on page 114. present_or_copy(list) See data clauses data on page 114. present_or_copyin(list) See data clauses data on page 114. present_or_copyout(list) See data clauses data on page 114. present_or_create(list) See data clauses data on page 114. deviceptr(list) See data clauses data on page 114. private(list) A copy of each item on the list will be created for each parallel gang. 112 S–2179–80Using OpenACC [5] firstprivate(list) The private versions of the items on the list will be initialized to the state of the associated item on the host. 5.3.2 loop The loop directive applies to the loop which immediately follows this directive. The directive can describe what type of parallelism to use to execute the loop and declare loop-private variables and arrays and reduction operations. #pragma acc loop [clause [[,] clause]...] For-loop-nest Clauses: collapse(scalar-integer-expression) Specifies that scalar-integer-expression nested loops are associated with the loop construct. scalar-integer-expression must be a constant positive integer expression. If more than one loop is associated with the loop construct, the iterations of all the associated loops are all scheduled according to the rest of the clauses. The trip count for all loops associated with the collapse clause must be computable and invariant in all the loops. A gang, worker, or vector clause on the directive may be applied to each loop or to the linearized iteration space as determined by the compiler. If no collapse clause is present, only the immediately following loop is associated with the loop directive. gang Specifies that the iterations of the associated loop or loops are to be executed in parallel by distributing the iterations among the gangs. The loop iterations must be data independent, except for variables specified in a reduction clause. worker Specifies that the iterations of the associated loop or loops are to be executed in parallel by distributing the iterations among the multiple workers within a single gang. The loop iterations must be data independent, except for variables specified in a reduction clause. It is implementation-defined whether a loop with the worker clause may contain a loop containing the gang clause. S–2179–80 113Cray C and C++ Reference Manual seq Specifies that the associated loop or loops are to be executed sequentially by the accelerator. Overrides any automatic compiler parallelization or vectorization. Default in context of a parallel region. vector Specifies that the iterations of the associated loop/s are to be executed in vector or SIMD mode. The operations will execute using vectors of the length specified or chosen for the parallel region. A loop with the vector clause may not contain a loop containing the gang or worker clause. private(list) A copy of each item on the list will be created for each iteration of the associated loop(s). reduction(operator:list) See parallel clauses parallel on page 110. 5.3.3 parallel_loop (Deferred implementation) The parallel_loop construct is a combined construct consisting of a parallel region and an accelerator loop which contains a single loop nest. This is identical to explicitly specifying a parallel directive containing a loop directive. It is currently supported in Fortran only. parallel_loop uses the combined set of clauses available with the parallel and loop directives. 5.3.4 data Starts an accelerator data region and defines scalars, arrays and subarrays to be allocated in the accelerator memory for the duration of the region. Clauses determine whether data is copied from the host to the accelerator memory upon region entry, and copied from the accelerator memory to the host memory upon region exit. Restrictions: A program should not branch into or out of the construct. A program should not depend on any ordering of the evaluations of the clauses. A program should not depend on any side effects of the evaluations of the clauses. At most, one if clause may appear. 114 S–2179–80Using OpenACC [5] Only symbols can appear in data clauses. If the symbol is an array, the entire array is moved. If a non-contiguous array section is specified, the smallest contiguous bounding region is moved. Support for arrays and pointers in C and C++ has been implemented for single dimensional arrays an single level pointers. Non-contiguous array sections are not supported at this time. #pragma acc data [clause [[,] clause]...] Structured-block Clauses: if(condition) (Deferred implementation) If the clause evaluates to nonzero in C or C++, the program allocates memory on, and moves data to and/or from the accelerator. If the clause evaluates to zero, no memory is allocated and no data is moved. async[ (scalar-integer-expression ) ] (Deferred implementation) copy(list) Combines the behavior of the copyin and copyout clauses. The copy clause causes the accelerator shared objects to be initialized to the host's memory state when the region starts and then causes the host's memory state to be updated with the accelerator memory state when the region ends. copyin(list) Allocate data in list on the accelerator and copy data from the host to the accelerator when entering the region. copyout(list) Allocate the data in list on the accelerator and copy the data from the accelerator to the host when exiting the region. create(list) Declares that the variables, arrays or subarrays in the list need to be allocated in the device memory, but the values in the host memory are not used on the accelerator, and any values computed and assigned on the accelerator are not used on the host. Data in this clause is not copied between the host and device memories. S–2179–80 115Cray C and C++ Reference Manual present(list) Causes the system to look for the list items on the accelerator. It is an error to not find the object on the accelerator. Array sections of assumed size or allocatable arrays are not supported. The array will not be found in the present table. present_or_copy(list) Causes the system to look for the list items on the accelerator. If the object is not found, the copy clause takes effect. Array sections of assumed size or allocatable arrays are not supported. The array will not be found in the present table. present_or_copyin(list) Causes the system to look for the list items on the accelerator. If the object is not found, the copyin clause takes effect. Array sections of assumed size or allocatable arrays are not supported. The array will not be found in the present table. present_or_copyout(list) Causes the system to look for the list items on the accelerator. If not found on the accelerator, the data is allocated in the accelerator memory and copied from the accelerator back to the host when the region exits. Array sections of assumed size or allocatable arrays are not supported. The array will not be found in the present table. present_or_create(list) Causes the system to look for the list items on the accelerator. If not found, the data is allocated in the accelerator. Array sections of assumed size or allocatable arrays are not supported. The array will not be found in the present table. deviceptr(list) Used to declare that the pointers in the list are device pointers, so the data need not be allocated or moved between the host and device for this pointer. In C and C++, the variables in list must be pointers. 116 S–2179–80Using OpenACC [5] 5.3.5 host_data Makes the device address of data available in host code. #pragma acc host_data [clause [[,] clause]...] Structured-block Clause: use_device(list) Use the device address of any variable or array in the list in code within the construct. May be used to pass the device address of variables or arrays to optimized procedures written in a lower-level API. The variables or arrays in list must be present in the accelerator memory due to data regions that contain this construct. 5.3.6 update The update directive is used within an explicit or implicit data region to update all or part of a host memory array with values from the corresponding array in accelerator memory, or to update all or part of an accelerator memory object with values from the corresponding object in host memory. This directive may not appear inside of a parallel region. The list items on the accelerator shall be visible in accelerator shared memory. This directive is executable. It shall not appear in place of the statement following an if, while, do, switch, or label in C, or in place of the statement following a logical if in Fortran. #pragma acc update [clause [[,] clause]...] Clauses: host(list) Specifies that the variables, arrays or subarrays in the list are to be copied from the accelerator device memory to the host memory. device(list) Specifies that the variables, arrays or subarrays in the list are to be copied from the host memory to the accelerator device memory. if(condition) (Deferred implementation) If the condition evaluates to nonzero the program moves data to/from the accelerator. If the clause evaluates to zero, no data is moved. async[ ( scalar-integer-expression ) ] (Deferred implementation) S–2179–80 117Cray C and C++ Reference Manual 5.4 OpenACC Runtime Routines The current release does not support OpenACC runtime routines. 5.5 OpenACC Environment Variables The current release does not support OpenACC environment variables. 5.6 Compiler Options These C/C++ compiler options affect OpenACC directives. -h [no]acc Enables or disables compiler recognition of OpenACC accelerator directives. By default, OpenACC is enabled. -h nopragma=acc Enables or disables compiler recognition of OpenACC accelerator directives. By default, OpenACC is enabled. 118 S–2179–80Using Cray Unified Parallel C (UPC) [6] Unified Parallel C (UPC) is a C language extension for parallel program development. UPC supports a Partitioned Global Address Space (PGAS) programming model. Cray supports the UPC Language Specification 1.2 and also supports some Cray specific functions as noted in the following sections. For more information about UPC functions, see intro_pgas(7), or refer to the appropriate UPC man page. For a description of the -h upc command line option, see Compiling and Linking UPC Code on page 120. You should be familiar with UPC and understand the differences between the published UPC Introduction and Language Specification paper and the current UPC specification. If you are not familiar with UPC, refer to the UPC home page at http://upc.gwu.edu/. Under the Publications link, select the Introduction to UPC and Language Specification paper. This paper is slightly outdated but contains valuable information about understanding and using UPC. The UPC home page also contains, under the Documentation link, the UPC Language Specification 1.2 paper. UPC allows you to explicitly specify parallel programming through language syntax rather than library functions such as those used in MPI and SHMEM by allowing you to read and write memory of other processes with simple assignment statements. Program synchronization occurs only when explicitly programmed; there is no implied synchronization. Note: UPC is a dialect of the C language. It is not available in C++. UPC allows you to maintain a view of your program as a collection of threads operating in a common global address space without burdening you with details of how parallelism is implemented on the machine (for example, as shared memory or as a collection of physically distributed memories). UPC data objects are private to a single thread or shared among all threads of execution. Each thread has a unique memory space that holds its private data objects, and access to a globally-shared memory space that is distributed across the threads. Thus, every part of a shared data object has an affinity to a single thread. S–2179–80 119Cray C and C++ Reference Manual Cray UPC is compatible with MPI and SHMEM. Note: UPC 1.2 supports a parallel I/O model which provides control over file synchronization. However, if you continue to use the regular C I/O routines, you must supply the controls as needed to remove race conditions. File I/O under UPC is very similar to standard C because one thread opens a file and shares the file handle, and multiple threads may read or write to the same file. 6.1 Cray Implementation Differences There is a false sharing hazard when referencing shared char and short integers. If two PEs store a char or short to the same 64-bit word in memory without synchronization, incorrect results can occur. It is possible for one PE's store to be lost. This is because these stores are implemented by reading the entire 64-bit word, inserting the char or short value and writing the entire word back to memory. The following output is a result of two PEs writing two different characters into the same word in memory without synchronization: Register Memory Initial Value 0x0000 PE 0 Reads 0x0000 0x0000 PE 1 Reads 0x0000 0x0000 PE 0 Inserts 3 0x3000 0x0000 PE 1 Inserts 7 0x0700 0x0000 PE 0 Writes 0x3000 0x3000 PE 1 Writes 0x0700 0x0700 Notice that the value stored by PE 0 has been lost. The final value intended was 0x3700. This situation is referred to as false sharing. It is the result of supporting data types that are smaller than the smallest type that can be individually read or written by the hardware. UPC programmers must take care when storing to shared char and short data that this situation does not occur. 6.2 Compiling and Linking UPC Code Compiling a PGAS application (UPC, Fortran 2008) requires the PrgEnv-cray module to be loaded. The -hupc option is required to enable recognition of UPC syntax because it is not part of the standard C language. The -X npes option can optionally be used to define the number of threads to use and statically set the value of the THREADS constant. See -X npes on page 61 for requirements regarding the use of the -X npes option. The following command creates an executable file: % cc -hupc hello.c -o hello 120 S–2179–80Using Cray Unified Parallel C (UPC) [6] An executable can be created by linking together various object files that were generated from source code written in standard C, UPC, and Fortran. Either cc or ftn can be used to link the object files: % cc -hupc x.o y.o z.o % ftn -hcaf x.o y.o z.o For dynamic linking, add the -dynamic option. For information about linking PGAS applications to use huge pages, see the intro_hugepages(1) man page. 6.3 Launching a UPC Application After compiling the UPC code, you run the program using the aprun command. Launch the application using 128 PEs: % aprun -n 128 ./hello If you use the –X npes compiler option, you must specify the same number of threads in the aprun command. The processing elements specified by npes are compute node cores/PEs. By default, each PE reserves 64 MB of symmetric heap space. To increase or decrease this amount, set the XT_SYMMETRIC_HEAP_SIZE environment variable to the desired number of bytes. The suffixes K, M, and G are permitted to simplify requests for large values: % export XT_SYMMETRIC_HEAP_SIZE=512M % aprun -n 128 ./hello The Cray XE UPC run time system uses GNI and DMAPP (low level libraries) to implement a logically shared, distributed memory programming model. The symmetric heap is mapped onto hugepages by DMAPP. It is advisable to also map the static data and/or private heap onto huge pages. See the intro_hugepages(1) man page. The Cray XT UPC run time system uses GASNet, which has a default shared segment size of 2 GB. If your program requires a total amount of shared memory (static plus dynamic) that is near or more than 2 GB per PE, then you may encounter a GASNet limit. This default may be inappropriate on nodes with 4 GB per PE, for example. The workaround is to set GASNET_MAX_SEGSIZE=4GB. 6.4 Cray Extensions Defined in cray_upc.h Cray supports extensions to UPC that are not part of the UPC Language Specification, version 1.2. Include upc_cray.h to use these extensions. S–2179–80 121Cray C and C++ Reference Manual 6.4.1 Collective Deallocation The UPC library includes collective allocation routines, but does not provide the analogous collective deallocation routines. These extensions permit collective deallocation. 6.4.1.1 upc_all_free() Deallocates memory allocated by the upc_all_alloc() function. 6.4.1.2 upc_all_lock_free() Deallocates all resources allocated by the upc_all_lock_alloc() function. 6.4.2 Node Affinity 6.4.2.1 upc_nodeof() Returns the index of the node of the thread that has affinity to the shared object pointed to by ptr. Similar to upc_threadof(). 6.4.2.2 NODES NODES is an expression with a value of type int; it specifies the number of nodes and has the same value on every thread in the job. 6.4.2.3 MYNODE MYNODE is an expression with a value of type int; it specifies the unique node index associated with the current thread and has the same value on all threads that are located on the same node. 6.4.3 Privatizability (Castability) Functions These functions assist with casting a pointer-to-shared to a normal pointer when the target is located on the same node. 6.4.3.1 upc_cast() Converts a pointer-to-shared to a pointer-to-local if possible. If not possible, a null pointer is returned. Pointers to data with affinity to the current thread always succeed. Null pointers-to-shared yield null pointers-to-local. 6.4.3.2 upc_castable() Returns TRUE if the pointer can be converted to a pointer-to-local, FALSE otherwise. Null pointers and pointers to data with affinity to the current thread return TRUE. 122 S–2179–80Using Cray Unified Parallel C (UPC) [6] 6.4.3.3 upc_thread_castable() Returns TRUE if all of the specified thread's shared data may be referenced via local addresses, FALSE otherwise. In the event that some but not all of the thread's data may be so referenced, returns FALSE. If threadnum == MYTHREAD, returns TRUE. This function allows for one castability check prior to a large number of references. 6.4.4 Shared String Handling Functions, Non-blocking Versions 6.4.4.1 upc_memcpy_nb(), upc_memcpy_nbi() Copies n characters from a shared object having affinity with one thread to a shared object having affinity with the same or another thread. 6.4.4.2 upc_memput_nb(), upc_memput_nbi() Copies n characters from an object on the calling thread to a shared object with affinity to any single thread. 6.4.4.3 upc_memget_nb(), upc_memget_nbi() Copies n characters from a shared object with affinity to any single thread to an object on the calling thread. 6.4.4.4 upc_sync_nb(), upc_test_nb() Synchronizes or tests completion status of non-blocking, memory copy functions, upc_memput_nb(), upc_memput_nb(), and upc_memput_nb(). 6.4.5 Timing These routines permit timing regions of UPC code. • upc_ticks_now() • upc_ticks_to_ns() S–2179–80 123Cray C and C++ Reference Manual 124 S–2179–80Using Cray C++ Libraries [7] Most of the standard C++ features are supported, except for the few mentioned in Unsupported Standard C++ Library Features. For information about C++ language conformance and exceptions, see Appendix B, Using Cray C and C++ Dialects on page 167. 7.1 Unsupported Standard C++ Library Features The Cray C++ compiler supports the C++ standard except for wide characters and multiple locales as follows: • String classes using basic string class templates with wide character types or that use the wstring standard template class • I/O streams using wide character objects • File-based streams using file streams with wide character types (wfilebuf, wifstream, wofstream, and wfstream) • Multiple localization libraries; Cray C++ supports only one locale Note: The C++ standard provides a standard naming convention for library routines. Therefore, classes or routines that use wide characters are named appropriately. For example, the fscanf and sprintf functions do not use wide characters, but the fwscanf and swprintf function do. S–2179–80 125Cray C and C++ Reference Manual 126 S–2179–80Using Cray C Extensions [8] 8.1 Complex Data Extensions Cray C extends the complex data facilities defined by standard C with these extensions: • Imaginary constants • Incrementing or decrementing _Complex data The Cray C compiler supports the Cray imaginary constant extension and is defined in the header file. This imaginary constant has the following form: Ri R is either a floating constant or an integer constant; no space or other character can appear between R and i. If you are compiling in strict conformance mode (-h conform), the Cray imaginary constants are not available. The following example illustrates imaginary constants: #include double complex z1 = 1.2 + 3.4i; double complex z2 = 5i; The other extension to the complex data facility allows the prefix– and postfixincrement and decrement operators to be applied to the _Complex data type. The operations affect only the real portion of a complex number. 8.2 fortran Keyword In extended mode, the identifier fortran is treated as a keyword. It specifies a storage class that can be used to declare a Fortran-coded external function. The use of the fortran keyword when declaring a function causes the compiler to verify that the arguments used in each call to the function are pass by addresses; any arguments that are not addresses are converted to addresses. As in any function declaration, an optional type-specifier declares the type returned, if any. Type int is the default; type void can be used if no value is returned (by a Fortran subroutine). The fortran storage class causes conversion of lowercase function names to uppercase, and, if the function name ends with an underscore character, the trailing underscore character is stripped from the function name. (Stripping the trailing underscore character is in keeping with UNIX practice.) S–2179–80 127Cray C and C++ Reference Manual Functions specified with a fortran storage class must not be declared elsewhere in the file with a static storage class. Note: The fortran keyword is not allowed in Cray C++. An example using the fortran keyword is shown in Cray C and Fortran Example on page 149. 8.3 Hexadecimal Floating-point Constants The Cray C compiler supports the standard hexadecimal floating constant notations and the Cray hexadecimal floating constant notation. The standard hexadecimal floating constants are portable and have sizes that are dependent upon the hardware. The remainder of this section discusses the Cray hexadecimal floating constant. The Cray hexadecimal floating constant feature is not portable, because identical hexadecimal floating constants can have different meanings on different systems. It can be used whenever traditional floating-point constants are allowed. The hexadecimal constant has the usual syntax: 0x (or 0X) followed by hexadecimal characters. The optional floating suffix has the same form as for normal floating constants: f or F (for float), l or L (for long), optionally followed by an i (imaginary). The constant must represent the same number of bits as its type, which is determined by the suffix (or the default of double). The constant's bit length is four times the number of hexadecimal digits, including leading zeros. The following example illustrates hexadecimal constant representation: 0x7f7fffff.f 32-bit float 0x0123456789012345. 64-bit double The value of a hexadecimal floating constant is interpreted as a value in the specified floating type. This uses an unsigned integral type of the same size as the floating type, regardless of whether an object can be explicitly declared with such a type. No conversion or range checking is performed. The resulting floating value is defined in the same way as the result of accessing a member of floating type in a union after a value has been stored in a different member of integral type. 128 S–2179–80Using Cray C Extensions [8] The following example illustrates hexadecimal floating-point constant representation that use Cray floating-point format: int main(void) { float f1, f2; double g1, g2; f1 = 0x3ec00000.f; f2 = 0x3fc00000.f; g1 = 0x40fa400100000000.; g2 = 0x40fa400200000000.; printf("f1 = %8.8g\n", f1); printf("f2 = %8.8g\n", f2); printf("g1 = %16.16g\n", g1); printf("g2 = %16.16g\n", g2); return 1; } This is the output for the previous example: f1 = 0.375 f2 = 1.5 g1 = 107520.0625 g2 = 107520.125 S–2179–80 129Cray C and C++ Reference Manual 130 S–2179–80Using Predefined Macros [9] The macros listed in this chapter are the Cray-specific predefined macros. To see the entire list of predefined macros, add -Wp,-list_final_macros -E to your cc command line. For example, if you have the file c.c, specify: % cc -Wp,-list_final_macros -E c.c > out Predefined macros can be divided into the following categories: • Macros required by the C and C++ standards (Macros Required by the C and C++ Standards on page 132) • Macros based on the host machine (Macros Based on the Host Machine on page 132) • Macros based on the target machine (Macros Based on the Target Machine on page 133) • Macros based on the compiler (Macros Based on the Compiler on page 133) • UPC macros (UPC Predefined Macros on page 134) Predefined macros provide information about the compilation environment. In this chapter, only those macros that begin with the underscore (_) character are defined when running in strict-conformance mode. Note: Any of the predefined macros except those required by the standard (see Macros Required by the C and C++ Standards on page 132) can be undefined by using the -U command line option; they can also be redefined by using the -D command line option. A large set of macros is also defined in the standard header files. S–2179–80 131Cray C and C++ Reference Manual 9.1 Macros Required by the C and C++ Standards The following macros are required by the C and C++ standards: Macro Description __TIME__ Time of translation of the source file. __DATE__ Date of translation of the source file. __LINE__ Line number of the current line in your source file. __FILE__ Name of the source file being compiled. __STDC__ Defined as the decimal constant 1 if compilation is in strict conformance mode; defined as the decimal constant 2 if the compilation is in extended mode. This macro is defined for Cray C and C++ compilations. __cplusplus Defined as 1 when the compiling Cray C++ code and undefined when compiling Cray C code. The __cplusplus macro is required by the ISO C++ standard, but not the ISO C standard. 9.2 Macros Based on the Host Machine The following macros provide information about the environment running on the host machine: Macro Description __linux Defined as 1. __linux__ Defined as 1. linux Defined as 1. __gnu_linux__ Defined as 1. 132 S–2179–80Using Predefined Macros [9] 9.3 Macros Based on the Target Machine The following macros provide information about the characteristics of the target machine: Macro x86 AVX _ADDR64 Defined as 1 if the targeted CPU has 64-bit address registers; if the targeted CPU does not have 64-bit address registers, the macro is not defined. __LITTLE_ENDIAN__ Defined as 1. Cray XE and Cray XK systems use little endian byte ordering. _LITTLE_ENDIAN Defined as 1. Cray XE and Cray XK systems use little endian byte ordering. _MAXVL_8 Defined as 16, the number of 8-bit elements that fit in an XMM register ("vector length"). Defined as 32, the number of 8-bit elements that fit in a YMM register ("vector length"). _MAXVL_16 Defined as 8. Defined as 16. _MAXVL_32 Defined as 4. Defined as 8. _MAXVL_64 Defined as 2. Defined as 4. _MAXVL_128 Defined as 0. Defined as 2. 9.4 Macros Based on the Compiler The following macros provide information about compiler features: Macro Description _RELEASE Defined as the major release level of the compiler. _RELEASE_MINOR Defined as the minor release level of the compiler. _RELEASE_STRING Defined as a string that describes the version of the compiler. _CRAYC Defined as 1 to identify the Cray C and C++ compilers. S–2179–80 133Cray C and C++ Reference Manual 9.5 UPC Predefined Macros The following macros provide information about UPC functions: Macro Description __UPC__ The integer constant 1, indicating a conforming implementation. __UPC_DYNAMIC_THREADS__ The integer constant 1 in the dynamic THREADS translation environment. __UPC_STATIC_THREADS__ The integer constant 1 in the static THREADS translation environment. 134 S–2179–80Running C and C++ Applications [10] To run applications, log in to a login node and set up your user environment. See the Cray Application Developer's Environment User's Guide for details on setting up your environment. In your working directory, load the appropriate modules, compile your programs, and launch them using the aprun command. Note: You need to be in a Lustre-mounted directory, such as /lus/nid00007/user1/myxtapps, before using the aprun command. To use the Cray C compiler, load the PrgEnv-cray module. Use the module list command to get a list of currently loaded modules. If another Programming Environment module is loaded, use the module swap command. For example, if PrgEnv-pgi is loaded, use this command: % module swap PrgEnv-pgi PrgEnv-cray Then use the cc -V command to verify that the Cray C compiler is available: % cc -V /opt/cray/xt-asyncpe/2.5/bin/cc: INFO: native target is being used Cray C++ : Version 7.1.0.129 Thu May 21, 2009 14:37:25 Compile and run your application. % cc -o simple simple.c % aprun -n 4 ./simple | sort Application 1024906 resources: utime 0, stime 0 hello from pe 0 of 4 hello from pe 1 of 4 hello from pe 2 of 4 hello from pe 3 of 4 If you specified the -X option on the cc command line, then the aprun -n option must specify the same number of processing elements (npes). Otherwise, you will receive a run time error. For additional information, see the Cray Application Developer's Environment User's Guide. S–2179–80 135Cray C and C++ Reference Manual 136 S–2179–80Debugging Cray C and C++ Code [11] The TotalView symbolic debugger is available to help you debug C and C++ codes (see TotalView Users Guide). In addition, the Cray C and C++ compilers provide the following features to help you in debugging codes: • The -G and -g compiler options provide symbol information about your source code for use by the TotalView debugger. For more information about these compiler options, see -G level and -g on page 46. • The -h [no]bounds option and the #pragma _CRI [no]bounds directive let you check pointer and array references. The -h [no]bounds option is described in -h [no]bounds (cc) on page 47. The #pragma _CRI [no]bounds directive is described in [no]bounds Directive on page 68. • The fast option optimizes code for use with Cray fast-track debugging and requires use of a debugger that supports fast-track debugging. For more information, see the lgdb(1) man page. • The #pragma _CRI message directive lets you add warning messages to sections of code where you suspect problems. The #pragma _CRI message directive is described in message Directive on page 70. • The #pragma _CRI [no]opt directive lets you selectively isolate portions of your code to optimize, or to toggle optimization on and off in selected portions of your code. The #pragma _CRI [no]opt directive is described in [no]opt Directive on page 72. S–2179–80 137Cray C and C++ Reference Manual 11.1 TotalView Debugger Some of the functions available in the TotalView debugger allow you to perform the following actions: • Set and clear breakpoints, which can be conditional, at both the source code level and the assembly code level • Examine core files • Step through a program, including across function calls • Reattach to the executable file after editing and recompiling • Edit values of variables and memory locations • Evaluate code fragments 11.2 Compiler Debugging Options Compiler options control the trade-offs between ease of debugging and compiler optimizations. The compiler produces internal debugger information (DWARF) at all times. The DWARF data provides function and line information to debuggers for tracebacks and breakpoints, as well as type and location information about data variables. These options are specified as follows: • -Gf With no DWARF, the executable is optimized and as small as possible, but cannot be easily debugged. Only assembly instructions will be visible and only global symbols will be available. • -Gp With partial DWARF and at least some optimization, tracebacks and limited breakpoints are available in the debugger. The source code will be visible and many more symbols will be available. The executable will be somewhat slower and larger in exchange for increased debugger functionality. • -g or -Gn With full DWARF and no optimizations, full debugging will be available, but at the cost of a slower and larger executable. Note: The -g or -G options may be specified on a per file basis so that only part of an application incurs the overhead of improved debugging. 138 S–2179–80Debugging Cray C and C++ Code [11] However, consider the following cases in which optimization is affected by the -Gp and -Gf debugging options: • Vectorization can be inhibited if a label exists within the vectorizable loop. • Vectorization can be inhibited if the loop contains a nested block and the -Gp option is specified. • When the -Gp option is specified, setting a breakpoint at the first statement in a vectorized loop allows you to stop and display at each vector iteration. However, setting a breakpoint at the first statement in an unrolled loop may not allow you to stop at each vector iteration. S–2179–80 139Cray C and C++ Reference Manual 140 S–2179–80Using Interlanguage Communication [12] The C and C++ compilers provide mechanisms for declaring external functions written in other languages. This enables you to write portions of an application in C, C++, Fortran, or assembly language, which can be useful in cases where the other languages provide performance advantages or utilities not available in C or C++. The calling sequence is described in detail on the callseq(3) man page. 12.1 Calls Between C and C++ Functions The following requirements apply when making calls between functions written in C and C++: • In Cray C++, the extern "C" linkage is required when declaring an external function that is written in Cray C or when declaring a Cray C++ function that is to be called from Cray C. Normally the compiler mangles function names to encode information about the function's prototype in the external name; this prevents direct access to these function names from a C function. The extern "C" keyword prevents the compiler from performing name mangling. • The program must be linked using the CC command. • The program's main routine must be C or C++ code compiled using the CC command. Objects can be shared between C and C++. There are some Cray C++ objects that are not accessible to Cray C functions (such as classes). The following object types can be shared directly: • Integral and floating types. • Structures and unions that are declared identically in C and C++. In order for structures and unions to be shared, they must be declared with identical members in the identical order. • Arrays and pointers to the above types. S–2179–80 141Cray C and C++ Reference Manual In the following example, a Cray C function (C_add_func) is called by the Cray C++ main program: #include extern "C" int C_add_func(int, int); int global_int = 123; main() { int res, i; cout << "Start C++ main" << endl; /* Call C function to add two integers and return result. */ cout << "Call C C_add_func" << endl; res = C_add_func(10, 20); cout << "Result of C_add_func = " << res << endl; cout << "End C++ main << endl; } The Cray C function (C_add_func) is as follows: #include extern int global_int; int C_add_func(int p1, int p2) { printf("\tStart C function C_add_func.\n"); printf("\t\tp1 = %d\n", p1); printf("\t\tp2 = %d\n", p2); printf("\t\tglobal_int = %d\n", global_int); return p1 + p2; } The output from the execution of the calling sequence illustrated in the preceding example is as follows: Start C++ main Call C C_add_func Start C function C_add_func. p1 = 10 p2 = 20 global_int = 123 Result of C_add_func = 30 End C++ main 142 S–2179–80Using Interlanguage Communication [12] 12.2 Calling Fortran Functions and Subroutines from C or C++ The following standard considerations apply when calling Fortran functions from C or C++. In addition, new interoperability features are supported under the more recent Fortran standards. These newer features are described in Calling a C or C++ Function from Fortran on page 151. 12.2.1 Requirements • Fortran uses the call-by-address convention. C and C++ use the call-by-value convention, which means that only pointers should be passed to Fortran subprograms. For more information, see Argument Passing on page 143. • Fortran arrays are in column-major order. C and C++ arrays are in row-major order. This indicates which dimension is indicated by the first value in an array element subscript. For more information, see Array Storage on page 144. • Single-dimension arrays of signed 32-bit integers and single-dimension arrays of 32-bit floating-point numbers are the only aggregates that can be passed as parameters without changing the arrays. • Fortran character pointers and character pointers from Cray C and C++ are incompatible. For more information, see Logical and Character Data on page 145. • Fortran logical values and the Boolean values from C and C++ are not fully compatible. For more information, see Logical and Character Data on page 145. • External C and C++ variables are stored in common blocks of the same name, making them readily accessible from Fortran programs if the C or C++ variable is in uppercase. • When declaring Fortran functions or objects in C or C++, the name must be specified in all uppercase letters, digits, or underscore characters and consist of 31 or fewer characters. • In Cray C, Fortran functions can be declared using the fortran keyword (see fortran Keyword on page 127). The fortran keyword is not available in Cray C++. Instead, Fortran functions must be declared by specifying extern "C". 12.2.2 Argument Passing Because Fortran subroutines expect arguments to be passed by pointers rather than by value, C and C++ functions called from Fortran subroutines must pass pointers rather than values. S–2179–80 143Cray C and C++ Reference Manual All argument passing in Cray C is strictly by value. To prepare for a function call between two Cray C functions, a copy is made of each actual argument. A function can change the values of its formal parameters, but these changes cannot affect the values of the actual arguments. It is possible, however, to pass a pointer. (All array arguments are passed by this method.) This capability is analogous to the Fortran method of passing arguments. In addition to passing by value, Cray C++ also provides passing by reference. 12.2.3 Array Storage C and C++ arrays are stored in memory in row-major order. Fortran arrays are stored in memory in column-major order. For example, the C or C++ array declaration int A[3][2] is stored in memory as: A[0][0] A[0][1] A[1][0] A[1][1] A[2][0] A[2][1] The previously defined array is viewed linearly in memory as: A[0][0] A[0][1] A[1][0] A[1][1] A[2][0] A[2][1] The Fortran array declaration INTEGER A(3,2) is stored in memory as: A(1,1) A(2,1) A(3,1) A(1,2) A(2,2) A(3,2) The previously defined array is viewed linearly in memory as: A(1,1) A(2,1) A(3,1) A(1,2) A(2,2) A(3,2) When an array is shared between Cray C, C++, and Fortran, its dimensions are declared and referenced in C and C++ in the opposite order in which they are declared and referenced in Fortran. Arrays are zero-based in C and C++ and are one-based in Fortran, so in C and C++ you should subtract 1 from the array subscripts that you would normally use in Fortran. For example, using the Fortran declaration of array A in the preceding example, the equivalent declaration in C or C++ is: int a[2][3]; 144 S–2179–80Using Interlanguage Communication [12] The following list shows how to access elements of the array from Fortran and from C or C++: Fortran C or C++ A(1,1) A[0][0] A(2,1) A[0][1] A(3,1) A[0][2] A(1,2) A[1][0] A(2,2) A[1][1] A(3,2) A[1][2] 12.2.4 Logical and Character Data Logical and character data need special treatment for calls between C or C++ and Fortran. Fortran has a character descriptor that is incompatible with a character pointer in C and C++. The techniques used to represent logical (Boolean) values also differ between Cray C, C++, and Fortran. Mechanisms you can use to convert one type to the other are provided by the fortran.h header file and conversion macros shown in the following list: Macro Description _btol Conversion utility that converts a 0 to a Fortran logical .FALSE. and a nonzero value to a Fortran logical .TRUE. _ltob Conversion utility that converts a Fortran logical .FALSE. to a 0 and a Fortran logical .TRUE. to a 1. 12.2.5 Accessing Named Common from C and C++ The following example demonstrates how external C and C++ variables are accessible in Fortran named common blocks. It shows a C or C++ function calling a Fortran subprogram, the associated Fortran subprogram, and the associated input and output. In this example, the C or C++ structure _st is accessed in the Fortran subprogram as common block ST. The Fortran common block ST will be converted to lower case with a trailing underscore added. The name of the structure and the converted Fortran common block name must match. The C and C++ structure member names and the Fortran common block member names do not have to match, as is shown in this example. S–2179–80 145Cray C and C++ Reference Manual The following Cray C main program calls the Fortran subprogram FCTN: #include struct { int i; double a[10]; long double d; } _st; main() { int i; /* initialize struct _st */ _st.I = 12345; for (i = 0; i < 10; i++) _st.a[i] = i; _st.d = 1234567890.1234567890L; /* print out the members of struct _st */ printf("In C: _st.i = %d, _st.d = %20.10Lf\n", _st.i, _st.d); printf("In C: _st.a = "); for (i = 0; i < 10; i++) printf("%4.1f", _st.a[i]); printf("\n\n"); /* call the fortran function */ FCTN(); } The following example is the Fortran subprogram FCTN called by the previous Cray C main program: C *********** Fortran subprogram (f.f): *********** SUBROUTINE FCTN COMMON /ST/STI, STA(10), STD INTEGER STI REAL STA DOUBLE PRECISION STD INTEGER I WRITE(6,100) STI, STD 100 FORMAT ('IN FORTRAN: STI = ', I5, ', STD = ', D25.20) WRITE(6,200) (STA(I), I = 1,10) 200 FORMAT ('IN FORTRAN: STA =', 10F4.1) END 146 S–2179–80Using Interlanguage Communication [12] The previous Cray C and Fortran examples are executed by the following commands, and they produce the output shown: % cc -c c.c % ftn -c f.f % ftn c.o f.o % ./a.out ST.i = 12345, ST.d = 1234567890.1234567890 In C: ST.a = 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 IN FORTRAN: STI = 12345, STD = .12345678901234567889D+10 IN FORTRAN: STA = 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 12.2.6 Accessing Blank Common from C or C++ Fortran includes the concept of a common block. A common block is an area of memory that can be referenced by any program unit in a program. A named common block has a name specified in names of variables or arrays stored in the block. A blank common block, sometimes referred to as blank common, is declared in the same way, but without a name. There is no way to access blank common from C or C++ similar to accessing a named common block. However, you can write a simple Fortran function to return the address of the first word in blank common to the C or C++ program and then use that as a pointer value to access blank common. S–2179–80 147Cray C and C++ Reference Manual The following example shows how Fortran blank common can be accessed using C or C++ source code: #include struct st { float a; float b[10]; } *ST; #ifdef __cplusplus extern "C" struct st *MYCOMMON(void); extern "C" void FCTN(void); #else fortran struct st *MYCOMMON(void); fortran void FCTN(void); #endif main() { int i; ST = MYCOMMON(); ST->a = 1.0; for (i = 0; i < 10; i++) ST->b[i] = i+2; printf("\n In C and C++\n"); printf(" a = %5.1f\n", ST->a); printf(" b = "); for (i = 0; i < 10; i++) printf("%5.1f ", ST->b[i]); printf("\n\n"); FCTN(); } This Fortran source code accesses blank common and is accessed from the C or C++ source code in the preceding example: SUBROUTINE FCTN COMMON // STA,STB(10) PRINT *, "IN FORTRAN" PRINT *, " STA = ",STA PRINT *, " STB = ",STB STOP END FUNCTION MYCOMMON() COMMON // A MYCOMMON = LOC(A) RETURN END This is the output of the previous C or C++ source code: a = 1.0 b = 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 11.0 148 S–2179–80Using Interlanguage Communication [12] This is the output of the previous Fortran source code: STA = 1. STB = 2., 3., 4., 5., 6., 7., 8., 9., 10., 11. 12.2.7 Cray C and Fortran Example Here is an example of a Cray C function that calls a Fortran subprogram. The Fortran subprogram example follows the Cray C function example, and the input and output from this sequence follows the Fortran subprogram example. Note: This example assumes that the Cray Fortran function is compiled with the -s default32 option enabled. The examples will not work if the -s default64 option is enabled. /* C program (main.c): */ #include #include #include /* Declare prototype of the Fortran function. Note the last */ /* argument passes the length of the first argument. */ fortran double FTNFCTN (char *, int *, int); double FLOAT1 = 1.6; double FLOAT2; /* Initialized in FTNFCTN */ main() { int clogical, ftnlogical, cstringlen; double rtnval; char *cstring = "C Character String"; /* Convert clogical to its Fortran equivalent */ clogical = 1; ftnlogical = _btol(clogical); /* Print values of variables before call to Fortran function */ printf(" In main: FLOAT1 = %g; FLOAT2 = %g\n", FLOAT1, FLOAT2); printf(" Calling FTNFCTN with arguments:\n"); printf(" string = \"%s\"; logical = %d\n\n", cstring, clogical); cstringlen = strlen(cstring); rtnval = FTNFCTN(cstring, &ftnlogical, cstringlen); /* Convert ftnlogical to its C equivalent */ clogical = _ltob(&ftnlogical); /* Print values of variables after call to Fortran function */ printf(" Back in main: FTNFCTN returned %g\n", rtnval); printf(" and changed the two arguments:\n"); printf(" string = \"%.*s\"; logical = %d\n", cstringlen, cstring, clogical); } C Fortran subprogram (ftnfctn.f): FUNCTION FTNFCTN(STR, LOG) S–2179–80 149Cray C and C++ Reference Manual REAL FTNFCTN CHARACTER*(*) STR LOGICAL LOG COMMON /FLOAT1/FLOAT1 COMMON /FLOAT2/FLOAT2 REAL FLOAT1, FLOAT2 DATA FLOAT2/2.4/ ! FLOAT1 INITIALIZED IN MAIN C PRINT CURRENT STATE OF VARIABLES PRINT*, ' IN FTNFCTN: FLOAT1 = ', FLOAT1, 1 ';FLOAT2 = ', FLOAT2 PRINT*, ' ARGUMENTS: STR = "', STR, '"; LOG = ', LOG C CHANGE THE VALUES FOR STR(ING) AND LOG(ICAL) STR = 'New Fortran String' LOG = .FALSE. FTNFCTN = 123.4 PRINT*, ' RETURNING FROM FTNFCTN WITH ', FTNFCTN PRINT* RETURN END The previous Cray C function and Fortran subprogram are executed by the following commands and produce the following output: % cc -c main.c % ftn -c ftnfctn.f % ftn main.o ftnfctn.o % ./a.out In main: FLOAT1 = 1.6; FLOAT2 = 2.4 Calling FTNFCTN with arguments: string = "C Character String"; logical = 1 IN FTNFCTN: FLOAT1 = 1.6; FLOAT2 = 2.4 ARGUMENTS: STR = "C Character String"; LOG = T RETURNING FROM FTNFCTN WITH 123.4 Back in main: FTNFCTN returned 123.4 and changed the two arguments: string = "New Fortran String"; logical = 0 150 S–2179–80Using Interlanguage Communication [12] 12.2.8 Calling a Fortran Program from Cray C++ The following example illustrates how a Fortran program can be called from a Cray C++ program: #include extern "C" int FORTRAN_ADD_INTS(int *arg1, int &arg2); main() { int num1, num2, res; cout << "Start C++ main" << endl << endl; //Call FORTRAN function to add two integers and return result. //Note that the second argument is a reference parameter so //it is not necessary to take the address of the //variable num2. num1 = 10; num2 = 20; cout << "Before Call to FORTRAN_ADD_INTS" << endl; res = FORTRAN_ADD_INTS(&num1, num2); cout << "Result of FORTRAN Add = " << res << endl << endl; cout << "End C++ main" << endl; } The Fortran program that is called from the Cray C++ main function in the preceding example is as follows: INTEGER FUNCTION FORTRAN_ADD_INTS(Arg1, Arg2) INTEGER Arg1, Arg2 PRINT *," FORTRAN_ADD_INTS, Arg1,Arg2 = ", Arg1, Arg2 FORTRAN_ADD_INTS = Arg1 + Arg2 END The output from the execution of the preceding example is as follows: Start C++ main Before Call to FORTRAN_ADD_INTS FORTRAN_ADD_INTS, Arg1,Arg2 = 10, 20 Result of FORTRAN Add = 30 End C++ main 12.3 Calling a C or C++ Function from Fortran In addition to calling Fortran functions and subroutines from C or C++ programs, C or C++ functions can be called from Fortran. Two methods can be used: either the Standard Fortran/C Interoperability on page 155 or the Portable Interoperability Mechanism on page 152, which provides a standard portable interoperability mechanism for Fortran and C programs. S–2179–80 151Cray C and C++ Reference Manual 12.3.1 Portable Interoperability Mechanism If you use the method documented in this section, keep in mind the information in Calling Fortran Functions and Subroutines from C or C++ on page 143. When calling a Cray C++ function from a Fortran program, observe the following rules: • The Cray C++ function must be declared with extern "C" linkage. • The program must be linked using the CC command. • The program's main routine must be C or C++ code compiled using the CC command. The example that follows illustrates a Fortran program, main.f, that calls a Cray C function, ctctn.c. The Cray C function being called, the commands required, and the associated input and output are also included. Note: This example assumes that the Cray Fortran program is compiled with the -s default32 option enabled, and will not work if the -s default64 option is enabled. 152 S–2179–80Using Interlanguage Communication [12] Example 14. Calling a C function from Fortran Fortran program main.f source code: C Fortran program (main.f): PROGRAM MAIN REAL CFCTN COMMON /FLOAT1/FLOAT1 COMMON /FLOAT2/FLOAT2 REAL FLOAT1, FLOAT2 DATA FLOAT1/1.6/ ! FLOAT2 INITIALIZED IN cfctn.c LOGICAL LOG CHARACTER*24 STR REAL RTNVAL C INITIALIZE VARIABLES STR(ING) AND LOG(ICAL) STR = 'Fortran Character String' LOG = .TRUE. C PRINT VALUES OF VARIABLES BEFORE CALL TO C FUNCTION PRINT*, 'In main.f: FLOAT1 = ', FLOAT1, 1 '; FLOAT2 = ', FLOAT2 PRINT*, 'Calling cfctn.c with these arguments: ' PRINT*, 'LOG = ', LOG PRINT*, 'STR = ', STR RTNVAL = CFCTN(STR, LOG) C PRINT VALUES OF VARIABLES AFTER CALL TO C FUNCTION PRINT*, 'Back in main.f:: cfctn.c returned ', RTNVAL PRINT*, 'and changed the two arguments to: ' PRINT*, 'LOG = ', LOG PRINT*, 'STR = ', STR END PROGRAM Compile main.f, creating main.o: % ftn -c main.f S–2179–80 153Cray C and C++ Reference Manual C function cfctn.c source code: /* C function (cfctn.c) */ #include #include #include #include float FLOAT1; /* Initialized in MAIN */ float FLOAT2 = 2.4; /* The slen argument passes the length of string in str */ float cfctn_(char * str, int *log, int slen) { int clog; float rtnval; char *cstring; /* Convert log passed from Fortran MAIN */ /* into its C equivalent */ cstring = malloc(slen+1); strncpy(cstring, str, slen); cstring[slen] = '\0'; clog = _ltob(log); /* Print the current state of the variables */ printf(" In CFCTN: FLOAT1 = %.1f; FLOAT2 = %.1f\n", FLOAT1, FLOAT2); printf(" Arguments: str = '%s'; log = %d\n", cstring, clog); /* Change the values for str and log */ strncpy(str, "C Character String ", 24); *log = 0; rtnval = 123.4; printf(" Returning from CFCTN with %.1f\n\n", rtnval); return(rtnval); } Compile cfctn.c, creating cfctn.o: % cc -c cfctn.c Link main.o and cfctn.o, creating executable interlang1: % ftn -o interlang1 main.o cfctn.o Run program interlang1: % ./interlang1 154 S–2179–80Using Interlanguage Communication [12] Program output: In main.f: FLOAT1 = 1.60000002 ; FLOAT2 = 2.4000001 Calling cfctn.c with these arguments: LOG = T STR = Fortran Character String In CFCTN: FLOAT1 = 1.6; FLOAT2 = 2.4 Arguments: str = 'Fortran Character String'; log = 1 Returning from CFCTN with 123.4 Back in main.f:: cfctn.c returned 123.400002 and changed the two arguments to: LOG = F STR = C Character String 12.3.2 Standard Fortran/C Interoperability For more information about C interoperability, see the current Fortran standard. The ISO_C_BINDING module provides interoperability between Fortran intrinsic types and C types. The ISO_C_BINDING module provides named constants which can be used as KIND type parameters, compatible with C types. In addition to the named constants required by the Fortran 2003 standard, Cray compiler provides, as an extension, definitions for 128-bit floating, and complex types. C_FLOAT128 and C_FLOAT128_COMPLEX correspond to C types __float128 and __float128 complex. S–2179–80 155Cray C and C++ Reference Manual 156 S–2179–80Implementation-defined Behavior [13] This chapter describes compiler behavior that is defined by the implementation according to the C and/or C++ standards. The standards require that the behavior of each particular implementation be documented, and define implementation-defined behavior as behavior that is dependent on the characteristics of the implementation for a correct program construct and correct data. 13.1 Messages All diagnostic messages issued by the compilers are reported through the Cray Linux Environment (CLE) message system. For information about messages issued by the compilers and for information about the Cray Linux Environment (CLE) message system, see Appendix C, Using the Compiler Message System on page 181. 13.2 Environment When argc and argv are used as parameters to the main function, the array members argv[0] through argv[argc-1] contain pointers to strings that are set by the command shell. The shell sets these arguments to the list of words on the command line used to invoke the compiler (the argument list). For further information about how the words in the argument list are formed, refer to the documentation on the shell in which you are running. For information about Cray Linux Environment (CLE) shells, see the sh(1) or csh(1) man page. A third parameter, char **envp, provides access to environment variables. The value of the parameter is a pointer to the first element of an array of null-terminated strings that matches the output of the env command. The array of pointers is terminated by a null pointer. The compiler does not distinguish between interactive devices and other, noninteractive devices. The library, however, may determine that stdin, stdout, and stderr (cin, cout, and cerr in Cray C++) refer to interactive devices and buffer them accordingly. S–2179–80 157Cray C and C++ Reference Manual 13.2.1 Identifiers The identifier (as defined by the standards) is merely a sequence of letters and digits. Specific uses of identifiers are called names. The Cray C compiler treats the first 255 characters of a name as significant, regardless of whether it is an internal or external name. The case of names, including external names, is significant. In Cray C++, all characters of a name are significant. 13.2.2 Types Table 12 summarizes Cray C and C++ types and the characteristics of each type. Representation is the number of bits used to represent an object of that type. Memory is the number of storage bits that an object of that type occupies. In the Cray C and C++ compilers, size, in the context of the sizeof operator, refers to the size allocated to store the operand in memory; it does not refer to representation, as specified in Table 12. Thus, the sizeof operator will return a size that is equal to the value in the Memory column of Table 12 divided by 8 (the number of bits in a byte). Table 12. Data Type Mapping Type Representation Size and Memory Storage Size (bits) bool (C++) 8 _Bool (C) 8 char 8 wchar_t 32 short 16 int 32 long 64 long long 64 float 32 double 64 long double 64 float complex 64 (each part is 32 bits) double complex 128 (each part is 64 bits) long double complex 128 (each part is 64 bits) _float128 128 158 S–2179–80Implementation-defined Behavior [13] Type Representation Size and Memory Storage Size (bits) _float128 complex 256 (each part is 128 bits) Pointers 64 Note: Vectorization of 8- and 16-bit data types is deferred. 13.2.3 Characters The full 8-bit ASCII code set can be used in source files. Characters not in the character set defined in the standard are permitted only within character constants, string literals, and comments. The -h [no]calchars option allows the use of the $ sign in identifier names. For more information about the -h [no]calchars option, see -h [no]calchars on page 29. A character consists of 8 bits. Up to 8 characters can be packed into a 64-bit word. A plain char type (that is, one that is declared without a signed or unsigned keyword) is treated as a signed type. Character constants and string literals can contain any characters defined in the 8-bit ASCII code set. The characters are represented in their full 8-bit form. A character constant can contain up to 8 characters. The integer value of a character constant is the value of the characters packed into a word from left to right, with the result right-justified, as shown in the following table: Table 13. Packed Characters Character Constant Integer Value 'a' 0x61 'ab' 0x6162 In a character constant or string literal, if an escape sequence is not recognized, the \ character that initiates the escape sequence is ignored, as shown in the following table: Table 14. Unrecognizable Escape Sequences Character Constant Integer Value Explanation '\a' 0x7 Recognized as the ASCII BEL character '\8' 0x38 Not recognized; ASCII value for 8 '\[' 0x5b Not recognized; ASCII value for [ '\c' 0x63 Not recognized; ASCII value for c S–2179–80 159Cray C and C++ Reference Manual 13.2.4 Wide Characters Wide characters are treated as signed 64-bit integer types. Wide character constants cannot contain more than one multibyte character. Multibyte characters in wide character constants and wide string literals are converted to wide characters in the compiler by calling the mbtowc() function. The current locale in effect at the time of compilation determines the method by which mbtowc() converts multibyte characters to wide characters, and the shift states required for the encoding of multibyte characters in the source code. If a wide character, as converted from a multibyte character or as specified by an escape sequence, cannot be represented in the extended execution character set, it is truncated. 13.2.5 Integers All integral values are represented in a two's complement format. For representation and memory storage requirements for integral types, see Table 12. When an integer is converted to a shorter signed integer, and the value cannot be represented, the result is the truncated representation treated as a signed quantity. When an unsigned integer is converted to a signed integer of equal length, and the value cannot be represented, the result is the original representation treated as a signed quantity. The bitwise operators (unary operator ~ and binary operators <<, >>, &, ^, and |) operate on signed integers in the same manner in which they operate on unsigned integers. The result of e1 >> e2, where e1 is a negative-valued signed integral value, is that e1 is right-shifted e2 bit positions; vacated bits are filled with 1s. This behavior can be modified by using the -h nosignedshifts option (see -h [no]signedshifts on page 29). Bits higher than the sixth bit are not ignored. The result of the / operator is the largest integer less than or equal to the algebraic quotient when either operand is negative and the result is a nonnegative value. If the result is a negative value, it is the smallest integer greater than or equal to the algebraic quotient. The / operator behaves the same way in C and C++ as in Fortran. The sign of the result of the percent (%) operator is the sign of the first operand. Integer overflow is ignored. Because some integer arithmetic uses the floating-point instructions, floating-point overflow can occur during integer operations. Division by 0 and all floating-point exceptions, if not detected as an error by the compiler, can cause a run time abort. 13.2.6 128-Bit Floating Point and 256-Bit Complex Predefined Types The Cray C and C++ Compilers now support 128-bit floating point and 256-bit complex predefined types using the X86-64 ABI definitions for type names and data layout. These types are sometimes referred to as "quad-precision". In C and C++, use __float128, and __float128 _complex. 160 S–2179–80Implementation-defined Behavior [13] In C and C++, long double remains identical to double – 64-bit IEEE, and not 80-bit extended precision. The base type itself uses 128 bits of storage with a guaranteed minimum alignment on a 128-bit boundary, little endian, has a 15-bit exponent, a 113-bit mantissa, and an exponent bias of 16383, and is compatible with the gcc implementation. C forms of intrinsic math functions offer full support for quad-precision types. See the intro_quad_precision(3i) man page for a complete list of intrinsic functions that support quad-precision. 13.2.7 Arrays and Pointers An unsigned long value can hold the maximum size of an array. The type size_t is defined to be a typedef name for unsigned long in the headers: malloc.h, stddef.h, stdio.h, stdlib.h, string.h, and time.h. If more than one of these headers is included, only the first defines size_t. A type long can hold the difference between two pointers to elements of the same array. The type ptrdiff_t is defined to be a typedef name for long in the header stddef.h. If a pointer type's value is cast to a signed or unsigned long int, and then cast back to the original type's value, the two pointer values will compare equal. Pointers on Cray Linux Environment (CLE) systems are byte pointers. Byte pointers use the same internal representation as integers; a byte pointer counts the numbers of bytes from the first address. A pointer can be explicitly converted to any integral type large enough to hold it. The result will have the same bit pattern as the original pointer. Similarly, any value of integral type can be explicitly converted to a pointer. The resulting pointer will have the same bit pattern as the original integral type. 13.2.8 Registers Use of the register storage class in the declaration of an object has no effect on whether the object is placed in a register. The compiler performs register assignment aggressively; that is, it automatically attempts to place as many variables as possible into registers. 13.2.9 Classes, Structures, Unions, Enumerations, and Bit Fields Accessing a member of a union by using a member of a different type results in an attempt to interpret, without conversion, the representation of the value of the member as the representation of a value in the different type. S–2179–80 161Cray C and C++ Reference Manual Members of a class or structure are packed into words from left to right. Padding is appended to a member to correctly align the following member, if necessary. Member alignment is based on the size of the member: • For a member bit field of any size, alignment is any bit position that allows the member to fit entirely within a 64–bit word. • For a member with a size less than 64 bits, alignment is the same as the size. For example, a char has a size and alignment of 8 bits; a float has a size and alignment of 32 bits. • For a member with a size equal to or greater than 64 bits, alignment is 64 bits. • For a member with array type, alignment is equal to the alignment of the element type. A plain int type bit field is treated as a signed int bit field. The values of an enumeration type are represented in the type signed int in C; they are a separate type in C++. 13.2.10 Qualifiers When an object that has volatile-qualified type is accessed, it is simply a reference to the value of the object. If the value is not used, the reference need not result in a load of the value from memory. 13.2.11 Declarators A maximum of 12 pointer, array, and/or function declarators are allowed to modify an arithmetic, structure, or union type. 13.2.12 Statements The compiler has no fixed limit on the maximum number of case values allowed in a switch statement. The Cray C++ compiler parses asm statements for correct syntax, but otherwise ignores them. 13.2.13 Exceptions In Cray C++, when an exception is thrown, the memory for the temporary copy of the exception being thrown is allocated on the stack and a pointer to the allocated space is returned. 162 S–2179–80Implementation-defined Behavior [13] 13.2.14 System Function Calls For a description of the form of the unsuccessful termination status that is returned from a call to exit(3), see the exit(3) man page. 13.3 Preprocessing The value of a single-character constant in a constant expression that controls conditional inclusion matches the value of the same character in the execution character set. No such character constant has a negative value. For each, 'a' has the same value in the two contexts: #if 'a' == 97 if ('a' == 97) The -I option and the method for locating included source files is described in -I incldir on page 53. The source file character sequence in a #include directive must be a valid or Cray Linux Environment (CLE) file name or path name. A #include directive may specify a file name by means of a macro, provided the macro expands into a source file character sequence delimited by double quotes or < and > delimiters, as follows: #define myheader "./myheader.h" #include myheader #define STDIO #include STDIO The macros __DATE__ and __TIME__ contain the date and time of the beginning of translation. For more information, refer to the description of the predefined macros in Chapter 9, Using Predefined Macros on page 131. The #pragma directives are described in Chapter 3, Using #pragma Directives on page 65. S–2179–80 163Cray C and C++ Reference Manual 164 S–2179–80Using Libraries and the Linker [A] This appendix describes the libraries that are available with the Cray C and C++ compilers and the linker. A.1 Cray C and C++ Libraries Libraries that support Cray C and C++ are automatically available when you use the CC or cc command to compile your programs. These commands automatically issue the appropriate directives to link the program with the appropriate functions. If your program strictly conforms to the C or C++ standards, you do not need to know library names and locations. If your program requires other libraries or if you want direct control over the linking process, more knowledge of the linker and libraries is necessary. The Standard Template Library (STL) is a C++ library of container classes, algorithms, and iterators; it provides many of the basic algorithms and data structures of computer science. The STL is a generic library, meaning that its components are heavily parameterized: almost every component in the STL is a template. Be sure you have a complete understanding of templates and how they work before using them. A.2 Linker When you use the cc or CC command to invoke the compiler, and the program compiles without errors, the linker is called. Specifying the -c option on the command line produces relocatable object files (*.o) without calling the linker. These relocatable object files can then be used as input to the linker command by specifying the file names on the appropriate linker command line. For example, the following command line compiles a file called target.c and produces the relocatable object file called target.o in your current working directory: cc -c target.c You can then use file target.o as input to the linker or save the file to use with other relocatable object files to compile and create a linked executable file (a.out by default). S–2179–80 165Cray C and C++ Reference Manual Because of the special code needed to handle templates, constructors, destructors, and other C++ language features, object files generated by using the CC command should be linked using the CC command. 166 S–2179–80Using Cray C and C++ Dialects [B] This appendix details the features of the C and C++ languages that are accepted by the Cray C and C++ compilers, including certain language dialects and anachronisms. Users should be aware of these details, especially users who are porting codes from other environments. B.1 C++ Language Conformance The Cray C++ compiler accepts the C++ language as defined by the ISO/IEC 14882:2003 standard, except for exported templates. The Cray C++ compiler also has a cfront compatibility mode, which duplicates a number of features and bugs of cfront. Complete compatibility is not guaranteed or intended. The mode allows programmers who have used cfront features to continue to compile their existing code (see General Directives on page 67). Command line options are also available to enable and disable anachronisms (see C++ Anachronisms Accepted on page 171) and strict standard-conformance checking (see Extensions Accepted in Normal C++ Mode on page 172, and Extensions Accepted in C or C++ Mode on page 173). The command line options are described in Chapter 2, Invoking the C and C++ Compilers on page 19. B.1.1 Supported C++ Language Features The following features, which are in the ISO/IEC 14882:2003 standard but not in traditional C++1 , are supported: • The dependent statement of an if, while, do-while, or for is considered to be a scope, and the restriction on having such a dependent statement be a declaration is removed. • The expression tested in an if, while, do-while, or for, as the first operand of a ? operator, or as an operand of the &&, ||, or ! operators may have a pointer-to-member type or a class type that can be converted to a pointer-to-member type in addition to the scalar cases permitted by the ARM. • Qualified names are allowed in elaborated type specifiers. 1 As defined in The Annotated C++ Reference Manual (ARM), by Ellis and Stroustrup, Addison Wesley, 1990. S–2179–80 167Cray C and C++ Reference Manual • A global-scope qualifier is allowed in member references of the form x.::A::B and p->::A::B. • The precedence of the third operand of the ? operator is changed. • If control reaches the end of the main() routine, and the main() routine has an integral return type, it is treated as if a return 0; statement was executed. • Pointers to arrays with unknown bounds as parameter types are diagnosed as errors. • A functional-notation cast of the form A() can be used even if A is a class without a (nontrivial) constructor. The temporary that is created gets the same default initialization to zero as a static object of the class type. • A cast can be used to select one out of a set of overloaded functions when taking the address of a function. • Template friend declarations and definitions are permitted in class definitions and class template definitions. • Type template parameters are permitted to have default arguments. • Function templates may have nontype template parameters. • A reference to const volatile cannot be bound to an rvalue. • Qualification conversions such as conversion from T** to T const * const are allowed. • Digraphs are recognized. • Operator keywords (for example, and or bitand) are recognized. • Static data member declarations can be used to declare member constants. • bool is recognized. • RTTI (run time type identification), including dynamic_cast and the typeid operator, is implemented. • Declarations in tested conditions (within if, switch, for, and while statements) are supported. • Array new and delete are implemented. • New-style casts (static_cast, reinterpret_cast, and const_cast) are implemented. • Definition of a nested class outside its enclosing class is allowed. • mutable is accepted on nonstatic data member declarations. • Namespaces are implemented, including using declarations and directives. Access declarations are broadened to match the corresponding using declarations. 168 S–2179–80Using Cray C and C++ Dialects [B] • The typename keyword is recognized. • explicit is accepted to declare nonconverting constructors. • The scope of a variable declared in the for-init-statement of a for loop is the scope of the loop (not the surrounding scope). • Member templates are implemented. • The new specialization syntax (using template <>) is implemented. • Type qualifiers, const and volatile referred to as cv-qualifiers are retained on rvalues (in particular, on function return values). • The distinction between trivial and nontrivial constructors has been implemented, as has the distinction between process overlay directives (PODs) and non-PODs with trivial constructors. • The linkage specification is treated as part of the function type (affecting function overloading and implicit conversions). • A typedef name can be used in an explicit destructor call. • Placement delete is supported. • An array allocated via a placement new can be deallocated via delete. • enum types are considered to be nonintegral types. • Partial specification of class templates is implemented. • Partial ordering of function templates is implemented. • Function declarations that match a function template are regarded as independent functions, not as “guiding declarations” that are instances of the template. • It is possible to overload operators using functions that take enum types and no class types. • Explicit specification of function template arguments is supported. • Unnamed template parameters are supported. • The new lookup rules for member references of the form x.A::B and p->A::B are supported. • The notation :: template (and –>template, etc.) is supported. • In a reference of the form f()->g(), with g a static member function, f() is evaluated. Likewise for a similar reference to a static data member. The ARM specifies that the left operand is not evaluated in such cases. • enum types can contain values larger than can be contained in an int. S–2179–80 169Cray C and C++ Reference Manual • Default arguments of function templates and member functions of class templates are instantiated only when the default argument is used in a call. • String literals and wide string literals have const type. • Argument-dependent (Koenig) lookup of function names is implemented. • Class and function names declared only in unqualified friend declarations are not visible except for functions found by argument-dependent lookup. • A void expression can be specified on a return statement in a void function. • reinterpret_cast allows casting a pointer to a member of one class to a pointer to a member of another class even when the classes are unrelated. • Two-phase name binding in templates as described in the Working Paper is implemented. • Putting a try/catch around the initializers and body of a constructor is implemented. • template parameters are implemented. • Universal character set escapes (e.g., \uabcd) are implemented. • extern inline functions are supported. • Covariant return types on overriding virtual functions are supported. 170 S–2179–80Using Cray C and C++ Dialects [B] B.2 C++ Anachronisms Accepted C++ anachronisms are enabled by using the -h anachronisms command line option (see -h [no]anachronisms (CC) on page 23). When anachronisms are enabled, the following anachronisms are accepted: • overload is allowed in function declarations. It is accepted and ignored. • Definitions are not required for static data members that can be initialized by using the default initialization. The anachronism does not apply to static data members of template classes; they must always be defined. • The number of elements in an array can be specified in an array delete operation. The value is ignored. • A single operator++() and operator--() function can be used to overload both prefix and postfix operations. • The base class name can be omitted in a base class initializer if there is only one immediate base class. • Assignment to the this pointer in constructors and destructors is allowed. This is only allowed if anachronisms are enabled and the assignment to this configuration parameter is enabled. • A bound function pointer (a pointer to a member function for a given object) can be cast to a pointer to a function. • A nested class name may be used as a non-nested class name if no other class of that name has been declared. The anachronism is not applied to template classes. • A reference to a non-const type may be initialized from a value of a different type. A temporary is created, it is initialized from the (converted) initial value, and the reference is set to the temporary. • A reference to a non-const class type may be initialized from an rvalue of the class type or a derived class thereof. No (additional) temporary is used. • A function with old-style parameter declarations is allowed and can participate in function overloading as though it were prototyped. Default argument promotion is not applied to parameter types of such functions when checking for compatibility, therefore, the following statements declare the overloading of two functions named f: int f(int); int f(x) char x; { return x; } Note: In C, this code is legal, but has a different meaning. A tentative declaration of f is followed by its definition. S–2179–80 171Cray C and C++ Reference Manual B.3 Extensions Accepted in Normal C++ Mode The following C++ extensions are accepted (except when strict standard conformance mode is enabled, in which case a warning or caution message may be issued): • A friend declaration for a class can omit the class keyword, as shown in the following example: class B; class A { friend B; // Should be "friend class B" }; • Constants of scalar type can be defined within classes, as shown in the following example: class A { const int size=10; int a[size]; }; • In the declaration of a class member, a qualified name can be used, as shown in the following example: struct A { int A::f(); // Should be int f(); } • An assignment operator declared in a derived class with a parameter type matching one of its base classes is treated as a “default” assignment operator; that is, such a declaration blocks the implicit generation of a copy assignment operator. This is cfront behavior that is known to be relied upon in at least one widely used library. Here is an example: struct A { }; struct B : public A { B& operator=(A&); }; By default, as well as in cfront compatibility mode, there will be no implicit declaration of B::operator=(const B&), whereas in strict-ANSI mode, B::operator=(A&) is not a copy assignment operator and B::operator=(const B&) is implicitly declared. • Implicit type conversion between a pointer to an extern "C" function and a pointer to an extern "C++" function is permitted. The following is an example: extern "C" void f(); // f's type has extern "C" linkage void (*pf)() // pf points to an extern "C++" function = &f; // error unless implicit conversion allowed 172 S–2179–80Using Cray C and C++ Dialects [B] • The ? operator, for which the second and third operands are string literals or wide string literals, can be implicitly converted to one of the following: char * wchar_t * In C++ string literals are const. There is a deprecated implicit conversion that allows conversion of a string literal to char *, dropping the const. That conversion, however, applies only to simple string literals. Allowing it for the result of a ? operation is an extension: char *p = x ? "abc" : "def"; B.4 Extensions Accepted in C or C++ Mode The following extensions are accepted in C or C++ mode except when strict standard conformance modes is enabled, in which case a warning or caution message may be issued. • The special lint comments /*ARGSUSED*/, /*VARARGS*/ (with or without a count of nonvarying arguments), and /*NOTREACHED*/ are recognized. • A translation unit (input file) can contain no declarations. • Comment text can appear at the ends of preprocessing directives. • Bit fields can have base types that are enum or integral types in addition to int and unsigned int. This corresponds to A.6.5.8 in the ANSI Common Extensions appendix. • enum tags can be incomplete as long as the tag name is defined and resolved by specifying the brace-enclosed list later. • An extra comma is allowed at the end of an enum list. • The final semicolon preceding the closing of a struct or union type specifier can be omitted. • A label definition can be immediately followed by a right brace ( } ). (Normally, a statement must follow a label definition.) • An empty declaration (a semicolon preceded by nothing) is allowed. • An initializer expression that is a single value and is used to initialize an entire static array, struct, or union does not need to be enclosed in braces. ANSI C requires braces. • In an initializer, a pointer constant value can be cast to an integral type if the integral type is large enough to contain it. • The address of a variable with register storage class may be taken. S–2179–80 173Cray C and C++ Reference Manual • In an integral constant expression, an integer constant can be cast to a pointer type and then back to an integral type. • In duplicate size and sign specifiers (for example, short short or unsigned unsigned) the redundancy is ignored. • Benign redeclarations of typedef names are allowed. That is, a typedef name can be redeclared in the same scope with the same type. • Dollar sign ($) characters can be accepted in identifiers by using the -h calchars command line option. This is not allowed by default. • Numbers are scanned according to the syntax for numbers rather than the pp-number syntax. Thus, 0x123e+1 is scanned as three tokens instead of one token that is not valid. If the -h conform option is specified, the pp-number syntax is used. • Assignment and pointer differences are allowed between pointers to types that are interchangeable but not identical, for example, unsigned char * and char *. This includes pointers to integral types of the same size (for example, int * and long *). Assignment of a string constant to a pointer to any kind of character is allowed without a warning. • Assignment of pointer types is allowed in cases where the destination type has added type qualifiers that are not at the top level (for example, int ** to const int **). Comparisons and pointer difference of such pairs of pointer types are also allowed. • In operations on pointers, a pointer to void is always implicitly converted to another type if necessary, and a null pointer constant is always implicitly converted to a null pointer of the right type if necessary. In ANSI C, these are allowed by some operators, and not by others (generally, where it does not make sense). • Pointers to different function types may be assigned or compared for equality (==) or inequality (!=) without an explicit type cast. This extension is not allowed in C++ mode. • A pointer to void can be implicitly converted to or from a pointer to a function type. • External entities declared in other scopes are visible: void f1(void) { extern void f(); } void f2() { f(); /* Using out of scope declaration */ } • In C mode, end-of-line comments (//) are supported. • A non-lvalue array expression is converted to a pointer to the first element of the array when it is subscripted or similarly used. 174 S–2179–80Using Cray C and C++ Dialects [B] • The fortran keyword. For more information, see fortran Keyword on page 127. • Cray hexadecimal floating point constants. For more information, see Hexadecimal Floating-point Constants on page 128. B.5 C++ Extensions Accepted in cfront Compatibility Mode The cfront compatibility mode is enabled by the -h cfront command-line option. The following extensions are accepted in cfront compatibility mode: • Type qualifiers on the this parameter are dropped in contexts such as in the following example: struct A { void f() const; }; void (A::*fp)() = &A::f; This is a safe operation. A pointer to a const function can be put into a pointer to non-const, because a call using the pointer is permitted to modify the object and the function pointed to will not modify the object. The opposite assignment would not be safe. • Conversion operators that specify a conversion to void are allowed. • A nonstandard friend declaration can introduce a new type. A friend declaration that omits the elaborated type specifier is allowed in default mode, however, in cfront mode the declaration can also introduce a new type name. An example follows: struct A { friend B; }; • The third operator of the ? operator is a conditional expression instead of an assignment expression. • A reference to a pointer type may be initialized from a pointer value without use of a temporary even when the reference pointer type has additional type qualifiers above those present in the pointer value. For example: int *p; const int *&r = p; // No temporary used • A reference can be initialized to NULL. • Because cfront does not check the accessibility of types, access errors for types are issued as warnings instead of errors. S–2179–80 175Cray C and C++ Reference Manual • When matching arguments of an overloaded function, a const variable with a value of 0 is not considered to be a null pointer constant. In general, in overload resolution, a null pointer constant must be entered as "0” to be considered a null pointer constant (e.g., '\0' is not considered a null pointer constant). • An alternate form of declaring pointer-to-member-function variables is supported, as shown in the following example: struct A { void f(int); static void sf(int); typedef void A::T3(int); // nonstd typedef decl typedef void T2(int); // std typedef }; typedef void A::T(int); // nonstd typedef decl T* pmf = &A::f; // nonstd ptr-to-member decl A::T2* pf = A::sf; // std ptr to static mem decl A::T3* pmf2 = &A::f; // nonstd ptr-to-member decl In this example, T is construed to name a function type for a nonstatic member function of class A that takes an int argument and returns void; the use of such types is restricted to nonstandard pointer-to-member declarations. The declarations of T and pmf in combination are equivalent to the following single standard pointer-to-member declaration: void (A::* pmf)(int) = &A::f; A nonstandard pointer-to-member declaration that appears outside of a class declaration, such as the declaration of T, is normally not valid and would cause an error to be issued. However, for declarations that appear within a class declaration, such as A::T3, this feature changes the meaning of a valid declaration. cfront version 2.1 accepts declarations, such as T, even when A is an incomplete type; so this case is also accepted. • Protected member access checking is not done when the address of a protected member is taken. For example: class B { protected: int i; }; class D : public B { void mf()}; void D::mf() { int B::* pmi1 = &B::i; // error, OK in cfront mode int D::* pmi2 = &D::i; // OK } Note: Protected member access checking for other operations (such as everything except taking a pointer-to-member address) is done normally. 176 S–2179–80Using Cray C and C++ Dialects [B] • The destructor of a derived class can implicitly call the private destructor of a base class. In default mode, this is an error but in cfront mode it is reduced to a warning. For example: class A { ~A(); }; class B : public A { ~B(); }; B::~B(){} // Error except in cfront mode • When disambiguation requires deciding whether something is a parameter declaration or an argument expression, the pattern type-name-or-keyword (identifier ...) is treated as an argument. For example: class A { A(); }; double d; A x(int(d)); A(x2); By default, int(d) is interpreted as a parameter declaration (with redundant parentheses), and so x is a function; but in cfront compatibility mode int(d) is an argument and x is a variable. The declaration A(x2) is also misinterpreted by cfront. It should be interpreted as the declaration of an object named x2, but in cfront mode it is interpreted as a function style cast of x2 to the type A. Similarly, the following declaration declares a function named xzy, that takes a parameter of type function taking no arguments and returning an int. In cfront mode, this is interpreted as a declaration of an object that is initialized with the value int(), which evaluates to 0. int xyz(int()); • A named bit field can have a size of 0. The declaration is treated as though no name had been declared. • Plain bit fields (such as bit fields declared with a type of int) are always signed. • The name given in an elaborated type specifier can be a typedef name that is the synonym for a class name. For example: typedef class A T; class T *pa; // No error in cfront mode • No warning is issued on duplicate size and sign specifiers, as shown in the following example: short short int i; // No warning in cfront mode S–2179–80 177Cray C and C++ Reference Manual • Virtual function table pointer-update code is not generated in destructors for base classes of classes without virtual functions, even if the base class virtual functions might be overridden in a further derived class. For example: struct A { virtual void f() {} A() {} ~A() {} }; struct B : public A { B() {} ~B() {f();} // Should call A::f according to ARM 12.7 }; struct C : public B { void f() {} } c; In cfront compatibility mode, B::~B calls C::f. • An extra comma is allowed after the last argument in an argument list. For example: f(1, 2, ); • A constant pointer-to-member function can be cast to a pointer-to-function, as in the following example. A warning is issued. struct A {int f();}; main () { int (*p)(); p = (int (*)())A::f; // Okay, with warning } • Arguments of class types that allow bitwise copy construction but also have destructors are passed by value like C structures, and the destructor is not called on the copy. In normal mode, the class object is copied into a temporary, the address of the temporary is passed as the argument, and the destructor is called on the temporary after the call returns. Because the argument is passed by value instead of by address, code like this compiled in cfront mode is not calling-sequence compatible with the same code compiled in normal mode. In practice, this is not much of a problem, since classes that allow bitwise copying usually do not have destructors. • A union member may be declared to have the type of a class for which the user has defined an assignment operator (as long as the class has no constructor or destructor). A warning is issued. • When an unnamed class appears in a typedef declaration, the typedef name may appear as the class name in an elaborated type specifier. For example: typedef struct { int i, j; } S; struct S x; // No error in cfront mode 178 S–2179–80Using Cray C and C++ Dialects [B] • Two member functions may be declared with the same parameter types when one is static and the other is nonstatic with a function qualifier. For example: class A { void f(int) const; static void f(int); // No error in cfront mode }; • The scope of a variable declared in the for-init-statement is the scope to which the for statement belongs. For example: int f(int i) { for (int j = 0; j < i; ++j) { /* ... */ } return j; // No error in cfront mode } • Function types differing only in that one is declared extern "C" and the other extern "C++" can be treated as identical: typedef void (*PF)(); extern "C" typedef void (*PCF)(); void f(PF); void f(PCF); By contrast, in standard C++, PF and PCF are different and incompatible types; PF is a pointer to an extern "C++" function whereas PCF is a pointer to an extern "C" function; and the two declarations of f create an overload set. • Functions declared inline have internal linkage. • enum types are regarded as integral types. • An uninitialized const object of non-POD class type is allowed even if its default constructor is implicitly declared as in the following example: struct A { virtual void f(); int i; }; const A a; • A function parameter type is allowed to involve a pointer or reference to array of unknown bounds. • If the user declares an operator= function in a class, but not one that can serve as the default operator=, and bitwise assignment could be done on the class, a default operator= is not generated. Only the user-written operator= functions are considered for assignments, so bitwise assignment is not done. S–2179–80 179Cray C and C++ Reference Manual 180 S–2179–80Using the Compiler Message System [C] This appendix describes how to use the message system to control and use messages issued by the compiler. Explanatory texts for messages can be displayed online through the use of the explain command. C.1 Expanding Messages with the explain Command You can use the explain command to display an explanation of any message issued by the compiler. The command takes as an argument, the message number, including the number's prefix. The prefix for Cray C and C++ is CC. In the following sample dialog, the cc command invokes the compiler on source file bug.c. Message CC-24 is displayed. The explain command displays the expanded explanation for this message. % cc bug.c CC-24 cc: ERROR File = bug.c, Line = 1 An invalid octal constant is used. int i = 018; ^ 1 error detected in the compilation of "bug.c". % explain CC-24 An invalid octal constant is used. Each digit of an octal constant must be between 0 and 7, inclusive. One or more digits in the indicated octal constant are outside of this range. Change each digit in the octal constant to be within the valid range. C.2 Controlling the Use of Messages This section summarizes the command line options that affect the issuing of messages from the compiler. S–2179–80 181Cray C and C++ Reference Manual C.2.1 Command Line Options Option Description -h errorlimit[=n] Specifies the maximum number of error messages the compiler prints before it exits. -h [no]message=n[:...] Enables or disables the specified compiler messages, overriding -h msglevel. -h msglevel_n Specifies the lowest severity level of messages to be issued. -h [no]msgs Enables or disables the writing of optimization messages to stderr. h [no]negmsgs Enables or disables the writing of messages to stderr that indicate why optimizations such as vectorization, inlining, or cloning did not occur in a given instance. -h report=args Generates optimization report messages. C.2.2 Environment Options for Messages The following are used by the message system. Variable Description NLSPATH Specifies the default value of the message system search path environment variable. LANG Identifies your requirements for native language, local customs, and coded character set with regard to the message system. MSG_FORMAT Controls the format in which you receive error messages. 182 S–2179–80Using the Compiler Message System [C] C.2.3 ORIG_CMD_NAME Environment Variable You can override the command name printed in the message. If the environment variable ORIG_CMD_NAME is set, the value of ORIG_CMD_NAME is used as the command name in the message. This functionality is provided for use with shell scripts that invoke the compiler. By setting ORIG_CMD_NAME to the name of the script, any message printed by the compiler appears as though it was generated by the script. For example, the following C shell script is named newcc: # setenv ORIG_CMD_NAME 'basename $0' cc $* A message generated by invoking newcc resembles the following: CC-8 newcc: ERROR File = x.c, Line = 1 A new-line character appears inside a string literal. Because the environment variable ORIG_CMD_NAME is set to newcc, this appears as the command name instead of cc in this message. ! Caution: The ORIG_CMD_NAME environment variable is not part of the message system. It is supported by the Cray C and C++ compilers as an aid to programmers. Other products, such as the Fortran compiler and the linker, may support this variable. However, you should not rely on support for this variable in any other product. You must be careful when setting the environment variable ORIG_CMD_NAME. If you set ORIG_CMD_NAME inadvertently, the compiler may generate messages with an incorrect command name. This may be particularly confusing if, for example, ORIG_CMD_NAME is set to newcc when the Fortran compiler prints a message. The Fortran message will look as though it came from newcc. C.3 Message Severity Each message issued by the compiler falls into one of the following categories of messages, depending on the severity of the error condition encountered or the type of information being reported. Category Meaning COMMENT Inefficient programming practices. NOTE Unusual programming style or the use of outmoded statements. CAUTION Possible user error. Cautions are issued when the compiler detects a condition that may cause the program to abort or behave unpredictably. S–2179–80 183Cray C and C++ Reference Manual Category Meaning WARNING Probable user error. Indicates that the program will probably abort or behave unpredictably. ERROR Fatal error; that is, a serious error in the source code. No binary output is produced. INTERNAL Problems in the compilation process. Please report internal errors immediately to the system support staff, so that a bug report can be filed. LIMIT Compiler limits have been exceeded. Normally you can modify the source code or environment to avoid these errors. If limit errors cannot be resolved by such modifications, please report these errors to the system support staff, so that bug report can be filed. INFO Useful additional information about the compiled program. INLINE Information about inline code expansion performed on the compiled code. SCALAR Information about scalar optimizations performed on the compiled code. VECTOR Information about vectorization optimizations performed on the compiled code. OPTIMIZATION Information about general optimizations. IPA_INFO Information about interprocedural optimizations. C.4 Common System Messages The errors in the following list can occur during the execution of a user program. The operating system detects them and issues the appropriate message. These errors are not detected by the compiler and are not unique to C and C++ programs; they may occur in any application program written in any language. • Operand Range Error An operand range error occurs when a program attempts to load or store in an area of memory that is not part of the user's area. This usually occurs when an invalid pointer is dereferenced. 184 S–2179–80Using the Compiler Message System [C] • Program Range Error A program range error occurs when a program attempts to jump into an area of memory that is not part of the user's area. This may occur, for example, when a function in the program mistakenly overwrites the internal program stack. When this happens, the address of the function from which the function was called is lost. When the function attempts to return to the calling function, it jumps elsewhere instead. • Error Exit An error exit occurs when a program attempts to execute an invalid instruction. This error usually occurs when the program's code area has been mistakenly overwritten with words of data (for example, when the program stores in a location pointed to by an invalid pointer). S–2179–80 185Cray C and C++ Reference Manual 186 S–2179–80Using Intrinsic Functions [D] The C and C++ intrinsic functions either allow for direct access to some hardware instructions or result in generation of inline code to perform some specialized functions. These intrinsic functions are processed completely by the compiler. In many cases, the generated code is one or two instructions. These are called functions because they are invoked with the syntax of function calls. To get access to most of the intrinsic functions, the Cray C++ compiler requires that either the intrinsics.h file be included or that the intrinsic functions that you want to call be explicitly declared. If the source code does not have an intrinsics.h statement and you cannot modify the code, you can use the -h prototype_intrinsics option instead. If you explicitly declare an intrinsic function, the declaration must agree with the documentation or the compiler treats the call as a call to a normal function, not the intrinsic function. The -h nointrinsics command line option causes the compiler to treat these calls as regular function calls and not as intrinsic function calls. There are built-in atomic memory intrinsic functions of the form __sync_* that do not require an include file nor any explicit declaration. The types of the arguments to intrinsic functions are checked by the compiler, and if any of the arguments do not have the correct type, a warning message is issued and the call is treated as a normal call to an external function. If your intention was to call an external function with the same name as an intrinsic function, you should change the external function name. The names used for the Cray C intrinsic functions are in the name space reserved for the implementation. Note: Several of these intrinsic functions have both a vector and a scalar version. If a vector version of an intrinsic function exists and the intrinsic is called within a vectorized loop, the compiler uses the vector version of the intrinsic. For details on whether it has a vector version, refer to the appropriate intrinsic function man page. The following sections groups the C and C++ intrinsics according to function and provides a brief description of each intrinsic in that group. For more information, see the corresponding man page. S–2179–80 187Cray C and C++ Reference Manual D.1 Atomic Memory Operations Atomic memory operations (AMOs), unlike other functions, cannot be interrupted by the system and can allow multiple threads to safely modify the same variable under certain conditions. The AMO intrinsics allow you to add, subtract, AND, NAND, OR, and XOR values together, or compare and swap values. Local AMOs operate on variables in the processor's local memory (cache domain); they do not use the network interface to access memory. Multiple threads using local atomic memory operations to access the same variable need to be running within the same processor cache domain, which implies that they must be running on the same node. Local AMOs are atomic with respect to each other. The compiler issues an error message if a user tries to apply a local AMO intrinsic to a Unified Parallel C or shared variable or Fortran coarray that is not local to the current thread. Global AMOs use the network interface to access variables in memory. The variables may or may not be in the processor's local cache domain. Global AMOs are atomic with respect to each other. Global AMOs are used to modify a Unified Parallel C (UPC) shared variable or Fortran coarray and are available only when compiling UPC (-hupc) or coarray Fortran (-hcaf). A global AMO uses a different mechanism for achieving atomicity than a local AMO, so local and global AMOs are not atomic with respect to each other. Global and local AMOs should not be used concurrently on the same memory location, without synchronization. It is possible to safely modify a variable using both atomic and non-atomic operations within a single UPC thread or Fortran image; however, if a thread or image modifies a variable with an atomic operation and a different thread or image concurrently modifies the same variable with a non-atomic operation, the result is indeterminate. 188 S–2179–80Using Intrinsic Functions [D] D.1.1 Local Atomic Memory Operations The following functions, defined in intrinsics.h, perform various local atomic memory operations: __builtin_ia32_lfence (Load fence) Insures that all memory loads issued before this intrinsic are visible in memory before any future loads are executed. __builtin_ia32_sfence (Store fence) Insures that all memory stores issued before this intrinsic are visible in memory before any future stores are executed. __builtin_ia32_mfence (Memory fence) Insures that all memory stores and loads issued before this intrinsic are visible in memory before any future stores or loads are executed. Functions built into the compiler do not require an include file, nor a specific compilation option for use. The following local atomic, built-in functions return the value of the object before the named operation occurs: Note: In this discussion, an object is an entity that is referred to by a pointer. A value is an actual number, bit mask, etc. that is not referred to by a pointer. The allowed object and value types are signed and unsigned integer types of 1, 2, 4, or 8 bytes. • The __sync_fetch_and_add function fetches the object pointed to by ptr, adds value, places the result into the object pointed to by ptr, and returns the old value of the object pointed to by ptr. • The __sync_fetch_and_sub function fetches the object pointed to by ptr, subtracts value, places the result into the object pointed to by ptr, and returns the old value of the object pointed to by ptr. • The __sync_fetch_and_or function fetches the object pointed to by ptr, ORs value, places the result into the object pointed to by ptr, and returns the old value of the object pointed to by ptr. • The __sync_fetch_and_and function fetches the object pointed to by ptr, ANDs value, places the result into the object pointed to by ptr, and returns the old value of the object pointed to by ptr. • The __sync_fetch_and_xor function fetches the object pointed to by *ptr, XORs value, places the result into the object pointed to by ptr, and returns the old value of the object pointed to by ptr. S–2179–80 189Cray C and C++ Reference Manual • The __sync_fetch_and_nand function fetches the object pointed to by ptr, NANDs value, places the result into the object pointed to by ptr, and returns the old value of the object pointed to by ptr. The following local atomic, built-in functions return the value of the object after the named operation occurs: • The __sync_add_and_fetch function adds value to the object pointed to by ptr and returns the new value of the object pointed to by ptr. • The __sync_sub_and_fetch function subtracts value from the object pointed to by ptr and returns the new value of the object pointed to by ptr. • The __sync_or_and_fetch function ORs value with the object pointed to by ptr and returns the new value of the object pointed to by ptr. • The __sync_and_and_fetch function ANDs value with the object pointed to by ptr and returns the new value of the object pointed to by ptr. • The __sync_xor_and_fetch function XORs value with the object pointed to by ptr and returns the new value of the object pointed to by ptr. • The __sync_nand_and_fetch function NANDs value with the current value of ptr and returns the new contents of ptr. D.1.2 Global Atomic Memory Operations Global atomic memory operations (global AMO) are typically used to atomically modify a Unified Parallel C (UPC) shared variable or Fortran coarray. The target of a global AMO can be located in a different cache domain, so a global AMO is not atomic with respect to memory operations performed locally within the target's cache domain. Therefore, the application must use synchronization to ensure that global AMOs and local memory operations are not used concurrently on the same memory location. The following intrinsics are defined in intrinsics.h. Functions without the _upc suffix accept both shared and non-shared pointers as the first argument. Functions with the _upc suffix accept only shared pointers as the first argument. Note: In this discussion, an object is an entity that is referred to by a pointer. A value is an actual number, bit mask, etc. that is not referred to by a pointer. • The _amo_aadd and _amo_aadd_upc functions (atomic add) add value to the object pointed to by ptr. • The _amo_aaddf and _amo_aaddf functions (atomic add and fetch) add value to the object pointed to by ptr and return the new value. • The _amo_afadd and _amo_afadd_upc functions (atomic fetch and add) add value to the object pointed to by ptr and return the old value of the object. 190 S–2179–80Using Intrinsic Functions [D] • The _amo_aax and _amo_aax_upc functions (atomic AND and XOR) AND the object pointed to by ptr with andMask, XOR the result with xorMask, and place the result into the object. • The _amo_afax and _amo_afax_upc functions (atomic fetch and AND and XOR) AND the object pointed to by ptr with andMask, XOR the result with xorMask, place the result into the object, and return the old value of the object. • The _amo_aandf and _amo_aandf_upc functions (atomic AND and fetch) AND the object pointed to by ptr with value, place the result into the object, and return the new value of the object. • The _amo_afand and _amo_afand_upc functions (atomic fetch and AND) AND the object pointed to by ptr with value, place the result into the object, and return the old value of the object. • The _amo_anandf and _amo_anandf_upc functions (atomic NAND and fetch) NAND the object pointed to by ptr with value, place the result into the object, and return the new value of the object. • The _amo_afnand and _amo_afnand_upc functions (atomic fetch and NAND) NAND the object pointed to by ptr with value, place the result into the object, and return the old value of the object. • The _amo_aorf and _amo_aorf_upc functions (atomic OR and fetch) OR the object pointed to by ptr with value, place the result into the object, and return the new value of the object. • The _amo_afor and _amo_afor_upc functions (atomic fetch and OR) OR the object pointed to by ptr with value, place the result into the object, and return the old value of the object. • The _amo_axorf and _amo_axorf_upc functions (atomic XOR and fetch) XOR the object pointed to by ptr with value, place the result into the object, and return the new value of the object. • The _amo_afxor and _amo_afxor_upc functions (atomic fetch and XOR) XOR the object pointed to by ptr with value, place the result into the object, and returns the old value of the object. • The _amo_acswap and _amo_acswap_upc functions (atomic compare and swap) compare and swap a value by replacing the contents of the object pointed to by ptr with replacementValue if compareValue is equal to the object pointed to by ptr and return the old value of the object. • The _amo_aswap and _amo_aswap_upc functions (atomic swap) swap a value by replacing the contents of the object pointed to by ptr with replacementValue. This function always returns the old value. S–2179–80 191Cray C and C++ Reference Manual • The _amo_aflush and _amo_aflush_upc functions (atomic flush) force *ptr to be written to memory. For more information, see the amo(3i) man page. D.2 Bit Operations The following intrinsic functions copy, count, or shift bits or computes the parity bit: _dshiftl Move the left most n bits of an integer into the right side of another integer, and return that integer. _dshiftr Move the right most n bits of an integer into the left side of another integer and return that integer. _pbit Copies the rightmost bit of a word to the n th bit, from the right, of another word. _pbits Copies the rightmost m bits of a word to another word beginning at bit n. _poppar Computes the parity bit for a variable. _popcnt _popcnt32 _popcnt64 Counts the number of set bits in 32-bit and 64-bit integer words. _leadz _leadz32 _leadz64 Counts the number of leading 0 bits in 32-bit and 64-bit integer words. _gbit _gbit returns the value of the n th bit from the right. _gbits Returns a value consisting of m bits extracted from a variable, beginning at n th bit from the right. D.3 Mask Operations The following intrinsic functions create bit masks: _mask Creates a left-justified or right-justified bit mask with all bits set to 1. _maskl Returns a left-justified bit mask with i bits set to 1. _maskr Returns a right-justified bit mask with i bits set to 1. 192 S–2179–80Using Intrinsic Functions [D] D.4 Miscellaneous Operations The following intrinsic functions perform various functions: _int_mult_upper Multiplies integers and returns the uppermost bits. For more information, see the int_mult_upper(3i) man page. _ranf Computes a pseudo-random floating-point number ranging from 0.0 through 1.0. _rtc Return a real-time clock value expressed in clock ticks. S–2179–80 193 Lustre File System Operations Manual - Version 1.8 Part No.: 821-0035-12 December 2010, Revision 01Please Recycle Copyright © 2007, 2010, Oracle and/or its af?liates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you ?nd any errors, please report them to us in writing. If this is software or related software documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS. Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-speci?c supplemental regulations. As such, the use, duplication, disclosure, modi?cation, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007). Oracle America, Inc., 500 Oracle Parkway, Redwood City, CA 94065. This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications which may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its af?liates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications. Oracle and Java are registered trademarks of Oracle and/or its af?liates. Other names may be trademarks of their respective owners. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark licensed through X/Open Company, Ltd. This software or hardware and documentation may provide access to or information on content, products, and services from third parties. Oracle Corporation and its af?liates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services. Oracle Corporation and its af?liates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services. Copyright © 2007, 2010, Oracle et/ou ses af?liés. Tous droits réservés. Ce logiciel et la documentation qui l’accompagne sont protégés par les lois sur la propriété intellectuelle. Ils sont concédés sous licence et soumis à des restrictions d’utilisation et de divulgation. Sauf disposition de votre contrat de licence ou de la loi, vous ne pouvez pas copier, reproduire, traduire, diffuser, modi?er, breveter, transmettre, distribuer, exposer, exécuter, publier ou af?cher le logiciel, même partiellement, sous quelque forme et par quelque procédé que ce soit. Par ailleurs, il est interdit de procéder à toute ingénierie inverse du logiciel, de le désassembler ou de le décompiler, excepté à des ?ns d’interopérabilité avec des logiciels tiers ou tel que prescrit par la loi. Les informations fournies dans ce document sont susceptibles de modi?cation sans préavis. Par ailleurs, Oracle Corporation ne garantit pas qu’elles soient exemptes d’erreurs et vous invite, le cas échéant, à lui en faire part par écrit. Si ce logiciel, ou la documentation qui l’accompagne, est concédé sous licence au Gouvernement des Etats-Unis, ou à toute entité qui délivre la licence de ce logiciel ou l’utilise pour le compte du Gouvernement des Etats-Unis, la notice suivante s’applique : U.S. GOVERNMENT RIGHTS. Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-speci?c supplemental regulations. As such, the use, duplication, disclosure, modi?cation, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007). Oracle America, Inc., 500 Oracle Parkway, Redwood City, CA 94065. Ce logiciel ou matériel a été développé pour un usage général dans le cadre d’applications de gestion des informations. Ce logiciel ou matériel n’est pas conçu ni n’est destiné à être utilisé dans des applications à risque, notamment dans des applications pouvant causer des dommages corporels. Si vous utilisez ce logiciel ou matériel dans le cadre d’applications dangereuses, il est de votre responsabilité de prendre toutes les mesures de secours, de sauvegarde, de redondance et autres mesures nécessaires à son utilisation dans des conditions optimales de sécurité. Oracle Corporation et ses af?liés déclinent toute responsabilité quant aux dommages causés par l’utilisation de ce logiciel ou matériel pour ce type d’applications. Oracle et Java sont des marques déposées d’Oracle Corporation et/ou de ses af?liés.Tout autre nom mentionné peut correspondre à des marques appartenant à d’autres propriétaires qu’Oracle. AMD, Opteron, le logo AMD et le logo AMD Opteron sont des marques ou des marques déposées d’Advanced Micro Devices. Intel et Intel Xeon sont des marques ou des marques déposées d’Intel Corporation. Toutes les marques SPARC sont utilisées sous licence et sont des marques ou des marques déposées de SPARC International, Inc. UNIX est une marque déposée concédée sous licence par X/Open Company, Ltd. Ce logiciel ou matériel et la documentation qui l’accompagne peuvent fournir des informations ou des liens donnant accès à des contenus, des produits et des services émanant de tiers. Oracle Corporation et ses af?liés déclinent toute responsabilité ou garantie expresse quant aux contenus, produits ou services émanant de tiers. En aucun cas, Oracle Corporation et ses af?liés ne sauraient être tenus pour responsables des pertes subies, des coûts occasionnés ou des dommages causés par l’accès à des contenus, produits ou services tiers, ou à leur utilisation. This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License. To view a copy of this license and obtain more information about Creative Commons licensing, visit Creative Commons Attribution-Share Alike 3.0 United States or send a letter to Creative Commons, 171 2nd Street, Suite 300, San Francisco, California 94105, USA.iii Contents Preface xxiii Part I Lustre Architecture 1. Introduction to Lustre 1–1 1.1 Introducing the Lustre File System 1–2 1.1.1 Lustre Key Features 1–3 1.2 Lustre Components 1–5 1.2.1 Lustre Networking (LNET) 1–6 1.2.2 Management Server (MGS) 1–7 1.3 Lustre Systems 1–7 1.4 Files in the Lustre File System 1–9 1.4.1 Lustre File System and Striping 1–11 1.4.2 Lustre Storage 1–12 1.4.2.1 OSS Storage 1–12 1.4.2.2 MDS Storage 1–12 1.4.3 Lustre System Capacity 1–13 1.5 Lustre Configurations 1–14 1.6 Lustre Networking 1–15 1.7 Lustre Failover and Rolling Upgrades 1–16iv Lustre 1.8 Operations Manual • December 2010 2. Understanding Lustre Networking 2–1 2.1 Introduction to LNET 2–1 2.2 Supported Network Types 2–2 2.3 Designing Your Lustre Network 2–3 2.3.1 Identify All Lustre Networks 2–3 2.3.2 Identify Nodes to Route Between Networks 2–3 2.3.3 Identify Network Interfaces to Include/Exclude from LNET 2–3 2.3.4 Determine Cluster-wide Module Configuration 2–4 2.3.5 Determine Appropriate Mount Parameters for Clients 2–4 2.4 Configuring LNET 2–5 2.4.1 Module Parameters 2–5 2.4.1.1 Using Usocklnd 2–7 2.4.1.2 OFED InfiniBand Options 2–8 2.4.2 Module Parameters - Routing 2–8 2.4.2.1 LNET Routers 2–11 2.4.3 Downed Routers 2–12 2.5 Starting and Stopping LNET 2–13 2.5.1 Starting LNET 2–13 2.5.1.1 Starting Clients 2–13 2.5.2 Stopping LNET 2–14Contents v Part II Lustre Administration 3. Installing Lustre 3–1 3.1 Preparing to Install Lustre 3–2 3.1.1 Supported Operating System, Platform and Interconnect 3–3 3.1.2 Required Lustre Software 3–4 3.1.3 Required Tools and Utilities 3–4 3.1.4 (Optional) High-Availability Software 3–4 3.1.5 Debugging Tools 3–5 3.1.6 Environmental Requirements 3–6 3.1.7 Memory Requirements 3–7 3.1.7.1 MDS Memory Requirements 3–7 3.1.7.2 OSS Memory Requirements 3–8 3.2 Installing Lustre from RPMs 3–10 3.3 Installing Lustre from Source Code 3–14 3.3.1 Patching the Kernel 3–15 3.3.1.1 Introducing the Quilt Utility 3–15 3.3.1.2 Get the Lustre Source and Unpatched Kernel 3–16 3.3.1.3 Patch the Kernel 3–17 3.3.2 Create and Install the Lustre Packages 3–18 3.3.3 Installing Lustre with a Third-Party Network Stack 3–20vi Lustre 1.8 Operations Manual • December 2010 4. Configuring Lustre 4–1 4.1 Configuring the Lustre File System 4–2 4.1.0.1 Simple Lustre Configuration Example 4–5 4.1.0.2 Module Setup 4–10 4.1.1 Scaling the Lustre File System 4–10 4.2 Additional Lustre Configuration 4–11 4.3 Basic Lustre Administration 4–11 4.3.1 Specifying the File System Name 4–12 4.3.2 Starting up Lustre 4–12 4.3.3 Mounting a Server 4–13 4.3.4 Unmounting a Server 4–14 4.3.5 Working with Inactive OSTs 4–14 4.3.6 Finding Nodes in the Lustre File System 4–15 4.3.7 Mounting a Server Without Lustre Service 4–16 4.3.8 Specifying Failout/Failover Mode for OSTs 4–16 4.3.9 Running Multiple Lustre File Systems 4–17 4.3.10 Setting and Retrieving Lustre Parameters 4–19 4.3.10.1 Setting Parameters with mkfs.lustre 4–19 4.3.10.2 Setting Parameters with tunefs.lustre 4–19 4.3.10.3 Setting Parameters with lctl 4–20 4.3.10.4 Reporting Current Parameter Values 4–21 4.3.11 Regenerating Lustre Configuration Logs 4–22 4.3.12 Changing a Server NID 4–23 4.3.13 Removing and Restoring OSTs 4–25 4.3.13.1 Removing an OST from the File System 4–25 4.3.13.2 Restoring an OST in the File System 4–27 4.3.14 Aborting Recovery 4–27 4.3.15 Determining Which Machine is Serving an OST 4–28Contents vii 4.4 More Complex Configurations 4–29 4.4.1 Failover 4–29 4.5 Operational Scenarios 4–30 4.5.1 Changing the Address of a Failover Node 4–31 5. Service Tags 5–1 5.1 Introduction to Service Tags 5–1 5.2 Using Service Tags 5–3 5.2.1 Installing Service Tags 5–3 5.2.2 Discovering and Registering Lustre Components 5–4 5.2.3 Service Tag Registration Information 5–7 6. Configuring Lustre - Examples 6–1 6.1 Simple TCP Network 6–1 6.1.1 Lustre with Combined MGS/MDT 6–1 6.1.1.1 Installation Summary 6–1 6.1.1.2 Configuration Generation and Application 6–2 6.1.2 Lustre with Separate MGS and MDT 6–3 6.1.2.1 Installation Summary 6–3 6.1.2.2 Configuration Generation and Application 6–3 6.1.2.3 Configuring Lustre with a CSV File 6–4viii Lustre 1.8 Operations Manual • December 2010 7. More Complicated Configurations 7–1 7.1 Multihomed Servers 7–1 7.1.1 Modprobe.conf 7–1 7.1.2 Start Servers 7–3 7.1.3 Start Clients 7–4 7.2 Elan to TCP Routing 7–5 7.2.1 Modprobe.conf 7–5 7.2.2 Start servers 7–5 7.2.3 Start clients 7–5 7.3 Load Balancing with InfiniBand 7–6 7.3.1 Setting Up modprobe.conf for Load Balancing 7–6 7.4 Multi-Rail Configurations with LNET 7–7 8. Failover 8–1 8.1 What is Failover? 8–1 8.1.1 Failover Capabilities 8–2 8.1.2 Types of Failover Configurations 8–2 8.2 Failover Functionality in Lustre 8–3 8.2.1 MDT Failover Configuration (Active/Passive) 8–4 8.2.2 OST Failover Configuration (Active/Active) 8–4 8.2.3 Lustre Failover and MMP 8–4 8.2.3.1 Working with MMP 8–5Contents ix 8.3 Configuring and Using Heartbeat with Lustre Failover 8–6 8.3.1 Creating a Failover Environment 8–6 8.3.1.1 Power Management Software 8–6 8.3.1.2 Power Equipment 8–7 8.3.2 Setting up the Heartbeat Software 8–7 8.3.2.1 Installing Heartbeat 8–8 8.3.2.2 Configuring Heartbeat 8–8 8.3.2.3 (Optional) Migrating a Heartbeat Configuration (v1 to v2) 8–13 8.3.3 Working with Heartbeat 8–14 8.3.3.1 Starting Heartbeat 8–14 8.3.3.2 Switching Resources Between Nodes 8–14 9. Configuring Quotas 9–1 9.1 Working with Quotas 9–1 9.1.1 Enabling Disk Quotas 9–2 9.1.1.1 Administrative and Operational Quotas 9–3 9.1.2 Creating Quota Files and Quota Administration 9–4 9.1.3 Quota Allocation 9–7 9.1.4 Known Issues with Quotas 9–10 9.1.4.1 Granted Cache and Quota Limits 9–10 9.1.4.2 Quota Limits 9–11 9.1.4.3 Quota File Formats 9–11 9.1.5 Lustre Quota Statistics 9–12 9.1.5.1 Interpreting Quota Statistics 9–14x Lustre 1.8 Operations Manual • December 2010 10. RAID 10–1 10.1 Considerations for Backend Storage 10–2 10.1.1 Selecting Storage for the MDS or OSTs 10–2 10.1.2 Reliability Best Practices 10–3 10.1.3 Performance Tradeoffs 10–4 10.1.4 Formatting Options for RAID Devices 10–4 10.1.4.1 Creating an External Journal 10–5 10.1.5 Handling Degraded RAID Arrays 10–6 10.2 Insights into Disk Performance Measurement 10–6 10.3 Lustre Software RAID Support 10–7 10.3.0.1 Enabling Software RAID on Lustre 10–7 11. Kerberos 11–1 11.1 What is Kerberos? 11–1 11.2 Lustre Setup with Kerberos 11–2 11.2.1 Configuring Kerberos for Lustre 11–2 11.2.1.1 Kerberos Distributions Supported on Lustre 11–2 11.2.1.2 Preparing to Set Up Lustre with Kerberos 11–3 11.2.1.3 Configuring Lustre for Kerberos 11–4 11.2.1.4 Configuring Kerberos 11–6 11.2.1.5 Setting the Environment 11–8 11.2.1.6 Building Lustre 11–9 11.2.1.7 Running GSS Daemons 11–10Contents xi 11.2.2 Types of Lustre-Kerberos Flavors 11–11 11.2.2.1 Basic Flavors 11–11 11.2.2.2 Security Flavor 11–12 11.2.2.3 Customized Flavor 11–13 11.2.2.4 Specifying Security Flavors 11–14 11.2.2.5 Mounting Clients 11–14 11.2.2.6 Rules, Syntax and Examples 11–15 11.2.2.7 Authenticating Normal Users 11–16 12. Network Interface Bonding 12–1 12.1 Network Bonding 12–1 12.2 Requirements 12–2 12.3 Using Lustre with Multiple NICs versus Bonding NICs 12–4 12.4 Bonding Module Parameters 12–5 12.5 Setting Up Bonding 12–5 12.5.1 Examples 12–9 12.6 Configuring Lustre with Bonding 12–11 12.6.1 Bonding References 12–11 13. Upgrading and Downgrading Lustre 13–1 13.1 Supported Upgrades 13–2 13.2 Lustre Interoperability 13–2 13.3 Upgrading Lustre 1.6.x to 1.8.x 13–3 13.3.1 Performing a Complete File System Upgrade 13–4 13.3.2 Performing a Rolling Upgrade 13–6 13.4 Upgrading Lustre 1.8.x to the Next Minor Version 13–8 13.5 Downgrading from Lustre 1.8.x to 1.6.x 13–8 13.5.1 Performing a Complete File System Downgrade 13–9 13.5.2 Performing a Rolling Downgrade 13–11xii Lustre 1.8 Operations Manual • December 2010 14. Lustre SNMP Module 14–1 14.1 Installing the Lustre SNMP Module 14–2 14.2 Building the Lustre SNMP Module 14–2 14.3 Using the Lustre SNMP Module 14–3 15. Backup and Restore 15–1 15.1 Backing up a File System 15–1 15.2 Backing up a Device (MDS or OST) 15–2 15.2.1 Backing Up the MDS 15–2 15.2.2 Backing Up an OST 15–3 15.3 Backing up Files 15–4 15.3.1 Backing up Extended Attributes 15–4 15.4 Restoring from a File-level Backup 15–5 15.5 Using LVM Snapshots with Lustre 15–7 15.5.1 Creating an LVM-based Backup File System 15–7 15.5.2 Backing up New/Changed Files to the Backup File System 15–9 15.5.3 Creating Snapshot Volumes 15–9 15.5.4 Restoring the File System From a Snapshot 15–10 15.5.5 Deleting Old Snapshots 15–12 15.5.6 Changing Snapshot Volume Size 15–12 16. POSIX 16–1 16.1 Introduction to POSIX 16–1 16.2 Installing POSIX 16–2 16.2.1 POSIX Installation Using a Quick Start Version 16–2 16.3 Building and Running a POSIX Compliance Test Suite on Lustre 16–3 16.3.1 Building the Test Suite from Scratch 16–3 16.3.2 Running the Test Suite Against Lustre 16–5 16.4 Isolating and Debugging Failures 16–6Contents xiii 17. Benchmarking 17–1 17.1 Bonnie++ Benchmark 17–2 17.2 IOR Benchmark 17–3 17.3 IOzone Benchmark 17–5 18. Lustre I/O Kit 18–1 18.1 Lustre I/O Kit Description and Prerequisites 18–1 18.1.1 Downloading an I/O Kit 18–2 18.1.2 Prerequisites to Using an I/O Kit 18–2 18.2 Running I/O Kit Tests 18–2 18.2.1 sgpdd_survey 18–3 18.2.1.1 Tuning sgpdd_survey 18–4 18.2.2 obdfilter_survey 18–5 18.2.2.1 Running obdfilter_survey Against a Local Disk 18–6 18.2.2.2 Running obdfilter_survey Against a Network 18–7 18.2.2.3 Running obdfilter_survey Against a Network Disk 18–8 18.2.2.4 Output Files 18–9 18.2.2.5 Script Output 18–10 18.2.2.6 Visualizing Results 18–10 18.2.3 ost_survey 18–11 18.2.4 stats-collect 18–12 18.3 PIOS Test Tool 18–14 18.3.1 Synopsis 18–15 18.3.2 PIOS I/O Modes 18–16 18.3.3 PIOS Parameters 18–17 18.3.4 PIOS Examples 18–20xiv Lustre 1.8 Operations Manual • December 2010 18.4 LNET Self-Test 18–21 18.4.1 Basic Concepts of LNET Self-Test 18–21 18.4.1.1 Modules 18–21 18.4.1.2 Utilities 18–22 18.4.1.3 Session 18–22 18.4.1.4 Console 18–22 18.4.1.5 Group 18–23 18.4.1.6 Test 18–23 18.4.1.7 Batch 18–24 18.4.1.8 Sample Script 18–25 18.4.2 LNET Self-Test Commands 18–26 18.4.2.1 Session 18–26 18.4.2.2 Group 18–27 18.4.2.3 Batch and Test 18–30 18.4.2.4 Other Commands 18–33 19. Lustre Recovery 19–1 19.1 Recovery Overview 19–2 19.1.1 Client Failure 19–2 19.1.2 Client Eviction 19–3 19.1.3 MDS Failure (Failover) 19–3 19.1.4 OST Failure (Failover) 19–4 19.1.5 Network Partition 19–5 19.1.6 Failed Recovery 19–5Contents xv 19.2 Metadata Replay 19–6 19.2.1 XID Numbers 19–6 19.2.2 Transaction Numbers 19–6 19.2.3 Replay and Resend 19–7 19.2.4 Client Replay List 19–7 19.2.5 Server Recovery 19–8 19.2.6 Request Replay 19–9 19.2.7 Gaps in the Replay Sequence 19–9 19.2.8 Lock Recovery 19–10 19.2.9 Request Resend 19–10 19.3 Reply Reconstruction 19–11 19.3.1 Required State 19–11 19.3.2 Reconstruction of Open Replies 19–11 19.4 Version-based Recovery 19–13 19.4.1 Delayed Recovery 19–14 19.4.2 Working with VBR 19–15 19.4.3 Tips for Using VBR 19–15 19.5 Recovering from Errors or Corruption on a Backing File System 19–16 19.6 Recovering from Corruption in the Lustre File System 19–18 19.6.1 Working with Orphaned Objects 19–22xvi Lustre 1.8 Operations Manual • December 2010 Part III Lustre Tuning, Monitoring and Troubleshooting 20. Lustre Tuning 20–1 20.1 Module Options 20–2 20.1.1 OSS Service Thread Count 20–2 20.1.1.1 Optimizing the Number of Service Threads 20–2 20.1.2 MDS Service Thread Count 20–3 20.2 LNET Tunables 20–4 20.2.1 Transmit and receive buffer size: 20–4 20.2.2 irq_affinity 20–4 20.3 Options for Formatting the MDT and OSTs 20–5 20.3.1 Planning for Inodes 20–5 20.3.2 Sizing the MDT 20–5 20.4 Overriding Default Formatting Options 20–6 20.4.1 Number of Inodes for the MDS 20–6 20.4.2 Inode Size for the MDS 20–6 20.4.3 Number of Inodes for an OST 20–7 20.5 Large-Scale Tuning for Cray XT and Equivalents 20–8 20.5.1 Network Tunables 20–8 20.6 Lockless I/O Tunables 20–9 20.7 Data Checksums 20–10Contents xvii 21. LustreProc 21–1 21.1 Proc Entries for Lustre 21–2 21.1.1 Locating Lustre File Systems and Servers 21–2 21.1.2 Lustre Timeouts 21–3 21.1.3 Adaptive Timeouts 21–5 21.1.3.1 Configuring Adaptive Timeouts 21–6 21.1.3.2 Interpreting Adaptive Timeouts Information 21–8 21.1.4 LNET Information 21–9 21.1.5 Free Space Distribution 21–11 21.1.5.1 Managing Stripe Allocation 21–11 21.2 Lustre I/O Tunables 21–12 21.2.1 Client I/O RPC Stream Tunables 21–12 21.2.2 Watching the Client RPC Stream 21–14 21.2.3 Client Read-Write Offset Survey 21–15 21.2.4 Client Read-Write Extents Survey 21–17 21.2.5 Watching the OST Block I/O Stream 21–19 21.2.6 Using File Readahead and Directory Statahead 21–20 21.2.6.1 Tuning File Readahead 21–20 21.2.6.2 Tuning Directory Statahead 21–21 21.2.7 OSS Read Cache 21–22 21.2.7.1 Using OSS Read Cache 21–22 21.2.8 OSS Asynchronous Journal Commit 21–24 21.2.9 mballoc History 21–27 21.2.10 mballoc3 Tunables 21–29 21.2.11 Locking 21–31 21.2.12 Setting MDS and OSS Thread Counts 21–32xviii Lustre 1.8 Operations Manual • December 2010 21.3 Debug Support 21–34 21.3.1 RPC Information for Other OBD Devices 21–37 21.3.1.1 Interpreting OST Statistics 21–38 21.3.1.2 llobdstat 21–40 21.3.1.3 Interpreting MDT Statistics 21–40 22. Lustre Monitoring 22–1 22.1 Lustre Monitoring Tool 22–1 22.2 Red Hat Cluster Manager 22–2 22.3 SNMP Monitoring 22–2 22.4 CollectL 22–3 22.5 Other Monitoring Options 22–3 23. Lustre Troubleshooting 23–1 23.1 Troubleshooting Lustre 23–2 23.1.1 Error Numbers 23–2 23.1.2 Error Messages 23–3 23.1.3 Lustre Logs 23–3 23.2 Reporting a Lustre Bug 23–4 23.3 Common Lustre Problems and Performance Tips 23–5 23.3.1 Recovering from an Unavailable OST 23–5 23.3.2 Write Performance Better Than Read Performance 23–6 23.3.3 OST Object is Missing or Damaged 23–7 23.3.4 OSTs Become Read-Only 23–8 23.3.5 Identifying a Missing OST 23–8 23.3.6 Improving Lustre Performance When Working with Small Files 23–10 23.3.7 Default Striping 23–11 23.3.8 Erasing a File System 23–12 23.3.9 How to Fix a Bad LAST_ID on an OST 23–13Contents xix 23.3.10 Reclaiming Reserved Disk Space 23–16 23.3.11 Considerations in Connecting a SAN with Lustre 23–16 23.3.12 Handling/Debugging "Bind: Address already in use" Error 23–17 23.3.13 Replacing An Existing OST or MDS 23–18 23.3.14 Handling/Debugging Error "- 28" 23–18 23.3.15 Triggering Watchdog for PID NNN 23–19 23.3.16 Handling Timeouts on Initial Lustre Setup 23–20 23.3.17 Handling/Debugging "LustreError: xxx went back in time" 23–21 23.3.18 Lustre Error: "Slow Start_Page_Write" 23–21 23.3.19 Drawbacks in Doing Multi-client O_APPEND Writes 23–22 23.3.20 Slowdown Occurs During Lustre Startup 23–22 23.3.21 Log Message ‘Out of Memory’ on OST 23–22 23.3.22 Number of OSTs Needed for Sustained Throughput 23–23 23.3.23 Setting SCSI I/O Sizes 23–23 23.3.24 Identifying Which Lustre File an OST Object Belongs To 23–24 24. Lustre Debugging 24–1 24.1 Lustre Debug Messages 24–2 24.1.1 Format of Lustre Debug Messages 24–3 24.1.2 Lustre Debug Messages Buffer 24–3 24.2 Tools for Lustre Debugging 24–4 24.2.1 Debug Daemon Option to lctl 24–6 24.2.1.1 lctl Debug Daemon Commands 24–7 24.2.2 Controlling the Kernel Debug Log 24–8 24.2.3 The lctl Tool 24–8 24.2.4 Finding Memory Leaks 24–10 24.2.5 Printing to /var/log/messages 24–10 24.2.6 Tracing Lock Traffic 24–10xx Lustre 1.8 Operations Manual • December 2010 24.2.7 Sample lctl Run 24–11 24.2.8 Adding Debugging to the Lustre Source Code 24–11 24.3 Troubleshooting with strace 24–14 24.4 Looking at Disk Content 24–15 24.4.1 Determine the Lustre UUID of an OST 24–16 24.4.2 Tcpdump 24–16 24.5 Ptlrpc Request History 24–16 24.6 Using LWT Tracing 24–17 Part IV Lustre for Users 25. Striping and I/O Options 25–1 25.1 Lustre File Striping 25–2 25.1.1 Advantages of Striping 25–2 25.1.1.1 Bandwidth 25–2 25.1.2 Disadvantages of Striping 25–3 25.1.2.1 Increased Overhead 25–3 25.1.2.2 Increased Risk 25–3 25.1.3 Stripe Size 25–4 25.2 Setting and Retrieving Striping Information 25–5 25.2.1 Setting File Layouts 25–9 25.2.2 Changing Striping for a Subdirectory 25–9 25.2.3 Using a Specific Striping Pattern/File Layout for a Single File 25–10 25.2.4 Creating a File on a Specific OST 25–10Contents xxi 25.3 Managing Free Space 25–11 25.3.1 Checking File System Free Space 25–11 25.3.2 Using Stripe Allocations 25–13 25.3.3 Round-Robin Allocator 25–13 25.3.4 Weighted Allocator 25–13 25.3.5 Adjusting the Weighting Between Free Space and Location 25–14 25.4 Handling Full OSTs 25–14 25.4.1 Checking File System Usage 25–14 25.4.2 Taking a Full OST Offline 25–15 25.4.3 Migrating Data within a File System 25–16 25.5 Creating and Managing OST Pools 25–18 25.5.1 Working with OST Pools 25–19 25.5.1.1 Using the lfs Command with OST Pools 25–20 25.5.2 Tips for Using OST Pools 25–21 25.6 Performing Direct I/O 25–21 25.6.1 Making File System Objects Immutable 25–21 25.7 Other I/O Options 25–22 25.7.1 Lustre Checksums 25–22 25.7.1.1 Changing Checksum Algorithms 25–23 25.8 Striping Using llapi 25–24xxii Lustre 1.8 Operations Manual • December 2010 26. Lustre Security 26–1 26.1 Using ACLs 26–1 26.1.1 How ACLs Work 26–1 26.1.2 Using ACLs with Lustre 26–2 26.1.3 Examples 26–3 26.2 Using Root Squash 26–4 26.2.1 Configuring Root Squash 26–4 26.2.2 Enabling and Tuning Root Squash 26–5 26.2.3 Syntax Error Handling 26–6 27. Lustre Operating Tips 27–1 27.1 Adding an OST to a Lustre File System 27–2 27.2 A Simple Data Migration Script 27–3 27.3 Adding Multiple SCSI LUNs on Single HBA 27–5 27.4 Failures Running a Client and OST on the Same Machine 27–5 27.5 Improving Lustre Metadata Performance While Using Large Directories 27–6 Part V Reference 28. User Utilities (man1) 28–1 28.1 lfs 28–2 28.2 lfs_migrate 28–13 28.3 lfsck 28–16 28.4 Filefrag 28–18 28.5 Mount 28–20 28.6 Handling Timeouts 28–20Contents xxiii 29. Lustre Programming Interfaces (man2) 29–1 29.1 User/Group Cache Upcall 29–1 29.1.1 Name 29–1 29.1.2 Description 29–2 29.1.2.1 Primary and Secondary Groups 29–2 29.1.3 Parameters 29–3 29.1.4 Data structures 29–3 30. Setting Lustre Properties (man3) 30–1 30.1 Using llapi 30–1 30.1.1 llapi_file_create 30–1 30.1.2 llapi_file_get_stripe 30–4 30.1.3 llapi_file_open 30–5 30.1.4 llapi_quotactl 30–6 30.1.5 llapi_path2fid 30–9 31. Configuration Files and Module Parameters (man5) 31–1 31.1 Introduction 31–1 31.2 Module Options 31–2 31.2.1 LNET Options 31–3 31.2.1.1 Network Topology 31–3 31.2.1.2 networks ("tcp") 31–5 31.2.1.3 routes (“”) 31–5 31.2.1.4 forwarding ("") 31–7 31.2.2 SOCKLND Kernel TCP/IP LND 31–8 31.2.3 QSW LND 31–10 31.2.4 RapidArray LND 31–11 31.2.5 VIB LND 31–12 31.2.6 OpenIB LND 31–14xxiv Lustre 1.8 Operations Manual • December 2010 31.2.7 Portals LND (Linux) 31–15 31.2.8 Portals LND (Catamount) 31–17 31.2.9 MX LND 31–19 32. System Configuration Utilities (man8) 32–1 32.1 mkfs.lustre 32–2 32.2 tunefs.lustre 32–5 32.3 lctl 32–8 32.4 mount.lustre 32–16 32.5 Additional System Configuration Utilities 32–19 32.5.1 lustre_rmmod.sh 32–19 32.5.2 e2scan 32–20 32.5.3 Utilities to Manage Large Clusters 32–21 32.5.4 Application Profiling Utilities 32–21 32.5.5 More /proc Statistics for Application Profiling 32–22 32.5.6 Testing / Debugging Utilities 32–23 32.5.7 Flock Feature 32–28 32.5.7.1 Example 32–28 32.5.8 l_getgroups 32–29 32.5.9 llobdstat 32–30 32.5.10 llstat 32–31 32.5.11 lst 32–33 32.5.12 plot-llstat 32–35 32.5.13 routerstat 32–36 32.5.14 ll_recover_lost_found_objs 32–37Contents xxv 33. System Limits 33–1 33.1 Maximum Stripe Count 33–1 33.2 Maximum Stripe Size 33–2 33.3 Minimum Stripe Size 33–2 33.4 Maximum Number of OSTs and MDTs 33–2 33.5 Maximum Number of Clients 33–2 33.6 Maximum Size of a File System 33–3 33.7 Maximum File Size 33–3 33.8 Maximum Number of Files or Subdirectories in a Single Directory 33–3 33.9 MDS Space Consumption 33–4 33.10 Maximum Length of a Filename and Pathname 33–4 33.11 Maximum Number of Open Files for Lustre File Systems 33–5 33.12 OSS RAM Size 33–5 Glossary Glossary–1 Index Index–1xxvi Lustre 1.8 Operations Manual • December 2010xxiii Preface This operations manual provides detailed information and procedures to install, configure and tune the Lustre file system. The manual covers topics such as failover, quotas, striping and bonding. The Lustre manual also contains troubleshooting information and tips to improve Lustre operation and performance. Using UNIX Commands This document might not contain information about basic UNIX® commands and procedures such as shutting down the system, booting the system, and configuring devices. Refer to the following for this information: ¦ Software documentation that you received with your system ¦ Solaris™ Operating System documentation, which is at: http://docs.sun.comxxiv Lustre 1.8 Operations Manual • December 2010 Shell Prompts Typographic Conventions Note – Characters display differently depending on browser settings. If characters do not display correctly, change the character encoding in your browser to Unicode UTF-8. A '\' (backslash) continuation character is used to indicate that commands are too long to fit on one text line. Shell Prompt C shell machine-name% C shell superuser machine-name# Bourne shell and Korn shell $ Bourne shell and Korn shell superuser # Typeface Meaning Examples AaBbCc123 The names of commands, files, and directories; on-screen computer output Edit your.login file. Use ls -a to list all files. % You have mail. AaBbCc123 What you type, when contrasted with on-screen computer output % su Password: AaBbCc123 Book titles, new words or terms, words to be emphasized. Replace command-line variables with real names or values. Read Chapter 6 in the User’s Guide. These are called class options. You must be superuser to do this. To delete a file, type rm filename.Preface xxv Third-Party Web Sites Oracle is not responsible for the availability of third-party web sites mentioned in this document. Oracle does not endorse and is not responsible or liable for any content, advertising, products, or other materials that are available on or through such sites or resources. Oracle will not be responsible or liable for any actual or alleged damage or loss caused by or in connection with the use of or reliance on any such content, goods, or services that are available on or through such sites or resources. Documentation, Support, and Training These web sites provide additional resources: ¦ Documentation http://docs.sun.com/ ¦ Support http://www.sun.com/support/ ¦ Training http://www.sun.com/training/xxvi Lustre 1.8 Operations Manual • December 2010Revision History BookTitle Part Number Rev Date Comments Lustre 1.8 Operations Manual 821-0035-10 A December 2010 First release of Lustre 1.8 manual Lustre 1.8 Operations Manual 821-0035-10 B October 2009 Second release of Lustre 1.8 manual Lustre 1.8 Operations Manual 821-0035-11 A February 2010 Third release of Lustre 1.8 manual Lustre 1.8 Operations Manual 821-0035-11 B March 2010 Fourth release of Lustre 1.8 manual Lustre 1.8 Operations Manual 821-0035-12 A December 2010 Fifth release of Lustre 1.8 manual (1.8.5)PART I Lustre Architecture Lustre is a storage-architecture for clusters. The central component is the Lustre file system, a shared file system for clusters. The Lustre file system is currently available for Linux and provides a POSIX-compliant UNIX file system interface. The Lustre architecture is used for many different kinds of clusters. It is best known for powering seven of the ten largest high-performance computing (HPC) clusters in the world with tens of thousands of client systems, petabytes (PBs) of storage and hundreds of gigabytes per second (GB/sec) of I/O throughput. Many HPC sites use Lustre as a site-wide global file system, servicing dozens of clusters on an unprecedented scale.1-1 C H A P T E R 1 Introduction to Lustre This chapter describes Lustre software and components, and includes the following sections: ¦ Introducing the Lustre File System ¦ Lustre Components ¦ Lustre Systems ¦ Files in the Lustre File System ¦ Lustre Configurations ¦ Lustre Networking ¦ Lustre Failover and Rolling Upgrades These instructions assume you have some familiarity with Linux system administration, cluster systems and network technologies.1-2 Lustre 1.8 Operations Manual • December 2010 1.1 Introducing the Lustre File System Lustre is a storage architecture for clusters. The central component is the Lustre file system, which is available for Linux and provides a POSIX-compliant UNIX file system interface. The Lustre architecture is used for many different kinds of clusters. It is best known for powering seven of the ten largest high-performance computing (HPC) clusters worldwide, with tens of thousands of client systems, petabytes (PB) of storage and hundreds of gigabytes per second (GB/sec) of I/O throughput. Many HPC sites use Lustre as a site-wide global file system, serving dozens of clusters on an unprecedented scale. The scalability of a Lustre file system reduces the need to deploy many separate file systems (such as one for each cluster). This offers significant storage management advantages, for example, avoiding maintenance of multiple data copies staged on multiple file systems. Hand in hand with aggregating file system capacity with many servers, I/O throughput is also aggregated and scales with additional servers. Moreover, throughput (or capacity) can be easily adjusted by adding servers dynamically. Lustre has been integrated with several vendor’s kernels. We offer Red Hat Enterprise Linux (RHEL) and SUSE Linux Enterprise (SUSE) kernels with Lustre patches.Chapter 1 Introduction to Lustre 1-3 1.1.1 Lustre Key Features The key features of Lustre include: ¦ Scalability: Lustre scales up or down with respect to the number of client nodes, disk storage and bandwidth. Currently, Lustre is running in production environments with up to 26,000 client nodes, with many clusters in the 10,000-20,000 client range. Other Lustre installations provide aggregated disk storage and bandwidth of up to 1,000 OSTs running on more than 450 OSSs. Several Lustre file systems with a capacity of 1 PB or more (allowing storage of up to 2 billion files) have been in use since 2006. ¦ Performance: Lustre deployments in production environments currently offer performance of up to 100 GB/s. In a test environment, a performance of 130 GB/s and 13,000 creates/s has been sustained. Lustre single client node throughput has been measured at 2 GB/s (max) and OSS throughput at 2.5 GB/s (max). Lustre has been run at 240 GB/sec on the Spider file system at Oak Ridge National Laboratories. ¦ POSIX compliance: The full POSIX test suite passes on Lustre clients. In a cluster, POSIX compliance means that most operations are atomic and clients never see stale data or metadata. ¦ High-availability: Lustre offers shared storage partitions for OSS targets (OSTs), and a shared storage partition for the MDS target (MDT). ¦ Security: In Lustre, it is an option to have TCP connections only from privileged ports. Group membership handling is server-based. POSIX access control lists (ACLs) are supported. ¦ Open source: Lustre is licensed under the GNU GPL. Additionally, Lustre offers these features: ¦ Interoperability: Lustre runs on a variety of CPU architectures and mixed-endian clusters and interoperability between adjacent Lustre software releases. ¦ Access control list (ACL): Currently, the Lustre security model follows a UNIX file system, enhanced with POSIX ACLs. Noteworthy additional features include root squash and connecting from privileged ports only. ¦ Quotas: User and group quotas are available for Lustre. ¦ OSS addition: The capacity of a Lustre file system and aggregate cluster bandwidth can be increased without interrupting any operations by adding a new OSS with OSTs to the cluster. ¦ Controlled striping: The default stripe count and stripe size can be controlled in various ways. The file system has a default setting that is determined at format time. Directories can be given an attribute so that all files under that directory (and recursively under any sub-directory) have a striping pattern determined by the attribute. Finally, utilities and application libraries are provided to control the striping of an individual file at creation time.1-4 Lustre 1.8 Operations Manual • December 2010 ¦ Snapshots: Lustre file servers use volumes attached to the server nodes. The Lustre software includes a utility (using LVM snapshot technology) to create a snapshot of all volumes and group snapshots together in a snapshot file system that can be mounted with Lustre. ¦ Backup tools: Lustre 1.6 includes two utilities supporting backups. One tool scans file systems and locates files modified since a certain timeframe. This utility makes modified files’ pathnames available so they can be processed in parallel by other utilities (such as rsync) using multiple clients. Another useful tool is a modified version of GNU tar (gtar) which can back up and restore extended attributes (i.e. file striping and pool membership) for Lustre. 1 ¦ Other current features of Lustre are described in detail in this manual. Future features are described in the Lustre roadmap. 1. Files backed up using the modified version of gtar are restored per the backed up striping information. The backup procedure does not use default striping rules.Chapter 1 Introduction to Lustre 1-5 1.2 Lustre Components A Lustre file system consists of the following basic components (see FIGURE 1-1). ¦ Metadata Server (MDS) - The MDS server makes metadata stored in one or more MDTs available to Lustre clients. Each MDS manages the names and directories in the Lustre file system(s) and provides network request handling for one or more local MDTs. ¦ Metadata Target (MDT) - The MDT stores metadata (such as filenames, directories, permissions and file layout) on an MDS. Each file system has one MDT. An MDT on a shared storage target can be available to many MDSs, although only one should actually use it. If an active MDS fails, a passive MDS can serve the MDT and make it available to clients. This is referred to as MDS failover. ¦ Object Storage Servers (OSS): The OSS provides file I/O service, and network request handling for one or more local OSTs. Typically, an OSS serves between 2 and 8 OSTs, up to 8 TB each 2 . The MDT, OSTs and Lustre clients can run concurrently (in any mixture) on a single node. However, a typical configuration is an MDT on a dedicated node, two or more OSTs on each OSS node, and a client on each of a large number of compute nodes. ¦ Object Storage Target (OST): The OST stores file data (chunks of user files) as data objects on one or more OSSs. A single Lustre file system can have multiple OSTs, each serving a subset of file data. There is not necessarily a 1:1 correspondence between a file and an OST. To optimize performance, a file may be spread over many OSTs. A Logical Object Volume (LOV), manages file striping across many OSTs. ¦ Lustre clients: Lustre clients are computational, visualization or desktop nodes that run Lustre software that allows them to mount the Lustre file system. The Lustre client software consists of an interface between the Linux Virtual File System and the Lustre servers. Each target has a client counterpart: Metadata Client (MDC), Object Storage Client (OSC), and a Management Client (MGC). A group of OSCs are wrapped into a single LOV. Working in concert, the OSCs provide transparent access to the file system. Clients, which mount the Lustre file system, see a single, coherent, synchronized namespace at all times. Different clients can write to different parts of the same file at the same time, while other clients can read from the file. Lustre includes several additional components, LNET and the MGS, described in the following sections. 2. In Lustre 1.8.2, 16 TB OSTs are supported on RHEL 5 using specific RPMs (with ext4-based ldiskfs).1-6 Lustre 1.8 Operations Manual • December 2010 FIGURE 1-1 Lustre components in a basic cluster 1.2.1 Lustre Networking (LNET) Lustre Networking (LNET) is an API that handles metadata and file I/O data for file system servers and clients. LNET supports multiple, heterogeneous interfaces on clients and servers. LNET interoperates with a variety of network transports through Network Abstraction Layers (NAL). Lustre Network Drivers (LNDs) are available for a number of commodity and high-end networks, including Infiniband, TCP/IP, Quadrics Elan, Myrinet (MX and GM) and Cray. In clusters with a Lustre file system, servers and clients communicate with one another over a custom networking API known as Lustre Networking (LNET), while the disk storage behind the MDSs and OSSs is connected to these servers using traditional SAN technologies. Key features of LNET include: ¦ RDMA, when supported by underlying networks such as Elan, Myrinet and InfiniBand. ¦ Support for many commonly-used network types such as InfiniBand and IP. ¦ High availability and recovery features enabling transparent recovery in conjunction with failover servers. ¦ Simultaneous availability of multiple network types with routing between them.Chapter 1 Introduction to Lustre 1-7 1.2.2 Management Server (MGS) The MGS stores configuration information for all Lustre file systems in a cluster. Each Lustre target contacts the MGS to provide information, and Lustre clients contact the MGS to retrieve information. The MGS requires its own disk for storage. However, there is a provision that allows the MGS to share a disk ("co-locate") with a single MDT. The MGS is not considered "part" of an individual file system; it provides configuration information for all managed Lustre file systems to other Lustre components. 1.3 Lustre Systems Lustre components work together as coordinated systems to manage file and directory operations in the file system (see FIGURE 1-2). FIGURE 1-2 Lustre system interaction in a file system1-8 Lustre 1.8 Operations Manual • December 2010 The characteristics of the Lustre system include: At scale, the Lustre cluster can include up to 1,000 OSSs and 100,000 clients (see FIGURE 1-3). FIGURE 1-3 Lustre cluster at scale Typical number of systems Performance Required attached storage Desirable hardware characteristics Clients 1-100,000 1 GB/sec I/O, 1,000 metadata ops/sec None None OSS 1-1,000 500-2.5 GB/sec File system capacity/OSS count Good bus bandwidth MDS 2 (2-100 in future) 3,000-15,000 metadata ops/sec 1-2% of file system capacity Adequate CPU power, plenty of memoryChapter 1 Introduction to Lustre 1-9 1.4 Files in the Lustre File System Traditional UNIX disk file systems use inodes, which contain lists of block numbers where file data for the inode is stored. Similarly, for each file in a Lustre file system, one inode exists on the MDT. However, in Lustre, the inode on the MDT does not point to data blocks, but instead, points to one or more objects associated with the files. This is illustrated in FIGURE 1-4. These objects are implemented as files on the OST file systems and contain file data. FIGURE 1-4 MDS inodes point to objects, ext3 inodes point to data1-10 Lustre 1.8 Operations Manual • December 2010 FIGURE 1-5 shows how a file open operation transfers the object pointers from the MDS to the client when a client opens the file, and how the client uses this information to perform I/O on the file, directly interacting with the OSS nodes where the objects are stored. FIGURE 1-5 File open and file I/O in Lustre If only one object is associated with an MDS inode, that object contains all of the data in that Lustre file. When more than one object is associated with a file, data in the file is "striped" across the objects. The MDS knows the layout of each file, the number and location of the file's stripes. The clients obtain the file layout from the MDS. Client do I/O against the stripes of a file by communicating directly with the relevant OSTs. The benefits of the Lustre arrangement are clear. The capacity of a Lustre file system equals the sum of the capacities of the storage targets. The aggregate bandwidth available in the file system equals the aggregate bandwidth offered by the OSSs to the targets. Both capacity and aggregate I/O bandwidth scale simply with the number of OSSs.Chapter 1 Introduction to Lustre 1-11 1.4.1 Lustre File System and Striping Striping allows parts of files to be stored on different OSTs, as shown in FIGURE 1-6. A RAID 0 pattern, in which data is "striped" across a certain number of objects, is used; the number of objects is called the stripe_count. Each object contains "chunks" of data. When the "chunk" being written to a particular object exceeds the stripe_size, the next "chunk" of data in the file is stored on the next target. FIGURE 1-6 Files striped with a stripe count of 2 and 3 with different stripe sizes File striping presents several benefits. One is that the maximum file size is not limited by the size of a single target. Lustre can stripe files over up to 160 targets, and each target can support a maximum disk use of 8 TB3 by a file. This leads to a maximum disk use of 1.48 PB4 by a file. Note that the maximum file size is much larger (2^64 bytes), but the file cannot have more than 1.48 PB2 of allocated data; hence a file larger than 1.48 PB2 must have many sparse sections. While a single file can only be striped over 160 targets, Lustre file systems have been built with almost 5000 targets, which is enough to support a 40 PB file system. Another benefit of striped files is that the I/O bandwidth to a single file is the aggregate I/O bandwidth to the objects in a file and this can be as much as the bandwidth of up to 160 servers. 3. In Lustre 1.8.2, 16 TB on RHEL 5. 4. In Lustre 1.8.2, 2.96 PB on RHEL 5.1-12 Lustre 1.8 Operations Manual • December 2010 1.4.2 Lustre Storage The storage attached to the servers is partitioned, optionally organized with logical volume management (LVM) and formatted as file systems. Lustre OSS and MDS servers read, write and modify data in the format imposed by these file systems. 1.4.2.1 OSS Storage Each OSS can manage multiple object storage targets (OSTs), one for each volume; I/O traffic is load-balanced against servers and targets. An OSS should also balance network bandwidth between the system network and attached storage to prevent network bottlenecks. Depending on the server's hardware, an OSS typically serves between 2 and 25 targets, with each target up to 8 terabytes (TBs) in size. 1.4.2.2 MDS Storage For the MDS nodes, storage must be attached for Lustre metadata, for which 1-2 percent of the file system capacity is needed. The data access pattern for MDS storage is different from the OSS storage: the former is a metadata access pattern with many seeks and read-and-writes of small amounts of data, while the latter is an I/O access pattern, which typically involves large data transfers. High throughput to MDS storage is not important. Therefore, we recommend that a different storage type be used for the MDS (for example FC or SAS drives, which provide much lower seek times). Moreover, for low levels of I/O, RAID 5/6 patterns are not optimal, a RAID 0+1 pattern yields much better results. Lustre uses journaling file system technology on the targets, and for a MDS, an approximately 20 percent performance gain can sometimes be obtained by placing the journal on a separate device. Typically, the MDS requires CPU power; we recommend at least four processor cores.Chapter 1 Introduction to Lustre 1-13 1.4.3 Lustre System Capacity Lustre file system capacity is the sum of the capacities provided by the targets. As an example, 64 OSSs, each with two 8-TB targets, provide a file system with a capacity of nearly 1 PB. If this system uses sixteen 1-TB SATA disks, it may be possible to get 50 MB/sec from each drive, providing up to 800 MB/sec of disk bandwidth. If this system is used as storage backend with a system network like InfiniBand that supports a similar bandwidth, then each OSS could provide 800 MB/sec of end-to-end I/O throughput. Note that the OSS must provide inbound and outbound bus throughput of 800 MB/sec simultaneously. The cluster could see aggregate I/O bandwidth of 64x800, or about 50 GB/sec. Although the architectural constraints described here are simple, in practice it takes careful hardware selection, benchmarking and integration to obtain such results. In a Lustre file system, storage is only attached to server nodes, not to client nodes. If failover capability is desired, then this storage must be attached to multiple servers. In all cases, the use of storage area networks (SANs) with expensive switches can be avoided, because point-to-point connections between the servers and the storage arrays normally provide the simplest and best attachments.1-14 Lustre 1.8 Operations Manual • December 2010 1.5 Lustre Configurations Lustre file systems are easy to configure. First, the Lustre software is installed, and then MDT and OST partitions are formatted using the standard UNIX mkfs command. Next, the volumes carrying the Lustre file system targets are mounted on the server nodes as local file systems. Finally, the Lustre client systems are mounted (in a manner similar to NFS mounts). The configuration commands listed below are for the Lustre cluster shown in FIGURE 1-7. On the MDS (mds.your.org@tcp0): mkfs.lustre --mdt --mgs --fsname=large-fs /dev/sda mount -t lustre /dev/sda /mnt/mdt On OSS1: mkfs.lustre --ost --fsname=large-fs --mgsnode=mds.your.org@tcp0 /dev/sdb mount -t lustre /dev/sdb/mnt/ost1 On OSS2: mkfs.lustre --ost --fsname=large-fs --mgsnode=mds.your.org@tcp0 /dev/sdc mount -t lustre /dev/sdc/mnt/ost2 FIGURE 1-7 A simple Lustre clusterChapter 1 Introduction to Lustre 1-15 1.6 Lustre Networking In clusters with a Lustre file system, the system network connects the servers and the clients. The disk storage behind the MDSs and OSSs connects to these servers using traditional SAN technologies, but this SAN does not extend to the Lustre client system. Servers and clients communicate with one another over a custom networking API known as Lustre Networking (LNET). LNET interoperates with a variety of network transports through Network Abstraction Layers (NAL). Key features of LNET include: ¦ RDMA, when supported by underlying networks such as Elan, Myrinet and InfiniBand. ¦ Support for many commonly-used network types such as InfiniBand and IP. ¦ High availability and recovery features enabling transparent recovery in conjunction with failover servers. ¦ Simultaneous availability of multiple network types with routing between them. LNET includes LNDs to support many network type including: ¦ InfiniBand: OpenFabrics versions 1.0 and 1.2, Mellanox Gold, Cisco, Voltaire, and Silverstorm ¦ TCP: Any network carrying TCP traffic, including GigE, 10GigE, and IPoIB ¦ Quadrics: Elan3, Elan4 ¦ Myrinet: GM, MX ¦ Cray: Seastar, RapidArray The LNDs that support these networks are pluggable modules for the LNET software stack. LNET offers extremely high performance. It is common to see end-to-end throughput over GigE networks in excess of 110 MB/sec, InfiniBand double data rate (DDR) links reach bandwidths up to 1.5 GB/sec, and 10GigE interfaces provide end-to-end bandwidth of over 1 GB/sec.1-16 Lustre 1.8 Operations Manual • December 2010 1.7 Lustre Failover and Rolling Upgrades Lustre offers a robust, application-transparent failover mechanism that delivers call completion. This failover mechanism, in conjunction with software that offers interoperability between versions, is used to support rolling upgrades of file system software on active clusters. The Lustre recovery feature allows servers to be upgraded without taking down the system. The server is simply taken offline, upgraded and restarted (or failed over to a standby server with the new software). All active jobs continue to run without failures, they merely experience a delay. Lustre MDSs are configured as an active/passive pair, while OSSs are typically deployed in an active/active configuration that provides redundancy without extra overhead, as shown in FIGURE 1-8. Often the standby MDS is the active MDS for another Lustre file system, so no nodes are idle in the cluster. FIGURE 1-8 Lustre failover configurations for OSSs and MDSsChapter 1 Introduction to Lustre 1-17 Although a file system checking tool (lfsck) is provided for disaster recovery, journaling and sophisticated protocols re-synchronize the cluster within seconds, without the need for a lengthy fsck. Lustre version interoperability between successive minor versions is guaranteed. As a result, the Lustre failover capability is used regularly to upgrade the software without cluster downtime. Note – Lustre does not provide redundancy for data; it depends exclusively on redundancy of backing storage devices. The backing OST storage should be RAID 5 or, preferably, RAID 6 storage. MDT storage should be RAID 1 or RAID 0+1.1-18 Lustre 1.8 Operations Manual • December 20102-1 C H A P T E R 2 Understanding Lustre Networking This chapter describes Lustre Networking (LNET) and supported networks, and includes the following sections: ¦ Introduction to LNET ¦ Supported Network Types ¦ Designing Your Lustre Network ¦ Configuring LNET ¦ Starting and Stopping LNET 2.1 Introduction to LNET In a Lustre network, servers and clients communicate with one another using LNET, a custom networking API which abstracts away all transport-specific interaction. In turn, LNET operates with a variety of network transports through Lustre Network Drivers (LNDs). The following terms are important to understanding LNET. ¦ LND: Lustre Network Driver. A modular sub-component of LNET that implements one of the network types. LNDs are implemented as individual kernel modules (or a library in userspace) and, typically, must be compiled against the network driver software. ¦ Network: A group of nodes that communicate directly with each other. The network is how LNET represents a single cluster. Multiple networks can be used to connect clusters together. Each network has a unique type and number (for example, tcp0, tcp1, or elan0). ¦ NID: Lustre Network Identifier. The NID uniquely identifies a Lustre network endpoint, including the node and the network type. There is an NID for every network which a node uses.2-2 Lustre 1.8 Operations Manual • December 2010 Key features of LNET include: ¦ RDMA, when supported by underlying networks such as Elan, Myrinet, and InfiniBand ¦ Support for many commonly-used network types such as InfiniBand and TCP/IP ¦ High availability and recovery features enabling transparent recovery in conjunction with failover servers ¦ Simultaneous availability of multiple network types with routing between them LNET is designed for complex topologies, superior routing capabilities and simplified configuration. 2.2 Supported Network Types LNET supports the following network types: ¦ TCP ¦ openib (Mellanox-Gold InfiniBand) ¦ cib (Cisco Topspin) ¦ iib (Infinicon InfiniBand) ¦ vib (Voltaire InfiniBand) ¦ o2ib (OFED - InfiniBand and iWARP) ¦ ra (RapidArray) ¦ Elan (Quadrics Elan) ¦ GM and MX (Myrinet) ¦ Cray SeastarChapter 2 Understanding Lustre Networking 2-3 2.3 Designing Your Lustre Network Before you configure Lustre, it is essential to have a clear understanding of the Lustre network topologies. 2.3.1 Identify All Lustre Networks A network is a group of nodes that communicate directly with one another. As previously mentioned in this manual, Lustre supports a variety of network types and hardware, including TCP/IP, Elan, varieties of InfiniBand, Myrinet and others. The normal rules for specifying networks apply to Lustre networks. For example, two TCP networks on two different subnets (tcp0 and tcp1) would be considered two different Lustre networks. 2.3.2 Identify Nodes to Route Between Networks Any node with appropriate interfaces can route LNET between different networks—the node may be a server, a client, or a standalone router. LNET can route across different network types (such as TCP-to-Elan) or across different topologies (such as bridging two InfiniBand or TCP/IP networks). 2.3.3 Identify Network Interfaces to Include/Exclude from LNET If not explicitly specified, LNET uses either the first available interface or a pre-defined default for a given network type. If there are interfaces that LNET should not use (such as administrative networks, IP over IB, and so on), then the included interfaces should be explicitly listed.2-4 Lustre 1.8 Operations Manual • December 2010 2.3.4 Determine Cluster-wide Module Configuration The LNET configuration is managed via module options, typically specified in /etc/modprobe.conf or /etc/modprobe.conf.local (depending on the distribution). To ease the maintenance of large clusters, you can configure the networking setup for all nodes using a single, unified set of options in the modprobe.conf file on each node. For more information, see the ip2nets option in Setting Up modprobe.conf for Load Balancing. Users of liblustre should set the accept=all parameter. For details, see Module Parameters. 2.3.5 Determine Appropriate Mount Parameters for Clients In mount commands, clients use the NID of the MDS host to retrieve their configuration information. Since an MDS may have more than one NID, a client should use the appropriate NID for its local network. If you are unsure which NID to use, there is a lctl command that can help. MDS On the MDS, run: lctl list_nids This displays the server's NIDs (networks configured to work with Lustre). Client On a client, run: lctl which_nid This displays the closest NID for the client.Chapter 2 Understanding Lustre Networking 2-5 Client with SSH Access From a client with SSH access to the MDS, run: mds_nids=`ssh the_mds lctl list_nids` lctl which_nid $mds_nids This displays, generally, the correct NID to use for the MDS in the mount command. Note – In the mds_nids command above, be sure to use the correct mark (`), not a straight quotation mark ('). Otherwise, the command will not work. 2.4 Configuring LNET This section describes how to configure LNET, including entries in the modprobe.conf file which tell LNET which NIC(s) will be configured to work with Lustre, and parameters that specify the routing that will be used with Lustre. Note – We recommend that you use dotted-quad IP addressing rather than host names. We have found this aids in reading debug logs, and helps greatly when debugging configurations with multiple interfaces. 2.4.1 Module Parameters LNET hardware and routing are configured via module parameters of the LNET and LND-specific modules. Parameters should be specified in the /etc/modprobe.conf or /etc/modules.conf file. This example specifies that the node should use a TCP interface and an Elan interface: options lnet networks=tcp0,elan0 Depending on the LNDs used, it may be necessary to specify explicit interfaces. For example, if you want to use two TCP interfaces (tcp0 and tcp1, for example), it is necessary to specify the module parameters and ethX interfaces, like this: options lnet networks=tcp0(eth0),tcp1(eth1) This modprobe.conf entry specifies: ¦ First Lustre network, tcp0, is configured on interface eth0 ¦ Second Lustre network, tcp1, is configured on interface eth12-6 Lustre 1.8 Operations Manual • December 2010 Note – The requirement to specify explicit interfaces is not consistent across all LNDs used with Lustre, and LND behavior may change over time. We recommend that if your multi-homed settings do not work, try specifying the ethX interfaces in the options lnet networks line. All LNET routers that bridge two networks are equivalent; their configuration is not primary or secondary. All available routers balance their overall load. With the router checker configured, Lustre nodes can detect router health status, avoid those that appear dead, and reuse the ones that restore service after failures. To do this, LNET routing must correspond exactly with the Linux nodes' map of alive routers. There is no hard limit on the number of LNET routers. Note – When multiple interfaces are available during the network setup, Lustre choose the 'best' route. Once the network connection is established, Lustre expects the network to stay connected. In a Lustre network, connections do not fail over to the other interface, even if multiple interfaces are available on the same node. Under Linux 2.6, the LNET configuration parameters can be viewed under /sys/module/; generic and acceptor parameters under lnet and LND-specific parameters under the corresponding LND name. Note – Depending on the Linux distribution, options with included commas may need to be escaped using single and/or double quotes. Worst-case quotes look like: options lnet'networks="tcp0,elan0"' 'routes="tcp [2,10]@elan0"' Additional quotes may confuse some distributions. Check for messages such as: lnet: Unknown parameter ‘'networks' After modprobe LNET, remove the additional single quotes (modprobe.conf in this case). Additionally, the refusing connection - no matching NID message generally points to an error in the LNET module configuration. Note – By default, Lustre ignores the loopback (lo0) interface. Lustre does not ignore IP addresses aliased to the loopback. In this case, specify all Lustre networks. The liblustre network parameters may be set by exporting the environment variables LNET_NETWORKS, LNET_IP2NETS and LNET_ROUTES. Each of these variables uses the same parameters as the corresponding modprobe option.Chapter 2 Understanding Lustre Networking 2-7 Note, it is very important that a liblustre client includes ALL the routers in its setting of LNET_ROUTES. A liblustre client cannot accept connections, it can only create connections. If a server sends remote procedure call (RPC) replies via a router to which the liblustre client has not already connected, then these RPC replies are lost. Note – Liblustre is not required or even recommended for running Lustre on Linux. Most users will not use liblustre. Instead, you should use the Lustre (VFS) client file system to mount Lustre directly. Liblustre does NOT support multi-threaded applications. Note – Liblustre is not widely tested as part of Lustre release testing, and is currently maintained only as a courtesy to the Lustre community. 2.4.1.1 Using Usocklnd Lustre now offers usocklnd, a socket-based LND that uses TCP in userspace. By default, liblustre is compiled with usocklnd as the transport, so there is no need to specially enable it. Use the following environmental variables to tune usocklnd’s behavior. configuration. Variable Description USOCK_SOCKNAGLE=N Turns the TCP Nagle algorithm on or off. Setting N to 0 (the default value), turns the algorithm off. Setting N to 1 turns the algorithm on. USOCK_SOCKBUFSIZ=N Changes the socket buffer size. Setting N to 0 (the default value), specifies the default socket buffer size. Setting N to another value (must be a positive integer) causes usocklnd to try to set the socket buffer size to the specified value. USOCK_TXCREDITS=N Specifies the maximum number of concurrent sends. The default value is 256. N should be set to a positive value. USOCK_PEERTXCREDITS=N Specifies the maximum number of concurrent sends per peer. The default value is 8. N should be set to a positive value and should not be greater than the value of the USOCK_TXCREDITS parameter. USOCK_NPOLLTHREADS=N Defines the degree of parallelism of usocklnd, by equaling the number of threads devoted to processing network events. The default value is the number of CPUs in the system. N should be set to a positive value.2-8 Lustre 1.8 Operations Manual • December 2010 2.4.1.2 OFED InfiniBand Options For the SilverStorm/Infinicon InfiniBand LND (iiblnd), the network and HCA may be specified, as in this example: options lnet networks="o2ib3(ib3)" This specifies that the node is on o2ib network number 3, using HCA ib3. 2.4.2 Module Parameters - Routing The following parameter specifies a colon-separated list of router definitions. Each route is defined as a network number, followed by a list of routers. route= Examples: options lnet 'networks="o2ib0"' 'routes="tcp0 192.168.10.[1-8]@o2ib0"' This is an example for IB clients to access TCP servers via 8 IB-TCP routers. options lnet 'ip2nets="tcp0 10.10.0.*; o2ib0(ib0) 192.168.10.[1-128]"' \ 'routes="tcp 192.168.10.[1-8]@o2ib0; o2ib 10.10.0.[1-8]@tcp0" USOCK_FAIR_LIMIT=N The maximum number of times that usocklnd loops processing events before the next polling occurs. The default value is 1, meaning that every network event has only one chance to be processed before polling occurs the next time. N should be set to a positive value. USOCK_TIMEOUT=N Specifies the network timeout (measured in seconds). Network options that are not completed in N seconds time out and are canceled. The default value is 50 seconds. N should be a positive value. USOCK_POLL_TIMEOUT=N Specifies the polling timeout; how long usocklnd ‘sleeps’ if no network events occur. N results in a slightly lower overhead of checking network timeouts and longer delay of evicting timed-out events. The default value is 1 second. N should be set to a positive value. USOCK_MIN_BULK=N This tunable is only used for typed network connections. Currently, liblustre clients do not use this usocklnd facility. Variable DescriptionChapter 2 Understanding Lustre Networking 2-9 This specifies bi-directional routing; TCP clients can reach Lustre resources on the IB networks and IB servers can access the TCP networks. For more information on ip2nets, Modprobe.conf. Note – Configure IB network interfaces on a different subnet than LAN interfaces. Best Practices for ip2nets, routes and networks Options For the ip2nets, routes and networks options, several best practices must be followed or configuration errors occur. Best Practice 1: If you add a comment to any of the above options, position the semicolon after the comment. If you fail to do so, some nodes are not properly initialized because LNET silently ignores everything following the '#' character (which begins the comment), until it reaches the next semicolon. This is subtle; no error message is generated to alert you to the problem. This example shows the correct syntax: options lnet ip2nets="pt10 192.168.0.[89,93] # comment with semicolon AFTER comment; \ pt11 192.168.0.[92,96] # comment In this example, the following is ignored: comment with semicolon AFTER comment This example shows the wrong syntax: options lnet ip2nets="pt10 192.168.0.[89,93]; # comment with semicolon BEFORE comment \ pt11 192.168.0.[92,96]; In this example, the following is ignored: comment with semicolon BEFORE comment pt11 192.168.0.[92,96]. Because LNET silently ignores pt11 192.168.0.[92,96], these nodes are not properly initialized. Best Practice 2: Do not add an excessive number of comments to these options. The Linux kernel has a limit on the length of string module options; it is usually 1KB, but may differ in vendor kernels. If you exceed this limit, errors result and the configuration specified by the user is not processed properly. Using Routing Parameters Across a Cluster To ease Lustre administration, the same routing parameters can be used across different parts of a routed cluster. For example, the bi-directional routing example above can be used on an entire cluster (TCP clients, TCP-IB routers, and IB servers): ¦ TCP clients would ignore o2ib0(ib0) 192.168.10.[1-128] in ip2nets since they have no such interfaces. Similarly, IB servers would ignore tcp0 192.168.0.*. But TCP-IB routers would use both since they are multi-homed.2-10 Lustre 1.8 Operations Manual • December 2010 ¦ TCP clients would ignore the route "tcp 192.168.10.[1-8]@o2ib0" since the target network is a local network. For the same reason, IB servers would ignore "o2ib 10.10.0.[1-8]@tcp0". ¦ TCP-IB routers would ignore both routes, because they are multi-homed. Moreover, the routers would enable LNet forwarding since their NIDs are specified in the 'routes' parameters as being routers. live_router_check_interval, dead_router_check_interval, auto_down, check_routers_before_use and router_ping_timeout In a routed Lustre setup with nodes on different networks such as TCP/IP and Elan, the router checker checks the status of a router. The auto_down parameter enables/disables (1/0) the automatic marking of router state. The live_router_check_interval parameter specifies a time interval in seconds after which the router checker will ping the live routers. In the same way, you can set the dead_router_check_interval parameter for checking dead routers. You can set the timeout for the router checker to check the live or dead routers by setting the router_ping_timeout parameter. The Router pinger sends a ping message to a dead/live router once every dead/live_router_check_interval seconds, and if it does not get a reply message from the router within router_ping_timeout seconds, it considers the router to be down. The last parameter is check_routers_before_use, which is off by default. If it is turned on, you must also give dead_router_check_interval a positive integer value. The router checker gets the following variables for each router: ¦ Last time that it was disabled ¦ Duration of time for which it is disabled The initial time to disable a router should be one minute (enough to plug in a cable after removing it). If the router is administratively marked as "up", then the router checker clears the timeout. When a route is disabled (and possibly new), the "sent packets" counter is set to 0. When the route is first re-used (that is an elapsed disable time is found), the sent packets counter is incremented to 1, and incremented for all further uses of the route. If the route has been used for 100 packets successfully, then the sent-packets counter should be with a value of 100. Set the timeout to 0 (zero), so future errors no longer double the timeout.Chapter 2 Understanding Lustre Networking 2-11 Note – The router_ping_timeout is consistent with the default LND timeouts. You may have to increase it on very large clusters if the LND timeout is also increased. For larger clusters, we suggest increasing the check interval. 2.4.2.1 LNET Routers All LNET routers that bridge two networks are equivalent. They are not configured as primary or secondary, and load is balanced across all available routers. With the router checker configured, Lustre nodes can detect router health status, avoid those that appear dead, and reuse the ones that restore service after failures. There are no hard requirements regarding the number of LNET routers, although there should enough to handle the required file serving bandwidth (and a 25% margin for headroom). Comparing 32-bit and 64-bit LNET Routers By default, at startup, LNET routers allocate 544M (i.e. 139264 4K pages) of memory as router buffers. The buffers can only come from low system memory (i.e. ZONE_DMA and ZONE_NORMAL). On 32-bit systems, low system memory is, at most, 896M no matter how much RAM is installed. The size of the default router buffer puts big pressure on low memory zones, making it more likely that an out-of-memory (OOM) situation will occur. This is a known cause of router hangs. Lowering the value of the large_router_buffers parameter can circumvent this problem, but at the cost of penalizing router performance, by making large messages wait for longer for buffers. On 64-bit architectures, the ZONE_HIGHMEM zone is always empty. Router buffers can come from all available memory and out-of-memory hangs do not occur. Therefore, we recommend using 64-bit routers.2-12 Lustre 1.8 Operations Manual • December 2010 2.4.3 Downed Routers There are two mechanisms to update the health status of a peer or a router: ¦ LNET can actively check health status of all routers and mark them as dead or alive automatically. By default, this is off. To enable it set auto_down and if desired check_routers_before_use. This initial check may cause a pause equal to router_ping_timeout at system startup, if there are dead routers in the system. ¦ When there is a communication error, all LNDs notify LNET that the peer (not necessarily a router) is down. This mechanism is always on, and there is no parameter to turn it off. However, if you set the LNET module parameter auto_down to 0, LNET ignores all such peer-down notifications. Several key differences in both mechanisms: ¦ The router pinger only checks routers for their health, while LNDs notices all dead peers, regardless of whether they are a router or not. ¦ The router pinger actively checks the router health by sending pings, but LNDs only notice a dead peer when there is network traffic going on. ¦ The router pinger can bring a router from alive to dead or vice versa, but LNDs can only bring a peer down.Chapter 2 Understanding Lustre Networking 2-13 2.5 Starting and Stopping LNET Lustre automatically starts and stops LNET, but it can also be manually started in a standalone manner. This is particularly useful to verify that your networking setup is working correctly before you attempt to start Lustre. 2.5.1 Starting LNET To start LNET, run: $ modprobe lnet $ lctl network up To see the list of local NIDs, run: $ lctl list_nids This command tells you the network(s) configured to work with Lustre If the networks are not correctly setup, see the modules.conf "networks=" line and make sure the network layer modules are correctly installed and configured. To get the best remote NID, run: $ lctl which_nid where is the list of available NIDs. This command takes the "best" NID from a list of the NIDs of a remote host. The "best" NID is the one that the local node uses when trying to communicate with the remote node. 2.5.1.1 Starting Clients To start a TCP client, run: mount -t lustre mdsnode:/mdsA/client /mnt/lustre/ To start an Elan client, run: mount -t lustre 2@elan0:/mdsA/client /mnt/lustre2-14 Lustre 1.8 Operations Manual • December 2010 2.5.2 Stopping LNET Before the LNET modules can be removed, LNET references must be removed. In general, these references are removed automatically when Lustre is shut down, but for standalone routers, an explicit step is needed to stop LNET. Run: lctl network unconfigure Note – Attempting to remove Lustre modules prior to stopping the network may result in a crash or an LNET hang. if this occurs, the node must be rebooted (in most cases). Make sure that the Lustre network and Lustre are stopped prior to unloading the modules. Be extremely careful using rmmod -f. To unconfigure the LNET network, run: modprobe -r Tip – To remove all Lustre modules, run: $ lctl modules | awk '{print $2}' | xargs rmmodPART II Lustre Administration Lustre administration includes the steps necessary to meet pre-installation requirements, and install and configure Lustre. It also includes advanced topics such as failover, quotas, bonding, benchmarking, Kerberos and POSIX.3-1 C H A P T E R 3 Installing Lustre Lustre installation involves two procedures, meeting the installation prerequisites and installing the Lustre software, either from RPMs or from source code. This chapter includes these sections: ¦ Preparing to Install Lustre ¦ Installing Lustre from RPMs ¦ Installing Lustre from Source Code Lustre can be installed from either packaged binaries (RPMs) or freely-available source code. Installing from the package release is straightforward, and recommended for new users. Integrating Lustre into an existing kernel and building the associated Lustre software is an involved process. For either installation method, the following are required: ¦ Linux kernel patched with Lustre-specific patches ¦ Lustre modules compiled for the Linux kernel ¦ Lustre utilities required for Lustre configuration Note – When installing Lustre and creating components on devices, a certain amount of space is reserved, so less than 100% of storage space will be available. Lustre servers use the ext3 file system to store user-data objects and system data. By default, ext3 file systems reserve 5% of space that cannot be used by Lustre. Additionally, Lustre reserves up to 400 MB on each OST for journal use 1 . This reserved space is unusable for general storage. For this reason, you will see up to 400 MB of space used on each OST before any file object data is saved to it. 1. Additionally, a few bytes outside the journal are used to create accounting data for Lustre.3-2 Lustre 1.8 Operations Manual • December 2010 3.1 Preparing to Install Lustre To successfully install and run Lustre, make sure the following installation prerequisites have been met: ¦ Supported Operating System, Platform and Interconnect ¦ Required Lustre Software ¦ Required Tools and Utilities ¦ (Optional) High-Availability Software ¦ Debugging Tools ¦ Environmental Requirements ¦ Memory RequirementsChapter 3 Installing Lustre 3-3 3.1.1 Supported Operating System, Platform and Interconnect Lustre 1.8 supports the following operating systems, platforms 2 and interconnects. To install Lustre from downloaded packages (RPMs), you must use a supported configuration. Note – Lustre clients running on architectures with different endianness are supported. One limitation is that the PAGE_SIZE kernel macro on the client must be as large as the PAGE_SIZE of the server. In particular, ia64 clients with large pages (up to 64kB pages) can run with i386 servers (4kB pages). If you are running i386 clients with ia64 servers, you must compile the ia64 kernel with a 4kB PAGE_SIZE (so the server page size is not larger than the client page size). 2. We encourage the use of 64-bit platforms. Configuration Component Supported Type Operating system OEL 5.4, i686 and x86_64 only (Lustre 1.8.2) OEL 5.3, i686 and x86_64 only (Lustre 1.8.1.1 and 1.8.2) Red Hat Enterprise Linux 5 SuSE Linux Enterprise Server 10 SuSE Linux Enterprise Server 11, i686 and x86_64 only (Lustre 1.8.1 and later) Linux kernel 2.6.16 or greater NOTE: Lustre does not support security-enhanced (SE) Linux (including clients and servers). Platform x86, IA-64, x86-64 (EM64 and AMD64) PowerPC architectures (for clients only) and mixed-endian clusters Interconnect TCP/IP Quadrics Elan 3 and 4 Myri-10G and Myrinet-2000 Mellanox InfiniBand (Voltaire, OpenIB, Silverstorm and any OFED-supported InfiniBand adapter)3-4 Lustre 1.8 Operations Manual • December 2010 3.1.2 Required Lustre Software To install Lustre, the following are required: ¦ Linux kernel patched with Lustre-specific patches (the patched Linux kernel is required only on the Lustre MDS and OSSs) ¦ Lustre modules compiled for the Linux kernel ¦ Lustre utilities required for Lustre configuration ¦ Lustre-specific tools (e2fsck and lfsck) used to repair a backing file system, available in the e2fsprogs package ¦ (Optional) Network-specific kernel modules and libraries (for example, kernel modules and libraries required for an InfiniBand interconnect) 3.1.3 Required Tools and Utilities Several third-party utilities are required: ¦ e2fsprogs: Lustre requires a recent version of e2fsprogs that understands extents. Use e2fsprogs-1.41-6 or later, available at: http://downloads.lustre.org/public/tools/e2fsprogs/ A quilt patchset of all changes to the vanilla e2fsprogs is available in e2fsprogs-{version}-patches.tgz. Note – Lustre-patched e2fsprogs utility only needs to be installed on machines that mount backend (ldiskfs) file systems, such as the OSS, MDS and MGS nodes. It does not need to be loaded on clients. ¦ Perl - Various userspace utilities are written in Perl. Any recent version of Perl will work with Lustre. 3.1.4 (Optional) High-Availability Software If you plan to enable failover server functionality with Lustre (either on an OSS or the MDS), you must add high-availability (HA) software to your cluster software. You can use any HA software package with Lustre. 3 For more information, see Failover. 3. In this manual, the Linux-HA (Heartbeat) package is referenced, but you can use any HA software.Chapter 3 Installing Lustre 3-5 3.1.5 Debugging Tools Lustre is a complex system and you may encounter problems when using it. You should have debugging tools on hand to help figure out how and why a problem occurred. A variety of diagnostic and analysis tools are available to debug issues with the Lustre software. Some of these are provided in Linux distributions, while others have been developed and are made available by the Lustre project. These in-kernel debug mechanisms are incorporated into the Lustre software: ¦ Debug logs ¦ Debug daemon ¦ /proc/sys/lnet/debug These tools are also provided with the Lustre software: ¦ lctl ¦ Lustre subsystem asserts ¦ lfs These general debugging tools are provided as a part of the standard Linux distribution: ¦ strace ¦ /var/log/messages ¦ Crash dumps ¦ debugfs These logging and data collection tools can be used to collect information for debugging Lustre kernel issues: ¦ kdump ¦ netconsole ¦ netdump To debug Lustre in a development environment, use: ¦ leak_finder.pl A variety of debuggers and analysis tools are available including: ¦ kgdb ¦ crash For detailed information about these debugging tools, see Tools for Lustre Debugging.3-6 Lustre 1.8 Operations Manual • December 2010 3.1.6 Environmental Requirements Make sure the following environmental requirements are met before installing Lustre: ¦ (Recommended) Provide remote shell access to clients. Although not strictly required to run Lustre, we recommend that all cluster nodes have remote shell client access, to facilitate the use of Lustre configuration and monitoring scripts. Parallel Distributed SHell (pdsh) is preferable, although Secure SHell (SSH) is acceptable. ¦ Ensure client clocks are synchronized. Lustre uses client clocks for timestamps. If clocks are out-of-sync between clients and servers, timeouts and client evictions will occur. Drifting clocks can also cause problems by, for example, making it difficult to debug multi-node issues or correlate logs, which depend on timestamps. We recommend that you use Network Time Protocol (NTP) to keep client and server clocks in sync with each other. For more information about NTP, see: http://www.ntp.org. ¦ Maintain uniform file access permissions on all cluster nodes. Use the same user IDs (UID) and group IDs (GID) on all clients. If use of supplemental groups is required, verify that the group_upcall requirements have been met. See User/Group Cache Upcall. ¦ (Recommended) Disable Security-Enhanced Linux (SELinux) on servers and clients. Lustre does not support SELinux. Therefore, disable the SELinux system extension on all Lustre nodes and make sure other security extensions, like Novell AppArmorand network packet filtering tools (such as iptables) do not interfere with Lustre.Chapter 3 Installing Lustre 3-7 3.1.7 Memory Requirements This section describes the memory requirements of Lustre. 3.1.7.1 MDS Memory Requirements MDS memory requirements are determined by the following factors: ¦ Number of clients ¦ Size of the directories ¦ Extent of load The amount of memory used by the MDS is a function of how many clients are on the system, and how many files they are using in their working set. This is driven, primarily, by the number of locks a client can hold at one time. The default maximum number of locks for a compute node is 100*num_cores, and interactive clients can hold in excess of 10,000 locks at times. For the MDS, this works out to approximately 2 KB per file, including the Lustre DLM lock and kernel data structures for it, just for the current working set. There is, by default, 400 MB for the file system journal, and additional RAM usage for caching file data for the larger working set that is not actively in use by clients, but should be kept "HOT" for improved access times. Having file data in cache can improve metadata performance by a factor of 10x or more compared to reading it from disk. Approximately 1.5 KB/file is needed to keep a file in cache. For example, for a single MDT on an MDS with 1,000 clients, 16 interactive nodes, and a 2 million file working set (of which 400,000 files are cached on the clients): File system journal = 400 MB 1000 * 4-core clients * 100 files/core * 2kB = 800 MB 16 interactive clients * 10,000 files * 2kB = 320 MB 1,600,000 file extra working set * 1.5kB/file = 2400 MB Thus, the minimum requirement for a system with this configuration is 4-GB RAM. However, additional memory may significantly improve performance 4 . If there are directories containing 1 million or more files, you may benefit significantly from having more memory. For example, in an environment where clients randomly access one of 10 million files, having extra memory for the cache significantly improves performance. 4. Having more RAM is always prudent, given the relatively low cost of this component compared to the total system cost.3-8 Lustre 1.8 Operations Manual • December 2010 3.1.7.2 OSS Memory Requirements When planning the hardware for an OSS node, consider the memory usage of several components in the Lustre system (i.e., journal, service threads, file system metadata, etc.). Also, consider the effect of the OSS read cache feature (new in Lustre 1.8), which consumes memory as it caches data on the OSS node. ¦ Journal size: By default, each Lustre ldiskfs file system has 400 MB for the journal size. This can pin up to an equal amount of RAM on the OSS node per file system. ¦ Service threads: The service threads on the OSS node pre-allocate a 1 MB I/O buffer for each ost_io service thread, so these buffers do not need to be allocated and freed for each I/O request. ¦ File system metadata: A reasonable amount of RAM needs to be available for file system metadata. While no hard limit can be placed on the amount of file system metadata, if more RAM is available, then the disk I/O is needed less often to retrieve the metadata. ¦ Network transport: If you are using TCP or other network transport that uses system memory for send/receive buffers, this must also be taken into consideration. ¦ Failover configuration: If the OSS node will be used for failover from another node, then the RAM for each journal should be doubled, so the backup server can handle the additional load if the primary server fails. ¦ OSS read cache: OSS read cache provides read-only caching of data on an OSS, using the regular Linux page cache to store the data. Just like caching from a regular file system in Linux, OSS read cache uses as much physical memory as is available. Because of these memory requirements, the following calculations should be taken as determining the absolute minimum RAM required in an OSS node.Chapter 3 Installing Lustre 3-9 Calculating OSS Memory Requirements The minimum recommended RAM size for an OSS with two OSTs is computed below: 1.5 MB per OST IO thread * 512 threads = 768 MB e1000 RX descriptors, RxDescriptors=4096 for 9000 byte MTU = 128 MB Operating system overhead = 512 MB 400 MB journal size * 2 OST devices = 800 MB 600 MB file system metadata cache * 2 OSTs = 1200 MB This consumes about 1,700 MB just for the pre-allocated buffers, and an additional 2 GB for minimal file system and kernel usage. Therefore, for a non-failover configuration, the minimum RAM would be 4 GB for an OSS node with two OSTs. While it is not strictly required, adding additional memory on the OSS will improve the performance of reading smaller, frequently-accessed files. For a failover configuration, the minimum RAM would be at least 6 GB. For 4 OSTs on each OSS in a failover configuration 10GB of RAM is reasonable. When the OSS is not handling any failed-over OSTs the extra RAM will be used as a read cache. As a reasonable rule of thumb, about 2 GB of base memory plus 1 GB per OST can be used. In failover configurations, about 2 GB per OST is needed.3-10 Lustre 1.8 Operations Manual • December 2010 3.2 Installing Lustre from RPMs This procedure describes how to install Lustre from the RPM packages. This is the easier installation method and is recommended for new users. Alternately, you can install Lustre directly from the source code. For more information on this installation method, see Installing Lustre from Source Code. Note – In all Lustre installations, the server kernel that runs on an MDS, MGS or OSS must be patched. However, running a patched kernel on a Lustre client is optional and only required if the client will be used for multiple purposes, such as running as both a client and an OST. Caution – Lustre contains kernel modifications which interact with storage devices and may introduce security issues and data loss if not installed, configured or administered properly. Before installing Lustre, be cautious and back up ALL data. Use this procedure to install Lustre from RPMs. 1. Verify that all Lustre installation requirements have been met. For more information on these prerequisites, see Preparing to Install Lustre. 2. Download the Lustre RPMs. a. On the Lustre download site, select your platform. The files required to install Lustre (kernels, modules and utilities RPMs) are listed for the selected platform. b. Download the required files. Use the Download Manager or download the files individually.Chapter 3 Installing Lustre 3-11 3. Install the Lustre packages. Some Lustre packages are installed on servers (MDS and OSSs), and others are installed on Lustre clients. Lustre packages must be installed in a specific order. Caution – For a non-production Lustre environment or for testing, a Lustre client and server can run on the same machine. However, for best performance in a production environment, dedicated clients are always best. Performance and other issues can occur when an MDS or OSS and a client are running on the same machine 5 . The MDS and MGS can run on the same machine. a. For each Lustre package, determine if it needs to be installed on servers and/or clients. Use TABLE 3-1 to determine where to install a specific package. Depending on your platform, not all of the listed files need to be installed. 5. Running the MDS and a client on the same machine can cause recovery and deadlock issues, and the performance of other Lustre clients to suffer. Running the OSS and a client on the same machine can cause issues with low memory and memory pressure. The client consume all of the memory and tries to flush pages to disk. The OSS needs to allocate pages to receive data from the client, but cannot perform this operation, due to low memory. This can result in OOM kill and other issues. TABLE 3-1 Lustre required packages, descriptions and installation guidance Lustre Package Description Install on servers Install on patchless clients Install on patched clients Lustre kernel RPMs kernel-lustre- Lustre-patched kernel package for RHEL 5 (i686, ia64 and x86_64) platform. X X* kernel-lustre-smp- Lustre-patched kernel package for SuSE Server 10 (x86_64) platform. X X* kernel-lustre-bigsmp- Lustre-patched kernel package for SuSE Server 10 (i686) platform. X X* kernel-ib- Lustre OFED package. Install if the network interconnect is InfiniBand. X X X* kernel-lustre-default- kernel-lustre-default-base- Lustre-patched kernel package for SuSE Server 11 (i686 and x86_64) platform. X X* Lustre module RPMs lustre-modules- Lustre modules for the patched kernel. X X*3-12 Lustre 1.8 Operations Manual • December 2010 b. Install the kernel, modules and ldiskfs packages. Use the rpm -ivh command to install the kernel, module and ldiskfs packages. For example: $ rpm -ivh kernel-lustre-smp- \ kernel-ib- \ lustre-modules- \ lustre-ldiskfs- c. Install the utilities/userspace packages. Use the rpm -ivh command to install the utilities packages. For example: $ rpm -ivh lustre- lustre-client-modules- Lustre modules for patchless clients. X Lustre utilities lustre- Lustre utilities package. This includes userspace utilities to configure and run Lustre. X X* lustre-ldiskfs- Lustre-patched backing file system kernel module package for the ext3 file system X e2fsprogs- Utilities package used to maintain the ext3 backing file system. X lustre-client- Lustre utilities for patchless clients X * Only install this kernel RPM if you want to patch the client kernel. You do not have to patch the clients to run Lustre. TABLE 3-1 Lustre required packages, descriptions and installation guidance Lustre Package Description Install on servers Install on patchless clients Install on patched clientsChapter 3 Installing Lustre 3-13 d. Install the e2fsprogs package. Use the rpm -ivh command to install the e2fsprogs package. For example: $ rpm -ivh e2fsprogs- If e2fsprogs is already installed on your Linux system, install the Lustre-specific e2fsprogs version by using rpm -Uvh to update the existing e2fsprogs package. For example: $ rpm -Uvh e2fsprogs- The rpm command options --force or --nodeps are not required to install or update the Lustre-specific e2fsprogs package. We specifically recommend that you not use these options. If errors are reported, notify Lustre Support by filing a bug. e. (Optional) If you want to add optional packages to your Lustre file system, install them now. Optional packages include file system creation and repair tools, debugging tools, test programs and scripts, Linux kernel and Lustre source code, and other packages. A complete list of optional packages for your platform is provided on the Lustre download site. 4. Verify that the boot loader (grub.conf or lilo.conf) has been updated to load the patched kernel. 5. Reboot the patched clients and the servers. a. If you applied the patched kernel to any clients, reboot them. Unpatched clients do not need to be rebooted. b. Reboot the servers. Once all machines have rebooted, go to Configuring Lustre to configure Lustre Networking (LNET) and the Lustre file system.3-14 Lustre 1.8 Operations Manual • December 2010 3.3 Installing Lustre from Source Code If you need to build a customized Lustre server kernel or are using a Linux kernel that has not been tested with the version of Lustre you are installing, you may need to build and install Lustre from source code. This involves several steps: ¦ Patching the core kernel ¦ Configuring the kernel to work with Lustre ¦ Creating Lustre and kernel RPMs from source code. Please note that the Lustre/kernel configurations available at the Lustre download site have been extensively tested and verified with Lustre. The recommended method for installing Lustre servers is to use these pre-built binary packages (RPMs). For more information on this installation method, see Installing Lustre from RPMs. Caution – Lustre contains kernel modifications which interact with storage devices and may introduce security issues and data loss if not installed, configured and administered correctly. Before installing Lustre, be cautious and back up ALL data. Note – When using third-party network hardware with Lustre, the third-party modules (typically, the drivers) must be linked against the Linux kernel. The LNET modules in Lustre also need these references. To meet these requirements, a specific process must be followed to install and recompile Lustre. See Installing Lustre with a Third-Party Network Stack, for an example showing how to install Lustre 1.6.6 using the Myricom MX 1.2.7 driver. The same process can be used for other third-party network stacks.Chapter 3 Installing Lustre 3-15 3.3.1 Patching the Kernel If you are using non-standard hardware, plan to apply a Lustre patch, or have another reason not to use packaged Lustre binaries, you have to apply several Lustre patches to the core kernel and run the Lustre configure script against the kernel. 3.3.1.1 Introducing the Quilt Utility To simplify the process of applying Lustre patches to the kernel, we recommend that you use the Quilt utility. Quilt manages a stack of patches on a single source tree. A series file lists the patch files and the order in which they are applied. Patches are applied, incrementally, on the base tree and all preceding patches. You can: ¦ Apply patches from the stack (quilt push) ¦ Remove patches from the stack (quilt pop) ¦ Query the contents of the series file (quilt series), the contents of the stack (quilt applied, quilt previous, quilt top), and the patches that are not applied at a particular moment (quilt next, quilt unapplied). ¦ Edit and refresh (update) patches with Quilt, as well as revert inadvertent changes, and fork or clone patches and show the diffs before and after work. A variety of Quilt packages (RPMs, SRPMs and tarballs) are available from various sources. Use the most recent version you can find. Quilt depends on several other utilities, e.g., the coreutils RPM that is only available in RedHat 9. For other RedHat kernels, you have to get the required packages to successfully install Quilt. If you cannot locate a Quilt package or fulfill its dependencies, you can build Quilt from a tarball, available at the Quilt project website: http://savannah.nongnu.org/projects/quilt For additional information on using Quilt, including its commands, see Introduction to Quilt and the quilt(1) man page.3-16 Lustre 1.8 Operations Manual • December 2010 3.3.1.2 Get the Lustre Source and Unpatched Kernel The Lustre Engineering Team has targeted several Linux kernels for use with Lustre servers (MDS/OSS) and provides a series of patches for each one. The Lustre patches are maintained in the kernel_patch directory bundled with the Lustre source code. Note – Each patch series has been tailored to a specific kernel version, and may or may not apply cleanly to other versions of the kernel. To obtain the Lustre source and unpatched kernel: 1. Verify that all of the Lustre installation requirements have been met. For more information on these prerequisites, see Preparing to Install Lustre. 2. Download the Lustre source code. On the Lustre download site, select a version of Lustre to download and then select Source as the platform. 3. Download the unpatched kernel. For convenience, Oracle maintains an archive of unpatched kernel sources at: http://downloads.lustre.org/public/kernels/ 4. To save time later, download e2fsprogs now. The source code for Oracle’s Lustre-enabled e2fsprogs distribution can be found at: http://downloads.lustre.org/public/tools/e2fsprogs/Chapter 3 Installing Lustre 3-17 3.3.1.3 Patch the Kernel This procedure describes how to use Quilt to apply the Lustre patches to the kernel. To illustrate the steps in this procedure, a RHEL 5 kernel is patched for Lustre 1.6.5.1. 1. Unpack the Lustre source and kernel to separate source trees. a. Unpack the Lustre source. For this procedure, we assume that the resulting source tree is in /tmp/lustre-1.6.5.1 b. Unpack the kernel. For this procedure, we assume that the resulting source tree (also known as the destination tree) is in /tmp/kernels/linux-2.6.18 2. Select a config file for your kernel, located in the kernel_configs directory (lustre/kernel_patches/kernel_config). The kernel_config directory contains the .config files, which are named to indicate the kernel and architecture with which they are associated. For example, the configuration file for the 2.6.18 kernel shipped with RHEL 5 (suitable for i686 SMP systems) is kernel-2.6.18-2.6-rhel5-i686-smp.config. 3. Select the series file for your kernel, located in the series directory (lustre/kernel_patches/series). The series file contains the patches that need to be applied to the kernel. 4. Set up the necessary symlinks between the kernel patches and the Lustre source. This example assumes that the Lustre source files are unpacked under /tmp/lustre-1.6.5.1 and you have chosen the 2.6-rhel5.series file). Run: $ cd /tmp/kernels/linux-2.6.18 $ rm -f patches series $ ln -s /tmp/lustre-1.6.5.1/lustre/kernel_patches/series/2.6-\ rhel5.series ./series $ ln -s /tmp/lustre-1.6.5.1/lustre/kernel_patches/patches . 5. Use Quilt to apply the patches in the selected series file to the unpatched kernel. Run: $ cd /tmp/kernels/linux-2.6.18 $ quilt push -av The patched destination tree acts as a base Linux source tree for Lustre.3-18 Lustre 1.8 Operations Manual • December 2010 3.3.2 Create and Install the Lustre Packages After patching the kernel, configure it to work with Lustre, create the Lustre packages (RPMs) and install them. 1. Configure the patched kernel to run with Lustre. Run: $ cd $ cp /boot/config-‘uname -r‘ .config $ make oldconfig || make menuconfig $ make include/asm $ make include/linux/version.h $ make SUBDIRS=scripts $ make include/linux/utsrelease.h 2. Run the Lustre configure script against the patched kernel and create the Lustre packages. $ cd $ ./configure --with-linux= $ make rpms This creates a set of .rpms in /usr/src/redhat/RPMS/ with an appended date-stamp. The SuSE path is /usr/src/packages. Note – You do not need to run the Lustre configure script against an unpatched kernel. Example set of RPMs: Note – If the steps to create the RPMs fail, contact Lustre Support by reporting a bug. See Reporting a Lustre Bug. lustre-1.6.5.1-2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_20081021.i686.rpm lustre-debuginfo-1.6.5.1-2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_20081021.i686.rpm lustre-modules-1.6.5.1-2.6.18_53.xx.xxel5_lustre.1.6.5.1.custom_20081021.i686.rpm lustre-source-1.6.5.1-2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_20081021.i686.rpmChapter 3 Installing Lustre 3-19 Note – Lustre supports several features and packages that extend the core functionality of Lustre. These features/packages can be enabled at the build time by issuing appropriate arguments to the configure command. For a list of supported features and packages, run ./configure –help in the Lustre source tree. The configs/ directory of the kernel source contains the config files matching each the kernel version. Copy one to .config at the root of the kernel tree. 3. Create the kernel package. Navigate to the kernel source directory and run: $ make rpm Example result: kernel-2.6.95.0.3.EL_lustre.1.6.5.1custom-1.i686.rpm Note – Step 3 is only valid for RedHat and SuSE kernels. If you are using a stock Linux kernel, you need to get a script to create the kernel RPM. 4. Install the Lustre packages. Some Lustre packages are installed on servers (MDS and OSSs), and others are installed on Lustre clients. For guidance on where to install specific packages, see TABLE 3-1, which lists required packages and for each package, where to install it. Depending on the selected platform, not all of the packages listed in TABLE 3-1 need to be installed. Note – Running the patched server kernel on the clients is optional. It is not necessary unless the clients will be used for multiple purposes, for example, to run as a client and an OST. Lustre packages should be installed in this order: a. Install the kernel, modules and ldiskfs packages. Navigate to the directory where the RPMs are stored, and use the rpm -ivh command to install the kernel, module and ldiskfs packages. $ rpm -ivh kernel-lustre-smp- \ kernel-ib- \ lustre-modules- \ lustre-ldiskfs- b. Install the utilities/userspace packages. Use the rpm -ivh command to install the utilities packages. For example: $ rpm -ivh lustre-3-20 Lustre 1.8 Operations Manual • December 2010 c. Install the e2fsprogs package. Make sure the e2fsprogs package downloaded in Step 4 is unpacked, and use the rpm -i command to install it. For example: $ rpm -i e2fsprogs- d. (Optional) If you want to add optional packages to your Lustre system, install them now. 5. Verify that the boot loader (grub.conf or lilo.conf) has been updated to load the patched kernel. 6. Reboot the patched clients and the servers. a. If you applied the patched kernel to any clients, reboot them. Unpatched clients do not need to be rebooted. b. Reboot the servers. Once all the machines have rebooted, the next steps are to configure Lustre Networking (LNET) and the Lustre file system. See Configuring Lustre. 3.3.3 Installing Lustre with a Third-Party Network Stack When using third-party network hardware, you must follow a specific process to install and recompile Lustre. This section provides an installation example, describing how to install Lustre 1.6.6 while using the Myricom MX 1.2.7 driver. The same process is used for other third-party network stacks, by replacing MX-specific references in Step 2 with the stack-specific build and using the proper --with option when configuring the Lustre source code. 1. Compile and install the Lustre kernel. a. Install the necessary build tools. GCC and related tools must also be installed. For more information, see Required Lustre Software. $ yum install rpm-build redhat-rpm-config $ mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS} $ echo '%_topdir %(echo $HOME)/rpmbuild' > .rpmmacros b. Install the patched Lustre source code. This RPM is available at the Lustre download page. $ rpm -ivh kernel-lustre-source-2.6.18-92.1.10.el5_lustre.1.6.6.x86_64.rpmChapter 3 Installing Lustre 3-21 c. Build the Linux kernel RPM. $ cd /usr/src/linux-2.6.18-92.1.10.el5_lustre.1.6.6 $ make distclean $ make oldconfig dep bzImage modules $ cp /boot/config-`uname -r` .config $ make oldconfig || make menuconfig $ make include/asm $ make include/linux/version.h $ make SUBDIRS=scripts $ make rpm d. Install the Linux kernel RPM. If you are building a set of RPMs for a cluster installation, this step is not necessary. Source RPMs are only needed on the build machine. $ rpm -ivh ~/rpmbuild/kernel-lustre-2.6.18-92.1.10.el5_lustre.1.6.6.x86_64.rpm $ mkinitrd /boot/2.6.18-92.1.10.el5_lustre.1.6.6 e. Update the boot loader (/etc/grub.conf) with the new kernel boot information. $ /sbin/shutdown 0 -r 2. Compile and install the MX stack. $ cd /usr/src/ $ gunzip mx_1.2.7.tar.gz (can be obtained from www.myri.com/scs/) $ tar -xvf mx_1.2.7.tar $ cd mx-1.2.7 $ ln -s common include $ ./configure --with-kernel-lib $ make $ make install3-22 Lustre 1.8 Operations Manual • December 2010 3. Compile and install the Lustre source code. a. Install the Lustre source (this can be done via RPM or tarball). The source file is available at the Lustre download page. This example shows installation via the tarball. $ cd /usr/src/ $ gunzip lustre-1.6.6.tar.gz $ tar -xvf lustre-1.6.6.tar b. Configure and build the Lustre source code. The ./configure --help command shows a list of all of the --with options. All third-party network stacks are built in this manner. $ cd lustre-1.6.6 $ ./configure --with-linux=/usr/src/linux --with-mx=/usr/src/mx-1.2.7 $ make $ make rpms The make rpms command output shows the location of the generated RPMs 4. Use the rpm -ivh command to install the RPMS. $ rpm -ivh lustre-1.6.6-2.6.18_92.1.10.el5_lustre.1.6.6smp.x86_64.rpm $ rpm -ivh lustre-modules-1.6.6-2.6.18_92.1.10.el5_lustre.1.6.6smp.x86_64.rpm $ rpm -ivh lustre-ldiskfs-3.0.6-2.6.18_92.1.10.el5_lustre.1.6.6smp.x86_64.rpm 5. Add the following lines to the /etc/modprobe.conf file. options kmxlnd hosts=/etc/hosts.mxlnd options lnet networks=mx0(myri0),tcp0(eth0) 6. Populate the myri0 configuration with the proper IP addresses. vim /etc/sysconfig/network-scripts/myri0 7. Add the following line to the /etc/hosts.mxlnd file. $ IP HOST BOARD EP_ID 8. Start Lustre. Once all the machines have rebooted, the next steps are to configure Lustre Networking (LNET) and the Lustre file system. See Configuring Lustre.4-1 C H A P T E R 4 Configuring Lustre You can use the administrative utilities provided with Lustre to set up a system with many different configurations. This chapter shows how to configure a simple Lustre system comprised of a combined MGS/MDT, an OST and a client, and includes the following sections: ¦ Configuring the Lustre File System ¦ Additional Lustre Configuration ¦ Basic Lustre Administration ¦ More Complex Configurations ¦ Operational Scenarios4-2 Lustre 1.8 Operations Manual • December 2010 4.1 Configuring the Lustre File System A Lustre file system consists of four types of subsystems – a Management Server (MGS), a Metadata Target (MDT), Object Storage Targets (OSTs) and clients. We recommend running these components on different systems, although, technically, they can co-exist on a single system. Together, the OSSs and MDS present a Logical Object Volume (LOV) which is an abstraction that appears in the configuration. It is possible to set up the Lustre system with many different configurations by using the administrative utilities provided with Lustre. Some sample scripts are included in the directory where Lustre is installed. If you have installed the Lustre source code, the scripts are located in the lustre/tests sub-directory. These scripts enable quick setup of some simple, standard Lustre configurations. Note – We recommend that you use dotted-quad IP addressing (IPv4) rather than host names. This aids in reading debug logs, and helps greatly when debugging configurations with multiple interfaces. 1. Define the module options for Lustre networking (LNET), by adding this line to the /etc/modprobe.conf file 1 . options lnet networks= This step restricts LNET to use only the specified network interfaces and prevents LNET from using all network interfaces. As an alternative to modifying the modprobe.conf file, you can modify the modprobe.local file or the configuration files in the modprobe.d directory. Note – For details on configuring networking and LNET, see Configuring LNET. 2. (Optional) Prepare the block devices to be used as OSTs or MDTs. Depending on the hardware used in the MDS and OSS nodes, you may want to set up a hardware or software RAID to increase the reliability of the Lustre system. For more details on how to set up a hardware or software RAID, see the documentation for your RAID controller or see Lustre Software RAID Support. 1. The modprobe.conf file is a Linux file that lives in /etc/modprobe.conf and specifies what parts of the kernel are loaded.Chapter 4 Configuring Lustre 4-3 3. Create a combined MGS/MDT file system. a. Consider the MDT size needed to support the file system. When calculating the MDT size, the only important factor is the number of files to be stored in the file system. This determines the number of inodes needed, which drives the MDT sizing. For more information, see Sizing the MDT and Planning for Inodes. Make sure the MDT is properly sized before performing the next step, as a too-small MDT can cause the space on the OSTs to be unusable. b. Create the MGS/MDT file system on the block device. On the MDS node, run: mkfs.lustre --fsname= --mgs --mdt The default file system name (fsname) is lustre. Note – If you plan to generate multiple file systems, the MGS should be on its own dedicated block device. 4. Mount the combined MGS/MDT file system on the block device. On the MDS node, run: mount -t lustre 5. Create the OST2 . On the OSS node, run: mkfs.lustre --ost --fsname= --mgsnode= You can have as many OSTs per OSS as the hardware or drivers allow. You should only use only 1 OST per block device. Optionally, you can create an OST which uses the raw block device and does not require partitioning. Note – Lustre currently supports block devices up to 16 TB on OEL 5/RHEL 5 (up to 8 TB on other distributions). If the device size is only slightly larger that 16 TB, we recommend that you limit the file system size to 16 TB at format time. If the device size is significantly larger than 16 TB, you should reconfigure the storage into devices smaller than 16 TB. We recommend that you not place partitions on top of RAID 5/6 block devices due to negative impacts on performance. 2. When you create the OST, you are defining a storage device ('sd'), a device number (a, b, c, d), and a partition (1, 2, 3) where the OST node lives.4-4 Lustre 1.8 Operations Manual • December 2010 6. Mount the OST. On the OSS node where the OST was created, run: mount -t lustre Note – To create additional OSTs, repeat Step 5 and Step 6. 7. Create the client (mount the file system on the client). On the client node, run: mount -t lustre :/ Note – To create additional clients, repeat Step 7. 8. Verify that the file system started and is working correctly by running the df, dd and ls commands on the client node. a. Run the lfs df -h command. [root@client1 /] lfs df -h The lfs df -h command lists space usage per OST and the MDT in human-readable format. b. Run the lfs df -ih command. [root@client1 /] lfs df -ih The lfs df -ih command lists inode usage per OST and the MDT. c. Run the dd command. [root@client1 /] cd /lustre [root@client1 /lustre] dd if=/dev/zero of=/lustre/zero.dat bs=4M count=2 The dd command verifies write functionality by creating a file containing all zeros (0s). In this command, an 8 MB file is created. d. Run the ls command. [root@client1 /lustre] ls -lsah The ls -lsah command lists files and directories in the current working directory. If you have a problem mounting the file system, check the syslogs for errors and also check the network settings. A common issue with newly-installed systems is hosts.deny or filewall rules that prevent connections on port 988.Chapter 4 Configuring Lustre 4-5 Tip – Now that you have configured Lustre, you can collect and register service tags in Lustre 1.8.3 and earlier versions. Note that service tags have been discontinued in Lustre 1.8.4 and later releases. For more information, see Service Tags. 4.1.0.1 Simple Lustre Configuration Example To see the steps in a simple Lustre configuration, follow this worked example in which a combined MGS/MDT and two OSTs are created. Three block devices are used, one for the combined MGS/MDS node and one for each OSS node. Common parameters used in the example are listed below, along with individual node parameters. Common Parameters Value Description MGS/MDS node 10.2.0.1@tcp0Node for the combined MGS/MDS file system temp Name of the Lustre file system network type TCP/IP Network type used for Lustre file system temp Node Parameters Value Description MGS/MDS node MGS/MDS node mdt1 MDS in Lustre file system temp block device /dev/sdb Block device for the combined MGS/MDS node mount point /mnt/mdt Mount point for the mdt1 block device (/dev/sdb) on the MGS/MDS node First OSS node OSS node oss1 First OSS node in Lustre file system temp OST ost1 First OST in Lustre file system temp block device /dev/sdc Block device for the first OSS node (oss1) mount point /mnt/ost1 Mount point for the ost1 block device (/dev/sdc) on the oss1 node Second OSS node OSS node oss2 Second OSS node in Lustre file system temp OST ost2 Second OST in Lustre file system temp block device /dev/sdd Block device for the second OSS node (oss2) mount point /mnt/ost2 Mount point for the ost2 block device (/dev/sdd) on the oss2 node4-6 Lustre 1.8 Operations Manual • December 2010 1. Define the module options for Lustre networking (LNET), by adding this line to the /etc/modprobe.conf file. options lnet networks=tcp 2. Create a combined MGS/MDT file system on the block device. On the MDS node, run: [root@mds /]# mkfs.lustre --fsname=temp --mgs --mdt /dev/sdb This command generates this output: Permanent disk data: Target: temp-MDTffff Index: unassigned Lustre FS: temp Mount type: ldiskfs Flags: 0x75 (MDT MGS needs_index first_time update ) Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr Parameters: mdt.group_upcall=/usr/sbin/l_getgroups checking for existing Lustre data: not found device size = 16MB 2 6 18 formatting backing filesystem ldiskfs on /dev/sdb target nametemp-MDTffff 4k blocks 0 options -i 4096 -I 512 -q -O dir_index,uninit_groups -F mkfs_cmd = mkfs.ext2 -j -b 4096 -L temp-MDTffff -i 4096 -I 512 -q -O dir_index,uninit_groups -F /dev/sdb Writing CONFIGS/mountdata Client node client node client1 Client in Lustre file system temp mount point /lustre Mount point for Lustre file system temp on the client1 node Node Parameters Value DescriptionChapter 4 Configuring Lustre 4-7 3. Mount the combined MGS/MDT file system on the block device. On the MDS node, run: [root@mds /]# mount -t lustre /dev/sdb /mnt/mdt This command generates this output: Lustre: temp-MDT0000: new disk, initializing Lustre: 3009:0:(lproc_mds.c:262:lprocfs_wr_group_upcall()) \ temp-MDT0000: group upcall set to /usr/sbin/l_getgroups Lustre: temp-MDT0000.mdt: set parameter \ group_upcall=/usr/sbin/l_getgroups Lustre: Server temp-MDT0000 on device /dev/sdb has started 4. Create the OSTs. In this example, the OSTs (ost1 and ost2) are being created or different OSSs (oss1 and oss2). a. Create ost1. On oss1 node, run: [root@oss1 /]# mkfs.lustre --ost --fsname=temp --mgsnode= 10.2.0.1@tcp0 /dev/sdc The command generates this output: Permanent disk data: Target: temp-OSTffff Index: unassigned Lustre FS: temp Mount type: ldiskfs Flags: 0x72 (OST needs_index first_time update) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=10.2.0.1@tcp checking for existing Lustre data: not found device size = 16MB 2 6 18 formatting backing filesystem ldiskfs on /dev/sdc target name temp-OSTffff 4k blocks 0 options -I 256 -q -O dir_index,uninit_groups -F mkfs_cmd = mkfs.ext2 -j -b 4096 -L temp-OSTffff -I 256 -q -O dir_index,uninit_groups -F /dev/sdc Writing CONFIGS/mountdata4-8 Lustre 1.8 Operations Manual • December 2010 b. Create ost2. On oss2 node, run: [root@oss2 /]# mkfs.lustre --ost --fsname=temp --mgsnode= 10.2.0.1@tcp0 /dev/sdd The command generates this output: Permanent disk data: Target: temp-OSTffff Index: unassigned Lustre FS: temp Mount type: ldiskfs Flags: 0x72 (OST needs_index first_time update) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=10.2.0.1@tcp checking for existing Lustre data: not found device size = 16MB 2 6 18 formatting backing filesystem ldiskfs on /dev/sdd target name temp-OSTffff 4k blocks 0 options -I 256 -q -O dir_index,uninit_groups -F mkfs_cmd = mkfs.ext2 -j -b 4096 -L temp-OSTffff -I 256 -q -O dir_index,uninit_groups -F /dev/sdc Writing CONFIGS/mountdata 5. Mount the OSTs. Mount each OST (ost1 and ost2), on the OSS where the OST was created. a. Mount ost1. On oss1 node, run: root@oss1 /] mount -t lustre /dev/sdc /mnt/ost1 The command generates this output: LDISKFS-fs: file extents enabled LDISKFS-fs: mballoc enabled Lustre: temp-OST0000: new disk, initializing Lustre: Server temp-OST0000 on device /dev/sdb has started Shortly afterwards, this output appears: Lustre: temp-OST0000: received MDS connection from 10.2.0.1@tcp0 Lustre: MDS temp-MDT0000: temp-OST0000_UUID now active, resetting orphansChapter 4 Configuring Lustre 4-9 b. Mount ost2. On oss2 node, run: root@oss2 /] mount -t lustre /dev/sdd /mnt/ost2 The command generates this output: LDISKFS-fs: file extents enabled LDISKFS-fs: mballoc enabled Lustre: temp-OST0000: new disk, initializing Lustre: Server temp-OST0000 on device /dev/sdb has started Shortly afterwards, this output appears: Lustre: temp-OST0000: received MDS connection from 10.2.0.1@tcp0 Lustre: MDS temp-MDT0000: temp-OST0000_UUID now active, resetting orphans 6. Create the client (mount the file system on the client). On the client node, run: root@client1 /] mount -t lustre 10.2.0.1@tcp0:/temp /lustre This command generates this output: Lustre: Client temp-client has started 7. Verify that the file system started and is working by running the df, dd and ls commands on the client node. a. Run the df command: [root@client1 /] lfs df -h This command generates output similar to this: Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 7.2G 2.4G 4.5G 35% / dev/sda1 99M 29M 65M 31% /boot tmpfs 62M 0 62M 0% /dev/shm 10.2.0.1@tcp0:/temp 30M 8.5M 20M 30% /lustre b. Run the dd command: [root@client1 /] cd /lustre [root@client1 /lustre] dd if=/dev/zero of=/lustre/zero.dat bs=4M count=2 This command generates output similar to this: 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.159628 seconds, 52.6 MB/s4-10 Lustre 1.8 Operations Manual • December 2010 c. Run the ls command: [root@client1 /lustre] ls -lsah This command generates output similar to this: total 8.0M 4.0K drwxr-xr-x 2 root root 4.0K Oct 16 15:27 . 8.0K drwxr-xr-x 25 root root 4.0K Oct 16 15:27 .. 8.0M -rw-r--r-- 1 root root 8.0M Oct 16 15:27 zero.dat 4.1.0.2 Module Setup Make sure the modules (like LNET) are installed in the appropriate /lib/modules directory. The mkfs.lustre utility tries to automatically load LNET (via the Lustre module) with the default network settings (using all available network interfaces). To change this default setting, use the network=... option to specify the network(s) that LNET should use: modprobe -v lustre "networks=XXX" For example, to load Lustre with multiple-interface support (meaning LNET will use more than one physical circuit for communication between nodes), load the Lustre module with the following network=... option: modprobe -v lustre "networks=tcp0(eth0),o2ib0(ib0)" where: tcp0 is the network itself (TCP/IP) eth0 is the physical device (card) that is used (Ethernet) o2ib0 is the interconnect (InfiniBand) 4.1.1 Scaling the Lustre File System A Lustre file system can be scaled by adding OSTs or clients. For instructions on creating additional OSTs see Step 4 and Step 5 above; for clients, see Step 7.Chapter 4 Configuring Lustre 4-11 4.2 Additional Lustre Configuration Once the Lustre file system is configured, it is ready for use. If additional configuration is necessary, several configuration utilities are available. For man pages and reference information, see: ¦ mkfs.lustre ¦ tunefs.lustre ¦ lctl ¦ mount.lustre System Configuration Utilities (man8) profiles utilities (e.g., lustre_rmmod, e2scan, l_getgroups, llobdstat, llstat, plot-llstat, routerstat, and ll_recover_lost_found_objs), and tools to manage large clusters, perform application profiling, and debug Lustre. 4.3 Basic Lustre Administration Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre administration tasks: ¦ Specifying the File System Name ¦ Starting up Lustre ¦ Mounting a Server ¦ Unmounting a Server ¦ Working with Inactive OSTs ¦ Finding Nodes in the Lustre File System ¦ Mounting a Server Without Lustre Service ¦ Specifying Failout/Failover Mode for OSTs ¦ Running Multiple Lustre File Systems ¦ Setting and Retrieving Lustre Parameters ¦ Regenerating Lustre Configuration Logs ¦ Changing a Server NID ¦ Removing and Restoring OSTs ¦ Changing a Server NID ¦ Aborting Recovery ¦ Determining Which Machine is Serving an OST4-12 Lustre 1.8 Operations Manual • December 2010 ¦ Failover ¦ Changing the Address of a Failover Node 4.3.1 Specifying the File System Name The file system name is limited to 8 characters. We have encoded the file system and target information in the disk label, so you can mount by label. This allows system administrators to move disks around without worrying about issues such as SCSI disk reordering or getting the /dev/device wrong for a shared target. Soon, file system naming will be made as fail-safe as possible. Currently, Linux disk labels are limited to 16 characters. To identify the target within the file system, 8 characters are reserved, leaving 8 characters for the file system name: -MDT0000 or -OST0a19 To mount by label, use this command: $ mount -t lustre -L This is an example of mount-by-label: $ mount -t lustre -L testfs-MDT0000 /mnt/mdt Caution – Mount-by-label should NOT be used in a multi-path environment. Although the file system name is internally limited to 8 characters, you can mount the clients at any mount point, so file system users are not subjected to short names. Here is an example: mount -t lustre uml1@tcp0:/shortfs /mnt/ 4.3.2 Starting up Lustre The startup order of Lustre components depends on whether you have a combined MGS/MDT or these components are separate. ¦ If you have a combined MGS/MDT, the recommended startup order is OSTs, then the MGS/MDT, and then clients. ¦ If the MGS and MDT are separate, the recommended startup order is: MGS, then OSTs, then the MDT, and then clients.Chapter 4 Configuring Lustre 4-13 Note – If an OST is added to a Lustre file system with a combined MGS/MDT, then the startup order changes slightly; the MGS must be started first because the OST needs to write its configuration data to it. In this scenario, the startup order is MGS/MDT, then OSTs, then the clients. 4.3.3 Mounting a Server Starting a Lustre server is straightforward and only involves the mount command. Lustre servers can be added to /etc/fstab: mount -t lustre The mount command generates output similar to this: /dev/sda1 on /mnt/test/mdt type lustre (rw) /dev/sda2 on /mnt/test/ost0 type lustre (rw) 192.168.0.21@tcp:/testfs on /mnt/testfs type lustre (rw) In this example, the MDT, an OST (ost0) and file system (testfs) are mounted. LABEL=testfs-MDT0000 /mnt/test/mdt lustre defaults,_netdev,noauto 0 0 LABEL=testfs-OST0000 /mnt/test/ost0 lustre defaults,_netdev,noauto 0 0 In general, it is wise to specify noauto and let your high-availability (HA) package manage when to mount the device. If you are not using failover, make sure that networking has been started before mounting a Lustre server. RedHat, SuSE, Debian (and perhaps others) use the _netdev flag to ensure that these disks are mounted after the network is up. We are mounting by disk label here—the label of a device can be read with e2label. The label of a newly-formatted Lustre server ends in FFFF, meaning that it has yet to be assigned. The assignment takes place when the server is first started, and the disk label is updated. Caution – Do not do this when the client and OSS are on the same node, as memory pressure between the client and OSS can lead to deadlocks. Caution – Mount-by-label should NOT be used in a multi-path environment.4-14 Lustre 1.8 Operations Manual • December 2010 4.3.4 Unmounting a Server To stop a Lustre server, use the umount command. For example, to stop ost0 on mount point /mnt/test, run: $ umount /mnt/test Gracefully stopping a server with the umount command preserves the state of the connected clients. The next time the server is started, it waits for clients to reconnect, and then goes through the recovery procedure. If the force (-f) flag is used, then the server evicts all clients and stops WITHOUT recovery. Upon restart, the server does not wait for recovery. Any currently connected clients receive I/O errors until they reconnect. Note – If you are using loopback devices, use the -d flag. This flag cleans up loop devices and can always be safely specified. 4.3.5 Working with Inactive OSTs To mount a client or an MDT with one or more inactive OSTs, run commands similar to this: client> mount -o exclude=testfs-OST0000 -t lustre uml1:/testfs\ /mnt/testfs client> cat /proc/fs/lustre/lov/testfs-clilov-*/target_obd To activate an inactive OST on a live client or MDT, use the lctl activate command on the OSC device. For example: lctl --device 7 activate Note – A colon-separated list can also be specified. For example, exclude= testfs-OST0000:testfs-OST0001.Chapter 4 Configuring Lustre 4-15 4.3.6 Finding Nodes in the Lustre File System There may be situations in which you need to find all nodes in your Lustre file system or get the names of all OSTs. To get a list of all Lustre nodes, run this command on the MGS: # cat /proc/fs/lustre/mgs/MGS/live/* Note – This command must be run on the MGS. In this example, file system lustre has three nodes, lustre-MDT0000, lustre-OST0000, and lustre-OST0001. cfs21:/tmp# cat /proc/fs/lustre/mgs/MGS/live/* fsname: lustre flags: 0x0 gen: 26 lustre-MDT0000 lustre-OST0000 lustre-OST0001 To get the names of all OSTs, run this command on the MDS: # cat /proc/fs/lustre/lov/-mdtlov/target_obd Note – This command must be run on the MDS. In this example, there are two OSTs, lustre-OST0000 and lustre-OST0001, which are both active. cfs21:/tmp# cat /proc/fs/lustre/lov/lustre-mdtlov/target_obd 0: lustre-OST0000_UUID ACTIVE 1: lustre-OST0001_UUID ACTIVE4-16 Lustre 1.8 Operations Manual • December 2010 4.3.7 Mounting a Server Without Lustre Service If you are using a combined MGS/MDT, but you only want to start the MGS and not the MDT, run this command: mount -t lustre -o nosvc The variable is the combined MGS/MDT. In this example, the combined MGS/MDT is testfs-MDT0000 and the mount point is mnt/test/mdt. $ mount -t lustre -L testfs-MDT0000 -o nosvc /mnt/test/mdt 4.3.8 Specifying Failout/Failover Mode for OSTs Lustre uses two modes, failout and failover, to handle an OST that has become unreachable because it fails, is taken off the network, is unmounted, etc. ¦ In failout mode, Lustre clients immediately receive errors (EIOs) after a timeout, instead of waiting for the OST to recover. ¦ In failover mode, Lustre clients wait for the OST to recover. By default, the Lustre file system uses failover mode for OSTs. To specify failout mode instead, run this command: $ mkfs.lustre --fsname= --ost --mgsnode= --param="failover.mode=failout" In this example, failout mode is specified for the OSTs on MGS uml1, file system testfs. $ mkfs.lustre --fsname=testfs --ost --mgsnode=uml1 --param= "failover.mode=failout" /dev/sdb Caution – Before running this command, unmount all OSTS that will be affected by the change in the failover/failout mode. Note – After initial file system configuration, use the tunefs.lustre utility to change the failover/failout mode. For example, to set the failout mode, run: $ tunefs.lustre --param failover.mode=failout Chapter 4 Configuring Lustre 4-17 4.3.9 Running Multiple Lustre File Systems There may be situations in which you want to run multiple file systems. This is doable, as long as you follow specific naming conventions. By default, the mkfs.lustre command creates a file system named lustre. To specify a different file system name (limited to 8 characters), run this command: mkfs.lustre --fsname= Note – The MDT, OSTs and clients in the new file system must share the same name (prepended to the device name). For example, for a new file system named foo, the MDT and two OSTs would be named foo-MDT0000, foo-OST0000, and foo-OST0001. To mount a client on the file system, run: mount -t lustre mgsnode:/ For example, to mount a client on file system foo at mount point /mnt/lustre1, run: mount -t lustre mgsnode:/foo /mnt/lustre1 Note – If a client(s) will be mounted on several file systems, add the following line to /etc/xattr.conf file to avoid problems when files are moved between the file systems: lustre.* skip Note – The MGS is universal; there is only one MGS per Lustre installation, not per file system. Note – There is only one file system per MDT. Therefore, specify --mdt --mgs on one file system and --mdt --mgsnode= on the other file systems.4-18 Lustre 1.8 Operations Manual • December 2010 A Lustre installation with two file systems (foo and bar) could look like this, where the MGS node is mgsnode@tcp0 and the mount points are /mnt/lustre1 and /mnt/lustre2. mgsnode# mkfs.lustre --mgs /mnt/lustre1 mdtfoonode# mkfs.lustre --fsname=foo --mdt \ --mgsnode=mgsnode@tcp0 /mnt/lustre1 ossfoonode# mkfs.lustre --fsname=foo --ost \ --mgsnode=mgsnode@tcp0 /mnt/lustre1 ossfoonode# mkfs.lustre --fsname=foo --ost \ --mgsnode=mgsnode@tcp0 /mnt/lustre2 mdtbarnode# mkfs.lustre --fsname=bar --mdt \ --mgsnode=mgsnode@tcp0 /mnt/lustre1 ossbarnode# mkfs.lustre --fsname=bar --ost \ --mgsnode=mgsnode@tcp0 /mnt/lustre1 ossbarnode# mkfs.lustre --fsname=bar --ost \ --mgsnode=mgsnode@tcp0 /mnt/lustre2 To mount a client on file system foo at mount point /mnt/lustre1, run: mount -t lustre mgsnode@tcp0:/foo /mnt/lustre1 To mount a client on file system bar at mount point /mnt/lustre2, run: mount -t lustre mgsnode@tcp0:/bar /mnt/lustre2Chapter 4 Configuring Lustre 4-19 4.3.10 Setting and Retrieving Lustre Parameters There are several options for setting parameters in Lustre. ¦ When the file system is created, using mkfs.lustre. See Setting Parameters with mkfs.lustre ¦ When a server is stopped, using tunefs.lustre. See Setting Parameters with tunefs.lustre ¦ When the file system is running, using lctl. See Setting Parameters with lctl Additionally, you can use lctl to retrieve Lustre parameters. See Reporting Current Parameter Values. 4.3.10.1 Setting Parameters with mkfs.lustre When the file system is created, parameters can simply be added as a --param option to the mkfs.lustre command. For example: $ mkfs.lustre --mdt --param="sys.timeout=50" /dev/sda 4.3.10.2 Setting Parameters with tunefs.lustre If a server (OSS or MDS) is stopped, parameters can be added using the --param option to the tunefs.lustre command. For example: $ tunefs.lustre --param="failover.node=192.168.0.13@tcp0" /dev/sda With tunefs.lustre, parameters are "additive" -- new parameters are specified in addition to old parameters, they do not replace them. To erase all old tunefs.lustre parameters and just use newly-specified parameters, run: $ tunefs.lustre --erase-params --param= The tunefs.lustre command can be used to set any parameter settable in a /proc/fs/lustre file and that has its own OBD device, so it can be specified as ..=. For example: $ tunefs.lustre --param mdt.group_upcall=NONE /dev/sda14-20 Lustre 1.8 Operations Manual • December 2010 4.3.10.3 Setting Parameters with lctl When the file system is running, the lctl command can be used to set parameters (temporary or permanent) and report current parameter values. Temporary parameters are active as long as the server or client is not shut down. Permanent parameters live through server and client reboots. Note – Lustre 1.8.4 adds the lctl list_param command, which enables users to list all parameters that can be set. See Listing Parameters. Setting Temporary Parameters Use the lctl set_param command to set temporary parameters on the node where it is run. These parameters map to items in /proc/{fs,sys}/{lnet,lustre}. The lctl set_param command uses this syntax: lctl set_param [-n] ..= For example: # lctl set_param osc.*.max_dirty_mb=1024 osc.myth-OST0000-osc.max_dirty_mb=32 osc.myth-OST0001-osc.max_dirty_mb=32 osc.myth-OST0002-osc.max_dirty_mb=32 osc.myth-OST0003-osc.max_dirty_mb=32 osc.myth-OST0004-osc.max_dirty_mb=32 Setting Permanent Parameters Use the lctl conf_param command to set permanent parameters. In general, the lctl conf_param command can be used to specify any parameter settable in a /proc/fs/lustre file, with its own OBD device. The lctl conf_param command uses this syntax (same as the mkfs.lustre and tunefs.lustre commands): ..=) Here are a few examples of lctl conf_param commands: $ mgs> lctl conf_param testfs-MDT0000.sys.timeout=40 $ lctl conf_param testfs-MDT0000.mdt.group_upcall=NONE $ lctl conf_param testfs.llite.max_read_ahead_mb=16 $ lctl conf_param testfs-MDT0000.lov.stripesize=2M $ lctl conf_param testfs-OST0000.osc.max_dirty_mb=29.15Chapter 4 Configuring Lustre 4-21 $ lctl conf_param testfs-OST0000.ost.client_cache_seconds=15 $ lctl conf_param testfs.sys.timeout=40 Caution – Parameters specified with the lctl conf_param command are set permanently in the file system’s configuration file on the MGS. Listing Parameters To list Lustre or LNET parameters that are available to set, use the lctl list_param command. For example: lctl list_param [-FR] . The following arguments are available for the lctl list_param command. -F Add '/', '@' or '=' for directories, symlinks and writeable files, respectively -R Recursively lists all parameters under the specified path For example: $ lctl list_param obdfilter.lustre-OST0000 4.3.10.4 Reporting Current Parameter Values To report current Lustre parameter values, use the lctl get_param command with this syntax: lctl get_param [-n] .. This example reports data on RPC service times. $ lctl get_param -n ost.*.ost_io.timeouts service : cur 1 worst 30 (at 1257150393, 85d23h58m54s ago) 1 1 1 1 This example reports the number of inodes available on each OST. # lctl get_param osc.*.filesfree osc.myth-OST0000-osc-ffff88006dd20000.filesfree=217623 osc.myth-OST0001-osc-ffff88006dd20000.filesfree=5075042 osc.myth-OST0002-osc-ffff88006dd20000.filesfree=3762034 osc.myth-OST0003-osc-ffff88006dd20000.filesfree=91052 osc.myth-OST0004-osc-ffff88006dd20000.filesfree=1296514-22 Lustre 1.8 Operations Manual • December 2010 4.3.11 Regenerating Lustre Configuration Logs If the Lustre system’s configuration logs are in a state where the file system cannot be started, use the writeconf command to erase them. After the writeconf command is run and the servers restart, the configuration logs are re-generated and stored on the MGS (as in a new file system). You should only use the writeconf command if: ¦ The configuration logs are in a state where the file system cannot start ¦ A server NID is being changed The writeconf command is destructive to some configuration items (i.e., OST pools information and items set via conf_param), and should be used with caution. To avoid problems: ¦ Shut down the file system before running the writeconf command ¦ Run the writeconf command on all servers (MDT first, then OSTs) ¦ Start the file system in this order: ¦ MGS (or the combined MGS/MDT) ¦ MDT ¦ OSTs ¦ Lustre clients Caution – Lustre 1.8 introduces the OST pools feature, which enables a group of OSTs to be named for file striping purposes. If you use OST pools, be aware that running the writeconf command erases all pools information (as well as any other parameters set via lctl conf_param). We recommend that the pools definitions (and conf_param settings) be executed via a script, so they can be reproduced easily after a writeconf is performed. To regenerate Lustre’s system configuration logs: 1. Shut down the file system in this order. a. Unmount the clients. b. Unmount the MDT. c. Unmount all OSTs. 2. Make sure the the MDT and OST devices are available.Chapter 4 Configuring Lustre 4-23 3. Run the writeconf command on all servers. Run writeconf on the MDT first, and then the OSTs. a. On the MDT, run: $ tunefs.lustre --writeconf b. On each OST, run: $ tunefs.lustre --writeconf 4. Restart the file system in this order. a. Mount the MGS (or the combined MGS/MDT). b. Mount the MDT. c. Mount the OSTs. d. Mount the clients. After the writeconf command is run, the configuration logs are re-generated as servers restart. 4.3.12 Changing a Server NID If you need to change the NID on the MDT or an OST, run the writeconf command to erase Lustre configuration information (including server NIDs), and then re-generate the system configuration using updated server NIDs. Change a server NID in these situations: ¦ New server hardware is added to the file system, and the MDS or an OSS is being moved to the new machine ¦ New network card is installed in the server ¦ You want to reassign IP addresses To change a server NID: 1. Update the LNET configuration in the /etc/modprobe.conf file so the list of server NIDs (lctl list_nids) is correct. The lctl list_nids command indicates which network(s) are configured to work with Lustre.4-24 Lustre 1.8 Operations Manual • December 2010 2. Shut down the file system in this order. a. Unmount the clients. b. Unmount the MDT. c. Unmount all OSTs. 3. Run the writeconf command on all servers. Run writeconf on the MDT first, and then the OSTs. a. On the MDT, run: $ tunefs.lustre --writeconf b. On each OST, run: $ tunefs.lustre --writeconf c. If the NID on the MGS was changed, communicate the new MGS location to each server. Run: tunefs.lustre --erase-param --mgsnode= --writeconf /dev/.. 4. Restart the file system in this order. a. Mount the MGS (or the combined MGS/MDT). b. Mount the MDT. c. Mount the OSTs. d. Mount the clients. After the writeconf command is run, the configuration logs are re-generated as servers restart, and server NIDs in the updated list_nids file are used.Chapter 4 Configuring Lustre 4-25 4.3.13 Removing and Restoring OSTs OSTs can be removed from and restored to a Lustre file system. Currently in Lustre, removing an OST really means that the OST is ‘deactivated’ in the file system, not permanently removed. A removed OST still appears in the file system; do not create a new OST with the same name. You may want to remove (deactivate) an OST and prevent new files from being written to it in several situations: ¦ Hard drive has failed and a RAID resync/rebuild is underway ¦ OST is nearing its space capacity 4.3.13.1 Removing an OST from the File System When removing an OST, remember that the MDT does not communicate directly with OSTs. Rather, each OST has a corresponding OSC which communicates with the MDT. It is necessary to determine the device number of the OSC that corresponds to the OST. Then, you use this device number to deactivate the OSC on the MDT. To remove an OST from the file system: 1. For the OST to be removed, determine the device number of the corresponding OSC on the MDT. a. List all OSCs on the node, along with their device numbers. Run: lctl dl | grep " osc " This is sample lctl dl | grep " osc " output: 11 UP osc lustre-OST-0000-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5 12 UP osc lustre-OST-0001-osc-cac94211 4ea5b30f-6a8e-55a0-7519-2f20318ebdb4 5 13 IN osc lustre-OST-0000-osc lustre-MDT0000-mdtlov_UUID 5 14 UP osc lustre-OST-0001-osc lustre-MDT0000-mdtlov_UUID 5 b. Determine the device number of the OSC that corresponds to the OST to be removed.4-26 Lustre 1.8 Operations Manual • December 2010 2. Temporarily deactivate the OSC on the MDT. On the MDT, run: $ mdt> lctl --device deactivate For example, based on the command output in Step 1, to deactivate device 13 (the MDT’s OSC for OST-0000), the command would be: $ mdt> lctl --device 13 deactivate This marks the OST as inactive on the MDS, so no new objects are assigned to the OST. This does not prevent use of existing objects for reads or writes. Note – Do not deactivate the OST on the clients. Do so causes errors (EIOs), and the copy out to fail. Caution – Do not use lctl conf_param to deactivate the OST. It permanently sets a parameter in the file system configuration. 3. Discover all files that have objects residing on the deactivated OST. Run: lfs find --obd {OST UUID} / 4. Copy (not move) the files to a new directory in the file system. Copying the files forces object re-creation on the active OSTs. 5. Move (not copy) the files back to their original directory in the file system. Moving the files causes the original files to be deleted, as the copies replace them. 6. Once all files have been moved, permanently deactivate the OST on the clients and the MDT. On the MGS, run: # mgs> lctl conf_param .osc.active=0 Note – A removed OST still appears in the file system; do not create a new OST with the same name.Chapter 4 Configuring Lustre 4-27 Temporarily Deactivating an OST in the File System You may encounter situations when it is necessary to temporarily deactivate an OST, rather than permanently deactivate it. For example, you may need to deactivate a failed OST that cannot be immediately repaired, but want to continue to access the remaining files on the available OSTs. To temporarily deactivate an OST: 1. Mount the Lustre file system. 2. On the MDS and all clients, run: # lctl set_param osc.--*.active=0 Clients accessing files on the deactivated OST receive an IO error (-5), rather than pausing until the OST completes recovery. 4.3.13.2 Restoring an OST in the File System Restoring an OST to the file system is as easy as activating it. When the OST is active, it is automatically added to the normal stripe rotation and files are written to it. To restore an OST: 1. Make sure the OST to be restored is running. 2. Reactivate the OST. On the MGS, run: # mgs> lctl conf_param .osc.active=1 4.3.14 Aborting Recovery You can abort recovery with either the lctl utility or by mounting the target with the abort_recov option (mount -o abort_recov). When starting a target, run: $ mount -t lustre -L -o abort_recov Note – The recovery process is blocked until all OSTs are available.4-28 Lustre 1.8 Operations Manual • December 2010 4.3.15 Determining Which Machine is Serving an OST In the course of administering a Lustre file system, you may need to determine which machine is serving a specific OST. It is not as simple as identifying the machine’s IP address, as IP is only one of several networking protocols that Lustre uses and, as such, LNET does not use IP addresses as node identifiers, but NIDs instead. To identify the NID that is serving a specific OST, run one of the following commands on a client (you do not need to be a root user): client$ lctl get_param osc.${fsname}-${OSTname}*.ost_conn_uuid For example: client$ lctl get_param osc.*-OST0000*.ost_conn_uuid osc.myth-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp - OR - client$ lctl get_param osc.*.ost_conn_uuid osc.myth-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp osc.myth-OST0001-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp osc.myth-OST0002-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp osc.myth-OST0003-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp osc.myth-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcpChapter 4 Configuring Lustre 4-29 4.4 More Complex Configurations If a node has multiple network interfaces, it may have multiple NIDs. When a node is specified, all of its NIDs must be listed, delimited by commas (,) so other nodes can choose the NID that is appropriate for their network interfaces. When failover nodes are specified, they are delimited by a colon (:) or by repeating a keyword (--mgsnode= or --failnode=). To obtain all NIDs from a node (while LNET is running), run: lctl list_nids This displays the server's NIDs (networks configured to work with Lustre). 4.4.1 Failover This example has a combined MGS/MDT failover pair on uml1 and uml2, and a OST failover pair on uml3 and uml4. There are corresponding Elan addresses on uml1 and uml2. uml1> mkfs.lustre --fsname=testfs --mdt --mgs \ --failnode=uml2,2@elan /dev/sda1 uml1> mount -t lustre /dev/sda1 /mnt/test/mdt uml3> mkfs.lustre --fsname=testfs --ost --failnode=uml4 \ --mgsnode=uml1,1@elan --mgsnode=uml2,2@elan /dev/sdb uml3> mount -t lustre /dev/sdb /mnt/test/ost0 client> mount -t lustre uml1,1@elan:uml2,2@elan:/testfs /mnt/testfs uml1> umount /mnt/mdt uml2> mount -t lustre /dev/sda1 /mnt/test/mdt uml2> cat /proc/fs/lustre/mds/testfs-MDT0000/recovery_status Where multiple NIDs are specified, comma-separation (for example, uml2,2@elan) means that the two NIDs refer to the same host, and that Lustre needs to choose the "best" one for communication. Colon-separation (for example, uml1:uml2) means that the two NIDs refer to two different hosts, and should be treated as failover locations (Lustre tries the first one, and if that fails, it tries the second one.)4-30 Lustre 1.8 Operations Manual • December 2010 Note – If you have an MGS or MDT configured for failover, perform these steps: 1. On the OST, list the NIDs of all MGS nodes at mkfs time. OST# mkfs.lustre --fsname sunfs --ost --mgsnode=10.0.0.1 --mgsnode=10.0.0.2 /dev/{device} 2. On the client, mount the file system. client# mount -t lustre 10.0.0.1:10.0.0.2:/sunfs /cfs/client/ 4.5 Operational Scenarios In the operational scenarios below, the management node is the MDS. The management server is co-located on the MDS and started with the MDT. Tip – All targets that are configured for failover must have some kind of shared storage among two server nodes. IP Network, Combined MGS/MDS, Single OST, No Failover On the MDS, run: mkfs.lustre --mgs --mdt --fsname= mount -t lustre On the OSS, run: mkfs.lustre --ost --mgsnode= --fsname= mount -t lustre On the client, run: mount -t lustre :/ Chapter 4 Configuring Lustre 4-31 IP Network, Failover MGS/MDS For failover, storage holding target data must be available as shared storage to failover server nodes. Failover nodes are statically configured as mount options. On the MDS, run: mkfs.lustre --mgs --mdt --fsname= \ --failover= mount -t lustre On the OSS, run: mkfs.lustre --ost --mgsnode=[,] \ --failover= --fsname= mount -t lustre On the client, run: mount -t lustre [,]:/ \ IP Network, Failover MGS/MDS and OSS On the MDS, run: mkfs.lustre --mgs --mdt \ --failover= --fsname= mount -t lustre On the OSS, run: mkfs.lustre --ost --mgsnode=[,] \ --failover= --fsname= mount -t lustre On the client, run: mount -t lustre [,]:/ \ 4.5.1 Changing the Address of a Failover Node To change the address of a failover node (e.g, to use node X instead of node Y), run this command on the OSS/OST partition: tunefs.lustre --erase-params --failnode= 4-32 Lustre 1.8 Operations Manual • December 20105-1 C H A P T E R 5 Service Tags This chapter describes the use of service tags with Lustre, and includes the following sections: ¦ Introduction to Service Tags ¦ Using Service Tags Note – Service tags have been discontinued in Lustre 1.8.4 and later releases. The information in this chapter is provided for the convenience of users on Lustre version 1.8.3 and earlier releases. 5.1 Introduction to Service Tags Service tags are part of an IT asset inventory management system provided by Oracle. A service tag is a unique identifier for a piece of hardware or software (gear) that enables usage data about the tagged item to be shared over a local network in standard XML format. The service tag program is used for a number of Oracle products, including hardware, software and services, and has now been implemented for Lustre. Service tags are provided for each MGS, MDS, OSS node and Lustre client. Using service tags enables automatic discovery and tracking of these system components, so administrators can better manage their Lustre environment.5-2 Lustre 1.8 Operations Manual • December 2010 Note – Service tags are used solely to provide an inventory list of system and software information to Oracle; they do not contain any personal information. Service tag components that communicate information are read-only and contained. They are not capable of accepting information and they cannot communicate with any other services on your system. For more information on service tags, see the Service Tag wiki and Service Tag FAQ.Chapter 5 Service Tags 5-3 5.2 Using Service Tags To begin using service tags with your Lustre system, download the service tag package and registration client. The entire service tag process can be easily managed from the Sun Inventory webpage. 5.2.1 Installing Service Tags Service tag packages (for RedHat and SuSE Linux) are downloadable from the Lustre downloads page. To download and install the service tags package: 1. Navigate to the Lustre download page and download the service tag package, sun-servicetag-1.1.4-1.i386.rpm1 , for Lustre. 2. Install the service tag package on all Lustre nodes (MGSs, MDSs, OSSs and clients). The service tag package includes several init.d scripts which are started on reboot (/etc/init.d/stosreg and /etc/init.d/psn start). This package also adds entries in the [x]inetd’s configuration scripts to provide remote access to the nodes needed to collect information. The script restarts [x]inetd (killall -HUP xinetd 1>/dev/null 2>&1). 3. If this is a new installation, format the OSTs, MDTs, MGSs and Lustre clients. 4. Mount the OSTs, MDTs, MGSs and Lustre clients, and verify that the Lustre file system is running normally. 1. This is the current service tag package. The version number is subject to change.5-4 Lustre 1.8 Operations Manual • December 2010 5.2.2 Discovering and Registering Lustre Components After installing the service tag package on all of your Lustre nodes, discover and register the Lustre components. To perform this procedure, Lustre must be fully configured and running. 1. Navigate to the Oracle/Sun Inventory page and download the Registration client (client.jnlp) by clicking Discover & Register. 2. Install Java Virtual Machine (Java VM) on the collection node. Java VM is available at the Java download site. 3. Start the Registration client, run: $ javaws client.jnlp The Registration Client utility launches. FIGURE 5-1 Registration ClientChapter 5 Service Tags 5-5 Note – The Registration client requires an X display to run. If the node from which you want to do the registration has no native X display, you can use SSH’s X forwarding to display the Registration client interface on your local machine. The registration process includes up to five steps. The first step is to discover the service tags created when you started Lustre. The Registration client looks for Sun products on your local subnet, by default. Alternately, you can specify another subnet, specific hosts or IP addresses. 4. Select an option to locate service tags and click Next. The Product Data screen displays Sun products (that support service tags) as they are located. For each product, the system name, product name, and version (if applicable) are listed. FIGURE 5-2 Product Data If the list of located products does not look complete, select Back and enter a more accurate search. Note – Located service tags are not limited to Lustre components. The Registration client locates any Sun product on your system that is supported in the Sun inventory management program.5-6 Lustre 1.8 Operations Manual • December 2010 5. Register the service tags or save them for later use. There are two options for registering service tags. ¦ Click Next to continue with the remaining steps 3-5 of the registration process, including authentication to the Inventory management website and uploading your service tags. ¦ Save the collected service tags and register them on another machine. This option is good if the system used to collect the service tags does not have Web access. Click Save As and enter a file where the tags should be saved. You can then move this file (using network copy, a USB key, etc.) to a machine with Web access. On the Web-access machine, navigate to Sun Inventory and click Discover & Register to start the Registration client. Select the ‘Locate Product on Other Subnets, Specific System or Load Previously Saved Data’ option and check the ‘File Name’ box. Enter (or navigate to) the file where the collected service tags were saved, click Next and follow the remaining steps 3-5 to complete the registration process, including authentication to the Inventory management website and uploading your service tags. 6. If you wish, navigate to Sun Inventory and log into your account to view and manage your IT assets. Note – For more information about service tags, see https://inventory.sun.com, which links to the http://wikis.sun.com/display/ServiceTag/Home wiki. This wiki includes an FAQ about the service tag program.Chapter 5 Service Tags 5-7 5.2.3 Service Tag Registration Information The service tag registration process collects the following product, registration agentry and system information. Data Name Description Product Information Lustre-specific information Node type (client, MDS, OSS or MGS) Instance identifier Unique identifier for that instance of the gear Product name Name of the gear Product identifier Unique identifier for the gear being registered Product vendor Vendor of the gear Product version Version of the gear Parent name Parent gear of the registered gear Parent identifier Unique identifier for the parent of the gear Customer tag Optional, customer-defined value Time stamp Day and time that the gear is registered Source Where the gear identifiers came from Container Name of the gear's container Registration Agentry Information Agentry Identifier Unique value for that instance of the agentry Agentry Version Value of the agentry Registry Identifier File version containing product registration information System Information Host System hostname System Operating System Release Operating system version Architecture Physical hardware architecture Platform Hardware platform Manufacturer Hardware manufacturer CPU manufacturer CPU manufacturer HostID System host ID Serial number System chassis serial number5-8 Lustre 1.8 Operations Manual • December 20106-1 C H A P T E R 6 Configuring Lustre - Examples This chapter provides Lustre configuration examples and includes the following section: ¦ Simple TCP Network 6.1 Simple TCP Network This chapter presents several examples of Lustre configurations on a simple TCP network. 6.1.1 Lustre with Combined MGS/MDT Below is an example is of a Lustre setup “datafs” having combined MDT/MGS with four OSTs and a number of Lustre clients. 6.1.1.1 Installation Summary ¦ Combined (co-located) MDT/MGS ¦ Four OSTs ¦ Any number of Lustre clients6-2 Lustre 1.8 Operations Manual • December 2010 6.1.1.2 Configuration Generation and Application 1. Install the Lustre RPMS (per Installing Lustre) on all nodes that are going to be part of the Lustre file system. Boot the nodes in Lustre kernel, including the clients. 2. Change modprobe.conf by adding the following line to it. options lnet networks=tcp 3. Configuring Lustre on MGS and MDT node. $ mkfs.lustre --fsname datafs --mdt --mgs /dev/sda 4. Make a mount point on MDT/MGS for the file system and mount it. $ mkdir -p /mnt/data/mdt $ mount -t lustre /dev/sda /mnt/data/mdt 5. Configuring Lustre on all four OSTs. mkfs.lustre --fsname datafs --ost --mgsnode=mds16@tcp0 /dev/sda mkfs.lustre --fsname datafs --ost --mgsnode=mds16@tcp0 /dev/sdd mkfs.lustre --fsname datafs --ost --mgsnode=mds16@tcp0 /dev/sda1 mkfs.lustre --fsname datafs --ost --mgsnode=mds16@tcp0 /dev/sdb Note – While creating the file system, make sure you are not using disk with the operating system. 6. Make a mount point on all the OSTs for the file system and mount it. $ mkdir -p /mnt/data/ost0 $ mount -t lustre /dev/sda /mnt/data/ost0 $ mkdir -p /mnt/data/ost1 $ mount -t lustre /dev/sdd /mnt/data/ost1 $ mkdir -p /mnt/data/ost2 $ mount -t lustre /dev/sda1 /mnt/data/ost2 $ mkdir -p /mnt/data/ost3 $ mount -t lustre /dev/sdb /mnt/data/ost3 $ mount -t lustre mdt16@tcp0:/datafs /mnt/datafsChapter 6 Configuring Lustre - Examples 6-3 6.1.2 Lustre with Separate MGS and MDT The following example describes a Lustre file system “datafs” having an MGS and an MDT on separate nodes, four OSTs, and a number of Lustre clients. 6.1.2.1 Installation Summary ¦ One MGS ¦ One MDT ¦ Four OSTs ¦ Any number of Lustre clients 6.1.2.2 Configuration Generation and Application 1. Install the Lustre RPMs (per Installing Lustre) on all the nodes that are going to be a part of the Lustre file system. Boot the nodes in the Lustre kernel, including the clients. 2. Change the modprobe.conf by adding the following line to it. options lnet networks=tcp 3. Start Lustre on the MGS node. $ mkfs.lustre --mgs /dev/sda 4. Make a mount point on MGS for the file system and mount it. $ mkdir -p /mnt/mgs $ mount -t lustre /dev/sda1 /mnt/mgs 5. Start Lustre on the MDT node. $ mkfs.lustre --fsname=datafs --mdt --mgsnode=mgsnode@tcp0 \ /dev/sda2 6. Make a mount point on MDT/MGS for the file system and mount it. $ mkdir -p /mnt/data/mdt $ mount -t lustre /dev/sda /mnt/data/mdt 7. Start Lustre on all the four OSTs. mkfs.lustre --fsname datafs --ost --mgsnode=mds16@tcp0 /dev/sda mkfs.lustre --fsname datafs --ost --mgsnode=mds16@tcp0 /dev/sdd mkfs.lustre --fsname datafs --ost --mgsnode=mds16@tcp0 /dev/sda1 mkfs.lustre --fsname datafs --ost --mgsnode=mds16@tcp0 /dev/sdb6-4 Lustre 1.8 Operations Manual • December 2010 8. Make a mount point on all the OSTs for the file system and mount it. $ mkdir -p /mnt/data/ost0 $ mount -t lustre /dev/sda /mnt/data/ost0 $ mkdir -p /mnt/data/ost1 $ mount -t lustre /dev/sdd /mnt/data/ost1 $ mkdir -p /mnt/data/ost2 $ mount -t lustre /dev/sda1 /mnt/data/ost2 $ mkdir -p /mnt/data/ost3 $ mount -t lustre /dev/sdb /mnt/data/ost3 $ mount -t lustre mdsnode@tcp0:/datafs /mnt/datafs 6.1.2.3 Configuring Lustre with a CSV File A new utility (script) - /usr/sbin/lustre_config can be used to configure Lustre 1.6 and later. This script enables you to automate formatting and setup of disks on multiple nodes. Describe your entire installation in a Comma Separated Values (CSV) file and pass it to the script. The script contacts multiple Lustre targets simultaneously, formats the drives, updates modprobe.conf, and produces HA configuration files using definitions in the CSV file. (The lustre_config -h option shows several samples of CSV files.) Note – The CSV file format is a file type that stores tabular data. Many popular spreadsheet programs, such as Microsoft Excel, can read from/write to CSV files. How lustre_config Works The lustre_config script parses each line in the CSV file and executes remote commands, like mkfs.lustre, to format each Lustre target in the Lustre cluster. Optionally, the lustre_config script can also: ¦ Verify network connectivity and hostnames in the cluster ¦ Configure Linux MD/LVM devices ¦ Modify /etc/modprobe.conf to add Lustre networking information ¦ Add the Lustre server information to /etc/fstab ¦ Produce configurations for Heartbeat or CluManagerChapter 6 Configuring Lustre - Examples 6-5 How to Create a CSV File Five different types of line formats are available to create a CSV file. Each line format represents a target. The list of targets with the respective line formats are described below: Linux MD device The CSV line format is: hostname, MD, md name, operation mode, options, raid level, component devices Where: Variable Supported Type hostname Hostname of the node in the cluster. MD Marker of the MD device line. md name MD device name, for example: /dev/md0 operation mode Operations mode, either create or remove. Default is create. options A ‘catchall’ for other mdadm options, for example, -c 128 raid level RAID level: 0, 1, 4, 5, 6, 10, linear and multipath. hostname Hostname of the node in the cluster. component devices Block devices to be combined into the MD device. Multiple devices are separated by space or by using shell extensions, for example: /dev/sd{a,b,c}6-6 Lustre 1.8 Operations Manual • December 2010 Linux LVM PV (Physical Volume) The CSV line format is: hostname, PV, pv names, operation mode, options Where: Linux LVM VG (Volume Group) The CSV line format is: hostname, VG, vg name, operation mode, options, pv paths Where: Variable Supported Type hostname Hostname of the node in the cluster. PV Marker of the PV line. pv names Devices or loopback files to be initialized for later use by LVM or to wipe the label, for example: /dev/sda Multiple devices or files are separated by space or by using shell expansions, for example: /dev/sd{a,b,c} operation mode Operations mode, either create or remove. Default is create. options A ‘catchall’ for other pvcreate/pvremove options, for example: -vv Variable Supported Type hostname Hostname of the node in the cluster. VG Marker of the VG line. vg name Name of the volume group, for example: ost_vg operation mode Operations mode, either create or remove. Default is create. options A ‘catchall’ for other vgcreate/rgremove options, for example: -s 32M pv paths Physical volumes to construct this VG, required by the create mode; multiple PVs are separated by space or by using shell expansions, for example: /dev/sd[k-m]1Chapter 6 Configuring Lustre - Examples 6-7 Linux LVM LV (Logical Volume) The CSV line format is: hostname, LV, lv name, operation mode, options, lv size, vg name Where: Variable Supported Type hostname Hostname of the node in the cluster. LV Marker of the LV line. lv name Name of the logical volume to be created (optional) or path of the logical volume to be removed (required by the remove mode). operation mode Operations mode, either create or remove. Default is create. options A ‘catchall’ for other lvcreate/lvremove options, for example: -i 2 -l 128 lv size Size [kKmMgGtT] to be allocated for the new LV. Default is megabytes (MB). vg name Name of the VG in which the new LV is created.6-8 Lustre 1.8 Operations Manual • December 2010 Lustre target The CSV line format is: hostname, module_opts, device name, mount point, device type, fsname, mgs nids, index, format options, mkfs options, mount options, failover nids Where: Note – In one node, all NIDs are delimited by commas (','). To use comma-separated NIDs in a CSV file, they must be enclosed in quotation marks, for example: "lustre-mgs2,2@elan" When multiple nodes are specified, they are delimited by a colon (':'). If you leave a blank, it is set to default. Variable Supported Type hostname Hostname of the node in the cluster. It must match uname -n module_opts Lustre networking module options. Use the newline character (\n) to delimit multiple options. device name Lustre target (block device or loopback file). mount point Lustre target mount point. device type Lustre target type (mgs, mdt, ost, mgs|mdt, mdt|mgs). fsname Lustre file system name (limit is 8 characters). mgs nids NID(s) of the remote mgs node, required for MDT and OST targets; if this item is not given for an MDT, it is assumed that the MDT is also an MGS (according to mkfs.lustre). index Lustre target index. format options A ‘catchall’ contains options to be passed to mkfs.lustre. For example: device-size, --param, and so on. mkfs options Format options to be wrapped with --mkfsoptions= and passed to mkfs.lustre. mount options If this script is invoked with -m option, then the value of this item is wrapped with --mountfsoptions= and passed to mkfs.lustre; otherwise, the value is added into /etc/ fstab failver nids NID(s) of the failover partner node.Chapter 6 Configuring Lustre - Examples 6-9 The lustre_config.csv file looks like: {mdtname}.{domainname},options lnet networks= tcp,/dev/sdb,/mnt/mdt,mgs|mdt {ost2name}.{domainname},options lnet networks= tcp,/dev/sda,/mnt/ost1,ost,,192.168.16.34@tcp0 {ost1name}.{domainname},options lnet networks= tcp,/dev/sda,/mnt/ost0,ost,,192.168.16.34@tcp0 Note – Provide a Fully Qualified Domain Name (FQDN) for all nodes that are a part of the file system in the first parameter of all the rows starting in a new line. For example: mdt1.clusterfs.com,options lnet networks= tcp,/dev/sdb,/mnt/mdt,mgs|mdt - AND - ost1.clusterfs.com,options lnet\ networks=tcp,/dev/sda,/mnt/ ost1,ost,,192.168.16.34@tcp06-10 Lustre 1.8 Operations Manual • December 2010 Using CSV with lustre_config Once you created the CSV file, you can start to configure the file system by using the lustre_config script. 1. List the available parameters. At the command prompt. Type: $ lustre_config lustre_config: Missing csv file! Usage: lustre_config [options] This script is used to format and set up multiple lustre servers from a csv file. Options: -h help and examples -a select all the nodes from the csv file to operate on -w hostname,hostname,... select the specified list of nodes (separated by commas) to operate on rather than all the nodes in the csv file -x hostname,hostname,... exclude the specified list of nodes (separated by commas) -t HAtype produce High-Availability software configurations The argument following -t is used to indicate the High-Availability software type. The HA software types which are currently supported are: hbv1 (Heartbeat version 1) and hbv2 (Heartbeat version 2). -n no net - don’t verify network connectivity and hostnames in the cluster -d configure Linux MD/LVM devices before formatting the Lustre targets -f force-format the Lustre targets using --reformat option OR you can specify --reformat in the ninth field of the target line in the csv file -m no fstab change - don’t modify /etc/fstab to add the new Lustre targets. If using this option, then the value of "mount options" item in the csv file will be passed to mkfs.lustre,else the value will be added into the /etc/fstab -v verbose mode csv file is a spreadsheet that contains configuration parameters (separated by commas) for each target in a Lustre clusterChapter 6 Configuring Lustre - Examples 6-11 Example 1: Simple Lustre configuration with CSV (use the following command): $ lustre_config -v -a -f lustre_config.csv This command starts the execution and configuration on the nodes or targets in lustre_config.csv, prompting you for the password to log in with root access to the nodes. To avoid this prompt, configure a shell like pdsh or SSH. After completing the above steps, the script makes Lustre target entries in the /etc/fstab file on Lustre server nodes, such as: /dev/sdb /mnt/mdtlustre defaults 0 0 /dev/sda /mnt/ostlustre defaults 0 0 2. Run mount /dev/sdb and mount /dev/sda to start the Lustre services. Note – Use the /usr/sbin/lustre_createcsv script to collect information on Lustre targets from running a Lustre cluster and generating a CSV file. It is a reverse utility (compared to lustre_config) and should be run on the MGS node. Example 2: More complicated Lustre configuration with CSV: For RAID and LVM-based configuration, the lustre_config.csv file looks like this: # Configuring RAID 5 on mds16.clusterfs.com mds16.clusterfs.com,MD,/dev/md0,,-c 128,5,/dev/sdb /dev/sdc /dev/sdd # configuring multiple RAID5 on oss161.clusterfs.com oss161.clusterfs.com,MD,/dev/md0,,-c 128,5,/dev/sdb /dev/sdc /dev/sdd oss161.clusterfs.com,MD,/dev/md1,,-c 128,5,/dev/sde /dev/sdf /dev/sdg # configuring LVM2-PV from the RAID5 from the above steps on oss161.clusterfs.com oss161.clusterfs.com,PV,/dev/md0 /dev/md1 # configuring LVM2-VG from the PV and RAID5 from the above steps on oss161.clusterfs.com oss161.clusterfs.com,VG,oss_data,,-s 32M,/dev/md0 /dev/md1 # configuring LVM2-LV from the VG, PV and RAID5 from the above steps on oss161.clusterfs.com oss161.clusterfs.com,LV,ost0,,-i 2 -I 128,2G,oss_data oss161.clusterfs.com,LV,ost1,,-i 2 -I 128,2G,oss_data6-12 Lustre 1.8 Operations Manual • December 2010 # configuring LVM2-PV on oss162.clusterfs.com oss162.clusterfs.com,PV, /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg # configuring LVM2-VG from the PV from the above steps on oss162.clusterfs.com oss162.clusterfs.com,VG,vg_oss1,,-s 32M,/dev/sdb /dev/sdc /dev/sdd oss162.clusterfs.com,VG,vg_oss2,,-s 32M,/dev/sde /dev/sdf /dev/sdg # configuring LVM2-LV from the VG and PV from the above steps on oss162.clusterfs.com oss162.clusterfs.com,LV,ost3,,-i 3 -I 64,1G,vg_oss2 oss162.clusterfs.com,LV,ost2,,-i 3 -I 64,1G,vg_oss1 #configuring Lustre file system on MDS/MGS, OSS and OST with RAID and LVM created above mds16.clusterfs.com,options lnet networks= tcp,/dev/md0,/mnt/mdt,mgs|mdt,,,,,,, oss161.clusterfs.com,options lnet networks= tcp,/dev/oss_data/ost0,/mnt/ost0,ost,,192.168.16.34@tcp0,,,, oss161.clusterfs.com,options lnet networks= tcp,/dev/oss_data/ost1,/mnt/ost1,ost,,192.168.16.34@tcp0,,,, oss162.clusterfs.com,options lnet networks= tcp,/dev/pv_oss1/ost2,/mnt/ost2,ost,,192.168.16.34@tcp0,,,, oss162.clusterfs.com,options lnet networks= tcp,/dev/pv_oss2/ost3,/mnt/ost3,ost,,192.168.16.34@tcp0,,,, $ lustre_config -v -a -d -f lustre_config.csv This command creates RAID and LVM, and then configures Lustre on the nodes or targets specified in lustre_config.csv. The script prompts you for the password to log in with root access to the nodes. After completing the above steps, the script makes Lustre target entries in the /etc/fstab file on Lustre server nodes, such as: For MDS | MDT: /dev/md0 /mnt/mdtlustre defaults00 For OSS: /pv_oss1/ost2 /mnt/ost2lustre defaults00 3. Start the Lustre services, run: mount /dev/sdb mount /dev/sda7-1 C H A P T E R 7 More Complicated Configurations This chapter describes more complicated Lustre configurations and includes the following sections: ¦ Multihomed Servers ¦ Elan to TCP Routing ¦ Load Balancing with InfiniBand ¦ Multi-Rail Configurations with LNET 7.1 Multihomed Servers If you are using multiple networks with Lustre, certain configuration settings are required. Throughout this section, a worked example is used to illustrate these settings. In this example, servers megan and oscar each have three TCP NICs (eth0, eth1, and eth2) and an Elan NIC. The eth2 NIC is used for management purposes and should not be used by LNET. TCP clients have a single TCP interface and Elan clients have a single Elan interface. 7.1.1 Modprobe.conf Options under modprobe.conf are used to specify the networks available to a node. You have the choice of two different options – the networks option, which explicitly lists the networks available and the ip2nets option, which provides a list-matching lookup. Only one option can be used at any one time. The order of LNET lines in modprobe.conf is important when configuring multi-homed servers. If a server node can be reached using more than one network, the first network specified in modprobe.conf will be used.7-2 Lustre 1.8 Operations Manual • December 2010 Networks On the servers: options lnet networks=tcp0(eth0, eth1),elan0 Elan-only clients: options lnet networks=elan0 TCP-only clients: options lnet networks=tcp0 Note – In the case of TCP-only clients, the first available non-loopback IP interface is used for tcp0 since the interfaces are not specified. ip2nets The ip2nets option is typically used to provide a single, universal modprobe.conf file that can be run on all servers and clients. An individual node identifies the locally available networks based on the listed IP address patterns that match the node's local IP addresses. Note that the IP address patterns listed in the ip2nets option are only used to identify the networks that an individual node should instantiate. They are not used by LNET for any other communications purpose. The servers megan and oscar have eth0 IP addresses 192.168.0.2 and .4. They also have IP over Elan (eip) addresses of 132.6.1.2 and .4. TCP clients have IP addresses 192.168.0.5-255. Elan clients have eip addresses of 132.6.[2-3].2, .4, .6, .8. modprobe.conf is identical on all nodes: options lnet 'ip2nets="tcp0(eth0,eth1)192.168.0.[2,4]; tcp0 \ 192.168.0.*; elan0 132.6.[1-3].[2-8/2]"' Note – LNET lines in modprobe.conf are only used by the local node to determine what to call its interfaces. They are not used for routing decisions. Because megan and oscar match the first rule, LNET uses eth0 and eth1 for tcp0 on those machines. Although they also match the second rule, it is the first matching rule for a particular network that is used. The servers also match the (only) Elan rule. The [2-8/2] format matches the range 2-8 stepping by 2; that is 2,4,6,8. For example, clients at 132.6.3.5 would not find a matching Elan network.Chapter 7 More Complicated Configurations 7-3 7.1.2 Start Servers For the combined MGS/MDT with TCP network, run: $ mkfs.lustre --fsname spfs --mdt --mgs /dev/sda $ mkdir -p /mnt/test/mdt $ mount -t lustre /dev/sda /mnt/test/mdt - OR - For the MGS on the separate node with TCP network, run: $ mkfs.lustre --mgs /dev/sda $ mkdir -p /mnt/mgs $ mount -t lustre /dev/sda /mnt/mgs For starting the MDT on node mds16 with MGS on node mgs16, run: $ mkfs.lustre --fsname=spfs --mdt --mgsnode=mgs16@tcp0 /dev/sda $ mkdir -p /mnt/test/mdt $ mount -t lustre /dev/sda2 /mnt/test/mdt For starting the OST on TCP-based network, run: $ mkfs.lustre --fsname spfs --ost --mgsnode=mgs16@tcp0 /dev/sda$ $ mkdir -p /mnt/test/ost0 $ mount -t lustre /dev/sda /mnt/test/ost07-4 Lustre 1.8 Operations Manual • December 2010 7.1.3 Start Clients TCP clients can use the host name or IP address of the MDS, run: mount –t lustre megan@tcp0:/mdsA/client /mnt/lustre Use this command to start the Elan clients, run: mount –t lustre 2@elan0:/mdsA/client /mnt/lustre Note – If the MGS node has multiple interfaces (for instance, cfs21 and 1@elan), only the client mount command has to change. The MGS NID specifier must be an appropriate nettype for the client (for example, a TCP client could use uml1@tcp0, and an Elan client could use 1@elan). Alternatively, a list of all MGS NIDs can be given, and the client chooses the correctd one. For example: $ mount -t lustre mgs16@tcp0,1@elan:/testfs /mnt/testfsChapter 7 More Complicated Configurations 7-5 7.2 Elan to TCP Routing Servers megan and oscar are on the Elan network with eip addresses 132.6.1.2 and .4. Megan is also on the TCP network at 192.168.0.2 and routes between TCP and Elan. There is also a standalone router, router1, at Elan 132.6.1.10 and TCP 192.168.0.10. Clients are on either Elan or TCP. 7.2.1 Modprobe.conf modprobe.conf is identical on all nodes, run: options lnet 'ip2nets="tcp0 192.168.0.*; elan0 132.6.1.*"' \ 'routes="tcp [2,10]@elan0; elan 192.168.0.[2,10]@tcp0"' 7.2.2 Start servers To start router1, run: modprobe lnet lctl network configure To start megan and oscar, run: $ mkfs.lustre --fsname spfs --mdt --mgs /dev/sda $ mkdir -p /mnt/test/mdt $ mount -t lustre /dev/sda /mnt/test/mdt $ mount -t lustre mgs16@tcp0,1@elan:/testfs /mnt/testfs 7.2.3 Start clients For the TCP client, run: mount -t lustre megan:/mdsA/client /mnt/lustre/ For the Elan client, run: mount -t lustre 2@elan0:/mdsA/client /mnt/lustre7-6 Lustre 1.8 Operations Manual • December 2010 7.3 Load Balancing with InfiniBand A Lustre file system contains OSSs with two InfiniBand HCAs. Lustre clients have only one InfiniBand HCA using OFED Infiniband ''o2ib'' drivers. Load balancing between the HCAs on the OSS is accomplished through LNET. 7.3.1 Setting Up modprobe.conf for Load Balancing To configure LNET for load balancing on clients and servers: 1. Set the modprobe.conf options. Depending on your configuration, set modprobe.conf options as follows: ¦ Dual HCA OSS server options lnet ip2nets= "o2ib0(ib0),o2ib1(ib1) 192.168.10.1.[101-102] ¦ Client with the odd IP address options lnet ip2nets=o2ib0(ib0) 192.168.10.[103-253/2] ¦ Client with the even IP address options lnet ip2nets=o2ib1(ib0) 192.168.10.[102-254/2] 2. Run the modprobe lnet command and create a combined MGS/MDT file system. The following commands create the MGS/MDT file system and mount the servers (MGS/MDT and OSS). modprobe lnet $ mkfs.lustre --fsname lustre --mgs --mdt $ mkdir -p $ mount -t lustre $ mount -t lustre $ mkfs.lustre --fsname lustre --mgs --mdt $ mkdir -p $ mount -t lustre $ mount -t lustre Chapter 7 More Complicated Configurations 7-7 For example: modprobe lnet $ mkfs.lustre --fsname lustre --mdt --mgs /dev/sda $ mkdir -p /mnt/test/mdt $ mount -t lustre /dev/sda /mnt/test/mdt $ mount -t lustre mgs@o2ib0:/lustre /mnt/mdt $ mkfs.lustre --fsname lustre --ost --mgsnode=mds@o2ib0 /dev/sda $ mkdir -p /mnt/test/mdt $ mount -t lustre /dev/sda /mnt/test/ost $ mount -t lustre mgs@o2ib0:/lustre /mnt/ost 3. Mount the clients. mount -t lustre :/ This example shows an IB client being mounted. mount -t lustre 192.168.10.101@o2ib0,192.168.10.102@o2ib1:/mds/client /mnt/lustre 7.4 Multi-Rail Configurations with LNET To aggregate bandwidth across both rails of a dual-rail IB cluster (o2iblnd) 1 using LNET, consider these points: ¦ LNET can work with multiple rails, however, it does not load balance across them. The actual rail used for any communication is determined by the peer NID. ¦ Multi-rail LNET configurations do not provide an additional level of network fault tolerance. The configurations described below are for bandwidth aggregation only. Network interface failover is planned as an upcoming Lustre feature. ¦ A Lustre node always uses the same local NID to communicate with a given peer NID. The criteria used to determine the local NID are: ¦ Fewest hops (to minimize routing), and ¦ Appears first in the "networks" or "ip2nets" LNET configuration strings 1. Multi-rail configurations are only supported by o2iblnd; other IB LNDs do not support multiple interfaces.7-8 Lustre 1.8 Operations Manual • December 2010 As an example, consider a two-rail IB cluster running the OFA stack (OFED) with these IPoIB address assignments. ib0 ib1 Servers 192.168.0.* 192.168.1.* Clients 192.168.[2-127].* 192.168.[128-253].* You could create these configurations: ¦ A cluster with more clients than servers. The fact that an individual client cannot get two rails of bandwidth is unimportant because the servers are the actual bottleneck. ip2nets="o2ib0(ib0), o2ib1(ib1)192.168.[0-1].* #all servers;\ o2ib0(ib0) 192.168.[2-253].[0-252/2]#even clients;\ o2ib1(ib1) 192.168.[2-253].[1-253/2]#odd clients" This configuration gives every server two NIDs, one on each network, and statically load-balances clients between the rails. ¦ A single client that must get two rails of bandwidth, and it does not matter if the maximum aggregate bandwidth is only (# servers) * (1 rail). ip2nets=" o2ib0(ib0) 192.168.[0-1].[0-252/2] #even servers;\ o2ib1(ib1) 192.168.[0-1].[1-253/2] #odd servers;\ o2ib0(ib0),o2ib1(ib1) 192.168.[2-253].* #clients" This configuration gives every server a single NID on one rail or the other. Clients have a NID on both rails. ¦ All clients and all servers must get two rails of bandwidth. ip2nets=” o2ib0(ib0),o2ib2(ib1) 192.168.[0-1].[0-252/2] #even servers;\ o2ib1(ib0),o2ib3(ib1) 192.168.[0-1].[1-253/2] #odd servers;\ o2ib0(ib0),o2ib3(ib1) 192.168.[2-253].[0-252/2)#even clients;\ o2ib1(ib0),o2ib2(ib1) 192.168.[2-253].[1-253/2)#odd clients" This configuration includes two additional proxy o2ib networks to work around Lustre's simplistic NID selection algorithm. It connects "even" clients to "even" servers with o2ib0 on rail0, and "odd" servers with o2ib3 on rail1. Similarly, it connects "odd" clients to "odd" servers with o2ib1 on rail0, and "even" servers with o2ib2 on rail1.8-1 C H A P T E R 8 Failover This chapter describes failover in a Lustre system and includes the following sections: ¦ What is Failover? ¦ Failover Functionality in Lustre ¦ Configuring and Using Heartbeat with Lustre Failover 8.1 What is Failover? A computer system is ''highly available'' when the services it provides are available with minimal downtime. In a highly-available system, if a failure condition occurs, such as the loss of a server or a network or software fault, the system’s services continue without interruption. Generally, we measure availability by the percentage of time the system is required to be available. Availability is accomplished by replicating hardware and/or software so that when a primary server fails or is unavailable, a standby server can be switched into its place to run applications and associated resources. This process, called failover, should be automatic and, in most cases, completely application-transparent. A failover hardware setup requires a pair of servers with a shared resource (typically a physical storage device, which may be based on SAN, NAS, hardware RAID, SCSI or FC technology). The method of sharing storage should be essentially transparent at the device level in that the same physical logical unit number (LUN) should be visible from both servers. To ensure high availability at the physical storage level, we encourage the use of RAID arrays to protect against drive-level failures.8-2 Lustre 1.8 Operations Manual • December 2010 8.1.1 Failover Capabilities To establish a highly-available Lustre file system, power management software or hardware and high availability (HA) software are used to provide the following failover capabilities: ¦ Resource fencing - Protects physical storage from simultaneous access by two nodes. ¦ Resource management - Starts and stops the Lustre resources as a part of failover, maintains the cluster state, and carries out other resource management tasks. ¦ Health monitoring - Verifies the availability of hardware and network resources and responds to health indications provided by Lustre. Although these capabilities can be provided by a variety of software and/or hardware solutions, the currently supported solution for Lustre is Heartbeat. For information about accessing the latest version of Heartbeat, see: www.sun.com/software/products/hpcsoftware/getit.jsp HA software is responsible for detecting failure of the primary Lustre server node and controlling the failover. Lustre works with any HA software that supports resource (I/O) fencing. For proper resource fencing, the HA software must be able to completely power off the failed server or disconnect it from the shared storage device. If two active nodes have access to the same storage device, data may be severely corrupted. 8.1.2 Types of Failover Configurations Nodes in a cluster can be configured for failover in several ways. They are often configured in pairs (for example, two OSTs attached to a shared storage device), but other failover configurations are also possible. Failover configurations include: ¦ Active/passive pair - In this configuration, the active node provides resources and serves data, while the passive node is usually standing by idle. If the active node fails, the passive node takes over and becomes active. ¦ Active/active pair - In this configuration, both nodes are active, each providing a subset of resources. In case of a failure, the second node takes over resources from the failed node. The active/passive configuration is seldom used for OST servers as it doubles hardware costs without improving performance. On the other hand, an active/active cluster configuration can improve performance by serving and providing arbitrary failover protection to a number of OSTs. In an active/active configuration, multiple OSS nodes are configured to serve the same OST, but only one OSS node can serve the OST at a time. The OST must never be active on more than one OSS at a time.Chapter 8 Failover 8-3 8.2 Failover Functionality in Lustre The failover functionality provided in Lustre supports the following failover scenario. When a client attempts to do I/O to a failed Lustre target, it continues to try until it receives an answer from any of the configured failover nodes for the Lustre target. A user-space application does not detect anything unusual, except that the I/O may take longer than usual to complete. Lustre failover requires two nodes configured as a failover pair, which must share one or more storage devices. Lustre can be configured to provide MDT or OST failover. ¦ For MDT failover, two MDSs are configured to serve the same MDT. Only one MDS node can serve an MDT at a time. ¦ For OST failover, multiple OSS nodes are configured to be able to serve the same OST. However, only one OSS node can serve the OST at a time. An OST can be moved between OSS nodes that have access to the same storage device using umount/mount commands. To add a failover partner to a Lustre configuration, the --failnode option is used. This can be done at creation time (using mkfs.lustre) or later when the Lustre system is active (using tunefs.lustre). For explanations of these utilities, see mkfs.lustre and tunefs.lustre. For a failover example, see More Complicated Configurations. Note – Failover is supported in Lustre only at the file system level. In a complete failover solution, support for system-level components, such as node failure detection or power control, is provided by a third party tool. Caution – OST failover functionality does not protect against corruption caused by a disk failure. If the storage media (i.e., physical disk) used for an OST fails, Lustre cannot recover it. We strongly recommend that some form of RAID be used for OSTs. Lustre functionality assumes that the storage is reliable, so it adds no extra reliability features.8-4 Lustre 1.8 Operations Manual • December 2010 8.2.1 MDT Failover Configuration (Active/Passive) Two MDSs are usually configured as an active/passive failover pair. Note that both nodes must have access to shared storage for the MDT(s) and the MGS. The primary (active) MDS manages the Lustre system metadata resources. If the primary MDS fails, the secondary (passive) MDS takes over these resources and serves the MDTs and the MGS. Note – In an environment with multiple file systems, the MDSs can be configured in a quasi active/active configuration, with each MDS managing metadata for a subset of the Lustre file system. 8.2.2 OST Failover Configuration (Active/Active) OSTs are usually configured in a load-balanced, active/active failover configuration. A failover cluster is built from two OSSs. Note – OSSs configured as a failover pair must have shared disks/RAID. In an active configuration, 50% of the available OSTs are assigned to one OSS and the remaining OSTs are assigned to the other OSS. Each OSS serves as the primary node for half the OSTs and as a failover node for the remaining OSTs. In this mode, if one OSS fails, the other OSS takes over all of the failed OSTs. The clients attempt to connect to each OSS serving the OST, until one of them responds. Data on the OST is written synchronously, and the clients replay transactions that were in progress and uncommitted to disk before the OST failure. 8.2.3 Lustre Failover and MMP The failover functionality in Lustre is supported by the multiple mount protection (MMP) feature, which protects the file system from being mounted simultaneously to more than one node. This feature is important in a shared storage environment (for example, when a failover pair of OSTs share a partition). Lustre's backend file system, ldiskfs, supports the MMP mechanism. A block in the file system is updated by a kmmpd daemon at one second intervals, and a sequence number is written in this block. If the file system is cleanly unmounted, then a special "clean" sequence is written to this block. When mounting the file system, ldiskfs checks if the MMP block has a clean sequence or not.Chapter 8 Failover 8-5 Even if the MMP block has a clean sequence, ldiskfs waits for some interval to guard against the following situations: ¦ If I/O traffic is heavy, it may take longer for the MMP block to be updated. ¦ If another node is trying to mount the same file system, a "race" condition may occur. With MMP enabled, mounting a clean file system takes at least 10 seconds. If the file system was not cleanly unmounted, then the file system mount may require additional time. Note – The MMP feature is only supported on Linux kernel versions >= 2.6.9. 8.2.3.1 Working with MMP On a new Lustre file system, MMP is automatically enabled by mkfs.lustre at format time if failover is being used and the kernel and e2fsprogs version support it. On an existing file system, a Lustre administrator can manually enable MMP when the file system is unmounted. Use the following commands to determine whether MMP is running in Lustre and to enable or disable the MMP feature. To determine if MMP is enabled, run: dumpe2fs -h |grep mmp Here is a sample command: dumpe2fs -h /dev/sdc | grep mmp Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent mmp sparse_super large_file uninit_bg To manually disable MMP, run: tune2fs -O ^mmp To manually enable MMP, run: tune2fs -O mmp When MMP is enabled, if ldiskfs detects multiple mount attempts after the file system is mounted, it blocks these later mount attempts and reports the time when the MMP block was last updated, the node name, and the device name of the node where the file system is currently mounted.8-6 Lustre 1.8 Operations Manual • December 2010 8.3 Configuring and Using Heartbeat with Lustre Failover This section describes how to configure Lustre failover using the Heartbeat cluster infrastructure daemon. 8.3.1 Creating a Failover Environment Lustre provides failover mechanisms only at the file system level. No failover support is provided for system-level components, such as node failure detection or power control, as would typically be provided in a complete failover solution. Additional tools are also needed to provide resource fencing, control and monitoring. 8.3.1.1 Power Management Software Lustre failover requires power control and management capability to verify that a failed node is shut down before I/O is directed to the failover node. This avoids double-mounting the two nodes, and the risk of unrecoverable data corruption. A variety of power management tools will work, but two packages that are commonly used with Lustre are STONITH and PowerMan. Shoot The Other Node In The HEAD (STONITH), is a set of power management tools provided with the Linux-HA package. STONITH has native support for many power control devices and is extensible. It uses expect scripts to automate control. PowerMan, available from the Lawrence Livermore National Laboratory (LLNL), is used to control remote power control (RPC) devices from a central location. PowerMan provides native support for several RPC varieties and expect-like configuration simplifies the addition of new devices. The latest versions of PowerMan are available at: sourceforge.net/projects/powerman For more information about PowerMan, go to: computing.llnl.gov/linux/powerman.htmlChapter 8 Failover 8-7 8.3.1.2 Power Equipment Lustre failover also requires the use of RPC devices, which come in different configurations. Lustre server nodes may be equipped with some kind of service processor that allows remote power control. If a Lustre server node is not equipped with a service processor, then a multi-port, Ethernet-addressable RPC may be used as an alternative. For recommended products, refer to the list of supported RPC devices on the PowerMan website. computing.llnl.gov/linux/powerman.html 8.3.2 Setting up the Heartbeat Software Lustre must be combined with high-availability (HA) software to enable a complete Lustre failover solution. Lustre can be used with different HA packages, including Heartbeat, the Linux-HA software. For current information about Heartbeat, see linux-ha.org/wiki. The Heartbeat package is one of the core components of the Linux-HA project. Heartbeat is highly-portable and runs on every known Linux platform, as well as FreeBSD and Solaris. This section describes how to install Heartbeat v2 and configure it with and without STONITH. Because Heartbeat v1 has simpler configuration files, which can be used with both Heartbeat v1 and v2, the configuration examples show how to configure Heartbeat using Heartbeat v1 configuration files. Heartbeat v2 adds monitoring and supports more complex cluster topologies, and the Heartbeat v2 configuration is stored as an XML file. To support users with Heartbeat v2, this section also includes a procedure to migrate Heartbeat v1 configuration files to v2.8-8 Lustre 1.8 Operations Manual • December 2010 8.3.2.1 Installing Heartbeat 1. Install Lustre (see Installing Lustre). 2. Install the Heartbeat packages. Heartbeat v2 requires several packages. This example uses Heartbeat v. 2.1.4. The required Heartbeat packages are, in order: ¦ heartbeat-stonith -> heartbeat-stonith-2.1.4-1.x86_64.rpm ¦ heartbeat-pils -> heartbeat-pils-2.1.4-1.x86_64.rpm ¦ heartbeat -> heartbeat-2.1.4-1.x86_64.rpm You can download the Heartbeat packages and guides covering basic setup and testing here: www.sun.com/software/products/hpcsoftware/getit.jsp Heartbeat packages are available for many Linux distributions. Additionally, Heartbeat has some dependencies on other packages. It is recommended that you use a package manager like yum, yast or aptitude to install the Heartbeat packages and resolve their package dependencies. 8.3.2.2 Configuring Heartbeat This section describes Heartbeat configuration and provides a worked example to illustrate the configuration steps. Note – Depending on the particular packaging, Heartbeat files may be located in a different directory or path than indicated in the following procedures.Chapter 8 Failover 8-9 For remote power control, both OSS nodes are equipped with a service processor (SP). The SPs are accessible over the network via their hostnames. Individual node parameters are listed below. Configuring Heartbeat without STONITH Note – This procedure describes Heartbeat configuration using a v1 configuration file, which can be used with both Heartbeat v1 and v2. See (Optional) Migrating a Heartbeat Configuration (v1 to v2) for an optional procedure to convert the v1 configuration file to an XML-formatted v2 configuration file. Note – Depending on the particular packaging, Heartbeat files may be located in a different directory or path than indicated in the following procedure. For example, they may be located in /etc/ha.d/ or /var/lib/heartbeat. Parameters Value Description First OSS node OSS node oss01 First OSS node in the Lustre file system OST ost01 First OST in the Lustre file system block device /dev/sda Block device for the first OSS node (oss01) mount point /mnt/ost1 Mount point for the oss01 block device (/dev/sda) on the oss01 node hostname oss01sp Hostname for the first OSS node’s SP Second OSS node OSS node oss02 Second OSS node in the Lustre file system OST ost02 Second OST in the Lustre file system block device /dev/sdb Block device for the second OSS node (oss02) mount point /mnt/ost02 Mount point for the ost02 block device (/dev/sdb) on the oss02 node hostname oss02sp Hostname for the second OSS node’s SP8-10 Lustre 1.8 Operations Manual • December 2010 To configure Heartbeat without STONITH: 1. Create (or edit) the Heartbeat configuration file, /etc/ha.d/ha.cf. This file must be identical on both nodes. In this example configuration (without STONITH configuration), the /etc/ha.d/ha.cf file looks like this: # log file settings # write debug output to /var/log/ha-debug debugfile /var/log/ha-debug # write log messages to /var/log/ha-log logfile /var/log/ha-log # use syslog to write to logfiles logfacility local0 # set some time-outs. these values are only recommendations, which depend e.g. on the OSS load # send keep-alive packages every 2 seconds keepalive 2 # wait 90 seconds before declaring a node dead deadtime 90 # write a warning to the logfile after 30 seconds without an answer from the failover node warntime 30 # wait for 120 seconds before declaring a node dead after heartbeat is brought up initdead 120 # define communication channels # use port 12345 to communicate with fail-over node udpport 12345 # use network interfaces eth0 and ib0 to detect a failed node bcast eth0 ib0 # Use manual failback auto_failback off # node names in this failover-pair. These names must match the output of `hostname` node oss01 node oss02Chapter 8 Failover 8-11 2. Define the resources that will be controlled by Heartbeat by editing the /etc/ha/d/haresources file. This file must be identical on both nodes. In this example configuration, the /etc/ha.d/haresources file looks like this: oss01 Filesystem::/dev/sda::/mnt/ost01::lustre oss02 Filesystem::/dev/sdb::/mnt/ost02::lustre The resource definition file tells Heartbeat that one file system resource is associated with oss01 and oss02. Each resource is defined on separate lines. The file system resource script takes three inputs separated by "::". The first parameter is the device name, the second is the mount point and the third is the file system type. Depending on the configuration, a resource can be more complex, e.g., software RAID needs to be assembled before the file system can be mounted. In this case, an haresources file may look like this: oss01 Raid1::/etc/mdadm.conf.oss::/dev/md1 Filesystem::/dev/md1::/mnt/ost01::lustre oss02 Raid1::/etc/mdadm.conf.oss::/dev/md2 Filesystem::/dev/md2::/mnt/ost02::lustre When a resource group is started by Heartbeat, the resources start from left to right. In this example, the RAID is assembled first, and the file system is mounted second. If the resource group is stopped, then the file system is unmounted first and the RAID is stopped second. Other resource scripts can be found in the /etc/ha.d/resource.d/ folder. 3. Create the /etc/ha.d/authkeys file and fix its permissions. This file must be identical on both nodes. In this example configuration, the authkeys file looks like this: auth 1 1 sha1 PutYourSuperSecretKeyHere Make sure that the permissions for this files are set to 0600, by running chmod 0600 /etc/ha.d/authkeys on both nodes. 4. Test the Heartbeat configuration. Run the following command on both nodes: service heartbeat start Check the log files on both nodes to find any problems and fix them. After the initial deadtime interval, you should see the nodes discover each other's state and start the Lustre resources associated with them.8-12 Lustre 1.8 Operations Manual • December 2010 Configuring Heartbeat with STONITH STONITH automates the process of power control and management. Expect scripts are dependent on the exact set of commands provided by each hardware vendor. As a result, any change in the power control hardware or firmware requires that STONITH be adjusted. Note – This procedure describes configuring Heartbeat using a v1 configuration file, which can be used with both Heartbeat v1 and v2. See (Optional) Migrating a Heartbeat Configuration (v1 to v2) for an optional procedure to convert the v1 configuration file to an XML-formatted v2 configuration file. Note – Depending on the particular packaging, Heartbeat files may be located in a different directory or path than indicated in the following procedure. For example, they may be located in /etc/ha.d/ or /var/lib/heartbeat. The heartbeat-stonith package comes with a number of pre-defined STONITH scripts for different power control hardware. Additionally, Heartbeat can be configured to run an external script. Heartbeat can be configured in two STONITH modes: ¦ One STONITH command for all nodes found in ha.cf: stonith ¦ One STONITH command per-node: stonith_host You can use an external script to kill each node, e.g.: stonith_host oss01 external foo /etc/ha.d/reset-nodeB stonith_host oss02 external foo /etc/ha.d/reset-nodeA To get the proper STONITH syntax, run: $ stonith -L The above command lists supported models. To list required parameters and specify the configuration filename, run: $ stonith -l -t To attempt a test, run: $ stonith -l -t To test STONITH, use a real hostname. To work with Heartbeat correctly, the external STONITH scripts should take the parameters {start|stop|status} and return 0 or 1.Chapter 8 Failover 8-13 To add STONITH functionality (using an ipmi service processor) to the configuration example, add the following lines to the /etc/ha.d/ha.cf configuration file: # define how a node can be powered off in case of a failure. more details below stonith_host oss01 external/ipmi oss02 oss02sp root changeme lanplus stonith_host oss02 external/ipmi oss01 oss01sp root changeme lanplus STONITH is only invoked if one of the failover nodes is no longer responding to Heartbeat messages and the cluster does stop resources in an orderly manner. If two cluster nodes can communicate, they usually shut down properly. This means that many tests do not produce a STONITH, for example: ¦ Calling init 0, shutdown, or reboot on a node will cause no STONITH ¦ Stopping Heartbeat on a node stops the resources cleanly and fails them over to the other node without invoking STONITH. 8.3.2.3 (Optional) Migrating a Heartbeat Configuration (v1 to v2) Heartbeat includes a script that enables v1 configuration files to be migrated to v2 XML configuration files. The script reads the v1 configuration files (ha.cf and haresources), and then writes an XML file to STDOUT. The script is $ /usr/lib/heartbeat/haresources2cib.py or $ /usr/lib64/heartbeat/haresources2cib.py To redirect the script output after the cib.xml file has been generated, it is recommended that you check the XML file and change some parameters, such as resource-stickiness and timeouts, to more appropriate values. For example: $ /usr/lib64/heartbeat/haresources2cib.py > cib.xml Then the cib.xml file should than be copied to /var/lib/heartbeat/crm/cib.xml on both failover nodes. To test the new configuration, start Heartbeat on both nodes and check the log files. Note – If a Heartbeat v2 configuration file is available on the system, it is not necessary to remove the v1 configuration files, as they are ignored.8-14 Lustre 1.8 Operations Manual • December 2010 8.3.3 Working with Heartbeat After Lustre and Heartbeat are correctly configured, the following commands can be used to control Heartbeat. 8.3.3.1 Starting Heartbeat To start Heartbeat, run this command on both failover nodes: service heartbeat start After a node fails, start Heartbeat manually and analyze the cause of the problem before taking over the failed resources. You should NOT start Heartbeat automatically after a node failure. 8.3.3.2 Switching Resources Between Nodes Depending on whether Heartbeat v1 or v2 configuration files are being used, there are different ways to switch resources between nodes. For Heartbeat v1 configuration files, two scripts are provided (hb_takeover and hb_standby), that make it easy to switch resources between failover nodes. Depending on your system, these scripts are located in /usr/lib/heartbeat/ or /usr/lib64/heartbeat/. The hb_takeover and hb_standby scripts take the following arguments: ¦ all -- take/fail over all resources ¦ foreign -- take/fail over foreign resources ¦ local -- take/fail over local resources only ¦ failback -- fail/take over foreign resources Performing an hb_takeover on the current node is equivalent to performing an hb_standby on the other node. For Heartbeat v2 configuration files, the crm_resource command is used to interact with Heartbeat's Cluster Resource Manager and switch resources between nodes. For more information on crm_resource, see: linux.die.net/man/8/crm_resourceChapter 8 Failover 8-15 To switch resources between nodes: 1. Generate a complete list of resources known to the Heartbeat cluster resource manager. Run: crm_resource --list 2. From the list, identify the group name for the resource to fail over. 3. Determine if and where the specified resource is running. Run: crm_resource -W -r 4. Migrate the resource to the host. Run: crm_resource -M -r -H 5. To un-migrate a resource, run: crm_resource -U -r 8-16 Lustre 1.8 Operations Manual • December 20109-1 C H A P T E R 9 Configuring Quotas This chapter describes how to configure quotas and includes the following sections: ¦ Working with Quotas ¦ Enabling Disk Quotas ¦ Creating Quota Files and Quota Administration ¦ Quota Allocation ¦ Known Issues with Quotas ¦ Lustre Quota Statistics 9.1 Working with Quotas Quotas allow a system administrator to limit the amount of disk space a user or group can use in a directory. Quotas are set by root, and can be specified for individual users and/or groups. Before a file is written to a partition where quotas are set, the quota of the creator's group is checked. If a quota exists, then the file size counts towards the group's quota. If no quota exists, then the owner's user quota is checked before the file is written. Similarly, inode usage for specific functions can be controlled if a user over-uses the allocated space. Lustre quota enforcement differs from standard Linux quota support in several ways: ¦ Quotas are administered via the lfs command (post-mount). ¦ Quotas are distributed (as Lustre is a distributed file system), which has several ramifications. ¦ Quotas are allocated and consumed in a quantized fashion. ¦ Client does not set the usrquota or grpquota options to mount. When quota is enabled, it is enabled for all clients of the file system; started automatically using quota_type or started manually with lfs quotaon.9-2 Lustre 1.8 Operations Manual • December 2010 Caution – Although quotas are available in Lustre, root quotas are NOT enforced. lfs setquota -u root (limits are not enforced) lfs quota -u root (usage includes internal Lustre data that is dynamic in size and does not accurately reflect mount point visible block and inode usage). 9.1.1 Enabling Disk Quotas Use this procedure to enable (configure) disk quotas in Lustre. 1. If you have re-complied your Linux kernel, be sure that CONFIG_QUOTA and CONFIG_QUOTACTL are enabled. Also, verify that CONFIG_QFMT_V1 and/or CONFIG_QFMT_V2 are enabled. Quota is enabled in all Linux 2.6 kernels supplied for Lustre. 2. Start the server. 3. Mount the Lustre file system on the client and verify that the lquota module has loaded properly by using the lsmod command. $ lsmod [root@oss161 ~]# lsmod Module Size Used by obdfilter 220532 1 fsfilt_ldiskfs 52228 1 ost 96712 1 mgc 60384 1 ldiskfs 186896 2 fsfilt_ldiskfs lustre 401744 0 lov 289064 1 lustre lquota 107048 4 obdfilter mdc 95016 1 lustre ksocklnd 111812 1 The Lustre mount command no longer recognizes the usrquota and grpquota options. If they were previously specified, remove them from /etc/fstab. When quota is enabled, it is enabled for all file system clients (started automatically using quota_type or manually with lfs quotaon). Note – Lustre with the Linux kernel 2.4 does not support quotas.Chapter 9 Configuring Quotas 9-3 To enable quotas automatically when the file system is started, you must set the mdt.quota_type and ost.quota_type parameters, respectively, on the MDT and OSTs. The parameters can be set to the string u (user), g (group) or ug for both users and groups. You can enable quotas at mkfs time (mkfs.lustre --param mdt.quota_type= ug) or with tunefs.lustre. As an example: tunefs.lustre --param ost.quota_type=ug $ost_dev Caution – If you are using mkfs.lustre --param mdt.quota_type=ug or tunefs.lustre --param ost.quota_type=ug, be sure to run the command on all OSTs and the MDT. Otherwise, abnormal results may occur. 9.1.1.1 Administrative and Operational Quotas Lustre has two kinds of quota files: ¦ Administrative quotas (for the MDT), which contain limits for users/groups for the entire cluster. ¦ Operational quotas (for the MDT and OSTs), which contain quota information dedicated to a cluster node. Lustre 1.6.5 introduced the v2 file format for administrative quota files, with continued support for the old file format (v1). The mdt.quota_type parameter also handles ‘1’ and ‘2’ options, to specify the Lustre quota versions that will be used. For example: --param mdt.quota_type=ug1 --param mdt.quota_type=u2 Lustre 1.6.6 introduced the v2 file format for operational quotas, with continued support for the old file format (v1). The ost.quota_type parameter handles ‘1’ and ‘2’ options, to specify the Lustre quota versions that will be used. For example: --param ost.quota_type=ug2 --param ost.quota_type=u1 For more information about the v1 and v2 formats, see Quota File Formats.9-4 Lustre 1.8 Operations Manual • December 2010 9.1.2 Creating Quota Files and Quota Administration Once each quota-enabled file system is remounted, it is capable of working with disk quotas. However, the file system is not yet ready to support quotas. If umount has been done regularly, run the lfs command with the quotaon option. If umount has not been done, perform these steps: 1. Take Lustre ''offline''. That is, verify that no write operations (append, write, truncate, create or delete) are being performed (preparing to run lfs quotacheck). Operations that do not change Lustre files (such as read or mount) are okay to run. Caution – When lfs quotacheck is run, Lustre must NOT be performing any write operations. Failure to follow this caution may cause the statistic information of quota to be inaccurate. For example, the number of blocks used by OSTs for users or groups will be inaccurate, which can cause unexpected quota problems. 2. Run the lfs command with the quotacheck option: # lfs quotacheck -ug /mnt/lustre By default, quota is turned on after quotacheck completes. Available options are: ¦ u — checks the user disk quota information ¦ g — checks the group disk quota information The lfs quotacheck command checks all objects on all OSTs and the MDS to sum up for every UID/GID. It reads all Lustre metadata and re-computes the number of blocks/inodes that each UID/GID has used. If there are many files in Lustre, it may take a long time to complete. Note – User and group quotas are separate. If either quota limit is reached, a process with the corresponding UID/GID cannot allocate more space on the file system. Note – When lfs quotacheck runs, it creates a quota file -- a sparse file with a size proportional to the highest UID in use and UID/GID distribution. As a general rule, if the highest UID in use is large, then the sparse file will be large, which may affect functions such as creating a snapshot.Chapter 9 Configuring Quotas 9-5 Note – For Lustre 1.6 releases before version 1.6.5, and 1.4 releases before version 1.4.12, if the underlying ldiskfs file system has not unmounted gracefully (due to a crash, for example), re-run quotacheck to obtain accurate quota information. Lustre 1.6.5 and 1.4.12 use journaled quota, so it is not necessary to run quotacheck after an unclean shutdown. In certain failure situations (e.g., when a broken Lustre installation or build is used), re-run quotacheck after checking the server kernel logs and fixing the root problem. The lfs command includes several command options to work with quotas: ¦ quotaon — enables disk quotas on the specified file system. The file system quota files must be present in the root directory of the file system. ¦ quotaoff — disables disk quotas on the specified file system. ¦ quota — displays general quota information (disk usage and limits) ¦ setquota — specifies quota limits and tunes the grace period. By default, the grace period is one week. Usage: lfs quotaon [-ugf] lfs quotaoff [-ug] lfs quota [-q] [-v] [-o obd_uuid] [-u|-g |uid|gname|gid>] lfs quota -t <-u|-g> lfs setquota <-u|--user|-g|--group> [-b ] [-B ] [-i ] [-I ] Examples: In all of the examples below, the file system is /mnt lustre. To turn on user and group quotas, run: $ lfs quotaon -ug /mnt/lustre To turn off user and group quotas, run: $ lfs quotaoff -ug /mnt/lustre To display general quota information (disk usage and limits) for the user running the command and his primary group, run: $ lfs quota /mnt/lustre9-6 Lustre 1.8 Operations Manual • December 2010 To display general quota information for a specific user ("bob" in this example), run: $ lfs quota -u bob /mnt/lustre To display general quota information for a specific user ("bob" in this example) and detailed quota statistics for each MDT and OST, run: $ lfs quota -u bob -v /mnt/lustre To display general quota information for a specific group ("eng" in this example), run: $ lfs quota -g eng /mnt/lustre To display block and inode grace times for user quotas, run: $ lfs quota -t -u /mnt/lustre To set user and group quotas for a specific user ("bob" in this example), run: $ lfs setquota -u bob 307200 309200 10000 11000 /mnt/lustre In this example, the quota for user "bob" is set to 300 MB (309200*1024) and the hard limit is 11,000 files. Therefore, the inode hard limit should be 11000. Note – For the Lustre command $ lfs setquota/quota ... the qunit for block is KB (1024) and the qunit for inode is 1. The quota command displays the quota allocated and consumed for each Lustre device. Using the previous setquota example, running this lfs quota command: $ lfs quota -u bob -v /mnt/lustre displays this command output: Disk quotas for user bob (uid 6000): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 0 30720 30920 - 0 10000 11000- lustre-MDT0000_UUID0 - 16384 - 0 - 2560- lustre-OST0000_UUID0 - 16384 - 0 - 0- lustre-OST0001_UUID0 - 16384 - 0 - 0-Chapter 9 Configuring Quotas 9-7 9.1.3 Quota Allocation In Lustre, quota must be properly allocated or users may experience unnecessary failures. The file system block quota is divided up among the OSTs within the file system. Each OST requests an allocation which is increased up to the quota limit. The quota allocation is then quantized to reduce the number of quota-related request traffic. By default, Lustre supports both user and group quotas to limit disk usage and file counts. The quota system in Lustre is completely compatible with the quota systems used on other file systems. The Lustre quota system distributes quotas from the quota master. Generally, the MDS is the quota master for both inodes and blocks. All OSTs and the MDS are quota slaves to the OSS nodes. To reduce quota requests and get reasonably accurate quota distribution, the transfer quota unit (qunit) between quota master and quota slaves is changed dynamically by the lquota module. The default minimum value of qunit is 1 MB for blocks and 2 for inodes. The proc entries to set these values are: /proc/fs/lustre/mds/lustre-MDT*/quota_least_bunit and /proc/fs/lustre/mds/lustre-MDT*/quota_least_iunit. The default maximum value of qunit is 128 MB for blocks and 5120 for inodes. The proc entries to set these values are quota_bunit_sz and quota_iunit_sz in the MDT and OSTs. Note – In general, the quota_bunit_sz value should be larger than 1 MB. For testing purposes, it can be set to 4 KB, if necessary. The file system block quota is divided up among the OSTs and the MDS within the file system. Only the MDS uses the file system inode quota. This means that the minimum quota for block is 1 MB* (the number of OSTs + the number of MDSs), which is 1 MB* (number of OSTs + 1). If you attempt to assign a smaller quota, users maybe not be able to create files. As noted, the default minimum quota for inodes is 2. The default is established at file system creation time, but can be tuned via /proc values (described below). The inode quota is also allocated in a quantized manner on the MDS. If we look at the setquota example again, running this lfs quota command: # lfs quota -u bob -v /mnt/lustre displays this command output:9-8 Lustre 1.8 Operations Manual • December 2010 Disk quotas for user bob (uid 500): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 30720* 30720 30920 6d23h56m44s10101*10000 11000 6d23h59m50s lustre-MDT0000_UUID0 - 1024 - 10101 - 10240 lustre-OST0000_UUID0 - 1024 - - - - lustre-OST0001_UUID30720*- 28872 - - - - The total quota limit of 30,920 is allotted to user bob, which is further distributed to two OSTs and one MDS. Note – Values appended with “*” show the limit that has been over-used (exceeding the quota), and receives this message Disk quota exceeded. For example: \ $ cp: writing `/mnt/lustre/var/cache/fontconfig/ beeeeb3dfe132a8a0633a017c99ce0-x86.cache’: Disk quota exceeded. The requested quota of 300 MB is divided across the OSTs. Note – It is very important to note that the block quota is consumed per OST and the MDS per block and inode (there is only one MDS for inodes). Therefore, when the quota is consumed on one OST, the client may not be able to create files regardless of the quota available on other OSTs. Additional information: Grace period — The period of time (in seconds) within which users are allowed to exceed their soft limit. There are four types of grace periods: ¦ user block soft limit ¦ user inode soft limit ¦ group block soft limit ¦ group inode soft limit The grace periods are applied to all users. The user block soft limit is for all users who are using a blocks quota.Chapter 9 Configuring Quotas 9-9 Soft limit — Once you are beyond the soft limit, the quota module begins to time, but you still can write block and inode. When you are always beyond the soft limit and use up your grace time, you get the same result as the hard limit. For inodes and blocks, it is the same. Usually, the soft limit MUST be less than the hard limit; if not, the quota module never triggers the timing. If the soft limit is not needed, leave it as zero (0). Hard limit — When you are beyond the hard limit, you get -EQUOTA and cannot write inode/block any more. The hard limit is the absolute limit. When a grace period is set, you can exceed the soft limit within the grace period if are under the hard limits. Lustre quota allocation is controlled by two variables, quota_bunit_sz and quota_iunit_sz referring to KBs and inodes, respectively. These values can be accessed on the MDS as /proc/fs/lustre/mds/*/quota_* and on the OST as /proc/fs/lustre/obdfilter/*/quota_*. The quota_bunit_sz and quota_iunit_sz variables are the maximum qunit values for blocks and inodes, respectively. At any time, module lquota chooses a reasonable qunit between the minimum and maximum values. The /proc values are bounded by two other variables quota_btune_sz and quota_itune_sz. By default, the *tune_sz variables are set at 1/2 the *unit_sz variables, and you cannot set *tune_sz larger than *unit_sz. You must set bunit_sz first if it is increasing by more than 2x, and btune_sz first if it is decreasing by more than 2x. Total number of inodes — To determine the total number of inodes, use lfs df -i (and also /proc/fs/lustre/*/*/filestotal). For more information on using the lfs df -i command and the command output, see Querying File System Space. Unfortunately, the statfs interface does not report the free inode count directly, but instead reports the total inode and used inode counts. The free inode count is calculated for df from (total inodes - used inodes). It is not critical to know a file system’s total inode count. Instead, you should know (accurately), the free inode count and the used inode count for a file system. Lustre manipulates the total inode count in order to accurately report the other two values. The values set for the MDS must match the values set on the OSTs. The quota_bunit_sz parameter displays bytes, however lfs setquota uses KBs. The quota_bunit_sz parameter must be a multiple of 1024. A proper minimum KB size for lfs setquota can be calculated as: Size in KBs = minimum_quota_bunit_sz * (number of OSTS + 1) = 1024 * (number of OSTs +1)9-10 Lustre 1.8 Operations Manual • December 2010 We add one (1) to the number of OSTs as the MDS also consumes KBs. As inodes are only consumed on the MDS, the minimum inode size for lfs setquota is equal to quota_iunit_sz. Note – Setting the quota below this limit may prevent the user from all file creation. 9.1.4 Known Issues with Quotas Using quotas in Lustre can be complex and there are several known issues. 9.1.4.1 Granted Cache and Quota Limits In Lustre, granted cache does not respect quota limits. In this situation, OSTs grant cache to Lustre client to accelerate I/O. Granting cache causes writes to be successful in OSTs, even if they exceed the quota limits, and will overwrite them. The sequence is: 1. A user writes files to Lustre. 2. If the Lustre client has enough granted cache, then it returns ‘success’ to users and arranges the writes to the OSTs. 3. Because Lustre clients have delivered success to users, the OSTs cannot fail these writes. Because of granted cache, writes always overwrite quota limitations. For example, if you set a 400 GB quota on user A and use IOR to write for user A from a bundle of clients, you will write much more data than 400 GB, and cause an out-of-quota error (-EDQUOT). Note – The effect of granted cache on quota limits can be mitigated, but not eradicated. Reduce the max_dirty_buffer in the clients (can be set from 0 to 512). To set max_dirty_buffer to 0: * In releases after Lustre 1.6.5, set: lctl set_param osc.*.max_dirty_mb=0 * In releases before Lustre 1.6.5, set: proc/fs/lustre/osc/*/max_dirty_mb; do echo 512 > $0Chapter 9 Configuring Quotas 9-11 9.1.4.2 Quota Limits Available quota limits depend on the Lustre version you are using. ¦ Lustre version 1.4.11 and earlier (for 1.4.x releases) and Lustre version 1.6.4 and earlier (for 1.6.x releases) support quota limits less than 4 TB. ¦ Lustre versions 1.4.12, 1.6.5 and later support quota limits of 4 TB and greater in Lustre configurations with OST storage limits of 4 TB and less. ¦ Future Lustre versions are expected to support quota limits of 4 TB and greater with no OST storage limits. 9.1.4.3 Quota File Formats Lustre 1.6.5 introduced the v2 file format for administrative quotas, with 64-bit limits that support large-limits handling. The old quota file format (v1), with 32-bit limits, is also supported. Lustre 1.6.6 introduced the v2 file format for operational quotas. A few notes regarding the current quota file formats: Lustre 1.6.5 and later use mdt.quota_type to force a specific administrative quota version (v2 or v1). ¦ For the v2 quota file format, (OBJECTS/admin_quotafile_v2.{usr,grp}) ¦ For the v1 quota file format, (OBJECTS/admin_quotafile.{usr,grp}) Lustre 1.6.6 and later use ost.quota_type to force a specific operational quota version (v2 or v1). ¦ For the v2 quota file format, (lquota_v2.{user,group}) ¦ For the v1 quota file format, (lquota.{user,group}) Lustre Version Quota Limit Per User/Per Group OST Storage Limit 1.4.11 and earlier < 4TB n/a 1.4.12 => 4TB <= 4TB of storage 1.6.4 and earlier < 4TB n/a 1.6.5 => 4TB <= 4TB of storage Future Lustre versions => 4TB No storage limit9-12 Lustre 1.8 Operations Manual • December 2010 The quota_type specifier can be used to set different combinations of administrative/operational quota file versions on a Lustre node: ¦ "1" - v1 (32-bit) administrative quota file, v1 (32-bit) operational quota file (default in releases before Lustre 1.6.5) ¦ "2" - v2 (64-bit) administrative quota file, v1 (32-bit) operational quota file (default in Lustre 1.6.5) ¦ "3" - v2 (64-bit) administrative quota file, v2 (64-bit) operational quota file (default in releases after Lustre 1.6.5) If quotas do not exist or look broken, then quotacheck creates quota files of a required name and format. If Lustre is using the v2 quota file format when only v1 quota files exist, then quotacheck converts old v1 quota files to new v2 quota files. This conversion is triggered automatically, and is transparent to users. If an old quota file does not exist or looks broken, then the new v2 quota file will be empty. In case of an error, details can be found in the kernel log of the corresponding MDS/OST. During conversion of a v1 quota file to a v2 quota file, the v2 quota file is marked as broken, to avoid it being used if a crash occurs. The quota module does not use broken quota files (keeping quota off). In most situations, Lustre administrators do not need to set specific versioning options. Upgrading Lustre without using quota_type to force specific quota file versions results in quota files being upgraded automatically to the latest version. The option ensures backward compatibility, preventing a quota file upgrade to a version which is not supported by earlier Lustre versions. 9.1.5 Lustre Quota Statistics Lustre includes statistics that monitor quota activity, such as the kinds of quota RPCs sent during a specific period, the average time to complete the RPCs, etc. These statistics are useful to measure performance of a Lustre file system. Each quota statistic consists of a quota event and min_time, max_time and sum_time values for the event. Quota Event Description sync_acq_req Quota slaves send a acquiring_quota request and wait for its return. sync_rel_req Quota slaves send a releasing_quota request and wait for its return. async_acq_req Quota slaves send an acquiring_quota request and do not wait for its return.Chapter 9 Configuring Quotas 9-13 async_rel_req Quota slaves send a releasing_quota request and do not wait for its return. wait_for_blk_quota (lquota_chkquota) Before data is written to OSTs, the OSTs check if the remaining block quota is sufficient. This is done in the lquota_chkquota function. wait_for_ino_quota (lquota_chkquota) Before files are created on the MDS, the MDS checks if the remaining inode quota is sufficient. This is done in the lquota_chkquota function. wait_for_blk_quota (lquota_pending_commit) After blocks are written to OSTs, relative quota information is updated. This is done in the lquota_pending_commit function. wait_for_ino_quota (lquota_pending_commit) After files are created, relative quota information is updated. This is done in the lquota_pending_commit function. wait_for_pending_blk_quota_req (qctxt_wait_pending_dqacq) On the MDS or OSTs, there is one thread sending a quota request for a specific UID/GID for block quota at any time. At that time, if other threads need to do this too, they should wait. This is done in the qctxt_wait_pending_dqacq function. wait_for_pending_ino_quota_req (qctxt_wait_pending_dqacq) On the MDS, there is one thread sending a quota request for a specific UID/GID for inode quota at any time. If other threads need to do this too, they should wait. This is done in the qctxt_wait_pending_dqacq function. nowait_for_pending_blk_quota_req (qctxt_wait_pending_dqacq) On the MDS or OSTs, there is one thread sending a quota request for a specific UID/GID for block quota at any time. When threads enter qctxt_wait_pending_dqacq, they do not need to wait. This is done in the qctxt_wait_pending_dqacq function. nowait_for_pending_ino_quota_req (qctxt_wait_pending_dqacq) On the MDS, there is one thread sending a quota request for a specific UID/GID for inode quota at any time. When threads enter qctxt_wait_pending_dqacq, they do not need to wait. This is done in the qctxt_wait_pending_dqacq function. quota_ctl The quota_ctl statistic is generated when lfs setquota, lfs quota and so on, are issued. adjust_qunit Each time qunit is adjusted, it is counted. Quota Event Description9-14 Lustre 1.8 Operations Manual • December 2010 9.1.5.1 Interpreting Quota Statistics Quota statistics are an important measure of a Lustre file system’s performance. Interpreting these statistics correctly can help you diagnose problems with quotas, and may indicate adjustments to improve system performance. For example, if you run this command on the OSTs: cat /proc/fs/lustre/lquota/lustre-OST0000/stats You will get a result similar to this: snapshot_time 1219908615.506895 secs.usecs async_acq_req 1 samples [us]32 32 32 async_rel_req 1 samples [us]5 5 5 nowait_for_pending_blk_quota_req(qctxt_wait_pending_dqacq) 1 samples [us] 2 2 2 quota_ctl 4 samples [us]80 3470 4293 adjust_qunit 1 samples [us]70 70 70 .... In the first line, snapshot_time indicates when the statistics were taken. The remaining lines list the quota events and their associated data. In the second line, the async_acq_req event occurs one time. The min_time, max_time and sum_time statistics for this event are 32, 32 and 32, respectively. The unit is microseconds ( s). In the fifth line, the quota_ctl event occurs four times. The min_time, max_time and sum_time statistics for this event are 80, 3470 and 4293, respectively. The unit is microseconds ( s). Involving Lustre Support in Quotas Analysis Quota statistics are collected in /proc/fs/lustre/lquota/.../stats. Each MDT and OST has one statistics proc file. If you have a problem with quotas, but cannot successfully diagnose the issue, send the statistics files in the folder to Lustre Support for analysis. To prepare the files: 1. Initialize the statistics data to 0 (zero). Run: lctl set_param lquota.${FSNAME}-MDT*.stats=0 lctl set_param lquota.${FSNAME}-OST*.stats=0 2. Perform the quota operation that causes the problem or degraded performance. 3. Collect all statistics in /proc/fs/lustre/lquota/ and send them to Lustre Support. Note the following: ¦ Proc quota entries are collected in these folders: /proc/fs/lustre/obdfilter/lustre-OSTXXXX/quota*Chapter 9 Configuring Quotas 9-15 - AND - /proc/fs/lustre/mds/lustre-MDTXXXX/quota* Proc quota entries are copied to /proc/fs/lustre/lquota. ¦ To maintain compatibility, old quota proc entries in the following folders are not deleted in the current Lustre release (although they may be deprecated in the future): /proc/fs/lustre/obdfilter/lustre-OSTXXXX/ - AND - /proc/fs/lustre/mds/lustre-MDTXXXX/ ¦ Only use the quota entries in /proc/fs/lustre/lquota/.9-16 Lustre 1.8 Operations Manual • December 201010-1 C H A P T E R 10 RAID This chapter describes software and hardware RAID, and includes the following sections: ¦ Considerations for Backend Storage ¦ Insights into Disk Performance Measurement ¦ Lustre Software RAID Support10-2 Lustre 1.8 Operations Manual • December 2010 10.1 Considerations for Backend Storage Lustre's architecture allows it to use any kind of block device as backend storage. The characteristics of such devices, particularly in the case of failures vary significantly and have an impact on configuration choices. This section surveys issues and recommendations regarding backend storage. 10.1.1 Selecting Storage for the MDS or OSTs MDS The MDS does a large amount of small writes. For this reason, we recommend that you use RAID1 for MDT storage. If you require more capacity for an MDT than one disk provides, we recommend RAID1 + 0 or RAID10. LVM is not recommended at this time for performance reasons. OST A quick calculation (shown below), makes it clear that without further redundancy, RAID5 is not acceptable for large clusters and RAID6 is a must. Take a 1 PB file system (2,000 disks of 500 GB capacity). The MTTF 1 of a disk is about 1,000 days. This means that the expected failure rate is 2000/1000 = 2 disks per day. Repair time at 10% of disk bandwidth is close to 1 day (500 GB at 5 MB/sec = 100,000 sec = 1 day). If we have a RAID 5 stripe that is 10 disks wide, then during 1 day of rebuilding, the chance that a second disk in the same array fails is about 9 / 1000 ~= 1/100. This means that, in the expected period of 50 days, a double failure in a RAID 5 stripe leads to data loss. So, RAID 6 or another double parity algorithm is necessary for OST storage. For better performance, we recommend that you create RAID sets with no more than 8 data disks (+1 or +2 parity disks) as this will provide more IOPS from having multiple independent RAID sets. 1. Mean Time to FailureChapter 10 RAID 10-3 File system: Use RAID5 with 5 or 9 disks or RAID6 with 6 or 10 disks, each on a different controller. The stripe width is the optimal minimum I/O size. Ideally, the RAID configuration should allow 1 MB Lustre RPCs to fit evenly on a single RAID stripe without an expensive read-modify-write cycle. Use this formula to determine the stripe_width. = * ( - ) <= 1 MB where is 1 for RAID5/RAID-Z and 2 for RAID6/RAID-Z2. If the RAID configuration does not allow to fit evenly into 1 MB, select , such that is close to 1 MB, but not larger. For example, RAID6 with 6 disks has 4 data and 2 parity disks, so we get: <= 1024kB/4; either 256kB, 128kB or 64kB The value must equal * ( - ). Use it for OST file systems only (not MDT file systems). $ mkfs.lustre --mountfsoptions="stripe=" ... External journal: Use RAID1 with two partitions of 400 MB (or more), each from disks on different controllers. To set up the journal device (/dev/mdJ), run: $ 'mke2fs -O journal_dev -b 4096 /dev/mdJ' Then run --reformat on the file system device (/dev/mdX), specifying the RAID geometry to the underlying ldiskfs file system, where: = / 4096 = / 4096: $ mkfs.lustre --reformat ... --mkfsoptions "-j -J device=/dev/mdJ -E stride=" /dev/mdX 10.1.2 Reliability Best Practices It is considered mandatory that you use disk monitoring software, so rebuilds happen without any delay. We recommend backups of the metadata file systems. This can be done with LVM snapshots or using raw partition backups.10-4 Lustre 1.8 Operations Manual • December 2010 10.1.3 Performance Tradeoffs Writeback cache can dramatically increase write performance on any type of RAID array. 2 Unfortunately, unless the RAID array has battery-backed cache (a feature only found in some higher-priced hardware RAID arrays), interrupting the power to the array may result in out-of-sequence writes. This causes problems for journaling. If writeback cache is enabled, a file system check is required after the array loses power. Data may also be lost because of this. Therefore, we recommend against the use of writeback cache when data integrity is critical. You should carefully consider whether the benefits of using writeback cache outweigh the risks. 10.1.4 Formatting Options for RAID Devices When formatting a file system on a RAID device, it is beneficial to specify additional parameters at the time of formatting. This ensures that the file system is optimized for the underlying disk geometry. Use the --mkfsoptions parameter to specify these options when formatting the OST or MDT. For RAID 5, RAID 6, RAID 1+0 storage, specifying the -E stride = option improves the layout of the file system metadata ensuring that no single disk contains all of the allocation bitmaps. The parameter is in units of 4096-byte blocks and represents the amount of contiguous data written to a single disk before moving to the next disk. This is applicable to both MDS and OST file systems. For more information on how to override the defaults while formatting MDS or OST file systems, see Options for Formatting the MDT and OSTs. 2. Client writeback cache improves performance for many small files or for a single, large file alike. However, if the cache is filled with small files, cache flushing is likely to be much slower (because of less data being sent per RPC), so there may be a drop-off in total throughput.Chapter 10 RAID 10-5 10.1.4.1 Creating an External Journal If you have configured a RAID array and use it directly as an OST, it houses both data and metadata. For better performance 3 , we recommend putting OST metadata on another journal device, by creating a small RAID 1 array and using it as an external journal for the OST. It is not known if external journals improve performance of MDTs. Currently, we recommend against using them for MDTs to reduce complexity. No more than 102,400 file system blocks will ever be used for a journal. For Lustre's standard 4 KB block size, this corresponds to a 400 MB journal. A larger partition can be created, but only the first 400 MB will be used. Additionally, a copy of the journal is kept in RAM on the OSS. Therefore, make sure you have enough memory available to hold copies of all the journals. To create an external journal, perform these steps for each OST on the OSS: 1. Create a 400 MB (or larger) journal partition (RAID 1 is recommended). In this example, /dev/sdb is a RAID 1 device, run: $ sfdisk -uC /dev/sdb << EOF > ,50,L > EOF 2. Create a journal device on the partition. Run: $ mke2fs -b 4096 -O journal_dev /dev/sdb1 3. Create the OST. In this example, /dev/sdc is the RAID 6 device to be used as the OST, run: $ mkfs.lustre --ost --mgsnode=mds@osib \ --mkfsoptions="-J device=/dev/sdb1" /dev/sdc 4. Mount the OST as usual. 3. Performance is affected because, while writing large sequential data, small I/O writes are done to update metadata. This small-sized I/O can affect performance of large sequential I/O with disk seeks.10-6 Lustre 1.8 Operations Manual • December 2010 10.1.5 Handling Degraded RAID Arrays Lustre 1.8.2 and later versions include functionality that notifies Lustre if an external RAID array has degraded performance (resulting in a degraded OST), either because a disk has failed and not been replaced, or because a disk was replaced and is undergoing a rebuild. To avoid a global performance slowdown due to a degraded OST, the MDS can avoid the OST for new object allocation if it is notified of the degraded state. The new file (called "degraded"), in /proc/fs/lustre/obdfilter/{OST}, marks the OST as degraded if it is written with a "1" (or any non-zero value), until a "0" is written to it. Therefore, "1" should be written to the file when the array becomes degraded and "0" should be written when the array becomes healthy. If the OST is remounted due to a reboot or other condition, the flag resets to "0". 10.2 Insights into Disk Performance Measurement Several tips and insights for disk performance measurement are provided below. Some of this information is specific to RAID arrays and/or the Linux RAID implementation. ¦ Performance is limited by the slowest disk. Before creating a software RAID array, benchmark all disks individually. We have frequently encountered situations where drive performance was not consistent for all devices in the array. Replace any disks that are significantly slower than the rest. ¦ Disks and arrays are very sensitive to request size. To identify the optimal request size for a given disk, benchmark the disk with different record sizes ranging from 4 KB to 1 to 2 MB. Note – Try to avoid sync writes; probably subsequent write would make the stripe full and no reads will be needed. Try to configure RAID arrays and the application so that most of the writes are full-stripe and stripe-aligned.Chapter 10 RAID 10-7 ¦ (Suggested) MDT setup for maximum performance. RAID1 with an internal journal and two disks from different controllers. If you need a larger MDT, create multiple RAID1 devices from pairs of disks, and then make a RAID0 array of the RAID1 devices. This ensures maximum reliability because multiple disk failures only have a small chance of hitting both disks in the same RAID1 device. Doing the opposite (RAID1 of a pair of RAID0 devices) has a 50% chance that even two disk failures can cause the loss of the whole MDT device. The first failure disables an entire half of the mirror and the second failure has a 50% chance of disabling the remaining mirror. 10.3 Lustre Software RAID Support A number of Linux kernels offer software RAID support, by which the kernel organizes disks into a RAID array. All Lustre-supported kernels have software RAID capability, but Lustre has added performance improvements to the RHEL 4 and RHEL 5 kernels that make operations even faster 4 . Therefore, if you are using software RAID functionality, we recommend that you use a Lustre-patched RHEL 4 or 5 kernel to take advantage of these performance improvements, rather than a SLES kernel. 10.3.0.1 Enabling Software RAID on Lustre This procedure describes how to set up software RAID on a Lustre system. It requires use of mdadm, a third-party tool to manage devices using software RAID. 1. Install Lustre, but do not configure it yet. See Installing Lustre. 2. Create the RAID array with the mdadm command. The mdadm command is used to create and manage software RAID arrays in Linux, as well as to monitor the arrays while they are running. To create a RAID array, use the --create option and specify the MD device to create, the array components, and the options appropriate to the array. Note – For best performance, we generally recommend using disks from as many controllers as possible in one RAID array. 4. These enhancements have mostly improved write performance.10-8 Lustre 1.8 Operations Manual • December 2010 To illustrate how to create a software RAID array for Lustre, the steps below include a worked example that creates a 10-disk RAID 6 array from disks /dev/dsk/c0t0d0 through c0tod4 and /dev/dsk/c1t0d0 through c1tod4. This RAID array has no spares. For the 10-disk RAID 6 array, there are 8 active disks. The chunk size must be chosen such that <= 1024KB/8. Therefore, the largest valid chunk size is 128KB. a. Create a RAID array for an OST. On the OSS, run: $ mdadm --create -c -l \ -n -x where: For the worked example, the command is: $ mdadm --create /dev/md10 -c 128 -l 6 -n 10 -x 0 \ /dev/dsk/c0t0d[01234] /dev/dsk/c1t0d[01234] This command output displays: mdadm: array /dev/md10 started. We also want an external journal on a RAID 1 device. We create this from two 400MB partitions on separate disks: /dev/dsk/c9t0d20p1 and /dev/dsk/c1t0d20p1 RAID array to create, in the form of /dev/mdX Size of each stripe piece on the array’s disks (in KB); discussed above. Architecture of the RAID array. RAID 5 and RAID 6 are commonly used for OSTs. Number of active disks in the array, including parity disks. Number of spare disks initially assigned to the array. More disks may be brought in via spare pooling (see below). List of the block devices used for the RAID array; wildcards may be used.Chapter 10 RAID 10-9 b. Create a RAID array for an external journal. On the OSS, run: $ mdadm --create -l -n \ -x where: For the worked example, the command is: $ mdadm --create /dev/md20 -l 1 -n 2 -x 0 /dev/dsk/c0t0d20p1 \ /dev/dsk/c1t0d20p1 This command output displays: mdadm: array /dev/md20 started. We now have two arrays - a RAID 6 array for the OST (/dev/md20), and a RAID 1 array for the external journal (/dev/md20). The arrays will now be re-synced, a process which re-synchronizes the various disks in the array so their contents match. The arrays may be used during the re-sync process (including formatting the OSTs), but performance will not be as high as usual. The re-sync progress may be monitored by reading the /proc/mdstat file. Next, you need to create a RAID array for an MDT. In this example, a RAID 10 array is created with 4 disks: /dev/dsk/c0t0d1, c0t0d3, c1t0d1, and c1t0d3. For smaller arrays, RAID 1 could be used. RAID array to create, in the form of /dev/mdX Architecture of the RAID array. RAID 1 is recommended for external journals. Number of active disks in the RAID array, including mirrors. Number of spare disks initially assigned to the RAID array. More disks may be brought in via spare pooling (see below). List of the block devices used for the RAID array; wildcards may be used.10-10 Lustre 1.8 Operations Manual • December 2010 c. Create a RAID array for an MDT. On the MDT, run: $ mdadm --create -l -n \ -x where: For the worked example, the command is: $ mdadm --create -l 10 -n 4 -x 0 /dev/md10 /dev/dsk/c[01]t0d[13] This command output displays: mdadm: array /dev/md10 started. If you creating many arrays across many servers, we recommend scripting this process. Note – Do not use the --assume-clean option when creating arrays. This could lead to data corruption on RAID 5 and will cause array checks to show errors with all RAID types. RAID array to create, in the form of /dev/mdX Architecture of the RAID array. RAID 1 or RAID 10 is recommended for MDTs. Number of active disks in the RAID array, including mirrors. Number of spare disks initially assigned to the RAID array. More disks may be brought in via spare pooling (see below). List of the block devices used for the RAID array; wildcards may be used.Chapter 10 RAID 10-11 3. Set up the mdadm tool. The mdadm tool enables you to monitor disks for failures (you will receive a notification). It also enables you to manage spare disks. When a disk fails, you can use mdadm to make a spare disk active, until such time as the failed disk is replaced. Here is an example mdadm.conf from an OSS with 7 OSTs including external journals. Note how spare groups are configured, so that OSTs without spares still benefit from the spare disks assigned to other OSTs. ARRAY /dev/md10 level=raid6 num-devices=10 UUID=e8926d28:0724ee29:65147008:b8df0bd1 spare-group=raids ARRAY /dev/md11 level=raid6 num-devices=10 spares=1 UUID=7b045948:ac4edfc4:f9d7a279:17b468cd spare-group=raids ARRAY /dev/md12 level=raid6 num-devices=10 spares=1 UUID=29d8c0f0:d9408537:39c8053e:bd476268 spare-group=raids ARRAY /dev/md13 level=raid6 num-devices=10 UUID=1753fa96:fd83a518:d49fc558:9ae3488c spare-group=raids ARRAY /dev/md14 level=raid6 num-devices=10 spares=1 UUID=7f0ad256:0b3459a4:d7366660:cf6c7249 spare-group=raids ARRAY /dev/md15 level=raid6 num-devices=10 UUID=09830fd2:1cac8625:182d9290:2b1ccf2a spare-group=raids ARRAY /dev/md16 level=raid6 num-devices=10 UUID=32bf1b12:4787d254:29e76bd7:684d7217 spare-group=raids ARRAY /dev/md20 level=raid1 num-devices=2 spares=1 UUID=bcfb5f40:7a2ebd50:b3111587:8b393b86 spare-group=journals ARRAY /dev/md21 level=raid1 num-devices=2 spares=1 UUID=6c82d034:3f5465ad:11663a04:58fbc2d1 spare-group=journals ARRAY /dev/md22 level=raid1 num-devices=2 spares=1 UUID=7c7274c5:8b970569:03c22c87:e7a40e11 spare-group=journals ARRAY /dev/md23 level=raid1 num-devices=2 spares=1 UUID=46ecd502:b39cd6d9:dd7e163b:dd9b2620 spare-group=journals ARRAY /dev/md24 level=raid1 num-devices=2 spares=1 UUID=5c099970:2a9919e6:28c9b741:3134be7e spare-group=journals ARRAY /dev/md25 level=raid1 num-devices=2 spares=1 UUID=b44a56c0:b1893164:4416e0b8:75beabc4 spare-group=journals ARRAY /dev/md26 level=raid1 num-devices=2 spares=1 UUID=2adf9d0f:2b7372c5:4e5f483f:3d9a0a25 spare-group=journals # Email address to notify of events (e.g. disk failures) MAILADDR admin@example.com10-12 Lustre 1.8 Operations Manual • December 2010 4. Set up periodic checks of the RAID array. We recommend checking the software RAID arrays monthly for consistency. This can be done using cron and should be scheduled for an idle period so performance is not affected. To start a check, write "check" into /sys/block/[ARRAY]/md/sync_action. For example, to check /dev/md10, run this command on the Lustre server: $ echo check > /sys/block/md10/md/sync_action 5. Format the OSTs and MDT, and continue with normal Lustre setup and configuration. For configuration information, see Configuring Lustre. Note – The default value of stripe_cache_size is 16 KB. These additional resources may be helpful when enabling software RAID on Lustre: ¦ md(4), mdadm(8), mdadm.conf(5) manual pages ¦ Linux software RAID wiki: http://linux-raid.osdl.org/ ¦ Kernel documentation: Documentation/md.txt11-1 C H A P T E R 11 Kerberos This chapter describes how to use Kerberos with Lustre and includes the following sections: ¦ What is Kerberos? ¦ Lustre Setup with Kerberos 11.1 What is Kerberos? Kerberos is a mechanism for authenticating all entities (such as users and services) on an “unsafe” network. Users and services, known as "principals", share a secret password (or key) with the Kerberos server. This key enables principals to verify that messages from the Kerberos server are authentic. By trusting the Kerberos server, users and services can authenticate one another. Caution – Kerberos is a future Lustre feature that is not available in current versions. If you want to test Kerberos with a pre-release version of Lustre, check out the Lustre source from the CVS repository and build it. For more information on checking out Lustre source code, see CVS.11-2 Lustre 1.8 Operations Manual • December 2010 11.2 Lustre Setup with Kerberos Setting up Lustre with Kerberos can provide advanced security protections for the Lustre network. Broadly, Kerberos offers three types of benefit: ¦ Allows Lustre connection peers (MDS, OSS and clients) to authenticate one another. ¦ Protects the integrity of the PTLRPC message from being modified during network transfer. ¦ Protects the privacy of the PTLRPC message from being eavesdropped during network transfer. Kerberos uses the “kernel keyring” client upcall mechanism. 11.2.1 Configuring Kerberos for Lustre This section describes supported Kerberos distributions and how to set up and configure Kerberos on Lustre. 11.2.1.1 Kerberos Distributions Supported on Lustre Lustre supports the following Kerberos distributions: ¦ MIT Kerberos 1.3.x ¦ MIT Kerberos 1.4.x ¦ MIT Kerberos 1.5.x ¦ MIT Kerberos 1.6 (not yet verified) On a number of operating systems, the Kerberos RPMs are installed when the operating system is first installed. To determine if Kerberos RPMs are installed on your OS, run: # rpm -qa | grep krb If Kerberos is installed, the command returns a list like this: krb5-devel-1.4.3-5.1 krb5-libs-1.4.3-5.1 krb5-workstation-1.4.3-5.1 pam_krb5-2.2.6-2.2Chapter 11 Kerberos 11-3 Note – The Heimdal implementation of Kerberos is not currently supported on Lustre, although it support will be added in an upcoming release. 11.2.1.2 Preparing to Set Up Lustre with Kerberos To set up Lustre with Kerberos: 1. Configure NTP to synchronize time across all machines. 2. Configure DNS with zones. 3. Verify that there are fully-qualified domain names (FQDNs), that are resolvable in both forward and reverse directions for all servers. This is required by Kerberos. 4. On every node, install flowing packages: ¦ libgssapi (version 0.10 or higher) Some newer Linux distributions include libgssapi by default. If you do not have libgssapi, build and install it from source: http://www.citi.umich.edu/projects/nfsv4/linux/libgssapi/libssapi-0.10.tar.gz ¦ keyutils11-4 Lustre 1.8 Operations Manual • December 2010 11.2.1.3 Configuring Lustre for Kerberos To configure Lustre for Kerberos: 1. Configure the client nodes. a. For each client node, create a lustre_root principal and generate the keytab. kadmin> addprinc -randkey lustre_root/client_host.domain@REALM kadmin> ktadd -e aes128-cts:normal lustre_root/client_host.domain@REALM b. Install the keytab on the client node. Note – For each client-OST pair, there is only one security context, shared by all users on the client. This protects data written by one user to be passed to an OST by another user due to asynchronous bulk I/O. The client-OST connection only guarantees message integrity or privacy; it does not authenticate users. 2. Configure the MDS nodes. a. For each MDS node, create a lustre_mds principal and generate the keytab. kadmin> addprinc -randkey lustre_mds/mdthost.domain@REALM kadmin> ktadd -e aes128-cts:normal lustre_mds/mdthost.domain@REALM b. Install the keytabl on the MDS node. 3. Configure the OSS nodes. a. For each OSS node, create a lustre_oss principal and generate the keytab. kadmin> addprinc -randkey lustre_oss/osthost.domain@REALM kadmin> ktadd -e aes128-cts:normal lustre_oss/osshost.domain@REALM b. Install the keytab on the OSS node. Tip – To avoid assigning a unique keytab to each client node, create a general lustre_root principal and keytab, and install the keytab on as many client nodes as needed. kadmin> addprinc -randkey lustre_root@REALM kadmin> ktadd -e aes128-cts:normal lustre_root@REALM Remember that if you use a general keytab, then one compromised client means that all client nodes are insecure.Chapter 11 Kerberos 11-5 General Installation Notes ¦ The host.domain should be the FQDN in your network. Otherwise, the server may not recognize any GSS request. ¦ To install a keytab entry on a node, use the ktutil 1 utility. ¦ Lustre supports these encryption types for MIT Kerberos 5, v1.4 and higher: ¦ des-cbc-crc ¦ des-cbc-md5 ¦ des3-hmac-sha1 ¦ aes128-cts ¦ aes256-cts ¦ arcfour-hmac-md5 For MIT Kerberos 1.3.x, only des-cbc-md5 works because of a known issue between libgssapi and the Kerberos library. Note – The encryption type (or enctype) is an identifier specifying the encryption, mode and hash algorithms. Each Kerberos key has an associated enctype that identifies the cryptographic algorithm and mode used when performing cryptographic operations with the key. It is important that the enctypes requested by the client are actually supported on the system hosting the client. This is the case if the defaults that control enctypes are not overridden. 1. Kerberos keytab file maintenance utility.11-6 Lustre 1.8 Operations Manual • December 2010 11.2.1.4 Configuring Kerberos To configure Kerberos to work with Lustre: 1. Modify the files for Kerberos: $ /etc/krb5.conf [libdefaults] default_realm = CLUSTERFS.COM [realms] CLUSTERFS.COM = { kdc = mds16.clustrefs.com admin_server = mds16.clustrefs.com } [domain_realm] .clustrefs.com = CLUSTERFS.COM clustrefs.com = CLSUTREFS.COM [logging] default = FILE:/var/log/kdc.log 2. Prepare the Kerberos database. 3. Create service principals so Lustre supports Kerberos authentication. Note – You can create service principals when configuring your other services to support Kerberos authentication.Chapter 11 Kerberos 11-7 4. Configure the client nodes. For each client node: a. Create a lustre_root principal and generate the keytab: kadmin> addprinc -randkey lustre_root/client_host.domain@REALM kadmin> ktadd -e aes128-cts:normal lustre_root/client_host.domain@REALM This process populates /etc/krb5.keytab, which is not human-readable. Use the ktutil program to read and modify it. b. Install the keytab. Note – There is only one security context for each client-OST pair, shared by all users on the client. This protects data written by one user to be passed to an OST by another user due to asynchronous bulk I/O. The client-OST connection only guarantees message integrity or privacy; it does not authenticate users. 5. Configure the MDS nodes. For each MDT node, create a lustre_mds principal, and generate and install the keytab. kadmin> addprinc -randkey lustre_mds/mdthost.domain@REALM kadmin> ktadd -e aes128-cts:normal lustre_mds/mdthost.domain@REALM 6. Configure the OSS nodes. For each OST node, create a lustre_oss principal, and generate and install the keytab. kadmin> addprinc -randkey lustre_oss/oss_host.domain@REALM kadmin> ktadd -e aes128-cts:normal lustre_oss/oss_host.domain@REALM To save the trouble of assigning a unique keytab for each client node, create a general lustre_root principal and its keytab, and then install the keytab on as many client nodes as needed. kadmin> addprinc -randkey lustre_root@REALM kadmin> ktadd -e aes128-cts:normal lustre_root@REALM Note – If one client is compromised, all client nodes become insecure. For more detailed information on installing and configuring Kerberos, see: http://web.mit.edu/Kerberos/krb5-1.6/#documentation11-8 Lustre 1.8 Operations Manual • December 2010 11.2.1.5 Setting the Environment Perform the following steps to configure the system and network to use Kerberos. System-wide Configuration 1. On each MDT, OST, and client node, add the following line to /etc/fstab to mount them automatically. nfsd /proc/fs/nfsd nfsd defaults 0 0 2. On each MDT and client node, dd the following line to /etc/request-key.conf. create lgssc * * /usr/sbin/lgss_keyring %o %k %t %d %c %u %g %T %P %S Networking If your network is not using SOCKLND or InfiniBand (and uses Quadrics, Elan or Myrinet, for example), configure a /etc/lustre/nid2hostname (simple script that translates a NID to a hostname) on each server node (MDT and OST). This is an example on an Elan cluster: #!/bin/bash set -x exec 2>/tmp/$(basename $0).debug # convert a NID for a LND to a hostname, for GSS for example # called with three arguments: lnd netid nid # $lnd will be string "QSWLND", "GMLND", etc. # $netid will be number in hex string format, like "0x16", etc. # $nid has the same format as $netid # output the corresponding hostname, or error message leaded by a '@' for error logging. lnd=$1 netid=$2 nid=$3Chapter 11 Kerberos 11-9 # uppercase the hex nid=$(echo $nid | tr '[abcdef]' '[ABCDEF]') # and convert to decimal nid=$(echo -e "ibase=16\n${nid/#0x}" | bc) case $lnd in QSWLND) # simply stick "mtn" on the front echo "mtn$nid" ;; *) echo "@unknown LND: $lnd" ;; esac 11.2.1.6 Building Lustre If you are compiling the kernel from the source, enable GSS during configuration: # ./configure --with-linux=path_to_linux_source --enable-gss - \ other-options When you enable Lustre with GSS, the configuration script checks all dependencies, like Kerberos and libgssapi installation, and in-kernel Sun RPC-related facilities. When you install lustre-xxx.rpm on target machines, RPM again checks for dependencies like Kerberos and libgssapi.11-10 Lustre 1.8 Operations Manual • December 2010 11.2.1.7 Running GSS Daemons If you turn on GSS between an MDT-OST or MDT-MDT, GSS treats the MDT as a client. You should run lgssd on the MDT. There are two types of GSS daemons: lgssd and lsvcgssd. Before starting Lustre, make sure they are running on each node: ¦ OST: lsvcgssd ¦ MDT: lsvcgssd ¦ CLI: none Note – Verbose logging can help you make sure Kerberos is set up correctly. To use verbose logging and run it in the foreground, run lsvcgssd -vvv -f -v increases the verbose level of a debugging message by 1. For example, to set the verbose level to 3, run lsvcgssd -v -v -v -f runs lsvcgssd in the foreground, instead of as daemon. We are maintaining a patch against nfs-utils, and bringing necessary patched files into the Lustre tree. After a successful build, GSS daemons are built under lustre/utils/gss and are part of lustre-xxxx.rpm.Chapter 11 Kerberos 11-11 11.2.2 Types of Lustre-Kerberos Flavors There are three major flavors in which you can configure Lustre with Kerberos: ¦ Basic Flavors ¦ Security Flavor ¦ Customized Flavor Select a flavor depending on your priorities and preferences. 11.2.2.1 Basic Flavors Currently, we support six basic flavors: null, plain, krb5n, krb5a, krb5i, and krb5p. Basic Flavor Authentication RPC Message Protection Bulk Data Protection Remarks null N/A N/A N/A* Almost no performance overhead. The on-wire RPC data is compatible with old versions of Lustre (1.4.x, 1.6.x). plain N/A null checksum (adler32) Carries checksum (which only protects data mutating during transfer, cannot guarantee the genuine author because there is no actual authentication). krb5n GSS/Kerberos5 null checksum (adler32) No protection of the RPC message, adler32 checksum protection of bulk data; light performance overhead.11-12 Lustre 1.8 Operations Manual • December 2010 11.2.2.2 Security Flavor A security flavor is a string that describes what kind of security transform is performed on a given PTLRPC connection. It covers two parts of messages, the RPC message and BULK data. You can set either part in one of the following modes: ¦ null – No protection ¦ integrity – Data integrity protection (checksum or signature) ¦ privacy – Data privacy protection (encryption) krb5a GSS/Kerberos5 partial integrity checksum (adler32) Only the header of the RPC message is integrity protected, adler32 checksum protection of bulk data, more performance overhead compared to krb5n. krb5i GSS/Kerberos5 integrity integrity [sha1] RPC message integrity protection algorithm is determined by actual Kerberos algorithms in use; heavy performance overhead. krb5p GSS/Kerberos5 privacy privacy [sha1/aes128] RPC message privacy protection algorithm is determined by actual Kerberos algorithms in use; heaviest performance overhead. * In Lustre 1.6.5, bulk data checksumming is enabled (by default) to provide integrity checking using the adler32 mechanism if the OSTs support it. Adler32 checksums offer lower CPU overhead than CRC32. Basic Flavor Authentication RPC Message Protection Bulk Data Protection RemarksChapter 11 Kerberos 11-13 11.2.2.3 Customized Flavor In most situations, you do not need a customized flavor, a basic flavor is sufficient for regular use. But to some extent, you can customize the flavor string. The flavor string format is: base_flavor[-bulk{nip}[:hash_alg[/cipher_alg]]] Here are some examples of customized flavors: plain-bulkn Use plain on the RPC message (null protection), and no protection on the bulk transfer. krb5i-bulkn Use krb5i on the RPC message, but do not protect the bulk transfer. krb5p-bulki Use krb5p on the RPC message, and protect data integrity of the bulk transfer. krb5p-bulkp:sha512/aes256 Use krb5p on the RPC message, and protect data privacy of the bulk transfer by algorithm SHA512 and AES256. Currently, Lustre supports these bulk data cryptographic algorithms: ¦ Hash: ¦ adler32 ¦ crc32 ¦ md5 ¦ sha1 / sha256 / sha384 / sha512 ¦ wp256 / wp384 / wp512 ¦ Cipher: ¦ arc4 ¦ aes128 / aes192 / aes256 ¦ cast128 / cast256 ¦ twofish128 / twofish25611-14 Lustre 1.8 Operations Manual • December 2010 11.2.2.4 Specifying Security Flavors If you have not specified a security flavor, the CLIENT-MDT connection defaults to plain, and all other connections use null. Specifying Flavors by Mount Options When mounting OST or MDT devices, add the mount option (shown below) to specify the security flavor: # mount -t lustre -o sec=plain /dev/sda1 /mnt/mdt/ This means all connections to this device will use the plain flavor. You can split this sec=flavor as: # mount -t lustre -o sec_mdt={flavor1},sec_cli={flavor1}/dev/sda \ /mnt/mdt/ This means connections from other MDTs to this device will use flavor1, and connections from all clients to this device will use flavor2. Specifying Flavors by On-Disk Parameters You can also specify the security flavors by specifying on-disk parameters on OST and MDT devices: # tune2fs -o security.rpc.mdt=flavor1 -o security.rpc.cli=flavor2 \ device On-disk parameters are overridden by mount options. 11.2.2.5 Mounting Clients Root on client node mounts Lustre without any special tricks.Chapter 11 Kerberos 11-15 11.2.2.6 Rules, Syntax and Examples The general rules and syntax for using Kerberos are: .srpc.flavor.[.]=flavor ¦ : This could be file system name or specific MDT/OST device name. For example, lustre, lustre-MDT0000, lustre-OST0001. ¦ : LNET network name of the RPC initiator. For example, tcp0, elan1, o2ib0. ¦ : This could be one of cli2mdt, cli2ost, mdt2mdt, or mdt2ost. In most cases, you do not need to specify the part. Examples: ¦ Apply krb5i on ALL connections: mgs> lctl conf_param lustre.srpc.flavor.default=krb5i ¦ For nodes in network tcp0, use krb5p. All other nodes use null. mgs> lctl conf_param lustre.srpc.flavor.tcp0=krb5p mgs> lctl conf_param lustre.srpc.flavor.default=null ¦ For nodes in network tcp0, use krb5p; for nodes in elan1, use plain; Among other nodes, clients use krb5i to MDT/OST, MDT use null to other MDTs, MDT use plain to OSTs. mgs> lctl conf_param lustre.srpc.flavor.tcp0=krb5p mgs> lctl conf_param lustre.srpc.flavor.elan1=plain mgs> lctl conf_param lustre.srpc.flavor.default.cli2mdt=krb5i mgs> lctl conf_param lustre.srpc.flavor.default.cli2ost=krb5i mgs> lctl conf_param lustre.srpc.flavor.default.mdt2mdt=null mgs> lctl conf_param lustre.srpc.flavor.default.mdt2ost=plain11-16 Lustre 1.8 Operations Manual • December 2010 11.2.2.7 Authenticating Normal Users On client nodes, non-root users must use kinit to access Lustre (just like other Kerberized applications). kinit is used to obtain and cache Kerberos ticket-granting tickets. Two requirements to authenticating users: ¦ Before kinit is run, the user must be registered as a principal with the Kerberos server (the Key Distribution Center or KDC). In KDC, the username is noted as username@REALM. ¦ The client and MDT nodes should have the same user database. To destroy the established security contexts before logging out, run lfs flushctx: # lfs flushctx [-k] Here -k also means destroy the on-disk Kerberos credential cache. It is equivalent to kdestroy. Otherwise, it only destroys established contexts in the Lustre kernel.12-1 C H A P T E R 12 Network Interface Bonding This chapter describes how to set up bonding with Lustre, and includes the following sections: ¦ Network Bonding ¦ Requirements ¦ Using Lustre with Multiple NICs versus Bonding NICs ¦ Bonding Module Parameters ¦ Setting Up Bonding ¦ Configuring Lustre with Bonding 12.1 Network Bonding Bonding, also known as link aggregation, trunking and port trunking, is a method of aggregating multiple physical network links into a single logical link for increased bandwidth. Several different types of bonding are supported in Linux. All these types are referred to as “modes,” and use the bonding kernel module. Modes 0 to 3 provide support for load balancing and fault tolerance by using multiple interfaces. Mode 4 aggregates a group of interfaces into a single virtual interface where all members of the group share the same speed and duplex settings. This mode is described under IEEE spec 802.3ad, and it is referred to as either “mode 4” or “802.3ad.” (802.3ad refers to mode 4 only. The detail is contained in Clause 43 of the IEEE 8 - the larger 802.3 specification. For more information, consult IEEE.)12-2 Lustre 1.8 Operations Manual • December 2010 12.2 Requirements The most basic requirement for successful bonding is that both endpoints of the connection must support bonding. In a normal case, the non-server endpoint is a switch. (Two systems connected via crossover cables can also use bonding.) Any switch used must explicitly support 802.3ad Dynamic Link Aggregation. The kernel must also support bonding. All supported Lustre kernels have bonding functionality. The network driver for the interfaces to be bonded must have the ethtool support. To determine slave speed and duplex settings, ethtool support is necessary. All recent network drivers implement it. To verify that your interface supports ethtool, run: # which ethtool /sbin/ethtool # ethtool eth0 Settings for eth0: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full/ 100baseT/Half 100baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: MII PHYAD: 1 Transceiver: internal Auto-negotiation: on Supports Wake-on: pumbg Wake-on: d Current message level: 0x00000001 (1) Link detected: yesChapter 12 Network Interface Bonding 12-3 # ethtool eth1 Settings for eth1: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: MII PHYAD: 32 Transceiver: internal Auto-negotiation: on Supports Wake-on: pumbg Wake-on: d Current message level: 0x00000007 (7) Link detected: yes To quickly check whether your kernel supports bonding, run: # grep ifenslave /sbin/ifup # which ifenslave /sbin/ifenslave Note – Bonding and ethtool have been available since 2000. All Lustre-supported kernels include this functionality.12-4 Lustre 1.8 Operations Manual • December 2010 12.3 Using Lustre with Multiple NICs versus Bonding NICs Lustre can use multiple NICs without bonding. There is a difference in performance when Lustre uses multiple NICs versus when it uses bonding NICs. Whether an aggregated link actually yields a performance improvement proportional to the number of links provided, depends on network traffic patterns and the algorithm used by the devices to distribute frames among aggregated links. Performance with bonding depends on: ¦ Out-of-order packet delivery This can trigger TCP congestion control. To avoid this, some bonding drivers restrict a single TCP conversation to a single adapter within the bonded group. ¦ Load balancing between devices in the bonded group. Consider a scenario with a two CPU node with two NICs. If the NICs are bonded, Lustre establishes a single bundle of sockets to each peer. Since ksocklnd bind sockets to CPUs, only one CPU moves data in and out of the socket for a uni-directional data flow to each peer. If the NICs are not bonded, Lustre establishes two bundles of sockets to the peer. Since ksocklnd spreads traffic between sockets, and sockets between CPUs, both CPUs move data.Chapter 12 Network Interface Bonding 12-5 12.4 Bonding Module Parameters Bonding module parameters control various aspects of bonding. Outgoing traffic is mapped across the slave interfaces according to the transmit hash policy. For Lustre, we recommend that you set the xmit_hash_policy option to the layer3+4 option for bonding. This policy uses upper layer protocol information if available to generate the hash. This allows traffic to a particular network peer to span multiple slaves, although a single connection does not span multiple slaves. $ xmit_hash_policy=layer3+4 The miimon option enables users to monitor the link status. (The parameter is a time interval in milliseconds.) It makes an interface failure transparent to avoid serious network degradation during link failures. A reasonable default setting is 100 milliseconds; run: $ miimon=100 For a busy network, increase the timeout. 12.5 Setting Up Bonding To set up bonding: 1. Create a virtual 'bond' interface by creating a configuration file in: /etc/sysconfig/network-scripts/ # vi /etc/sysconfig/ \ network-scripts/ifcfg-bond0 2. Append the following lines to the file. DEVICE=bond0 IPADDR=192.168.10.79 # Use the free IP Address of your network NETWORK=192.168.10.0 NETMASK=255.255.255.0 USERCTL=no BOOTPROTO=none ONBOOT=yes12-6 Lustre 1.8 Operations Manual • December 2010 3. Attach one or more slave interfaces to the bond interface. Modify the eth0 and eth1 configuration files (using a VI text editor). a. Use the VI text editor to open the eth0 configuration file. # vi /etc/sysconfig/network-scripts/ifcfg-eth0 b. Modify/append the eth0 file as follows: DEVICE=eth0 USERCTL=no ONBOOT=yes MASTER=bond0 SLAVE=yes BOOTPROTO=none c. Use the VI text editor to open the eth1 configuration file. # vi /etc/sysconfig/network-scripts/ifcfg-eth1 d. Modify/append the eth1 file as follows: DEVICE=eth1 USERCTL=no ONBOOT=yes MASTER=bond0 SLAVE=yes BOOTPROTO=none 4. Set up the bond interface and its options in /etc/modprobe.conf. Start the slave interfaces by your normal network method. # vi /etc/modprobe.conf a. Append the following lines to the file. alias bond0 bonding options bond0 mode=balance-alb miimon=100 b. Load the bonding module. # modprobe bonding # ifconfig bond0 up # ifenslave bond0 eth0 eth1 5. Start/restart the slave interfaces (using your normal network method). Note – You must modprobe the bonding module for each bonded interface. If you wish to create bond0 and bond1, two entries in modprobe.conf are required.Chapter 12 Network Interface Bonding 12-7 The examples below are from RedHat systems. For setup use: /etc/sysconfig/networking-scripts/ifcfg-* The OSDL website referenced below includes detailed instructions for other configuration methods, instructions to use DHCP with bonding, and other setup details. We strongly recommend you use this website. http://linux-net.osdl.org/index.php/Bonding 6. Check /proc/net/bonding to determine status on bonding. There should be a file there for each bond interface. # cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.0.3 (March 23, 2006) Bonding Mode: load balancing (round-robin) MII Status: up MII Polling Interval (ms): 0 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth0 MII Status: up Link Failure Count: 0 Permanent HW addr: 4c:00:10:ac:61:e0 Slave Interface: eth1 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:14:2a:7c:40:1d12-8 Lustre 1.8 Operations Manual • December 2010 7. Use ethtool or ifconfig to check the interface state. ifconfig lists the first bonded interface as “bond0.” ifconfig bond0 Link encap:Ethernet HWaddr 4C:00:10:AC:61:E0 inet addr:192.168.10.79 Bcast:192.168.10.255 \ Mask:255.255.255.0 inet6 addr: fe80::4e00:10ff:feac:61e0/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:3091 errors:0 dropped:0 overruns:0 frame:0 TX packets:880 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:314203 (306.8 KiB) TX bytes:129834 (126.7 KiB) eth0 Link encap:Ethernet HWaddr 4C:00:10:AC:61:E0 inet6 addr: fe80::4e00:10ff:feac:61e0/64 Scope:Link UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:1581 errors:0 dropped:0 overruns:0 frame:0 TX packets:448 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:162084 (158.2 KiB) TX bytes:67245 (65.6 KiB) Interrupt:193 Base address:0x8c00 eth1 Link encap:Ethernet HWaddr 4C:00:10:AC:61:E0 inet6 addr: fe80::4e00:10ff:feac:61e0/64 Scope:Link UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:1513 errors:0 dropped:0 overruns:0 frame:0 TX packets:444 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:152299 (148.7 KiB) TX bytes:64517 (63.0 KiB) Interrupt:185 Base address:0x6000Chapter 12 Network Interface Bonding 12-9 12.5.1 Examples This is an example of modprobe.conf for bonding Ethernet interfaces eth1 and eth2 to bond0: # cat /etc/modprobe.conf alias eth0 8139too alias scsi_hostadapter sata_via alias scsi_hostadapter1 usb-storage alias snd-card-0 snd-via82xx options snd-card-0 index=0 options snd-via82xx index=0 alias bond0 bonding options bond0 mode=balance-alb miimon=100 options lnet networks=tcp alias eth1 via-rhine # cat /etc/sysconfig/network-scripts/ifcfg-bond0 DEVICE=bond0 BOOTPROTO=none NETMASK=255.255.255.0 IPADDR=192.168.10.79 # (Assign here the IP of the bonded interface.) ONBOOT=yes USERCTL=no ifcfg-ethx # cat /etc/sysconfig/network-scripts/ifcfg-eth0 TYPE=Ethernet DEVICE=eth0 HWADDR=4c:00:10:ac:61:e0 BOOTPROTO=none ONBOOT=yes USERCTL=no IPV6INIT=no PEERDNS=yes MASTER=bond0 SLAVE=yes12-10 Lustre 1.8 Operations Manual • December 2010 In the following example, the bond0 interface is the master (MASTER) while eth0 and eth1 are slaves (SLAVE). Note – All slaves of bond0 have the same MAC address (Hwaddr) – bond0. All modes, except TLB and ALB, have this MAC address. TLB and ALB require a unique MAC address for each slave. $ /sbin/ifconfig bond0Link encap:EthernetHwaddr 00:C0:F0:1F:37:B4 inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0 TX packets:3286647 errors:1 dropped:0 overruns:1 carrier:0 collisions:0 txqueuelen:0 eth0Link encap:EthernetHwaddr 00:C0:F0:1F:37:B4 inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0 TX packets:1643167 errors:1 dropped:0 overruns:1 carrier:0 collisions:0 txqueuelen:100 Interrupt:10 Base address:0x1080 eth1Link encap:EthernetHwaddr 00:C0:F0:1F:37:B4 inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0 TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 Interrupt:9 Base address:0x1400Chapter 12 Network Interface Bonding 12-11 12.6 Configuring Lustre with Bonding Lustre uses the IP address of the bonded interfaces and requires no special configuration. It treats the bonded interface as a regular TCP/IP interface. If needed, specify “bond0” using the Lustre networks parameter in /etc/modprobe options lnet networks=tcp(bond0) 12.6.1 Bonding References We recommend the following bonding references: In the Linux kernel source tree, see documentation/networking/bonding.txt http://linux-ip.net/html/ether-bonding.html http://www.sourceforge.net/projects/bonding This is the bonding SourceForge website: http://linux-net.osdl.org/index.php/Bonding This is the most extensive reference and we highly recommend it. This website includes explanations of more complicated setups, including the use of DHCP with bonding.12-12 Lustre 1.8 Operations Manual • December 201013-1 C H A P T E R 13 Upgrading and Downgrading Lustre The chapter describes how to upgrade and downgrade between different Lustre versions and includes the following sections: ¦ Supported Upgrades ¦ Lustre Interoperability ¦ Upgrading Lustre 1.6.x to 1.8.x ¦ Upgrading Lustre 1.8.x to the Next Minor Version ¦ Downgrading from Lustre 1.8.x to 1.6.x13-2 Lustre 1.8 Operations Manual • December 2010 13.1 Supported Upgrades For Lustre 1.8.x, the following upgrades are supported: ¦ Lustre 1.6.x (latest version) to Lustre 1.8.x (latest version) ¦ Lustre 1.8.x (any minor version) to Lustre 1.8.x (latest version) 13.2 Lustre Interoperability Lustre interoperability enables 1.8.x servers ("new" servers) to work with 1.6.x clients ("old" clients), 1.6.x servers ("old" servers) to work with 1.8.x clients ("new" clients), and "mixed" environments with 1.6.x and 1.8.x servers. For example, half of each OSS failover pair could be upgraded to enable a quick reversion to 1.6 by powering down the 1.8 servers. This table describes interoperability between Lustre clients, OSTs and MDTs with different versions of Lustre installed. Lustre Component Interoperability with Other Lustre Components Clients • Old, live clients can communicate with old/new/mixed servers • Old clients can start up using old/new/mixed servers • New clients can start up using old/new/mixed servers Note - Old clients cannot mount a file system that was created by a new MDT. OSTs • Old OSTs can communicate with new clients/MDT • New OSTs can only be started after the MGS has been started (typically this means "after the MDT has been upgraded") MDTs • Old MDT can communicate with new clients • New, co-located MGS/MDT can be started at any point • New, non co-located MDT can be started after the MGS startsChapter 13 Upgrading and Downgrading Lustre 13-3 13.3 Upgrading Lustre 1.6.x to 1.8.x Two upgrade paths are supported to meet the upgrade requirements of different Lustre environments. ¦ Complete file system - All servers and clients are shut down and upgraded at the same time. See Performing a Complete File System Upgrade. ¦ Rolling upgrade - Individual servers (or their failover partners) and clients are upgraded one at a time, so the file system never goes down. See Performing a Rolling Upgrade. Note – If you upgrade some Lustre components to 1.8.x but not others (such as running 1.8 clients in a file system with 1.6 OSTs), and run a mixed environment, you may see one or more warnings similar to this: LustreError: 3877:0:(socklnd_cb.c:2228:ksocknal_recv_hello()) Unknown protocol version (2.x expected) from 192.168.2.43 This warning is given when the 1.6 and 1.8 components use different protocols. It can be safely ignored because the Lustre components negotiate a common protocol. In this example, the 1.8 clients fall back to use the 1.6 protocol with the 1.6 OSTs. Note – In a Lustre system upgraded to 1.8, extents are disabled on the MDT. Leaving extents disabled does not cause problems, but it does report a warning to the system logs. Extents can be enabled on the MDT, but doing so results in a file system that is no longer compatible with 1.6 host services. To enable extents, run: # tune2fs -O extents /path/to/mdt/device13-4 Lustre 1.8 Operations Manual • December 2010 13.3.1 Performing a Complete File System Upgrade This procedure describes a complete file system upgrade in which 1.8.x Lustre packages are installed on multiple 1.6.x servers and clients, requiring a file system shut down. If you want to upgrade one Lustre component at a time and avoid the shutdown, see Performing a Rolling Upgrade. Tip – In a Lustre upgrade, the package install and file system unmount steps are reversible; you can do either step first. To minimize downtime, this procedure first performs the 1.8.x package installation, and then unmounts the file system. 1. Make a complete, restorable file system backup before upgrading Lustre. 2. Install the 1.8.x packages on the Lustre servers and/or clients. Some or all servers can be upgraded. Some or all clients can be upgraded. For help determining where to install a specific package, see TABLE 3-1 (Lustre packages, descriptions and installation guidance). a. Install the kernel, modules and ldiskfs packages. For example: $ rpm -ivh kernel-lustre-smp- \ kernel-ib- \ lustre-modules- \ lustre-ldiskfs- b. Upgrade the utilities/userspace packages. For example: $ rpm -Uvh lustre- c. If a new e2fsprogs package is available, upgrade it. For example: $ rpm -Uvh e2fsprogs- Use e2fsprogs-1.41-6 or later, available at: http://downloads.lustre.org/public/tools/e2fsprogs/ d. (Optional) If you want to add optional packages to your Lustre system, install them now.Chapter 13 Upgrading and Downgrading Lustre 13-5 3. Shut down the file system. Shut down the components in this order: clients, then the MDT, then OSTs. Unmounting a block device causes Lustre to be shut down on that node. a. Unmount the clients. On each client node, run: umount b. Unmount the MDT. On the MDS node, run: umount c. Unmount the OSTs (be sure to unmount all OSTs). On each OSS node, run: umount 4. Unload the old Lustre modules by either: ¦ Rebooting the node - OR - ¦ Removing the Lustre modules manually. Run lustre_rmmod several times and use lsmod to check the currently loaded modules. 5. Start the upgraded file system. Start the components in this order: OSTs, then the MDT, then clients. a. Mount the OSTs (be sure to mount all OSTs). On each OSS node, run: mount -t lustre b. Mount the MDT. On the MDS node, run: mount -t lustre c. Mount the file system on the clients. On each client node, run: mount -t lustre :/ If you have a problem upgrading Lustre, contact us via the Bugzilla bug tracker.13-6 Lustre 1.8 Operations Manual • December 2010 13.3.2 Performing a Rolling Upgrade This procedure describes a rolling upgrade in which one Lustre component (server or client) is upgraded and restarted at a time while the file system is running. If you want to upgrade the complete Lustre file system or multiple components at a time, requiring a file system shutdown, see Performing a Complete File System Upgrade. Note – The suggested upgrade order is the MGS first, then OSTs, then the MDT, and then clients. These are general guidelines, and specific upgrade requirements can be found in the release notes for a given Lustre version. If no particular restrictions are stated, then the suggested upgrade order may be rearranged; bear in mind that the suggested order is the most heavily tested by the Lustre team. Note – If the Lustre component to be upgraded is an OSS in a failover pair, follow these special upgrade steps to minimize downtime: 1. Fail over the server to its peer server, so the file system remains available. 2. Install the Lustre 1.8.x packages on the idle server. 3. Unload the old Lustre modules on the idle server by either: Rebooting the node - OR - Removing the Lustre modules manually by running the lustre_rmmod command several times and checking the currently loaded modules with the lsmod command. 4. Fail back services to the idle (now upgraded) server. 5. Repeat Steps 1 to 4 on the peer server. This limits the outage (per OSS) to a single server for as long as it takes to fail over.Chapter 13 Upgrading and Downgrading Lustre 13-7 1. Make a complete, restorable file system backup before upgrading Lustre. 2. Install the 1.8.x packages on the Lustre component (server or client). For help determining where to install a specific package, see TABLE 3-1 (Lustre packages, descriptions and installation guidance). a. Install the kernel, modules and ldiskfs packages. For example: $ rpm -ivh kernel-lustre-smp- \ kernel-ib- \ lustre-modules- \ lustre-ldiskfs- b. Upgrade the utilities/userspace packages. For example: $ rpm -Uvh lustre- c. If a new e2fsprogs package is available, upgrade it. For example: $ rpm -Uvh e2fsprogs- Use e2fsprogs-1.41-6 or later, available at: http://downloads.lustre.org/public/tools/e2fsprogs/ d. (Optional) If you want to add optional packages to your Lustre system, install them now. 3. Unload the old Lustre modules by either: ¦ Rebooting the node - OR - ¦ Removing the Lustre modules manually. Run lustre_rmmod several times and use lsmod to check the currently-loaded modules. 4. If the upgraded component is a server, fail back services to it. If you have a problem upgrading Lustre, contact us via the Bugzilla bug tracker.13-8 Lustre 1.8 Operations Manual • December 2010 13.4 Upgrading Lustre 1.8.x to the Next Minor Version To upgrade Lustre 1.8.x to the next minor version, for example, Lustre 1.8.0.1 > 1.8.x, follow these procedures: ¦ To upgrade the complete file system or multiple file system components at the same time, requiring a file system shutdown, see Performing a Complete File System Upgrade ¦ To upgrade one Lustre component (server or client) at a time, while the file system is running, see Performing a Rolling Upgrade 13.5 Downgrading from Lustre 1.8.x to 1.6.x This section describes how to downgrade from Lustre 1.8.x to 1.6.x. Only file systems that were upgraded from 1.6.x can be downgraded to 1.6.x. A file system that was created or reformatted under Lustre 1.8.x cannot be downgraded. Two paths are available to meet the downgrade requirements of different Lustre environments. ¦ Complete file system - File system is shut down and all servers and clients are downgraded at once. See Performing a Complete File System Downgrade. ¦ Individual servers / clients - Individual servers and clients are downgraded one at a time and restarted (a "rolling downgrade"), so the file system never goes down. See Performing a Rolling Downgrade.Chapter 13 Upgrading and Downgrading Lustre 13-9 13.5.1 Performing a Complete File System Downgrade This procedure describes a complete file system downgrade in which 1.6.x Lustre packages are installed on multiple 1.8.x servers and clients, requiring a file system shut down. If you want to upgrade one Lustre component at a time and avoid the shutdown, see Performing a Rolling Downgrade. Tip – In a Lustre downgrade, the package install and file system unmount steps are reversible; you can do either step first. To minimize downtime, this procedure first performs the 1.6.x package installation, and then unmounts the file system. 1. Make a complete, restorable file system backup before downgrading Lustre. 2. Verify that 1.6.x packages are installed on the Lustre servers and/or clients. a. Check that the kernel, modules and ldiskfs packages are installed. The 1.6.x kernel, modules and ldiskfs packages should be on all nodes because of the earlier upgrade to 1.8.x, unless they were removed after the upgrade. If it is necessary to install kernel, modules or ldiskfs packages, use the rpm -ivh command. For example: $ rpm -ivh kernel-lustre-smp- \ kernel-ib- \ lustre-modules- \ lustre-ldiskfs- For help determining where to install a specific package, see TABLE 3-1 (Lustre packages, descriptions and installation guidance). b. Install the utilities/userspace packages, using the --oldpackage option. For example: rpm -Uvh --oldpackage lustre- Note – You do not need to downgrade or take any action with e2fsprogs.13-10 Lustre 1.8 Operations Manual • December 2010 3. Shut down the file system. Shut down the components in this order: clients, then the MDT, then OSTs. Unmounting a block device causes Lustre to be shut down on that node. a. Unmount the clients. On each client node, run: umount b. Unmount the MDT. On the MDS node, run: umount c. Unmount the OSTs (be sure to unmount all OSTs). On each OSS node, run: umount 4. Unload the old Lustre modules by either: ¦ Rebooting the node - OR - ¦ Removing the Lustre modules manually. Run lustre_rmmod several times and use lsmod to check the currently loaded modules. 5. Start the downgraded file system. Start the components in this order: OSTs, then the MDT, then clients. a. Mount the OSTs (be sure to mount all OSTs). On each OSS node, run: mount -t lustre b. Mount the MDT. On the MDS node, run: mount -t lustre c. Mount the file system on the clients. On each client node, run: mount -t lustre :/ If you have a problem downgrading Lustre, contact us via the Bugzilla bug tracker.Chapter 13 Upgrading and Downgrading Lustre 13-11 13.5.2 Performing a Rolling Downgrade This procedure describes a rolling downgrade in which one Lustre component (server or client) is downgraded and restarted at a time while the file system is running. If you want to downgrade the complete Lustre file system or multiple components at a time, requiring a file system shutdown, see Performing a Complete File System Downgrade. Note – If the Lustre component to be downgraded is an OSS in a failover pair, follow these special downgrade steps to minimize downtime: 1. Fail over the server to its peer server, so the file system remains available. 2. Install the Lustre 1.8.x packages on the idle server. 3. Unload the old Lustre modules on the idle server by either: Rebooting the node - OR - Removing the Lustre modules manually by running the lustre_rmmod command several times and checking the currently loaded modules with the lsmod command. 4. Fail back services to the idle (now upgraded) server. 5. Repeat Steps 1 to 4 on the peer server. This limits the outage (per OSS) to a single server for as long as it takes to fail over.13-12 Lustre 1.8 Operations Manual • December 2010 1. Make a complete, restorable file system backup before downgrading Lustre. 2. Install the 1.6.x packages on the Lustre component (server or client). For help determining where to install a specific package, see TABLE 3-1 (Lustre packages, descriptions and installation guidance). a. Install the kernel, modules and ldiskfs packages. For example: $ rpm -ivh kernel-lustre-smp- \ kernel-ib- \ lustre-modules- \ lustre-ldiskfs- b. Downgrade the utilities/userspace packages, using the --oldpackage option. For example: $ rpm -Uvh --oldpackage lustre- Note – You do not need to downgrade or take any action with e2fsprogs. 3. Unload the old Lustre modules by either: ¦ Rebooting the node - OR - ¦ Removing the Lustre modules manually. Run lustre_rmmod several times and use lsmod to check the currently-loaded modules. 4. If the upgraded component is a server, fail back services to it. If you have a problem upgrading Lustre, contact us via the Bugzilla bug tracker.14-1 C H A P T E R 14 Lustre SNMP Module The Lustre SNMP module reports information about Lustre components and system status, and generates traps if an LBUG occurs. The Lustre SNMP module works with the net-snmp. The module consists of a plug-in (lustresnmp.so), which is loaded by the snmpd daemon, and a MIB file (Lustre-MIB.txt). This chapter describes how to install and use the Lustre SNMP module, and includes the following sections: ¦ Installing the Lustre SNMP Module ¦ Building the Lustre SNMP Module ¦ Using the Lustre SNMP Module14-2 Lustre 1.8 Operations Manual • December 2010 14.1 Installing the Lustre SNMP Module To install the Lustre SNMP module: 1. Locate the SNMP plug-in (lustresnmp.so) in the base Lustre RPM and install it. /usr/lib/lustre/snmp/lustresnmp.so 2. Locate the MIB (Lustre-MIB.txt) in /usr/share/lustre/snmp/mibs/Lustre-MIB.txt and append the following line to snmpd.con. dlmod lustresnmp /usr/lib/lustre/snmp/lustresnmp.so 3. You may need to copy Lustre-MIB.txt to a different location to use few tools. For this, use either of these commands. ~/.snmp/mibs /usr/local/share/snmp/mibs 14.2 Building the Lustre SNMP Module To build the Lustre SNMP module, you need the net-snmp-devel package. The default net-snmp install includes a snmpd.conf file. 1. Complete the net-snmp setup by checking and editing the snmpd.conf file, located in /etc/snmp /etc/snmp/snmpd.conf 2. Build the Lustre SNMP module from the Lustre src.rpm ¦ Install the Lustre source ¦ Run ./configure ¦ Add the --enable-snmp optionChapter 14 Lustre SNMP Module 14-3 14.3 Using the Lustre SNMP Module Once the Lustre SNMP module in installed and built, use it for purposes: ¦ For all Lustre components, the SNMP module reports a number and total and free capacity (usually in bytes). ¦ Depending on the component type, SNMP also reports total or free numbers for objects like OSD and OSC or other files (LOV, MDC, and so on). ¦ The Lustre SNMP module provides one read/write variable, sysStatus, which starts and stops Lustre. ¦ The sysHealthCheck object reports status either as healthy' or 'not healthy' and provides information for the failure. ¦ The Lustre SNMP module generates traps on the detection of LBUG (lustrePortalsCatastropeTrap), and detection of various OBD-specific healthchecks (lustreOBDUnhealthyTrap).14-4 Lustre 1.8 Operations Manual • December 201015-1 C H A P T E R 15 Backup and Restore Lustre provides backups at the file system-level, device-level and file-level. This chapter describes how to backup and restore on Lustre, and includes the following sections: ¦ Backing up a File System ¦ Backing up a Device (MDS or OST) ¦ Backing up Files ¦ Restoring from a File-level Backup ¦ Using LVM Snapshots with Lustre 15.1 Backing up a File System Backing up a complete file system gives you full control over the files to back up, and allows restoration of individual files as needed. File system-level backups are also the easiest to integrate into existing backup solutions. File system backups are performed from a Lustre client (or many clients working parallel in different directories) rather than on individual server nodes; this is no different than backing up any other file system. However, due to the large size of most Lustre file systems, it is not always possible to get a complete backup. We recommend that you back up subsets of a file system. This includes subdirectories of the entire file system, filesets for a single user, files incremented by date, and so on.15-2 Lustre 1.8 Operations Manual • December 2010 15.2 Backing up a Device (MDS or OST) In some cases, it is useful to do a full, device-level backup of an individual device (MDS or OST), before replacing hardware, performing maintenance, etc. Doing full device-level backups ensures that all of the data is preserved in the original state and is the easiest method of doing a backup. Note – A device-level backup of the MDS is especially important because, if it fails permanently, the entire file system would need to be restored. If hardware replacement is the reason for the backup or if a spare storage device is available, it is possible to do a raw copy of the MDS or OST from one block device to the other, as long as the new device is at least as large as the original device. To do this, run: dd if=/dev/{original} of=/dev/{new} bs=1M If hardware errors cause read problems on the original device, use the command below to allow as much data as possible to be read from the original device while skipping sections of the disk with errors: dd if=/dev/{original} of=/dev/{new} bs=4k conv=sync,noerror count= {original size in 4kB blocks} Even in the face of hardware errors, the ext3 file system is very robust and it may be possible to recover the file system data after running e2fsck -f on the new device. 15.2.1 Backing Up the MDS This procedure provides another way to back up the MDS. 1. Make a mount point for the file system. Run: mkdir -p /mnt/mds 2. Mount the file system. Run: mount -t ldiskfs {mdsdev} /mnt/mds 3. Change to the mount point being backed up. Run: cd /mnt/mdsChapter 15 Backup and Restore 15-3 4. Back up the EAs. Run: getfattr -R -d -m '.*' -P . > ea.bak Note – In most distributions, the getfattr command is part of the "attr" package. If the getfattr command returns errors like Operation not supported, then the kernel does not correctly support EAs. Stop and use a different backup method or contact us for assistance. 5. Verify that the ea.bak file has properly backed up the EA data on the MDS. Without this EA data, the backup is not useful. Look at this file with "more" or a text editor. For each file, it should have an item similar to this: # file: ROOT/mds_md5sum3.txt trusted.lov= 0s0AvRCwEAAABXoKUCAAAAAAAAAAAAAAAAAAAQAAEAAADD5QoAAAAAAAAAAAAAAAAA AAAAAAEAAAA= 6. Back up all file system data. Run: tar czvf {backup file}.tgz --sparse . Note – In Lustre 1.6.7 and later, the --sparse option reduces the size of the backup file. Be sure to use it so the tar command does not mistakenly create an archive full of zeros. 7. Change directory out of the mounted file system. Run: cd - 8. Unmount the file system. Run: umount /mnt/mds Note – When restoring an MDT backup on a different node as part of an MDT migration, you also have to change server NIDs and use the --writeconf command to re-generate the configuration logs. See Changing a Server NID and Regenerating Lustre Configuration Logs. 15.2.2 Backing Up an OST Follow the same procedure as Backing Up the MDS (except skip Step 5) and, for each OST device file system, replace mds with ost in the commands.15-4 Lustre 1.8 Operations Manual • December 2010 15.3 Backing up Files In other cases, it is desirable to back up only the file data on an MDS or OST instead of the entire device, e.g., if the device is very large but has little data in it, if the configuration of the parameters of the ext3 filesystem need to be changed, to use less space for the backup, etc. In this situation, it is possible to mount the ext3 filesystem directly from the storage device, and do a file-level backup. Lustre MUST STOP be stopped on this node. 15.3.1 Backing up Extended Attributes In Lustre, each OST object has an extended attribute (EA) that contains the MDT inode number and stripe index for the object. The EA’s striping information includes the location of file data on the OSTs and OST pool membership. The EA data must be backed up or the file backup will not be useful. Current backup tools do not properly save the EA data, so the following extra steps are required. 1. Make a mountpoint for the file system. mkdir /mnt/mds 2. Mount the filesystem. mount -t ldiskfs {olddev} /mnt/mds 3. Change to the mountpoint being backed up. cd /mnt/mds 4. Back up the extended attributes. getfattr -R -d -m '.*' -P . > ea.bak In most distributions, the getfattr command is part of the "attr" package. If the getfattr command returns errors like "Operation not supported", then your kernel does not support EAs correctly. Stop and use a different backup method or submit a Bugzilla ticket. 5. Verify that the ea.bak file has properly backed up the EA data on the MDS. You can look at this file with "more" or a text editor. For each file, it should have an item similar to this # file: ROOT/mds_md5sum3.txt trusted.lov=0s0AvRCwEAAABXoKUCAAAAAAAAQAAEAAADD5QoAAAAAAAAAAEAAAA=Chapter 15 Backup and Restore 15-5 6. Back up all file system data. tar czvf {backup file}.tgz --sparse . 7. Change out of the mounted file system. cd - 8. Unmount the file system. umount /mnt/mds 9. Print the file system label and write it down. e2label {olddev} The same process should be followed on each MDS or OST file system. 15.4 Restoring from a File-level Backup To restore data from a file-level backup, you need to format the device, restore the file data and then restore the EA data. 1. Format the new device. Run: mkfs.lustre {--mdt|--ost} {other options} {newdev} 2. Mount the file system. Run: mount -t ldiskfs {newdev} /mnt/mds 3. Change to the new file system mount point. Run: cd /mnt/mds 4. Restore the file system backup. Run: tar xzvpf {backup file} --sparse 5. Restore the file system extended attributes. Run: setfattr --restore=ea.bak 6. Verify that the extended attributes were restored. If this is not correct, then all data in the files will be lost, and would show up as all files in the filesystem having zero length. getfattr -d -m ".*" ROOT/mds_md5sum3.txt trusted.lov=0s0AvRCwEAAABXoKUCAAAAAAAAQAAEAAADD5QoAAAAAAAAAEAAAA=15-6 Lustre 1.8 Operations Manual • December 2010 7. Remove the (now invalid) recovery logs. Run: rm OBJECTS/* CATALOGS 8. Change out of the MDS file system. cd - 9. Unmount the MDS file system. umount /mnt/mds If the file system was used between the time the backup was made and when it was restored, then the lfsck tool (part of Lustre e2fsprogs) can be run to ensure the file system is coherent. If all of the device file systems were backed up at the same time after the entire Lustre file system was stopped, this is not necessary. The file system should be immediately usable even if lfsck is not run, though there will be I/O errors reading from files that are present on the MDS but not the OSTs, and files that were created after the MDS backup will not be accessible/visible.Chapter 15 Backup and Restore 15-7 15.5 Using LVM Snapshots with Lustre If you want to perform disk-based backups (because, for example, access to the backup system needs to be as fast as to the primary Lustre file system), you can use the Linux LVM snapshot tool to maintain multiple, incremental file system backups. Because LVM snapshots cost CPU cycles as new files are written, taking snapshots of the main Lustre file system will probably result in unacceptable performance losses. You should create a new, backup Lustre file system and periodically (e.g., nightly) back up new/changed files to it. Periodic snapshots can be taken of this backup file system to create a series of "full" backups. Note – Creating an LVM snapshot is not as reliable as making a separate backup, because the LVM snapshot shares the same disks as the primary MDT device, and depends on the primary MDT device for much of its data. If the primary MDT device becomes corrupted, this may result in the snapshot being corrupted. 15.5.1 Creating an LVM-based Backup File System Use this procedure to create a backup Lustre file system for use with the LVM snapshot mechanism. 1. Create LVM volumes for the MDT and OSTs. Create LVM devices for your MDT and OST targets. Make sure not to use the entire disk for the targets; save some room for the snapshots. The snapshots start out as 0 size, but grow as you make changes to the current file system. If you expect to change 20% of the file system between backups, the most recent snapshot will be 20% of the target size, the next older one will be 40%, etc. Here is an example: cfs21:~# pvcreate /dev/sda1 Physical volume "/dev/sda1" successfully created cfs21:~# vgcreate volgroup /dev/sda1 Volume group "volgroup" successfully created cfs21:~# lvcreate -L200M -nMDT volgroup Logical volume "MDT" created cfs21:~# lvcreate -L200M -nOST0 volgroup Logical volume "OST0" created cfs21:~# lvscan ACTIVE '/dev/volgroup/MDT' [200.00 MB] inherit ACTIVE '/dev/volgroup/OST0' [200.00 MB] inherit15-8 Lustre 1.8 Operations Manual • December 2010 2. Format the LVM volumes as Lustre targets. In this example, the backup file system is called “main” and designates the current, most up-to-date backup. cfs21:~# mkfs.lustre --mdt --fsname=main /dev/volgroup/MDT No management node specified, adding MGS to this MDT. Permanent disk data: Target: main-MDTffff Index: unassigned Lustre FS: main Mount type: ldiskfs Flags: 0x75 (MDT MGS needs_index first_time update ) Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr Parameters: checking for existing Lustre data device size = 200MB formatting backing filesystem ldiskfs on /dev/volgroup/MDT target name main-MDTffff 4k blocks 0 options -i 4096 -I 512 -q -O dir_index -F mkfs_cmd = mkfs.ext2 -j -b 4096 -L main-MDTffff -i 4096 -I 512 -q -O dir_index -F /dev/volgroup/MDT Writing CONFIGS/mountdata cfs21:~# mkfs.lustre --ost --mgsnode=cfs21 --fsname=main /dev/volgroup/OST0 Permanent disk data: Target: main-OSTffff Index: unassigned Lustre FS: main Mount type: ldiskfs Flags: 0x72 (OST needs_index first_time update ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=192.168.0.21@tcp checking for existing Lustre data device size = 200MB formatting backing filesystem ldiskfs on /dev/volgroup/OST0 target name main-OSTffff 4k blocks 0 options -I 256 -q -O dir_index -F mkfs_cmd = mkfs.ext2 -j -b 4096 -L main-OSTffff -I 256 -q -O dir_index -F /dev/ volgroup/OST0 Writing CONFIGS/mountdata cfs21:~# mount -t lustre /dev/volgroup/MDT /mnt/mdtChapter 15 Backup and Restore 15-9 cfs21:~# mount -t lustre /dev/volgroup/OST0 /mnt/ost cfs21:~# mount -t lustre cfs21:/main /mnt/main 15.5.2 Backing up New/Changed Files to the Backup File System At periodic intervals e.g., nightly, back up new and changed files to the LVM-based backup file system. cfs21:~# cp /etc/passwd /mnt/main cfs21:~# cp /etc/fstab /mnt/main cfs21:~# ls /mnt/main fstab passwd 15.5.3 Creating Snapshot Volumes Whenever you want to make a "checkpoint" of the main Lustre file system, create LVM snapshots of all target MDT and OSTs in the LVM-based backup file system. You must decide the maximum size of a snapshot ahead of time, although you can dynamically change this later. The size of a daily snapshot is dependent on the amount of data changed daily in the main Lustre file system. It is likely that a two-day old snapshot will be twice as big as a one-day old snapshot. You can create as many snapshots as you have room for in the volume group. If necessary, you can dynamically add disks to the volume group. The snapshots of the target MDT and OSTs should be taken at the same point in time. Make sure that the cronjob updating the backup file system is not running, since that is the only thing writing to the disks. Here is an example: cfs21:~# modprobe dm-snapshot cfs21:~# lvcreate -L50M -s -n MDTb1 /dev/volgroup/MDT Rounding up size to full physical extent 52.00 MB Logical volume "MDTb1" created cfs21:~# lvcreate -L50M -s -n OSTb1 /dev/volgroup/OST0 Rounding up size to full physical extent 52.00 MB Logical volume "OSTb1" created15-10 Lustre 1.8 Operations Manual • December 2010 After the snapshots are taken, you can continue to back up new/changed files to "main". The snapshots will not contain the new files. cfs21:~# cp /etc/termcap /mnt/main cfs21:~# ls /mnt/main fstab passwd termcap 15.5.4 Restoring the File System From a Snapshot Use this procedure to restore the file system from an LVM snapshot. 1. Rename the LVM snapshot. Rename the file system snapshot from "main" to "back" so you can mount it without unmounting "main". This is recommended, but not required. Use the --reformat flag to tunefs.lustre to force the name change. For example: cfs21:~# tunefs.lustre --reformat --fsname=back --writeconf /dev/volgroup/MDTb1 checking for existing Lustre data found Lustre data Reading CONFIGS/mountdata Read previous values: Target: main-MDT0000 Index: 0 Lustre FS: main Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr Parameters: Permanent disk data: Target: back-MDT0000 Index: 0 Lustre FS: back Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr Parameters: Writing CONFIGS/mountdata cfs21:~# tunefs.lustre --reformat --fsname=back --writeconf /dev/volgroup/OSTb1 checking for existing Lustre data found Lustre dataChapter 15 Backup and Restore 15-11 Reading CONFIGS/mountdata Read previous values: Target: main-OST0000 Index: 0 Lustre FS: main Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=192.168.0.21@tcp Permanent disk data: Target: back-OST0000 Index: 0 Lustre FS: back Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=192.168.0.21@tcp Writing CONFIGS/mountdata When renaming an FS, we must also erase the last_rcvd file from the snapshots cfs21:~# mount -t ldiskfs /dev/volgroup/MDTb1 /mnt/mdtback cfs21:~# rm /mnt/mdtback/last_rcvd cfs21:~# umount /mnt/mdtback cfs21:~# mount -t ldiskfs /dev/volgroup/OSTb1 /mnt/ostback cfs21:~# rm /mnt/ostback/last_rcvd cfs21:~# umount /mnt/ostback 2. Mount the file system from the LVM snapshot. For example: cfs21:~#mount-tlustre/dev/volgroup/MDTb1/mnt/mdtback cfs21:~# mount -t lustre /dev/volgroup/OSTb1 /mnt/ostback cfs21:~# mount -t lustre cfs21:/back /mnt/back 3. Note the old directory contents, as of the snapshot time. For example: cfs21:~/cfs/b1_5/lustre/utils# ls /mnt/back fstab passwds15-12 Lustre 1.8 Operations Manual • December 2010 15.5.5 Deleting Old Snapshots To reclaim disk space, you can erase old snapshots as your backup policy dictates. Run: lvremove /dev/volgroup/MDTb1 15.5.6 Changing Snapshot Volume Size You can also extend or shrink snapshot volumes if you find your daily deltas are smaller or larger than expected. Run: lvextend -L10G /dev/volgroup/MDTb1 Note – Extending snapshots seems to be broken in older LVM. It is working in LVM v2.02.01.16-1 C H A P T E R 16 POSIX This chapter describes how to install and run the POSIX compliance suite of file system tests and includes the following sections: ¦ Introduction to POSIX ¦ Installing POSIX ¦ Building and Running a POSIX Compliance Test Suite on Lustre ¦ Isolating and Debugging Failures 16.1 Introduction to POSIX Portable Operating System Interface (POSIX) is a set of standard, operating system interfaces based on the Unix OS. POSIX defines file system behavior on single UNIX node. Although used mainly with UNIX systems, the POSIX standard can apply to any operating system. POSIX specifies the user and software interfaces to the OS. Required program-level services include basic I/O (file, terminal, and network) services. POSIX also defines a standard threading library API which is supported by most modern operating systems. POSIX in a cluster means that most of the operations are atomic. Clients cannot see the metadata. POSIX offers strict mandatory locking which gives guarantee of semantics. Users do not have control on these locks. Note – Lustre is not completely POSIX-compliant, so test results may show some errors. If you have questions about test results, contact our QE and Test Team (lustre-koala-team@sun.com).16-2 Lustre 1.8 Operations Manual • December 2010 16.2 Installing POSIX Several quick start versions of the POSIX compliance suite are available to download. Each version is gcc- and architecture-specific. You need to determine which version of gcc you are running locally ({{{gcc -v}}}) and then download the appropriate tarball. If a package is not available for your particular combination of gcc+architecture, see Building and Running a POSIX Compliance Test Suite on Lustre. The following quick start versions are provided: ¦ one-step-gcc2.96-i686.tgz ¦ one-step-gcc2.96-ia64.tgz ¦ one-step-gcc3.04-i686.tgz ¦ one-step-gcc3.2-i686.tgz 16.2.1 POSIX Installation Using a Quick Start Version Use this procedure to install POSIX using a quick start version. 1. Download the POSIX scripts into /usr/src/posix. ¦ Test script: one-step-gcc-.tgz ¦ Quick start script: one-step-setup.sh Both scripts are available at: http://downloads.lustre.org/public/tools/benchmarks/posix/ 2. Launch the setup script. Run: cd /usr/src/posix sh one-step-setup.sh 3. Edit the configuration file /mnt/lustre/TESTROOT/tetexec.cfg with appropriate values for your system.Chapter 16 POSIX 16-3 4. Save the TESTROOT for running Lustre tests. Run: cd /mnt/lustre tar zcvf /usr/src/posix/TESTROOT.tgz TESTROOT Note – The quick start installation procedure only works with the paths /home/tet and /mnt/lustre. If you want to change the paths, follow the steps in Building and Running a POSIX Compliance Test Suite on Lustre and create a new tarball. 5. Launch the test suite. Run: su - vxs0 . ../profile tcc -e -a /mnt/lustre/TESTROOT -s scen.exec -p 16.3 Building and Running a POSIX Compliance Test Suite on Lustre This section describes how to build and run a POSIX compliance test suite for a compiler and architecture for which we do not provide a quick start package. 16.3.1 Building the Test Suite from Scratch This section describes building a POSIX compliance suite to test a Lustre file system. 1. Download all POSIX files in http://downloads.lustre.org/public/tools/benchmarks/posix ¦ tet_vsxgen_3.02.tgz ¦ lts_vsx-pcts2.0beta2.tgz ¦ install.sh ¦ myscen.bld ¦ myscen.exec Note – We now use the latest release of the LSB-VSX POSIX test suite (lts_vsx-pcts2.0beta2.tgz) and the generic TET/VSXgen framework (tet_vsxgen_3.02.tgz). In this release, the issue of "getgroups() did not return NGROUPS_MAX" has been fixed.16-4 Lustre 1.8 Operations Manual • December 2010 2. DO NOT configure or mount a Lustre file system yet. 3. Run the {{{install.sh}}} script and select /home/tet for the root directory for the test suite installation. Say 'y' to install the users and groups. Accept the defaults to install the packages. 4. Create a temporary directory to hold the POSIX tests while they are being built. Run: mkdir -p /mnt/lustre/TESTROOT;chown vsx0.vsxg0 !$ 5. Log in as the test user. Run: su - vsx0 6. Build the test suite. Run: ../setup.sh Most of the default answers are correct, except the root directory from which to run the testsets. For this you should specify /mnt/lustre/TESTROOT. For "Install pseudolanguages?", answer 'n'. 7. When the script prompts "Install scripts into TESTROOT/BIN..?", do not stop the script from running (this does not work). Instead, use another terminal to replace the existing files with the downloaded files. Enter: cp .../myscen.bld /home/tet/test_sets/scen.bld cp .../myscen.exec /home/tet/test_sets/scen.exec This confines the tests that are run to those relevant for file systems, avoiding hours of running other tests on sockets, math, stdio, libc, shell, etc. 8. Continue with the installation at this point. Answer 'y' to the "Build testsets" question. The script builds and installs all file system tests and then runs them all. Although the script is running the files on a local file system, this is a valuable baseline for comparison with the behavior of Lustre. The results are put into /home/tet/test_sets/results/0002e/journal. It is suggested that you rename or symlink this directory to /home/tet/test_sets/results/ext3/journal (or the name of the local file system that the test was run on). Running the full test should only take about 5 minutes.Chapter 16 POSIX 16-5 9. Answer 'n' to re-running just the failed tests. The results (in a table) are in /home/tet/test_sets/results/report. 10. Save the test suite for later use, to run additional tests on a Lustre file system. Tar up the tests to avoid rebuilding them each time. Enter: tar cvzf TESTROOT.tgz -C /mnt/lustre TESTROOT Tip – At this time, you probably want to remove the installed tests, to save a bit of space and, more importantly, to avoid confusion if you forget to mount your Lustre file system before running the tests. 16.3.2 Running the Test Suite Against Lustre 1. As root, set up your Lustre file system, mounted on /mnt/lustre (e.g., sh llmount.sh) and untar the POSIX tests back to their home. Enter: tar --same-owner -xzpvf /path/to/tarball/TESTROOT.tgz -C /mnt/lustre 2. As the vsx0 user, you can re-run the tests as many times as necessary. If you are newly su'd or logged in as the vsx0 user, you need to source the environment with '. profile' so your path and other environment is set up correctly. To run the tests, enter: . /home/tet/profile tcc -e -s scen.exec -a /mnt/lustre/TESTROOT -p Each new result is put in a new directory under /home/tet/test_sets/results and given a directory name similar to 0004e, an increasing number that ends with e for test execution or b for building the tests). 3. To look at a formatted report, enter: vrpt results/0004e/journal | less Some tests are "Unsupported", "Untested", or "Not In Use", which does not necessarily indicate a problem.16-6 Lustre 1.8 Operations Manual • December 2010 4. To compare two test results, run: vrptm results/ext3/journal results/0004e/journal | less This is more interesting than looking at the result of a single test, since it helps find test failures that are specific to the file system instead of the Linux VFS or kernel. Up to 6 test results can be compared at one time. It is often useful to rename the results directory to have more meaningful names (such as before_unlink_fix). 16.4 Isolating and Debugging Failures When failures occur, you need to gather information about what is happening at runtime. For example, some tests may cause kernel panics depending on your configuration. ¦ The POSIX compliance suite does not have debugging enabled by default, so it is useful to turn on the debugging options of VSX. Two important debug options reside in the tetexec.cfg configuration file, under the TESTROOT directory: ¦ VSX_DBUG_FILE=output_file - If you are running the test under UML with hostfs support, use a file on the hostfs as the debug output file. In the case of a crash, the debug output is then be safely written to the debug file. Note – The default value for this option puts the debug log under your test directory in /mnt/lustre/TESTROOT, which may not be useful if you experience a kernel panic and lustre (or your machine) crashes. ¦ VSX_DBUG_FLAGS=xxxxx - For detailed information about debug flags, refer to the documentation included with the POSIX test suite. The following example causes VSX to output all debug messages: VSX_DBUG_FLAGS=t:d:n:f:F:L:l,2:p:P ¦ VSX is based on the TET framework which provides common libraries for VSX. You can have TET print verbose debug messages by inserting the -T option when running the tests: tcc -Tall5 -e -s scen.exec -a /mnt/lustre/TESTROOT -p 2>&1 | tee /tmp/POSIX-command-line-output.log ¦ VSX prints detailed messages in the report for failed tests. This includes the test strategy, the kind of operations done by the test suite, and what is going wrong.Chapter 16 POSIX 16-7 Each subtest (e.g., 'access', 'create') usually contains a number of single tests. The report shows exactly which single test fails. In this case, you can find more information directly from the VSX source code. For example, if the fifth single test of subtest chmod failed, you could look at the source: /home/tet/test_sets/tset/POSIX.os/files/chmod/chmod.c ...which contains a single test array: public struct tet_testlist tet_testlist[] = { test1, 1, test2, 2, test3, 3, test4, 4, test5, 5, test6, 6, test7, 7, test8, 8, test9, 9, test10, 10, test11, 11, test12, 12, test13, 13, test14, 14, test15, 15, test16, 16, test17, 17, test18, 18, test19, 19, test20, 20, test21, 21, test22, 22, test23, 23, NULL, 0 }; If this single test is causing problems, as in the case of a kernel panic, or if you are trying to isolate a single failure, it may be useful to edit the tet_testlist array down to the single test in question and then recompile the test suite. Then, you can create a new tarball of the resulting TESTROOT directory, named appropriately (e.g, TESTROOT-chmod-5-only.tgz) and re-run the POSIX suite using the steps above. It may also be helpful to edit the scen.exec file to run only the test set in question: all "total tests in POSIX.os 1" /tset/POSIX.os/files/chmod/T.chmod16-8 Lustre 1.8 Operations Manual • December 2010 Note – Rebuilding individual POSIX tests is not straightforward due to the reliance on tcc. One option is to substitute edited source files into the source tree while following the manual installation procedure described above and let the existing POSIX install scripts do the work. The installation scripts (specifically /home/tet/test_sets/run_testsets.sh), contain relevant commands to build the test suite -- something akin to tcc -p -b -s $HOME/scen.bld $* -- but these commands may not work outside the scripts. Let us know if you get better mileage rebuilding these tests.17-1 C H A P T E R 17 Benchmarking The benchmarking process involves identifying the highest standard of excellence and performance, learning and understanding these standards, and finally adapting and applying them to improve the performance. Benchmarks are most often used to provide an idea of how fast any software or hardware runs. Complex interactions between I/O devices, caches, kernel daemons, and other OS components result in behavior that is difficult to analyze. Moreover, systems have different features and optimizations, so no single benchmark is always suitable. The variety of workloads that these systems experience also adds in to this difficulty. One of the most widely researched areas in storage subsystem is file system design, implementation, and performance. This chapter describes benchmark suites to test Lustre and includes the following sections: ¦ Bonnie++ Benchmark ¦ IOR Benchmark ¦ IOzone Benchmark17-2 Lustre 1.8 Operations Manual • December 2010 17.1 Bonnie++ Benchmark Bonnie++ is a benchmark suite that having aim of performing a number of simple tests of hard drive and file system performance. Then you can decide which test is important and decide how to compare different systems after running it. Each Bonnie++ test gives a result of the amount of work done per second and the percentage of CPU time utilized. There are two sections to the program's operations. The first is to test the I/O throughput in a fashion that is designed to simulate some types of database applications. The second is to test creation, reading, and deleting many small files in a fashion similar to the usage patterns. Bonnie++ is a benchmark tool that test hard drive and file system performance by sequential I/O and random seeks. Bonnie++ tests file system activity that has been known to cause bottlenecks in I/O-intensive applications. To install and run the Bonnie++ benchmark: 1. Download the most recent version of the Bonnie++ software: http://www.coker.com.au/bonnie++/ 2. Install and run the Bonnie++ software (per the ReadMe file accompanying the software). Sample output: Version 1.03 --Sequential Output-- --Sequential Input- --Random-- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP mds 2G 3811822 21245 10 51967 10 90.00 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 510 0 +++++ +++ 283 1 465 0 +++++ +++ 291 1 mds,2G,,,38118,22,21245,10,,,51967,10,90.0,0,16,510,0,+++++,+++,28 3,1,465,0,+++++,+++,291,1Chapter 17 Benchmarking 17-3 Version 1.03 --Sequential Output-- --Sequential Input- --Random-- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP mds 2G 27460 92 41450 25 21474 10 19673 60 52871 10 88.0 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 29681 99 +++++ +++ 30412 90 29568 99 +++++ +++ 28077 82 mds,2G,27460,92,41450,25,21474,10,19673,60,52871,10,88.0,0,16,2968 1,99,+++++,+++,30412,90,29568,99,+++++,+++,28077,82 17.2 IOR Benchmark The IOR_survey script tests the performance of the Lustre file system. It uses IOR (Interleaved or Random), a script used for testing performance of parallel file systems using various interfaces and access patterns. IOR uses MPI for process synchronization. Under the control of compile-time defined constants (and, to a lesser extent, environment variables), I/O is done via MPI-IO. The data are written and read using independent parallel transfers of equal-sized blocks of contiguous bytes that cover the file with no gaps and that do not overlap each other. The test consists of creating a new file, writing it with data, then reading the data back. The IOR benchmark, developed by LLNL, tests system performance by focusing on parallel/sequential read/write operations that are typical of scientific applications. To install and run the IOR benchmark: 1. Satisfy the prerequisites to run IOR. a. Download lam 7.0.6 (local area multi-computer): http://www.lam-mpi.org/7.0/download.php b. Obtain a Fortran compiler for the Fedora Core 4 operating system. c. Download the most recent version of the IOR software: http://sourceforge.net/projects/ior-sio17-4 Lustre 1.8 Operations Manual • December 2010 2. Install the IOR software (per the ReadMe file and User Guide accompanying the software). 3. Run the IOR software. In user mode, use the lamboot command to start the lam service and use appropriate Lustre-specific commands to run IOR (described in the IOR User Guide). Sample Output: IOR-2.9.0: MPI Coordinated Test of Parallel I/O Run began: Fri Sep 29 11:43:56 2006 Command line used: ./IOR -w -r -k -O lustrestripecount 10 –o test Machine: Linux mds Summary: api = POSIX test filename = test access = single-shared-file clients = 1 (1 per node) repetitions = 1 xfersize = 262144 bytes blocksize = 1 MiB aggregate filesize= 1 MiB access bw(MiB/s) block(KiB)xfer(KiB) open(s)wr/rd(s)close(s)iter ------ --------- --------- -------- -------------------------- write 173.89 1024.00 256.00 0.0000300.0057010.0000160 read 278.49 1024.00 256.00 0.0000090.0035660.0000120 Max Write: 173.89 MiB/sec (182.33 MB/sec) Max Read: 278.49 MiB/sec (292.02 MB/sec) Run finished: Fri Sep 29 11:43:56 2006Chapter 17 Benchmarking 17-5 17.3 IOzone Benchmark IOZone is a file system benchmark tool which generates and measures a variety of file operations. Iozone has been ported to many machines and runs under many operating systems. Iozone is useful to perform a broad file system analysis of a vendor’s computer platform. The benchmark tests file I/O performance for the operations like read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read/write, pread/pwrite variants, aio_read, aio_write, mm, etc. The IOzone benchmark tests file I/O performance for the following operations: read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read/write, pread/pwrite variants, aio_read, aio_write, and mmap. To install and run the IOzone benchmark: 1. Download the most recent version of the IOZone software from this location: http://www.iozone.org 2. Install the IOZone software (per the ReadMe file accompanying the IOZone software).17-6 Lustre 1.8 Operations Manual • December 2010 3. Run the IOZone software (per the ReadMe file accompanied with the IOZone software). Sample Output Iozone: Performance Test of File I/O Version $Revision: 3.263 $ Compiled for 32 bit mode. Build: linux Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins, Al Slater, Scott Rhine, Mike Wisner, Ken Goss, Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Erik Habbinga, Kris Strecker, Walter Wong. Run began: Fri Sep 29 15:37:07 2006 Network distribution mode enabled. Command line used: ./iozone -+m test.txt Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 512 4 194309 406651 728276 792701 715002 498592 638351 700365 587235 190554 378448 686267 765201 iozone test complete.18-1 C H A P T E R 18 Lustre I/O Kit This chapter describes the Lustre I/O kit and PIOS performance tool, and includes the following sections: ¦ Lustre I/O Kit Description and Prerequisites ¦ Running I/O Kit Tests ¦ PIOS Test Tool ¦ LNET Self-Test 18.1 Lustre I/O Kit Description and Prerequisites The Lustre I/O kit is a collection of benchmark tools for a Lustre cluster. The I/O kit can be used to validate the performance of the various hardware and software layers in the cluster and also as a way to find and troubleshoot I/O issues. The I/O kit contains three tests. The first surveys basic performance of the device and bypasses the kernel block device layers, buffer cache and file system. The subsequent tests survey progressively higher layers of the Lustre stack. Typically with these tests, Lustre should deliver 85-90% of the raw device performance. It is very important to establish performance from the “bottom up” perspective. First, the performance of a single raw device should be verified. Once this is complete, verify that performance is stable within a larger number of devices. Frequently, while troubleshooting such performance issues, we find that array performance with all LUNs loaded does not always match the performance of a single LUN when tested in isolation. After the raw performance has been established, other software layers can be added and tested in an incremental manner.18-2 Lustre 1.8 Operations Manual • December 2010 18.1.1 Downloading an I/O Kit You can download the I/O kit from: http://downloads.lustre.org/public/tools/lustre-iokit/ In this directory, you will find two packages: ¦ lustre-iokit consists of a set of developed and supported by the Lustre group. ¦ scali-lustre-iokit is a Python tool maintained by Scali team, and is not discussed in this manual. 18.1.2 Prerequisites to Using an I/O Kit The following prerequisites must be met to use the Lustre I/O kit: ¦ password-free remote access to nodes in the system (normally obtained via ssh or rsh) ¦ Lustre file system software ¦ sg3_utils for the sgp_dd utility 18.2 Running I/O Kit Tests As mentioned above, the I/O kit contains these test tools: ¦ sgpdd_survey ¦ obdfilter_survey ¦ ost_survey ¦ stats-collectChapter 18 Lustre I/O Kit 18-3 18.2.1 sgpdd_survey Use the sgpdd_survey tool to test bare metal performance, while bypassing as much of the kernel as possible. This script requires the sgp_dd package, although it does not require Lustre software. This survey may be used to characterize the performance of a SCSI device by simulating an OST serving multiple stripe files. The data gathered by this survey can help set expectations for the performance of a Lustre OST exporting the device. The script uses sgp_dd to carry out raw sequential disk I/O. It runs with variable numbers of sgp_dd threads to show how performance varies with different request queue depths. The script spawns variable numbers of sgp_dd instances, each reading or writing a separate area of the disk to demonstrate performance variance within a number of concurrent stripe files. The device(s) used must meet one of the two tests described below: SCSI device: Must appear in the output of sg_map (make sure the kernel module "sg" is loaded) Raw device: Must appear in the output of raw -qa If you need to create raw devices in order to use the sgpdd_survey tool, note that raw device 0 cannot be used due to a bug in certain versions of the "raw" utility (including that shipped with RHEL4U4.) You may not mix raw and SCSI devices in the test specification. Caution – The sgpdd_survey script overwrites the device being tested, which results in the LOSS OF ALL DATA on that device. Exercise caution when selecting the device to be tested.18-4 Lustre 1.8 Operations Manual • December 2010 The sgpdd_survey script must be customized according to the particular device being tested and also according to the location where it should keep its working files. Customization variables are described explicitly at the start of the script. When the sgpdd_survey script runs, it creates a number of working files and a pair of result files. All files start with the prefix given by the script variable ${rslt}. ${rslt}_.summary same as stdout ${rslt}__* tmp files ${rslt}_.detail collected tmp files for post-mortem The summary file and stdout should contain lines like this: total_size 8388608K rsz 1024 thr 1 crg 1 180.45 MB/s 1 x 180.50 \ =/ 180.50 MB/s The number immediately before the first MB/s is bandwidth, computed by measuring total data and elapsed time. The remaining numbers are a check on the bandwidths reported by the individual sgp_dd instances. If there are so many threads that the sgp_dd script is unlikely to be able to allocate I/O buffers, then "ENOMEM" is printed. If one or more sgp_dd instances do not successfully report a bandwidth number, then "failed" is printed. 18.2.1.1 Tuning sgpdd_survey To get large I/O (1 MB) to disk, it may be necessary to tune several sgpdd_survey parameters as specified: /sys/block/sdN/queue/max_sectors_kb = 4096 /sys/block/sdN/queue/max_phys_segments = 256 /proc/scsi/sg/allow_dio = 1 /sys/module/ib_srp/parameters/srp_sg_tablesize = 255Chapter 18 Lustre I/O Kit 18-5 18.2.2 obdfilter_survey The obdfilter_survey script processes sequential I/O with varying numbers of threads and objects (files) by using lctl to drive the echo_client connected to local or remote obdfilter instances or remote obdecho instances. It can be used to characterize the performance of the following Lustre components: OSTs The script exercises one or more instances of obdfilter directly. The script may run on one or more nodes, for example, when the nodes are all attached to the same multi-ported disk subsystem. Tell the script the names of all obdfilter instances (which should be up and running already). If some instances are on different nodes, specify their hostnames too (for example, node1:ost1). Alternately, you can pass parameter case=disk to the script. (The script automatically detects the local obdfilter instances.) All obdfilter instances are driven directly. The script automatically loads the obdecho module (if required) and creates one instance of echo_client for each obdfilter instance. Network The script drives one or more instances of the obdecho server via instances of echo_client running on one or more nodes. Pass the parameters case=network and target='''' to the script. For each nework case, the script does the required setup. Striped File System Over the Network The script drives one or more instances of obdfilter via instances of echo_client running on one or more nodes. Tell the script the names of the OSCs (which should be up and running). Alternately, you can pass the parameter case=netdisk to the script. The script will use all of the local OSCs. Note – The obdfilter_survey script is NOT scalable to hundreds of nodes since it is only intended to measure individual servers, not the scalability of the entire system.18-6 Lustre 1.8 Operations Manual • December 2010 Note – The obdfilter_survey script must be customized, depending on the components under test and where the script’s working files should be kept. Customization variables are clearly described in the script (Customization Variables section). In particular, refer to the maximum supported value ranges for customization variables. 18.2.2.1 Running obdfilter_survey Against a Local Disk The obdfilter_survey script supports automatic and manual runs against a local disk. Obdfilter-survey profiles the overall throughput of storage hardware 1 , by sending ranges of workloads to the OSTs (varied in thread counts and I/O sizes). When the obdfilter_survey script is complete, it provides information on the performance abilities of the storage hardware and shows the saturation points. If you use plot scripts on the data, this information is shown graphically. To run the obdfilter_survey script, create a normal Lustre configuration; no special setup is needed. To perform an automatic run: 1. Set up the Lustre file system. 2. Verify that the obdecho.ko module is present. 3. Run the obdfilter_survey script with the parameter case=disk. For example: $ nobjhi=2 thrhi=2 size=1024 case=disk sh obdfilter-survey To perform a manual run: 1. List all OSTs you want to test. (You do not have to specify an MDS or LOV.) 2. On all OSSs, run: $ mkfs.lustre --fsname spfs --mdt --mgs /dev/sda Caution – Write tests are destructive. This test should be run before the Lustre file system is started. If you do this, you will not need to reformat to restart Lustre system. However, if the obdfilter_survey test is terminated before it completes, you may have to remove objects from the disk. 1. The sgpdd-survey profiles individual disks. This script is destructive, and should not be run anywhere you want to preserve existing data.Chapter 18 Lustre I/O Kit 18-7 3. Determine the obdfilter instance names on all Lustre clients. The device names appear in the fourth column of the lctl dl command output. For example: $ pdsh -w oss[01-02] lctl dl |grep obdfilter |sort oss01: 0 UP obdfilter oss01-sdb oss01-sdb_UUID 3 oss01: 2 UP obdfilter oss01-sdd oss01-sdd_UUID 3 oss02: 0 UP obdfilter oss02-sdi oss02-sdi_UUID 3 ... In this example, the obdfilter instance names are oss01-sdb, oss01-sdd, and oss02-sdi. Since you are driving obdfilter instances directly, set the shell array variable, targets, to the names of the obdfilter instances. For example: targets='oss01:oss01-sdb oss01:oss01-sdd oss02:oss02-sdi'\ ./obdfilter-survey 18.2.2.2 Running obdfilter_survey Against a Network The obdfilter_survey script can only be run automatically against a network; no manual test is supported. To run the network test, a specific Lustre setup is needed. Make sure that these configuration requirements have been met. ¦ Install all Lustre modules, including obdecho. ¦ Start lctl and check the device list, which must be empty. ¦ Use a password-less entry between the client and server machines, to avoid having to type the password. To perform an automatic run: 1. Run the obdfilter_survey script with the parameters case=netdisk and targets= ''''. For example: $ nobjhi=2 thrhi=2 size=1024 targets="" \ case=network sh obdfilter-survey On the server side, you can see the statistics at: /proc/fs/lustre/obdecho//stats where 'echo_srv' is the obdecho server created by the script.18-8 Lustre 1.8 Operations Manual • December 2010 18.2.2.3 Running obdfilter_survey Against a Network Disk The obdfilter_survey script can be run automatically or manually against a network disk. To run the network disk test, create a Lustre configuration using normal methods; no special setup is needed. To perform an automatic run: 1. Set up the Lustre file system with the required OSTs. 2. Verify that the obdecho.ko module is present. 3. Run the obdfilter_survey script with the parameter case=netdisk. For example: $ nobjhi=2 thrhi=2 size=1024 case=netdisk sh obdfilter-survey To perform a manual run: 1. Run the obdfilter_survey script and tell the script the names of all echo_client instances (which should be up and running already). $ nobjhi=2 thrhi=2 size=1024 targets=" ..." \ sh obdfilter-surveyChapter 18 Lustre I/O Kit 18-9 18.2.2.4 Output Files When the obdfilter_survey script runs, it creates a number of working files and a pair of result files. All files start with the prefix given by ${rslt}. The obdfilter_survey script iterates over the given number of threads and objects performing the specified tests and checks that all test processes have completed successfully. Note – The obdfilter_survey script may not clean up properly if it is aborted or if it encounters an unrecoverable error. In this case, a manual cleanup may be required, possibly including killing any running instances of 'lctl' (local or remote), removing echo_client instances created by the script and unloading obdecho. File Description ${rslt}.summary Same as stdout ${rslt}.script_* Per-host test script files ${rslt}.detail_tmp* Per-OST result files ${rslt}.detail Collected result files for post-mortem18-10 Lustre 1.8 Operations Manual • December 2010 18.2.2.5 Script Output The summary file and stdout of the obdfilter_survey script contain lines such as: ost 8 sz 67108864K rsz 1024 obj 8 thr 8 write 613.54 [ 64.00, 82.00] Where: Note – Although the numbers of threads and objects are specified per-OST in the customization section of the script, the reported results are aggregated over all OSTs. 18.2.2.6 Visualizing Results It is useful to import the obdfilter_survey script summary data (it is fixed width) into Excel (or any graphing package) and graph the bandwidth versus the number of threads for varying numbers of concurrent regions. This shows how the OSS performs for a given number of concurrently-accessed objects (files) with varying numbers of I/Os in flight. It is also extremely useful to record average disk I/O sizes during each test. These numbers help locate pathologies in the system when the file system block allocator and the block device elevator. The plot-obdfilter script (included) is an example of processing output files to a .csv format and plotting a graph using gnuplot. Variable Supported Type ost8 Total number of OSTs being tested. sz 67108864K Total amount of data read or written (in KB). rsz 1024 Record size (size of each echo_client I/O, in KB). obj 8 Total number of objects over all OSTs. thr 8 Total number of threads over all OSTs and objects. write Test name. If more tests have been specified, they all appear on the same line. 613.54 Aggregate bandwidth over all OSTs (measured by dividing the total number of MB by the elapsed time). [64, 82.00] Minimum and maximum instantaneous bandwidths on an individual OST.Chapter 18 Lustre I/O Kit 18-11 18.2.3 ost_survey The ost_survey tool is a shell script that uses lfs setstripe to perform I/O against a single OST. The script writes a file (currently using dd) to each OST in the Lustre file system, and compares read and write speeds. The ost_survey tool is used to detect misbehaving disk subsystems. Note – We have frequently discovered wide performance variations across all LUNs in a cluster. To run the ost_survey script, supply a file size (in KB) and the Lustre mount point. For example, run: $ ./ost-survey.sh 10 /mnt/lustre Average read Speed: 6.73 Average write Speed: 5.41 read - Worst OST indx 0 5.84 MB/s write - Worst OST indx 0 3.77 MB/s read - Best OST indx 1 7.38 MB/s write - Best OST indx 1 6.31 MB/s 3 OST devices found Ost index 0 Read speed 5.84 Write speed 3.77 Ost index 0 Read time 0.17 Write time 0.27 Ost index 1 Read speed 7.38 Write speed 6.31 Ost index 1 Read time 0.14 Write time 0.16 Ost index 2 Read speed 6.98 Write speed 6.16 Ost index 2 Read time 0.14 Write time 0.1618-12 Lustre 1.8 Operations Manual • December 2010 18.2.4 stats-collect The stats-collect utility contains the following scripts used to collect application profiling information from Lustre clients and servers: ¦ lstat.sh - script for a single node that is run on each profile node ¦ gather_stats_everywhere.sh - script that collect statistics ¦ config.sh - script that contains customized configuration descriptions The stats-collect utility requires: ¦ Lustre to be installed and set up on your cluster ¦ SSH and SCP access to these nodes without requiring a password Configuring stats-collect Configuring the stats-collect utility is simple - all of the profiling configuration VARs are in the config.sh script. XXXX_INTERVAL is the profiling interval where the value of interval means: ¦ 0 - gather statistics at start and stop only ¦ N - gather statistics every N seconds If XXX_INTERVAL is not specified, then XXX statistics are not collected. XXX can be VMSTAT, SERVICE, BRW, SDIO, MBALLOC, IO, JBD, CLIENT Running stats-collect The gather_stats_everywhere.sh script should be run in three phases: ¦ sh gather_stats_everywhere.sh config.sh start Starts statistics collection on each node specified in the config.sh script. ¦ sh gather_stats_everywhere.sh config.sh stop Stops collecting statistics on each node. If is provided, it creates a profile tarball /tmp/. ¦ sh gather_stats_everywhere.sh config.sh analyse log_tarball.tgz csv Analyzes the log_tarball and create a csv tarball for this profiling tarball.Chapter 18 Lustre I/O Kit 18-13 Examples To collect profile information: 1. Start the collect profile daemon on each node. sh gather_stats_everywhere.sh config.sh start 2. Run your test. 3. Stop the collect profile daemon on each node, clean up the temporary file and create a profiling tarball. sh gather_stats_everywhere.sh config.sh stop log_tarball.tgz 4. Create a csv file according to the profile. sh gather_stats_everywhere.sh config.sh analyse log_tarball.tgz csv18-14 Lustre 1.8 Operations Manual • December 2010 18.3 PIOS Test Tool The PIOS test tool is a parallel I/O simulator for Linux and Solaris. PIOS generates I/O on file systems, block devices and zpools similar to what can be expected from a large Lustre OSS server when handling the load from many clients. The program generates and executes the I/O load in a manner substantially similar to an OSS, that is, multiple threads take work items from a simulated request queue. It forks a CPU load generator to simulate running on a system with additional load. PIOS can read/write data to a single shared file or multiple files (default is a single file). To specify multiple files, use the --fpp option. (It is better to measure with both single and multiple files.) If the final argument is a file, block device or zpool, PIOS writes to RegionCount regions in one file. PIOS issues I/O commands of size ChunkSize. The regions are spaced apart Offset bytes (or, in the case of many files, the region starts at Offset bytes). In each region, RegionSize bytes are written or read, one ChunkSize I/O at a time. Note that: ChunkSize <= Regionsize <= Offset Multiple runs can be specified with comma separated lists of values for ChunkSize, Offset, RegionCount, ThreadCount, and RegionSize. Multiple runs can also be specified by giving a starting (low) value, increase (in percent) and high value for each of these arguments. If a low value is given, no value list or value may be supplied. Every run is given a timestamp, and the timestamp and offset are written with every chunk (to allow verification). Before every run, PIOS executes the pre-run shell command. After every run, PIOS executes the post-run command. Typically, this is used to clear and collect statistics for the run, or to start and stop statistics gathering during the run. The timestamp is passed to both pre-run and post-run. For convenience, PIOS understands byte specifiers and uses: K,k for kilobytes (2<<10) M,m for megabytes (2<<20) G,g for gigabytes (2<<30) T,t for terabytes (2<<40) Download the PIOS test tool at: http://downloads.lustre.org/public/tools/benchmarks/pios/Chapter 18 Lustre I/O Kit 18-15 18.3.1 Synopsis pios [--chunksize|-c =values, (--chunksize_low|-a =value --chunksize_high|-b =value --chunksize_incr|-g =value)] [--offset|-o =values, (--offset_low|-m =value --offset_high|-q =value --offset_incr|-r =value)] [--regioncount|-n =values, (--regioncount_low|-i =value --regioncount_high|-j =value --regioncount_incr|-k =value)] [--threadcount|-t =values, (--threadcount_low|-l =value --threadcount_high|-h =value --threadcount_incr|-e =value)] [--regionsize|-s =values, (--regionsize_low|-A =value --regionsize_high|-B =value --regionsize_incr|-C =value)] [--directio|-d, --posixio|-x, --cowio|-w} [--cleanup|-L --threaddelay|-T =ms --regionnoise|-I ==shift --chunknoise|-N =bytes -fpp|-F ] [--verify|-V =values] [--prerun|-P =pre-command --postrun|-R =post-command] [--path|-p =output-file-path]18-16 Lustre 1.8 Operations Manual • December 2010 18.3.2 PIOS I/O Modes There are several supported PIOS I/O modes: POSIX I/O: This is the default operational mode where I/O is done using standard POSIX calls, such as pwrite/pread. This mode is valid on both Linux and Solaris. DIRECT I/O: This mode corresponds to the O_DIRECT flag in open(2) system call, and it is currently applicable only to Linux. Use this mode when using PIOS on the ldiskfs file system on an OSS. COW I/O: This mode corresponds to the copy overwrite operation where file system blocks that are being overwritten were copied to shadow files. Only use this mode if you want to see overhead of preserving existing data (in case of overwrite). This mode is valid on both Linux and Solaris.Chapter 18 Lustre I/O Kit 18-17 18.3.3 PIOS Parameters PIOS has five basic parameters to determine the amount of data that is being written. ChunkSize(c): Amount of data that a thread writes in one attempt. ChunkSize should be a multiple of file system block size. RegionSize(s): Amount of data required to fill up a region. PIOS writes a chunksize of data continuously until it fills the regionsize. RegionSize should be a multiple of ChunkSize. RegionCount(n): Number of regions to write in one or multiple files. The total amount of data written by PIOS is RegionSize x RegionCount. ThreadCount(t): Number of threads working on regions.18-18 Lustre 1.8 Operations Manual • December 2010 Offset(o): Distance between two successive regions when all threads are writing to the same file. In the case of multiple files, threads start writing in files at Offset bytes. Parameter Description --chunknoise = N N is a byte specifier. When performing an I/O task, add a random signed integer in the range [-N,N] to the chunksize. All regions are still fully written. This randomizes the I/O size to some extent. --chunksize = N[,N2,N3...] N is a byte specifier and performs I/O in chunks of N kilo-, mega-, giga- or terabyte. You can give a comma separated list of multiple values. This argument is mutually exclusive with --chunksize_low. Note that each thread allocates a buffer of size chunksize + chunknoise for use during the run. --chunksize_low=L --chunksize_high=H --chunksize_incr=F Performs a sequence of operations starting with a chunksize of L, increasing it by F% each time until chunksize exceeds H. --cleanup Removes files that were created during the run. If there is an encounter for existing files, they are over-written. --directio --posixio --cowio One of these arguments must be passed to indicate if DIRECT I/O, POSIX I/O or COW I/O is used. --offset=O[,O2,O3...] The argument is a byte specifier or a list of specifiers. Each run uses regions at offset multiple of O in a single file. If the run targets multiple files, then the I/O writes at offset O in each file. --offset_low=OL --offset_high=OH --offset_inc=PH The arguments are byte specifiers. They generate runs with a range of offsets starting at OL, increasing P% until the region size exceeds OH. Each of these arguments is exclusive with the offset argument. --prerun=”pre-command” Before each run, executes the pre-command as a shell command through the system(3) call. The timestamp of the run is appended as the last argument to the pre-command string. Typically, this is used to clear statistics or start a data collection script when the run starts. --postrun=”post-command” After each run, executes the post-command as a shell command through the system(3) call. The timestamp of the run is appended as the last argument to the pre-command string. Typically, this is used to append statistics for the run or close an open data collection script when the run completes.Chapter 18 Lustre I/O Kit 18-19 --regioncount=N[,N2,N3...] PIOS writes to N regions in a single file or block device or to N files. --regioncount_low=RL --regioncount_high=RH --regioncount_inc=P Generate runs with a range of region counts starting at TL, increasing P% until the thread count exceeds RH. Each of these arguments is exclusive with the regioncount argument. --regionnoise=k When generating the next I/O task, do not select the next chunk in the next stream, but shift a random number with a maximum noise of shifting k regions ahead. The run will complete when all regions are fully written or read. This merely introduces a randomization of the ordering. --regionsize=S[,S2,S3...] The argument is a byte specifier or a list of byte specifiers. During the run(s), write S bytes to each region. --regionsize_low=RL --regionsize_high=RH --regionsize_inc=P The arguments are byte specifiers. Generate runs with a range of region sizes starting at TL, increasing P% until the region size exceeds RH. Each argument is exclusive with the regionsize argument. --threadcount=T[,T2,T3...] PIOS runs with T threads performing I/O. A sequence of values may be given. --threadcount_low=TL --threadcount_high=TH --threadcount_inc=TP Generate runs with a range of thread counts starting at TL, increasing TP% until the thread count exceeds TH. Each of these arguments is exclusive with the threadcount argument. --threaddelay=ms A random amount of noise not exceeding ms is inserted between the time that a thread identifies as the next chunk it needs to read or write and the time it starts the I/O. --fpp Where threads write to files: • fpp indicates files per process behavior where threads write to multiple files. • sff indicates single shared files where all threads write to the same file. --verify-V=timestamp [,timestamp2,timestamp3]|- -verify|-V Verify a written file or set of files. A single timestamp or sequence of timestamps can be given for each run, respectively. If no argument is passed, the verification is done from timestamps read from the first location of files previously written in the test. If sequence is given, then each run verifies the timestamp accordingly. If a single timestamp is given, then it is verified with all files written. Parameter Description18-20 Lustre 1.8 Operations Manual • December 2010 18.3.4 PIOS Examples To create a 1 GB load with a different number of threads: In one file: pios -t 1,2,4, 8,16, 32, 64, 128 -n 128 -c 1M -s 8M -o 8M \ --load=posixio -p /mnt/lustre In multiple files: pios -t 1,2,4, 8,16, 32, 64, 128 -n 128 -c 1M -s 8M -o 8M \ --load=posixio,fpp -p /mnt/lustre To create a 1 GB load with a different number of chunksizes on ldiskfs with direct I/O: In one file: pios -t 32 -n 128 -c 128K, 256K, 512K, 1M, 2M, 4M -s 8M -o 8M \ --load=directio -p /mnt/lustre In multiple files: pios -t 32 -n 128 -c 128K, 256K, 512K, 1M, 2M, 4M -s 8M -o 8M \ --load=directio,fpp -p /mnt/lustre To create a 32 MB to 128 MB load with different RegionSizes on a Solaris zpool: In one file: pios -t 8 -n 16 -c 1M -A 2M -B 8M -C 100 -o 8M --load=posixio -p \ /myzpool/ In multiple files: pios -t 8 -n 16 -c 1M -A 2M -B 8M -C 100 -o 8M --load=posixio, \ fpp -p /myzpool/ To read and verify timestamps: Create a load with PIOS: pios -t 40 -n 1024 -c 256K -s 4M -o 8M --load=posixio -p /mnt/lustre Keep the same parameters to read: pios -t 40 -n 1024 -c 256K -s 4M -o 8M --load=posixio -p \ /mnt/lustre --verifyChapter 18 Lustre I/O Kit 18-21 18.4 LNET Self-Test LNET self-test helps site administrators confirm that Lustre Networking (LNET) has been properly installed and configured, and that underlying network software and hardware are performing according to expectations. LNET self-test is a kernel module that runs over LNET and LNDs. It is designed to: ¦ Test the connection ability of the Lustre network ¦ Run regression tests of the Lustre network ¦ Test performance of the Lustre network Note – Apart from the performance impact, LNET self-test is invisible to Lustre. 18.4.1 Basic Concepts of LNET Self-Test This section describes basic concepts of LNET self-test, utilities and a sample script. 18.4.1.1 Modules To run LNET self-test, these modules must be loaded: libcfs, lnet, lnet_selftest and one of the klnds (i.e, ksocklnd, ko2iblnd...). To load all necessary modules, run modprobe lnet_selftest (recursively loads the modules on which LNET self-test depends. The LNET self-test cluster has two types of nodes: ¦ Console node - A single node that controls and monitors the test cluster. It can be any node in the test cluster. ¦ Test nodes - The nodes that run tests. Test nodes are controlled by the user via the console node; the user does not need to log into them directly. The console and test nodes require all previously-listed modules to be loaded. (The userspace test node does not require these modules.)18-22 Lustre 1.8 Operations Manual • December 2010 Note – Test nodes can be in either kernel or userspace. A console user can invite a kernel test node to join the test session by running lst add_group NID, but the user cannot actively add a userspace test node to the test-session. However, the console user can passively accept a test node to the test session while the test node runs lstclient to connect to the console. 18.4.1.2 Utilities LNET self-test has two user utilities, lst and lstclient. ¦ lst - The user interface for the self-test console (run on the console node). It provides a list of commands to control the entire test system, such as create session, create test groups, etc. ¦ lstclient - The userspace LNET self-test program (run on a test node). lstclient is linked with userspace LNDs and LNET. lstclient is not needed if a user just wants to use kernel space LNET and LNDs. 18.4.1.3 Session In the context of LNET self-test, a session is a test node that can be associated with only one session at a time, to ensure that the session has exclusive use. Almost all operations should be performed in a session context. From the console node, a user can only operate nodes in his own session. If a session ends, the session context in all test nodes is destroyed. The console node can be used to create, change or destroy a session (new_session, end_session, show_session). For more information, see Session. 18.4.1.4 Console The console node is the user interface of the LNET self-test system, and can be any node in the test cluster. All self-test commands are entered from the console node. From the console node, a user can control and monitor the status of the entire test cluster (session). The console node is exclusive, meaning that a user cannot control two different sessions (LNET self-test clusters) on one node.Chapter 18 Lustre I/O Kit 18-23 18.4.1.5 Group An LNET self-test group is just a named collection of nodes. There are no restrictions on group membership, i.e., a node can be included in any number of groups, and any number of groups can exist in a single LNET self-test session. Each node in a group has a rank, determined by the order in which it was added to the group, which is used to establish test traffic patterns. A user can only control nodes in his/her session. To allocate nodes to the session, the user needs to add nodes to a group (of the session). All nodes in a group can be referenced by group's name. A node can be allocated to multiple groups of a session. Note – A console user can associate kernel space test nodes with the session by running lst add_group NIDs, but a userspace test node cannot be actively added to the session. However, the console user can passively "accept" a test node to associate with test session while the test node running lstclient connects to the console node, i.e: lstclient --sesid CONSOLE_NID --group NAME). 18.4.1.6 Test A test generates network load between two arbitrary groups of nodes - the test's "from" and "to" groups. When a test is running, each node in the "from" group sends requests to nodes in the "to" group, and receive responses in return. This activity is designed to mimic Lustre RPC traffic, i.e. the "from" group acts like a set of clients and the "to" group acts like a set of servers. The traffic pattern and test intensity is determined several properties, including test type, distribution of test nodes, concurrency of test, RDMA operation type, etc. Several of the available test parameters are described below. ¦ Type: The test type determines the message pattern for a single request/response. Supported types are: ¦ Ping: Small request / small response. Pings only generate small messages. They are useful to determine latency and small message overhead, and to simulate Lustre metadata traffic. ¦ brw: Small request / bulk / small response. Brws include an additional phase where bulk data is either fetched from the request sender (brw write) or sent back to it (brw read) before the response is returned. The size of the bulk transfer is a test parameter. Brw tests are useful to determine network bandwidth and to simulate Lustre I/O traffic.18-24 Lustre 1.8 Operations Manual • December 2010 ¦ Distribution: Determines which nodes in the "to" group communicate with each node in the "from" group. It allows you to specify a wide range of topologies, including one-to-one and all-to-all. Distribution divides the "from" group into subsets, which are paired with equivalent subsets from the "to" group so only nodes in matching subsets communicate. For example: --distribute 1:1 This is the default setting. Each "from" node communicates with the same rank (modules "to" group size) "to" node. Note that if there are more "from" nodes than "to" nodes, some "from" nodes may share the same "to" nodes. Also, if there are more "to" nodes than "from" nodes, some higher-ranked "to" nodes will be idle. --distribute 1:n (where 'n' is the size of the "to" group). Each "from" node communicates with every node in the "to" group. ¦ Concurrency: Determines how many requests each "from" node in a test keeps on the wire. 18.4.1.7 Batch A batch is an arbitrary collection of tests which are started and stopped together; they run in parallel. Each test should belong to a batch; tests should not exist individually. Users can control a test batch (run, stop); they cannot control individual tests. Tests in a batch are non-destructive to the file system, and can be run in a normal Lustre environment (provided the performance impact is acceptable). The simplest batch might contain only a single test - running brw to determine whether network bandwidth will be an I/O bottleneck. In this example, the "to" group is comprised of Lustre OSSes and the "from" group includes the compute nodes. Adding an second test to perform pings from a login node to the MDS could tell you how much checkpointing would affect the ls -l process.Chapter 18 Lustre I/O Kit 18-25 18.4.1.8 Sample Script These are the steps to run a sample LNET self-test script simulating the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an InfiniBand network (connected via LNET routers). In this example, half the clients are reading and half the clients are writing. 1. Load libcfs.ko, lnet.ko, ksocklnd.ko and lnet_selftest.ko on all test nodes and the console node. 2. Run this script on the console node: #!/bin/bash export LST_SESSION=$$ lst new_session read/write lst add_group servers 192.168.10.[8,10,12-16]@tcp lst add_group readers 192.168.1.[1-253/2]@o2ib lst add_group writers 192.168.1.[2-254/2]@o2ib lst add_batch bulk_rw lst add_test --batch bulk_rw --from readers --to servers \ brw read check=simple size=1M lst add_test --batch bulk_rw --from writers --to servers \ brw write check=full size=4K # start running lst run bulk_rw # display server stats for 30 seconds lst stat servers & sleep 30; kill $! # tear down lst end_session Note – This script can be easily adapted to pass the group NIDs by shell variables or command line arguments (making it good for general-purpose use).18-26 Lustre 1.8 Operations Manual • December 2010 18.4.2 LNET Self-Test Commands The LNET self-test (lst) utility is used to issue LNET self-test commands. The lst utility takes a number of command line arguments. The first argument is the command name and subsequent arguments are command-specific. 18.4.2.1 Session This section lists lst session commands. Process Environment (LST_SESSION) The lst utility uses the LST_SESSION environmental variable to identify the session locally on the self-test console node. This should be a numeric value that uniquely identifies all session processes on the node. It is convenient to set this to the process ID of the shell both for interactive use and in shell scripts. Almost all lst commands require LST_SESSION to be set. new_session [--timeout SECONDS] [--force] NAME Creates a new session. –-timeout SECONDS Console timeout value of the session. The session ends automatically if it remains idle (i.e., no commands are issued) for this period. --force Ends conflicting sessions. This determines who “wins” when one session conflicts with another. For example, if there is already an active session on this node, then this attempt to create a new session fails unless the -force flag is specified. However, if the -force flag is specified, then the other session is ended. Similarly, if this session attempts to add a node that is already “owned” by another session, the -force flag allows this session to “steal” the node. name A human-readable string to print when listing sessions or reporting session conflicts. $ export LST_SESSION=$$ $ lst new_session --force liangzhenChapter 18 Lustre I/O Kit 18-27 end_session Stops all operations and tests in the current session and clears the session’s status. $ lst end_session show_session Shows the session information. This command prints information about the current session. It does not require LST_SESSION to be defined in the process environment. $ lst show_session 18.4.2.2 Group This section lists lst group commands. add_group NAME NIDs [NIDs...] Creates the group and adds a list of test nodes to the group. update_group NAME [--refresh] [--clean STATE] [--remove NIDs] Updates the state of nodes in a group or adjusts a group’s membership. This command is useful if some nodes have crashed and should be excluded from the group. NAME Name of the group. NIDs A string that may be expanded into one or more LNET NIDs. $ lst add_group servers 192.168.10.[35,40-45]@tcp $ lst add_group clients 192.168.1.[10-100]@tcp 192.168.[2,4].\ [10-20]@tcp –-refresh Refreshes the state of all inactive nodes in the group. –-clean STATUS Removes nodes with a specified status from the group. Status may be: active The node is in the current session. busy The node is now owned by another session. down The node has been marked down.18-28 Lustre 1.8 Operations Manual • December 2010 list_group [NAME] [--active] [--busy] [--down] [--unknown] [--all] Prints information about a group or lists all groups in the current session if no group is specified. unknown The node’s status has yet to be determined. invalid Any state but active. –-remove NIDs Removes specified nodes from the group. $ lst update_group clients --refresh $ lst update_group clients --clean busy $ lst update_group clients --clean invalid // \ invalid == busy || down || unknown $ lst update_group clients --remove 192.168.1.[10-20]@tcp NAME The name of the group. –-active Lists the active nodes. –-busy Lists the busy nodes. –-down Lists the down nodes. –-unknown Lists unknown nodes. –-all Lists all nodes. $ lst list_group 1) clients 2) servers Total 2 groups $ lst list_group clients ACTIVE BUSY DOWN UNKNOWN TOTAL 3 1 2 0 6 $ lst list_group clients --all 192.168.1.10@tcp Active 192.168.1.11@tcp Active 192.168.1.12@tcp Busy 192.168.1.13@tcp Active 192.168.1.14@tcp DOWN 192.168.1.15@tcp DOWN Total 6 nodes $ lst list_group clients --busy 192.168.1.12@tcp Busy Total 1 nodeChapter 18 Lustre I/O Kit 18-29 del_group NAME Removes a group from the session. If the group is referred to by any test, then the operation fails. If nodes in the group are referred to only by this group, then they are kicked out from the current session; otherwise, they are still in the current session. $ lst del_group clients Userland client (lstclient --sesid NID --group NAME) Use lstclient to run the userland self-test client. lstclient should be executed after creating a session on the console. There are only two mandatory options for lstclient: Also, lstclient has a mandatory option that enforces LNET to behave as a server (start acceptor if the underlying NID needs it, use privileged ports, etc.): --server_mode For example: Client1 $ lstclient --sesid 192.168.1.52@tcp |--group clients --server_mode Note – Only the super user is allowed to use the --server_mode option. –-sesid NID The first console’s NID. –-group NAME The test group to join. Console $ lst new_session testsession Client1 $ lstclient --sesid 192.168.1.52@tcp --group clients18-30 Lustre 1.8 Operations Manual • December 2010 18.4.2.3 Batch and Test This section lists lst batch and test commands. add_batch NAME The default batch (named “batch”) is created when the session is started. However, the user can specify a batch name by using add_batch: $ lst add_batch bulkperf add_test --batch BATCH [--loop #] [--concurrency #] [--distribute #:#] from GROUP --to GROUP TEST ... Adds a test to batch. For now, TEST can be brw and ping: –-loop # Loop count of the test. –-concurrency # Concurrency of the test. –-from GROUP The source group (test client). –-to GROUP The target group (test server). –-distribute #:# The distribution of nodes in clients and servers. The first number of distribute is a subset of client (count of nodes in the “from” group). The second number of distribute is a subset of server (count of nodes in the “to” group); only nodes in two correlative subsets will talk. The following examples are illustrative: Clients: (C1, C2, C3, C4, C5, C6) Server: (S1, S2, S3) --distribute 1:1 (C1->S1), (C2->S2), (C3->S3), (C4->S1), (C5->S2), (C6->S3) \ /* -> means test conversation */ --distribute 2:1 (C1,C2->S1), (C3,C4->S2), (C5,C6->S3) --distribute 3:1 (C1,C2,C3->S1), (C4,C5,C6->S2), (NULL->S3) --distribute 3:2 (C1,C2,C3->S1,S2), (C4,C5,C6->S3,S1) --distribute 4:1 (C1,C2,C3,C4->S1), (C5,C6->S2), (NULL->S3) --distribute 4:2 (C1,C2,C3,C4->S1,S2), (C5, C6->S3, S1) --distribute 6:3 (C1,C2,C3,C4,C5,C6->S1,S2,S3)Chapter 18 Lustre I/O Kit 18-31 There are only two test types: list_batch [NAME] [--test INDEX] [--active] [--invalid] [--server] Lists batches in the current session or lists client|server nodes in a batch or a test. –-ping There are no private parameters for the ping test. –-brw The brw test can have several options: read | write Read or write. The default is read. size=# | #K | #M I/O size can be bytes, KB or MB (i.e., size=1024, size=4K, size=1M. The default is 4K bytes. check=full | simple A data validation check (checksum of data). The default is no-check. As an example: $ lst add_group clients 192.168.1.[10-17]@tcp $ lst add_group servers 192.168.10.[100-103]@tcp $ lst add_batch bulkperf $ lst add_test --batch bulkperf --loop 100 \ --concurrency 4 --distribute 4:2 --from clients \ brw WRITE size=16K // add brw (WRITE, 16 KB) test to batch bulkperf, \ the test will run in 4 workitem, each // 192.168.1.[10-13] will write to 192.168.10.[100,101] // 192.168.1.[14-17] will write to 192.168.10.[102,103] –-test INDEX Lists tests in a batch. If no option is used, all tests in the batch are listed. If the option is used, only specified tests in the batch are listed. $ lst list_batch bulkperf $ lst list_batch bulkperf Batch: bulkperf Tests: 1 State: Idle ACTIVE BUSY DOWN UNKNOWN TOTAL client 8 0 0 0 8 server 4 0 0 0 4 Test 1(brw) (loop: 100, concurrency: 4) ACTIVE BUSY DOWN UNKNOWN TOTAL client 8 0 0 0 8 server 4 0 0 0 4 $ lst list_batch bulkperf --server --active 192.168.10.100@tcp Active 192.168.10.101@tcp Active 192.168.10.102@tcp Active 192.168.10.103@tcp Active18-32 Lustre 1.8 Operations Manual • December 2010 run NAME Runs the batch. $ lst run bulkperf stop NAME Stops the batch. $ lst stop bulkperf query NAME [--test INDEX] [--timeout #] [--loop #] [--delay #] [--all] Queries the batch status. –-test INDEX Only queries the specified test. The test INDEX starts from 1. –-timeout # The timeout value to wait for RPC. The default is 5 seconds. –-loop # The loop count of the query. –-delay # The interval of each query. The default is 5 seconds. –-all The list status of all nodes in a batch or a test. $ lst run bulkperf $ lst query bulkperf --loop 5 --delay 3 Batch is running Batch is running Batch is running Batch is running Batch is running $ lst query bulkperf --all 192.168.1.10@tcp Running 192.168.1.11@tcp Running 192.168.1.12@tcp Running 192.168.1.13@tcp Running 192.168.1.14@tcp Running 192.168.1.15@tcp Running 192.168.1.16@tcp Running 192.168.1.17@tcp Running $ lst stop bulkperf $ lst query bulkperf Batch is idleChapter 18 Lustre I/O Kit 18-33 18.4.2.4 Other Commands This section lists other lst commands. ping [-session] [--group NAME] [--nodes NIDs] [--batch name] [--server] [--timeout #] Sends a “hello” query to the nodes. –-session Pings all nodes in the current session. –-group NAME Pings all nodes in a specified group. –-nodes NIDs Pings all specified nodes. –-batch NAME Pings all client nodes in a batch. –-server Sends RPC to all server nodes instead of client nodes. This option is only used with batch NAME. –-timeout # The RPC timeout value. $ lst ping 192.168.10.[15-20]@tcp 192.168.1.15@tcp Active [session: liang id: 192.168.1.3@tcp] 192.168.1.16@tcp Active [session: liang id: 192.168.1.3@tcp] 192.168.1.17@tcp Active [session: liang id: 192.168.1.3@tcp] 192.168.1.18@tcp Busy [session: Isaac id: 192.168.10.10@tcp] 192.168.1.19@tcp Down [session: id: LNET_NID_ANY] 192.168.1.20@tcp Down [session: id: LNET_NID_ANY]18-34 Lustre 1.8 Operations Manual • December 2010 stat [--bw] [--rate] [--read] [--write] [--max] [--min] [--avg] " " [--timeout #] [--delay #] GROUP|NIDs [GROUP|NIDs] The collection performance and RPC statistics of one or more nodes. Specifying a group name (GROUP) causes statistics to be gathered for all nodes in a test group. For example: $ lst stat servers where servers is the name of a test group created by lst add_group Specifying a NID range (NIDs) causes statistics to be gathered for selected nodes. For example: $ lst stat 192.168.0.[1-100/2]@tcp Currently, only LNET performance statistics are available. 2 By default, all statistics information is displayed. Users can specify additional information with these options. 2. In the future, more statistics will be supported. –-bw Displays the bandwidth of the specified group/nodes. –-rate Displays the rate of RPCs of the specified group/nodes. –-read Displays the read statistics of the specified group/nodes. –-write Displays the write statistics of the specified group/nodes. –-max Displays the maximum value of the statistics. –-min Displays the minimum value of the statistics. –-avg Displays the average of the statistics. –-timeout # The timeout of the statistics RPC. The default is 5 seconds. –-delay # The interval of the statistics (in seconds). $ lst run bulkperf $ lst stat clients [LNet Rates of clients] [W] Avg: 1108 RPC/s Min: 1060 RPC/s Max: 1155 RPC/s [R] Avg: 2215 RPC/s Min: 2121 RPC/s Max: 2310 RPC/s [LNet Bandwidth of clients] [W] Avg: 16.60 MB/s Min: 16.10 MB/s Max: 17.1 MB/s [R] Avg: 40.49 MB/s Min: 40.30 MB/s Max: 40.68 MB/sChapter 18 Lustre I/O Kit 18-35 show_error [--session] [GROUP]|[NIDs] ... Lists the number of failed RPCs on test nodes. –-session Lists errors in the current test session. With this option, historical RPC errors are not listed. $ lst show_error clients clients 12345-192.168.1.15@tcp: [Session: 1 brw errors, 0 ping errors] \ [RPC: 20 errors, 0 dropped, 12345-192.168.1.16@tcp: [Session: 0 brw errors, 0 ping errors] \ [RPC: 1 errors, 0 dropped, Total 2 error nodes in clients $ lst show_error --session clients clients 12345-192.168.1.15@tcp: [Session: 1 brw errors, 0 ping errors] Total 1 error nodes in clients18-36 Lustre 1.8 Operations Manual • December 201019-1 C H A P T E R 19 Lustre Recovery This chapter describes how to recover Lustre, and includes the following sections: ¦ Recovery Overview ¦ Metadata Replay ¦ Reply Reconstruction ¦ Version-based Recovery ¦ Recovering from Errors or Corruption on a Backing File System ¦ Recovering from Corruption in the Lustre File System19-2 Lustre 1.8 Operations Manual • December 2010 19.1 Recovery Overview Lustre's recovery support is responsible for dealing with node or network failure and returning the cluster to a consistent, performant state. Because Lustre allows servers to perform asynchronous update operations to the on-disk file system (i.e., the server can reply without waiting for the update to synchronously commit to disk), the clients may have state in memory that is newer than what the server can recover from disk after a crash. A handful of different types of failures can cause recovery to occur: ¦ Client (compute node) failure ¦ MDS failure (and failover) ¦ OST failure (and failover) ¦ Transient network partition Currently, all Lustre failure and recovery operations are based on the concept of connection failure; all imports or exports associated with a given connection are considered to fail if any of them fail. For information on Lustre recovery, see Metadata Replay. For information on recovering from a corrupt file system, see Recovering from Errors or Corruption on a Backing File System. For information on resolving orphaned objects, a common issue after recovery, see . 19.1.1 Client Failure Lustre's support for recovery from client failure is based on lock revocation and other resources, so surviving clients can continue their work uninterrupted. If a client fails to timely respond to a blocking lock callback from the Distributed Lock Manager (DLM) or fails to communicate with the server in a long period of time (i.e., no pings), the client is forcibly removed from the cluster (evicted). This enables other clients to acquire locks blocked by the dead client's locks, and also frees resources (file handles, export data) associated with that client. Note that this scenario can be caused by a network partition, as well as an actual client node system failure. Network Partition describes this case in more detail.Chapter 19 Lustre Recovery 19-3 19.1.2 Client Eviction If a client is not behaving properly from the server's point of view, it will be evicted. This ensures that the whole file system can continue to function in the presence of failed or misbehaving clients. An evicted client must invalidate all locks, which in turn, results in all cached inodes becoming invalidated and all cached data being flushed. Reasons why a client might be evicted: ¦ Failure to respond to a server request in a timely manner ¦ Blocking lock callback (i.e., client holds lock that another client/server wants) ¦ Lock completion callback (i.e., client is granted lock previously held by another client) ¦ Lock glimpse callback (i.e., client is asked for size of object by another client) ¦ Server shutdown notification (with simplified interoperability) ¦ Failure to ping the server in a timely manner, unless the server is receiving no RPC traffic at all (which may indicate a network partition). 19.1.3 MDS Failure (Failover) Highly-available (HA) Lustre operation requires that the metadata server have a peer configured for failover, including the use of a shared storage device for the MDT backing file system. The actual mechanism for detecting peer failure, power off (STONITH) of the failed peer (to prevent it from continuing to modify the shared disk), and takeover of the Lustre MDS service on the backup node depends on external HA software such as Heartbeat. It is also possible to have MDS recovery with a single MDS node. In this case, recovery will take as long as is needed for the single MDS to be restarted. When clients detect an MDS failure (either by timeouts of in-flight requests or idle-time ping messages), they connect to the new backup MDS and use the Metadata Replay protocol. Metadata Replay is responsible for ensuring that the backup MDS re-acquires state resulting from transactions whose effects were made visible to clients, but which were not committed to the disk. The reconnection to a new (or restarted) MDS is managed by the file system configuration loaded by the client when the file system is first mounted. If a failover MDS has been configured (using the --failnode= option to mkfs.lustre or tunefs.lustre), the client tries to reconnect to both the primary and backup MDS until one of them responds that the failed MDT is again available. At that point, the client begins recovery. For more information, see Metadata Replay.19-4 Lustre 1.8 Operations Manual • December 2010 Transaction numbers are used to ensure that operations are replayed in the order they were originally performed, so that they are guaranteed to succeed and present the same filesystem state as before the failure. In addition, clients inform the new server of their existing lock state (including locks that have not yet been granted). All metadata and lock replay must complete before new, non-recovery operations are permitted. In addition, only clients that were connected at the time of MDS failure are permitted to reconnect during the recovery window, to avoid the introduction of state changes that might conflict with what is being replayed by previously-connected clients. 19.1.4 OST Failure (Failover) When an OST fails or has communication problems with the client, the default action is that the corresponding OSC enters recovery, and I/O requests going to that OST are blocked waiting for OST recovery or failover. It is possible to administratively mark the OSC as inactive on the client, in which case file operations that involve the failed OST will return an IO error (-EIO). Otherwise, the application waits until the OST has recovered or the client process is interrupted (e.g. ,with CTRL-C). The MDS (via the LOV) detects that an OST is unavailable and skips it when assigning objects to new files. When the OST is restarted or re-establishes communication with the MDS, the MDS and OST automatically perform orphan recovery to destroy any objects that belong to files that were deleted while the OST was unavailable. For more information, see Working with Orphaned Objects. While the OSC to OST operation recovery protocol is the same as that between the MDC and MDT using the Metadata Replay protocol, typically the OST commits bulk write operations to disk synchronously and each reply indicates that the request is already committed and the data does not need to be saved for recovery. In some cases, the OST replies to the client before the operation is committed to disk (e.g. truncate, destroy, setattr, and I/O operations in very new versions of Lustre), and normal replay and resend handling is done, including resending of the bulk writes. In this case, the client keeps a copy of the data available in memory until the server indicates that the write has committed to disk. To force an OST recovery, unmount the OST and then mount it again. If the OST was connected to clients before it failed, then a recovery process starts after the remount, enabling clients to reconnect to the OST and replay transactions in their queue. When the OST is in recovery mode, all new client connections are refused until the recovery finishes. The recovery is complete when either all previously-connected clients reconnect and their transactions are replayed or a client connection attempt times out. If a connection attempt times out, then all clients waiting to reconnect (and their transactions) are lost.Chapter 19 Lustre Recovery 19-5 Note – If you know an OST will not recover a previously-connected client (if, for example, the client has crashed), you can manually abort the recovery using this command: lctl --device abort_recovery To determine an OST’s device number and device name, run the lctl dl command. Sample lctl dl command output is shown below: 7 UP obdfilter ddn_data-OST0009 ddn_data-OST0009_UUID 1159 In this example, 7 is the OST device number. The device name is ddn_data-OST0009. In most instances, the device name can be used in place of the device number. 19.1.5 Network Partition Network failures may be transient. To avoid invoking recovery, the client tries, initially, to re-send any timed out request to the server. If the resend also fails, the client tries to re-establish a connection to the server. Clients can detect harmless partition upon reconnect if the server has not had any reason to evict the client. If a request was processed by the server, but the reply was dropped (i.e., did not arrive back at the client), the server must reconstruct the reply when the client resends the request, rather than performing the same request twice. 19.1.6 Failed Recovery In the case of failed recovery, a client is evicted by the server and must reconnect after having flushed its saved state related to that server, as described in Client Eviction, above. Failed recovery might occur for a number of reasons, including: ¦ Failure of recovery ¦ In Lustre 1.8, recovery fails if the operations of one client directly depend on the operations of another client that failed to participate in recovery. Otherwise, Version Based Recovery (VBR) allows recovery to proceed for all of the connected clients, and only missing clients are evicted. ¦ Manual abort of recovery ¦ Manual eviction by the administrator19-6 Lustre 1.8 Operations Manual • December 2010 19.2 Metadata Replay Highly available Lustre operation requires that the MDS have a peer configured for failover, including the use of a shared storage device for the MDS backing file system. When a client detects an MDS failure, it connects to the new MDS and uses the metadata replay protocol to replay its requests. Metadata replay ensures that the failover MDS re-accumulates state resulting from transactions whose effects were made visible to clients, but which were not committed to the disk. 19.2.1 XID Numbers Each request sent by the client contains an XID number, which is a client-unique, monotonically increasing 64-bit integer. The initial value of the XID is chosen so that it is highly unlikely that the same client node reconnecting to the same server after a reboot would have the same XID sequence. The XID is used by the client to order all of the requests that it sends, until such a time that the request is assigned a transaction number. The XID is also used in Reply Reconstruction to uniquely identify per-client requests at the server. 19.2.2 Transaction Numbers Each client request processed by the server that involves any state change (metadata update, file open, write, etc., depending on server type) is assigned a transaction number by the server that is a target-unique, monontonically increasing, server-wide 64-bit integer. The transaction number for each file system-modifying request is sent back to the client along with the reply to that client request. The transaction numbers allow the client and server to unambiguously order every modification to the file system in case recovery is needed. Each reply sent to a client (regardless of request type) also contains the last committed transaction number that indicates the highest transaction number committed to the file system. The backing file systems that Lustre uses (ext3/4, ZFS) enforce the requirement that any earlier disk operation will always be committed to disk before a later disk operation, so the last committed transaction number also reports that any requests with a lower transaction number have been committed to disk.Chapter 19 Lustre Recovery 19-7 19.2.3 Replay and Resend Lustre recovery can be separated into two distinct types of operations: replay and resend. Replay operations are those for which the client received a reply from the server that the operation had been successfully completed. These operations need to be redone in exactly the same manner after a server restart as had been reported before the server failed. Replay can only happen if the server failed; otherwise it will not have lost any state in memory. Resend operations are those for which the client never received a reply, so their final state is unknown to the client. The client sends unanswered requests to the server again in XID order, and again awaits a reply for each one. In some cases, resent requests have been handled and committed to disk by the server (possibly also having dependent operations committed), in which case, the server performs reply reconstruction for the lost reply. In other cases, the server did not receive the lost request at all and processing proceeds as with any normal request. These are what happen in the case of a network interruption. It is also possible that the server received the request, but was unable to reply or commit it to disk before failure. 19.2.4 Client Replay List All file system-modifying requests have the potential to be required for server state recovery (replay) in case of a server failure. Replies that have an assigned transaction number that is higher than the last committed transaction number received in any reply from each server are preserved for later replay in a per-server replay list. As each reply is received from the server, it is checked to see if it has a higher last committed transaction number than the previous highest last committed number. Most requests that now have a lower transaction number can safely be removed from the replay list. One exception to this rule is for open requests, which need to be saved for replay until the file is closed so that the MDS can properly reference count open-unlinked files.19-8 Lustre 1.8 Operations Manual • December 2010 19.2.5 Server Recovery A server enters recovery if it was not shut down cleanly. If, upon startup, if any client entries are in the last_rcvd file for any previously connected clients, the server enters recovery mode and waits for these previously-connected clients to reconnect and begin replaying or resending their requests. This allows the server to recreate state that was exposed to clients (a request that completed successfully) but was not committed to disk before failure. In the absence of any client connection attempts, the server waits indefinitely for the clients to reconnect. This is intended to handle the case where the server has a network problem and clients are unable to reconnect and/or if the server needs to be restarted repeatedly to resolve some problem with hardware or software. Once the server detects client connection attempts - either new clients or previously-connected clients - a recovery timer starts and forces recovery to finish in a finite time regardless of whether the previously-connected clients are available or not. If no client entries are present in the last_rcvd file, or if the administrator manually aborts recovery, the server does not wait for client reconnection and proceeds to allow all clients to connect. As clients connect, the server gathers information from each one to determine how long the recovery needs to take. Each client reports its connection UUID, and the server does a lookup for this UUID in the last_rcvd file to determine if this client was previously connected. If not, the client is refused connection and it will retry until recovery is completed. Each client reports its last seen transaction, so the server knows when all transactions have been replayed. The client also reports the amount of time that it was previously waiting for request completion so that the server can estimate how long some clients might need to detect the server failure and reconnect. If the client times out during replay, it attempts to reconnect. If the client is unable to reconnect, REPLAY fails and it returns to DISCON state. It is possible that clients will timeout frequently during REPLAY, so reconnection should not delay an already slow process more than necessary. We can mitigate this by increasing the timeout during replay.Chapter 19 Lustre Recovery 19-9 19.2.6 Request Replay If a client was previously connected, it gets a response from the server telling it that the server is in recovery and what the last committed transaction number on disk is. The client can then iterate through its replay list and use this last committed transaction number to prune any previously-committed requests. It replays any newer requests to the server in transaction number order, one at a time, waiting for a reply from the server before replaying the next request. Open requests that are on the replay list may have a transaction number lower than the server's last committed transaction number. The server processes those open requests immediately. The server then processes replayed requests from all of the clients in transaction number order, starting at the last committed transaction number to ensure that the state is updated on disk in exactly the same manner as it was before the crash. As each replayed request is processed, the last committed transaction is incremented. If the server receives a replay request from a client that is higher than the current last committed transaction, that request is put aside until other clients provide the intervening transactions. In this manner, the server replays requests in the same sequence as they were previously executed on the server until either all clients are out of requests to replay or there is a gap in a sequence. 19.2.7 Gaps in the Replay Sequence In some cases, a gap may occur in the reply sequence. This might be caused by lost replies, where the request was processed and committed to disk but the reply was not received by the client. It can also be caused by clients missing from recovery due to partial network failure or client death. In the case where all clients have reconnected, but there is a gap in the replay sequence the only possibility is that some requests were processed by the server but the reply was lost. Since the client must still have these requests in its resend list, they are processed after recovery is finished. In the case where all clients have not reconnected, it is likely that the failed clients had requests that will no longer be replayed. In Lustre 1.8 and later, version-based recovery (VBR) is used to determine if a request following a transaction gap is safe to be replayed. Each item in the file system (MDS inode or OST object) stores on disk the number of the last transaction in which it was modified. Each reply from the server contains the previous version number of the objects that it affects. During VBR replay, the server matches the previous version numbers in the resend request against the current version number. If the versions match, the request is the next one that affects the object and can be safely replayed. For more information, see Version-based Recovery.19-10 Lustre 1.8 Operations Manual • December 2010 19.2.8 Lock Recovery If all requests were replayed successfully and all clients reconnected, clients then do lock replay locks -- that is, every client sends information about every lock it holds from this server and its state (whenever it was granted or not, what mode, what properties and so on), and then recovery completes successfully. Currently, Lustre does not do lock verification and just trusts clients to present an accurate lock state. This does not impart any security concerns since Lustre 1.x clients are trusted for other information (e.g. user ID) during normal operation also. After all of the saved requests and locks have been replayed, the client sends an MDS_GETSTATUS request with last-replay flag set. The reply to that request is held back until all clients have completed replay (sent the same flagged getstatus request), so that clients don't send non-recovery requests before recovery is complete. 19.2.9 Request Resend Once all of the previously-shared state has been recovered on the server (the target file system is up-to-date with client cache and the server has recreated locks representing the locks held by the client), the client can resend any requests that did not receive an earlier reply. This processing is done like normal request processing, and, in some cases, the server may do reply reconstruction.Chapter 19 Lustre Recovery 19-11 19.3 Reply Reconstruction When a reply is dropped, the MDS needs to be able to reconstruct the reply when the original request is re-sent. This must be done without repeating any non-idempotent operations, while preserving the integrity of the locking system. In the event of MDS failover, the information used to reconstruct the reply must be serialized on the disk in transactions that are joined or nested with those operating on the disk. 19.3.1 Required State For the majority of requests, it is sufficient for the server to store three pieces of data in the last_rcvd file: ¦ XID of the request ¦ Resulting transno (if any) ¦ Result code (req->rq_status) For open requests, the "disposition" of the open must also be stored. 19.3.2 Reconstruction of Open Replies An open reply consists of up to three pieces of information (in addition to the contents of the "request log"): ¦ File handle ¦ Lock handle ¦ mds_body with information about the file created (for O_CREAT) The disposition, status and request data (re-sent intact by the client) are sufficient to determine which type of lock handle was granted, whether an open file handle was created, and which resource should be described in the mds_body.19-12 Lustre 1.8 Operations Manual • December 2010 Finding the File Handle The file handle can be found in the XID of the request and the list of per-export open file handles. The file handle contains the resource/FID. Finding the Resource/fid The file handle contains the resource/fid. Finding the Lock Handle The lock handle can be found by walking the list of granted locks for the resource looking for one with the appropriate remote file handle (present in the re-sent request). Verify that the lock has the right mode (determined by performing the disposition/request/status analysis above) and is granted to the proper client.Chapter 19 Lustre Recovery 19-13 19.4 Version-based Recovery Lustre 1.8 introduces the Version-based Recovery (VBR) feature, which improves Lustre reliability in cases where client requests (RPCs) fail to replay during recovery 1 . In pre-1.8 versions of Lustre, if the MGS or an OST went down and then recovered, a recovery process was triggered in which clients attempted to replay their requests. Clients were only allowed to replay RPCs in serial order. If a particular client could not replay its requests, then those requests were lost as well as the requests of clients later in the sequence. The ''downstream'' clients never got to replay their requests because of the wait on the earlier client’s RPCs. Eventually, the recovery period would time out (so the component could accept new requests), leaving some number of clients evicted and their requests and data lost. With VBR, the recovery mechanism does not result in the loss of clients or their data, because changes in inode versions are tracked, and more clients are able to reintegrate into the cluster. With VBR, inode tracking looks like this: ¦ Each inode 2 stores a version, that is, the number of the last transaction (transno) in which the inode was changed. ¦ When an inode is about to be changed, a pre-operation version of the inode is saved in the client’s data. ¦ The client keeps the pre-operation inode version and the post-operation version (transaction number) for replay, and sends them in the event of a server failure. ¦ If the pre-operation version matches, then the request is replayed. The post-operation version is assigned on all inodes modified in the request. Note – An RPC can contain up to four pre-operation versions, because several inodes can be involved in an operation. In the case of a ''rename'' operation, four different inodes can be modified. 1. There are two scenarios under which client RPCs are not replayed: (1) Non-functioning or isolated clients do not reconnect, and they cannot replay their RPCs, causing a gap in the replay sequence. These clients get errors and are evicted. (2) Functioning clients connect, but they cannot replay some or all of their RPCs that occurred after the gap caused by the non-functioning/isolated clients. These clients get errors (caused by the failed clients). With VBR, these requests have a better chance to replay because the "gaps" are only related to specific files that the missing client(s) changed. 2. Usually, there are two inodes, a parent and a child.19-14 Lustre 1.8 Operations Manual • December 2010 During normal operation, the server: ¦ Updates the versions of all inodes involved in a given operation ¦ Returns the old and new inode versions to the client with the reply When the recovery mechanism is underway, VBR follows these steps: 1. VBR only allows clients to replay transactions if the affected inodes have the same version as during the original execution of the transactions, even if there is gap in transactions due to a missed client. 2. The server attempts to execute every transaction that the client offers, even if it encounters a re-integration failure. 3. When the replay is complete, the client and server check if a replay failed on any transaction because of inode version mismatch. If the versions match, the client gets a successful re-integration message. If the versions do not match, then the client is evicted. VBR recovery is fully transparent to users. It may lead to slightly longer recovery times if the cluster loses several clients during server recovery. 19.4.1 Delayed Recovery With VBR, it is possible to recover clients even after the server’s recovery window closes. This is known as delayed recovery. This feature is useful if clients have become temporarily unavailable during recovery (e.g., because of a network partition). Note – In Lustre 1.8, the delayed recovery feature is available as a preview, and is turned off by default. It is designed for use with future versions of Lustre, to help with disconnected operations.Chapter 19 Lustre Recovery 19-15 19.4.2 Working with VBR In Lustre 1.8, the VBR feature is built into the Lustre recovery functionality. It cannot be disabled. Delayed recovery can be enabled with the --enable-delayed-recovery option: ./configure ... --enable-delayed-recovery During reboot, a list of new messages is displayed. CWARN("RECOVERY: service %s, %d recoverable clients, last_transno "LPU64"\n"); was updated with number delayed clients: CWARN("RECOVERY: service %s, %d recoverable clients, %d delayed clients, last_transno "LPU64"\n"); Note – There should be no delayed clients until delayed recovery is enabled. These are some VBR messages that may be displayed: DEBUG_REQ(D_WARNING, req, "Version mismatch during replay\n"); This message indicates why the client was evicted. No action is needed. CWARN("%s: version recovery fails, reconnecting\n"); This message indicates why the recovery failed. No action is needed. These are some VBR messages that may be displayed if delayed recovery is enabled: CWARN("RECOVERY: service %s, %d recoverable clients, %d delayed clients, last_transno "LPU64"\n"); This controls the number of delayed clients. There should be 0 delayed clients without delayed recovery enabled). CWARN("%s: NID %s (%s) export was already marked as delayed and will wait for end of recovery\n"); The old client is trying to reconnect, but it will wait for end of the server’s recovery period. No action is needed. 19.4.3 Tips for Using VBR VBR will be successful for clients which do not share data with other client. Therefore, the strategy for reliable use of VBR is to store a client’s data in its own directory, where possible. VBR can recover these clients, even if other clients are lost.19-16 Lustre 1.8 Operations Manual • December 2010 19.5 Recovering from Errors or Corruption on a Backing File System When an OSS, MDS, or MGS server crash occurs, it is not necessary to run e2fsck on the file system. Ext3 journaling ensures that the file system remains coherent. The backing file systems are never accessed directly from the client, so client crashes are not relevant. The only time it is REQUIRED that e2fsck be run on a device is when an event causes problems that ext3 journaling is unable to handle, such as a hardware device failure or I/O error. If the ext3 kernel code detects corruption on the disk, it mounts the file system as read-only to prevent further corruption, but still allows read access to the device. This appears as error "-30" (EROFS) in the syslogs on the server, e.g.: Dec 29 14:11:32 mookie kernel: LDISKFS-fs error (device sdz): ldiskfs_lookup: unlinked inode 5384166 in dir #145170469 Dec 29 14:11:32 mookie kernel: Remounting filesystem read-only In such a situation, it is normally required that e2fsck only be run on the bad device before placing the device back into service. In the vast majority of cases, Lustre can cope with any inconsistencies it finds on the disk and between other devices in the file system. Note – lfsck is rarely required for Lustre operation. For problem analysis, it is strongly recommended that e2fsck be run under a logger, like script, to record all of the output and changes that are made to the file system in case this information is needed later. If time permits, it is also a good idea to first run e2fsck in non-fixing mode (-n option) to assess the type and extent of damage to the file system. The drawback is that in this mode, e2fsck does not recover the file system journal, so there may appear to be file system corruption when none really exists. To address concern about whether corruption is real or only due to the journal not being replayed, you can briefly mount and unmount the ext3 filesystem directly on the node with Lustre stopped (NOT via Lustre), using a command similar to: mount -t ldiskfs /dev/{ostdev} /mnt/ost; umount /mnt/ost This causes the journal to be recovered.Chapter 19 Lustre Recovery 19-17 The e2fsck utility works well when fixing file system corruption (better than similar file system recovery tools and a primary reason why ext3 was chosen over other file systems for Lustre). However, it is often useful to identify the type of damage that has occurred so an ext3 expert can make intelligent decisions about what needs fixing, in place of e2fsck. root# {stop lustre services for this device, if running} root# script /tmp/e2fsck.sda Script started, file is /tmp/e2fsck.sda root# mount -t ldiskfs /dev/sda /mnt/ost root# umount /mnt/ost root# e2fsck -fn /dev/sda # don't fix file system, just check for corruption : [e2fsck output] : root# e2fsck -fp /dev/sda # fix filesystem using "prudent" answers (usually 'y') In addition, the e2fsprogs package contains the lfsck tool, which does distributed coherency checking for the Lustre file system after e2fsck has been run. Running lfsck is NOT required in a large majority of cases, at a small risk of having some leaked space in the file system. To avoid a lengthy downtime, it can be run (with care) after Lustre is started.19-18 Lustre 1.8 Operations Manual • December 2010 19.6 Recovering from Corruption in the Lustre File System In cases where the MDS or an OST becomes corrupt, you can run a distributed check on the file system to determine what sort of problems exist. Use lfsck to correct any defects found. 1. Stop the Lustre file system. 2. Run e2fsck -f on the individual MDS / OST that had problems to fix any local file system damage. We recommend running e2fsck under script, to create a log of changes made to the file system in case it is needed later. After e2fsck is run, bring up the file system, if necessary, to reduce the outage window. 3. Run a full e2fsck of the MDS to create a database for lfsck. It is critical to use the -n option for a mounted file system, otherwise you will corrupt the file system. e2fsck -n -v --mdsdb /tmp/mdsdb /dev/{mdsdev} The mdsdb file can grow fairly large, depending on the number of files in the file system (10 GB or more for millions of files, though the actual file size is larger because the file is sparse). It is quicker to write the file to a local file system due to seeking and small writes. Depending on the number of files, this step can take several hours to complete. Example e2fsck -n -v --mdsdb /tmp/mdsdb /dev/sdb e2fsck 1.39.cfs1 (29-May-2006) Warning: skipping journal recovery because doing a read-only filesystem check. lustre-MDT0000 contains a file system with errors, check forced. Pass 1: Checking inodes, blocks, and sizes MDS: ost_idx 0 max_id 288 MDS: got 8 bytes = 1 entries in lov_objids MDS: max_files = 13 MDS: num_osts = 1 mds info db file written Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Free blocks count wrong (656160, counted=656058).Chapter 19 Lustre Recovery 19-19 Fix? no Free inodes count wrong (786419, counted=786036). Fix? no Pass 6: Acquiring information for lfsck MDS: max_files = 13 MDS: num_osts = 1 MDS: 'lustre-MDT0000_UUID' mdt idx 0: compat 0x4 rocomp 0x1 incomp 0x4 lustre-MDT0000: ******* WARNING: Filesystem still has errors ******* 13 inodes used (0%) 2 non-contiguous inodes (15.4%) # of inodes with ind/dind/tind blocks: 0/0/0 130272 blocks used (16%) 0 bad blocks 1 large file 296 regular files 91 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets -------- 387 files 4. Make this file accessible on all OSTs, either by using a shared file system or copying the file to the OSTs. The pdcp command is useful here. The pdcp command (installed with pdsh), can be used to copy files to groups of hosts. Pdcp is available here: http://sourceforge.net/projects/pdsh 5. Run a similar e2fsck step on the OSTs. The e2fsck --ostdb command can be run in parallel on all OSTs. e2fsck -n -v --mdsdb /tmp/mdsdb --ostdb /tmp/{ostNdb} \ /dev/{ostNdev} The mdsdb file is read-only in this step; a single copy can be shared by all OSTs. Note – If the OSTs do not have shared file system access to the MDS, a stub mdsdb file, {mdsdb}.mdshdr, is generated. This can be used instead of the full mdsdb file.19-20 Lustre 1.8 Operations Manual • December 2010 Example: [root@oss161 ~]# e2fsck -n -v --mdsdb /tmp/mdsdb --ostdb \ /tmp/ostdb /dev/sda e2fsck 1.39.cfs1 (29-May-2006) Warning: skipping journal recovery because doing a read-only filesystem check. lustre-OST0000 contains a file system with errors, check forced. Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Free blocks count wrong (989015, counted=817968). Fix? no Free inodes count wrong (262088, counted=261767). Fix? no Pass 6: Acquiring information for lfsck OST: 'lustre-OST0000_UUID' ost idx 0: compat 0x2 rocomp 0 incomp 0x2 OST: num files = 321 OST: last_id = 321 lustre-OST0000: ******* WARNING: Filesystem still has errors ******* 56 inodes used (0%) 27 non-contiguous inodes (48.2%) # of inodes with ind/dind/tind blocks: 13/0/0 59561 blocks used (5%) 0 bad blocks 1 large file 329 regular files 39 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets -------- 368 filesChapter 19 Lustre Recovery 19-21 6. Make the mdsdb file and all ostdb files available on a mounted client and run lfsck to examine the file system. Optionally, correct the defects found by lfsck. script /root/lfsck.lustre.log lfsck -n -v --mdsdb /tmp/mdsdb --ostdb /tmp/{ost1db} /tmp/{ost2db} ... /lustre/mount/point Example: script /root/lfsck.lustre.log lfsck -n -v --mdsdb /home/mdsdb --ostdb /home/{ost1db} \ /mnt/lustre/client/ MDSDB: /home/mdsdb OSTDB[0]: /home/ostdb MOUNTPOINT: /mnt/lustre/client/ MDS: max_id 288 OST: max_id 321 lfsck: ost_idx 0: pass1: check for duplicate objects lfsck: ost_idx 0: pass1 OK (287 files total) lfsck: ost_idx 0: pass2: check for missing inode objects lfsck: ost_idx 0: pass2 OK (287 objects) lfsck: ost_idx 0: pass3: check for orphan objects [0] uuid lustre-OST0000_UUID [0] last_id 288 [0] zero-length orphan objid 1 lfsck: ost_idx 0: pass3 OK (321 files total) lfsck: pass4: check for duplicate object references lfsck: pass4 OK (no duplicates) lfsck: fixed 0 errors By default, lfsck reports errors, but it does not repair any inconsistencies found. lfsck checks for three kinds of inconsistencies: ¦ Inode exists but has missing objects (dangling inode). This normally happens if there was a problem with an OST. ¦ Inode is missing but OST has unreferenced objects (orphan object). Normally, this happens if there was a problem with the MDS. ¦ Multiple inodes reference the same objects. This can happen if the MDS is corrupted or if the MDS storage is cached and loses some, but not all, writes. If the file system is in use and being modified while the --mdsdb and --ostdb steps are running, lfsck may report inconsistencies where none exist due to files and objects being created/removed after the database files were collected. Examine the lfsck results closely. You may want to re-run the test.19-22 Lustre 1.8 Operations Manual • December 2010 19.6.1 Working with Orphaned Objects The easiest problem to resolve is that of orphaned objects. When the -l option for lfsck is used, these objects are linked to new files and put into lost+found in the Lustre file system, where they can be examined and saved or deleted as necessary. If you are certain the objects are not useful, run lfsck with the -d option to delete orphaned objects and free up any space they are using. To fix dangling inodes, use lfsck with the -c option to create new, zero-length objects on the OSTs. These files read back with binary zeros for stripes that had objects re-created. Even without lfsck repair, these files can be read by entering: dd if=/lustre/bad/file of=/new/file bs=4k conv=sync,noerror Because it is rarely useful to have files with large holes in them, most users delete these files after reading them (if useful) and/or restoring them from backup. Note – You cannot write to the holes of such files without having lfsck re-create the objects. Generally, it is easier to delete these files and restore them from backup. To fix inodes with duplicate objects, use lfsck with the -c option to copy the duplicate object to a new object and assign it to a file. One file will be okay and the duplicate will likely contain garbage. By itself, lfsck cannot tell which file is the usable one.PART III Lustre Tuning, Monitoring and Troubleshooting The part includes chapters describing how to tune, debug and troubleshoot Lustre.20-1 C H A P T E R 20 Lustre Tuning This chapter contains information to tune Lustre for better performance and includes the following sections: ¦ Module Options ¦ LNET Tunables ¦ Options for Formatting the MDT and OSTs ¦ Large-Scale Tuning for Cray XT and Equivalents ¦ Lockless I/O Tunables ¦ Data Checksums20-2 Lustre 1.8 Operations Manual • December 2010 20.1 Module Options Many options in Lustre are set by means of kernel module parameters. These parameters are contained in the modprobe.conf file (On SuSE, this may be modprobe.conf.local). 20.1.1 OSS Service Thread Count The oss_num_threads parameter enables the number of OST service threads to be specified at module load time on the OSS nodes: options ost oss_num_threads={N} After startup, the minimum and maximum number of OSS thread counts can be set via the {service}.thread_{min,max,started} tunable. To change the tunable at runtime, run: lctl {get,set}_param {service}.thread_{min,max,started} For details, see Setting MDS and OSS Thread Counts. 20.1.1.1 Optimizing the Number of Service Threads An OSS can have a minimum of 2 service threads and a maximum of 512 service threads. The number of service threads is a function of how much RAM and how many CPUs are on each OSS node (1 thread / 128MB * num_cpus). If the load on the OSS node is high, new service threads will be started in order to process more requests concurrently, up to 4x the initial number of threads (subject to the maximum of 512). For a 2GB 2-CPU system, the default thread count is 32 and the maximum thread count is 128. Increasing the size of the thread pool may help when: ¦ Several OSTs are exported from a single OSS ¦ Back-end storage is running synchronously ¦ I/O completions take excessive time due to slow storage Decreasing the size of the thread pool may help if: ¦ Clients are overwhelming the storage capacity ¦ There are lots of "slow I/O" or similar messagesChapter 20 Lustre Tuning 20-3 Increasing the number of I/O threads allows the kernel and storage to aggregate many writes together for more efficient disk I/O. The OSS thread pool is shared—each thread allocates approximately 1.5 MB (maximum RPC size + 0.5 MB) for internal I/O buffers. It is very important to consider memory consumption when increasing the thread pool size. Drives are only able to sustain a certain amount of parallel I/O activity before performance is degraded, due to the high number of seeks and the OST threads just waiting for I/O. In this situation, it may be advisable to decrease the load by decreasing the number of OST threads. Determining the optimum number of OST threads is a process of trial and error, and varies for each particular configuration. Variables include the number of OSTs on each OSS, number and speed of disks, RAID configuration, and available RAM. You may want to start with a number of OST threads equal to the number of actual disk spindles on the node. If you use RAID, subtract any dead spindles not used for actual data (e.g., 1 of N of spindles for RAID5, 2 of N spindles for RAID6), and monitor the performance of clients during usual workloads. If performance is degraded, increase the thread count and see how that works until performance is degraded again or you reach satisfactory performance. Note – If there are too many threads, the latency for individual I/O requests can become very high and should be avoided. Set the desired maximum thread count permanently using the method described above. 20.1.2 MDS Service Thread Count The mds_num_threads parameter enables the number of MDS service threads to be specified at module load time on the MDS node: options mds mds_num_threads={N} After startup, the minimum and maximum number of MDS thread counts can be set via the {service}.thread_{min,max,started} tunable. To change the tunable at runtime, run: lctl {get,set}_param {service}.thread_{min,max,started} For details, see Setting MDS and OSS Thread Counts. At this time, no testing has been done to determine the optimal number of MDS threads. The default value varies, based on server size, up to a maximum of 32. The maximum number of threads (MDS_MAX_THREADS) is 512.20-4 Lustre 1.8 Operations Manual • December 2010 Note – The OSS and MDS automatically start new service threads dynamically, in response to server load within a factor of 4. The default value is calculated the same way as before. Setting the _mu_threads module parameter disables automatic thread creation behavior. 20.2 LNET Tunables This section describes LNET tunables. 20.2.1 Transmit and receive buffer size: With Lustre release 1.4.7 and later, ksocklnd now has separate parameters for the transmit and receive buffers. options ksocklnd tx_buffer_size=0 rx_buffer_size=0 If these parameters are left at the default value (0), the system automatically tunes the transmit and receive buffer size. In almost every case, this default produces the best performance. Do not attempt to tune these parameters unless you are a network expert. 20.2.2 irq_affinity By default, this parameter is on. In the normal case of an SMP system, we would like network traffic to remain local to a single CPU. This helps to keep the processor cache warm and minimizes the impact of context switches. This is especially helpful when an SMP system has more than one network interface and ideal when the number of interfaces equals the number of CPUs. If you have an SMP platform with a single fast interface such as 10GB Ethernet and more than two CPUs, you may see performance improve by turning this parameter off. As always, you should test to compare the impact.Chapter 20 Lustre Tuning 20-5 20.3 Options for Formatting the MDT and OSTs The backing file systems on an MDT and OSTs are independent of one another, so the formatting parameters for them should not be same. The size of the MDS backing file system depends solely on how many inodes you want in the total Lustre file system. It is not related to the size of the aggregate OST space. 20.3.1 Planning for Inodes Each time you create a file on a Lustre file system, it consumes one inode on the MDS and one inode for each OST object that the file is striped over (normally it is based on the default stripe count option -c, but this may change on a per-file basis). In ext3/ldiskfs file systems, inodes are pre-allocated, so creating a new file does not consume any of the free blocks. However, this also means that the format-time options should be conservative, as it is not possible to increase the number of inodes after the file system is formatted. It is possible to add OSTs with additional space and inodes to the file system. To be on the safe side, plan for 4 KB per inode on the MDT. This is the default value. For the OST, the amount of space taken by each object depends entirely upon the usage pattern of the users/applications running on the system. Lustre, by necessity, defaults to a very conservative estimate for the object size (16 KB per object). You can almost always increase this value for file system installations. Many Lustre file systems have average file sizes over 1 MB per object. 20.3.2 Sizing the MDT When calculating the MDS size, the only important factor is the average size of files to be stored in the file system. If the average file size is, for example, 5 MB and you have 100 TB of usable OST space, then you need at least (100 TB * 1024 GB/TB * 1024 MB/GB / 5 MB/inode) = 20 million inodes. We recommend that you have twice the minimum, that is, 40 million inodes in this example. At the default 4 KB per inode, this works out to only 160 GB of space for the MDS. Conversely, if you have a very small average file size, 4 KB for example, Lustre is not very efficient. This is because you consume as much space on the MDS as on the OSTs. This is not a very common configuration for Lustre.20-6 Lustre 1.8 Operations Manual • December 2010 20.4 Overriding Default Formatting Options To override the default formatting options for any of the Lustre backing file systems, use the --mkfsoptions='backing fs options' argument to mkfs.lustre to pass formatting options to the backing mkfs. For all options to format backing ext3 and ldiskfs filesystems, see the mke2fs(8) man page; this section only discusses several Lustre-specific options. 20.4.1 Number of Inodes for the MDS The number of inodes on the MDS is determined at format time based on the total size of the file system to be created. The default MDS inode ratio is one inode for every 4096 bytes of file system space. To override the inode ratio, use the option -i . For example, use --mkfsoptions="-i 4096" to create one inode per 4096 bytes of file system space. Alternately, if you are specifying an absolute number of inodes, use the -N option. You should not specify the -i option with an inode ratio below one inode per 1024 bytes in order to avoid unintentional mistakes. Instead, use the -N option. For example, by default, a 2 TB MDS will have 512M inodes. The largest currently-supported file system size is 16 TB, which would hold 4B inodes, the maximum possible number of inodes with ldiskfs. With an MDS inode ratio of 1024 bytes per inode, a 2 TB MDS would hold 2B inodes, and a 4 TB MDS would hold 4B inodes, which is the maximum number of inodes currently supported by ext3. 20.4.2 Inode Size for the MDS Lustre uses "large" inodes on backing file systems to efficiently store Lustre metadata with each file. On the MDS, each inode is at least 512 bytes in size (by default), while on the OST each inode is 256 bytes in size. Lustre (or more specifically the backing ext3 file system), also needs sufficient space left for other metadata like the journal (up to 400 MB), bitmaps and directories. There are also a few regular files that Lustre uses to maintain cluster consistency. To specify a larger inode size, use the -I option. We do NOT recommend specifying a smaller-than-default inode size, as this can lead to serious performance problems; and you cannot change this parameter after formatting the file system. The inode ratio must always be larger than the inode size.Chapter 20 Lustre Tuning 20-7 20.4.3 Number of Inodes for an OST For OST file systems, it is normally advantageous to take local file system usage into account. Try to minimize the number of inodes on each OST, while keeping enough margin for potential variance in future usage. This helps reduce the format and e2fsck time, and makes more space available for data. The current default is to create one inode per 16 KB of space in the OST file system, but in many environments, this is far too many inodes for the average file size. As a good rule of thumb, the OSTs should have at least: num_ost_inodes = 4 * * / You can specify the number of inodes on the OST file systems via the -N option to --mkfs options. Alternately, if you know the average file size, then you can also specify the OST inode count for the OST file systems via -i . For example, if the average file size is 16 MB and there are, by default 4 stripes per file, then --mkfsoptions='-i 1048576' would be appropriate. Note – In addition to the number of inodes, e2fsck runtime on OSTs is affected by a number of other variables: size of the file system, number of allocated blocks, distribution of allocated blocks on the disk, disk speed, CPU speed, and amount of RAM on the server. Reasonable e2fsck runtimes (without serious file system problems), are expected to take five minutes to two hours. For more details on formatting MDT and OST file systems, see Formatting Options for RAID Devices.20-8 Lustre 1.8 Operations Manual • December 2010 20.5 Large-Scale Tuning for Cray XT and Equivalents This section only applies to Cray XT3 Catamount nodes, and explains parameters used with the kptllnd module. If it does not apply to your setup, ignore it. 20.5.1 Network Tunables With a large number of clients and servers possible on these systems, tuning various request pools becomes important. We are making changes to the ptllnd module. Parameter Description max_nodes max_nodes is the maximum number of queue pairs, and, therefore, the maximum number of peers with which the LND instance can communicate. Set max_nodes to a value higher than the product of the total number of nodes and maximum processes per node. Max nodes > (Total # Nodes) * (max_procs_per_node) Setting max_nodes to a lower value than described causes Lustre to throw an error. Setting max_nodes to a higher value, causes excess memory to be consumed. max_procs_per_node max_procs_per_node is the maximum number of cores (CPUs), on a single Catamount node. Portals must know this value to properly clean up various queues. LNET is not notified directly when a Catamount process aborts. The first information LNET receives is when a new Catamount process with the same Cray portals NID starts and sends a connection request. If the number of processes with that Cray portals NID exceeds the max_procs_per_node value, LNET removes the oldest one to make space for the new one.Chapter 20 Lustre Tuning 20-9 20.6 Lockless I/O Tunables The lockless I/O tunable feature allows servers to ask clients to do lockless I/O (liblustre-style where the server does the locking) on contended files. The lockless I/O patch introduces these tunables: ¦ OST-side: /proc/fs/lustre/ldlm/namespaces/filter-lustre-* contended_locks - If the number of lock conflicts in the scan of granted and waiting queues at contended_locks is exceeded, the resource is considered to be contended. contention_seconds - The resource keeps itself in a contended state as set in the parameter. max_nolock_bytes - Server-side locking set only for requests less than the blocks set in the max_nolock_bytes parameter. If this tunable is set to zero (0), it disables server-side locking for read/write requests. ¦ Client-side: /proc/fs/lustre/llite/lustre-* contention_seconds - llite inode remembers its contended state for the time specified in this parameter. These two tunables combine to set the size of the ptllnd request buffer pool. The buffer pool must never drop an incoming message, so proper sizing is very important. Ntx Ntx helps to size the transmit (tx) descriptor pool. A tx descriptor is used for each send and each passive RDMA. The max number of concurrent sends == 'credits'. Passive RDMA is a response to a PUT or GET of a payload that is too big to fit in a small message buffer. For servers, this only happens on large RPCs (for instance, where a long file name is included), so the MDS could be under pressure in a large cluster. For routers, this is bounded by the number of servers. If the tx pool is exhausted, a console error message appears. Credits Credits determine how many sends are in-flight at once on ptllnd. Optimally, there are 8 requests in-flight per server. The default value is 128, which should be adequate for most applications. Parameter Description20-10 Lustre 1.8 Operations Manual • December 2010 ¦ Client-side statistics: The /proc/fs/lustre/llite/lustre-*/stats file has new rows for lockless I/O statistics. lockless_read_bytes and lockless_write_bytes - To count the total bytes read or written, the client makes its own decisions based on the request size. The client does not communicate with the server if the request size is smaller than the min_nolock_size, without acquiring locks by the client. 20.7 Data Checksums To avoid the risk of data corruption on the network, a Lustre client can perform end-to-end data checksums 1 . Be aware that at high data rates, checksumming can impact Lustre performance. 1. This feature computes a 32-bit checksum of data read or written on both the client and server, and ensures that the data has not been corrupted in transit over the network.21-1 C H A P T E R 21 LustreProc This chapter describes Lustre /proc entries and includes the following sections: ¦ Proc Entries for Lustre ¦ Lustre I/O Tunables ¦ Debug Support The proc file system acts as an interface to internal data structures in the kernel. Proc variables can be used to control aspects of Lustre performance and provide information.21-2 Lustre 1.8 Operations Manual • December 2010 21.1 Proc Entries for Lustre This section describes /proc entries for Lustre. 21.1.1 Locating Lustre File Systems and Servers Use the proc files on the MGS to locate the following: ¦ All known file systems # cat /proc/fs/lustre/mgs/MGS/filesystems spfs lustre ¦ The server names participating in a file system (for each file system that has at least one server running) # cat /proc/fs/lustre/mgs/MGS/live/spfs fsname: spfs flags: 0x0 gen: 7 spfs-MDT0000 spfs-OST0000 All servers are named according to this convention: - This can be shown for live servers under /proc/fs/lustre/devices: # cat /proc/fs/lustre/devices 0 UP mgs MGS MGS 11 1 UP mgc MGC192.168.10.34@tcp 1f45bb57-d9be-2ddb-c0b0-5431a49226705 2 UP mdt MDS MDS_uuid 3 3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4 4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 7 5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5 6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5 7 UP lov lustre-clilov-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa04 8 UP mdc lustre-MDT0000-mdc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05 9 UP osc lustre-OST0000-osc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05 10 UP osc lustre-OST0001-osc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05Chapter 21 LustreProc 21-3 Or from the device label at any time: # e2label /dev/sda lustre-MDT0000 21.1.2 Lustre Timeouts Lustre uses two types of timeouts. ¦ LND timeouts that ensure point-to-point communications complete in finite time in the presence of failures. These timeouts are logged with the S_LND flag set. They may not be printed as console messages, so you should check the Lustre log for D_NETERROR messages, or enable printing of D_NETERROR messages to the console (echo + neterror > /proc/sys/lnet/printk). Congested routers can be a source of spurious LND timeouts. To avoid this, increase the number of LNET router buffers to reduce back-pressure and/or increase LND timeouts on all nodes on all connected networks. You should also consider increasing the total number of LNET router nodes in the system so that the aggregate router bandwidth matches the aggregate server bandwidth. ¦ Lustre timeouts that ensure Lustre RPCs complete in finite time in the presence of failures. These timeouts should always be printed as console messages. If Lustre timeouts are not accompanied by LNET timeouts, then you need to increase the lustre timeout on both servers and clients. Specific Lustre timeouts are described below. /proc/sys/lustre/timeout This is the time period that a client waits for a server to complete an RPC (default is 100s). Servers wait half of this time for a normal client RPC to complete and a quarter of this time for a single bulk request (read or write of up to 1 MB) to complete. The client pings recoverable targets (MDS and OSTs) at one quarter of the timeout, and the server waits one and a half times the timeout before evicting a client for being "stale." Note – Lustre sends periodic ‘PING’ messages to servers with which it had no communication for a specified period of time. Any network activity on the file system that triggers network traffic toward servers also works as a health check. /proc/sys/lustre/ldlm_timeout This is the time period for which a server will wait for a client to reply to an initial AST (lock cancellation request) where default is 20s for an OST and 6s for an MDS. If the client replies to the AST, the server will give it a normal timeout (half of the client timeout) to flush any dirty data and release the lock.21-4 Lustre 1.8 Operations Manual • December 2010 Note – When adaptive timeouts are enabled, the ldlm_timeout tunable is not used. /proc/sys/lustre/fail_loc This is the internal debugging failure hook. See lustre/include/linux/obd_support.h for the definitions of individual failure locations. The default value is 0 (zero). sysctl -w lustre.fail_loc=0x80000122 # drop a single reply /proc/sys/lustre/dump_on_timeout This triggers dumps of the Lustre debug log when timeouts occur. The default value is 0 (zero). /proc/sys/lustre/dump_on_eviction This triggers dumps of the Lustre debug log when an eviction occurs. The default value is 0 (zero). By default, debug logs are dumped to the /tmp folder; this location can be changed via /proc.Chapter 21 LustreProc 21-5 21.1.3 Adaptive Timeouts Lustre 1.8 introduces an adaptive mechanism to set RPC timeouts. This feature causes servers to track actual RPC completion times, and to report estimated completion times for future RPCs back to clients. The clients use these estimates to set their future RPC timeout values. If server request processing slows down for any reason, the RPC completion estimates increase, and the clients allow more time for RPC completion. If RPCs queued on the server approach their timeouts, then the server sends an early reply to the client, telling the client to allow more time. In this manner, clients avoid RPC timeouts and disconnect/reconnect cycles. Conversely, as a server speeds up, RPC timeout values decrease, allowing faster detection of non-responsive servers and faster attempts to reconnect to a server's failover partner. Note – In Lustre 1.8, adaptive timeouts are enabled, by default. In earlier Lustre versions supporting adaptive timeouts (1.6.5 through 1.6.7.x), this feature was disabled, by default. In previous Lustre versions, the static obd_timeout (/proc/sys/lustre/timeout) value was used as the maximum completion time for all RPCs; this value also affected the client-server ping interval and initial recovery timer. Now, with adaptive timeouts, obd_timeout is only used for the ping interval and initial recovery estimate. When a client reconnects during recovery, the server uses the client's timeout value to reset the recovery wait period; i.e., the server learns how long the client had been willing to wait, and takes this into account when adjusting the recovery period.21-6 Lustre 1.8 Operations Manual • December 2010 21.1.3.1 Configuring Adaptive Timeouts One of the goals of adaptive timeouts is to relieve users from having to tune the obd_timeout value. In general, obd_timeout should no longer need to be changed. However, there are several parameters related to adaptive timeouts that users can set. In most situations, the default values should be used. The following parameters can be set persistently system-wide using lctl conf_param on the MGS. For example, lctl conf_param work1.sys.at_max= 1500 sets the at_max value for all servers and clients using the work1 file system. Note – Nodes using multiple Lustre file systems must use the same at_* values for all file systems.) Parameter Description at_min Sets the minimum adaptive timeout (in seconds). Default value is 0. The at_min parameter is the minimum processing time that a server will report. Clients base their timeouts on this value, but they do not use this value directly. If you experience cases in which, for unknown reasons, the adaptive timeout value is too short and clients time out their RPCs (usually due to temporary network outages), then you can increase the at_min value to compensate for this. Ideally, users should leave at_min set to its default. at_max Sets the maximum adaptive timeout (in seconds). The at_max parameter is an upper-limit on the service time estimate, and is used as a 'failsafe' in case of rogue/bad/buggy code that would lead to never-ending estimate increases. If at_max is reached, an RPC request is considered 'broken' and should time out. Setting at_max to 0 causes adaptive timeouts to be disabled and the old fixed-timeout method (obd_timeout) to be used. This is the default value in Lustre 1.6.5. NOTE: It is possible that slow hardware might validly cause the service estimate to increase beyond the default value of at_max. In this case, you should increase at_max to the maximum time you are willing to wait for an RPC completion. at_history Sets a time period (in seconds) within which adaptive timeouts remember the slowest event that occurred. Default value is 600.Chapter 21 LustreProc 21-7 In Lustre 1.8, adaptive timeouts are enabled, by default. To disable adaptive timeouts, at run time, set at_max to 0. On the MGS, run: $ lctl conf_param .sys.at_max=0 Note – Changing adaptive timeouts status at runtime may cause transient timeout, reconnect, recovery, etc. at_early_margin Sets how far before the deadline Lustre sends an early reply. Default value is 5 * . at_extra Sets the incremental amount of time that a server asks for, with each early reply. The server does not know how much time the RPC will take, so it asks for a fixed value. Default value is 30 † . When a server finds a queued request about to time out (and needs to send an early reply out), the server adds the at_extra value. If the time expires, the Lustre client enters recovery status and reconnects to restore it to normal status. If you see multiple early replies for the same RPC asking for multiple 30-second increases, change the at_extra value to a larger number to cut down on early replies sent and, therefore, network load. ldlm_enqueue_min Sets the minimum lock enqueue time. Default value is 100. The ldlm_enqueue time is the maximum of the measured enqueue estimate (influenced by at_min and at_max parameters), multiplied by a weighting factor, and the ldlm_enqueue_min setting. LDLM lock enqueues were based on the obd_timeout value; now they have a dedicated minimum value. Lock enqueues increase as the measured enqueue times increase (similar to adaptive timeouts). ‡ * This default was chosen as a reasonable time in which to send a reply from the point at which it was sent. † This default was chosen as a balance between sending too many early replies for the same RPC and overestimating the actual completion time. ‡ Currently, the only way to change the ldlm_enqueue_min value is as a module parameter to the ptlrpc module (i.e., not with a conf_param setting). For example, adding a line "options ptlrpc ldlm_enqueue_min=200" to /etc/modprobe.conf will permanently change it; "echo 200 >>/sys/module/ptlrpc/parameters/ldlm_enqueue_min" will temporarily change it. Parameter Description21-8 Lustre 1.8 Operations Manual • December 2010 21.1.3.2 Interpreting Adaptive Timeouts Information Adaptive timeouts information can be read from /proc/fs/lustre/*/timeouts files (for each service and client) or with the lctl command. This is an example from the /proc/fs/lustre/*/timeouts files: cfs21:~# cat /proc/fs/lustre/ost/OSS/ost_io/timeouts This is an example using the lctl command: $ lctl get_param -n ost.*.ost_io.timeouts This is the sample output: service : cur 33 worst 34 (at 1193427052, 0d0h26m40s ago) 1 1 33 2 The ost_io service on this node is currently reporting an estimate of 33 seconds. The worst RPC service time was 34 seconds, and it happened 26 minutes ago. The output also provides a history of service times. In the example, there are 4 "bins" of adaptive_timeout_history, with the maximum RPC time in each bin reported. In 0-150 seconds, the maximum RPC time was 1, with the same result in 150-300 seconds. From 300-450 seconds, the worst (maximum) RPC time was 33 seconds, and from 450-600s the worst time was 2 seconds. The current estimated service time is the maximum value of the 4 bins (33 seconds in this example). Service times (as reported by the servers) are also tracked in the client OBDs: cfs21:# lctl get_param osc.*.timeouts last reply : 1193428639, 0d0h00m00s ago network : cur 1 worst 2 (at 1193427053, 0d0h26m26s ago) 1 1 1 1 portal 6 : cur 33 worst 34 (at 1193427052, 0d0h26m27s ago) 33 33 33 2 portal 28 : cur 1 worst 1 (at 1193426141, 0d0h41m38s ago) 1 1 1 1 portal 7 : cur 1 worst 1 (at 1193426141, 0d0h41m38s ago) 1 0 1 1 portal 17 : cur 1 worst 1 (at 1193426177, 0d0h41m02s ago) 1 0 0 1 In this case, RPCs to portal 6, the OST_IO_PORTAL (see lustre/include/lustre/lustre_idl.h), shows the history of what the ost_io portal has reported as the service estimate.Chapter 21 LustreProc 21-9 Server statistic files also show the range of estimates in the normal min/max/sum/sumsq manner. cfs21:~# lctl get_param mdt.*.mdt.stats ... req_timeout 6 samples [sec] 1 10 15 105 ... 21.1.4 LNET Information This section describes /proc entries for LNET information. /proc/sys/lnet/peers Shows all NIDs known to this node and also gives information on the queue state. # cat /proc/sys/lnet/peers nid refs state max rtr min tx min queue 0@lo 1 ~rtr 0 0 0 0 0 0 192.168.10.35@tcp1 ~rtr 8 8 8 8 6 0 192.168.10.36@tcp1 ~rtr 8 8 8 8 6 0 192.168.10.37@tcp1 ~rtr 8 8 8 8 6 0 The fields are explained below: Field Description refs A reference count (principally used for debugging) state Only valid to refer to routers. Possible values: • ~ rtr (indicates this node is not a router) • up/down (indicates this node is a router) • auto_fail must be enabled max Maximum number of concurrent sends from this peer rtr Routing buffer credits. min Minimum routing buffer credits seen. tx Send credits. min Minimum send credits seen. queue Total bytes in active/queued sends.21-10 Lustre 1.8 Operations Manual • December 2010 Credits work like a semaphore. At start they are initialized to allow a certain number of operations (8 in this example). LNET keeps a track of the minimum value so that you can see how congested a resource was. If rtr/tx is less than max, there are operations in progress. The number of operations is equal to rtr or tx subtracted from max. If rtr/tx is greater that max, there are operations blocking. LNET also limits concurrent sends and router buffers allocated to a single peer so that no peer can occupy all these resources. /proc/sys/lnet/nis # cat /proc/sys/lnet/nis nid refs peer max tx min 0@lo 3 0 0 0 0 192.168.10.34@tcp 4 8 256 256 252 Shows the current queue health on this node. The fields are explained below: Subtracting max – tx yields the number of sends currently active. A large or increasing number of active sends may indicate a problem. # cat /proc/sys/lnet/nis nid refs peer max tx min 0@lo 2 0 0 0 0 10.67.73.173@tcp 4 8 256 256 253 Field Description nid Network interface refs Internal reference counter peer Number of peer-to-peer send credits on this NID. Credits are used to size buffer pools max Total number of send credits on this NID. tx Current number of send credits available on this NID. min Lowest number of send credits available on this NID. queue Total bytes in active/queued sends.Chapter 21 LustreProc 21-11 21.1.5 Free Space Distribution Free-space stripe weighting, as set, gives a priority of "0" to free space (versus trying to place the stripes "widely" -- nicely distributed across OSSs and OSTs to maximize network balancing). To adjust this priority (as a percentage), use the qos_prio_free proc tunable: $ cat /proc/fs/lustre/lov/-mdtlov/qos_prio_free Currently, the default is 90%. You can permanently set this value by running this command on the MGS: $ lctl conf_param -MDT0000.lov.qos_prio_free=90 Setting the priority to 100% means that OSS distribution does not count in the weighting, but the stripe assignment is still done via weighting. If OST 2 has twice as much free space as OST 1, it is twice as likely to be used, but it is NOT guaranteed to be used. Also note that free-space stripe weighting does not activate until two OSTs are imbalanced by more than 20%. Until then, a faster round-robin stripe allocator is used. (The new round-robin order also maximizes network balancing.) 21.1.5.1 Managing Stripe Allocation The MDS uses two methods to manage stripe allocation and determine which OSTs to use for file object storage: ¦ QOS Quality of Service (QOS) considers an OST’s available blocks, speed, and the number of existing objects, etc. Using these criteria, the MDS selects OSTs with more free space more often than OSTs with less free space. ¦ RR Round-Robin (RR) allocates objects evenly across all OSTs. The RR stripe allocator is faster than QOS, and used often because it distributes space usage/load best in most situations, maximizing network balancing and improving performance. Whether QOS or RR is used depends on the setting of the qos_threshold_rr proc tunable. The qos_threshold_rr variable specifies a percentage threshold where the use of QOS or RR becomes more/less likely. The qos_threshold_rr tunable can be set as an integer, from 0 to 100, and results in this stripe allocation behavior: ¦ If qos_threshold_rr is set to 0, then QOS is always used ¦ If qos_threshold_rr is set to 100, then RR is always used ¦ The larger the qos_threshold_rr setting, the greater the possibility that RR is used instead of QOS21-12 Lustre 1.8 Operations Manual • December 2010 21.2 Lustre I/O Tunables The section describes I/O tunables. /proc/fs/lustre/llite/-/max_cache_mb # cat /proc/fs/lustre/llite/lustre-ce63ca00/max_cached_mb 128 This tunable is the maximum amount of inactive data cached by the client (default is 3/4 of RAM). 21.2.1 Client I/O RPC Stream Tunables The Lustre engine always attempts to pack an optimal amount of data into each I/O RPC and attempts to keep a consistent number of issued RPCs in progress at a time. Lustre exposes several tuning variables to adjust behavior according to network conditions and cluster size. Each OSC has its own tree of these tunables. For example: $ ls -d /proc/fs/lustre/osc/OSC_client_ost1_MNT_client_2 /localhost /proc/fs/lustre/osc/OSC_uml0_ost1_MNT_localhost /proc/fs/lustre/osc/OSC_uml0_ost2_MNT_localhost /proc/fs/lustre/osc/OSC_uml0_ost3_MNT_localhost $ ls /proc/fs/lustre/osc/OSC_uml0_ost1_MNT_localhost blocksizefilesfreemax_dirty_mb ost_server_uuid stats ... and so on. RPC stream tunables are described below. /proc/fs/lustre/osc/