Les cookies nous permettent de vous proposer nos services plus facilement. En utilisant nos services, vous nous donnez expressément votre accord pour exploiter ces cookies.En savoir plus OK

Clustering and IASPs for Higher Availability on the IBM iSeries Server

Au format "texte" :

ibm.com/redbooks Domino for iSeries Sizing and Performance Tuning Wilfried Blankertz Christina Fasth Kim Greene Deb Landon Brandon Rau Colin Stamp Correctly sizing a server with the Workload Estimator Tips and techniques for tuning Domino for iSeries Information on iSeries Dedicated Server for Domino Front cover International Technical Support Organization Domino for iSeries Sizing and Performance Tuning April 2002 SG24-5162-01 © Copyright International Business Machines Corporation 1999, 2002. All rights reserved. Note to U.S Government Users - Documentation related to restricted rights - Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp. Second Edition (April 2002) This edition applies to Release 5.0.8 of Domino for iSeries for use with OS/400 Version 5 Release 1 (V5R1) and later. The first edition of this redbook was published in October 1999 and was originally entitled Lotus Domino for AS/400: Performance, Tuning, and Capacity Planning. It applied to Release 4.6.3 of Domino for AS/400 for use with OS/400 Version 4 Release 2 and later. Comments may be addressed to: IBM Corporation, International Technical Support Organization Dept. JLU Building 107-2 3605 Highway 52N Rochester, Minnesota 55901-7829 When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you. Take Note! Before using this information and the product it supports, be sure to read the general information in “Special notices” on page xi. © Copyright IBM Corp. 1999, 2002 iii Contents Special notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi IBM trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xv The team that wrote this edition of the Redbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xv The team that wrote the first edition of this redbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Special notice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii Summary of changes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Second Edition, April 2002. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Chapter 1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Conclusions and references . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Chapter 2. Overview of Lotus Domino for iSeries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.1 Why use Domino for iSeries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1.1 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1.2 Reliability and availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1.3 Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1.4 Proven security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 Planning a Domino for iSeries server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2.1 Developing a Notes naming structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2.2 Choosing the right replication topology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2.3 How security relates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2.4 What remains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.3 The value of sizing, performance analysis, and tuning . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3.1 Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.4 Domino architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.4.1 Layer 0: The kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.4.2 Layer 1: Kernel extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.4.3 Layer 2: System and core applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.4.4 Layer 3: Layered end-user applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.5 Tools available . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.5.1 Collection Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.2 Performance Management/400 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.3 Performance Tools (5722-PT1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.4 iSeries Performance Explorer (PEX). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.5.5 Management Central . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.5.6 BEST/1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.5.7 Server.Load . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.5.8 Performance Navigator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.5.9 NotesBench . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.5.10 GroupSizr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.5.11 WEBSizr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.5.12 LoadRunner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.5.13 Tivoli Manager for Domino . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.5.14 Tivoli Application Response Measurement (ARM) . . . . . . . . . . . . . . . . . . . . . . . 16 iv Domino for iSeries Sizing and Performance Tuning 2.6 NSD for OS/400 (Dump utility) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.6.1 Activating the NSD tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Chapter 3. Sizing Domino for iSeries using the Workload Estimator . . . . . . . . . . . . . 19 3.1 Introduction to Workload Estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.2 Benchmarks to determine the load of specific workloads . . . . . . . . . . . . . . . . . . . . . . . 21 3.3 Workload Estimator terms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.3.1 Workload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.3.2 Instance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.4 Domino sizing concepts in the Workload Estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.4.1 Concurrent users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.4.2 Number of partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.4.3 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.5 Domino mail concepts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.5.1 Mail access types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.6 Domino application concepts in Workload Estimator . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.6.1 Example application comparisons. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.6.2 Database capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.7 Characterizing existing applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.8 Additional options for more accurate estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.9 Using IBM Workload Estimator for iSeries results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.10 Additional tips and insights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.11 Consolidating Domino servers from other platforms . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.11.1 Capabilities of various iSeries and AS/400 models. . . . . . . . . . . . . . . . . . . . . . . 48 3.12 Additional resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Chapter 4. Basic concepts of performance analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.1 Defining a Domino for iSeries performance methodology. . . . . . . . . . . . . . . . . . . . . . . 54 4.1.1 Relationship between Domino and the iSeries server . . . . . . . . . . . . . . . . . . . . . 55 4.2 Understanding Domino on the iSeries server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.2.1 Transactions in Domino . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.2.2 Domino base functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.2.3 Domino advanced functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.3 Domino tasks on the iSeries server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.4 Application development performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.5 Lotus Domino overall performance on the iSeries server . . . . . . . . . . . . . . . . . . . . . . . 59 4.6 The Queuing Multiplier (QM) curve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.7 Dividing the CPU utilization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.8 A brief discussion of threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 4.8.1 Threads on the iSeries server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.8.2 Other Domino task that use threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 4.9 A brief discussion on run attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 4.9.1 Run priority . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 4.9.2 How a change in run priority affects a job. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.9.3 Job priority considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.9.4 Time slice parameter and tuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.9.5 How the system manages job and run priorities . . . . . . . . . . . . . . . . . . . . . . . . . . 67 4.10 Reviewing iSeries performance using CL commands. . . . . . . . . . . . . . . . . . . . . . . . . 67 4.10.1 Work with System Status (WRKSYSSTS) command . . . . . . . . . . . . . . . . . . . . . 68 4.10.2 Information about thread state transitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.10.3 Work with Active Jobs (WRKACTJOB) command . . . . . . . . . . . . . . . . . . . . . . . 70 4.10.4 Work with System Activity (WRKSYSACT) command . . . . . . . . . . . . . . . . . . . . 71 4.10.5 Work with Disk Status (WRKDSKSTS) command . . . . . . . . . . . . . . . . . . . . . . . 73 v 4.10.6 Observing network performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 4.10.7 Work with System Values (WRKSYSVAL) command. . . . . . . . . . . . . . . . . . . . . 79 4.10.8 Display System Log (DSPLOG) command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 4.10.9 Start System Service Tools (STRSST) command . . . . . . . . . . . . . . . . . . . . . . . 82 4.11 Collecting performance data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.11.1 Starting Collection Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.11.2 Using Work with System Activity (WRKSYSACT) with an output file . . . . . . . . . 89 4.12 Management Central performance monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 4.12.1 Monitoring real-time system performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 4.12.2 Viewing graph history of monitor data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 4.12.3 Exporting Graph History to PC files for analysis and printing . . . . . . . . . . . . . . 103 4.12.4 Combining Management Central’s CPU Utilization metrics . . . . . . . . . . . . . . . 104 4.12.5 Job monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 4.13 Reviewing data using Performance Tools/400 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 4.13.1 Print System Report (PRTSYSRPT) command . . . . . . . . . . . . . . . . . . . . . . . . 114 4.13.2 Print Component Report (PRTCPTRPT) command . . . . . . . . . . . . . . . . . . . . . 114 4.13.3 Print Transaction Report (PRTTNSRPT) command . . . . . . . . . . . . . . . . . . . . . 114 4.13.4 Print Activity Report (PRTACTRPT) command. . . . . . . . . . . . . . . . . . . . . . . . . 115 4.13.5 Convert Performance Thread Data (CVTPFRTHD) command . . . . . . . . . . . . . 116 4.14 Performance Explorer (PEX). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 4.14.1 Add PEX Definition (ADDPEXDFN) command . . . . . . . . . . . . . . . . . . . . . . . . . 116 4.14.2 Start PEX Session (STRPEX) command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 4.14.3 End PEX Session (ENDPEX) command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 4.14.4 Print PEX Report (PRTPEXRPT) command. . . . . . . . . . . . . . . . . . . . . . . . . . . 117 4.14.5 Performance Explorer trace points for Domino. . . . . . . . . . . . . . . . . . . . . . . . . 117 4.14.6 TPROF PEX trace. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Chapter 5. Domino logs and statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 5.1 The commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 5.1.1 The show tasks command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 5.1.2 Understanding the Domino statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 5.1.3 Show Domino open database statistics (sh DBS). . . . . . . . . . . . . . . . . . . . . . . . 136 5.1.4 Platform-dependent statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 5.1.5 Show Domino Transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 5.2 The reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 5.2.1 The Log database (log.nsf) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 5.2.2 The Statistics and Reporting database (statrep.nsf). . . . . . . . . . . . . . . . . . . . . . 149 5.3 Combining performance data from Domino and iSeries . . . . . . . . . . . . . . . . . . . . . . . 152 5.4 Monitoring the Notes Remote Procedure Calls (NRPC) . . . . . . . . . . . . . . . . . . . . . . . 156 5.4.1 Using a network sniffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 5.4.2 Lotus Client NRPC Monitoring Tool Client_Clock. . . . . . . . . . . . . . . . . . . . . . . . 157 Chapter 6. Tuning the iSeries server for Lotus Domino . . . . . . . . . . . . . . . . . . . . . . . 161 6.1 Tuning the iSeries server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 6.2 Automatic performance adjustment (QPFRADJ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 6.2.1 The interactive iSeries tuning chart. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 6.2.2 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 6.3 Determining Domino workload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 6.4 Choosing which memory pool to use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 6.4.1 Using expert cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 6.4.2 Activity levels of memory pools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 6.4.3 Faulting rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 6.5 Choosing which processor priority to use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 vi Domino for iSeries Sizing and Performance Tuning 6.5.1 Changing the run priority of Domino tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 6.6 DASD: System ASP, user ASPs, and independent ASPs . . . . . . . . . . . . . . . . . . . . . 175 6.7 Unleashing iSeries work management on Domino servers. . . . . . . . . . . . . . . . . . . . . 176 6.7.1 Configuring a Domino server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 6.7.2 Starting a Domino server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 6.7.3 Working with Domino jobs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 6.8 Online backup with BRMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 6.9 Network tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 6.9.1 Maximum Transmission Unit (MTU) Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 6.9.2 TCP/IP buffer size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 6.9.3 Port filtering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 6.9.4 Duplex. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 6.9.5 TCPONLY for Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 Chapter 7. Tuning Lotus Domino for better performance on iSeries . . . . . . . . . . . . . 183 7.1 Basics of Domino tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 7.2 Relating Domino tasks to iSeries jobs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 7.3 Domino parameters that affect performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 7.4 Domino Database indexing: Controlling the UPDATE task. . . . . . . . . . . . . . . . . . . . . 185 7.4.1 Components of the Domino database indexer . . . . . . . . . . . . . . . . . . . . . . . . . . 185 7.4.2 UPDATE task CPU utilization in a normal environment . . . . . . . . . . . . . . . . . . . 188 7.4.3 The UPDATE task and CPU utilization in a problem environment . . . . . . . . . . . 188 7.4.4 Affecting the UPDATE task behavior through the view properties . . . . . . . . . . . 190 7.4.5 Detecting UPDATE task problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 7.4.6 Making the changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 7.4.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 7.5 TCP/IP between Domino servers on the same iSeries . . . . . . . . . . . . . . . . . . . . . . . . 199 7.6 GTR search engine version 3.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 7.7 Using the Extended Directory Catalog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 7.7.1 Comparing EDC to a standard directory catalog (DirCat). . . . . . . . . . . . . . . . . . 199 7.7.2 Using Extended Directory Catalog to improve performance. . . . . . . . . . . . . . . . 200 7.8 Investigating mail backlogs and performance issues . . . . . . . . . . . . . . . . . . . . . . . . . 200 7.9 Domino console logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 7.10 Number of users per Domino server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 7.11 Using a separate partition for the R5 SMTP MTA server . . . . . . . . . . . . . . . . . . . . . 202 7.12 Excessive translations on iSeries and zSeries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 7.12.1 Opening up any suspect job using the new Domino panel group. . . . . . . . . . . 202 7.13 Optimizing mail: White space, compact, and quotas . . . . . . . . . . . . . . . . . . . . . . . . 205 7.13.1 Mail file size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 7.14 Domino memory management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 7.14.1 NSF_Buffer_Pool_Size or Nsf_Buffer_Pool_Size_mb . . . . . . . . . . . . . . . . . . . 211 7.14.2 How queues relate to the NSF_Buffer_Pool_Size . . . . . . . . . . . . . . . . . . . . . . 219 7.15 Controlling the details of Domino logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 7.16 iSeries environment variable settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 7.17 Domino for iSeries specific notes.ini parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 7.18 CORBA and Java servlets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 Chapter 8. Understanding the Domino server jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 8.1 Domino server jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 8.2 Domino server jobs (always necessary) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 8.2.1 ADMINP: Administration process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 8.2.2 AMGR: Agent manager. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 8.2.3 QNNINSTS: The ‘Watchdog’. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 vii 8.2.4 ROUTER: The mail router. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 8.2.5 SERVER: The main server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 8.2.6 STATLOG: Database activity logger. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244 8.2.7 UPDATE: The Indexer task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244 8.3 Domino server jobs (often used) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 8.3.1 CLADMIN: Cluster administration process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 8.3.2 CLDBDIR: Cluster Database Directory Manager . . . . . . . . . . . . . . . . . . . . . . . . 246 8.3.3 CLREPL: Cluster Replicator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 8.3.4 COLLECT: Statistics collector. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 8.3.5 EVENT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 8.3.6 HTTP: The Web server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 8.3.7 LOGASIO: Log asynchronous I/O process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 8.3.8 REPLICA: The replication task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 8.3.9 SCHED: Schedule Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 8.3.10 SMTP: Internet Mail Router. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 8.4 Domino server jobs (very likely not used) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 8.4.1 BILLING . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 8.4.2 CALCONN: Calendar connector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254 8.4.3 QNNINADD: Directory synchronization add-in task . . . . . . . . . . . . . . . . . . . . . . 255 8.4.4 REPORT: Statistic reporter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 8.4.5 STATS: Statistics on demand via e-mail. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 8.5 Domino server database housekeeping jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 8.5.1 COMPACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 8.5.2 UPDALL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 8.5.3 Fixup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 8.6 Series SMTP: AnyMail/400 Mail Server Framework (MSF) . . . . . . . . . . . . . . . . . . . . 258 8.7 Domino server QNNxxxx programs and QSYSWRK . . . . . . . . . . . . . . . . . . . . . . . . . 258 Chapter 9. Integration with DB2 performance tips and techniques . . . . . . . . . . . . . . 259 9.1 Choosing a storage container . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 9.1.1 Domino as a storage container . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 9.1.2 DB2 UDB as a data storage container . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 9.1.3 Why Domino does not scale like DB2 UDB . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262 9.2 Integrating Domino and DB2 UDB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 9.2.1 Integration options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 9.2.2 Connection pooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 9.3 Questions to consider when integrating Domino and DB2 . . . . . . . . . . . . . . . . . . . . . 272 Chapter 10. Clustering and partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 10.1 Domino partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274 10.1.1 Why partition Domino on iSeries servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274 10.1.2 How Domino partitioning works. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 10.1.3 Planning for Domino partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276 10.1.4 Partitioning tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 10.1.5 Communication between Domino servers on the same iSeries server. . . . . . . 278 10.2 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278 10.2.1 iSeries clustering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 10.2.2 ClusterProven Domino for iSeries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280 10.2.3 Domino clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 10.2.4 Why cluster Domino servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 10.2.5 How a Domino cluster works. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 10.2.6 ICM and the Domino HTTP Clustering components and technology . . . . . . . . 287 10.2.7 Planning for Domino clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 viii Domino for iSeries Sizing and Performance Tuning 10.2.8 Domino clustering limitations and recommendations . . . . . . . . . . . . . . . . . . . . 290 10.2.9 Mail database integration for high availability . . . . . . . . . . . . . . . . . . . . . . . . . . 291 10.2.10 Remote access with iNotes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 10.2.11 Application hub server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 10.2.12 Network performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 10.2.13 Server_Availability_Index and Server_Availability_Threshold . . . . . . . . . . . . 293 10.2.14 Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 10.2.15 Server-based tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 10.2.16 Clustering tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 Chapter 11. Internet and intranet performance tips. . . . . . . . . . . . . . . . . . . . . . . . . . . 301 11.1 Internet and intranet protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 11.2 HTTP performance tuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 11.2.1 HTTP terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 11.2.2 Optimizing HTTP threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 11.2.3 Optimizing HTTP cache settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308 11.2.4 Optimizing HTTP logging for performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310 11.2.5 Best practices for Domino HTTP server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310 11.2.6 Summary of HTTP issues, architecture, and tuning possibilities . . . . . . . . . . . 315 11.2.7 Performance tuning of time-out values on the Domino Web server . . . . . . . . . 315 11.2.8 Full Text Search information in a document over the Web . . . . . . . . . . . . . . . . 317 11.3 Structure of the SMTP MTA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 11.3.1 SMTP performance improvements in Domino R5. . . . . . . . . . . . . . . . . . . . . . . 319 11.3.2 Best practices for SMTP messaging using partitioning. . . . . . . . . . . . . . . . . . . 323 11.4 Other Internet and intranet protocols and functions . . . . . . . . . . . . . . . . . . . . . . . . . 327 11.4.1 IMAP or POP3 on Domino Server within a cluster and failover . . . . . . . . . . . . 327 11.4.2 IMAP server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 11.4.3 POP3 server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329 11.4.4 LDAP server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330 11.4.5 NNTP server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334 11.4.6 iNotes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337 Chapter 12. The iSeries Dedicated Server for Domino . . . . . . . . . . . . . . . . . . . . . . . . 341 12.1 Dedicated Server for Domino . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 12.2 Dedicated Server performance behavior and capacity . . . . . . . . . . . . . . . . . . . . . . . 342 12.2.1 Enhancements with OS/400 V5R1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 12.2.2 Dedicated Server for Domino workload environments . . . . . . . . . . . . . . . . . . . 345 12.2.3 Dedicated server capacity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348 12.3 Evaluating iSeries application integration on Dedicated Server for Domino . . . . . . . 350 12.3.1 DB2 database processing defined . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351 12.3.2 Accessing external databases on the same iSeries server. . . . . . . . . . . . . . . . 354 12.3.3 Accessing external databases on a different iSeries server . . . . . . . . . . . . . . . 355 12.3.4 Java and Domino integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356 12.3.5 WebSphere and Domino integration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356 12.3.6 File serving on a Dedicated Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 12.3.7 Backup Recovery and Media Services on a Dedicated Server. . . . . . . . . . . . . 358 12.3.8 Logical partitioning (LPAR) on a Dedicated Server. . . . . . . . . . . . . . . . . . . . . . 360 12.3.9 Linux on a Dedicated Server for Domino . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 12.4 Observing Dedicated Server performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362 12.4.1 Operations Navigator: Management Central. . . . . . . . . . . . . . . . . . . . . . . . . . . 362 12.5 Observing Domino processing with Performance Explorer. . . . . . . . . . . . . . . . . . . . 365 12.6 Sizing a Dedicated Server for your Domino application . . . . . . . . . . . . . . . . . . . . . . 368 12.6.1 Special tuning for the Dedicated Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369 ix Chapter 13. Domino transaction logging on iSeries . . . . . . . . . . . . . . . . . . . . . . . . . . 371 13.1 Storage management on iSeries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372 13.1.1 Single-level storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372 13.1.2 When data is actually written to disk. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373 13.1.3 Auxiliary storage pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373 13.2 Domino transaction logging and how it operates . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 13.2.1 Performance improvement due to Domino transaction logging . . . . . . . . . . . . 376 13.3 Performance tests with TxL on Domino for iSeries . . . . . . . . . . . . . . . . . . . . . . . . . . 381 13.3.1 Test environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381 13.3.2 Test results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382 13.3.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382 13.4 When to use Domino for iSeries transaction logging . . . . . . . . . . . . . . . . . . . . . . . . 384 13.4.1 Incremental online backup with BRMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384 13.5 Miscellaneous tips for Domino transaction logging . . . . . . . . . . . . . . . . . . . . . . . . . . 385 13.5.1 What is the Database Instance ID (DBIID) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385 13.5.2 How transaction logging technically works . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386 Appendix A. Capacity planning for Domino server using BEST/1 . . . . . . . . . . . . . . . 391 The need for capacity planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 Capacity planning prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 Basic capacity planning process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 CPW values versus Domino mail user ratings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 Capacity planning unit of measure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 Business transaction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 Domino server workload definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 What the different workloads are . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 Splitting up the workloads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 Other Domino workload concerns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 BEST/1 for Domino server capacity planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399 iSeries Domino server modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400 Measuring workload and system resource utilization . . . . . . . . . . . . . . . . . . . . . . . . . . 403 Creating a model using measured data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404 Validating the BEST/1 model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413 Saving the BEST/1 model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423 Appendix B. SMTP and AnyMail/400 Mail Server Framework (MSF) jobs . . . . . . . . . 427 Appendix C. Important notes.ini parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435 Steps for changing the notes.ini file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436 Description of notes.ini variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436 Appendix D. Domino R5 statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453 Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477 IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477 Other resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479 Referenced Web sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480 How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481 IBM Redbooks collections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483 x Domino for iSeries Sizing and Performance Tuning © Copyright IBM Corp. 1999, 2002 xi Special notices References in this publication to IBM products, programs or services do not imply that IBM intends to make these available in all countries in which IBM operates. Any reference to an IBM product, program, or service is not intended to state or imply that only IBM's product, program, or service may be used. Any functionally equivalent program that does not infringe any of IBM's intellectual property rights may be used instead of the IBM product, program or service. Information in this book was developed in conjunction with use of the equipment specified, and is limited in application to those specific hardware and software products and levels. IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to the IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785. Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact IBM Corporation, Dept. 600A, Mail Drop 1329, Somers, NY 10589 USA. Such information may be available, subject to appropriate terms and conditions, including in some cases, payment of a fee. The information contained in this document has not been submitted to any formal IBM test and is distributed AS IS. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk. Any pointers in this publication to external Web sites are provided for convenience only and do not in any manner serve as an endorsement of these Web sites. xii Domino for iSeries Sizing and Performance Tuning IBM trademarks The following terms are trademarks of the International Business Machines Corporation in the United States and/or other countries: Other company trademarks The following terms are trademarks of other companies: Tivoli, Manage. Anything. Anywhere.,The Power To Manage., Anything. Anywhere.,TME, NetView, Cross-Site, Tivoli Ready, Tivoli Certified, Planet Tivoli, and Tivoli Enterprise are trademarks or registered trademarks of Tivoli Systems Inc., an IBM company, in the United States, other countries, or both. In Denmark, Tivoli is a trademark licensed from Kjøbenhavns Sommer - Tivoli A/S. C-bus is a trademark of Corollary, Inc. in the United States and/or other countries. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and/or other countries. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States and/or other countries. PC Direct is a trademark of Ziff Communications Company in the United States and/or other countries and is used by IBM Corporation under license. ActionMedia, LANDesk, MMX, Pentium and ProShare are trademarks of Intel Corporation in the United States and/or other countries. UNIX is a registered trademark in the United States and other countries licensed exclusively through The Open Group. e (logo)® AS/400® CICS® DB2 Universal Database™ DRDA® IBM® MQSeries® Network Station™ OS/2® PartnerWorld® Redbooks™ SP1® VisualInfo™ zSeries™ cc:Mail® Notes® iNotes™ Lotus Sametime™ Redbooks (logo)™ AS/400e™ ClusterProven™ DFS™ DXT™ IMS™ Net.Commerce™ OfficeVision® OS/400® Perform™ SecureWay® Tivoli® WebSphere® Lotus® Lotusphere® Sametime® Lotus Discovery Server™ Mobile Notes™ AIX® Balance® DB2® Distributed Relational Database Architecture™ e (logo)® iSeries™ Netfinity® OfficeVision/400™ PAL® pSeries™ SP™ VisualAge® xSeries™ 1-2-3® Lotus Notes® Domino™ Lotus QuickPlace ™ QuickPlace™ IBM trademarks xiii SET, SET Secure Electronic Transaction, and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC. Other company, product, and service names may be trademarks or service marks of others. xiv Domino for iSeries Sizing and Performance Tuning © Copyright IBM Corp. 1999, 2002 xv Preface Explore the methodologies and approaches to assist in providing optimal performance of Lotus Domino for iSeries! This IBM Redbook targets technical professionals who are responsible for, or must advise on, the installation and administration of Lotus Domino servers on IBM ~ iSeries servers. It can also be used by performance specialists to gain knowledge on how to collect, measure, analyze, and extrapolate performance data from Domino servers. This redbook offers you the ability to translate these methodologies into your own situation and come up with specific performance analysis strategies and a set of tuning parameters for the best possible performance. This redbook shows a detailed approach for:
Estimating an appropriate configuration for a new iSeries server to run Domino
Measuring Domino application performance
Tuning Domino servers and OS/400 resources for optimal use
Understanding the impact of many configuration settings
Considering partitioning, clustering, the use of text search, indexing, and views
Improving throughput of the integration of Domino and DB2 UDB for iSeries
Analyzing the impact of the various Domino server tasks
Tuning Domino HTTP server workloads The team that wrote this edition of the Redbook This second edition of the Redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, Rochester Center. Wilfried Blankertz is an Technical Support Specialist for iSeries in IBM EMEA region Central located in Frankfurt/Germany. From 1995 to 1998, he was assigned to the International Technical Support Organization, Rochester Center. Here, he wrote extensively and taught IBM classes worldwide on all areas of AS/400 Groupware solutions and Systems Management. Before joining the ITSO, he worked as a systems engineer in IBM Germany supporting customers with the AS/400 system and its predecessor systems (IBM System /3, /32, /34, /36, and /38) for over 26 years. While he still focuses on iSeries technical support, he is also a Certified Lotus Professional for Domino Administration for Domino R5 Application Development and Domino Administration. Note: This redbook reflects the IBM ~ iSeries server name. Throughout this redbook, we use the shortened version “iSeries” to refer to both AS/400e and iSeries servers. Although the examples in this redbook were conducted using the iSeries running OS/400 V5R1, they should work the same way on AS/400e servers running OS/400 V5R1. xvi Domino for iSeries Sizing and Performance Tuning Christina Fasth is a Technical Support Specialist for iSeries in IBM Sweden, where she provides operational and software support to customers. Before joining IBM in 1996, she worked with IBM Systems S/3, S/3X, and AS/400 as a Systems Operator, Application Developer, System Engineering, and IT Manager. Her areas of expertise include Domino for iSeries and Client Servers. Kim Greene, president of Kim Greene Consulting, Inc., specializes in Domino for iSeries consulting, customized education, and performance. She has over four years of experience with Domino and 11 years of experience with the AS/400 and iSeries platform. Kim specializes in iSeries Domino performance analysis, system, and application tuning, enterprise integration with back-end applications and data, and Java and Domino integration. She is also a frequent writer for technology magazines and speaks at many conferences. Previously, she was employed at IBM in Rochester, Minnesota, in the PartnerWorld for Developers (PWD) organization helping IBM Business Partners incorporate Domino into their existing applications. She worked on several areas of OS/400 performance in the AS/400 development laboratory. Deb Landon is an IT Specialist at the International Technical Support Organization, Rochester Center, focusing on Domino for iSeries She writes extensively and teaches IBM classes worldwide on Domino for iSeries. Before joining the ITSO in November of 2000, she was a member of the PartnerWorld for developers, iSeries team supporting business partners in the area of Domino for iSeries. She was a member of the original team who created the highly successful Domino Days event that has since been replicated worldwide. Brandon Rau is a Staff Software Engineer in the Rochester Support Center. He has seven years of experience with AS/400 and the iSeries specializing in the areas of performance and work management. He as written performance articles published in the iSeries magazine, The Link, and Rochester Support Center Newsletter. Colin Stamp has been with Lotus/IBM for six years, where he works as a Senior Lotus Service Manager (LSM) and a Senior FSS consultant and specialist, in the field resolving performance and capacity issues for both customers and for Lotus Professional Services Consultancy colleagues on ongoing projects. Prior to joining the LSM team as a Senior Messaging, Communications, Platforms, Performance & High Availability specialist, he worked with the Lotus SoftSwitch Team in a Senior Support, Troubleshooter, and Consultancy capacity. Previously, he worked 10 years for Data General as a Senior Communications Support Engineer. Colin has written a number of white papers around Domino (transaction logging, High Availability, Mailfile usage). Preface xvii Thanks to the following people for their invaluable contributions to this project: Dave Herbeck Marcy Howerter Dave Johnson Don Morrison Joseph H. Peterson Domino for iSeries development James Cioffi Michael P Martin Richard Smitherman Domino for iSeries Center of Competency Michael Gilley Walter Scanlan Domino for iSeries Support The team that wrote the first edition of this redbook The original version of this redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, Rochester Center. Gottfried Schimunek Justine Middleton Collier Johnson Chris Larsen Craig Nash Petri Nuutinen Rick Underwood Mark Van Houten Thanks to the following people for their invaluable contributions to the first edition of this project: Tom Budnik Terry O’Brien Jeff Tenner IBM Domino for AS/400 Development Wilfried Blankertz IBM Germany, AS/400 Technical Support Center George Gaylord IBM Chicago, USA Mike Denney Kim Greene Chuck Stupca IBM Rochester, USA xviii Domino for iSeries Sizing and Performance Tuning Special notice This publication is intended to help IBM Business Partners and IBM Global Service personnel to measure, analyze, and extrapolate performance data from Lotus Domino servers running on the iSeries server and to do tuning to improve performance of the servers. The information in this publication is not intended as the specification of any programming interfaces that are provided by the iSeries server. See the PUBLICATIONS section of the IBM Programming Announcement for iSeries for more information about what publications are considered to be product documentation. Comments welcome Your comments are important to us! We want our IBM Redbooks to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways:
Use the online Contact us review redbook form found at: ibm.com/redbooks
Send your comments in an Internet note to: redbook@us.ibm.com
Mail your comments to the address on page ii. © Copyright IBM Corp. 1999, 2002 xix Summary of changes This section describes the technical changes made in this edition of the book and in previous editions. This edition may also include minor corrections and editorial changes that are not identified. Summary of Changes for SG24-5162-01 for Domino for iSeries Sizing and Performance Tuning as created or updated on April 3, 2002. Second Edition, April 2002 This revision reflects the addition, deletion, or modification of new and changed information described below. New information
Chapter 3, “Sizing Domino for iSeries using the Workload Estimator” on page 19, is completely new.
The contents of the former Chapter 7, “Capacity Planning for Domino Server” have been moved to Appendix A, “Capacity planning for Domino server using BEST/1” on page 391.
Chapter 7, “Tuning Lotus Domino for better performance on iSeries” on page 183, now contains the former Section 6.2 with many changes and additions.
New paragraphs and sections have been added to all other chapters. Changed information
Many pieces of information have been changed to reflect the differences due to the enhancements of the Domino releases after Domino for AS/400 R4.6 through Domino for iSeries R5.08, as well as OS/400 Version 5 Release 1. xx Domino for iSeries Sizing and Performance Tuning © Copyright IBM Corp. 1999, 2002 1 Chapter 1. Introduction Lotus Domino for iSeries has excellent, scalable performance. It has been proven to be a superior platform for server consolidation since its release. It has been the performance leader as measured by the publicly available NotesBench Benchmark. However, users do not typically use the NotesBench workload in their environments. Therefore, some guidance on how to optimize performance in customer environments is required. Integrating Domino with commercial applications and data in relational databases also needs careful consideration when maximizing throughput and providing the best possible response times. 1 2 Domino for iSeries Sizing and Performance Tuning 1.1 Conclusions and references This section describes briefly some of the most important topics covered in this book. It also points you directly to the appropriate place in this redbook for finding more details.
What size of an iSeries server do you need to support 1000 Domino users? Unfortunately, we cannot answer this question in a single sentence without knowing more details. See Chapter 3, “Sizing Domino for iSeries using the Workload Estimator” on page 19, to learn about the additional details you need.
If you know Lotus Domino very well, but wonder how you can use OS/400 functions on a IBM ~iSeries server to analyze why Domino or other applications are not performing well, read Chapter 4, “Basic concepts of performance analysis” on page 53. Lotus Domino provides several logs and statistics reports, are described in Chapter 5, “Domino logs and statistics” on page 127. This information can assist you in understanding bottlenecks and the throughput of your server.
When you use partitioning to create multiple servers, you are able to allocate different priorities and memory pools to each server. You can find information about memory storage pools in 6.4, “Choosing which memory pool to use” on page 166, and 6.5, “Choosing which processor priority to use” on page 173.
Run separate functions in separate Domino partitions. You may find it much easier to monitor performance and tune effectively if functions, such as mail and applications, are run in different Domino servers or partitions. Partitioning allows you to configure multiple Domino servers on one iSeries server. You can find information about partitioning in 7.10, “Number of users per Domino server” on page 201, and 10.1, “Domino partitioning” on page 274. Do not run server tasks that you do not need.
If your installation is not using calendaring and scheduling, it is a waste of resources to run the Calendar Connector (CALCONN) task on the server. Another function to look at is directory synchronization. Directory synchronization is automatically turned on when you configure a Domino server on the iSeries. If you will not use this function, choose to not enable the function when you configure your Domino server or change your existing Domino configure to remove its function. You remove tasks by editing the notes.ini file to take them out of the line ServerTasks. See Chapter 8, “Understanding the Domino server jobs” on page 229, to learn about the usage of the most important Domino server tasks. © Copyright IBM Corp. 1999, 2002 3 Chapter 2. Overview of Lotus Domino for iSeries This chapter explains the benefits of running Lotus Domino on an iSeries server. It describes the planning steps that you have to take when implementing Domino. Plus, it outlines some tools that you can use for monitoring performance. 2 4 Domino for iSeries Sizing and Performance Tuning 2.1 Why use Domino for iSeries Domino for iSeries is a powerful solution for building and deploying both business and e-business applications. As a server, the iSeries server provides reliability and scalability, which are essential in the dynamic and often unpredictable e-business environment. Domino for iSeries has the power and tools to enhance existing business applications and reach new levels of collaboration and coordination. 2.1.1 Scalability Within a single architecture, the iSeries spans a vast performance spectrum. On the high-end, running on a 24-way iSeries model, Domino for iSeries provides the processing strength of 27 partitions, accommodating 100,500 active mail users in a NotesBench audit1. That is 27 Domino for iSeries servers on one IBM iSeries! Each separate partitioned server can provide one or more functions of Domino for iSeries. Many customers run multiple Domino servers on a single iSeries. A typical example involves multiple Domino servers where one Domino server or partition provides mail function, another application serving, and another HTTP serving. Scalability is extremely important since the workload on any Domino server increases. Disk space demand increases rapidly as e-mail users become more proficient with managing their messages and calendars and Domino applications are added to the system. As the sizes of Domino databases increase, the server view indexer tasks need more disk space and CPU cycles to run. During the life cycle of the iSeries server, you may need to add memory, disk capacity, and additional processors. There is no better platform to address these growth issues than the iSeries server. 2.1.2 Reliability and availability Lotus Domino for iSeries takes advantage of the reliability and availability features of iSeries, such as RAID-5, mirrored disk units, and integrated backup capability. Each Domino for iSeries server runs as an application in its own subsystem. The unique iSeries architecture makes it safe to run Domino for iSeries and mission-critical business applications on the same machine. With a 99.97% proven availability rating, the iSeries is the most available server in the world for running Domino applications! On top of this, there are additional iSeries server hardware features to help keep the server running. An uninterruptible power supply (UPS) and backup procedures are also recommended to ensure maximum reliability. 2.1.3 Integration Domino for iSeries includes integration between Domino and DB2 UDB for iSeries databases. Domino for iSeries maximizes its integrated access to OS/400, allowing direct, real-time access to DB2 UDB for iSeries. Data can be accessed through several methods and provide tight integration between Domino and DB2 UDB for iSeries. Domino has a very rich set of integration options available for you to choose from. You can use one of the graphical tools that allow you to map your Domino environment to DB2 or program your own application logic. The tools include Domino Enterprise Connection Services (DECS) and Lotus Enterprise Integrator (LEI). If you choose to use one of the programming options, you can choose from the @DB functions, LotusScript:Data Object (LS:DO), Lotus Connector LotusScript Extensions (LC LSX), or Java Database Connectivity (JDBC). For more details on the performance aspects of these various integration options, refer to Chapter 9, “Integration with DB2 performance tips and techniques” on page 259. 1 See http://www.notesbench.org Chapter 2. Overview of Lotus Domino for iSeries 5 2.1.4 Proven security Integrated, flexible security is a strength of both Domino and the iSeries server. Domino has built-in security features such as execution control lists (ECL) and access control lists (ACL). In addition, Domino security policies can be setup on the server along with access controlled directly in the Domino server document. iSeries comes with support for some of the functions that would traditionally be found in a fire wall: IP Packet filtering, Network Address Translation (NAT), and SOCKS support. The iSeries server provides the flexibility you need in today’s networked world to have function and security right where you need it. 2.2 Planning a Domino for iSeries server Are you ready to move to the latest Web server technology known as Domino? Are you ready to jump right in and install the software? Well, not just yet. There are still some significant steps in planning that you need to accomplish before you implement your Domino for iSeries environment. There are several key areas that you want to be sure to include. Due to the scalability of the iSeries server, along with rich function sets available in Domino for iSeries, planning enables increased performance and application flexibility. The functionality of Domino for iSeries requires sufficient planning and testing. While the product is easy to install, the science is in determining the organizational requirements and architecting Domino for iSeries to best meet those organizational requirements. As you find more uses for Domino, you need to account for the resources it takes to support those new uses. This is why planning and preparation are so important. As you will notice in the following chapters, the functionality of Domino for iSeries is broad. 2.2.1 Developing a Notes naming structure Hierarchical naming schemes are the backbone to designing a solid Domino for iSeries infrastructure. One of the first planning steps to address is designing your naming structure. Do you want to have several organizational units or just one? The impact of this decision alone affects how you implement Domino for iSeries. The hierarchical naming convention used by Domino is common name (CN)/organizational unit (OU)/organization (O). An example of this naming convention is Jane Doe/Southeast/Widgets, where Jane Doe is the common name, Southeast is the Organizational Unit, and Widgets is the organization name (Figure 2-1). 6 Domino for iSeries Sizing and Performance Tuning Figure 2-1 Canonical format for names Only one common name and one organization are allowed. However, you can have multiple organizational units in your naming scheme. The OU is optional. Therefore, a naming scheme of CN/O is perfectly acceptable for small companies with a single Domino for iSeries administrator. However, companies are in the business to grow. Planning for company acquisitions and growth needs to be in the equation. When planning for common names, generally shorter names provide for greater ease of use. Before you implement your hierarchical naming scheme, it is important to do some planning. Draw a pyramid with the organization name at the top, and one or more levels of OUs below, as shown in Figure 2-1. Each level in the pyramid represents one OU certifier ID. Maintaining many certifiers and passwords can be a significant administrative task on the CPU. Multiple OUs also create excessively long Domino names that contain multiple slashes. Using departments as OUs has both advantages and disadvantages. It can be difficult to create and maintain an OU for each department because people can move between departments frequently. However, since employees in the same department work together and use the same databases and send e-mail to one another, the advantage of departments as OUs is their use as implicit groups for security and possibly for mail. Alternatively, you may consider a hierarchy that consists of a company, division, and geographic location, or perhaps a naming scheme that aligns with your Notes system administrators. For example, if your Domino administrators are distributed geographically, you may want to create an OU for each location. The naming convention provides a logical security grouping and a potential distribution of the administrative functions within an enterprise. 2.2.2 Choosing the right replication topology Replication and mail routing are highly used functions on the Domino for iSeries server. Designing a replication and mail routing topology is a major step in planning your Domino for iSeries environment. Engineering an efficient topology allows your Domino for iSeries server to perform at its highest level of reliability and efficiency while minimizing costs. Southeast Southwest Nick Lee Peter Brown Jane Doe Widgets Organization Unit (OU) Organization (O) Common Name (CN) Organization Unit (OU) Jane Doe/Southeast/Widgets (CN/OU/O) Chapter 2. Overview of Lotus Domino for iSeries 7 The factors you need to consider when choosing a replication topology are:
Size of your Notes and Domino Network: How many servers will be required to accommodate the number of users?
Location of your Notes and Domino servers: Are all your servers scattered in several locations or are most of them in one building? Perhaps in one iSeries server?
Amount and size of the e-mail data being sent: Is the average database (Notes Storage File) size only 1 or 2 MB in size or over 50 to 100 MB in size?
Keeping data current: Is there a need to keep your databases current or can databases go without replication for a period of days?
The number of clients that will access the data: Does your organization have several hundred or several thousand users per server? If you plan your Domino topology early, your organization's environment will run more efficiently. You will also be prepared for unlimited future growth. Such issues as mail routing and replication are primary methods used for passing information between Domino servers. They are scheduled tasks that can consume many CPU cycles. 2.2.3 How security relates Lotus Domino includes many layers of security. These layers include keeping your data secure on the Internet, mail routing, and replication within your organization. The categories of security are:
Server authentication
Database access control lists (ACL)
Form/view access control lists
Document level access
Field level encryption 2.2.4 What remains The following list contains more questions that you need to answer:
Will you be accessing the Internet? If so, you should consider using a firewall.
Do you need encryption capability? If so, secure sockets layer (SSL) needs to be enabled.
Where will your Domino for iSeries server be located? Is it behind a firewall?
How do you register new users, and what are the key databases you need to be aware of when planning registration?
Should you use advanced Domino services such as clustering, partitioning, or billing? We attempt to address these questions throughout this redbook. Your decision to implement decentralized or centralized administrative functions is fundamental to this discussion. 8 Domino for iSeries Sizing and Performance Tuning 2.3 The value of sizing, performance analysis, and tuning Figure 2-2 represents the framework for this redbook. In addition, it provides a methodology for upfront sizing, long-term performance analysis, and tuning. We explore the methodology in greater detail in Chapter 3, “Sizing Domino for iSeries using the Workload Estimator” on page 19. Figure 2-2 Method — Performance, tuning, and capacity planning This book provides information about the implementation, management, and control of Domino for iSeries. You may ask why we are including implementation in this book. The functionality of Lotus Domino is quite diverse when compared to other applications. As a result, how the Domino for iSeries environment is architected or implemented is imperative. Never lose sight of the goal of performance monitoring in an organization, which is: To develop an estimate of the system resources required to deliver performance in accordance with a documented Service Level Agreement and based on a forecast level of business activity at some specified date in the future. Your ability to performance tune and size Domino for iSeries is directly related to how it is implemented and how it has been architected. As you may imagine, Domino for iSeries has many features and functions. Each of these directly impacts the performance of the server in different ways. Due to the scalability and reliability of the iSeries server, many customers deploy all or many of the function sets of Domino on a single iSeries server. Knowing the impact that this type of architecture has on the system is necessary to understand what the system is or is not doing at any one time. Implementing Domino for iSeries on a single iSeries server is different from implementing Domino on several distributed Windows/NT servers. Understand the business requirements for using the Domino for iSeries environment. Analyze Domino statistics and logs — PM/Domino (determine measurements needed to track objectives). Domino for iSeries Sizing, Performance Analysis, and Tuning Tune Domino for iSeries using results from analyzing the iSeries information, along with Domino statistics and logs. Sizing Domino for iSeries based on the projected number of users and function sets employed. Employ advanced function sets in Domino for iSeries to meet performance objectives. Establish performance objectives. Observe iSeries performance data using the 57xx-PT1 tool. Chapter 2. Overview of Lotus Domino for iSeries 9 Because of the iSeries subsystem architecture, multiple Domino servers can be very easily deployed on a single iSeries server. The iSeries server is the most popular multi-user business computing system in the world. Lotus Domino is the most popular groupware solution. The three pillars forming the foundation upon which Domino for iSeries is built supports the pervasive theme of “simplicity through integration”. These pillars are:
Reliability: 99.97% uptime, use of subsystems, automatic recovery
Scalability: The most scalable Domino servers available today — up, down, and across
Integration: Database, administration, mail, directory synchronization, save, and restore 2.3.1 Audience The intent of this book is to provide traditional iSeries server administrators a methodology to manage the performance and capacity of their Domino for iSeries environment. It attempts to outline the issues involved in implementing the many functions of Domino while providing maximum throughput. In addition, it provides a framework for upfront sizing, long-term performance analysis, and tuning. Traditionally, the iSeries server administrator evaluates resources in two primary environments of the iSeries server: interactive and batch workloads. Through the management of system resources in each of these environments, the server administrator provides better throughput for the end users. The same approach applies to Domino end users. As Domino functions are deployed, their impact on system resources can be measured and analyzed. Consequently, the architecture or implementation of Domino functions on the iSeries server is critical in understanding how the system is performing or will perform in the future. Due to the functionality of Domino for iSeries, this book is primarily directed toward the traditional iSeries server administrator. However, the content of the book provides a much needed understanding to Domino administrators on how a particular set of functions can be better deployed on the iSeries server. 2.4 Domino architecture Figure 2-3 illustrates the layered architecture of Lotus Domino and the large amount of functionality it delivers. Lotus Domino offers much more flexibility than simply an e-mail application. The layered view accentuates the tight integration of core services and applications. It contrasts sharply with other products that are assembled from disparate sources and patched together with ad hoc interfaces. This section and the following sections outline a few specific examples of services at different layers of the architecture to show the design. 10 Domino for iSeries Sizing and Performance Tuning Figure 2-3 Understanding the Domino for iSeries architecture 2.4.1 Layer 0: The kernel Layer 0 contains the document-centric object store. Each document, or note2, is a compound structure of mixed data types arranged in fields that can be arbitrarily modified and extended. A document may contain text, rich text, binary blobs (attachments, ActiveX or Java Applets, for instance), encryption keys, doclinks, and so on. Each Domino database contains a collection of documents, and includes meta-organizing structures for display, security, retrieval, and access rights to the individual documents. 2.4.2 Layer 1: Kernel extensions Layer 1 is logically distinct from Layer 0. However, it is tightly bound to the object store. Critical platform services are encapsulated within this layer. These include security, replication, routing, formula languages, view indexing, and document editing and rendering. Domino Architecture Layer 3 Layer 2 Layer 1 C and C++ Layer 0 Java Beans SDK Programmability LotusScript Java VB and VBA NET NEM (UI Layer) OS Services On Disk Structure Memory Management Learning Space Sametime iNotes Web Access QuickPlace Domino.Doc C & S Mail Directory Browser Discussion Lotus Hosting Layered Applications Third-Party Applications Lotus Enterprise Integrator FT Search Free Time System Applications Log Billing Admin News (NNTP) Router (MTAs) Kernel Extensions NIF/VIEW Replication BSAFE Security Editor Object Store (NFS) Kernel 2 A single document as seen by a Notes user is normally stored as multiple notes. Chapter 2. Overview of Lotus Domino for iSeries 11 Each Layer 1 module is self-contained, with common interfaces to each other and the object store. As a result, new modules can be introduced into Layer 1 or existing ones swapped out. For example, the SMTP Message Transfer Agent (MTA), the router for Internet mail, was inserted in Release 4 to provide an alternative mail router. Later, with Domino R5, a native solution for SMTP was implemented to provide reliable and scalable mail router. The same ability can be used for search engines, security key rings (Public Key Infrastructures), and so on, as your needs require the adoption of new or standards-based components. 2.4.3 Layer 2: System and core applications Layer 2 includes core collaborative applications that are coordinated by supporting system applications. Core applications (mail, calendar and scheduling, discussion, document management, and so on) are the first level of services directly visible to end users. As standard Domino applications, they can be modified and extended by Domino developers, but usually depend on kernel routines. System applications control and monitor these and all other applications. They include the LDAP-compliant directory service (Domino Directory), Logging, Administration, Billing, Free Time Lookup, and others. 2.4.4 Layer 3: Layered end-user applications Layer 3 applications are not distinguishable technically from Layer 2 applications. Layer 3 applications also rely on the kernel and the Domino Directory (formerly called Public Address Book or Names and Address Book). However, most of them further extend core mail, calendar and scheduling, discussion features, workflow into higher level, task-specific applications. The growing set of Domino applications shows the way complex functions evolve from lower architectural layers, while adding design and user interface elements of their own. Thousands of existing third-party Notes applications also fit into this layer. An example is a Sales Force Automation Domino application that utilizes security, workflow, document management, and SameTime to generate self-help Web applications for end users. 2.5 Tools available There are a number of tools for sizing, performance analysis, and tuning that are available for the iSeries Domino environment. This section lists a few of the tools that are available:
iSeries and Domino tools – Collection Services – Performance Management (PM/400) – Performance Tools/400 – Performance Explorer – Management Central – BEST/1 – Server.Load
Third-party tools for iSeries and Domino – PM/Domino – Performance Navigator – NotesBench – GroupSizr – WEBSizr – LoadRunner – Tivoli 12 Domino for iSeries Sizing and Performance Tuning Comparisons between the native iSeries statistics and the Lotus Domino statistics are further explained throughout this redbook. 2.5.1 Collection Services Collection Services is a data collection tool that is native to the iSeries. It can provide significant help in troubleshooting performance problems. Data analysis can be performed against this data collection online or offline. More information is provided on this important function of OS/400 in Chapter 4, “Basic concepts of performance analysis” on page 53. Collection Services, formerly known as Performance Monitor, is a part of the operating system OS/400. To analyze the data acquired by collection services, utilize Management Central (see 2.5.5, “Management Central” on page 13, and 4.12, “Management Central performance monitor” on page 89) or the Performance Tools/400, Licensed Program Product (LPP) 5722-PT1 (or 5769-PT1 before V5R1). See 2.5.3, “Performance Tools (5722-PT1)” on page 12, and 4.13, “Reviewing data using Performance Tools/400” on page 114, for more information. 2.5.2 Performance Management/400 Performance Management/400 (PM/400) presents a broad array of benefits to you. As a systems management tool, PM/400 ensures that you get the most from your system by regularly monitoring performance and growth. These measurements allow you, your IBM Business Partner, or IBM to more easily plan for future system growth and identify potential resource constraints. Measuring system performance helps you make decisions that affect your budget and human resource plans. PM/400 is automated and self-managing, which allows for ease of use. PM/400 gathers the non-proprietary performance and capacity data from your iSeries server and sends it to IBM. At a cost, you can receive this data in a series of reports and graphs that show the growth and performance of your system. You can view less detailed data for free online at: http://www-1.ibm.com/servers/eserver/iseries/pm400/ PM/400 is automatically installed on your iSeries server or AS/400 system. It is more than a service; it is a partnership between you and IBM or your IBM Business Partner. This partnership helps you achieve the best return from your iSeries investment. Contact your service provider for more information. You can also visit the Web site at: http://www-1.ibm.com/services/its/us/mus14d1b.html 2.5.3 Performance Tools (5722-PT1) Performance Tools/400 (5722-PT1) manages and analyzes performance data, providing real-time and long-term views, as well as detailed analyses. You can order this product with one of two features: the Manager feature or the Agent feature. The Agent feature contains a subset of the functions in the Manager feature. Performance Tools has the capability to report, analyze, and model system performance. The Advisor function can analyze the system and give recommendations to improve performance. The performance capabilities in both the Manager and Agent features allow you to:
Manage performance data
Monitor performance in real-time interactively
Display performance data
Analyze performance data
Create a BEST/1 capacity planning growth model Chapter 2. Overview of Lotus Domino for iSeries 13 Performance Tools/400 is a licensed software product from IBM. The costs varies depending on the processor group. 2.5.4 iSeries Performance Explorer (PEX) The iSeries Performance Explorer (PEX) is a set of performance collection and reporting functions and commands that became available on the RISC versions of OS/400. The PEX functions, which are part of a total iSeries performance management methodology, should not be the first set of performance tools used when analyzing a performance problem. The Performance Explorer and Collection Services are separate from one another and produce two completely different sets of database files that contain grouped sets of collected data. Both collections of data can run at the same time. However, this should be kept to a minimum because the impact to the system is significant with both collections active. As with Collection Services, the capability to collect Performance Explorer data is shipped with OS/400. The ability to analyze this data requires the purchase of Performance Tools for AS/400, LPP 5722-PT1. You can find further information on general PEX usage in 4.14, “Performance Explorer (PEX)” on page 116. You can also find more information for specific PEX trace points for Domino in 4.14.5, “Performance Explorer trace points for Domino” on page 117. 2.5.5 Management Central Management Central monitors provide the ability to monitor system-wide metrics, as well as focus in on specific jobs. Prior to V5R1, the monitors were classified in one group that monitored various performance statistics across the system. These monitors are now called system monitors. System monitors provide real-time monitoring along with the ability to view historical data on such areas as disk arm utilization and interactive feature utilization to name a few. The job monitors can monitor the activity of a single job or numerous jobs by job name, user, job name, subsystem or server type. You can learn more about Management Central monitors in 4.12, “Management Central performance monitor” on page 89. 2.5.6 BEST/1 The BEST/1 Capacity Planning tool can be used today to create models of various workloads to estimate the processor requirements along with disk, memory, input/output processors (IOP), and other requirements. This tool contains two predefined workloads for Domino that are based on NotesBench. It is best to collect your own performance data to feed into the BEST/1 tool for accurate capacity planning results. BEST/1 is part of the OS/400 Performance Tools Licensed Program Product (5722-PT1). As with every performance estimate (whether a rule of thumb or a sophisticated model), you always need to remember that it is an estimate. This is especially true with a robust product, like Domino for iSeries, that offers so many different capabilities with a chameleon-like ability to conform to characteristics and demands. The typical disclaimers that go with any performance estimate, such as "your experience may vary", are especially true with Domino. Note: After OS/400 V5R1, BEST/1 will no longer be a part of Performance Tools and will no longer be an offered LPP for OS/400. 14 Domino for iSeries Sizing and Performance Tuning 2.5.7 Server.Load Lotus Server.Load is a tool that measures and simulates various Domino server capacity and response metrics. Server.Load can be used to simulate the load that Notes client-based applications place on a Domino server. It allows you to select a workload and run it against a target server. The workloads (also called tests or scripts in this redbook) simulate the behavior of a Domino workstation to server. Server.Load offers Domino administrators the flexibility of selecting a built-in script, creating a custom script from a library of commands, or submitting commands manually. For example, you can select to run the built-in Simple Mail Routing script and simulate users on a client reading and sending mail. You can also create a custom script that creates and opens a Notes mail database and populates it with notes. If you want to test or run individual commands, such as deleting notes from a database or even running remote server console commands, you can do this as well. Server.Load also enables you to monitor server metrics in real time by using the Real Time monitor window. Server.Load provides the following features:
An easy to use interface for running and customizing scripts and measuring server performance
The choice of using built-in or custom scripts for the workloads
A command line to submit manual commands
Real-time control of test environment and variables
Real-time characterization of server performance and capacity metrics
Server.Load help database Starting with R5.06, Server.Load is shipped as part of the client CD or download kit. It consists of the tool sload.exe and the documentation sload50.nsf, which can be located in the Notes data directory when installed. Server.Load installs as part of the Administration Client. 2.5.8 Performance Navigator Performance Navigator is a graphical PC Tool for iSeries Performance Management and Capacity Planning that runs on Windows 95, 98, NT, and 2000. It has an intuitive interface combined with advanced statistical capabilities to access the iSeries performance data and reports dozens of performance metrics. It has a dynamic linear regression algorithm for trend analysis, capacity planning, and growth prediction. It enables you to sort through the data by month, week, day, and interval. This is a chargeable program product that runs on your workstation to analyze the QMPGDATA library from the iSeries server. For more information, see the Web site: http://www.mpginc.com 2.5.9 NotesBench NotesBench provides a collection of both benchmarks and workloads for evaluating the performance of Notes R5 servers. NotesBench is a third-party consortium organization with a mission to measure vendor server platform performance with Lotus Domino servers. NotesBench is available to hardware vendors only. The statistics can be used for such applications as Server.Planner. The workloads are software components that simulate the behavior of Notes workstation-to-server or server-to-server operations. They return measurements that let you evaluate server performance in relation to the cost of ownership of the server. Chapter 2. Overview of Lotus Domino for iSeries 15 NotesBench provides:
A command-line user interface for running the benchmarks
Scripts used in the benchmarks
Special variables for notes.ini files
A checklist of disclosure information
A checklist for auditors
The Using NotesBench guide
Related Domino and Notes documentation NotesBench gives you an objective way to evaluate the performance of different platforms running Notes. It is intended for testing engineers at hardware and software vendors. NotesBench requires that vendors run the NotesBench tests and publish their performance results. Each vendor runs the same tests in the same manner, and the tests can be audited. For more information, see the Web site: http://www.notesbench.org 2.5.10 GroupSizr GroupSizr (formerly known as and NoteSizr) from Technovations is an application used to size, benchmark, and analyze Lotus Notes servers, systems, and applications. GroupSizr can be used to simulate many real Lotus Notes clients using a single PC. The tool is capable of running a sequence of commands against any specified server or set of servers and any given database and any Notes application. GroupSizr produces concise and easy-to-understand results and reports, in addition to providing run-time analysis of test performance data. GroupSizr is particularly suitable for simulating a large amount of session requests within a limited duration of time, for example, on Monday at 8:00 a.m. This is a PC-based Licensed Program Product. However, you can download an evaluation copy by visiting the Web site at: http://www.technovations.com 2.5.11 WEBSizr WebSizr is a performance analysis and sizing framework for HTTP-based servers and applications. Application developers use WebSizr to engineer high-performance Web applications. WebSizr helps the architect and systems engineer to easily characterize the performance, scalability, and sizing of:
HTTP-based Web servers
Internet or intranet applications
Electronic commerce applications This is a PC-based licensed program product. However, you can download an evaluation copy by going to the Web site at: http://www.technovations.com 2.5.12 LoadRunner LoadRunner is a load testing tool that predicts system behavior and performance. It exercises an entire enterprise infrastructure by emulating thousands of users to isolate problems, optimize performance, and accelerate deployment. The Virtual User Generator lets you record both two-tier and three-tier client/server traffic into a clear and readable script. The dynamic monitoring of all virtual users allows you to organize and control how they test individually and how they interact during a scenario. The server monitor allows you to observe the database or application server during the load test. Summary statistics are presented with clear, attractive graphs and reports that help analyze load scenario results and pinpoint performance bottlenecks. 16 Domino for iSeries Sizing and Performance Tuning LoadRunner is a licensed product. While the costs for this product is rather expensive, it provides a wide array of functionality. For more information, visit the site at: http://www-heva.mercuryinteractive.com/products/loadrunner/ 2.5.13 Tivoli Manager for Domino Tivoli Manager for Domino provides the ability to monitor hundreds of Domino servers from a real-time graphical interface, all of which can be done from a single console. The latest release of Tivoli Manager for Domino is version 3.1. The latest release contains many new enhancements that include:
Web-based interface
Additional monitors
Added features for performance stations
Integration with Tivoli Data Protection
New graphic and reporting tools You can find more information and news updates on Tivoli Manager for Domino at:
http://www.tivoli.com
http://www.tivoli.com/news 2.5.14 Tivoli Application Response Measurement (ARM) The Application Response Measurement API is a set of standard API calls, agreed upon by a number of prominent IT companies, that allow an application-centric focus on performance. The ARM API allows you to monitor the network that connects a client to a server and record the interval from when a request leaves the client until a response arrives at the client. You can use a program that generates a typical client workload and make time measurements within the program. You can also measure the utilization for all the hardware components involved in providing service to the user. And you can assume that if these values are at reasonable levels, the user response time must be acceptable. This tool is a part of the "Enterprise" of Tivoli tools. For more information, visit: http://www.tivoli.com/support/downloads/arm_sdk/armguide.html ARM is intended for the application developers who wants to know how to instrument an application for transaction monitoring using the standard Application Response Measurement API function calls. Version 2.0 of the ARM Software Developer's Kit has been developed with the help of the ARM Working Group of the Computer Measurement Group (CMG). The 2.0 ARM SDK, including this documentation, is available on CD and from the CMG Web site: http://www.cmg.org/regions/cmgarmw This Web site also contains information on performance measurement agents that use the data generated by the ARM API function calls and the latest information regarding future updates or changes to the API. A public discussion list for ARM, cmgarm is now available by sending e-mail to: cmgarm@cmg.org To subscribe, send the following message to majordomo@cmg.org: subscribe cmgarm Chapter 2. Overview of Lotus Domino for iSeries 17 2.6 NSD for OS/400 (Dump utility) The Notes Server Dump (NSD) is a Lotus tool designed to gather information about a failing Domino Server. The tool extrapolates the critical data during the life of the Domino server. Upon failure, the tool produces a dump file. This file, when presented to Lotus support is intended to provide all the Domino information a Lotus Quality Engineer needs to troubleshoot just why the server failed. This was originally designed to be run with various UNIX systems. The NSD tool has been ported to run on OS/400 to diagnose Domino for iSeries. The tool collects and provides the following data to Lotus:
Active jobs
Environment variables
Job call stack
Job locks
Display open files
Dump of semaphores, shared memory, and message queues
Dump of notes.ini file
Commitment control status
Display communications status
Job mutexes (can be seen as locks)
Job activation group 2.6.1 Activating the NSD tool NSD can be activated by editing the notes.ini file and adding the line: CleanupScriptPath=call qnotes/nsd When the server is started, the information is dumped to a Stream File (STMF) in the Data directory of the Domino server. The file name is nsd_all_os400_(julian date)_(time). Consider this example nsd_all_os400_19980115_15:22:21. The file name is often truncated on a green-screen display because it is relatively long. In a Client Access green screen, if you click the file and then click the F22 key, you see the full file name. 18 Domino for iSeries Sizing and Performance Tuning © Copyright IBM Corp. 1999, 2002 19 Chapter 3. Sizing Domino for iSeries using the Workload Estimator This chapter is designed to offer guidelines to help use all of the functionality of the IBM Workload Estimator for iSeries (WLE) when sizing Domino to arrive at the most accurate estimate possible. Please keep in mind that even after reading this chapter and having a better understanding of how to use the WLE to achieve a more accurate estimate, it is necessary that you treat the results and recommendations of the Workload Estimator as estimates. For more detailed sizing information, we encourage you to use the BEST/1 modeling tool. The BEST/1 tool is designed for performance specialists (including customers) to be used for complete capacity planning. The BEST/1 capacity planner is part of the IBM Performance Tools/400 product, which is a licensed program for the iSeries and AS/400. For more information, see Appendix A, “Capacity planning for Domino server using BEST/1” on page 391. Another alternative is to use the Performance Management/400 integrated with the Workload Estimator (see 2.5.2, “Performance Management/400” on page 12, for more information). 3 20 Domino for iSeries Sizing and Performance Tuning 3.1 Introduction to Workload Estimator The purpose of the IBM Workload Estimator for iSeries (WLE) is to provide recommendations for estimating an appropriate configuration for an iSeries or an AS/400e server running one or more workloads associated with e-business or collaboration, such as Domino. It is a Java servlet-based Web application on the Internet that you can find at: http://www.ibm.com/eserver/iseries/support/estimator/index.html1 Figure 3-1 Workload Estimator conditions of usage The majority of the use of the Workload Estimator is to size new iSeries or AS/400e models. It is, however, possible to estimate for an upgrade of an existing system with a pre-existing system load by adding additional work. There are two ways to enter data about an existing system into the Workload Estimator: 1 This URL has been directed recently to be consistent with other iSeries Web pages. The former URL http://as400service.ibm.com/estimator will redirect you to the current site. Tip: Before you start using the Workload Estimator for the first time, we highly recommend that you print and read carefully the help documentation, which opens up automatically in a separate browser window after you accept the page with conditions of usage (see Figure 3-1) as well as through the tutorial provided with the tool. The tutorials can be downloaded as PDF files. Chapter 3. Sizing Domino for iSeries using the Workload Estimator 21
You can select the Existing workload option on the Workload Selection page and complete the questions asked about the existing resources of the system.
Import existing system data directly into the Workload Estimator from Performance Management/400 (PM/400). The recommendation or Selected System reported by the Workload Estimator includes the model, processor, interactive feature, CPW capacity, memory, and then number and capacity of disk storage (DASD) necessary to support the set of workloads defined by the user. When providing the sizing recommendation, the Estimator always defaults to recommend the latest technology systems that are available. Even though these detailed resource requirements for each WLE recommendation are given, it is important to remember that just as with every performance estimate (whether a rule of thumb or a sophisticated model), you always need to treat it as an estimate and not a detailed capacity planner. This is particularly true with a robust system like the iSeries server that offers so many different capabilities where each installation will have unique performance characteristics and demands. The sizing recommendations that are generated by the Workload Estimator began with benchmark measurements and performance tests based on well-defined repeatable workloads. For Domino, most of the measurements that were performed were the NotesBench consortium benchmark tests. Through the experience of running these measurements with large numbers of users, as well as feedback from internal and external customers and business partners, certain performance projections for each resource have been developed. These are used in the calculation for a recommended system. To arrive at a recommended system, the Workload Estimator goes through a series of questions to collect the necessary information to calculate the resource requirements of your solution. As you become more familiar with the Workload Estimator, you can provide more specific information for particular workloads, thereby leading to a more accurate estimation. The Workload Estimator is constantly being updated to implement the latest versions and releases of the workloads. New function is being added on a regular basis to keep the information and projections current. Periodically review the online version of the Workload Estimator to keep up-to-date with the most accurate information at http://www.ibm.com/eserver/iseries/support/supporthome.nsf/document/16533356, which always describes the latest changes. 3.2 Benchmarks to determine the load of specific workloads A few of the factors needed to determine a system's overall impact are the number of active or concurrent users, the type of work being performed, the frequency by which transactions are performed by the users, and the complexity of the applications being deployed. Although there are other factors to consider to determine the total impact on the system (communication lines, I/O processors, bus speeds), the parameters mentioned in the previous paragraph are the critical factors for the Workload Estimator. A series of questions are asked in WLE to acquire a combination of these factors to determine the resource requirements to support the load of your solution. Because of the diversity of business applications being deployed today, the best way to appropriately size a system for a given application is to test the behavior of many system configurations to see how they perform. Testing would require extensive hardware and people resources. Because of this, detailed testing is not typically performed by customers or IBM business partners. 22 Domino for iSeries Sizing and Performance Tuning A practical alternative is to select a workload that defines the type and frequency of the transactions performed by users and use this workload to put a load on a system. After the workload is defined, you can then extrapolate the results to a real user environment without having to run the test on many different system configurations to see how they each perform. This is only meaningful in conjunction with the well-defined workload for the benchmark. In most cases, the benchmark result does not correlate to the exact type of “real” users working in your organization. An example of this is the NotesBench workload used to measure the impact of mail users on a Domino server. The NotesBench consortium was established to define standard workloads for measuring Lotus Domino and Notes Benchmarks on different platforms. One specific benchmark that is of interest when using the Workload Estimator, especially for sizing a messaging workload, is the R5Mail workload2. The R5Mail script is as follows:
Events occurring every 15 minutes – Read five documents – Update two documents – Delete two documents – Scroll a view once – Open and close one database – Open and close one view – Send one memo to three recipients – Do three lookups on the Domino directory
Events occurring every 90 minutes – Schedule one appointment – Send one invitation For many benchmarks being published today, there is a multiplication factor used to project from the benchmark “virtual” user to a “real” user. As a general rule of thumb, a NotesBench R5Mail user equals approximately one-half of the complexity of a typical mail user. This means that in the Estimator, the number of typical users supported by a system is equal to one half of the number of NotesBench users reported for the same system. For example, if an iSeries Model 840-2461 was tested with 100,500 concurrent NotesBench users, then that same system should be able to support 50,250 typical mail users. You can find the iSeries audited NotesBench results and other useful NotesBench information at: http://www.notesbench.org Aside from viewing the existing iSeries and AS/400 benchmarks that have been published to help give you an idea of the data behind the Domino workload in the Estimator, this is also a good reference point to start when investigating possible migrations from other platforms. It is possible to compare the NotesBench results published from another platform for a specific workload with the results of the same workload on the iSeries to understand how to input your existing workload on another platform into the Estimator. 2 Earlier NotesBench audits as well as sizing aids for Domino for AS/400 used a different workload based on Domino Release 4.5. Note: Although the typical user comes much closer to an average user in reality than the NotesBench R5Mail user does, the workload generated by real users may deviate considerably! This is most significantly true when it comes to applications other than mail, but even the real life mail workload can differ substantially. Chapter 3. Sizing Domino for iSeries using the Workload Estimator 23 3.3 Workload Estimator terms When using Workload Estimator for sizing a Domino solution, it is important to use WLE terms as they were intended for use within the tool. This section includes a discussion of a some terms that could be interpreted differently within the context of Domino. 3.3.1 Workload On the Workload Selection page (shown in Figure 3-2), the term workload refers to the entire set of parameters defined within the pages that correspond to one of the types of workload selected. It is possible to select more than one of the same type of workload. This allows the user to size more than one Domino application per session of the Workload Estimator. Specifically to Domino, if mail and applications are selected, the workload, as the Estimator understands, is both of those Domino uses combined. Because many different applications can make up your total Domino solution, the workload selection could include as many different Domino workloads as your total business solution needs to define all of the applications. Similarly, for a Domino mail solution, it is possible to select multiple Domino workloads to account for each of the types of mail access. Alternatively, since each Domino workload that is defined in the Estimator contains both a mail profile and an application profile, it is possible to only have one Domino workload defined. For example, you could either define one Domino workload that details your messaging or mail solution and another Domino workload that details your end-to-end business application like a Web shopper. Or you could combine both into a single workload. Figure 3-2 Workload Selection page from Workload Estimator 24 Domino for iSeries Sizing and Performance Tuning 3.3.2 Instance Some Domino administrators use the term instance to refer to a Domino server. For example, you may have multiple instances of Domino on a single iSeries server. In Workload Estimator, the term instance in the phrase “the number of partitions being used by this instance of the Domino workload” (as shown in item 7 in Figure 3-3 on page 25) actually refers to specific Domino workloads defined in the Workload Estimator such as “Domino #1” in Figure 3-2. This instance may be one of multiple applications deployed on the same Domino server or across multiple Domino servers. Therefore, in this reference to the word instance in the Workload Estimator, the user should think about the specific usage type that is being defined, not the instance of the Domino server. 3.4 Domino sizing concepts in the Workload Estimator Due to the diverse Domino solutions being deployed, there are multiple ways to use Workload Estimator to size a Domino solution. The following sections describe in more detail certain Domino specific parameters that will help when you are faced with the task of estimating the load of a total Domino business solution and recommending an appropriate iSeries to support the solution. The Domino workload within the Workload Estimator has questions designed to acquire the factors of your Domino environment to appropriately determine the recommended system. The default values are suggested values for brand new Domino customers who do not know how their users will end up using the application. For those customers and business partners who are familiar with their Domino application and their user's habits, this section offers some definitions and helpful hints on how to interpret a few of the Domino workload concepts used to determine the recommended system. 3.4.1 Concurrent users The number of registered users is the easiest to understand and always a known input variable. Note, however, even though this should be the first question to ask, there is no way to come to a proper estimation, if the number of users is the only information you have. The number of active or concurrent users is very important when sizing the customer's solution. There are a couple of ways to report the total number of active users that a customer will be deploying. The defaults for concurrency, as shown in Figure 3-3, are set to values that are suggested for new Domino customers who do not know the frequency that their users will use the applications. Again, if you accept the defaults, very likely it may not match your environment. If you know the percentage of users that are active during peak times, adjusting the concurrency rate for the application accordingly will result in a much more accurate recommendation. Tip: We highly recommend that you look at every single parameter and try to understand its significance and how it might vary depending on your specific environment. The more often you accept the default values, the less likely the resulting suggested configuration will be useful. Chapter 3. Sizing Domino for iSeries using the Workload Estimator 25 Figure 3-3 Workload Estimator Mail Definition When you adjust the concurrency ratings within WLE Domino workloads, it is important to understand how the concurrency ratings are used. By selecting one Domino workload, choosing both mail and applications, and leaving the Estimator defaults for concurrency, the highest total concurrency rating for all of your users is 65%. This is due to the 50% default mail concurrency and the 15% default application concurrency rating. For example, if you have 1000 users and all users work with mail and a Domino application, then the mail and application profiles would each have 1000 registered users. WLE would then calculate using the concurrencies of each Domino usage type that 500 users would be using mail and 150 users would be using the Domino application concurrently. This totals 650 users actively interacting with the server at once. Sixty-five percent (650 out of 1000) of the users would then be active on the system driving the utilization. It is probably not appropriate to specify each of the usage types as being 100% busy if you also specify that all 1000 users are registered users of mail, the user-written applications, and Domino.Doc. This would give a concurrency rating of 200%, which is very unlikely. Table 3-1 shows the specific default values for concurrent users for each Domino usage type. The default concurrency values shown in Table 3-1 are based on feedback from IBM representatives and Business Partners who are familiar with the way that current customers are typically running as well as direct feedback from internal and external customers. Note, however, that these values may be significantly different to your own environment. 26 Domino for iSeries Sizing and Performance Tuning Table 3-1 Default concurrency values In order to determine the total number of concurrent users for each Domino usage type being defined in Workload Estimator, use a 15 minute interval. This means that an active user is any user who has performed an activity that made a request to the server within a 15 minute interval. If a user was active at anytime in that 15 minute interval, they should be counted as a concurrent user. Please note that this value does not correspond to the default time-out (Server_Session_Timeout), where inactive users are automatically disconnected after the specified time. If you use WLE for existing Domino installations, there are a couple of ways to help determine users' concurrency. For connected Notes Client users, SH STAT SERVER shows the peak number of users. This value would be a great starting point for the number of concurrent mail users. SH STAT SERVER does not show how much work the users are generating. Therefore, this would not be the place to determine users' complexity. For Web type users, HTTP is connectionless from a Domino statistics standpoint and, therefore, the statistics cannot show any numbers for “connected users”, but the value Domino.Threads.Active.Peak can give some indication of concurrency. The 15 minute interval recommendation is a rule of thumb used to determine the number of concurrent users of the Domino application being deployed. This rule of thumb also applies to remote users. For example, if all of the remote users were replicating every 15 minutes, then they would be 100% concurrent. It is important to note in this circumstance that disconnected users should be classified as casual mail users. 3.4.2 Number of partitions The number of partitions calculated in Workload Estimator refers to the number of Domino partitions. This metric does not refer to the number LPAR partitions being configured on the iSeries server. By using the default of *CALC to determine the number of partitions, WLE uses a value of 3000 concurrently active users per Domino partition. Depending on the workload generated by each user, it may be possible to support more users per partition. The default of 3000 users per partition is a good starting point3. Domino usage type Default concurrency Notes Client Mail access 50% WebMail access 15% IMAP Mail access 15% POP3 Mail access 15% iNotes Access for Outlook 15% Applications 15% 3 With Domino R4.6 and before, there was a certain maximum workload which could be handled efficiently by each Domino server. This made it necessary for many administrator’s to limit the number of mail users, per server. At the time, the recommendation was not to exceed 500 to 700 users per server. With Domino R5, the implementation has been changed, so each server can support a much higher workload. Chapter 3. Sizing Domino for iSeries using the Workload Estimator 27 There are of course instances in which overriding the Estimator default would be appropriate. In a case where you have a total of 3000 users, instead of putting all 3000 users into one partition, then you could put 1500 users in each of two partitions, to allow easier management of the servers. Especially for an application like mail, where each user owns a particular database, a server becomes difficult to manage if the number of databases exceeds a certain limit. 3.4.3 Clustering It is important to note that clustering defined for a Domino workload is actually Domino clustering, not OS/400 clustering. Two or more Domino servers on the same or different hardware (and same or different operating systems) can be grouped as a cluster. Important for the workload is a task called cluster replication (see 8.3.3, “CLREPL: Cluster Replicator” on page 246). This task ensures that changes applied to a database that have a replica on a server in the same cluster are replicated immediately4. When Domino databases are clustered, the server performs real-time replication for any changes that occur to the databases. Therefore, clustering introduces additional processing requirements that are used in sizing. For that reason, and since there are many ways to configure clustering, it is important to interpret and specify the appropriate information in WLE. In reality, not all databases on clustered servers need to be enabled for cluster replication. The assumption for clustering in WLE, however, is whichever clustering configuration is chosen for a particular Domino workload instance applies to all of the databases defined in that instance. For example, if a Domino workload is defined with mail and applications, the clustering configuration is applied to the total number of all mail databases and all other databases on that server. Your solution may be designed in such a way that clustering is deployed on a more granular level, so that mail and application databases are not clustered together. It is necessary to define multiple Domino workloads for each usage type and specify the appropriate clustering configuration for each. Similarly, if there are some user databases that are not clustered at all, you should define a separate Domino workload. The “Clustering with other Domino Servers” option refers to both being clustered to another server and having another server clustered onto the system being sized. If the system is being used by another system for clustering, it is also necessary to fill in the amount of disk storage needed for clustering. “Cluster on the same iSeries server” refers to the case where you have multiple Domino partitions on the same iSeries server using Domino clustering. If the partitions are clustered on the same system, the Estimator will automatically double the number of partitions that have been either calculated (*CALC) or that the user has specified. In addition, the Estimator calculates the amount of disk storage needed for clustering by adding the amount of disk storage required for the total number of mail databases, as well as any application databases that are defined. Here is how WLE would calculate the resources needed for clustering based on the input to the specific clustering questions: 1 partition (3000 concurrent users) + 1 partition (clustered on the same system) ______________________ 2 total partitions 4 As opposed to “normal” replication that only takes place periodically. 28 Domino for iSeries Sizing and Performance Tuning 3000 mail databases (3000 concurrent users) + 3000 mail databases (3000 replicated databases for clustering) _______________________ 6000 total mail databases 3.5 Domino mail concepts As noted previously in the NotesBench benchmark discussion, it may not be completely accurate to extrapolate NotesBench mail users to real life users. The first thing to note is that WLE uses the NotesBench benchmark value for the number of users that will fit on a system. Then it divides that by 2. Although this is done in an attempt to more accurately depict an average mail user, not all of the users working with mail server today are average or “typical”. Therefore, the user complexity question in Workload Estimator gives you the ability to further classify the mail users, as the casual, moderate, and heavy values are given a type of weighting factor. The default values of 10 casual, 70 moderate, and 20 heavy mail users is a good starting point if the organization did not use Notes mail before. The default mixture was chosen to represent a “typical” mail environment. This “typical” user projected for a given system will be approximately equal to one half of the number of Mail and Calendering users (MCU) specified for that system. For a more detailed discussion of the mail user metrics used for projections, see 3.10, “Additional tips and insights” on page 44. If you use many agents and extra function with your mail solutions, there is only a fine line between a heavy mail user as defined in Workload Estimator and a Custom Application user. One possible solution to sizing a very complex messaging solution is to use the Application profile for Domino in WLE. For example, a mail profile with 100% heavy Notes Client mail users represents the same load as an Application profile with an application rating of 1.3. If your mail user profile is very complex, with many views and agents being deployed, it may be appropriate to use the Domino Application profile. Just as it is important to use caution when sizing very heavy mail customers, it is also important to avoid overestimating the number of casual users. The casual user does e-mail only, rarely sending or receiving attachments. Therefore, casual users can quickly become more complex, for example, by starting to use the calendaring and scheduling features as they become more experienced with Notes mail. New Notes users will generally advance from being casual users to moderate users in only a few months. 3.5.1 Mail access types Another factor that has an impact on the workload generated by mail users is the access type. Mail access types currently supported in the Workload Estimator include:
Lotus Notes Client
iNotes Access for Microsoft Outlook Note: The mail profile should be used in most cases to estimate a mail workload. You should only consider using the Application profile in cases when you know your Domino mail solution very well and understand the differences to a standard mail environment. However, do not use the mail profile to estimate the load of a custom written or purchased Domino application. Chapter 3. Sizing Domino for iSeries using the Workload Estimator 29
iNotes Web Access
WebMail
IMAP
POP3 As noted above, most of the mail access type profiles in Workload Estimator are based on information collected from internal benchmarks using the NotesBench suite of workloads. The recommendations made by WLE are not directly comparable with the audited NotesBench data. Because the various NotesBench workloads create differing amounts of work per user, they have been normalized for the Workload Estimator. For a specific mail access type, the assumption is that each user would do the same amount of work within 15 minutes. The difference would be in the way the user's mail client connects to the server. With the many choices of mail clients that are available, you could support multiple mail access types with your mail and calendaring solution. When using WLE to size such a heterogeneous solution, you use multiple Domino workloads to define each separate mail access type. Important mail factors The number of concurrent mail users has a significant affect on the overall recommendation for a Domino mail workload. Based on feedback from customers, the default concurrency ratings for each of the mail access types have been set to represent a typical mail environment. For a new Domino mail user, it is usually appropriate to keep the defaults. If you are familiar with the way your mail users are working, see 3.4.1, “Concurrent users” on page 24, for general Domino sizing. Extremely large databases Another factor that has some affect on the performance and sizing of a Domino mail solution is the size of the actual mail databases being deployed along with the size of the Domino directory. It has been noted that as the size of these databases grows, the more complex the average user becomes, which results in heavier loads. The disk storage per registered user needed for the mail databases parameter in WLE currently effects only the disk capacity recommended by the user. For extreme environments with gigantic mail databases (for example, over 500 MB) and a huge Domino directory (for example, over 1 GB), it is possible to account for the extra load by defining a Generic Workload. In the Generic Workload profile, there is an option to add processor resources as appropriate to any type of work being deployed. 30 Domino for iSeries Sizing and Performance Tuning The Generic Workload can be a very versatile workload. It provides a controlled manner in which the user is allowed to enter various raw parameters of a workload. These parameters include such things as the interactive and non-interactive CPW processing requirements of a workload, the amount of memory, the number of disk arms, and the amount of disk storage needed by the workload as shown in Figure 3-4. Figure 3-4 Generic workload panel When you define mail users with databases larger than 200 MB, it is most likely appropriate to use the heavy mail setting for those users. Finally, as new versions of mail clients, Domino releases, and OS/400 releases are announced, measurements and analysis are done to keep WLE up-to-date with the latest technology being deployed on the iSeries. Therefore, it is important to continue to use the latest version of the WLE that is available. 3.6 Domino application concepts in Workload Estimator When we talk about Domino applications, we mean custom written or purchased Domino databases that implement certain collaboration logic. The logic of those applications is typically more complex like Mail and Calendaring. Chapter 3. Sizing Domino for iSeries using the Workload Estimator 31 Sizing Domino applications is a very complex task due to the varying types of applications as well as the varying ways in which applications can be designed. Because of these variations, each application places quite different demands on system components such as CPU, memory, and disk. The application discussion concentrates on the interpretation of some of the application concepts discussed in WLE as well as a short discussion on the Domino application rating used to rate the complexity of an application. To accompany the application rating discussion there are also a few questions that may help in determining a rating of an application if it does not compare to any of the examples included in WLE. 3.6.1 Example application comparisons Today no good tools exist on any Domino platform to simulate the load characteristics of an application and map the performance to a specific configuration for a desired number of users. As stated earlier, the most accurate way to size a system for a specific application is to benchmark the application on a desired platform with the exact resource configuration. Since that option is not always feasible or available, Workload Estimator directs you to arrive at a ballpark estimate of the iSeries capacity you will need by providing examples of several applications and assisting the user in comparing their application to one or more of the examples. There are currently seven example applications that represent typical Domino applications being deployed as shown in Table 3-2. For a more detailed definition on some of the example Domino applications, go to: http://www-1.ibm.com/servers/eserver/iseries/domino/d4appsz.htm Table 3-2 User-written application examples Once the comparison is made, you make an educated guess at an application rating for your own application. If your application fits closely to one of the specific examples in WLE, choose the appropriate example application from the application type list as shown in Figure 3-5. The application rating is then selected by WLE, based on which application type was chosen. The application rating found in Workload Estimator is a complexity rating of the entire application. More specifically, the rating is used to determine resource requirement projections for the different applications. Application example Rating Custom (default) 6 QuickPlace Light Publishing 3.5 Web Document Library Application 2.4 Customer Relationship Management 5.5 Web Shopping Application 6 Web Shopping DB2/400 Application 6.3 Domino.Doc (light) 0.5 Domino.Doc (medium) 1.5 Domino.Doc (heavy) 4.5 32 Domino for iSeries Sizing and Performance Tuning Using the application rating scale Because there are a wide variety of Domino applications being deployed today, a majority of them are probably not going to compare exactly to the examples described in the Estimator. Therefore, the ratings specified for the example applications in Workload Estimator can be used as references when comparing those applications to your applications. Once you determine whether your application places more or less demand on the system resources than the example applications, you can select a custom application rating. If you selected the custom application type, it is up to you to specify the application rating. An application rating of 1 would represent a very light application; a rating of 3 would be a moderate application; and something around an 8 would be a fairly heavy application. Figure 3-5 Workload Estimator Application Definition As new technology becomes available, Domino applications continue to grow in complexity. For that reason, the application rating scale does not stop at 8. There is room in the scale to rate an application as heavy as 11. Here is an example of how to use the sample applications to compare with your application and project an appropriate application rating. The first task is to figure out which application most resembles your solution. Once this is completed, there are a couple of ways to proceed:
If it is determined that your application users will generate similar activities as the sample application, but at a different rate, then use an application rating that is proportionately more or less in the same ratio as the rate difference. Chapter 3. Sizing Domino for iSeries using the Workload Estimator 33 For example, perhaps you are planning to implement a Web shopping application, so that the users are completing a group of transactions similar to the one described for Web Shopper, but are going to be completing the sequence at a rate of 180 seconds instead of 90 seconds. Then this application would use a rating that is approximately half the rating of Web Shopper. Your application is half as heavy as the example.
It is also possible that the rate and sequence of transactions for the expected workload is quite similar to an example in the Estimator, but the users will be completing heavier or lighter transactions. There is a little more guess work that is required to go into this comparison. It is important in this example to have a good idea of the complexity of each of the transactions that are completed. It is then up to you to decide how much of a difference the transactions are compared to the example application script. Section 3.7, “Characterizing existing applications” on page 35, includes some questions and metrics that are useful when trying to make the complexity classification of an application. Users versus HTTP hit rates When defining a mail or application profile in Workload Estimator, it is necessary to specify a number of active5 users. For Web applications, such as the Web Shopping example, it can be difficult to predict the number of concurrently active users. From an HTTP standpoint, it is a bit easier to discuss hit rates. The HTTP protocol does not have the concept of a session and, in many cases, the browser users don’t even need to authenticate. The Web master knows how often a certain URL has been visited, but it is difficult to find out whether 100 hits have been performed by a single person or by 100 different people. When you use one of the artificial Web applications listed in Table 3-2 on page 31 to rate the complexity of your real application, you also need to specify a number of concurrently active users to determine how often they are called. Those applications are described in detail in the Domino for AS/400: Application Sizing Examples white paper at: http://www-1.ibm.com/servers/eserver/iseries/domino/d4appsz.htm The white paper can help you to calculate the active users, if you know only the HTTP hit rate. Table 2, “Web Document Library Estimated Sizing” in the white paper (partially shown here in Table 3-3), allows you to determine a ratio between concurrent users and hits per second by dividing those two numbers. That way you find out that (according to this model application) 1 hit per second is generated by 15 active users, or in other words, 100 active users produce 6.6 hits per second. For example, assume you consider your own Web application to have the same complexity as the Web Document Library sample application and you expect 120 hits per second. Applying the ration described above would result in 1800 (120 times 15) concurrently active users. You can specify 1800 registered users and concurrency rate of 100% or 3600 registered with 50% active users. Table 3-3 Web Document Library Estimated Sizing 5 The Workload Estimator asks first for registered users, but what really counts are the active users. This is based on the concurrently active percentage multiplied by the number of registered users. Model Processor feature # N-way # Concurrent users Hits per second Hits per day 270 2422 1 247 16.5 1,422,720 2423 1 397 26.5 2,286,720 2424 2 777 51.8 4,475,520 34 Domino for iSeries Sizing and Performance Tuning This means that (concurrently active) users generate various numbers of hits during an active session with the application. Hit rate values, along with the equivalent number of concurrent users corresponding to those hit rates, are included in the definition and description of the Web applications in the white paper. Domino.Doc now part of the Domino application profile Note, Domino.Doc is now included in the Domino application profile. In older versions of the WLE, there was a separate profile for Domino.Doc. If you are used to using the Domino.Doc profile of the Domino workload, it will be simple to transform the information into the Application profile. There were originally three different complexities of the Domino.Doc application, that had different rates in which the script was completed. These three profiles are still included in the Estimator. They are now application-type examples to be used separately or to be used for comparisons with other custom written applications. If you are used to combining complexities of your Domino.Doc users with the complexity sliding bar, there are two ways to represent this in the Estimator. The easiest and most accurate way is to define two or three separate Domino workloads and define the different Domino.Doc user complexity groups each in a separate workload. It is also possible to make some estimates at what the average complexity your users would be and use only one Domino workload. This way introduces another level of uncertainty and should be used with caution. 3.6.2 Database capacity In this context, database refers to a DB2 UDB database and its associated processes, not a Notes database. Only the time spent in DB2 database code as well as access methods, such as SQL and Query/400, is considered to be DB2 database processing. The majority of the processing in these methods is during the accessing of database records through SQL, ODBC, JDBC, Query/400, DRDA, and other database access methods. One specific case in WLE when this metric is important is when sizing a V5R1 Dedicated Server for Domino iSeries model. For an in-depth discussion on the V5R1 Dedicated Server for Domino, refer to the white paper at: http://www.ibm.com/eserver/iseries/domino/dsd.htm The reason for discussing the database capacity as part of the general Domino sizing concepts of WLE is because of the way it is now used in V5R1 of OS/400 for Dedicated Server for Domino systems. The V5R1 Dedicated Server for Domino logic allows for complementary workloads to be deployed as long as certain criteria are met. To account for this, the Estimator now allows for complementary workloads to be projected onto a Dedicated Server for Domino, but with these requirements in place:
Domino must be the majority of the projected total work defined in WLE.
Projected DB2 UDB utilization is less than the rated DB2 capacity for the model. The rated DB2 capacity is equal to 15% of the CPU for Dedicated Server for Domino models. For more information on optimizing an application's use of DB2 database processing, refer to the iSeries Information Center for V5R1 at: http://publib.boulder.ibm.com/html/as400/v5r1/ic2924/index.htm 820 2425 1 397 26.4 2,286,720 Model Processor feature # N-way # Concurrent users Hits per second Hits per day Chapter 3. Sizing Domino for iSeries using the Workload Estimator 35 3.7 Characterizing existing applications Because of the complexity and variety in the Domino applications being deployed, here are some questions and characteristics that would be useful to classify an application that was not comparable with any of the examples found in WLE:
Where is the accessed data located? – In small databases? – In a large number of databases? – In DB2 databases?
How is the data structured for accessibility?
How is the load distributed on the server? If the answers to these questions lead to an observation that your solution is quite complex, it is appropriate to suggest a fairly high application rating. The following sections offer some examples of design metrics that may have an impact on the server load or capacity as well as user response times and can help answer the questions above. Some application characteristics do not necessarily add extra load to the server that, in turn, affects the total performance, but have an affect on the user response time. Too many views or agents One metric that could potentially have quite an impact on server performance if not tuned appropriately is the number of views. If your application has hundreds of views and is constantly refreshing them, the application could prove to be heavier than previously noted. Similarly, the number of agents in an application could cause a significant performance hit. The number of add-in tasks that are active in the application also has a significant impact on the server load. DB2 access and network overhead Another factor that could impact the actual load put on the server by an application is the extent to which the application accesses backend data like DB2. The more network layers and connections that are involved in accessing the data, the bigger the potential is for longer response time. After you determine if the application has any of these characteristics, and to what extent, you can now select an appropriate application rating. There is no exact science to calculate an application rating based on the design components of an application. However, having knowledge of the metrics discussed here will help you to determine the appropriate input values when using WLE. Applications currently running on an other platform If you currently have a Domino application today running on a platform other than iSeries, there are a couple of ways to obtain useful information needed to define the profile of the Domino solution in WLE. As mentioned previously 3.2, “Benchmarks to determine the load of specific workloads” on page 21, it is possible to use the published NotesBench results from a different platform to help characterize and define a specific workload for WLE. 36 Domino for iSeries Sizing and Performance Tuning 3.8 Additional options for more accurate estimation Aside from the actual Workload definitions, there are additional input capable parameters in the tool that give the user options to direct the recommended system. To access these additional parameters, go to the Navigation bar found in the top left corner of every WLE screen. Choose Edit followed by Options. This opens up the Options page (Figure 3-6) where you can adjust assumptions that apply to the entire iSeries environment that you are estimating. Figure 3-6 Input-capable parameters in WLE These options affect both the size of the projected workload or workloads and the manner in which the Estimator combines multiple workloads. Here are some descriptions to help aid in using these options to guide the Workload Estimator in recommending an appropriate system for your solution. Target Processor Utilization This parameter affects how the tool selects systems. The Estimator attempts to select a system that will provide processor utilization no greater than the specified Target Processor Utilization. To select a system that will provide adequate room for incidental growth as well as base operating system resources, the Workload Estimator default is 70%. Chapter 3. Sizing Domino for iSeries using the Workload Estimator 37 You may change this value based on specific needs. However, we do not recommend setting the default utilization above 70%. This could result in performance problems during peak usage times, when there are “spikes” in the system utilization. It is important to remember that, when you adjust this option, the recommendations of the WLE are estimates. This means that some range should be allowed for around the projected CPU Utilization. Target Interactive Feature Utilization The Estimator attempts to select a system whose interactive feature will be used by less than or equal to the specified amount. The default for this parameter is 100%. Disk Storage Percent Full For each workload defined, the Estimator calculates the minimum amount of disk storage that will fulfill the workload requirements. Before you select a system for the estimation, the Workload Estimator adds together the storage for each workload, plus extra storage requirements for base operating system requirements. If you want the Estimator to calculate extra disk storage as a percentage of the total required, you can lower this parameter. This is because the parameter is defaulted to 100%. This means that the Estimator assumes that the workload or workloads can consume all of the disk storage available for the selected system. If you lower this value, for example to 75%, then the Estimator would “save” 25% of the disk storage available to the system and only allow the workloads to consume 75% of the storage available. This would be an appropriate option to change if you knew that, for example, you were sizing a mail workload and your users' mail databases were projected to grow in size. Disk Storage Type The 10K RPM drives have a faster latency than the 7K RPM drives. This means that they have a faster spinning rate for reading and writing to disk. Potentially, the 10K drives could result in approximately ten to fifteen percent more throughput with most workloads. Target Family The new iSeries servers with the new, faster processors result in better performance over the previous 170, 720, 730, and 740 models. The Workload Estimator will default to the highest available technology. Unless there is a specific case in which you want to size your solution on a previous technology system, we suggest that you leave this parameter with the default. 38 Domino for iSeries Sizing and Performance Tuning 3.9 Using IBM Workload Estimator for iSeries results The Selected System that is recommended by WLE is a processor model that will support the processing, disk, and memory requirements calculated from the questions answered in each of the workloads defined. All of the systems capable of supporting the work defined are identified and reported in the Selected System drop-down list as shown in Figure 3-7. Figure 3-7 Selected Systems page The smallest system model capable of supporting the resources required for the workload or workloads that is defined, as well as meeting the criteria specified in 3.8, “Additional options for more accurate estimation” on page 36, is presented as the recommended system. The Selected System panel actually includes the specific processor model, interactive feature, processor Commercial Processing Workload (CPW), interactive CPW, database capacity, processor utilization, memory, and the required number of disks and disk capacity. Through the options described in 3.8, “Additional options for more accurate estimation” on page 36, the user can modify how the WLE arrives at these recommendations for customizing. One suggestion is to lower the Target Processor Utilization. This would allow for projecting growth. It would also allow for spikes in the system utilization during peak hours of operation. Chapter 3. Sizing Domino for iSeries using the Workload Estimator 39 For specific cases on the opposite end of the spectrum, if you know that yours is very constant and there are no major peak times, it is possible to set the target utilizations a bit higher. After you set the appropriate options, it is time to recommend a system that is capable of deploying the defined solution. There are a couple of resources that are reported by the Workload Estimator that need a bit of explanation on how they refer to a Domino solution. Processor CPW The Processor CPW, reported in the results of the Workload Estimator (Figure 3-8), should not be treated as the total CPW in all cases. In some instances, such as the Dedicated Server for Domino, this metric is actually the non-Domino processor CPW. For a more detailed discussion on the Dedicated Server for Domino models and the non-Domino processor CPW, see the Dedicated Server for Domino white paper “Enhanced V5R1 Processing Capability for the iSeries Dedicated Server for Domino”, which is available on the Web at: http://www-1.ibm.com/servers/eserver/iseries/domino/pdf/dsdjavav5r1.pdf Figure 3-8 Processor CPW 40 Domino for iSeries Sizing and Performance Tuning Database Capacity The Database Capacity metric reported for the Selected System shown in Figure 3-9 is a measure of the percentage of the overall CPU that is used to perform the DB2 database (not Domino database) processing. Database processing generally includes processing time in SQL, ODBC, JDBC, Query/400, DRDA, and other database access methods. If the database CPU is fully consumed, this does not mean that all of the capacity on the system is being utilized. It is possible to add more work other than database processing to take advantage of the remaining cycles. Figure 3-9 Database Capacity Chapter 3. Sizing Domino for iSeries using the Workload Estimator 41 Disk Drives The Disk Drives (arms) value (Figure 3-10) is the number of disk drives required to support the defined workload. This number is determined by the speed of the drives and the number of disk operations per second necessary to do the work. It does not take into account the disk drive size available. You should always configure at least this number of disk drives, even though you do not need that much capacity. Figure 3-10 Disk Drives (arms) 42 Domino for iSeries Sizing and Performance Tuning Capacity (GB) The Capacity shown in Figure 3-11 is a separate calculation based on the amount of disk storage defined for the mail databases plus the storage needed for any application databases. The calculations for disk capacity and the number of disk drives are done separately. Only taking one into consideration could result in a number that is too low for the other. For example, by selecting large disk drives to satisfy the capacity requirements of WLE, there may not be enough arms for the best performance. It is also worth noting that the value reported for the Selected System requirements is a minimum requirement. By including more drives, the system will have better parallel access to the data, which would reduce the data access time significantly. Figure 3-11 Capacity (GB) Chapter 3. Sizing Domino for iSeries using the Workload Estimator 43 Growth Solution Growth Solution (Figure 3-12) is also included in the Workload Estimator Results. It is selected by applying the growth rate, defaulted to 50%, to all of the system resources used to determine the initial recommended system. It is possible, if the Immediate Solution is capable of supporting 50% more resource, to have a Growth Solution be the same system as the Immediate Solution. Figure 3-12 Growth Solution 44 Domino for iSeries Sizing and Performance Tuning Further information on both the Immediate and Growth Solutions is summarized in Figure 3-13. Figure 3-13 Immediate and Growth Solutions 3.10 Additional tips and insights The Estimator help text for the Domino workload is very helpful in answering questions about the basic information on the workload. It is impossible to cover all of the Domino material that is used in sizing and performance measurements necessary. This section covers a few of the items that are not found in detail in the WLE help text, but that should be useful when using the WLE to help size a Domino solution. Dedicated Server for Domino For specific Dedicated Server for Domino information, see Chapter 12, “The iSeries Dedicated Server for Domino” on page 341. You should also see the Dedicated Server for Domino white paper “Enhanced V5R1 Processing Capability for the iSeries Dedicated Server for Domino”, which is available on the Web at: http://www-1.ibm.com/servers/eserver/iseries/domino/pdf/dsdjavav5r1.pdf CPW versus CIW The Commercial Processing Workload rating of a system is determined based on measurements using a synthetical workload and not based on a real existing application. CPW is designed to evaluate the system and associated software in a commercial environment. It is not representative of any specific environment, but generally applies to the commercial computing environment. Chapter 3. Sizing Domino for iSeries using the Workload Estimator 45 The applications that follow the CPW patterns tend to have many jobs running short transactions using relatively few CPU instructions and spending much time waiting for I/O to complete. Usually this is an environment where more time is spent by system code performing DB2 operations, rather than by the application executing processor instructions. CPW ratings can be used best when the application you are sizing is expected to be used for traditional business uses such as order entry, billing, etc. Domino applications are not sized well using CPW ratings, because typically the ratio of processing time to I/O time for Domino is very different from CPW. When sizing a Domino application, it is more important to consider the necessary CPU cycles per user per time versus the time waiting for I/O. For this reason, the Compute Intensive Workload (CIW) metric has been introduced for newer iSeries models to improve Domino sizing projections. CIW ratings are used when most of the work by the application is done within the application code instead of system services. For example, a Domino Mail and Calendar workload might spend 80% of the total processing time outside of OS/400, while the CPW workload spends most of its time in OS/400 database code. Since the instruction path length tends to be longer than CPW-like workloads, it is very likely that processors will spend less time switching from task to task. Therefore, it is not as important to have a big cache. The cache affect is also due in part to the data being accessed. Because CPW does more I/O, the cache plays a bigger role than it does for CIW. Some L2 cache is very important for Domino, especially for applications. That is why we always recommend that you choose those iSeries or AS/400 models for Domino that have L2 cache (see Table 3-4 on page 48 and Table 3-5 on page 48). But Domino applications that are more CPU-intensive and more comparable to CIW benefit more from additional processor speed rather than a bigger cache. Also, because CIW-like applications spend more time on individual processors, they tend to scale better on multiprocessor systems, than mail and CPW does. You can find further information on CIW in Appendix A of iSeries Performance Capabilities Reference Version 5 Release 1, SC41-0607. You can also find CPW and CIW ratings for iSeries servers in Appendix D of the same manual. Typical, Simple, or Mail and Calendaring Users When using Workload Estimator to size for a Domino mail solution, it is important to understand the relationships between these three metrics:
Simple Mail Users (SMU)
Mail and Calendaring Users
Typical Mail Users Due to the NotesBench audit rules, we are not allowed to publish results generated by NotesBench workloads without having them officially audited. Because it is not possible to audit results for every possible server configuration, the SMU and MCU metrics are used.
Simple Mail Users – Open mail database that contains documents that are 1 KB in size – Open the current view – Open five documents in the mail file – Categorize two of the documents – Compose two new mail memos 1 KB in size (every 90 minutes) – Mark several documents for deletion – Delete documents marked for deletion – Close the view 46 Domino for iSeries Sizing and Performance Tuning
Mail and Calendaring Users – Open the mail database that contains documents that are 10 KB in size – Open the current view – Open five documents in the mail file – Categorize two of the documents – Compose two new mail memos 10 KB in size (every 90 minutes) – Mark several documents for deletion – Delete documents marked for deletion – Create one appointment (every 90 minutes) – Schedule one meeting invitation (every 90 minutes) – Close the view We decided that these two scripts do not represent a typical mail workload. Therefore, we have determined that a typical user is three times as heavy as a SMU and twice as heavy as an MCU. This means that a given server that can support x number of Typical users can support 3x SMU and 2x MCU. To translate these metrics into Workload Estimator terminology, the equivalent of a Typical user is a moderate mail user. Registered users versus active users Note, that so far we were only talking about active users. Most people asking for the proper size of a Domino server will start similarly to: “We will have xx Domino users ....” The number xx includes typically all users who will use the Domino server at some point in time. In the terminology of the Workload Estimator, these are called registered users. The number of registered users is the first question to be answered for each WLE workload. However, this number alone cannot be used as a base for sizing, unless you respond to the following questions. The first question and probably most important questions is: “What is the percentage of concurrently active users?” Multiplying that value with the registered users brings you the number of active users. For example, the Model 820-2457 is rated at 6660 MCU at 70% CPU. If you specify 6660 users in Workload Estimator6, the Estimator will project a utilization of 69.9% CPU utilization of the 820-2457 at 50% concurrency. If the concurrency rating is changed to 100%, the number of users projected at 70% is 3330, or one half of the MCU number. 3.11 Consolidating Domino servers from other platforms Many times we hear a question like this: “How does an iSeries (or AS/400) Model xxx compare to server yyy running operating system zzz with a processor of nn MHz clock speed?” Most often this question comes up when someone plans to consolidate multiple servers from a PC or UNIX platform to one or more iSeries servers. The correct answer is always: “It depends!”. It depends on:
The complexity and type of Domino application you are planning to run, as detailed in the previous sections.
The entire server configuration. That is, processor clock speed (“MHz”) can only set an upper limit of the potential capabilities at most. Many other factors play an important role and can further limit the capabilities if they are not sized appropriately, such as size of memory, the number (not size) and speed of disk drives, and the capacity of communication lines. 6 Not selecting any additional options such as clustering or other applications. Chapter 3. Sizing Domino for iSeries using the Workload Estimator 47 While these components can be set by the person who configures the hardware, there are other factors depending on the internal architecture and design of the server. Such factors include: bus speed, internal data transfer band width, multitasking overhead, and others. Having said that, what can you do to estimate the appropriate size of an iSeries server, when you already know what server is needed on a different platform? Similar to what we said for sizing a completely new server, you can use benchmark data to provide an order of magnitude, as long as you are aware of the limitations of this method. How important is the processor clock rate (MHz)? Due to the dramatic developments in the area of processors during the last decade, it seems to many people that the clock rate of the processor (MHz) appears as the one and only indication of the speed of a computer system. However, the processor is the most important component and is not alone responsible for performance. There are some facts that describe where the processor speed is more or less important:
Capacity is affected by a complicated combination of processor, I/O, and overall efficiency factors. All applications need a different proportion of these resources. That's why we use CPW, CIW, MCU, or other kinds of well-defined workloads to classify applications.
MHz primarily affects response time of CPU intensive transactions. Some applications are more processor intensive than others. More importantly, however, is the fact that a short response time measured for a single transaction does not indicate at all how the server will be able to handle hundreds or thousands of users.
On servers with multiple processors, “adding up the MHz” will underestimate the capacity of iSeries servers and probably overestimate the capacity of servers on some other platforms due the relative efficiencies of the architectures.
When comparing to existing servers, it is necessary to know at which CPU utilization they work and take it into account. You can use the Domino platform statistics (see 5.1.4, “Platform-dependent statistics” on page 137) to find out how the utilization of the processor can raise during peek times. What about CPW? The Commercial Processing Workload should not be used to estimate for three main reasons:
Most, if not all, Domino applications show a different behavior than the one defined in CPW.
The CPW published for the Dedicated Server for Domino applies only to the non-Domino workload running on a Dedicated Server for Domino. See 12.2, “Dedicated Server performance behavior and capacity” on page 342, for more information.
CPW is normally not published for any non-iSeries platform. Simple Mail Users or Mail and Calendaring Users If the existing servers have a NotesBench audit published, you can roughly compare the Simple Mail Users with the NotesBench R4.5 Mail users and the Mail and Calendaring Users with NotesBench R5 Mail and Calendaring Users. While this method is probably the most accurate, there still some restrictions to consider:
The term users in SMU, MCU, and NotesBench users should be considered as a benchmark unit rather than real users. The typical mail users, as described in “Typical, Simple, or Mail and Calendaring Users” on page 45, may come close to reality in a mail-only environment, but should never be used to estimate the workload for other Domino applications. 48 Domino for iSeries Sizing and Performance Tuning
Many Domino applications tend to have a more CPU intensive workload, which is more comparable to CIW as it is to mail users.
The benchmarks assume a balanced configuration, that is an appropriate proportion of CPU, memory, and disk resources, which may not be true for an existing server.
When comparing to existing servers, their resources may not be fully utilized. When keeping the above points in mind, comparing the NotesBench and MCU ratings of existing servers is still the most reliable method to estimate the appropriate configuration of an iSeries server. In addition the CPU clock rating, together with actual CPU rating can be used to verify the necessary number and type of iSeries processors. In any case, the Workload Estimator should also be used to ensure a balanced system, as described in the previous sections or this chapter. The following sections should help you to gather some information about the different iSeries models and compare it to benchmark data published bey the NotesBench consortium on the Web at: http://www.notesbench.org 3.11.1 Capabilities of various iSeries and AS/400 models Table 3-4 and Table 3-5 on page 48 show the Mail and Calendaring Users (estimated at 70% CPU capacity) for the those 270 8xx Models announced in 2001. The other tables on pages 49 through 50 show the SMU and MCU ratings of older iSeries and AS/400 models. You may use the tables to compare the potential capacities of the different iSeries models. Although the ratings Mail and Calendaring Users and Simple Mail Users are not officially the same as the NotesBench ratings, the tables can give you a very rough idea to compare the capabilities of a certain iSeries model with another server which does have published NotesBench ratings. Table 3-4 and Table 3-5 describe performance capabilities of the current (as of the year 2001) iSeries models as they relate to MCU. Table 3-4 iSeries Dedicated Server for Domino performance details Table 3-5 Further iSeries servers Model Proc. feature CPUs Chip speed L2 Cache MCU rating Typical mail 820 2458 4 600 MHz 4 MB 11,810 5,905 2457 2 600 MHz 4 MB 6,660 3,330 2456 1 600 MHz 2 MB 3,110 1,555 270 2454 2 600 MHz 4 MB 6,660 3,330 2452 1 540 MHz 2 MB 3,070 1,535 Model Proc. feature CPUs Chip speed L2 Cache MCU rating Typical mail 840 2461 24 600 MHz 16MB 77,800 38,900 2420 24 500 MHz 8 MB 55,610 27,805 2418 12 500 MHz 8 MB 28,430 24,215 830 2403 8 540 MHz 4 MB 22,900 11,450 Chapter 3. Sizing Domino for iSeries using the Workload Estimator 49 Table 3-6 shows Domino mail workloads for the previous generation of the 270 and 8xx models. MCU results are based on 70% CPU utilization and should not be compared to the audited NotesBench numbers. Table 3-6 Domino workloads on first generation of 270 and 8xx models Table 3-7 shows Domino mail workloads for 170 and 7xx models. Table 3-7 Mail workloads for 170 and 7xx models 2402 4 540 MHz 4 MB 10,680 5,340 2400 2 400 MHz 2 MB 4,490 2,245 820 2438 4 600 MHz 4 MB 11,810 5,905 2437 2 600 MHz 4 MB 6,660 3,330 2436 1 600 MHz 2 MB 3,110 1,555 270 2434 2 600 MHz 4 MB 6,660 3,330 2432 1 540 MHz 2 MB 3,070 1,535 Model Proc. feature CPUs Chip speed L2 Cache MCU rating Typical mail Model Proc. feature CPUs Chip speed L2 Cache MCU rating Typical Mail Simple mail 840 2420 24 500 MHz 8 MB 55,610 27,810 83,420 840 2418 12 500 MHz 8 MB 28,430 14,220 42,650 830 2403 8 540 MHz 4 MB 20,910 10,460 31,370 830 2402 4 540 MHz 4 MB 10,680 5,340 16,020 830 2400 2 540 MHz 2 MB 4,490 2,250 6,060 820 2398 4 500 MHz 4 MB 9,890 4,950 14,840 820 2397 2 500 MHz 4 MB 5,610 2,810 8,420 820 2396 1 450 MHz 2 MB 2,570 1,290 3,860 820 2395 1 400 MHz - 1,650 830 2,480 270 2253 2 450 MHz 4 MB 5,050 2,530 7,580 270 2252 1 450 MHz 2 MB 2,570 1,290 3,860 270 2250 1 400 MHz - 1,600 800 2,400 270 2248 1 400 MHz - 810 410 1,220 Model Proc. feature CPUs Chip speed L2 Cache MCU rating Typical mail Simple mail 740 2070 12 262 MHz 4 MB 17,860 8,930 26,790 740 2069 8 262 MHz 4 MB 12,350 6,180 18,530 730 2068 8 262 MHz 4 MB 10,960 5,480 16,440 50 Domino for iSeries Sizing and Performance Tuning Table 3-8 shows the performance capabilities on 6xx and Sxx models. These results are at 70% CPU utilization and should not be compared to the audited NotesBench results. Use these figures when comparing AS/400 systems only. Table 3-8 Simple mail users on 6xx and Sxx models 730 2067 4 262 MHz 4 MB 6,900 3,450 10,350 730 2066 2 262 MHz 4 MB 3,530 1,760 5,290 730 2065 1 262 MHz 4 MB 18,70 940 2,810 720 2064 4 255 MHz 4 MB 4,920 2,460 7,380 720 2063 2 200 MHz 4 MB 2,570 1280 3,850 720 2062 1 200 MHz 4 MB 1,340 670 2,010 720 2061 1 200 MHz - 940 470 1,410 170 2388 2 255 MHz 4 MB 3,150 1580 4730 170 2385 1 252 MHz 4 MB 1,550 780 2,330 170 2292 1 200 MHz - 940 470 1,410 170 2291 1 200 MHz - 450 220 670 170 2290 1 200 MHz - 280 140 420 170 2289 1 200 MHz - 190 100 290 Model Proc. feature CPUs Typical mail Simple mail 600 2129 1 33 100 2134 1 43 130 2135 1 66 200 2136 1 116 350 S10 2118 1 66 200 2119 1 116 350 620 2179 1 123 370 2180 1 166 500 2181 1 300 900 2182 2 666 2000 S20 2161 1 166 500 2163 1 300 900 2165 2 666 2,000 2166 4 1,133 3,400 640 2237 1 466 1,400 Model Proc. feature CPUs Chip speed L2 Cache MCU rating Typical mail Simple mail Chapter 3. Sizing Domino for iSeries using the Workload Estimator 51 3.12 Additional resources The following Web sites offer you additional resources in sizing Domino for iSeries:
Dedicated Server for Domino white papers: http://www.ibm.com/eserver/iseries/domino/dsd.htm
Application sizing examples: http://www-1.ibm.com/servers/eserver/iseries/domino/d4appsz.htm
NotesBench consortium: http://www.notesbench.org 2238 2 800 2,400 2239 4 1,533 4,600 S30 2257 1 466 1,400 2258 2 800 2,400 2259 4 1,533 4,600 650 2240 8 2,900 8,700 2188 8 5,500 16,500 2243 12 4,166 12,500 2189 12 7,966 23,900 S40 2207 8 5,500 16,500 2261 12 4,166 12,500 228 12 7,966 23,900 Model Proc. feature CPUs Typical mail Simple mail 52 Domino for iSeries Sizing and Performance Tuning © Copyright IBM Corp. 1999, 2002 53 Chapter 4. Basic concepts of performance analysis This chapter provides recommendations on evaluating Domino performance. The primary tool used for collecting performance data on iSeries is Collection Services (previously called Performance Monitor). If a performance problem cannot be identified by the data collected with Collection Services, the next tool to use is the Performance Explorer. The collection functions and related commands in Collection Services and Performance Explorer are part of the OS/400 operating system. The reporting and analyzing functions for Collection Services and Performance Explorer are part of Performance Tools for AS/400 Licensed Program Product (5769-PT1). Performance Tools is a separate product that must be purchased. Its cost is based on the system model. Contact your sales representative for more information. An iSeries server can be viewed as a stable, well-balanced system where every job has a slot (run priority) of its own. System jobs in the iSeries server support all basic functions required by the applications. All Domino tasks run as high-priority batch jobs. Some have characteristics that are comparable to traditional (5250-based) interactive jobs. Changing a run priority of a job, without careful planning, may work for some time. In the long run, it may cause unpredictable results. Therefore, you should not add applications without proper preparation. As with any performance related activity, there are some general considerations that need to be applied when doing performance tuning for Domino on the iSeries server. Performance evaluation and management for Domino for iSeries is similar to performance tuning on the iSeries server. However, special considerations are given to Domino itself and to its performance management. 4 54 Domino for iSeries Sizing and Performance Tuning 4.1 Defining a Domino for iSeries performance methodology Anytime you enter a performance-oriented work environment, you need to consider some general guidelines or what you may call a “high-level view”. To help with this view, you need to:
Develop objectives such as customer expectations
Define short-term and long-term goals
Implement a good strategy for deployment of your objectives
Set a benchmark for evaluation of your level of progress When we talk about objectives, we refer to a clear definition of what the customer expects in a manner that can be measured. It is very important that the customer clearly understands their performance objectives before the development of your performance management strategy. Performance analysis can be a long process. A well-defined strategy can make a difference in the degree of your success. Performance management generally consists of performance measurements and a trend analysis. Some type of performance problem may already exist that may have warranted your involvement. Setting up a performance management solution for Domino on the iSeries server is more complex than the traditional iSeries performance tuning. You first need some understanding of Lotus Domino. This is essential to understanding the relationship between the processes that are taking place in Domino and how these processes affect the performance of the iSeries server. Because only limited tools are currently available to provide the analysis of Domino for iSeries, you may be required to gather some of the necessary information in making your performance determinations. This redbook was developed to assist you with that process. This redbook refers to iSeries performance measurement components such as Queuing Multiplier, CPU Service Time Equation, and so on. If you require a better understanding of the components of iSeries Performance Management, we recommend that you see AS/400 Performance Management, SG24-4735, and Performance Tools for iSeries Version 5, SC41-5340. The best practice for understanding Domino Performance is to have a good understanding of the information in the Domino 5 Administration Help (help5_admin.nsf) database and the iSeries specific Domino for AS/400 Help (as400hlp.nsf). Lotus Domino System Administration classes offered by Lotus Education are also available. The majority of the performance tuning parameters available for Lotus Domino is found in detail in the handbook or taught in a System Administration class. When gathering performance information on Domino on the iSeries server, basic iSeries techniques, in conjunction with some additional gathering of Domino information, provide a good foundation for making performance management decisions. Defining performance gathering objectives Consider the following points when collecting performance data:
Collect performance data against a set of agreed-to defined objectives. These objectives set goals and expectations and should focus on specific performance issues.
Plan on meeting one objective at a time. You may start out with an objective of end-user response time and be asked to pursue optimal transactional throughput. There is a direct relation between your specific objective and the way that the data is collected for that objective.
Select the criteria for data collection that is a direct result of the performance objective definition. It should include such things as duration, type of information to be collected, and so on. Chapter 4. Basic concepts of performance analysis 55
Select measured data showing a medium-to-heavy workload that the customer feels accurately reflects the typical workload.
Define a peak period (or critical applications) to measure.
Ensure measurements are made under similar conditions. That is, the same workload, number of active workstations, and so on must be valid for purposes of comparison. Making valid performance data evaluations and recommendations Consider the following points when evaluating the data:
Consider how performance changes may affect the rest of the business organization such as different user groups.
Involve the necessary decision making individuals with discussion-based feedback on issues such as cost, objective clarification, and time schedules.
Evaluate business growth on additional resources. Performance information that you gather now is critical in trend analysis of future system requirements.
Evaluate the pattern of your system workload. Know whether it is steady, increasing, or decreasing, and what percentage of utilization you are using on your system.
Take several measurements before reaching a conclusion. You need a comparative measurement to find acceptable or unacceptable system performance. Implementing performance improvement changes Consider the following points when implementing performance changes:
Give any change that is made ample time under various working conditions. Managing system performance is a continuous process involving system changes that may not produce an immediate performance improvement.
Make and measure changes in a controlled environment.
Make only one change at a time. This allows you to gauge the impact of each change. 4.1.1 Relationship between Domino and the iSeries server Under OS/400, Domino is considered an application that runs within its own subsystem. However, the ramification of putting Domino on your iSeries server can be far reaching. Because Domino has its own server, it has the capability of generating tasks that can compete for system resources with limited administrator control. Domino, by default, runs everything at the same priority. Run priority is further described in 4.9.1, “Run priority” on page 64. This can have a major impact on an iSeries server that has to share resources between Domino and additional business applications. Domino application developers can create complex workflow or other GUI-based client/server applications. They can also route information automatically and exchange information with enterprise software such as relational databases and transaction processing systems. Today, by adding Domino, the iSeries server enters into a more complex, collaborative environment. This environment requires performance specialists to consider processes that may require looking beyond standard performance techniques. As you come to understand, Domino acts as its own entity on the iSeries server, running Domino transactions to meet the Domino server requirements as well as the requirements of the users. Transactions are also run for any external system requirements such as requests from the OS/400 operating system. Because Domino is so diverse, Domino on the iSeries server can be optimized, with proper planning, to take advantage of OS/400 assets such as scalability. The iSeries server 56 Domino for iSeries Sizing and Performance Tuning can also be optimized to take advantage of Lotus Domino assets such as database clustering. Using the loopback option to cluster Lotus Domino databases on a single iSeries server provides an example of efficiency and high availability that is unmatched on other platforms. The main function of the Domino server is to service its users by providing database access, mail routing, and so on. Frequently, these tasks require interaction with the OS/400 operating system, such as the AnyMail FrameWork for routing messages to the Internet or when Domino users access DB2 UDB for iSeries databases directly through Domino. Although these and other interfaces to the OS/400 operating system use system resources, the Domino server tasks themselves should be the primary focus of performance considerations. If you understand some of the components of Domino, you should understand the advantages that Domino can provide under OS/400, as well as the performance implications. Some of the Domino components are:
Transactions
Domino objects
Domino base function definitions 4.2 Understanding Domino on the iSeries server The following sections explore the Domino components in greater detail. 4.2.1 Transactions in Domino In Domino, a transaction is an action that takes place. It requires a response from the Domino server to complete the task, for example, open, read, update, and close. Generally, Domino activities are created from the use of several transaction requests to complete a specific function. Transactions can be used to provide server functions as well as database manipulation. Here, we do not focus so much on the specific function. The focus is on the average response times for a task, which may be comprised of several functions. The goal is to find the average response time of a specific task and of all of the tasks collectively. 4.2.2 Domino base functions The Domino server has a variety of tasks available to provide functionality. There are some tasks that are common to most implementations. We consider these tasks to be the base functions of Domino. Depending on the function of the task, it is considered either as an interactive task or a background task. For performance tuning purposes, the Domino tasks shown in Table 4-1 are identified as a base Domino function. Table 4-1 How base tasks are executed within Domino Tasks How tasks are executed One Router thread (Mail Router) Background task One Replica Task (Replication) Background task AdminP running Background task Events and Reporting enabled Background task One Updater (Indexing) Interactive task C&S enabled (Calendar & Schedule) Interactive task Chapter 4. Basic concepts of performance analysis 57 Table 4-1 indicates how base tasks are run within Domino. Interactive tasks are directed more toward responding to a Notes Client request. Some tasks are configured with multiple threads from within Domino. 4.2.3 Domino advanced functions The following Domino tasks are considered to be the advanced tasks in Domino. You can increase performance by isolating resources and priorities when these advanced services are configured on individual servers.
Clustering and failover Clustering Lotus Domino databases together on one iSeries server provides high availability and failover that can be managed and optimized for performance from a single point of administration. By combining Domino clustering with the ability to configure multiple Domino servers on the same iSeries server, you provide a superior combination from a performance standpoint because clustered database updates do not have to travel across the network.
SMTP and HTTP By creating SMTP- and HTTP-specific Domino servers as separate servers, you can use the native iSeries performance tools to adjust the priority of these servers to a lower level if desired. With SMTP configured on an individual Domino server, there may be some additional administration. However, if a problem arises with Internet mail, you can recycle the server without affecting the active Lotus Notes users. By configuring the Lotus Domino HTTP on an individual server, you can minimize the effect of HTTP requests on Lotus Notes users. Also, if an HTTP request begins to loop, you can recycle the HTTP server without affecting the active Lotus Notes users. 4.3 Domino tasks on the iSeries server Tasks are specified within the notes.ini file load when the Domino server is started. Domino tasks are batch immediate. Therefore, there are no interactive jobs to see. Instead, you must monitor the actual Domino threads. This is accomplished through the standard iSeries Performance Tools. The initial startup of the Server Tasks causes approximately 8 to 10 threads to activate on the OS/400 operating system. Each thread reserves about 4K of memory. When additional users sign on, up to 250 threads become active. This causes the iSeries CPU resource utilization to increase for a short period of time. One of the most important points to keep in mind when you are working with performance for Domino for iSeries is that you have to deal with two memory managers. By design, the OS/400 operating system has its own memory manager. The Work with System Status (WRKSYSSTS) command provides access to main storage where you can increase pool sizes, the maximum number of active jobs in the pool, and so on. Domino also has a memory management capability that is controlled by parameters in the notes.ini file. If you tune only for iSeries performance, you may see limited or no performance improvement. You must address Domino memory management in conjunction with the iSeries performance tuning. Having the parameters set correctly and working together for both memory management tools provides the best performance results. Agent Manager running Interactive task Tasks How tasks are executed 58 Domino for iSeries Sizing and Performance Tuning Because there is no way to directly see how Domino is performing on the iSeries server, the best way to approach performance is to collect and analyze data from both the Domino and OS/400 environments. By using the Show Transactions command from the Domino console, you can see the actual transactions that are taking place on the Domino server. You can then look at the Domino Log Statistics for that transaction to see the time it took to complete the transaction. You can then analyze the iSeries performance data collected on CPU utilization and memory during the same time frame by using standard iSeries performance reporting techniques. Although there is no direct comparison, by evaluating the performance factors of both environments, with a little practice, you can develop a better understanding of the relationship between the Domino and OS/400 environments. In Domino, interactive and background tasks run at the same priority. The following tasks can help you tune Domino and the iSeries server: 1. Put Domino in its own system pool. This makes it is much easier to estimate the workload and, therefore, make a prediction on the affect of increased workload. 2. Make a base performance evaluation after the Domino server is started. As the Domino server goes through ramp up and clients begin to access the server, threads are spawned that cause increased CPU utilization for a short period of time. Information collected during this time frame can be misleading as you evaluate the performance data. 3. Pay attention to the notes.ini file. The correct balance of the iSeries performance parameters and the Domino performance parameters are essential to overall system performance. 4.4 Application development performance When you consider the affects of application responsiveness in a Domino environment, there are several factors that can impact performance:
The number and frequency of users accessing the application
Access to a backend relational database by the application
Client type (simple browser, Lotus Notes Client)
Client and server operations
Domino server architecture
Server deployment topology Although all of the factors mentioned above are important, the factor that has the most impact is the actual design of the applications. By building a complex application, you increase the processing and calculations required by the server, which can compromise performance. Developers must balance the need for clean, simple functionality with the need for performance. There is no single means of maximizing the performance of a Domino application. The desire for application performance does not always leave room for design elegance or maintainability. When designing an application, always start at the high level, where you can make performance considerations from the architectural perspective. The following list contains some application design factors that affect performance. You should carefully consider these points when developing an application with performance in mind.
Requirement for agents execution
Actions within forms
Field validation
Using formulas in views
Dynamic data versus static data Chapter 4. Basic concepts of performance analysis 59
Indexing requirements – Frequency of indexing – Full text indexing Further information on application development performance for Domino can be found in Performance Considerations for Domino Applications, SG24-5602. 4.5 Lotus Domino overall performance on the iSeries server Today, more customers are buying iSeries advanced servers to use solely as a Domino platform. However, customers frequently want to maximize resources on their current iSeries server. In this environment, you may have a mix of applications with performance criteria that vary over time. This can cause some conflict in performance requirements. From a system resource utilization viewpoint, it is important to keep these different processing requirements in mind. The Domino installation must be architected from a perspective that keeps the total iSeries server environment in mind. The iSeries hardware and software resources are limited and must be distributed proportionately according to overall organizational objectives. Placing Lotus Domino on an iSeries server that is already servicing business applications has a significant impact on the performance objectives of that system and requires additional consideration in the future performance management of that iSeries server. Figure 4-1 shows an example of a performance tuning flowchart. Figure 4-1 Objectives, measurement, and analysis for good performance AS/400 Performance Objectives Prior to Domino Installation Performance Measurement Prior to Domino Installation Performance Evaluation Prior to Domino Installation Domino Installation AS/400 Performance Objectives For Domino Installation Performance Analysis Domino Performance Tuning Performance Measurement After Domino Installation Performance Evaluation After Domino Installation iSeries iSeries 60 Domino for iSeries Sizing and Performance Tuning A good performance analysis should be completed on the target system before installing Lotus Domino. By properly tuning the iSeries server, you eliminate the possibility of mistaking any pre-existing performance concerns with Domino installation performance allowances. Once the Domino installation is complete, performance measurement and analysis should continue on a frequent basis for an extended period of time. The conclusion from this analysis may change over time. However, it provides the necessary foundation for making proper performance adjustments. 4.6 The Queuing Multiplier (QM) curve Domino on the iSeries, as with any application, is affected by many of the same performance factors that are common to the iSeries operating system. It is important to give some considerations to these common performance factors to make sure that you have a firm understanding of the iSeries performance requirements and the affect that they have on the Domino server. System performance may decrease primarily because of the workload. However, it can also be affected by improper system tuning. On the iSeries server, wait times, including waiting for CPU and disk I/O, are associated with queuing for the server. The higher the server utilization is, the greater the wait or queuing time is. In general, the time it takes for a request to be serviced depends on three primary parameters:
The number of waiters in the line ahead of a new request
The number of servers responding to requests
The service time to complete a request once given to the server, which is a function of the speed of the server and the amount of work to do The Queuing Multiplier is an important factor when projecting the impact of adding work or additional hardware on current system performance. As the utilization of the a server increases, queuing accounts for a part of the longer work (for request) completion. The Queuing Multiplier is a measure of the affect of queuing. Table 4-2 shows a simple example of describing the affects of queuing in practice. Let us assume that a transaction completes in one second when CPU utilization is 10%. If the CPU utilization is 50%, it takes two seconds to complete the transaction. If the CPU utilization is 90%, it takes 10 seconds for the transaction to complete. Table 4-2 How queuing factors affect response time Using a simple example, assume that the CPU is 67% utilized. The mathematical equation says the QM for a single CPU is three. A QM of 3 means, on the average, that there are a total of three requests in the queue (you and two requests for work ahead of you). CPU utilization Queuing Multiplier Response time 10 1.11 1 second 50 2 2 seconds 70 3.3 3.3 seconds 90 10 10 seconds Chapter 4. Basic concepts of performance analysis 61 The Queuing Multiplier values used in the formulas for disk and CPU service time are shown graphically in Figure 4-2. This shows the utilization at various rates and the significance of the knee in the curve for each type of N-way processor. Remember, after the knee, the service times become less stable and may increase dramatically for small utilization changes. Figure 4-2 Queuing Multiplier versus CPU utilization 4.7 Dividing the CPU utilization This section discusses CPU usage for jobs run on the iSeries server. The amount of CPU cycles used are divided among all the jobs in the system. In this example, we assume:
Batch jobs use 30% of the CPU.
Interactive jobs use 20% of the CPU.
Domino jobs use 50% of the CPU. This is demonstrated in Figure 4-3. Note: You may expect performance problems as the Queuing Multiplier increases above 4. This is a conservative guideline since multi-processor systems can provide acceptable performance with higher Queuing Multiplier values. The CPU queuing multiplier is shown on the Job Summary Report of Performance Tools. 40 50 60 70 80 90 95 0 5 10 15 20 1-way CPU 2-way CPU 4-way CPU 8-way CPU 12-way CPU QM=1/(1-(U**n)) CPU UTILIZATION QUEUING MULTIPLIER 62 Domino for iSeries Sizing and Performance Tuning Figure 4-3 Dividing the CPU with all jobs in the system To introduce more jobs, you need a larger CPU. Therefore, it is necessary to choose the correct iSeries configuration for the number of users and the jobs that the users will perform. Figure 4-4 shows how 50% of the CPU used by Domino can be shared between tasks:
The server consumes 45% of the CPU.
The router task consumes 25% of the CPU.
The update task consumes 30% of the CPU. Figure 4-4 Dividing the CPU time used between the Domino tasks 4.8 A brief discussion of threads This section provides a brief description of:
Thread concepts
Threads on the iSeries server
Domino server threads You can find additional information by searching for threads on the iSeries Information Center site at: http://publib.boulder.ibm.com/html/as400/v5r1/ic2924/index.htm We discuss threads because any Domino subsystem is actually a multi-threaded client/server application. The term thread is shorthand for “thread of control”. A thread is an independent unit of execution within a program. Using threads is a programming technique to run various parts of an application concurrently. For example, one thread may be writing a file while another thread is receiving input from a communication line. A process is the container for the memory and resources of the program. Process resources are accessible to all threads. Each piece of work on the system is performed in a job and each job has a unique name within the system. All jobs, with the exception of system jobs, run within subsystems. A job enters the subsystem from any of the work entries, such as, a job queue entry, workstation entry, communications entry, autostart job entry, or prestart job entry. BATCH JOBS 30.0% INTERACTIVE JOBS 20.0% DOMINOxx 50.0% BATCH JOBS INTERACTIVE JOBS DOMINOxx Router Task 25.0% Server Task 45.0% Update Task 30.0% Router Task Server Task Update Task Chapter 4. Basic concepts of performance analysis 63 A job is a collection of one or more threads. Each job has at least one thread, which is identified as the initial thread. The initial thread is created when the job starts. The job may also have additional threads, identified as secondary threads, depending on the applications that are used by a job. A thread inherits the run attributes from a job initiating the thread. They all start at the same run priority and use a time slice of the same length. The thread is an independent unit of dispatchable work. Each thread has its own run environment, such as a call stack. However, a thread shares many of the resources that are assigned to the job. The identifier for the thread is unique in the job to which the thread belongs. Controlling work in the system is performed mainly at the job level. Most commands and application programming interfaces (APIs) operate against the entire job. Resources are owned at the thread level or the job level. The job serves as an owner for resources that are shared with all threads within the same job. Similarly, some attributes that determine how work is processed are defined at the thread level and some are defined at the job level. Individual attributes and interfaces need to be reviewed to determine the scope of the attribute. Information about how work is processed is described throughout this redbook. When the information about the work is referred to as a job, the information applies to all threads as a single unit within the job. When the information about the work applies to an individual thread within a job, the information refers to only the thread. An example of a topic that refers to threads rather than jobs is the information about maximum activity levels. Because each thread is an independent unit of work, the activity level of a storage pool applies to threads rather than jobs. However, the maximum active counts associated with a subsystem and the subsystem work entries apply to jobs. Therefore, a thread is used in information about storage pool activity levels. A job is used in information about subsystem maximum active counts. 4.8.1 Threads on the iSeries server On the iSeries server, a job represents a process. Each process has at least one thread (a task) in which the program runs its code. The first thread in a process is referred to as the initial thread. Some processes are capable of supporting additional threads, which are called secondary threads. Since V4R2, OS/400 supports a kernel thread model. Kernel threads are separate tasks associated with a process. The kernel thread model uses a pre-emptive scheduling policy in which the operating system decides which thread is eligible to share the processor. In a kernel thread model, there is a one-to-one mapping between program threads and process threads. The cost in system resources for activating a new thread is far less than for a job. The job activation and termination is not associated with a thread. Asynchronous I/O completion ports (IOCP) You may have already heard of this support in another term, thread pools. Asynchronous I/O completion ports allows Domino to use fewer threads to support users under the SERVER job. This means that instead of having one thread per client user, a small group or pool of threads handles the work for all users. This means less CPU utilization for the Domino server and a reduction in the amount of memory that is required to handle the client users. 64 Domino for iSeries Sizing and Performance Tuning We highly recommend that you take advantage of this support if possible because the performance improvements can be substantial:
10 to 30% improvement in CPU utilization
20 to 24% reduction in memory required You can tune the number of threads in the pool by specifying the following line in your notes.ini file: SERVER_POOL_TASKS=n By default, the number of threads is 40 per port. In our testing, 40 threads were able to handle 3000 clients with high efficiency. If your server handles fewer clients concurrently, you can improve performance by decreasing the number of threads in the pool proportionately. This support is available in QMU 5.0.1.02 and later releases of Domino for iSeries. It requires OS/400 level V3R3 or higher. 4.8.2 Other Domino task that use threads Other Domino tasks include such jobs as Router, Replica Agent Manager, and so on. Some jobs are single threaded, and some are multi-threaded based on organizational requirements. 4.9 A brief discussion on run attributes A job entering the iSeries server receives its run-time attributes from the class description. This provides the ability for certain Domino tasks or the entire Domino server to use different classes and, therefore, allows more flexibility in adjusting run-time attributes of jobs. Some of the run attributes or parameters that are important to work management are:
Machine run priority (RUNPTY)
Purge (PURGE)
Time slice (TIMESLICE)
Default maximum wait time (DFTWAIT)
Maximum processing time (CPUTIME)
Maximum temporary storage (MAXTMPSTG)
Maximum threads Each of these attributes contributes to how and when a thread is processed. 4.9.1 Run priority Run priority is a value that ranges from 0 (highest priority) through 99 (lowest priority). It represents the importance of a job when it competes with other jobs for machine resources. This value represents the relative (not absolute) importance of the job. For example, a job with a run priority of 25 is not twice as important as one with a run priority of 50. The value is the highest run priority allowed for any thread within the job. Individual threads within the job may have a lower priority. Changing the run priority for the job affects all threads within the job. Note: Maximum processing unit time and maximum temporary storage limits can help prevent an erroneous program from impairing system performance. However, a sufficient amount of processing time and temporary storage must be given to allow the job to complete. Chapter 4. Basic concepts of performance analysis 65 Run priority can also be seen as a variable that defines the rank of a thread. A thread having a high priority number is processed later than one having a small priority number. For example, the system threads run on a priority level of 0, where batch jobs run on a priority of 50. By default, all Domino-related threads run on a priority level of 20, on the same priority level with interactive threads. The following sequence of events is a high-level view of how the iSeries work management handles threads: 1. A job enters the system and starts at least one thread. 2. A thread wants to utilize CPU. It is placed on a queue. 3. The CPU signals its readiness to process anything. 4. The gatekeeper on the queue selects the thread with the highest run priority out of the queue and provides it with access to the CPU. 5. The thread uses the CPU until it has finished with the CPU or it hits the end of its time slice and is thrown out of the CPU. See 4.9.4, “Time slice parameter and tuning” on page 66, for more details. The iSeries server has classes for batch-like jobs and for interactive jobs. By default, the batch jobs run at a lower run priority than the interactive jobs. This mechanism provides the higher priority threads CPU cycles even if the low priority jobs seem to use up all of the available CPU. Figure 4-5 shows the graphical representation of the CPU utilization between January 2001 and July 2001 for ACME, Inc. Figure 4-5 How different run priorities use the CPU The chart in Figure 4-5 shows a summary of the total CPU Utilization per run priority. Pay attention to the period 03/01. Jobs on run priority 50 used 25 percent of the CPU, which left 70 percent for jobs on run priority level 20. This is interpreted as meaning either:
There were no jobs running on priority level 50.
Jobs on run priority level 20 monopolized the system resources. During the period 05/01, a different situation occurred. Jobs on run priority level 50 used 70 percent of the CPU cycles, which left only 17 percent for the jobs running on priority level 20. ACME, Inc Who Uses the CPU 01/01 02/01 03/01 04/01 05/01 06/01 07/01 0 20 40 60 80 100 120 RUNPTY 20 RUNPTY 50 RUNPTY 0 66 Domino for iSeries Sizing and Performance Tuning In both time periods, the overall CPU utilization was about the same. However, different tasks received the CPU time. High priority jobs suffered when low priority jobs flourished. Likewise the high priority jobs flourished when the low priority jobs sufferered. The important point is to complete the work, not the amount of CPU utilization. 4.9.2 How a change in run priority affects a job When you lower a run priority of a thread, you make the thread stay on a queue longer than before. When you raise the run priority of a thread, you may decrease the time that the thread waits in a queue, provided that there are now fewer jobs with equal or higher priority competing for resources. Therefore, it gains control of the CPU more often and completes the work faster. All threads that have the same run priority level are treated equally. If all threads on the system have the same run priority, the CPU is shared according to the first come, first served basis. 4.9.3 Job priority considerations Normally, the priority for jobs that run in batch environments should be lower than the priority for jobs in interactive environments. The time slice should also be small enough to prevent a looping program from dominating the processor time and activity level in the storage pool. 4.9.4 Time slice parameter and tuning Time slice is a performance tuning parameter. The time slice value is specified in the job class discussed in 4.9, “A brief discussion on run attributes” on page 64. The value represents the amount of processing unit time that each thread in a job is allowed to use while processing a transaction. It does not represent the elapsed time of a transaction. If a thread fails to complete a transaction in the specified time slice, one of the following actions occurs:
If no threads of equal or higher priority are on the ineligible queue, the thread is given another time slice, remains in main storage, and continues the transaction.
If threads of equal or higher priority are on the ineligible queue, the thread currently running is removed from main storage and placed on the ineligible queue. A thread from the ineligible queue is moved into main storage and processing continues.
If the job is interactive and a time slice end pool is specified for the job, the thread is moved to the time slice end pool. Note: Changing the priority does not necessarily slow down or speed up a job. Only on a busy system do high priority jobs gain access to resources earlier and prevent low priority jobs form running as fast as possible. If a high priority job starts looping, other jobs with lower or equal priority will stop working. Note: You can use the QDYNPTYADJ system value to maintain high performance of batch job processing on iSeries servers. For more information about this system value, see 4.10.7, “Work with System Values (WRKSYSVAL) command” on page 79. Chapter 4. Basic concepts of performance analysis 67 When observing job transitions, we usually advise that you complete a transaction during a single time slice. This reduces the number of times the job enters and leaves main storage. Therefore, the active-to-ineligible transitions should be 0. However, long running transactions should not occupy activity levels for the entire transaction. Establish a time slice value that allows 90% of the transactions in your environment to finish in a single time slice. An example of calculating a time slice value is: 3 × the average CPU per transaction 4.9.5 How the system manages job and run priorities The iSeries server attempts to allocate main storage as efficiently as possible. A job may not use the same amount of resources each time it runs. For example, if there are several active jobs on your system, a job spends more time re-establishing the resources needed for running than if a dedicated system environment is used. The system uses the run priorities assigned to different jobs to assist in managing main storage. Therefore, high priority jobs use less system resource than low priority jobs. 4.10 Reviewing iSeries performance using CL commands To review iSeries server performance interactively, you can use the following CL commands for each area:
Processor load Use the following commands to determine if there are too many jobs in the system or if some jobs are using a large percentage of processor time: – Work with Active Jobs (WRKACTJOB) – Work with System Activity (WRKSYSACT), which is a Performance Tools command
Main storage Use the Work with System Status (WRKSYSSTS) command to review faulting and the Wait-to-Ineligible transitions.
Disk Use the Work with Disk Status (WRKDSKSTS) command to determine if there are either too few arms or if the arms are too slow.
Communications Use the following Performance Tools to find slow lines, errors on the line, or too many users for the line: – Performance Tools/400 Advisor – Performance Tools/400 Component Report – Communications Trace
IOPs Use the following Performance Tools to determine if any input/output processors (IOPs) are not balanced or if there are not enough IOPs: – Performance Tools/400 Advisor – Performance Tools/400 Component Report
Applications Use the Performance Tools or the Work with Object Locks (WRKOBJLCK) command to investigate locks and mutual exclusion (mutexes). 68 Domino for iSeries Sizing and Performance Tuning 4.10.1 Work with System Status (WRKSYSSTS) command You should observe and balance the overall (system wide) performance before you focus on a Domino performance problem. Domino performance is only a relatively small part of the overall performance. If the entire system is functioning poorly, there is no reason to focus on Domino performance. The Work with System Status display provides a real-time access to data that is also shown in the System Report Storage Pool Utilization section. Use this display to both observe and change the paging fault rates and job transition rates. Pay special attention to the following indicators (in decreasing order of priority):
Non-database fault rates in the machine pool
Non-database fault rates in all of the other pools
Page rates in all of the pools
Transition rates in all of the pools The following examples may help you to understand the performance impact of faulting:
The response time of an interactive transaction is affected by any faults that occur during that transaction. Each fault adds from 10 to 30 milliseconds to the end-user response time. For example, if the disk response time is 20 milliseconds and the transaction has five faults per transaction, add about 0.1 seconds to the total response time.
Each fault uses a certain amount of the CPU power. As more faults occur, more CPU is consumed for unproductive work. If the faulting rate of your system is close to the poor end of the faulting guideline tables, approximately 10% to 20% of the CPU is used for faulting. Adding main storage to reduce the faulting rate also lowers the CPU utilization. Therefore, more processing power is available to handle more transactions.
With the increasing faulting rate, the amount of disk I/O also increases. If you have only a few arms, these faults can cause the disk utilizations to increase more rapidly than if you have many disk arms. As your disk arm (actuator) utilization increases, the time to process disk I/Os increases and the response times is longer. While using the Work with System Status display (see Figure 4-6) to analyze a Domino performance problem, you should concentrate on two storage pools. Press the F11 key two times to see the pool names. These storage pools are:
*MACHINE pool: This is the pool in which the OS/400 jobs and microcode tasks run. Normally, this pool should have the rate of non-DB faults set as close to zero as possible.
Other pools: This is the pool to which the Domino jobs are routed. The shipped value for this is the *BASE pool. Look at the subsystem descriptions for Domino subsystems to see which storage pool is used by the jobs. Focus on that storage pool. Then, determine the following information:
The faulting rate in the *Machine pool (if the rate is not acceptable, see “The interactive iSeries tuning chart” on page 164)
The faulting rate in the storage pool used for Domino jobs
The percentage of disk used Note: When tuning the system, make sure that the machine pool is adjusted separately from the other pools. Chapter 4. Basic concepts of performance analysis 69 Figure 4-6 Work With System Status display 4.10.2 Information about thread state transitions You can balance your main memory and CPU utilization by allocating the available memory and setting the activity levels in the storage pools. An active thread exists in main storage and processes work requested by the application. A thread in the wait state needs a resource that is not available. An ineligible thread has work to do, but the system is unable to accept more work at that time. When a thread is waiting for a resource, it may wait in main storage or it may be removed from main storage until the resource is available. The terms short wait, short wait extended, and long wait are used to describe a thread waiting for a system resource. When observing job transitions, we advise that you allow a transaction to complete during a single time slice. This reduces the number of times the job enters and leaves main storage. Therefore, the active-to-ineligible transitions should be 0. However, long running transactions should not occupy activity levels for the entire transaction. You should establish a time slice value that allows 90% of the transactions in your environment to finish in a single time slice. An example of a time slice value is: 3 × the average CPU per transaction Short wait A thread in short wait holds an available activity level while waiting for an activity to occur. A thread can remain in short wait for a maximum of two seconds. When using remote lines, avoid actions that cause short waits because they cause the wait time in main storage to be much longer for a thread than if the thread is waiting for resources at a local workstation. Work with System Status ACME01 17/12/2001 06:15:00 % CPU used . . . . . . . : 26.2 System ASP . . . . . . . : 106.0 G Elapsed time . . . . . . : 00:15:05 % system ASP used . . . : 76.8542 Jobs in system . . . . . : 776 Total aux stg . . . . . : 106.0 G % perm addresses . . . . : .007 Current unprotect used . : 7345 M % temp addresses . . . . : .015 Maximum unprotect . . . : 7930 M Sys Pool Reserved Max ----DB----- --Non-DB--- Act- Wait- Act- Pool Size K Size K Act Fault Pages Fault Pages Wait Inel Inel 1 668800 280884 +++++ .0 .0 .0 .0 11.8 .0 .0 2 1741704 20 123 .0 .0 .0 .0 47.2 .0 .0 3 47184 0 9 .0 .0 .0 .0 .0 .0 .0 4 1260104 0 17 .0 .0 .9 1.9 23.6 .0 .0 5 1000800 316 1775 .0 .0 .0 .0 3139 .0 .0 Bottom ===> F21=Select assistance level 70 Domino for iSeries Sizing and Performance Tuning Short wait extended A thread is in short wait extended if it is in short wait for the maximum two seconds. After it is in short wait for two seconds and activity has not occurred, the system cancels the short wait, takes the thread out of the activity level, and puts the thread into a long wait. In the performance reports, this thread state transition is called a short wait extended. Long wait A thread that immediately leaves the activity level is in what is called long wait. A specialized form of long wait, called key/think wait, occurs outside the activity level when a thread completes a work assignment and returns to request more work. This is the amount of time it usually takes the user to decide what data should be entered and to type the data. When the thread receives a new assignment, it attempts to run again. If no activity level space is available, the thread becomes ineligible. Other examples of long waits are:
Record lock conflicts (when two or more threads attempt to lock the same record of a file)
Distributed Data Management data requests
Data queue requests
Tape read Ineligible queue and job priority Sometimes threads gain priority if they are held by locks. Normally, if a thread enters the long wait state by other than a lock conflict, it is placed behind all other threads of equal priority already on the ineligible queue. This is called a first-in, first-out priority queue. However, if a thread becomes ineligible after a short wait extended or a long wait caused by a lock conflict, it is placed in front of threads of equal priority already on the ineligible queue. The most common reasons for this change to normal queue placement are:
The thread enters a long wait as a result of a lock conflict because it was active (referring to objects in main storage) before the conflict occurred. If the wait is short (and many are), you may get the thread back into an activity level before all of the objects in use by the thread are removed from main storage.
When the thread is granted the lock, it leaves the wait state. If other threads on the ineligible queue want to use the same object, they must wait until the object is available again. Therefore, you want to use threads holding locks on objects and make them available for other threads to use. To accomplish this, the thread moves ahead of any potential requesters. 4.10.3 Work with Active Jobs (WRKACTJOB) command The WRKACTJOB command shows system performance by measuring aspects such as CPU usage and response time. The threads column shows you the number of threads for each job. Figure 4-7 shows an example of the Work with Active Jobs display. Note: Locks can be thread scoped or job scoped. Locks that are job scoped are shared by all of the threads that are in that job. Chapter 4. Basic concepts of performance analysis 71 Figure 4-7 Work With Active Jobs display To view the Work with Active Jobs display, type WRKACTJOB on any command line, and press the Enter key. To start (or end) an automatic refresh of information on the display that you are viewing, press the F19 key. The information is updated automatically at time intervals that are specified on the INTERVAL parameter. See CL Reference, SC41-5722, for details about the WRKACTJOB command. While using the WRKACTJOB command, position the cursor on a column and press the F16 key. The items that appear are sorted according to the entries in that column. Use the F11 key to toggle between the different views of the WRKACTJOB display. 4.10.4 Work with System Activity (WRKSYSACT) command The WRKSYSACT command is a part of the Performance Tools for iSeries (5722-PT1) licensed program and actually provides an enhancement to the Work with Active Jobs display (see Figure 4-8). It is the only tool that shows both jobs and internal OS/400 tasks at the same time on the display. By default, the jobs are sequenced by CPU usage. You can also sequence the display by I/O usage. The WRKSYSACT command is a part of Performance Tools/400 licensed program product. Figure 4-8 shows the panel displayed by this command when you use the parameter OUTPUT(*), the default, or OUTPUT(*BOTH). Work with Active Jobs ACME01 CPU %: 97.8 Elapsed time: 00:00:01 Active jobs: 260 Opt Subsystem/Job User Number Type CPU % Threads DOMINO QSYS 514292 SBS .0 1 ADMINP QNOTES 514325 BCI .0 1 AMGR QNOTES 514321 BCI .0 1 AMGR QNOTES 514324 BCI .0 3 CALCONN QNOTES 514331 BCI .0 1 EVENT QNOTES 514334 BCI 1.1 8 QNNINSTS QNOTES 514293 BCH .0 1 REPLICA QNOTES 514307 BCI .0 1 REPORT QNOTES 514335 BCI .0 1 ROUTER QNOTES 514310 BCI 4.2 4 SCHED QNOTES 514328 BCI .0 1 SERVER QNOTES 514300 BCI 15.9 257 SERVER QNOTES 514343 BCI 45.6 256 SERVER QNOTES 514344 BCI 22.9 256 UPDATE QNOTES 514315 BCI .7 1 UPDATE QNOTES 514318 BCI 1.3 1 ===> F21=Display instructions/keys Important: WRKSYSACT shows individual measurements for each time interval (snap shots), where WRKACTJOB calculates average values until you press F10. However, if you use the parameter OUTPUT(*FILE) or OUPUT(*BOTH), a database file will be generated with entries for each measurement interval. See 4.11.2, “Using Work with System Activity (WRKSYSACT) with an output file” on page 89. 72 Domino for iSeries Sizing and Performance Tuning Figure 4-8 Work With System Activity display What to look for The Performance Tools WRKSYSACT command provides a list of active jobs that used CPU in the last few seconds. You can determine the specific time by measuring the time interval between uses of the refresh key, PF10. The list is sorted by the amount of CPU seconds used. A high priority job (identified by a low number) may use too much CPU (>50%) for an extended period of time. This may cause poor response time for the entire system. Note the following points:
The Pty column shows the run priority of the job.
The CPU Util column shows the CPU utilization of the job.
The Total Sync I/O column shows the total amount of synchronous I/O processing that is required by the job. A low amount of synchronous I/O processing is important, because a job must wait for the completion of the synchronous I/O operation before continuing.
The Total Async I/O column shows the total amount of asynchronous I/O processing that is required by the job. The amount of asynchronous I/O processing is not as important as the amount of synchronous I/O processing, because a job can continue processing immediately after requesting an asynchronous I/O operation. In a way, an asynchronous I/O operation is similar to a batch job. After submitting the job, you do not have to worry about its completion.
The DB CPU Util column is new since V4R5 and shows the amount of DB2 processing performed by this job. This is important on a iSeries Dedicated Server for Domino, which has a strict limitation of not allowing more than 15% of the total CPU used for DB2 processing. This value allows you to identify jobs doing too much DB2 processing. See 12.3.1, “DB2 database processing defined” on page 351. Work with System Activity ACME01 11/09/01 15:33:00 Automatic refresh in seconds . . . . . . . . . . . . . . . . . 5 Elapsed time . . . . : 00:00:13 Average CPU util . . : .5 Number of CPUs . . . : 4 Maximum CPU util . . : .7 Overall DB CPU util : .0 Minimum CPU util . . : .4 Type options, press Enter. 1=Monitor job 5=Work with job Total Total DB Job or CPU Sync Async CPU Opt Task User Number Thread Pty Util I/O I/O Util QYPSJSVR QYPSJSVR 164955 00000004 10 .0 0 0 .0 QYPSPFRCOL QSYS 164938 0000000A 1 .0 0 3 .0 QYPSPFRCOL QSYS 164938 00000009 1 .0 4 0 .0 RMMCHDATAC 0 .0 5 0 .0 QYPSPFRCOL QSYS 164938 0000000B 1 .0 0 10 .0 Bottom F3=Exit F10=Update list F11=View 2 F12=Cancel F19=Automatic refresh F24=More keys Chapter 4. Basic concepts of performance analysis 73
Entering 1 on the Opt column allows you to monitor this job only. Entering 5 in the Opt column, allows you to access the Work With Job display. You can monitor up to 20 jobs and tasks at a single time. By pressing the F11 key, you can select from three different displays: the summary, synchronous I/O details, and asynchronous I/O details. The following information about jobs and CPU utilization provides additional tips to improve performance:
If a job or small set of jobs is using a large percentage of the CPU, check the job priority (PTY). If the priority of the job that is using too much CPU is a lower number (higher priority) than the jobs with poor performance, you may want to specify a larger RUNPTY. Use option 5 (Work with job) and option 40 (Change job) to specify a larger RUNPTY for the job or jobs that are using too much CPU.
If the job that is using too much CPU is an interactive job that is running a batch function (such as compile or test case), the user can submit work to batch or change the priority to 50.
If the CPU utilization is high (greater than 80%), but all jobs seem to have an equal amount of CPU usage, the system may have too many active jobs. 4.10.5 Work with Disk Status (WRKDSKSTS) command Figure 4-9 shows an example of the Work with Disk Status display. Figure 4-9 Work With Disk Status command display Elapsed time: 00:00:17 Size % I/O Request Read Write Read Write % Unit Type (M) Used Rqs Size (K) Rqs Rqs (K) (K) Busy 1 6607 4194 80.8 2.3 4.9 .2 2.1 4.0 5.0 0 2 6607 3145 81.0 1.2 6.8 .4 .8 12.5 4.0 2 3 6607 3145 80.8 .8 17.8 .1 .6 4.0 21.3 0 4 6607 4194 80.8 6.7 6.4 .4 6.2 21.0 5.3 0 5 6713 6442 80.8 3.6 5.0 .2 3.3 12.0 4.4 0 6 6607 3145 80.8 6.5 4.2 .2 6.3 6.0 4.2 0 7 6607 3145 80.8 2.2 4.9 .1 2.0 10.6 4.4 0 8 6607 4194 80.8 18.1 4.5 .3 17.7 15.3 4.3 0 9 6713 6442 80.8 6.7 7.6 1.3 5.3 8.3 7.5 2 10 6607 3145 80.8 4.7 6.0 .0 4.6 116.0 4.6 0 11 6607 3145 81.0 1.3 5.0 .1 1.2 18.0 3.8 2 12 6607 3145 81.0 1.2 6.6 .1 1.1 28.0 4.4 0 13 6607 4194 80.8 .7 5.5 .1 .5 5.3 5.6 2 More... Command ===> F3=Exit F5=Refresh F12=Cancel F24=More keys Note: This display may vary up to 20% from actual values. CL Reference, SC41-5722, contains a description of the Work with Disk Status (WRKDSKSTS) command and formatting information. 74 Domino for iSeries Sizing and Performance Tuning Before you observe the disk status, tune your system. When viewing the Work with Disk Status display, observe the percent of busy data. Each unit (arm) should be less than 50% busy. If each unit is between 50% and 70% busy, you may experience variable response times. If each unit is more than 70% busy, you do not have enough actuators (arms) to provide good performance. The actuator is the device within an auxiliary storage device that moves the read and write heads. If you have a well-tuned system with actuators that exceed 50% busy, you should increase the number of disk actuators. It is possible to experience inadequate performance even if only one actuator exceeds the 50% busy guideline. This can be caused by the placement of frequently used data on a single storage device. If this occurs on your system, use the Performance Tools for iSeries licensed program disk report to determine which data is causing heavy usage. You can save, delete, or restore some objects to improve performance. An actuator may exceed the 50% guideline for a short period of time. This condition can be caused by a batch job that is accessing data. If the data is not concentrated on a particular storage device, the high level of use should move from actuator to actuator while the batch job is running. Also, an actuator in an auxiliary storage pool (ASP) may be used heavily. Typically, this is not considered to exceed the guidelines. If you observe this activity, do not change the disk configuration. 4.10.6 Observing network performance While observing the performance, it is important not to forget to verify that the network is not the bottleneck. If you had all the CPU power in the world but a bad network, your users would suffer from poor response times or low transaction throughput. The PING command The PING command sends an echo to an IP host to determine whether the host is accessible. The response time shows how long the round trip message takes. This is the first tool to use to identify connectivity problems. The PING command can be issued on any workstation that supports TCP/IP, for example:
Workstation client command line
OS/400 command line
UNIX client In Figure 4-10, the PING command sends a data packet of 256 bytes five times in a row. The minimum, maximum, and average response times are all 10 milliseconds. Figure 4-10 PING command output example If you experience response times between a hundred and a thousand milliseconds or even time-outs, use the Trace Route (TRACERT) command to obtain additional information. PING RMTSYS(ACME01) Verifying connection to host system ACME01 at address 1.1.1.1. PING reply 1 from 1.2.3.4 took 10 ms. 256 bytes. TTL 64. PING reply 2 from 1.2.3.4 took 10 ms. 256 bytes. TTL 64. PING reply 3 from 1.2.3.4 took 10 ms. 256 bytes. TTL 64. PING reply 4 from 1.2.3.4 took 10 ms. 256 bytes. TTL 64. PING reply 5 from 1.2.3.4 took 10 ms. 256 bytes. TTL 64. Round-trip (in milliseconds) min/avg/max = 10/10/10 Connection verification statistics: 5 of 5 successful (100 %). Chapter 4. Basic concepts of performance analysis 75 The JPING tool JPING is a GUI tool that is along the same lines as NPING but has additional features over NotesCONNECT (NPING). You can download this tool from the Web at: http://www.notes.net/sandbox.nsf?OpenDatabase The NPING tool NPING is a GUI tool to test the various TCP/IP port connections to a particular server. You can download the tool by accessing it from the Web at: http://www.notes.net/sandbox.nsf?OpenDatabase Trace Route (TRACERT) command The Trace Route (TRACERT) command is similar to the PING command except that it reports the round trip time by intermediate hosts. It provides information about how each connection between the source system and the target system is performing. An example of the TRACERT output is shown in Figure 4-11. Figure 4-11 TRACERT command output example Work with TCP/IP Network Status (NETSTAT) command Use the Work with TCP/IP Network Status (WRKTCPSTS) command, also known as Network Status (NETSTAT), to obtain information about the status of TCP/IP network routes, interfaces, TCP connections, and UDP ports on your local system. You can also use NETSTAT to end TCP/IP connections and to start or end TCP/IP interfaces. If Internet Protocol (IP) over SNA (IPS) is enabled, NETSTAT displays information about the IP over SNA interfaces, routes, and connections. You can also use NETSTAT to end IP over SNA connections and to start or end IP over SNA interfaces. To use this command, the TCP/IP protocol stack or IP over SNA must be active. If neither is active, NETSTAT sends an escape message. Use the display shown in Figure 4-12 to view the status of your TCP/IP and IP over SNA interfaces, routes, and connections. C:\>tracert acme01 Tracing route to acme01.acmeland.acme.com [1.1.1.1] over a maximum of 30 hops: 1 <10 ms <10 ms <10 ms acme1342.acmeland.acme.com [1.2.3.4] 2 20 ms 10 ms <10 ms acmebb11.acmeland.acme.com [1.4.5.11] 3 <10 ms 10 ms 51 ms acme01.acmeland.acme.com [1.1.1.1] Trace complete. C:\> Note: The three times reported for each line show that there were three attempts for each connection. 76 Domino for iSeries Sizing and Performance Tuning Figure 4-12 Work With Network Status selection display To select an option, type the option number and press Enter. To run a command, type the command and press Enter. For assistance in typing the command, type the command and press F4. For assistance in selecting commands, press F4 without typing anything. Option 1: Work with TCP/IP Interface Status Select option 1 to view the Work with TCP/IP Interface Status display (Figure 4-13). Figure 4-13 Work with TCP/IP Interface Status display Use this display to:
View detailed information about a particular interface
View a list of the routes associated with a particular interface Work with TCP/IP Network Status System: ACME01 Select one of the following: 1. Work with TCP/IP interface status 2. Display TCP/IP route information 3. Work with TCP/IP connection status Selection or command ===> F3=Exit F4=Prompt F9=Retrieve F12=Cancel Work with TCP/IP Interface Status Type options, press Enter. 5=Display details 8=Display associated routes 9=Start 10=End 12=Work with configuration status 14=Display multicast groups Internet Network Line Interface Opt Address Address Description Status x.y.z9.ab2 a.b.ze.192 ACMELINE Active x.y.z9.ab5 a.b.ze.192 ACMELINE Active x.y.z9.ab5 a.b.ze.192 ACMELINE Active x.y.z9.ab6 a.b.ze.192 ACMELINE Active x.y.z9.ab7 a.b.ze.192 ACMELINE Active x.y.z9.ab8 a.b.ze.192 ACMELINE Active x.y.z9.ab3 a.b.ze.192 *VIRTUALIP Inactive x.y.z33.13 a.b.c33.0 TRNLINE Active x0.1.1.1 x0.a.y.0 ACMELINE1 Active x0.1.1.4 x0.a.y.0 ACMELINE1 Active x0.1.1.5 x0.a.y.0 ACMELINE1 Active x0.1.1.6 x0.a.y.0 ACMELINE1 Active F3=Exit F4=Prompt F5=Refresh F11=Display line information F12= F13=Sort by column F24=More keys Chapter 4. Basic concepts of performance analysis 77
Start or end a TCP/IP interface
Work with configuration status
View multicast host groups associated with your interface Option 2: Display TCP/IP Route Information Select option 2 to view the routes defined for the system (Figure 4-14). The IP address of the route destination is the ultimate destination reached by using this route. When used in combination with the subnet mask and the type of service values, the route destination identifies a route to a network or system. This route destination IP address can be the address of any of these items:
Directly attached network or subnetwork
Host system
Remote subnetwork
Remote network
Default route Figure 4-14 Display TCP/IP Route Information display The Subnet Mask column is the actual value of the subnet mask in dotted-decimal notation. The Next Hop column is the Internet address of the first system on the path from your system to the route destination. The value that appears is either *DIRECT or a dotted-decimal Internet address. *DIRECT is the next hop value of a route that is automatically created. When an interface is added to this system, a route to the network that the interface attaches to is also created. These routes to interfaces have a next hop value of *DIRECT. For all other routes, the next hop value is the dotted-decimal Internet address (for example, 9.12.4.6) of the Internet Protocol (IP) router that forwards data to the next system in the path to the route destination. Display TCP/IP Route Information Type options, press Enter. 5=Display details Route Subnet Next Route Opt Destination Mask Hop Available a.b.ccc.d0 255.255.255.0 *DIRECT *YES a.b.cc.dd3 *HOST *DIRECT *NO a.b.cc.dd2 255.255.255.192 *DIRECT *YES a.b.cc.dd2 255.255.255.192 *DIRECT *YES a.b.cc.dd2 255.255.255.192 *DIRECT *YES a.b.cc.dd2 255.255.255.192 *DIRECT *YES a.b.cc.dd2 255.255.255.192 *DIRECT *YES a.b.cc.dd2 255.255.255.192 *DIRECT *YES aa.bbb.ccc.128 255.255.255.128 *DIRECT *NO aa.bb.c.0 255.255.255.0 *DIRECT *NO aa.b.c.0 255.255.255.0 *DIRECT *YES aa.b.c.0 255.255.255.0 *DIRECT *YES aa.b.c.0 255.255.255.0 *DIRECT *YES F3=Exit F5=Refresh F6=Print list F9=Command line F11=Display route type F12=Cancel F13=Sort by column F24= 78 Domino for iSeries Sizing and Performance Tuning Because the IP router forwards the data, your system and the route destination do not have to be on the same network. However, the IP router must be on a network or subnetwork to which your system is directly connected. The Route Available column indicates whether this route is available. One of three possible values appears:
*YES: The route is available.
*NO: The route is not available.
*DOD: The route is used for point-to-point (PPP) dial-on-demand, but is currently not available. Option 3: Work with TCP/IP Connection Status Select option 3 to view or end a TCP/IP connection between a local system and a remote system (Figure 4-15). Figure 4-15 Work with TCP/IP Connection Status Display The Local Internet address field shows which connections are displayed in the Work with TCP/IP Connection Status list. The following information appears in this field:
*ALL appears when all connections defined on the local system are displayed. This is the default selection when the connections list is first displayed.
An Internet address appears when a subset of the list is displayed. Only connections with a local Internet address equal to this Internet address or with a local Internet address of “*” are shown in the list. Note: If the status of a route changes from unavailable to available, a period of time elapses before your system recognizes that the route is available. To make your system recognize that the route is available, use the Verify TCP/IP Connection (VFYTCPCNN) command (PING) to verify the connection to the router. If this does not work, notify the system operator of the problem. Work with TCP/IP Connection Status System: ACME01 Local internet address . . . . . . . . . . . : *ALL Type options, press Enter. 4=End 5=Display details Remote Remote Local Opt Address Port Port Idle Time State * * ftp-con > 019:01:37 Listen * * telnet 001:48:05 Listen * * smtp 042:36:54 Listen * * domain 042:37:04 Listen * * domain 042:37:04 *UDP * * tftp 042:36:55 *UDP * * snmp 042:36:57 *UDP * * drda 042:37:05 Listen * * as-svrmap 001:00:53 Listen * * exec 042:36:59 Listen * * lpd 042:36:59 Listen More... F5=Refresh F11=Display byte counts F13=Sort by column F14=Display port numbers F22=Display entire field F24=More keys Chapter 4. Basic concepts of performance analysis 79 The contents of this field can be changed by using F15 to subset by local address. The Remote Address column shows the Internet address of the remote host. An asterisk (*) is shown if either your iSeries server is waiting for a connection to open or if the list entry is for a UDP socket. The Remote Port column shows either the remote host port number or well-known port name. An asterisk (*) is shown if either your iSeries server is waiting for a connection to open or if the list entry is for a UDP socket. The Local Port column shows your local system port number or well-known port name. The Idle Time column shows the approximate length of time since the last activity on this connection. The time is shown in the format hhh:mm:ss (hours, minutes, seconds). 4.10.7 Work with System Values (WRKSYSVAL) command System values contain specifications that you can use to control or change the overall operation of your system. You can display one system value at a time. A list of the system values (shipped with the system) and their initial contents are available in OS/400 Work Management for Version 5 Release 1, SC41-5306. Use the WRKSYSVAL command to verify the setting the system values described in the following sections. QTOTJOB system value The QTOTJOB value controls the total number of jobs for which the auxiliary storage is allocated during IPL. The correct setting of this system value can be obtained by entering the WRKSYSSTS command. Pay attention to the value displayed in the Jobs in system field. The number of jobs in the system should never be greater than the value specified for the QTOTJOB system value. Add 15% to the number in the Jobs in system field. Before you set this to the QTOTJOB system value, consider the following words of caution:
Clear the output queues regularly. OS/400 reserves auxiliary storage for a job as long as there is at least one spooled output file for that job even though the job is inactive. The more files there are in the output queues, the more jobs you see on the Work with System Status display.
If you have a high number of spooled files on the system while using the WRKSYSSTS command and you add 15% more to set the QTOTJOB value, you may significantly increase the time it takes to IPL the system. Performance is also affected when using any system functions that search through the system-wide work control block table (WCBT). These functions include the Work with Active Jobs (WRKACTJOB), Work with Jobs (WRKJOB), and Start Subsystem (STRSBS) commands.
Consider using the OS/400 Operational Assistant options to clean the obsolete spooled files such as old program dumps from the system. Do this by entering GO CLEANUP on any command line. You can suspect a wrong value of the QTOTJOB system value if the system seems to slow down periodically for no apparent reason. A heavy batch job would be one visible reason. The system delay situation normally lasts a couple minutes after which the normal processing continues until the previously created job structures are used up and a new system delay situation arises. The value shipped with the operating system is 30, which normally is not large enough. Note: A change to this system value takes effect only after the next IPL. 80 Domino for iSeries Sizing and Performance Tuning QADLTOTJ system value The QADLTOTJ system value specifies the additional number of jobs that need storage allocated when the initial number of jobs (the system value QTOTJOB) is reached. Auxiliary storage is allocated whenever the number of jobs in the system exceeds the number for which storage has been allocated. QACTJOB system value The QACTJOB value controls the initial number of active jobs for which auxiliary storage is allocated during an IPL. The correct setting of this value can be determined by entering the WRKACTJOB command. On the right, top corner of the display is the number of active jobs in the system. Determine the highest amount of the active jobs during a busy day, and add 10% to the number. This number is the correct value for the QACTJOB system value. The number of active jobs should not be greater than this value. You can suspect a wrong value for the QACTJOB system value if the system seems to fall asleep periodically for no apparent reason. The slowed performance situation normally lasts a couple of minutes after which the normal processing continues until the amount of previously created job structures are used up and a new delay situation arises. The value shipped with the operating system is 20, which normally is not large enough. QADLACTJ system value The QADLACTJ system value specifies the additional number of active jobs that need storage allocated when the initial number of active jobs (the system value QACTJOB) is reached. Auxiliary storage is allocated whenever the number of active jobs exceeds the storage which has already been allocated. The amount of storage allocated for each job is approximately 110 KB. QMAXACTLVL system value The QMAXACTLVL value determines the maximum activity level of the system. This is the number of all of the jobs that can compete at the same time for main storage and processor resources. If a job cannot be processed because no activity levels are available, the job is held until another job reaches a time slice end or a long wait. See Chapter 14 of OS/400 Work Management for Version 5 Release 1, SC41-5306, for information about job state transitions. Note: A change to this system value takes effect immediately. Note: A change to this system value takes effect only after the next IPL. Note: A change to this system value takes effect immediately. Important: You must keep QACTJOB, QTOTJOB, QADLACTJ, and QADLTOTJ at reasonable values. If you make QACTJOB and QTOTJOB excessively high, the IPL is slower due to excessive storage allocation. If you make QACTJOB and QTOTJOB too small for your environment and you make QADLTOTJ and QADLACTJ excessively large, run-time performance can be impacted. Important: This system value should be set to *NOMAX and the activity levels should be controlled at the pool level. Chapter 4. Basic concepts of performance analysis 81 QMCHPOOL system value The QMCHPOOL system value affects the size of the machine storage pool. The machine storage pool contains the highly-shared microcode and operating system programs. You must be careful when changing the size for this storage pool because system performance may be impaired if the storage pool is too small. This value can be changed by the performance adjustment support when the QPFRADJ system value is set to 1, 2, or 3. You can also change the setting of the QMCHPOOL system value by using the Work with System Status display as described previously. QDYNPTYADJ system value The QDYNPTYADJ system value controls whether the priority of interactive jobs is dynamically adjusted to maintain high performance of batch job processing on iSeries server model hardware. This value is only effective on systems that are rated for both interactive and non-interactive throughput and have dynamic priority scheduling enabled (see the QDYNPTYSCD system value in the following section). QDYNPTYSCD system value The QDYNPTYSCD system value allows you to turn on and turn off the dynamic priority scheduler. The task scheduler uses this value to determine the algorithm for scheduling jobs running on the system. QPRCMLTTSK system value The system value QPRCMLTTSK allows you to turn on and turn off the Processor multi-tasking capability. If it is enabled, more than one set of task data will reside in each CPU. Not all iSeries hardware will support Processor multi-tasking. If the system does not support Processor multi-tasking and it is changed to on, it will be turned back off during the next IPL. Some workloads may experience increased performance due to caching implications. On a partitioned system (LPAR, not Domino partitioned servers), this system value controls the behavior of all partitioned systems and is changed from the primary partition only. Also, certain guest operating systems, such as LINUX, do not support QPRCMLTTSK set to 1. 4.10.8 Display System Log (DSPLOG) command The Display Log (DSPLOG) command shows the system history log (QHST). The history log contains information about the operation of the system and system status. The system history log contains the messages sent to the log, the date and time the message was sent, and the name of the job that sent it. Note: A change to this system value takes effect immediately. Note: A change to this system value takes effect immediately. The value shipped with the operating system is 20000KB. Note: A change to this system value takes effect at the next IPL. Note: A change to this system value takes effect at the next IPL. Note: A change to this system value takes effect at the next IPL. 82 Domino for iSeries Sizing and Performance Tuning 4.10.9 Start System Service Tools (STRSST) command System Service Tools rarely provide performance-related information. However, they may provide you with additional information. For example, a large amount of errors on a communications line tends to cause error-recovery procedures, which may cause performance problems. From the SST screen shown in Figure 4-16, select option 1 to start a Service Tool. Figure 4-16 Selecting to start a service tool The Start a Service Tool display appears as shown in Figure 4-17. Figure 4-17 Start a Service Tool Select option 1 to use a product activity log. You can see such activities as:
Analyze log Select option 1 (analyze log) to display or print a summary of product activity entries. This summary is useful for analyzing intermittent and multiple error conditions.
Display or print by log ID Select option 2 to display or print data from the product activity log by log identifier. The log ID is a unique identifier that ties together all data related to a single error condition.
Change log sizes Select option 3 to verify or change the amount of storage on a disk unit used for product activity log data. System Service Tools (SST) Select one of the following: 1. Start a service tool 2. Work with active service tools 3. Work with disk units 4. Work with diskette data recovery Start a Service Tool Warning: Incorrect use of this service tool can cause damage to data in this system. Contact your service representative for assistance. Select one of the following: 1. Product activity log 2. Trace Licensed Internal Code 3. Work with communications trace 4. Display/Alter/Dump 5. Licensed Internal Code log 6. Main storage dump manager 7. Hardware service manager Chapter 4. Basic concepts of performance analysis 83
Work with removable media lifetime statistics Select option 4 to display, print, or delete the statistical data logged for the lifetime use of a removable media. Lifetime is the total length of time one of these media allows for information to be read from or written to it.
Display or print removable media session statistics Select option 5 to display or print the statistical data logged for a session of a removable media. Session is the length of time one of these media is in position to be read from or written to (read/write heads loaded).
Reference code description Select this option to display or print the description of a reference code. 4.11 Collecting performance data Through V4R5, the Start Performance Monitor STRPFRMON command was available with the OS/400 operating system to allow you to collect performance data. At V5R1, Collection Services replaces Performance Monitor. STRPFRMON is no longer available at V5R1. Collection Services collects performance data for all jobs and tasks running on the system. It allows you to continuously collect performance data with minimal overhead and control what type or types of data are collected. Collection Services uses one or two jobs to collect data. QYPSPFRCOL is the primary job of Collection Services. It collects data into a management collection object (*MGTCOL). The second job, CRTPFRDTA, is only active if you have chosen the create the database files while the collection is active. Both of these jobs will run under the QSYSWRK subsystem. For more information on Collection Services, visit the Web site at: http://publib.boulder.ibm.com/html/as400/v5r1/ic2924/index.htm 4.11.1 Starting Collection Services Collection Services can be started by using any of the following methods:
Operations Navigator
Performance Tools menu options
Management Central APIs Operations Navigator In Operations Navigator, Collection Services can be started in several different ways. The information described here only shows one way in which it can be started. As shown in Figure 4-18, you can right-click the system name, choose Collections Services and then Start Collecting to begin the steps necessary to start Collection Services. 84 Domino for iSeries Sizing and Performance Tuning Figure 4-18 Starting Collection Services For the purposes of collecting sample performance data for Domino performance analysis, Figure 4-19 and Figure 4-20 are set to their default parameters. Figure 4-19 shows the available parameters that control how the data is collection. From this window, you can specify or change the interval at which to collect the data, when to cycle the collection, and how long to retain the data among other options. Chapter 4. Basic concepts of performance analysis 85 Figure 4-19 General tab from starting Collection Services Figure 4-20 describes the data that is going to be collected by Collection Services. It is recommended that you leave this parameter set. Changing the parameter to customer allows you to collect different varieties of data which is fine. But eliminating certain points of collection may provide you with insufficient data to do performance analysis. By clicking OK, Collection Services is submitted to QSYSWRK and data is collected. You also have the option to schedule a collection from these screens by choosing the Schedule button. To end the collection, choose Stop Collecting from the menu shown in Figure 4-18. 86 Domino for iSeries Sizing and Performance Tuning Figure 4-20 Data to Collect tab from starting Collection Services Performance Tools menu options By entering the GO PERFORM command on an iSeries command line, you see the IBM Performance Tools for iSeries display (Figure 4-21). The Performance Tools menu is only available if Performance Tools for AS/400 Licensed Program Product (5769-PT1) is installed on the system. Chapter 4. Basic concepts of performance analysis 87 Figure 4-21 Performance Tools main menu To start Collection Services, select option 2 from the menu. Figure 4-22 Collection Services status display Figure 4-22 provides the current status of Collection Services. Here you see: PERFORM IBM Performance Tools for iSeries System: ACME01 Select one of the following: 1. Select type of status 2. Collect performance data 3. Print performance report 4. Capacity planning/modeling 5. Performance utilities 6. Configure and manage tools 7. Display performance data 8. System activity 9. Performance graphics 10. Advisor 70. Related commands Selection or command ===> F3=Exit F4=Prompt F9=Retrieve F12=Cancel F13=Information Assistant F16=System main menu (C) COPYRIGHT IBM CORP. 1981, 2001. Collect Performance Data ASSLH 10/15/01 10:47:25 Collection Services status: Status . . . . . . . . . . . . : Started Collection object . . . . . . . : Q288000001 Library . . . . . . . . . . . : MYLIB Started . . . . . . . . . . . . : 10/15/01 00:00:01 Default collection interval . . : 00:00:15 Retention period . . . . . . . : 30 days 00 hours Cycle time . . . . . . . . . . : 00:00:00 Cycle interval . . . . . . . . : 24 Collection profile . . . . . . : *STANDARDP Select one of the following: 1. Start collecting data 2. Stop collecting data Selection or command ===> F3=Exit F4=Prompt F5=Refresh F9=Retrieve F12=Cancel 88 Domino for iSeries Sizing and Performance Tuning
Status of the collection (started or ended)
Name of the management collection object
Library that contains the management collection object
Interval in which the data is being collected
How long the management collection will reside on the system
Time the collection started
Collection cycle interval
Type of data being collected To continue starting a collection, choose option 1. The Start Collecting Data panel (Figure 4-23) provides the list of parameter necessary to collect performance data. To keep the data centralized, change the library parameter to a library of your choice. All other parameters can be left alone. Figure 4-23 Parameter menu for starting Collection Services When you press Enter at this screen, a new collection is started. To end the collection, you go into the menu options as shown in the previous steps but choose option 2 (Stop collecting data). You can see this option in Figure 4-22. Management Central APIs If neither Operations Navigator nor Performance Tools are available and PM/400 (Performance Management) is not actively collecting data, there are Management Central APIs that provide the functionality to start and end Collections Services. Starting Collection Services The Start Collector (QYPSSTRC QypsStartCollector) API starts the collection. If a collection is not currently active, the collection will use the defaults to start collecting data. If there is a collection active it will use the settings from the currently active collection and start a new collection using the attributes from the current collection if no new ones have been specified. Start Collecting Data Type choices, press Enter. Library . . . . . . . . . . . . . . MYLIB Name Collection interval (minutes) . . . 15 0.25, 0.5, 1, 5, 15, 30, 60 Retention period: Days . . . . . . . . . . . . . . . 30 *PERM, 0-30 Hours . . . . . . . . . . . . . . 0 0-23 Cycling: Time to synchronize cycle . . . . 00:00:00 HH:MM:SS Frequency to cycle collections . . 24 1-24 Create database files . . . . . . . *YES *YES, *NO Collection profile . . . . . . . . . *STANDARDP *MINIMUM, *STANDARD, *STANDARDP, *ENHCPCPLN F3=Exit F12=Cancel Chapter 4. Basic concepts of performance analysis 89 Use the following command to start Collection Services: CALL PGM(QYPSSTRC) PARM('*PFR ' '*STANDARDP' X'00000000') Ending Collection Services The End Collector (QYPSENDC QypsEndCollector) API ends an active collection. Use the following command to end Collection Services: CALL PGM(QYPSENDC) PARM('*PFR ' X'00000000') Changing System Collector Attributes Change System Collector Attributes (QYPSCSCA, QypsChgSysCollectorAttributes) changes system collector attributes. This API provides the default values for Collection Services. To learn more about these APIs and other Management Central APIs, go to the iSeries Information Center and search for Management Central APIs. The Information Center is located on the Web at: http://publib.boulder.ibm.com/pubs/html/as400/v5r1/ic2924/index.htm 4.11.2 Using Work with System Activity (WRKSYSACT) with an output file The WRKSYSACT command allows you to interactively work with the jobs and tasks currently running in the system. At a first glance, the information shown by this command looks very similar to the Work with Active Jobs (WRKACTJOB) panel. However, the WRKSYSACT has several advantages over WRKACTJOB:
It uses less system resources.
Besides user jobs, it also shows system tasks.
The resource utilization shown is not cumulative and shows as snapshot of each measurement interval. When used with Automatic Refresh (F19) function, it shows live information for the very second, you are watching it. However, WRKSYSACT is only available, when the OS/400 Performance Tools (5722-PT1) are installed on your system. Besides having the ability to view this data on the display station, the user can also direct the data to be stored in a database file for future use. To direct the data to a database file, specify either of the following commands: OUTPUT(*FILE) OUTPUT(*BOTH) This allows you to produce a report as described in 4.13.4, “Print Activity Report (PRTACTRPT) command” on page 115. 4.12 Management Central performance monitor Through Management Central, you can manage and monitor your Domino server performance in an easy and understandable way. Note: There must be six additional spaces after *PFR and before the ending quote on the first parameter. Note: There must be six additional spaces after *PFR and before the ending quote on the first parameter. 90 Domino for iSeries Sizing and Performance Tuning System and Job monitors gather and present real-time performance data for your Domino servers. You can use monitors to see your server’s performance as it happens at a single system or across multiple systems and groups of systems. In contrast, you can use Collection Services to collect performance data for later analysis. Collection Services allows you to analyze multiple sets of true performance data for a longer period of your system performance history. Using the Graph History window, you can see a graphical view of the metrics that you collected for an extended period of time. You can use the Graph History function as long as you collect data with Collection Services; you do not need to have a system monitor running. For information about Collection Services, see 4.11, “Collecting performance data” on page 83. This section does not cover all the functions in Management Central performance monitors. For more detailed and complete information, we recommend that you read Managing OS/400 with Operations Navigator V5R1, SG24-6226. 4.12.1 Monitoring real-time system performance The system monitor graphs present real-time system performance data in a graphical interface that you can directly manipulate to gather different or more detailed data. Monitors allow you to collect performance data simultaneously for a wide variety of system metrics, for any system or system group, and for any length of time. To monitor real-time system performance, you start with creating a system monitor. Creating a new system monitor You can define a new job monitor by using one of the following methods:
Expand Management Central. Select Monitors. Then right-click System and select New Monitor.
Right-click a system under My Connections. Then select Monitors->System.
Expand Endpoint Systems under Management Central and select a system. Then select Monitors->System.
Expand System Groups under Management Central and select a system group. Then select Monitors->System. Each of these methods activates the New Monitor window shown in Figure 4-24. Chapter 4. Basic concepts of performance analysis 91 Figure 4-24 System monitor General On the General page, you enter the name of the monitor and a brief description of the monitor. Metrics The Metrics page for New Monitor or Monitor Properties allows you to select the metrics that you want to monitor. You can view and change information about the collection interval, the maximum graphing value, and the display time for each metric. You can define one or two thresholds for each metric. The thresholds are independent of each other and provide a way to monitor for different conditions in one monitor. For example, you may monitor for a less severe condition and send a command to page the system operator. Or, you may monitor for a more severe condition and send a command page to start ending certain jobs. A threshold consists of a trigger value and, optionally, a reset value. You can specify an OS/400 command to be run when the threshold is triggered or reset. Before V4R5, you could choose from the following CPU Utilization metrics:
CPU Utilization (Average): This metric shows you the percentage of available processing unit time that is being consumed by all jobs on your iSeries server. This metric includes all work done on your system, both interactive and non-interactive.
CPU Utilization (Interactive Jobs): Formerly known as CPU Utilization (Interactive), this metric shows you the percentage of available processing unit time that is being consumed on the system for all jobs of the type I (interactive). It also helps you manage your interactive users' work compared to total CPU utilization capacity. The resulting data is relative to achieving good interactive response time and the amount of CPU left for non-interactive jobs. 92 Domino for iSeries Sizing and Performance Tuning Type I jobs include: – Twinaxial data link control (TDLC) – 5250 remote workstation – 3270 remote workstation – SNA pass-through – 5250 telnet
CPU Utilization Basic (Average): This metric shows the percentage of available processing unit time that is being consumed by all jobs on the system. Unlike the CPU Utilization (Average) metric mentioned above, this metric does not track detailed data. In addition to those metrics available before V4R5, the set of CPU Utilization metrics now includes three new metrics. These new CPU Utilization metrics include:
CPU Utilization (Interactive Feature): This metric is designed to help you monitor and manage your iSeries server's interactive use. It determines whether a particular job is doing interactive work and measures the system's overall interactive workload. This new metric was designed especially with the new iSeries Interactive Feature in mind, but can be used with other iSeries models where an interactive capability is of interest, such as iSeries servers. It complements the existing Management Central metrics such as CPU Utilization (Interactive Jobs) and CPU Utilization (Average). The new metric shows you when your system is approaching its interactive limits: – Signed-on 5250 workstation jobs – Autostart jobs, prestart jobs, or jobs submitted to a batch job queue that run I/O operations on a 5250 workstation
CPU Utilization (Database Capability): This metric is intended to help you monitor your iSeries server's database use. Using this metric, you can see how much of your system CPU is consumed by database activities and which jobs contribute the most to this use. In addition, you can find detailed data for each job, including the number of milliseconds of CPU used by that job in database processing during the particular sample interval being graphed.
CPU Utilization (Secondary Workloads): This metric is designed for use on iSeries dedicated servers. It measures how much CPU is being used on the system for work other than the primary workload for which the system is designed, which can include database activity. For example, this metric can be used on the iSeries Dedicated Server for Domino to see how much non-Domino work is being done on the system. The amount of CPU used by secondary workloads is currently reported only on Dedicated Servers for Domino systems running V4R5 or later and is a portion of the total CPU utilization capacity. On other systems and servers, the value is ignored and appears as 0%. This metric does not track detailed data. All three new metrics have default threshold values set, that are good starting values.
Interactive Response Time (Average): This includes the average response time for interactive (5250) jobs on the system. Second level information shows the jobs having the highest average response time.
Interactive Response Time (Maximum): This includes the highest response time for interactive (5250) jobs on the system while the monitor is active. Second level information shows the jobs having the highest response time. Note: You have to enable these metrics to enable them for use. Chapter 4. Basic concepts of performance analysis 93
Transaction Rate (Average): This includes the average number of transactions per second completed by all jobs active on the system. Second level information shows the jobs having the highest rate.
Transaction Rate (Interactive): This includes the average number of transactions per second completed by 5250 jobs active on the system. Second level information shows the jobs having the highest rate.
Batch Logical Database I/O: The average number of logical database input/output (I/O) operations currently performed by all non-5250 (batch) jobs on the system. Second level information shows the jobs performing the highest I/Os.
Disk Arm Utilization (Average): The average percentage of disk arm busy doing I/O operations for all disks on the system. Second level information shows information for each disk arm.
Disk Arm Utilization (Maximum): The maximum percentage of disk arm busy doing I/O operations for all disks on the system. Second level information shows information for each disk arm.
Disk Storage (Average): The average percentage of disk arm storage that is full on your system during the time you collect the data. Second level information shows information for each disk arm.
Disk Storage (Maximum): The highest percentage of disk arm storage that is full on your system during the time you collect the data. Second level information shows information for each disk arm.
Disk IOP Utilization (Average): The average percent busy the disk input/output processors (IOPs) are on your system during the time you collect the data. Second level information shows information for each IOP.
Disk IOP Utilization (Maximum): The maximum percent busy the disk input/output processors (IOPs) are on your system during the time you collect the data. Second level information shows information for each IOP.
Communication IOP Utilization (Average): The average percent busy the communication (LAN, WAN, ...) IOPs are on your system during the time you collect the data. Second level information shows information for each IOP.
Communication IOP Utilization (Maximum): The maximum percent busy the communication (LAN, WAN, ...) IOPs are on your system during the time you collect the data. Second level information shows information for each IOP.
Machine Pool Faults (Average): The average number of faults per second occurring in the machine pool of the system during the time you collect the data. Only Licensed Internal Code runs in the Machine pool.
User Pool Faults (Average): The average number of faults per second occurring in all of the user pools on the system during the time you collect the data. Second level information shows information for each pool.
User Pool Faults (Maximum): The maximum number of faults per second occurring in all of the user pools on the system during the time you collect the data. Second level information shows information for each pool.
Communication Line Utilization (Average): The average percentage of line utilization for all non-LAN lines active during the time you collect the data. Line utilization is an approximation of the actual amount of data transmitted compared to the theoretical maximum line speed configured on the line description object. Second level information shows information for each non-LAN line. A non-LAN line is supports binary synchronous, asynchronous, IDLC, X.25, LAPD, or SDLC protocols. 94 Domino for iSeries Sizing and Performance Tuning
Communication Line Utilization (Maximum): The maximum percentage of line utilization for all non-LAN lines active during the time you collect the data. Second level information shows information for each non-LAN line.
LAN Utilization (Average): The average percentage of line utilization for all LAN (Token-Ring and Ethernet) lines active during the time you collect the data. Line utilization is an approximation of the actual amount of data transmitted compared to the theoretical maximum line speed configured on the line description object. Second level information shows information for each LAN line.
LAN Utilization (Maximum): The maximum percentage of line utilization for all LAN lines active during the time you collect the data. Second level information shows information for each LAN line. Figure 4-25 shows an example of the metrics that can be added for monitoring Domino server performance. For examples of combining metrics to target specific types of performance information, see 4.12.4, “Combining Management Central’s CPU Utilization metrics” on page 104. Figure 4-25 System monitor metrics Actions The Actions page for New Monitor or Monitor Properties allows you to specify the actions to occur when a threshold is triggered and when a threshold is reset.
Log event: Adds an entry to the event log on the central system when the threshold is triggered or reset. The entry includes the date and time the event occurred, the endpoint system being monitored, the metric being collected, and the monitor that logged the event.
Open event log: Displays the event log when a trigger or reset event occurs. This automatically brings up the Event Log window if it is not currently displayed on your PC.
Open monitor: Displays a list of systems that are being monitored for the specified metrics and a list of the values for the specified metrics as they are collected for each Chapter 4. Basic concepts of performance analysis 95 system when a trigger or reset event occurs. This automatically brings up the Monitor window if it is not currently displayed on the PC.
Sound alarm: Sounds an alarm on the PC when the threshold for the monitor is triggered. Figure 4-26 shows an example of System monitor Action page. Figure 4-26 System monitor: Action page Systems and Groups The Systems and Groups page for New Monitor or Monitor Properties shows you a list of endpoint systems or system groups. The list can include systems from which the monitor is currently collecting data. The list can also include systems on which the monitor is not currently running. You can also add or remove endpoint systems or system groups from this page. Click Browse to add systems or groups to your list. Click Remove to remove selected systems or groups from the Selected systems and groups list. Figure 4-27 shows an example of the systems and groups panel. 96 Domino for iSeries Sizing and Performance Tuning Figure 4-27 System monitor: System and Groups Modifying an existing system monitor To modify an existing system monitor, follow these steps: 1. Expand Management Central and Monitors. 2. Double-click System. 3. Select the monitor you want to modify and then select Properties. The Monitor Profile can be changed easily through the Properties option of the Monitor, even while the Monitor is active. Figure 4-28 shows an example of the window where you modify a system monitor. Figure 4-28 Modifying a system monitor Chapter 4. Basic concepts of performance analysis 97 Starting the Real-time monitor To start the Real-time monitor, double-click the selected monitor, or select the monitor and double-click the green arrow icon (start). Once you start a monitor, you are free to perform other tasks on your server, in Operations Navigator, or on your PC. In fact, you could turn your PC off! It continues to monitor your systems and perform any threshold commands or actions you specified. Your monitor will run until you decide to stop it. If the monitor window is closed, the monitor is not stopped. If you want to end the monitor, go to the Operations Navigator session, and end the monitor. When a Real-time Monitor is started, new threads are associated with the monitor in the Collection Services job QYPSPFRCOL and Management Central job QYPSSRV, and in QYPSJSVR jobs in QSYSWRK subsystem. For more information about the Collection Services job, see 4.11.1, “Starting Collection Services” on page 83. Real-time performance data The premier feature of monitoring is the graphical view. It shows information based on the metrics selection you make, such as average CPU utilization and average interactive response time throughout the network. Once the first-level information, known as a metric, is graphed, the details about the specific sample or data point can be viewed using the drill-down capabilities. There can be anywhere from 1 to 3 levels of additional data for some of the data points. This drill-down capability gives administrators performance information from across the network. Figure 4-29 shows an example of the real-time monitoring graphical view. Time zone considerations: If your systems that are controlled by Management Central are located in different time zones, you must:
Ensure that your system values controlling system date (QDATE) and time (QTIME) are set correctly.
Be sure that the system value for Coordinated Universal Time Offset (QUTCOFFSET), which reflects the time difference from GMT (Greenwich Mean Time), is set correctly.
Consider the impact of changes resulting from transition to and from daylight saving time. 98 Domino for iSeries Sizing and Performance Tuning Figure 4-29 Real-time monitor graphical view The jobs listed in the monitor can be managed and worked on by integrating the Work Management function. For example, you can monitor for the jobs that take up the most CPU, then simply right-click a job, and select Hold or End and the job will be held or ended immediately. Figure 4-30 shows an example of how you can hold a job from the real-time monitor view. Chapter 4. Basic concepts of performance analysis 99 Figure 4-30 Holding a job Tip: You have more than one server on the system and want to know which server owns the job. In this case, simply right-click a job and select Job-> Properties-> Tab Active 1, where you can see which subsystem the job is running under as shown in Figure 4-31. 100 Domino for iSeries Sizing and Performance Tuning Figure 4-31 Job properties for the server Monitors also allow you to set and log events based on thresholds. A threshold is defined by the creator of the monitor to be a point at which the operator or an automation routine needs to be involved. For example, upon reaching a threshold, the administrator can define an OS/400 message to be sent to a particular user or a corrective program to be run. When the creator of the monitor uses these thresholds, they can also select to log events. These include two types of events (triggered or reset), and include information such as when, where, and what caused the event. Figure 4-32 shows an example of the Threshold 1 setting for CPU Utilization (Average). Figure 4-33 shows the monitor graphical view in the background with the Trigger mark for the CPU Utilization (Average) and the Event Log with the Trigger and its properties. Chapter 4. Basic concepts of performance analysis 101 Figure 4-32 Threshold 1 setting for CPU Utilization (Average) Figure 4-33 Event triggered by Threshold 1 for CPU Utilization (Average) 102 Domino for iSeries Sizing and Performance Tuning 4.12.2 Viewing graph history of monitor data In V5R1, you can display a graphical history of performance metrics shown in the system monitors. You can view collected data for the metrics over a long period of time whenever you want, instead of only viewing real-time data. The amount of data that is available to be graphed is determined by the settings that you selected in the Collection Services properties, specifically the Collection retention period. For more information about Collection Services, see 4.11.1, “Starting Collection Services” on page 83. If you want to use the Graph History function to see data that was collected days ago, weeks ago, or months ago, you need to have PM/400 activated. PM/400 is presented in 2.5.2, “Performance Management/400” on page 12. Without PM/400 enabled, the graph data field supports one to seven days. With PM/400 enabled, you can define how long your management collection objects remain on the system. For more information about PM/400, select File and View Trend Analysis on the Graph History window. This takes you to the PM/400 Web site. Graph History performance data The Graph History shows a view of the metrics that you collected for a long period of time for any of the available metrics as compared to the System Monitor window, which shows you real-time data for the last hour. The Graph History window allows you to display one graph at a time. However, you can display more than one Graph History window to make comparisons between systems. For examples of combining metrics to target specific types of performance information, see 4.12.4, “Combining Management Central’s CPU Utilization metrics” on page 104. You can view the graph history by following these steps: 1. In Operations Navigator Management Central, you can view the graph history. To do this, right-click either a system monitor or a Collection Services object and select Graph History. Then choose the metric and period you want to view. 2. Click Refresh to see the graphical view. 3. To open a new window, select File->New Graph History Window. Figure 4-34 shows an example of the three windows with the Graph History view. Chapter 4. Basic concepts of performance analysis 103 Figure 4-34 Graph History view Collection points on the graph line are shown by three different graphics that correspond to the three levels of data that are available:
A square collection point means the data includes both the detailed information and properties information.
A triangular collection point represents summarized data that contains detailed information.
A circular collection point represents data that contains no detailed information or properties information. 4.12.3 Exporting Graph History to PC files for analysis and printing The monitor does not support printing. However, with the new capabilities to view historical data, it can be useful to obtain a printed version of the graph. The Graph History allows you to export all of the data displayed for the selected metrics, the systems and groups, and the date and time range specified, into a file onto your PC. These PC files provide a history of your data and allow you to work with the data in a spreadsheet program or other application. The Graph History data can be exported to the following types of PC files:
Comma separated variables (.csv)
Lotus 1-2-3 compatible (.csv)
ASCII tab delimited text (.txt)
Microsoft Excel 97 (.xls)
Web page (.html) 104 Domino for iSeries Sizing and Performance Tuning The export function only exports what is shown in the graph. It would not export all of the data for time periods outside of what is displayed. 4.12.4 Combining Management Central’s CPU Utilization metrics Use Management Central's CPU Utilization metrics (or any other performance metrics) alone or in concert to target specific types of performance information. The information you want to gather from your systems may be more complex than a single metric can provide. Running metrics in combination allows you to analyze system data from multiple perspectives. Uses for CPU Utilization (Interactive Feature) One of the best ways to interpret this metric is to use it in combination with others. For example, you might use:
CPU Utilization (Interactive Feature) with CPU Utilization (Average): This combination gives you data about total system CPU use as well as interactive feature CPU use. This can be helpful in interpreting overall system activity.
CPU Utilization (Interactive Feature) with CPU Utilization (Interactive Jobs): Jobs of type I are a subset of all jobs included in those analyzed by the CPU Utilization (Interactive Feature) metric. As a result, this combination provides an accurate indication of interactive activity taking place on your system. This also helps you relate your system interactive capability to total CPU capability both at the system and job levels. Uses for CPU Utilization (Database Capability) CPU Utilization (Database Capability) can help you identify and resolve the impact of complex queries that are performed while other performance-critical applications are active. You may get the most value out of this specialized metric when you use it in combination with others to understand system database activity relative to other aspects of system activity. For example, you might use:
CPU Utilization (Database Capability) with CPU Utilization (Average): This combination gives you an accurate perspective of database activity relative to overall CPU activity on your system.
CPU Utilization (Database Capability) in addition to Disk Arm Utilization (Average) or Disk Storage (Average): This mix of performance metrics provides valuable information in an environment where the affects of database activity are important relative to disk utilization.
CPU Utilization (Database Capability) together with Communication IOP Utilization (Average): You may choose to use this combination of performance metrics when analyzing a server application (such as Web serving or ODBC) for time spent in communications versus data storage and retrieval. Tip: You could manage the retention of output from collection services, using the retention period as permanent. The Graphing/Export capability can then be used to transfer data into a Lotus or Excel spreadsheet and the data can be manipulated to build your own Trend Analyses over multiple collection periods. Chapter 4. Basic concepts of performance analysis 105 Uses for CPU Utilization (Secondary Workloads) Much like the CPU Utilization Basic (Average) metric, CPU Utilization (Secondary Workloads) does not provide job-level detail. You can use CPU Utilization (Secondary Workloads) with other metrics, however, to obtain the additional job-level data if you so require. For example, you can use:
CPU Utilization (Secondary Workloads) with CPU Utilization (Database Capability): This combination of metrics helps you monitor those Domino applications running ODBC or @DB commands to access DB2 Universal Database for iSeries data.
CPU Utilization (Secondary Workloads) with CPU Utilization (Average), CPU Utilization (Interactive Jobs), and with other similar metrics: These metrics, when used together, provide a summary of the secondary work taking place on your iSeries server. CPU Utilization (Secondary Workloads) does not provide job-level detail. However, you can use this metric with other metrics to obtain detailed job-level information about activity on your system. You may discover that different metrics, when used with CPU Utilization (Secondary Workloads), provide related job-level detail that best suits your performance data needs. 4.12.5 Job monitoring The ability to monitor for jobs across systems was added to Management Central in V5R1, which gives you the possibility to monitor your Domino server jobs running on one or more iSeries servers. For example, you can create a job monitor to monitor one or more systems that provide services for your Domino server applications. Or, you can create a job monitor to monitor batch jobs that run in the background. The term job monitor should not be confused with the term Performance Monitor (with OS/400 releases before V5R1), which is now called Collection Services (see 4.11, “Collecting performance data” on page 83). Where Collection Services collects performance data typically over a longer period of time, job monitors alert you when the jobs you are interested in do not behave the way you want. Creating a new job monitor You can define a new job monitor by using in one of the following methods:
Expand Management Central. Select Monitors. Right-click Job and select New Monitor.
Expand Basic Operations. Select Jobs. Scan the list of jobs displayed in the right pane and right-click a job. Click Monitor. You may need to use the Basic Operations Include function to find the job you are looking for.
Expand Work Management. Open the appropriate job grouping (Active Jobs, Server Jobs, Job Queues, or Subsystems) to view a list of jobs in the right pane. Right-click a job and select Monitor.
In Management Central, expand Monitors. Select Job. Right-click an existing monitor and select New Based On, which can save steps in defining a similar monitor from the beginning. Each of these methods activates the new monitor Properties window shown in Figure 4-35. Note: Each V5R1 OS/400 system is shipped with a sample job monitor – Sample Domino Server Monitor. It is provided to assist you in getting started with job monitoring and needs at least a system name added to become operational. 106 Domino for iSeries Sizing and Performance Tuning General The General page includes the name of the monitor, a brief description of the monitor, and the jobs and servers to monitor. On the Jobs to monitor subtab, you can specify the jobs to monitor based on any combination of job name, user, subsystem, and job type criteria. You can use an asterisk (*) in the job name, user name, or subsystem name as a substitute for any number of characters, anywhere within the name. Or you can select All to monitor all job names. You can select a job type from the list, or select All to monitor all job types. Each row in the Selected jobs list shows a combination of criteria that must be met for a job to be monitored. A job is monitored if it meets the criteria in at least one row. Click Add to add the specified criteria to the Selected jobs list. Click Remove to remove a selected row or rows of criteria from the Selected jobs list. Figure 4-35 shows Server as the selected job name for user QNOTES in subsystem Domsvr2 to be monitored. Figure 4-35 Job monitor: Jobs to monitor On the Servers to monitor subtab (see Figure 4-36), you can specify jobs by their server names. Select from the list of available servers on the Servers to monitor page. One of the available servers is the Domino server. If you add that one, you will monitor all Domino server jobs on your system. Note: The selections you make on the Job to monitor and Servers to monitor pages will be combined. For example, if you select a job with the subsystem name for one of two Domino servers and add Domino servers to be monitored, you will see data from both of the servers. Chapter 4. Basic concepts of performance analysis 107 Figure 4-36 Job monitor: Servers to monitor Metrics The Metrics page for New Monitor or Monitor Properties allows you to view and change information about each metric to be monitored. You can select metrics to monitor and define thresholds for metric values. The lower portion of the dialog changes based on the selected metric in the Metrics to monitor list. You can define one or two thresholds for each metric. The thresholds are independent of each other and provide a way to monitor for different conditions in one monitor. For example, you may monitor for a less severe condition and send a command to page the system operator or you may monitor for a more severe condition and send a command page to start ending certain jobs. A threshold consists of a trigger value, and optionally, a reset value. You can specify an OS/400 command to be run when the threshold is triggered or reset. For each job, the following metrics can be monitored:
Job Count: Monitor for a specific number of jobs matching the job selection specified under the General tab.
Job Status: Monitor for jobs in any selected status, such as Completed, Disconnected, Ending, Held while running, or Initial thread held.
Job Log Message: Monitor for messages based on any combination of Message ID, Type, and Minimum severity.
Job numeric values: – CPU utilization: The percentage of available processing unit time used by each job that is being monitored on this system. – Logical I/O rate: The number of logical I/O actions, per second, by each job that is being monitored on this system. 108 Domino for iSeries Sizing and Performance Tuning – Disk I/O rate: The average number of I/O operations, per second, performed by each job that is being monitored on this system. The value in this column is the sum of the asynchronous and synchronous disk I/O operations. – Communications I/O rate: The number of communications I/O (read and write) actions, per second, by each job that is being monitored on this system. – Transaction rate: The number of transactions per second by each job that is being monitored on this system. This is meaningful only for 5250 workstation jobs. – Transaction time: The total transaction time for each job that is being monitored on this system. This is meaningful only for 5250 workstation jobs. – Thread count: The number of active threads in each job that is being monitored on this system. – Page fault rate: The average number of times, per second, that an active program in each job, being monitored on this system, refers to an address that is not in main storage.
Summary numeric values: – CPU utilization: The percentage of available processing unit time used by all jobs monitored on this system. For multiple-processor systems, this is the average percent busy for all processors. – Logical I/O rate: The number of logical I/O actions (for example read, write and update) per second, by all jobs monitored on this system. – Disk I/O rate: The average number of physical I/O operations, per second, performed by all jobs monitored on this system. The value in this column is the sum of the asynchronous and synchronous disk I/O operations. – Communications I/O rate: The number of communications I/O (read and write) actions, per second, by all jobs monitored on this system. – Transaction rate: The number of transactions per second by all jobs monitored on this system. This is meaningful only for 5250 workstation jobs. – Transaction time: The total transaction time for all jobs monitored on this system. This is meaningful only for 5250 workstation jobs. – Thread count: The number of active threads for all jobs monitored on this system. – Page fault rate: The average number of times, per second, that active programs in all jobs monitored on this system refer to an address that is not in main storage. Figure 4-37 shows an example of the job monitor Metrics tab, where CPU Percent Utilization is monitored and will create a trigger if CPU% is > (greater than) 10%. Note: Be careful to monitor the smallest number of metrics that will give you the information you need. Monitoring a large number of metrics may have a negative performance impact on your system. Chapter 4. Basic concepts of performance analysis 109 Figure 4-37 Job monitor: Metrics Collection Interval When you are setting thresholds for the metrics you have selected to monitor, you should consider how often you want the data to be collected or sampled. For example, sampling every 5 minutes or even 15 minutes might be right for your environment. You can use the same or different collection interval values for each metric. This interval works in conjunction with the Duration parameter under Threshold on the Metrics page. That is, you specify the time interval value on the Collection Interval page, such as 5 minutes. For Duration on the Metrics page, you specify two intervals. This means the threshold condition is examined every 5 minutes, but the condition must last for two intervals or 10 minutes. If you want to monitor numeric and status metrics for less than 5 minutes, you must select Use different collection interval for each metric type. Figure 4-38 shows an example of the job monitor Collection Interval tab set to 5 minutes. 110 Domino for iSeries Sizing and Performance Tuning Figure 4-38 Job monitor: Collection Interval Actions When you have specified the threshold values for your job monitor, you can click the Actions tab to select event logging and PC actions to be taken when a threshold is triggered or reset. The actions that you can select are:
Log event: Adds an entry to the event log on the central system when the threshold is triggered or reset. The entry includes the date and time the event occurred, the endpoint system being monitored, the metric being collected, and the monitor that logged the event.
Open event log: Displays the event log when a trigger or reset event occurs. This automatically brings up the Event Log window if it is not currently displayed on PC.
Open monitor: Displays a list of systems that are being monitored for the specified metrics and a list of the values for the specified metrics as they are collected for each system when a trigger or reset event occurs. This automatically brings up the Monitor window if it is not currently displayed on the PC.
Sound alarm: Sounds an alarm on the PC when the threshold for the monitor is triggered.
Run OS/400 command: A check mark indicates whether a command has been defined for a threshold triggered or reset. If you have specified an OS/400 command to run when the threshold for this monitor is triggered or reset, those commands run only during times that actions are applied. This option cannot be changed from the Actions page. If you do not want the command to run, you can remove the command from the Metrics page. In the example shown in Figure 4-39, the Apply thresholds and actions and when to apply values are modified. Chapter 4. Basic concepts of performance analysis 111 Figure 4-39 Job monitor: Action Systems and Groups Use the Systems and Groups page to view and change the endpoint systems and groups on which you want the job monitor to run. When you start the monitor, the default system or systems to start on will be those specified here. If the activity has already been started, changes made on this page become active when you click OK. The activity is stopped on endpoint systems that are no longer in the list. The activity is started on systems that you added to the list. Figure 4-40 shows an example of the job monitor Systems and Groups display. Figure 4-40 Job monitor: Systems and Groups 112 Domino for iSeries Sizing and Performance Tuning Sharing This is very important, depending on whether you want other OS/400 users to view and manage your monitor. The sharing options that are available to the creator of the monitor are:
None: Other users cannot view this monitor.
Read-Only: Other users can view and use this monitor, but cannot make any changes to it. They can, however, create a new monitor based on this one and specify their own share values on that monitor.
Controlled: Other users can stop, start, and view this monitor and use it as a template for a new monitor. Only the owner can change the level of sharing, and others cannot change any of the monitors parameters. Starting a job monitor When your monitor is defined, it appears in Management Central under its specific job monitor type. Right-click the named monitor and select Start. Just as for a system monitor, once you start a job monitor, you are free to do other tasks on your server, in Operations Navigator, or on your PC. In fact, you could even turn off your PC, in which case it would continue to monitor your systems and perform any threshold commands or actions you specified. Your monitor will run until you decide to stop it. Figure 4-41 shows an example of how you can start job monitor. Figure 4-41 Starting a job monitor Viewing job monitor results After you have defined and started your monitor, you are ready to view your monitor results. Double-click the monitor name to open the Job Monitor window. In the Job Monitor window, you can see the overall status of the monitor and the target systems on which the monitor is running. Chapter 4. Basic concepts of performance analysis 113 In the example shown in Figure 4-42, we defined the monitor for the server job in subsystem Domsvr2 and specified a threshold for the server job that averages 10 percent of CPU over a five minute period. Figure 4-42 Job monitoring CPU percent utilization Viewing a job monitor event log The Event Log window displays a list of threshold trigger and reset events for job monitors. In our example in Figure 4-39, we specified under the Action tab that we want to add events to the Event log. As the server job reached CPU 33.69%, an event was added to the Event Log, as shown in Figure 4-43. Figure 4-43 Job Monitor and Event Log Tip: In the Job monitor view, you can select Options and Columns F11 to customize your view. For example, if you have more than one Domino server on the system, include the subsystem name and you can see which Domino server owns the server jobs. 114 Domino for iSeries Sizing and Performance Tuning 4.13 Reviewing data using Performance Tools/400 Performance Tools for iSeries Licensed Program Product (5722-PT1) provides the ability to:
Summarize the performance data into reports
Analyze performance data
Perform system modeling based on performance data Refer to the following publications for a description of these and other capabilities of the performance tools and how to use them:
Performance Tools for iSeries, SC41-5340
BEST/1 Capacity Planning Tool, SC41-5341 4.13.1 Print System Report (PRTSYSRPT) command The PRTSYSRPT command creates and prints a system operation overview report from the performance data collected by Collection Services. The report is written to the printer file QPPTSYSR. The system workload, resource utilization, storage pool utilization, disk utilization, and communications summary are shown in the report. 4.13.2 Print Component Report (PRTCPTRPT) command The PRTCPTRPT command creates a report that provides more detailed information than in the System Report. Detailed information regarding the system performance of each component is reported. This detailed report is produced from the performance data collected by Collection Services and shows the data by job, user, pool, disk, IOP, local workstation, and exception. The report is written to the printer file QPPTCPTR. Jobs can be selectively included in the report or excluded from the report, based on a variety of job details and interval times. 4.13.3 Print Transaction Report (PRTTNSRPT) command The PRTTNSRPT command creates and prints performance reports that show detailed information about the transactions that occurred during the time that the performance data was collected. These reports require that trace data is collected. To collect trace data, use the STRPFRTRC command (Figure 4-44). You should selectively include or exclude jobs from the reports, based on a variety of job details and interval times. Chapter 4. Basic concepts of performance analysis 115 Figure 4-44 STRPFRTRC command parameters Change Job Type (CHGJOBTYP) command Use the CHGJOBTYP command to alter the job type for jobs that appear on the reports produced by using the PRTTNSRPT command. For example, use this command to change the SERVER job type from batch to interactive to get statistics about the number of transactions on the Domino server. 4.13.4 Print Activity Report (PRTACTRPT) command The PRTACTRPT command creates reports based on the data collected by the Work With System Activity (WRKSYSACT) command. Start Performance Trace (STRPFRTRC) Type choices, press Enter. Size . . . . . . . . . . . . . . *CALC 128-998000, *CALC, *MAX Omit trace points . . . . . . . *NONE *NONE, *RSCMGT Additional Parameters Job types . . . . . . . . . . . *DFT *NONE, *ALL, *DFT, *ASJ... + for more values Job trace interval . . . . . . . 0.5 .1 - 9.9 seconds Bottom F3=Exit F4=Prompt F5=Refresh F12=Cancel F13=How to use this display F24=More keys Note: The Transaction Report provides Domino-related information down to the service program level only. At the time this redbook was written, none of the tools provided information at the module level. Note: The number of transactions obtained with this approach is not the same value that you find with using Domino Statistics and Logs. However, it may assist you by providing a number that represents an amount of work. Then, you can use that number for growth planning purposes. 116 Domino for iSeries Sizing and Performance Tuning 4.13.5 Convert Performance Thread Data (CVTPFRTHD) command The CVTPFRTHD command converts performance data records collected by Collection Services. The specified member (MBR parameter) of the QAPMJOBL database file contains records with thread-level performance data. Use the CVTPFRTHD command to convert this data and write the resulting records to a member by the same name (MBR parameter) in the QAPMTJOB file. The output file member contains records with job-level performance data, which is a total of the performance information for all threads running within the job. The input file (QAPMJOBL) must exist in the library specified on the LIB parameter. If the QAPMTJOB file does not exist in the specified library, it is created automatically. A file member by the name specified (MBR parameter) is automatically added to file QAPMTJOB if it does not already exist. 4.14 Performance Explorer (PEX) Performance Explorer is a tool that helps find the causes of performance problems that cannot be identified by using tools that do general performance monitoring. Using the Performance Explorer requires more specialized knowledge. You need to:
Have an idea of where the problem may be on the iSeries server
Set up PEX to collect data in specific areas
Interpret the PEX data For more information on the Performance Explorer, refer to the following resources:
Performance Tools for iSeries, SC41-5340
AS/400 Performance Explorer Tips and Techniques, SG24-4781
iDoctor for iSeries site at: http://www.as400service.ibm.com/i_dir/idoctor.nsf You can find another example on using PEX in 12.5, “Observing Domino processing with Performance Explorer” on page 365. Performance Explorer is probably the last tool that you use to analyze a performance problem. Using PEX requires some special skills. It is not the tool with which to start your analysis. The following list shows the process (or the cycle) for using the PEX tool: 1. Add the PEX Definition (ADDPEXDFN). 2. Start Performance Explorer (STRPEX). 3. End Performance Explorer (ENDPEX). 4. Print the PEX Report (PRTPEXRPT). 4.14.1 Add PEX Definition (ADDPEXDFN) command The ADDPEXDFN command allows you to add a new Performance Explorer definition to the system. Each definition is stored as a member in the QAPEXDFN file in library QUSRSYS. A Performance Explorer definition identifies the performance data that is collected during a Performance Explorer session. A session can be started using the Start Performance Explorer (STRPEX) command. When starting a new session, a Performance Explorer definition name is required. 4.14.2 Start PEX Session (STRPEX) command The STRPEX command allows you to start a new Performance Explorer session. Or, you can resume a previously suspended Performance Explorer session. Chapter 4. Basic concepts of performance analysis 117 4.14.3 End PEX Session (ENDPEX) command The ENDPEX command allows you to stop collecting data. The command requires a session name to accompany the request that identifies which instance of the Performance Explorer session to end. You can either end the data collection session or suspend the data collection session. If you choose to end the session, the collected data is put into a single physical file or multiple database files, or it is deleted, based on the value specified for the data option (DTAOPT) parameter. If you choose to suspend the collection of performance data, the session remains active. To resume data collection for a suspended session, specify OPTION(*RESUME) on a subsequent call of the STRPEX command. 4.14.4 Print PEX Report (PRTPEXRPT) command The PRTPEXRPT command allows you to print a formatted listing of the data that is collected by the Performance Explorer and is saved across a set of physical files in a particular library. 4.14.5 Performance Explorer trace points for Domino Starting with Domino 5.06a and OS/400 V4R4, Domino has added trace points to further assist in resolving performance-related problems that cannot be seen using other tools. There are currently six trace points. The trace points are enabled through the notes.ini file and the data is collected using PEX. PEX trace points for Domino The PEX trace points are enabled by adding a line, DEBUG_OS400_PEX with a value equal to 1, 2, 3, or 4 to the notes.ini file. It enables the PEX trace points with varying degrees of collection information, 1 being the least amount of data collected and 4 being the most data. The trace point structure is designed to collect data for the selected level along with levels less than the selected value. For example, setting DEBUG_OS400_PEX=2 collects data for the QIBM_QLNT_OPENDB, QIBM_QLNT_OPENCOLL, QIBM_QLNT_UPDATECOLL, and QIBM_QLNT_HTTP_URL trace points. This structure was implemented to reduce the amount of data that can be logged as a result of using QIBM_QLNT_OPENDBALL. Table 4-3 describes the six trace points and the information they provide when enabled. Table 4-3 PEX trace points for Domino Restriction: You must have read authority for each Performance Explorer database file in the specified library. Trace point name Level Type of information collected QIBM_QLNT_OPENDB 1 Databases opened from the Notes Client. Logs the database name, user name, and the user’s IP address. QIBM_QLNT_OPENCOLL 1 Collection (view) opened from the Notes Client. Logs the database name, view name, user name, and the user’s IP address. QIBM_QLNT_UPDATECOLL 1 Collection (view) refreshed or updated from the Notes Client. Logs the database name, view name, user name, and the user’s IP address. 118 Domino for iSeries Sizing and Performance Tuning Installing the PEX trace points for Domino For releases prior to V5R1, PTFs are required to install the trace points for Domino. Table 4-4 indicates the PTFs for each release of OS/400 that are needed. Table 4-4 PTFs required to install PEX trace points for Domino Enabling the PEX trace points Enabling PEX trace points is accomplished by adding a line to the notes.ini file. To do this, you add DEBUG_OS400_PEX=X, where X equals the desired collection level. The new line can be added anywhere in the file. To modify the notes.ini file, use the Work with Domino Servers (WRKDOMSVR) command and choose option 13 (Edit NOTES.INI) next to the applicable Domino server (Figure 4-45). QIBM_QLNT_HTTP_URL 2 URLs that are received by the HTTP server. This includes HTML pages, images, etc. You will see multiple entries for one page request due to sending all the page elements such as images. It also logs the URL requested. QIBM_QLNT_SVR_AGENT 3 Scheduled Server Agents that are ran through AMGR. Logs the database name, agent name, and the time the agent took to complete. QIBM_QLNT_OPENDBALL 4 Database opens internally in Domino. This will log many more opens than just QIBM_QLNT_OPENDB since it logs opens that Domino does internally. This can be used to see what databases Web clients or agents may be accessing. However, every open will not be logged here since every open does not pass through this trace point. If this is enabled, then your trace will likely consist of mostly these trace points. Logs the database name. OS400 version and release PTF V4R4 SF64028* V4R5 SF64108 V5R1 and later No PTF is required. * These PTFs can be applied immediately and do not have any activation or special instructions. Trace point name Level Type of information collected Chapter 4. Basic concepts of performance analysis 119 Figure 4-45 Work with Domino Servers command Figure 4-46 shows the editing view for the notes.ini file that appears next. Add DEBUG_OS400_PEX to the bottom of the notes.ini file with the desired logging level. Once complete press F3 twice to save and exit this screen. The newly added changes will not take effect until the Domino server has been stopped and started or you’ve performed the Domino command RESTART SERVER. Figure 4-46 Adding DEBUG_OS400_PEX to notes.ini Work with Domino Servers System: ACME01 Type options, press Enter. 1=Start server 2=Change server 5=Display console 6=End server 7=Submit command 8=Work console 9=Work server jobs 11=Change current directory 12=Work object links 13=Edit NOTES.INI Domino Domino Opt Server Subsystem Status *HTTPSETUP QDOMINOHT *ENDED MAILDOMXX MAILDOMXX *ENDED MAILDOM02 MAILDOM02 *ENDED 13 MAILSVR MAILSVR *STARTED APPLSVR APPLSVR *STARTED Bottom Parameters or command ===> F3=Exit F4=Prompt F5=Refresh F9=Retrieve F11=Display path F12=Cancel F17=Top F18=Bottom F22=Display entire field F24=More keys Edit File: /domino/mailsvr/NOTES.INI Record : 80 of 91 by 10 Column : 1 59 by 126 Control : CMD ....+....1....+....2....+....3....+....4....+....5....+....6....+....7....+....8....+... .9....+....0....+....1....+....2....+.. TRANSLOG_AutoFixup=1 TRANSLOG_UseAll=0 TRANSLOG_Style=0 TRANSLOG_Performance=2 TRANSLOG_Status=1 MTEnabled=0 SCHEDULE_VERSION=3 WebAdminSetup=503 DominoConfigLevel=3 TRANSLOG_MaxSize=1028 TRANSLOG_Path=logdir DEBUG_OS400_PEX=2 ************End of Data******************** F2=Save F3=Save/Exit F12=Exit F15=Services F16=Repeat find F17=Repeat change F19=Left F20=Right 120 Domino for iSeries Sizing and Performance Tuning After all necessary collections are made, go back into the notes.ini file and comment out DEBUG_OS400_PEX until it is needed again. To comment out this entry, place a semicolon in front of the line as shown in Figure 4-47. Figure 4-47 Commenting out DEBUG_OS400_PEX Creating a PEX definition for Domino A PEX definition identifies what data needs to be collected. Depending on the Domino and the OS/400 release, one of three commands shown in Table 4-5 need to be performed to add the PEX definition as a member to file QAPEXDFN in library QUSRSYS. Table 4-5 ADDPEXDFN commands for V5R1, V4R5, and V4R4 Executing one of these commands creates a PEX definition called Domino. The definition name does not have to be Domino. It can be called any name. Collecting a PEX trace for Domino in its entirety We discussed the necessary components needed to run a PEX trace on Domino. This section walks you through making the collection: 1. Add DEBUG_OS400_PEX with the desired level of collection to the notes.ini file. See “Enabling the PEX trace points” on page 118 for further information. Edit File: /domino/mailsvr/NOTES.INI Record : 80 of 91 by 10 Column : 1 59 by 126 Control : CMD ....+....1....+....2....+....3....+....4....+....5....+....6....+....7....+....8....+... .9....+....0....+....1....+....2....+.. DominoConfigLevel=3 TRANSLOG_MaxSize=1028 TRANSLOG_Path=logdir ;DEBUG_OS400_PEX=2 ************End of Data******************** F2=Save F3=Save/Exit F12=Exit F15=Services F16=Repeat find F17=Repeat change F19=Left F20=Right Note: Adding and commenting out DEBUG_OS400_PEX requires that you stop and start the Domino server before the changes will take effect. OS/400 VRM Command V5R1 ADDPEXDFN DFN(DOMINO) TYPE(*TRACE) JOB(*ALL) MAXSTG(100000) TRCTYPE(*SLTEVT) SLTEVT(*YES) OSEVT((*DOMTRCDTA)) V4R5 ADDPEXDFN DFN(DOMINO) TYPE(*TRACE) JOB(*ALL) MAXSTG(100000) TRCTYPE(*SLTEVT) SLTEVT(*YES) OSEVT(*MIEV12) V4R4 ADDPEXDFN DFN(DOMINO) TYPE(*TRACE) JOB(*ALL) MAXSTG(100000) TRCTYPE(*SLTEVT) SLTEVT(*YES) OSEVT(*MIEV12) Important: You must end and restart the Domino server for this to take effect. Chapter 4. Basic concepts of performance analysis 121 2. Create a library for the data to be dumped into when you end the trace. To do so, use the Create Library command: CRTLIB LIB(MYLIB) The library’s name can be a name of your choice. 3. Add the desired PEX definition based on the level of OS/400 you are running on. See “Creating a PEX definition for Domino” on page 120 for further information. 4. Start the PEX trace using the following command: STRPEX SSNID(MYTRACE) DFN(DOMINO) The PEX trace is now active. To verify whether any events have been logged, use the ENDPEX command and press Enter. Figure 4-48 shows an example of an ENDPEX display. On the far right-hand side of the screen, you see the Event Count column. This column indicates the number of events that have been collected in the active trace so far. Figure 4-48 ENDPEX display On average, 100 mail users collect 10M worth of raw PEX data in 15 minutes, so keep this in mind when deciding whether to end an active PEX collection. Also note that based on the current PEX definition we defined, we can collect up to 100 MB worth of PEX trace data before the trace goes into a suspended state. If a larger data collection size is needed, you can increase the MAXDATA parameter on the ADDPEXDFN command up to 4 GB at V5R1. Note: For the SSNID (Session ID) parameter of the STRPEX command, you can call the session name any name you choose. Attention: If the iDoctor for iSeries PEX Analyzer tool is installed on your system, you could use the QYPINT/STRIDOCCOL command to automatically start and end the PEX collection for you. Simply put the name of the definition in the PROBLEM parameter, the length you want the trace to run in the DURATION parameter, and change the PRTPEX parameter to “yes”. This command does all the work for you. It starts and ends the PEX trace along with printing the PEX data. If you do not have the iDoctor for iSeries PEX Analyzer installed, but want more information on the tool, go to: http://www.as400service.ibm.com/i_dir/idoctor.nsf Select Performance Explorer Session Type option, press Enter. 1=Select Event Option Session User Type State count MYTRACE MYUSER *TRACE ACTIVE 29311 Bottom F3=Exit F5=Refresh F12=Cancel 122 Domino for iSeries Sizing and Performance Tuning 5. Ending PEX can be a long running process and since a good portion of the machines in production today are iSeries servers with a limited amount of interactive workload capability, we recommend that you end PEX by submitting a batch job as shown here: SBMJOB CMD(ENDPEX SSNID(MYTRACE) DTALIB(MYLIB)) JOB(ENDPEX) Based on this example, you now have a batch job running called ENDPEX. This job must complete before you continue on to step 6 and should not be ended manually at anytime, or the PEX data will be lost. 6. Since the ENDPEX job has finished, you are now ready to print the data using the PRTPEXRPT command. This also can be a long running job. Therefore, it should be submitted as a batch job to print the PEX data as shown here: SBMJOB CMD(PRTPEXRPT MBR(MYTRACE) LIB(MYLIB) TYPE(*TRACE)) JOB(PRTPEXRPT) Once the PRTPEXRPT job finishes, it produces a spooled file named QPVPERPT. The information in the spooled file resembles the data shown in Figure 4-49. The spooled file is 150 columns wide so not all the information is viewable on the screen at one time. Figure 4-49 PRTPEXRPT spooled file 7. Obviously, analyzing the data is the most important part of the entire process. All the necessary information to analyze the problem is contained within the spooled output (QPVPERPT) created from running the PRTPEXRPT command. If further analysis is needed, the PEX data is stored in DB2 files so SQL and Query/400 can be used to extrapolate the data further. Example 4-1 shows an SQL statement that can be used over the PEX DB2 files to further extrapolate the data. The iDoctor for iSeries, a new function provided for OS/400 V5R1, allows you to further examine the results of PEX Analyzer. For more information on iDoctor, see http://www.as400service.ibm.com/i_dir/idoctor.nsf Example 4-1 SQL statement that can be used over the DB2 files select qayperuni.qrntsr+qaypetidx.qtitim microseconds, qaypemiusr.qrecn, qmudta, qaypetaski.qtsjnm, qtsjus, qtsjnb from qaypemiusr, qaypetidx, qayperuni, qaypetaski Display Spooled File File . . . . . : QPVPERPT Page/Line 8/17 Control . . . . . Columns 1 - 130 Find . . . . . . *...+....1....+....2....+....3....+....4....+....5....+....6....+....7....+....8....+....9....+....0....+....1....+....2....+....3 Address Offset Object Name Obj Seg PRE NPgs LIC-Pgm--Offset MI-Pgm-----Offset NAGP PI T ST T ST FIX Unit Sector Span SKP XCH EXID IE ss.mmm P M Task ID Parent-Pgm HLL-No CurrentPgm RC Delta Run Cycles Event PTY WaitSleep Cycles WODSC RS 34.635 00 0 0000000015F1 0 587865 MEV12 API QIBM_QLNT_HTTP_URL 0 00**/homepage.ns 34.637 00 0 0000000015F1 223433 811298 MEV12 API QIBM_QLNT_OPENDB_ALL0 00**/xenon/notes 34.658 00 0 0000000015F1 2696295 3507593 MEV12 API QIBM_QLNT_OPENDB_ALL0 00**names.nsf 34.674 00 0 0000000015F1 1953123 5460716 MEV12 API QIBM_QLNT_OPENDB_ALL0 00**/xenon/notes 34.686 00 0 0000000015F1 1504802 6965518 MEV12 API QIBM_QLNT_OPENDB_ALL0 00**names.nsf 34.687 00 0 0000000015F1 178390 7143908 MEV12 API QIBM_QLNT_OPENDB_ALL0 00**/xenon/notes 34.700 00 0 0000000015F1 1580501 8724409 MEV12 API QIBM_QLNT_OPENDB_ALL0 00**/xenon/notes 34.790 00 0 0000000015F2 0 369295 MEV12 API QIBM_QLNT_HTTP_URL 0 00**/homepage.ns 34.792 00 0 0000000015F2 141695 510990 MEV12 API QIBM_QLNT_OPENDB_ALL0 00**/xenon/notes 34.794 00 0 0000000015F3 0 385495 MEV12 API QIBM_QLNT_HTTP_URL 0 00**/xenon/notes 34.827 00 0 0000000015F4 0 340035 MEV12 API QIBM_QLNT_HTTP_URL 0 00**/homepage.ns 34.828 00 0 0000000015F4 134818 474853 MEV12 API QIBM_QLNT_OPENDB_ALL0 00**/xenon/notes 35.096 00 0 0000000015F2 0 1458298 MEV12 API QIBM_QLNT_OPENDB_ALL0 00**/xenon/notes 35.115 00 0 0000000015F4 0 1392245 MEV12 API QIBM_QLNT_OPENDB_ALL0 00**/xenon/notes 43.110 00 0 0000000015DD 0 233407 MEV12 API QIBM_QLNT_OPENDB_ALL0 00**/xenon/notes More... F3=Exit F12=Cancel F19=Left F20=Right F24=More keys Chapter 4. Basic concepts of performance analysis 123 where (qaypemiusr.qrecn = qaypetidx.qrecn) and (qaypetaski.qtscnt = qaypetidx.qtitct) order by qaypemiusr.qrecn Figure 4-50 shows a view of how the SQL statement in Example 4-1 would appear if it was run against PEX data. Further information containing the job name, job user, and job number is shown to the right of the information shown in Figure 4-50. Figure 4-50 Output from the SQL statement Both analyses show you the job responsible for the event, the API (trace point) that was triggered, and the specific information associated with the event triggered. Whether it is a view, agent, or URL causing performance problems, these techniques will help you find patterns in the data and help isolate the problem. 4.14.6 TPROF PEX trace In the event the system is not on a release of OS/400 or Domino that supports the Domino PEX trace points, Trace Profile (TPROF) PEX trace can be very useful in isolating CPU consumption problems. This trace is also beneficial to be used before enabling the Domino PEX trace points. It can uncover the problem or help pinpoint the level of Domino PEX trace points that need to be collected. TPROF is a PEX trace that helps identify system-wide CPU “hotspots” within application programs, service programs, OS/400 programs, procedures and modules, and SLIC procedures. The PEX data is collected by using the Base PEX event PMCO. The data is formatted as if you had collected *PROFILE PEX data. Running a TPROF PEX trace To run a TPROF PEX trace, you need to perform the following steps: 1. Create a library for the data to be stored using the CRTLIB command: CRTLIB LIB(MYLIB) 2. Add a PEX definition that describes the data to be collected: Display Data Data width . . . . . . : 316 Position to line . . . . . Shift to column . . . . . . ....+....1....+....2....+....3....+....4....+....5....+....6....+....7....+....8....+....9....+...10....+...11....+...12....+...13. Timestamp expression Rec No User data 2001-03-01-12.57.34.634696 1 API QIBM_QLNT_HTTP_URL 0 * 00**/homepage.nsf?Open 2001-03-01-12.57.34.636504 2 API QIBM_QLNT_OPENDB_ALL0 * 00**/xenon/notes/data/homepage.nsf 2001-03-01-12.57.34.658376 3 API QIBM_QLNT_OPENDB_ALL0 * 00**names.nsf 2001-03-01-12.57.34.673912 4 API QIBM_QLNT_OPENDB_ALL0 * 00**/xenon/notes/data/names.nsf 2001-03-01-12.57.34.685920 5 API QIBM_QLNT_OPENDB_ALL0 * 00**names.nsf 2001-03-01-12.57.34.687352 6 API QIBM_QLNT_OPENDB_ALL0 * 00**/xenon/notes/data/names.nsf 2001-03-01-12.57.34.699928 7 API QIBM_QLNT_OPENDB_ALL0 * 00**/xenon/notes/data/homepage.nsf 2001-03-01-12.57.34.790408 8 API QIBM_QLNT_HTTP_URL 0 * 00**/homepage.nsf/homePage.gif?OpenImageResource 2001-03-01-12.57.34.791552 9 API QIBM_QLNT_OPENDB_ALL0 * 00**/xenon/notes/data/homepage.nsf 2001-03-01-12.57.34.793664 10 API QIBM_QLNT_HTTP_URL 0 * 00**/xenon/notes/data/domino/icons/ecblank.gif 2001-03-01-12.57.34.826896 11 API QIBM_QLNT_HTTP_URL 0 * 00**/homepage.nsf/smSpacer.gif?OpenImageResource 2001-03-01-12.57.34.827976 12 API QIBM_QLNT_OPENDB_ALL0 * 00**/xenon/notes/data/homepage.nsf 2001-03-01-12.57.35.095536 13 API QIBM_QLNT_OPENDB_ALL0 * 00**/xenon/notes/data/homepage.nsf 2001-03-01-12.57.35.115440 14 API QIBM_QLNT_OPENDB_ALL0 * 00**/xenon/notes/data/homepage.nsf 2001-03-01-12.57.43.110456 15 API QIBM_QLNT_OPENDB_ALL0 * 00**/xenon/notes/data/names.nsf 2001-03-01-12.57.43.133040 16 API QIBM_QLNT_OPENDB_ALL0 * 00**names.nsf 2001-03-01-12.57.43.145712 17 API QIBM_QLNT_OPENDB_ALL0 * 00**/xenon/notes/data/names.nsf 2001-03-01-12.57.43.167616 18 API QIBM_QLNT_OPENDB_ALL0 * 00**names.nsf More... F3=Exit F12=Cancel F19=Left F20=Right F21=Split F22=Width 80 Cannot show this data at the display station. 124 Domino for iSeries Sizing and Performance Tuning ADDPEXDFN DFN(MYTRACE) TYPE(*TRACE) JOB(*ALL) TASK(*ALL) MAXSTG(100000) INTERVAL(5) TRCTYPE(*SLTEVT) SLTEVT(*YES) BASEVT(*PMCO) 3. Start the PEX trace: STRPEX SSNID(MYTRACE) DFN(MYTRACE01) 4. Duplicate the performance problem. 5. End the PEX trace: SBMJOB CMD(ENDPEX SSNID(MYTRACE01) DTALIB(MYLIB)) JOB(ENDPEX). 6. Once the job submitted in step 5 has ended, print the PEX data using PRTPEXRPT as shown below. This creates a spooled file under the PRTPEXRPT job. SBMJOB CMD(PRTPEXRPT MBR(MYTRACE01) LIB(MYLIB) TYPE(*PROFILE) PROFILEOPT(*SAMPLECOUNT *PROCEDURE)) JOB(PRTPEXRPT) Note: Since most iSeries machines are server machines with limited amounts of interactive capabilities, the Submit Job command is used. Chapter 4. Basic concepts of performance analysis 125 Analyzing the TPROF trace The output produced from step 6 in the previous section produces a report that shows which programs, modules, or procedures the system spent the most time in, based on hit counts. This is very useful in isolating a point in which a job or jobs are spending most of their time. Figure 4-51 provides an example of the TPROF PEX trace output. Figure 4-51 Sample page from a TPROF PEX report In the example shown in Figure 4-51, note the user program entry that is underlined. By reviewing this program and the functions it performs within the application, you can improve the overall performance of the system. Most PEX collections are not always this informative, which may require more in-depth analysis by looking at the patterns within the data to see how it all fits together. Collecting PEX data with the Domino PEX trace points may be needed. Histogram Hit Hit Cum Start Map Stmt Name Cnt % % Addr Flag Numb ------------------------------------------------------------------------------------------------- ** 15683 6.3 6.3 FFFFFFFFB31D04D4 ++ 0 HMFREHS/#hmfrehs * 10323 4.1 10.4 FFFFFFFFB1618FE0 ++ 0 QUMUGA/pSpinWait__11QuMutexGateFv * 9667 3.9 14.3 30FAE17D640197E4 == 0 ACME DATALINK/CheckValidKey * 9205 3.7 18.0 FFFFFFFFB31E3140 ++ 0 HMALCHS/#hmalchs * 7797 3.1 21.1 1E88ABC9373296D0 == 0 LIBNOTES OSSEM/OSLockSem 6241 2.5 23.6 1E88ABC93733BDC0 == 0 LIBNOTES MEMLOCK/LockHandle 6124 2.5 26.0 1E88ABC937329490 == 0 LIBNOTES OSSEM/OSUnlockSem 4771 1.9 27.9 1E88ABC93732BC40 == 0 LIBNOTES OSSEM/OSUnlockSpin 4387 1.8 29.7 1E88ABC937350A00 == 0 LIBNOTES THREAD/OSGetNativeThreadID 3860 1.5 31.2 FFFFFFFFFF800690 ++ 0 CFNSBLA/bla_NORTHSTAR 3532 1.4 32.6 1E88ABC93732BE20 == 0 LIBNOTES OSSEM/StaticHangEnable 3051 1.2 33.9 1E88ABC9373C5790 == 0 LIBNOTES VARRAY/OSAddressInVARRAY 2971 1.2 35.1 FFFFFFFFFF027030 ++ 0 STRNGEAO/memmovwteaofree 2912 1.2 36.2 1E88ABC9375C84B4 == 0 LIBNOTES MD/MDTransform 2611 1.0 37.3 1E88ABC93733EF00 == 0 LIBNOTES MEMORY/LockMemHandle 2499 1.0 38.3 222A3D696B00ABE0 == 0 QP0WPTHR/pthread_getthreadid_np 2410 1.0 39.2 1E88ABC9376DD9A0 == 0 LIBNOTES DBUNK/DbUNKGetName 2386 1.0 40.2 1E88ABC93733C490 == 0 LIBNOTES MEMLOCK/LockMem 2225 0.9 41.1 FFFFFFFF802830C0 ++ 0 PDCEACTV/__ct__14PdcEventActiveFCUlT1CQ _14PdcEventActive6CRTestPC6TDTask 2189 0.9 42.0 00CA6A6DC4006190 == 0 QC2UTIL1 QC2ALLOC/malloc 2074 0.8 42.8 30FAE17D640199F0 == 0 ACME DATALINK/GetValidName 2073 0.8 43.6 1E88ABC937391500 == 0 LIBNOTES BPOOL/OSBBlockAddr 1916 0.8 44.4 FFFFFFFFFF0030A0 ++ 0 CFSCV/scv0 1823 0.7 45.1 1E88ABC93733B640 == 0 LIBNOTES MEMLOCK/OSUnlockObject 1753 0.7 45.8 1E88ABC93733B7C0 == 0 LIBNOTES MEMLOCK/OSLockObject 1748 0.7 46.5 1E88ABC937A13500 == 0 LIBNOTES ODS/ODSToOrFromHost 1714 0.7 47.2 1E88ABC93733C330 == 0 LIBNOTES MEMLOCK/UnlockMem 1650 0.7 47.9 80000000000030A0 ++ 0 HVSCVVEC/hvscvvec 1641 0.7 48.5 FFFFFFFFFE6C2980 ++ 0 MASOUBLA/masounlockmutexbla 1608 0.6 49.2 1E88ABC93733BD50 == 0 LIBNOTES MEMLOCK/UnlockHandle 1573 0.6 49.8 1E88ABC937394040 == 0 LIBNOTES CMEM/Cmovmem 1520 0.6 50.4 1E88ABC937325B80 == 0 LIBNOTES OSSEM/OSLockReadSem 1484 0.6 51.0 FFFFFFFFFE6C26C0 ++ 0 MASOLBLA/masolockmutexbla 126 Domino for iSeries Sizing and Performance Tuning © Copyright IBM Corp. 1999, 2002 127 Chapter 5. Domino logs and statistics Domino includes a set of commands and statistical reports to monitor server performance. The commands provide an interactive method for checking performance statistics. The statistical reports provide a method of collecting the performance statistics on a scheduled interval. The statistical reports are stored in the default database – Statistics and Reporting Database (statrep.nsf). Gathering statistics requires system resources. Keeping what you collect to a minimum improves the performance of your Domino server. 5 128 Domino for iSeries Sizing and Performance Tuning 5.1 The commands Of the numerous Domino console commands, the following five commands are probably the most useful when monitoring performance and investigating a performance problem:
Show tasks
Show stat
Show database
Show platform statistics
Show trans The commands can be run by using any of these methods:
Enter the Work with Domino Servers (WRKDOMSVR) command on any OS/400 command line. Then, select the server, type option 8 (Work console), press Enter and enter the Domino console command show xxxx.
Starting from a command line on the iSeries server, enter the Work With Domino Console (WRKDOMCLS) command. Then, select the server, and enter the Domino console command show xxxx.
Start the Domino Administrator, and choose the Server tab and Console. Enter the command show xxxx.
Run the Submit Domino Command (SBMDOMCMD) command from an OS/400 command line or from a CL program. Then, enter the Domino console command show xxxx and the server name. In this case, you do not see the output of the command immediately. You can either redirect the output to a stream file for further analysis or see the results in the notes log LOG.NSF. To redirect the output of a Domino console command into a text file, append the greater than sign (>) followed by a file name to the command: show xxxx >filename The output file is stored in the Domino server directory within the iSeries Integrated File System (IFS). This is specially useful if you wrote a CL program that periodically writes a certain statistic value into a text file. Domino commands and their parameters can be abbreviated as long as the remaining characters are not ambiguous. For example, the command show tasks can also be entered in the form sh ta. 5.1.1 The show tasks command The show tasks Domino console command displays:
The status of the active server tasks
The Domino server release
The name of the server
The path of the Notes program directory
Optionally the thread number of each task Chapter 5. Domino logs and statistics 129 Figure 5-1 shows an example of the output from running the show tasks console command. Figure 5-1 The show tasks output An undocumented debug parameter provides the iSeries server job and thread number associated with tasks running under the SERVER job. The debug parameter helps map a thread to a user or a system function Figure 5-2 shows an example of the output from running the show task debug console command. Lotus Domino (r) Server (Release 5.0.8 for OS/400) 10/19/2001 09:44:48 AM Server name: applsvr/sg5162b Server directory: /domino/applsvr Partition: APPLSVR Elapsed time: 00:27:18 Transactions/minute: Last minute: 18; Last hour: 6; Peak: 39 Peak # of sessions: 2 at 10/19/2001 09:22:41 AM Transactions: 202 Availability Index: 100 (state: AVAILABLE) Message Tracking: Not Enabled Shared mail: Not Enabled Number of Mailboxes: 1 Pending mail: 0 Dead mail: 0 Waiting Tasks: 0 Transactional Logging: Not Enabled Task Description Database Server Perform console commands Database Server Listen for connect requests on TCPIP Database Server Load Monitor is idle Database Server Database Directory Manager Cache Refresher is idle Database Server Idle task Database Server Idle task Database Server Perform Database Cache maintenance Database Server Idle task Database Server Idle task Database Server Server for Notes Admin/sg5162b on TCPIP Database Server Server for Donald Duck/sg5162b on TCPIP Admin Process Idle Event Monitor Idle Calendar Connector Idle Schedule Manager Idle Admin Process Idle Agent Manager Executive '1': Idle Agent Manager Idle Stats Idle Indexer Idle Router Idle Replicator Idle HTTP Web Server Listening on port(s) 8081 DECS Server Idle 130 Domino for iSeries Sizing and Performance Tuning Figure 5-2 The show tasks debug results In the Task column, there is a server task with a thread identified as 17853:0000: 003E. On the iSeries server, the thread number is eight digits; only the last four digits are shown here. In this example, you can see that the thread is used by Notes Admin/sg5162b, so you know who to talk to if there is a performance problem or if a function is not being used properly. Lotus Domino (r) Server (Release 5.0.8 for OS/400) 11/05/2001 08:08:06 AM Server name: applsvr/sg5162b Server directory: /domino/applsvr Partition: APPLSVR Elapsed time: 00:03:30 Transactions/minute: Last minute: 1; Last hour: 5; Peak: 9 Peak # of sessions: 2 at 11/05/2001 08:06:19 AM Transactions: 19 Availability Index: 100 (state: AVAILABLE) Message Tracking: Not Enabled Shared mail: Not Enabled Number of Mailboxes: 1 Pending mail: 0 Dead mail: 0 Waiting Tasks: 0 Thread Pools: Enabled Transactional Logging: Enabled Task Description 17853:0012: 0003 Database Server Perform console commands 17853:0013: 0004 Database Server Listen for connect requests on TCPIP 17853:0014: 0005 Database Server Load Monitor is idle 17853:0015: 0006 Database Server Database Directory Manager Cache Refresher is idle 17853:0016: 0007 Database Server Thread pool member on TCPIP .. 17853:003D: 002E Database Server Thread pool member on TCPIP 17853:003E: 002F Database Server Thread pool utility task on TCPIP 17853:003F: 0031 Database Server Idle task 17853:0040: 0032 Database Server Idle task 17853:0041: 0033 Database Server Perform Database Cache maintenance 17853:0042: 0034 Database Server Idle task .. 17853:0049: 003B Database Server Idle task 17853:004A: 003C Database Server Recovery Manager Checkpoint Thread 17853:004B: 003D Database Server Recovery Manager DB Flushing Thread 17853:0000: 003E Database Server Server for Notes Admin/sg5162b on TCPIP Event Monitor Idle Calendar Connector Idle Schedule Manager Idle Admin Process Idle Agent Manager Executive '1': Idle Agent Manager Idle Stats Idle Indexer Idle Router Idle Replicator Idle HTTP Web Server Listening on port(s) 8081 DECS Server Idle Chapter 5. Domino logs and statistics 131 To analyze a performance problem, display the call stack for the thread you suspect that has a performance issue. To find out what job 17853:0000 003E is doing, perform the following steps: 1. Use the Work with Domino Servers (WRKDOMSVR) command to display a list of all Domino servers. 2. Select option 9 (Work server jobs) next to the server name. 3. Select option 12 (Work with threads) next to the job name. 4. Select option 10 (Display call stack) next to the thread 0000003E. You see a panel similar to the Display Call Stack example shown in Figure 5-3. Figure 5-3 Call stack for thread 0000003E Display Call Stack: Panel group update to improve readability If you have Domino 5.03 or higher installed, you may improve the analysis of the call stack for Domino server tasks. A new panel group named GWVJOB is provided in library QNOTES. This panel group enables a more readable display of the Call Stack. To replace the panel group QGWVJOB in QSYS with the panel group shipped with the Lotus Domino product, enter the CL commands in the order shown below on any OS/400 command line: CRTLIB GWVLIB TEXT('Panel Group for better Domino Call Stack display') CRTDUPOBJ GWVJOB QNOTES *PNLGRP GWVLIB QGWVJOB CHGSYSLIBL GWVLIB Display Call Stack System: AS06 Job: SERVER User: QNOTES Number: 017853 Thread: 0000003E Program Rqs or Lvl Procedure Library Statement Instruction < tth_parm_t QSYS 0000001092 < eate_part2 QSYS 0000001587 < eadWrapper QNOTES 0000000008 Scheduler QNOTES 0000000032 < tilityTask QNOTES 0000000024 < elayThread QNOTES 0000000001 < nix_usleep QNOTES 0000000006 select QSYS 0000004667 Bottom F3=Exit F10=Update stack F11=Display activation group F12=Cancel F16=Job menu F17=Top F18=Bottom F22=Display entire name Tip: In 7.12, “Excessive translations on iSeries and zSeries” on page 202, you can see another example that explains how to display the call stack and shows a problem with excessive CCSID translations. 132 Domino for iSeries Sizing and Performance Tuning Figure 5-4 shows the Call Stack view with improved readability. Figure 5-4 Call stack view with improved readability 5.1.2 Understanding the Domino statistics The Domino console command show stat shows server statistics that relate to:
The number of transactions
The number of concurrent users
The available amount of disk space
The memory available and used
Mail, replication, and network activity
Other useful information You can use this command without any parameters to show all available statistics, or select a group or a single value by specifying its name. To display only a group of statistics, specify the group name (the characters on the left of the period) followed by an asterisk (*) as a wildcard. You can find a complete list of statistic descriptions in the Domino events4.nsf database, in the Names & Messages, Statistics Names view. For monitoring performance and investigating a performance problem, we recommend that you run the following commands: Display Call Stack System: AS06 Job: SERVER User: QNOTES Number: 017839 Thread: 0000003E ILE ILE Procedure Statement Module program LE_Create_Thread2__FP12crtth_parm_t 0000001092 QLECRTTH QLESPI pthread_create_part2 0000001587 QP0WSPTHR QP0WPINT ThreadWrapper 0000000008 THREAD LIBNOTES Scheduler 0000000035 SCHED SERVER OSDelayThread 0000000001 THREAD LIBNOTES unix_usleep 0000000006 USLEEP LIBNOTES select 0000004667 QP0LLIB1 QP0LLIB1 Bottom F3=Exit F10=Update stack F11=Display instruction F12=Cancel F16=Job menu F17=Top F18=Bottom F22=Display entire name Important: The panel group that is used for this function only works well with Domino jobs. If you attempt to display the call stacks of non-Domino jobs while using this panel group, you will receive incomplete information. To correct this problem, you must sign off and sign back on or run the command: CHGSYSLIBL LIB(GWVLIB) OPTION(*REMOVE) Chapter 5. Domino logs and statistics 133
show stat Server.Trans.PerMinute
show stat Server.Users
show stat Server.Sessions.Dropped
show stat Database.BufferPool*
show stat Domino
show database Domino transactions per minute The show stat Server.Trans.PerMinute statistic provides the number of transactions that occurred on the server during the last minute. This information is useful for monitoring overall server use. Figure 5-5 shows an example of the output from running the show stat Server.Trans.PerMinute console command. Figure 5-5 The show stat Server.Trans.PerMinute statistic Number of active Notes users The show stat console command can also display a group of statistic values by specifying the wildcard character "*". For example, show stat Server.Users* provides the current number of active Notes sessions as well as the peaks and the times when they appeared. It includes all users and all server tasks connecting to the server (for example, the Replicator). It shows information about peak times and is useful for monitoring overall server use. The example in Figure 5-6 shows three users that are currently active on the server. On March 13 at 11:42 AM, the most users (65) were active on the server (since the server was started or the statistics were reset). On March 14 at 02:01 PM, the most users (17) performed any network operation to this server in one minute. And on March 14 at 09:14 AM, the most users (26) performed any network operation to this server within a five minute time frame. Figure 5-6 The show stat Server.Users* statistic The parameters shown in Figure 5-6 are explained here:
Server.Users: This specifies the number of active Notes Sessions. This includes all users and all server tasks connecting to the server (for example, the Replicator). In the example shown in Figure 5-6, three users are currently connected to the server. Note: To reset a statistic that is cumulative, you can use Set Statistics statisticname. Statisticname is a required parameter that names the statistic to be reset. You can't use wildcards (*) with this argument. For example, Set Stat Server.Trans.Total resets the Server.Trans.Total statistic to 0. Server.Trans.PerMinute = 2185 Server.Users = 3 Server.Users.1MinPeak = 17 Server.Users.1MinPeakTime = 03/14 02:01:55 PM Server.Users.5MinPeak = 26 Server.Users.5MinPeakTime = 03/14 09:14:03 AM Server.Users.Peak = 65 Server.Users.PeakTime = 03/13 11:42:02 AM 134 Domino for iSeries Sizing and Performance Tuning
Server.Users.1MinPeak: This specifies the number of users who have, in the last minute, performed a Notes transaction (some network activity caused, for example, when opening a database or document on the server). This value does not specify the number of users or transactions in the last minute. The Domino server checks this data regularly and updates the value only if the number is greater than the previous value.
Server.Users.1MinPeakTime: This value shows the date/time at which the maximum number of users have done a network operation in the past minute. Note that Server.Users.1MinPeak specifies the number of users during this past minute. Users who have not done any transaction so far, but are waiting for a transaction, are also counted in this list.
Server.Users.5MinPeak and Server.Users.5MinPeakTime: This shows the number of users and the time, respectively, at which the server had the most users performing a transaction during the past five minutes.
Server.Users.Peak and Server.Users.PeakTime: This shows the maximum number of users and the time, respectively, for users who had an active connection to the server. Sessions dropped The show stat Server.Sessions.Dropped statistic provides the number of users who gave up trying to connect to the server (usually by restarting or pressing Ctrl-Break). This statistic is useful for monitoring server response time. The Server.Sessions.Dropped statistic indicates poor server performance if the number of dropped sessions is high on a regular basis. Resolving a problem with dropped sessions may require:
Cleaning up and reducing the size of databases
Moving databases and users to a different server
Adding a server
Adding memory
Adding DASD
Upgrading the processor Figure 5-7 shows an example of the output from running the show stat Server.Sessions.Dropped server command. Figure 5-7 The show stat Server.Sessions.Dropped command Domino database buffer pool statistics Monitor how much NSF buffer pool your server uses by running the show stat database command. If BufferPoolPeak is routinely more than 95% of the maximum, you may need to increase the NSF_Buffer_POOL_Size. For more information about the NSF_Buffer_POOL_Size, see 7.14.1, “NSF_Buffer_Pool_Size or Nsf_Buffer_Pool_Size_mb” on page 211. Also monitor Database.BufferPool.PerCentReadsInBuffer. In our tests under heavy workload, we saw a BufferPoolPeak of 100%, while PerCentReadsInBuffer was under 95%. Under a light load, BufferPoolPeak was under 10%, while PerCentReadsInBuffer was over 97%. Server.Sessions.Dropped = 0 Chapter 5. Domino logs and statistics 135 When the maximum size of NSFBufferPool is reached, old entries are taken out of the pool to make room for the new entries. At this point, the Database.BufferPool.PerCentReadsInBuffer statistic will begin to decrease. This is not good because a low number in this pool indicates that there is not enough memory to cache most of the reads. The higher the percentage is, the better. This statistic represents the cumulative percentage of time that the buffer pool was used to service requests. The Domino server show stat database.Database.BufferPool* command presents a screen like the example shown in Figure 5-8. Figure 5-8 The show stat database.Database.BufferPool* command The parameters shown in Figure 5-8 are explained here:
Database.BufferPool.Maximum.Megabytes This is the maximum allowed in the NSF Buffer Pool. It defaults to one-fourth (¼) of available physical RAM. If the notes.ini parameter NSF_BUFFER_POOL_SIZE=nnnnnn is set, this parameter overrides the default.
Database.BufferPool.MM.Reads This is the cumulative number of times the buffer has been read.
Database.BufferPool.MM.Writes This is the cumulative number of times that data has been written to the buffer.
Database.BufferPool.Peak.megabytes This is the largest that the pool has grown to, so far. This value will never decrease until the Notes server is restarted.
Database.BufferPool.PerCentReadsInBuffer This is the cumulative percentage of time that the buffer pool was used to service requests. The higher this number is, the better. HTTP server statistics The show stat domino command monitors HTTP server activities on the Domino server. The Domino show stat Domino.Config.ActiveThreads* console command shows the server document settings (Figure 5-9) for Number active threads and Minimum active threads. Figure 5-9 The show stat Domino.Config.ActiveThreads* command The parameters in Figure 5-9 are explained here:
Domino.Config.ActiveThreads.Max: This is the maximum number of active threads servicing HTTP requests.
Domino.Config.ActiveThreads.Min: This is the minimum number of active threads servicing HTTP requests. Database.Database.BufferPool.Maximum.Megabytes = 300 Database.Database.BufferPool.MM.Reads = 0 Database.Database.BufferPool.MM.Writes = 0 Database.Database.BufferPool.Peak.Megabytes = 3 Database.Database.BufferPool.PerCentReadsInBuffer = 97.28 Domino.Config.ActiveThreads.Max = 40 Domino.Config.ActiveThreads.Min = 20 136 Domino for iSeries Sizing and Performance Tuning The Domino show stat Domino.Threads* console command shows the peak number of active threads.
Domino.Threads.Active.Peak: This indicates the peak number of active threads since Domino was started.
Domino.Threads.Peak.Total: This value represents the total peak number of active threads since Domino was started.
Domino.Threads.Peak.Time: This value indicates the time during the peak of the total thread statistics from above. Domino.Config.ActiveThreads.Max should be greater than Domino.Theads.Active.Peak.
Domino.Config*: If you run the show stat domino.config* command, you will see a list of how HTTP server is configured. This list may be useful if you don’t have access to the server document in the Domino directory. Figure 5-10 shows an example of the output from Domino show stat domino.config* console command. Figure 5-10 The show stat domino.config* command 5.1.3 Show Domino open database statistics (sh DBS) The Show Open Databases (show DBS) command is a tool for monitoring the performance of open databases. This command returns the information shown in Figure 5-11. Domino.Config.ActiveThreads.Max = 40 Domino.Config.ActiveThreads.Min = 20 Domino.Config.AllowDirectoryLinks = 1 Domino.Config.Directory.CGI = /domino/applsvr/domino/cgi-bin Domino.Config.Directory.HTML = /domino/applsvr/domino/html Domino.Config.Directory.Icons = /domino/applsvr/domino/icons Domino.Config.Directory.JavaRoot = /domino/applsvr/domino/java Domino.Config.DNSLookup = 0 Domino.Config.HomeURL = /homepage.nsf?Open Domino.Config.HostName = Domino.Config.Image.Format = GIF Domino.Config.Image.Interlaced = 1 Domino.Config.Log.Access = access Domino.Config.Log.Error = error Domino.Config.Log.Filter = Domino.Config.Log.TimeStamp = 0 Domino.Config.PortNumber = 8081 Domino.Config.PortStatus = 1 Domino.Config.SSL.KeyFile = /domino/applsvr/keyfile.kyr Domino.Config.SSL.PortNumber = 443 Domino.Config.SSL.Status = 0 Domino.Config.Timeout.CGI = 5 Domino.Config.Timeout.IdleThread = 0 Domino.Config.Timeout.Input = 2 Domino.Config.Timeout.Output = 20 Domino.Config.URLpath.CGI = /cgi-bin Domino.Config.URLpath.Icons = /icons Domino.Config.URLpath.JavaRoot = /domjava Domino.Config.URLpath.Servlet = /servlet Domino.Config.View.Lines = 30 Domino.Config.WelcomePage = default.htm Chapter 5. Domino logs and statistics 137 Figure 5-11 The show database command The columns shown in Figure 5-11 are explained here:
Refs: The number of times the database has been opened (the DBHANDLE count for the database).
Mod: Whether the database has been modified, but not yet flushed to disk.
FDs: The number of file descriptors currently being used for the database.
LockWaits: The number of times a user has had to wait for a lock on the database (read or write).
AvgWait: The average wait time in milliseconds for each wait.
#Waiters: The number of waiters currently on the database lock. (This number changes rapidly.)
MaxWaiters: The maximum number of waiters ever on the database lock. 5.1.4 Platform-dependent statistics New with Domino 5.0.3 for iSeries is the integration of platform and Domino statistics. Platform statistics are enabled by adding PLATFORM_STATISTICS_ENABLED=1 to the notes.ini file and restarting your server. This feature is disabled by default. After you enable the platform-dependent statistics in notes.ini, you can only see them on the console through the show stat platform command. The next step is to make sure those statistics are also recorded in the statrep.nsf database so you can analyze them. To collect performance data from Domino, you need to configure a couple of things as explained in the following process: 1. From your Collection Server (usually a hub in a large environment), open the Statistics & Events database (events4.nsf). This database allows you to set up the collection interval of your Domino servers. 2. Click Server Statistic Collection on the left. This displays a Collection Document. Edit this document or create a new one if one does not exist. Figure 5-12 shows the Statistics & Events database views. Database Name Refs Mod FDs LockWaits/AvgWait #Waiters MaxWaiters /domino/applsvr/statrep.nsf 1 Y 1 0 0 0 0 /domino/applsvr/events4.nsf 8 N 1 0 0 0 0 /domino/applsvr/busytime.nsf 1 N 1 0 0 0 0 /domino/applsvr/statmail.nsf 1 N 1 0 0 0 0 /domino/applsvr/mail.box 1 N 1 0 0 0 0 /domino/applsvr/decsadm.nsf 3 N 1 0 0 0 0 /domino/applsvr/names.nsf 55 Y 9 0 0 0 1 /domino/applsvr/log.nsf 1 Y 1 0 0 0 0 Note: To display LockWaits and AvgWait values, you must temporarily add the setting COLLECT_DB_LOCK_WAITS=1 to your server's notes.ini file. This setting consumes server resources, so you should remove it after you view Show DBS statistics. 138 Domino for iSeries Sizing and Performance Tuning Figure 5-12 Statistics & Events database views 3. Once the Server Statistic Collection document is opened for editing, click the Basics tab as shown in Figure 5-13. Be sure to enter the server name of the server that will collect the statistics. Select the option to collect statistics from All servers in this domain or specify the names of the servers from which you want to collect performance data. Once this is complete, click the Options tab. Figure 5-13 Server Statistic Collection document 4. Under the Options tab, click the Log statistics a database check box and specify the database that will gather the statistics as shown in Figure 5-14. We recommend you collect the statistics with the statrep.nsf database, and for performance purposes, collect statistics every 15 minutes with alarms every 60 minutes. Chapter 5. Domino logs and statistics 139 Figure 5-14 Server Statistic Collection document options 5. Edit the notes.ini file of the server collecting the statistics by adding add the line PLATFORM_STATISTICS_ENABLED=1. Figure 5-18 on page 149 shows an example of how to edit the notes.ini file using Operations Navigator. You can also use option 13 in the Work with Domino Servers (WRKDOMSVR) panel to edit the notes.ini file. This ensures the Domino servers gathers platform statistics unique to iSeries, like total CPU utilization, number of page faults per second, number of Wait-to-Ineligible transitions for threads in the memory pool, and CPU utilization per Domino tasks, which is very important for performance analysis. 6. After you save the notes.ini file, replicate your events4.nsf database to all servers in your domain by issuing the command: push events4.nsf Here, server name is the destination server. Make sure to specify a fully qualified server name. Otherwise, you see the message “Access control is set in events4 to not allow replication from events4.nsf.” 7. After all servers receive the updated file, recycle your servers (end and restart). When they restart, they will begin collecting their Domino statistics in the statrep.nsf file of your destination server. Metrics are collected continuously, unless they are disabled or reset, or the server is stopped. The server resets this information each time it is started. OS/400 platform statistics include performance metrics for CPU, memory, disk, and process utilization. The Statistics and Reports database (statrep.nsf) includes the platform view, Show Platform Statistics. However, this does not include iSeries-specific data, except the overall CPU utilization. If you want to see other iSeries related statistics, you need to create an additional view. See 5.3, “Combining performance data from Domino and iSeries” on page 152. The Statistics and Reports database is further described in 5.2.2, “The Statistics and Reporting database (statrep.nsf)” on page 149. System statistics System statistics (Platform.System.TotalUtil.*) indicate the platform system total utilization. It is the current average CPU utilization for the system. The minimum, average, and maximum utilization for each of the system statistics values also available. System statistics collect the same information for each partition, and the same information is reported for each Domino server. The values reported in each server may vary due to timing of the collection. 140 Domino for iSeries Sizing and Performance Tuning Total CPU utilization The total CPU utilization includes the entire workload of the iSeries server, that is all Domino servers and all non-Domino jobs. For the CPU utilization of specific Domino tasks of a single server, see “Process statistics” on page 140. The average CPU percent utilization of all processors on the system is reported by the value of Platform.System.TotalUtil. The smallest TotalUtil sampling for a session is contained in Platform.System.TotalUtil.Min. The average of all TotalUtil samplings for a session can be found in Platform.System.TotalUtil.Avg. This is calculated by the finding the sum of all TotalUtil samplings divided by the number of samplings. The largest TotalUtil sampling for a session is reported with Platform.System.TotalUtil.Max. Memory statistics Memory statistics (Platform.Memory.*) are collected and reported for the storage pool in which the Domino server is running. If more than one server (partition) shares the storage pool, the memory statistics reported include all servers. The values reported in each server may vary due to timing of the collection. The Platform.memory* statistics include:
The amount of memory in kilobytes allocated to the pool that this Domino partition is running in. – Platform.Memory.KBSize
The number of page faults per second in the memory pool this server is using. – Platform.Memory.FaultsPerSec – Platform.Memory.FaultsPerSec.Min – Platform.Memory.FaultsPerSec.Avg – Platform.Memory.FaultsPerSec.Max
The number of pages per second read to or written from the disk with the memory pool this server is using. – Platform.Memory.PagesPerSec – Platform.Memory.PagesPerSec.min – Platform.Memory.PagesPerSec.avg – Platform.Memory.PagesPerSec.max
The number of Wait-to-Ineligible transitions for threads in the memory pool this server is using. Any value greater than zero indicates your max active in the memory pool is too low. – Platform.Memory.WaitToIneligible Process statistics Process statistics (Platform.Process.*) are independent from partition to partition; no values are shared. Some of the server tasks, such as AMGR, REPLICA, SERVER, and UPDATE, can have statistics collected and reported for up to four instances of the process. If an instance of a process is not present, 0 is reported. Any new process takes two collections before it is reported by statistics. This applies for processes started when the server is started or processes started later with the load command. Chapter 5. Domino logs and statistics 141 If a process is removed, the metric for that process goes to 0. If a new process begins, it uses the first unused statistic slot. For example, if there are three AMGR processes, process 2 ends, so you only receive statistics from Platform.Process.AMGR.1.Util and Platform.Process.AMGR.3.Util. If another AMGR starts, it reports its statistics to the first unused statistic slot, which is Platform.Process.AMGR.2.Util. The Platform.process* statistics include:
The CPU utilization of the first ADMINP, AMGR, HTTP, REPLICA, ROUTER, SERVER, and UPDATE process of this partition. This is the percentage of the sample interval that the process used the CPU. – Platform.Process.ADMINP.1.Util – Platform.Process.AMGR.1.Util – Platform.Process.HTTP.1.Util – Platform.Process.REPLICA.1.Util – Platform.Process.ROUTER.1.Util – Platform.Process.SERVER.1.Util – Platform.Process.UPDATE.1.Util
The CPU utilization of the second AMGR, REPLICA, SERVER, and UPDATE process of this partition. This is the percentage of the sample interval that the process used the CPU. – Platform.Process.AMGR.2.Util – Platform.Process.REPLICA.2.Util – Platform.Process.SERVER.2.Util – Platform.Process.UPDATE.2.Util
The CPU utilization of the third AMGR, REPLICA, SERVER, and UPDATE of this partition. This is the percentage of the sample interval that the process used the CPU. – Platform.Process.AMGR.3.Util – Platform.Process.REPLICA.3.Util – Platform.Process.SERVER.3.Util – Platform.Process.UPDATE.3.Util
The CPU utilization of the fourth AMGR, REPLICA, SERVER, and UPDATE process of this partition. This is the percentage of the sample interval that the process used the CPU. – Platform.Process.AMGR.4.Util – Platform.Process.REPLICA.4.Util – Platform.Process.SERVER.4.Util – Platform.Process.UPDATE.4.Util Disk statistics Disk statistics (Platform.LogicalDisk.*) are collected and reported for the whole system. All configure disks in all ASPs are collected. The Percent Used statistic is the percent of all space of all configured disks that are in use. The Percent Busy statistic is an average of the percent busy time for every configured disk. The Platform.LogicalDisk* statistics include:
The average of the percent time that all configured disks in all ASPs are busy. The first “_Total” refers to all physical disks, and the second “_Total” refers to all logical disks. For the iSeries, read it as the total of all disks configured. – Platform.LogicalDisk._Total.1._Total.1.PctTime = .1 – Platform.LogicalDisk._Total.1._Total.1.PctTime.avg = 1.4 – Platform.LogicalDisk._Total.1._Total.1.PctTime.max = 45.7 – Platform.LogicalDisk._Total.1._Total.1.PctTime.min = 0 142 Domino for iSeries Sizing and Performance Tuning
The percentage of all configured disks in all ASPs that is currently allocated for data. – Platform.LogicalDisk._Total.1._Total.1.PctUsed = 15.6 Figure 5-15 shows an example of the output from running the show stat platform console command. Figure 5-15 The show stat platform results The Domino console platform command Platform statistics collection is controlled by the PLATFORM command. The following keywords can be used with the PLATFORM command:
Time: Display or change the current collection interval
Reset: Reset collection of the platform statistics
Wait: Stop collection of platform statistics
Run: Start collection of platform statistics Platform.LogicalDisk._Total.1._Total.1.PctTime = .1 Platform.LogicalDisk._Total.1._Total.1.PctTime.avg = 1.4 Platform.LogicalDisk._Total.1._Total.1.PctTime.max = 45.7 Platform.LogicalDisk._Total.1._Total.1.PctTime.min = 0 Platform.LogicalDisk._Total.1._Total.1.PctUsed = 15.6 Platform.Memory.FaultsPerSec = .3 Platform.Memory.FaultsPerSec.avg = .4 Platform.Memory.FaultsPerSec.max = 8.1 Platform.Memory.FaultsPerSec.min = 0 Platform.Memory.KBSize = 3,600,740 Platform.Memory.PagesPerSec = .3 Platform.Memory.PagesPerSec.avg = 2.1 Platform.Memory.PagesPerSec.max = 63.8 Platform.Memory.PagesPerSec.min = 0 Platform.Memory.WaitToIneligible = 0 Platform.Process.ADMINP.1.Util = 0 Platform.Process.AMGR.1.Util = 0 Platform.Process.AMGR.2.Util = 0 Platform.Process.AMGR.3.Util = 0 Platform.Process.AMGR.4.Util = 0 Platform.Process.HTTP.1.Util = 0 Platform.Process.REPLICA.1.Util = 0 Platform.Process.REPLICA.2.Util = 0 Platform.Process.REPLICA.3.Util = 0 Platform.Process.REPLICA.4.Util = 0 Platform.Process.ROUTER.1.Util = 0 Platform.Process.SERVER.1.Util = .8 Platform.Process.SERVER.2.Util = 0 Platform.Process.SERVER.3.Util = 0 Platform.Process.SERVER.4.Util = 0 Platform.Process.UPDATE.1.Util = 0 Platform.Process.UPDATE.2.Util = 0 Platform.Process.UPDATE.3.Util = 0 Platform.Process.UPDATE.4.Util = 0 Platform.System.TotalUtil = 2.2 Platform.System.TotalUtil.avg = 1.6 Platform.System.TotalUtil.max = 8.5 Platform.System.TotalUtil.min = 0 Chapter 5. Domino logs and statistics 143 For example, to start a new performance data monitoring session with a sampling rate of five minutes, type the following commands at the console: 1. platform time 5 The server collects performance data every five minutes. 2. platform reset Statistic values sampled before this command was issued are not used in calculating the average, maximum, or minimum. 5.1.5 Show Domino Transactions The show transactions, or in its short form sh tran, Domino console command provides a performance statistics summary for each Domino kernel transaction that is run. For each type of transaction, it displays the total number of NRPC transactions, the minimum and maximum duration of the transaction, the total time to perform all transactions, and the average time to perform the transaction. Figure 5-16 shows an example of the output from running the show trans console command. 144 Domino for iSeries Sizing and Performance Tuning Figure 5-16 The show trans output Each column is explained here:
Function: The kernel transactions provided by Domino.
Count: The number of times the function was run.
Min: The minimum CPU time in milliseconds it took to run the function.
Max: The maximum CPU time in milliseconds it took to run the function.
Total: The total CPU time in milliseconds spent running the function.
Average: Indicates the total per count. Lotus Domino (r) Server (Release 5.0.8 for OS/400) 10/19/2001 01:59:06 PM Server name: applsvr/sg5162b Server directory: /domino/applsvr Partition: APPLSVR Elapsed time: 04:41:37 Transactions/minute: Last minute: 0; Last hour: 0; Peak: 50 Peak # of sessions: 3 at 10/19/2001 09:47:40 AM Transactions: 294 Availability Index: 100 (state: AVAILABLE) Message Tracking: Not Enabled Shared mail: Not Enabled Number of Mailboxes: 1 Pending mail: 0 Dead mail: 0 Waiting Tasks: 0 Thread Pools: Enabled Transactional Logging: Not Enabled Function Count Min Max Total Average OPEN_DB 14 1 21 63 4 GET_SPECIAL_NOTE_ID 6 0 1 2 0 OPEN_NOTE 105 1 269 1734 16 UPDATE_NOTE 4 7 31 61 15 DB_INFO_GET 2 1 1 2 1 DB_MODIFIED_TIME 8 0 1 7 0 DB_REPLINFO_GET 4 0 1 1 0 CLOSE_DB 14 0 9 16 1 CLOSE_COLLECTION 21 0 1 11 0 OPEN_COLLECTION 21 3 361 1387 66 UPDATE_FILTERS 2 0 1 1 0 READ_ENTRIES 22 1 126 424 19 LOCATE_NOTE 1 0 0 0 0 FIND_BY_KEY 1 1 1 1 1 NIF_OPEN_NOTE 4 3 4 14 3 NAME_LOOKUP 4 5 11 26 6 NAME_GET_AB 1 2 2 2 2 #81 1 5 5 5 5 GET_NAMED_OBJECT_ID 12 0 1 9 0 POLL_DEL_SEQNUM 15 1 2 27 1 GET_MULT_NOTE_INFO_BY_UNI 1 1 1 1 1 SERVER_TIME 1 1 1 1 1 DB_QUOTA_GET 2 1 2 3 1 FOLDER_GETIDTABLE 4 1 4 11 2 START_SERVER 11 61 701 2033 184 GET_UNREAD_NOTE_TABLE 9 0 1 8 0 GET_DBOPTIONS 2 0 0 0 0 FINDDESIGN_NOTES 1 0 0 0 0 GET_DBINFOFLAGS 1 0 0 0 0 Chapter 5. Domino logs and statistics 145 Tracking these statistics in a graph helps to determine if transactions require more or less CPU time as your use of the server grows. The average of all transactions in CPU milliseconds may be useful for identifying a Queuing Multiplier:
A good response is less than (<) 1000 milliseconds.
An average response is greater than (>) 1000 milliseconds and less than (<) 5000 milliseconds.
A poor response is greater than (>) 5000 milliseconds. See 5.4, “Monitoring the Notes Remote Procedure Calls (NRPC)” on page 156, for a description how to analyze the transactions generated by a single Notes client. 5.2 The reports The Domino server provides services and tasks that create and report information about the Domino system. This information comes in two forms: statistics and events. Statistics show the status of processes running on the system. Events are generated when something takes place on the system. For example, the event “Replicating files with servername” occurs when a file replicates with a specified server. Domino collects and stores statistics in two databases:
Notes log database (log.nsf) is created automatically when you start a server for the first time. The Notes log database contains server events, such as replications performed, mail routed, and databases used. In addition to standard information that is logged for a server, you can customize what is logged for a server. For monitoring performance, customization is discussed in “Controlling what is logged in the database” on page 146.
Statistics and Reporting Database (statrep.nsf) is created automatically when the server tasks Event or Collect is started for the first time. The Statistics and Reporting Database (statrep.nsf) contains the statistical reports that are created based on the server tasks running on the Domino server. For example, if the HTTP task is not running, no statistics related to the Web server are recorded. We describe the capabilities of statrep.nsf in 5.2.2, “The Statistics and Reporting database (statrep.nsf)” on page 149. See also 5.3, “Combining performance data from Domino and iSeries” on page 152. Combining statistic into a graph with iSeries server performance monitor data (such as CPU utilization, pool size, database/non-database faults, average disk response time per I/O operation, and average line utilization) can provide a method for tracking performance based on server usage and available system resources. Note: For Internet workloads, use the show stat command to monitor statistics. For example, enter these commands at the server console:
SH STAT SMTP
SH STAT POP3
SH STAT IMAP
SH STAT LDAP
SH STAT Domino (for HTTP server statistics) 146 Domino for iSeries Sizing and Performance Tuning 5.2.1 The Log database (log.nsf) Every Domino server has a Log database that reports all server activity and provides detailed information about databases and users on the server. The terms Notes Log database and Log File are synonymous. The Notes Log database is more for server maintenance activities versus monitoring performance or investigating performance issues. Check the Notes Log database for problems that relate to replication, mail routing, modem communications, and other scheduled server tasks. Setting up and maintaining the Notes log database is handled by the notes.ini setting log: Log Syntax: Log=logfilename, log_option, not_used, days, size This syntax controls the name of the log file, whether events are logged to the server console, and when documents are automatically deleted from the log file. Note the following details:
logfilename: The log database file name, usually log.nsf.
log_option: Log options: – 1 = Log to the console – 2 = Force database fixup when opening the log file – 4 = Full document scan (as opposed to quick scan or open)
not_used: Always set to zero; this parameter is not currently used.
days: The number of days to retain log documents.
size: The size of log text in event documents. The default is Log=log.nsf, 1, 0, 7, 40000. The log file (log.nsf) is deleted in seven days and can contain up to 40,000 words. All log information is also sent to the console. Log=log.nsf, 1, 0, 7, 40000 We recommend that you run with the default Log setting. If you decide to keep the log file for more days or it contains more words, functions that access or update the log file can decrease in performance. Accessing the Notes Log database The Notes Log Database (log.nsf) can be accessed or viewed by using one of these methods:
From the Lotus Notes client workspace, click File-> Database-> Open. At the Open Database display, select the server and Notes Log (file name = log.nsf). Click Open.
Start the Domino Administrator client and click the Files tab. To open the database, double-click Notes Log (filename=log.nsf). Controlling what is logged in the database The notes.ini settings affect the amount and type of information that appears in the Notes log database (log.nsf). The notes.ini file provides all of the initialization settings for a Domino server. Each time the server starts, it checks this file for information on which server tasks run and how to set up the server environment. Tip: If the log option should be both 1 and 2, simply add the options and it becomes log option 3. Chapter 5. Domino logs and statistics 147 For monitoring performance, the following notes.ini settings are recommended:
LOG_AGENTMANAGER=1 (no default) Specifies whether the start of agent execution is recorded in the log file and shown on the server console: 0 Do not log agent execution events. 1 Log agent execution events.
LOG_MAILROUTING=10 (default is 20) Specifies the level of information recorded in the log file and controls other logging actions: 0 Defaults to 20. 10 Displays only the errors, warning, and major routing events, for example, startup and shutdown, number of messages transferred to x, and occurrences of database compacting. Successful deliveries and transfers are not recorded in the log file. 20 Same as for 10, except that successful deliveries and transfers are logged. 30 Displays thread information. 40 Displays transfer messages, message queues, and full document information for mail.box.
LOG_REPLICATION=2 (no default) Specifies whether the start and end of replication sessions are recorded in the log file and displayed on the console: 0 Do not log replication events. 1 Log server replication events. 2 Log replication activity at the database level. 3 Log replication activity at the level of database elements (views, document, etc.). 4 Log replication activity at the field level. 5 Log summary information.
LOG_SESSIONS=0 (no default) Specifies whether individual sessions are recorded in the log file and displayed on the console: 0 Do not log individual sessions. 1 Log individual sessions.
LOG_TASKS=1 (no default) Specifies whether the current status of server tasks is recorded in the log file and displayed on the console: 0 Do not send status information. 1 Send the status of server tasks to the log file and console.
LOG_VIEW_EVENTS=0 (no default) Specifies whether messages created when views are rebuilt are recorded in the log file: 0 Do not log messages when views are rebuilt. 1 Log messages when views are rebuilt.
LOG_UPDATE=0 (no default Level of index logging: 0 Records when indexer starts and shuts down. 1 Records when the Indexer starts and shuts down + when Indexer updates views + full text indexes for specific databases. 2 As 1 + record names of views the Indexer is updating. 148 Domino for iSeries Sizing and Performance Tuning
MAIL_LOG_TO_MISCEVENTS=0 (no default) Determines whether all mail event messages are displayed in the Miscellaneous Events view of the log file: 0 Does not display mail events in the Miscellaneous Events view. 1 Displays mail events in the Miscellaneous Events view.
NO_FORCE_ACTIVITY_LOGGING=0 (no default) Controls whether the Statlog task automatically enables activity logging on all databases: 0 Allows automatic activity logging on all databases. 1 Prevents automatic activity logging on all databases.
PHONELOG=0 (default 2) Specifies whether phone calls are recorded in the log file: 0 Does not record phone calls to the log file. 1 Records all calls, except those that fail because of a busy signal. 2 Records all phone calls.
SERVER_SHOW_PERFORMANCE=1 (no default) Specifies whether server performance events are displayed on the console: 0 Records server performance events in the log file. 1 Displays server performance events on the console.
SHOW_TASK_DETAIL=1 (no default) Specifies whether the name of the current executing transaction is added to the session message: 0 Do not include the name. 1 Include the name. Figure 5-17 shows an example of the notes.ini settings. Figure 5-17 Server Configuration Document for controlling notes.ini settings Editing notes.ini settings There are multiple ways that you can edit the notes.ini file. We outline the three most common ways here: Note: The settings listed in Figure 5-17 pertain to logging statistics for performance. For problem determination work, you may need to change the settings. Chapter 5. Domino logs and statistics 149
Start the Domino administrator, and click the Configuration tab. Expand the server section. Edit or add the configuration document. Add or change to the notes.ini section of the configuration document.
Select option 13 from the Work with Domino Servers (WRKDOMSVR) command to edit the notes.ini file for a particular server.
Through Operations Navigator, the graphical user interface to the iSeries, you can work with the Domino servers on a given server. Right-click the server of your choice and choose Properties. Then, you can edit the notes.ini file. Figure 5-18 shows an example of editing the notes.ini file using Operations Navigator. Figure 5-18 Editing notes.ini using Operations Navigator 5.2.2 The Statistics and Reporting database (statrep.nsf) The Domino server continuously updates statistics. To view system statistics at any time, you use the show stat command at the server console, as explained in 5.1.2, “Understanding the Domino statistics” on page 132. To use statistics to monitor the Domino server over a period of time at a specified interval, you can use the Collect task, which collects statistics and places the information into the Statistics database (statrep.nsf). The Domino server also provides monitor documents that you use to configure statistic thresholds. When the Collect task collects a statistic and places it into the Statistics database (statrep.nsf), it compares the statistic to the threshold configured in the monitor document. The first time the statistic reaches the specified threshold in the monitor document or in the Statistic Names document, an alarm is generated. An Alarm report is nothing more than a document informing you that a statistic has reached its threshold. To set up an alarm, you must specify a collection alarm interval in the Server Statistic Collection document. Alarm documents are automatically created in the Server - Analysis Tab - Statistics Reports - Alarms view in the Domino Administrator. 150 Domino for iSeries Sizing and Performance Tuning To notify you about important system information, the Domino server has a notification facility called the Event task. The Event task sends notification about the event to a destination configured in the Event Notification document. You can create statistics and events monitoring documents for your particular server and network needs. To do this, you can create Statistic Monitors and Event Notification documents from the Statistics & Events view on the Configuration tab in the Domino Administrator, or create a new event notification in the database for Statistics & Events (events4.nsf). If you do not set up an Event Notification document, you will not be notified that an event has occurred. Accessing the Statistics and Reporting database The Statistics and Reporting database can be accessed or viewed by using one of the following methods:
Start from the Lotus Notes client workspace, and click File-> Database-> Open. Then, select the server and Statistics Reports (file name = statrep.nsf). Click Open.
Start the Domino Administrator client, and click the Files tab. To open the database, double-click Statistics Reports (file name=statrep.nsf).
Start the Domino Administrator client and click the Server tab. Then, click Analysis and expand Statistics report. Controlling what is logged in the database (statrep.nsf) The following Domino server jobs (tasks) record events and statistics in the Statistics and Reporting Database:
The Event server job
The Collector server job To configure which events to monitor and where to report them, create a Statistics Monitor Document and an Event Notification Document in the Statistics and Events Database (events4.nsf) for each type of event that you want to monitor. Configuring statistics monitors and events To add documents, perform the following steps: 1. Starting from the Lotus Notes workspace, click File. 2. Select Database. 3. Click Open. 4. On the Open Database display, select the server and Statistics & Events Database (file name = events4.nsf). 5. Click Open. 6. On the Statistics & Events display, select the Statistic view. 7. Under Statistic, click the Monitors view. 8. To add a statistic to monitor, click the Add Statistic Monitor button. As shown in Figure 5-19, more than 700 users on the server can trigger a statistic server event of Warning (High). Chapter 5. Domino logs and statistics 151 Figure 5-19 Statistic Monitor document example To determine what to do with the statistic server event of Warning (High), you need an Event Notification document. To create the Event Notification document, perform the following tasks: 1. At the Statistics & Events display, select the Event Monitors view. 2. To add an event to the monitor, click the Add Event Monitor button. Figure 5-20 shows an example of an Event Monitor document. Figure 5-20 Event Notification document example In Figure 5-20, a statistical event of Warning (High) mails a notification to the user “Redbook”. The Collector server task The Server Statistic Collection document in Statistics & Events database (events4.nsf) lets you set up a server that collects statistics from a list of specified servers. If you do not configure this document, the Collect task collects, by default, statistics only from the server that has the Collect task running on it. To optimize performance, use the Collector server job to collect statistics of all servers because it impacts only one server in the environment. This frees the other servers in your environment for end-user access. 152 Domino for iSeries Sizing and Performance Tuning The content of the statistical reports is the same information you see by running the show stat command. The statistical report is basically a collection of show stat commands run over a period of time at a specified interval. The Event server task The Event server job runs by default. At the server console, you can stop the Event server task by entering the tell event quit command and start the Event server task by using the load event command. To optimize performance, minimize the number of monitors used by the Event server task to meet your organization’s requirement for monitoring server events. If your organization does not require server event monitoring, do not run the Event server task. 5.3 Combining performance data from Domino and iSeries To combine performance data from the iSeries with the Domino server or servers, use the Statistic Collector to sample information from one or multiple Domino servers running on the iSeries server. For example, if you have two Domino servers on one iSeries server, you can use one of the servers to collect statistics from both the servers to the same statistics database. You can also monitor your whole Domino server environment performance combined with the iSeries performance data. The statistic database and collector job is further described in 5.2.2, “The Statistics and Reporting database (statrep.nsf)” on page 149. To get the statistics records, you must add PLATFORM_STATISTICS_ENABLED=1 to notes.ini. You must also create a Server Statistic Collection document where you define the collecting server and the server or servers to collect data from, as well as the database to receive reports and the collecting report interval. The Domino server Collect job must be running at the collecting server. You can read more about platform statistics in 5.1.4, “Platform-dependent statistics” on page 137. Figure 5-21 shows an example of the Server Statistic Collection document. It has been configured to collect statistics from two Domino servers at two different domains, with the collecting server applsvr/sg5162b. Both servers are on the same iSeries. Figure 5-21 Server Statistic Collection document The Report task: The Report task no longer exists in R5. It has been replaced by the Collect task, which must be loaded manually or added to the ServerTasks line in the notes.ini. If you attempt to load the Report task, the error message appears: “Error attempting to load or run report.exe: Unable to locate program.” Chapter 5. Domino logs and statistics 153 Modifying the Statistics Reports database for iSeries and Domino data In the Statistics and Reports Database (statrep.nsf), there are standard views that come with the Domino application. If you select to look at the Platform view, the only OS/400 statistics available are Platform.System.TotalUtil and Platform.System.TotalUtil.avg, as shown in Figure 5-22. Figure 5-22 Statistics and Reports Database: Standard views To view more OS/400 platform statistics in the statistics database, you can modify the Platform view or a better way is to create one or more additional views. To prevent your modification from being lost by a replace design or upgrade, we recommend that you: 1. Create a new copy of the Statistics and Reports Database template (statrep5.ntf). 2. Name the new copy statrep400.ntf. 3. Do the modifications to your template. 4. Create your own statistics database based on statrep400.ntf. 5. Configure the Statistics Collection document to use your statrep400 database as the receiver. Here, we show an example of how you can create an additional view to your new database template, statrep400.ntf, that shows OS/400 platform statistics and Domino statistics together. The intent is to give you an idea of how you can modify the views to gather different sets of statistics and to provide you with performance data that you need in your environment. 1. Start the Domino Designer and open the database template statrep400.ntf. 2. Create a copy of the Statistics Reports\Platform view. 3. Rename the copy to iSeries Platform Statistics\iSeries and Domino Statistics. 4. Open the new view for design changes, remove the fields you don’t need in this view. Add OS/400 statistics fields (Platform.x.x.x.) and Domino fields that you want to have in the same view. 5. Save your changes. Section 5.1.4, “Platform-dependent statistics” on page 137, provides information about the OS/400 statistics fields (Platform.x.x.x) that are available and can be added to the view. For detailed information about how to create/modify database views, see the Domino Designer Help Guide. Figure 5-23 shows you the statrep400.ntf template opened in Domino Designer. 154 Domino for iSeries Sizing and Performance Tuning Figure 5-23 Statrep400.ntf template: Opened in Domino Designer Figure 5-24 shows you the statrep400.ntf template with the newly copied and the renamed view iSeries Platform Statistics\iSeries and Domino statistics. Figure 5-24 Statrep400.ntf with new iSeries Platform Statistics\iSeries and Domino statistics view Chapter 5. Domino logs and statistics 155 Figure 5-25 shows you the Statistics and Report template, statrep400.ntf, with iSeries Platform Statistics and Domino statistic fields. The Transactions per minute and Active users are Domino statistics, and the rest are OS/400 statistics fields. Figure 5-25 Statrep400.ntf with iSeries Platform Statistics and Domino statistic fields In Figure 5-26, you see the Statistics and Report database, statrep400.nsf, with iSeries Platform Statistics and Domino statistic data. The transactions per minute and active users are Domino statistics, and the rest are OS/400 statistics data. Figure 5-26 Statistics and Report database with combined iSeries and Domino statistics 156 Domino for iSeries Sizing and Performance Tuning 5.4 Monitoring the Notes Remote Procedure Calls (NRPC) How do you know what actually is happening at the network level with Notes Remote Procedure Calls to and from a workstation. How can you monitor the actual traffic and know if your proposed application will actually scale, or indeed whether your network can take the strain? This section answers these questions. 5.4.1 Using a network sniffer One option to help you trace frames and packets is to use a network sniffer. A remarkably good one is available as shareware from: http://www.ethereal.com Ethereal Ethereal is a GUI network protocol analyzer. It lets you interactively browse packet data from a live network or from a previously saved capture file. Ethereal knows how to read libpcap capture files, including those of tcpdump. In addition, Ethereal can read capture files from:
Snoop (including Shomiti) and atmsnoop
LanAlyzer
Sniffer (compressed or uncompressed)
Microsoft Network Monitor
AIX's iptrace
NetXray
Sniffer Pro
Etherpeek
RADCOM's WAN/LAN analyzer
Lucent/Ascend router debug output
HP-UX's nettl
The dump output from Toshiba's ISDN routers
The output from i4btrace from the ISDN4BSD project
The output in IPLog format from the Cisco Secure Intrusion Detection System
Pppd logs (pppdump format) There is no need to tell Ethereal what type of file you are reading; it determines the file type by itself. Ethereal is also capable of reading any of these file formats if they are compressed using gzip. Ethereal recognizes this directly from the file; the “.gz” extension is not required for this purpose. Three-view window (whole, line, and hex) Like other protocol analyzers, Ethereal's main window shows three views of a packet. It shows a summary line that briefly describes what the packet is. A protocol tree is shown that allows you to drill down to the exact protocol or field in which you interested. Finally, a hex dump shows you exactly what the packet looks like when it goes over the wire. Multiple and DIY display filters In addition, Ethereal has some features that make it unique. It can assemble all the packets in a TCP conversation and show you the ASCII (or EBCDIC, or hex) data in that conversation. Display filters in Ethereal are very powerful; more fields are filterable in Ethereal than in other protocol analyzers, and the syntax you can use to create your filters is richer. As Ethereal progresses, expect more and more protocol fields to be allowed in display filters. Chapter 5. Domino logs and statistics 157 Packet capturing and compressed file support Packet capturing is performed with the pcap library. The capture filter syntax follows the rules of the pcap library. This syntax is different from the display filter syntax. Compressed file support uses (and therefore requires) the zlib library. If the zlib library is not present, Ethereal will compile, but will be unable to read compressed files. 5.4.2 Lotus Client NRPC Monitoring Tool Client_Clock Notes also has its own built-in monitoring capability. This is an unsupported, but reasonably well documented utility, that is well-known as a useful debug parameter. This parameter analyzes traffic to transaction level and shows the duration in seconds. For more information, see Technote 163971 and this site regarding Client_Clock: http://domino-8.prominic.com/A55711/ref/notesini.nsf/5a0c8fb630f30a6bc12569a1005f8 07f/7a4872639fb90313c12567d70052ce32!OpenDocument&Highlight=0,CLIENT_CLOCK Overview of NRPC Monitor The Notes Client uses NRPCs to communicate with the server. Ping-ponging RPC calls (repeating back and forth) are expensive because they generate network traffic as well as unnecessary server load. The goal is to reduce as much ping-ponging RPC traffic within the design of your application as possible, shorten the data payload lengths, and reduce the frequency that you send packets in the first place. In summary, the goal is to minimize volume and frequency. Setting up monitoring To monitor RPC traffic from a Notes workstation, add three new lines to the notes.ini file as shown in Table 5-1. The table shows the three client notes.ini variables you need to set to enable and capture NRPCs. Table 5-1 Unsupported Client_Clock parameters and syntax On startup, you see two boxes, the console and the re-directed log, with some additional information for example: (54-997) NIF_OPEN_NOTE: 2119 ms. [48+9370=9418] Table 5-2 shows a quick breakout to help you interpret the results of NRPC tracing. Important: Before you enable any debug parameters of this type, it is imperative that you test them out on a Test system and test network. There may be issues surrounding their use or special precautions that you must consider. Debug parameters often require a large amount of disk space, dependent upon traffic volumes. Lotus could also change the code without notice completely altering the affect and implications of use. Parameter Description Client_Clock = 1 Activates N-RPC tracing Debug_Console = 1 DOS Window monitoring real-time traffic Debug_Outfile= Pipes the trace output to a text file 158 Domino for iSeries Sizing and Performance Tuning Table 5-2 RFC command stream breakout and some additional comments There are a lot of different things that happen between the client and the server. However, if you follow the commands, and you know the code you wrote, then as with any other application debugger, you should be able to make sense of it. Here are some general hints and tips for monitoring NRPC or any Network traffic:
Simplify what you are trying to trace. Start easy and work you way to the more complex items as you become familiar with the process.
Disable workstation replication to stop any unwanted NRPCs from being merged.
Pause between each key stroke or action so that you can separate/debug commands. Remember that the tighter the code is, the less traffic there is and the better the performance is. We now analyze an example involving LotusScript code in the QueryClose event of a form. This code opens a view in the current database, retrieves a document by key in the current database, and finally creates a new document in another database. Table 5-3 shows the LotusScript statements issued along with the resulting series of NRPCs. Table 5-3 LotusScript showing as RPC commands using Client_Clock De-Code: Meaning (54-997) Sequence Number NIF_OPEN_NOTE= Interpretation of the call 2119 ms. Time in milliseconds to complete the call 48 Number of bytes sent 9370 Number of bytes returned 9418 Total of bytes for this call [48+9370=9418] Data sent+data received=total data in bytes LotusScript statement RPCs Set MyView= db.GetView("LookupView") (332-1952) OPEN_COLLECTION: 514 ms.[32+612=644] (333-1953) READ_ENTRIES: 2143 ms. [46+7254=7300] (334-1955) CLOSE_COLLECTION: 0 ms. [12+0=12] (335-1955) OPEN_NOTE: 1538 ms. [28+2502=2530] (336-1956) OPEN_NOTE: 1273 ms. [28+2502=2530] Set MyDoc= MyView.GetDocumentByKey("key", True) (337-1958) FIND_BY_KEY: 356 ms. [32+34=66] (338-1958) READ_ENTRIES: 411 ms. [46+94=140] Chapter 5. Domino logs and statistics 159 For more information, see Performance Considerations for Domino Applications, SG24-5602. You should also see 5.1.5, “Show Domino Transactions” on page 143. (339-1959) OPEN_NOTE: 425 ms. [28+342=370] Call newDoc.Save(True,True) (340-1959) OPEN_DB: 459 ms. [126+210=336] (341-1959) DB_INFO_GET: 640 ms. [14+140=154] (342-1960) UPDATE_NOTE: 557 ms. [566+56=622] (343-1961) CLOSE_DB: 0 ms. [14+0=14] LotusScript statement RPCs 160 Domino for iSeries Sizing and Performance Tuning © Copyright IBM Corp. 1999, 2002 161 Chapter 6. Tuning the iSeries server for Lotus Domino With Domino and the iSeries working together, there are many different variables that can affect overall performance. To help you understand the full picture, we need to address both Domino and iSeries performance. This chapter is designed to address the OS/400 perspective of the equation so you can have a full understanding of how to analyze and influence the performance of Domino applications and functions running on the iSeries server. In Chapter 7, “Tuning Lotus Domino for better performance on iSeries” on page 183, you can learn how the Domino administrator can effect the throughput of one or more Domino servers running on an iSeries server. This chapter includes these topics:
Overview of resource tuning the iSeries with respect to special behaviors of Lotus Domino
OS/400 automatic performance adjustment
When to use separate memory pools
Setting the run time priority of Domino tasks
Using online backup with BRMS
Verifying the configuration of your network 6 162 Domino for iSeries Sizing and Performance Tuning 6.1 Tuning the iSeries server Working on and fixing a performance problem is similar to solving a three-dimensional puzzle. All parts of the puzzle initially appear to be unconnected. Changing one part (for example, increasing a priority) seems to cause unexpected changes and problems with what was thought of as a totally unconnected part of the whole puzzle! But, after solving several such puzzles, you begin to grasp a pattern. Solving a Domino performance problem is a challenging task. It requires an open mind and a “holistic” approach (exclude nothing from your initial enquiries). It also requires the ability to admit your earlier hypothesis may have been wrong and you may need to start with a new line of thinking. The most common cause of failure in diagnosis is to choose the wrong path and never come back for another look around. Before you attempt to fix or tune Domino performance issues, you must have a solid base to work from, that is the platform, which in this case is the iSeries server. The base platform itself must be tuned properly (as far as you can) before you can even start to determine what is causing the Domino performance problem. Domino is simply an application suite that has “work” and “memory management” functions of its own and is designed (on the iSeries) to run in a dedicated subsystem of its own. Tuning Lotus Domino for iSeries and tuning the iSeries server, therefore, have to go together. First, tune the iSeries server to provide Domino with (what you think are) the resources it needs. Then tune Domino. Then perhaps come back and re-tune the iSeries a little more. And then possibly go back to re-tune Domino again and so on until everything is balanced and running in harmony. This chapter explains how to provide those iSeries server resources to the Domino application. Any actual tuning (changes to) the iSeries server should only be attempted after a very careful analysis of the overall performance characteristics of the system is completed. This is described in Chapter 4, “Basic concepts of performance analysis” on page 53. Tuning the iSeries server involves these steps: 1. Measure the performance. 2. Change one parameter or setting at a time. 3. Measure the performance again, compare the results, and go back to step 2, if you are not satisfied yet. Tuning the system means that you try to allocate the available system resources to users or batch jobs that are in most need of those resources. Remember, there is an absolute and finite limit on the number of resources in the system, and all the jobs in the system compete for available resources. Once all resources have been used up, all “tuning” really achieves is to take resources away from one job and give them to another. You cannot increase available resources, obviously, but only re-allocate them. Hopefully you will find one job or subsystem over-allocated with spare resources, and you can safely re-allocate those spare resources to a job that is starved of resources and everyone is happy. But eventually, once all jobs have the correct resources that they need, you will be at the point where one job is stealing resources from another. That is the time when you must upgrade the hardware to gain additional resources. In truth, you should have seen this coming from your monitoring of the system before the workload grew large enough to eat up all the available system resources. System tuning also involves making every attempt to minimize or eliminate the unnecessary system work, for example: Chapter 6. Tuning the iSeries server for Lotus Domino 163
Monitor the number and arrival rate of messages in the job logs and history logs.
Monitor all error logs in the service tools (SST).
Avoid multiple sign on and sign off functions.
Clear all output queues, old history files, and all old journal entries on a regular basis. Perhaps the easiest way to affect the system performance is to evenly distribute the system’s workload, for example:
Move heavy batch jobs to run during the night or other off shift time.
Perform system save activities during a lower system usage time.
Assign the programmers a subsystem and storage pool of their own, reserved only for their work.
Keep the workload on the system as constant as possible. Once you have tried re-distributing the workload and tuning as much as possible, and it still does not meet your performance goals, it’s time to buy more hardware! Upgrading your system’s hardware may seem like an expensive solution. But when you consider the costs of hiring and maintaining a staff with the required skills to streamline your applications (and with the skills to perform a Performance Audit and produce a concise but accurate report), the cost of upgrading the system hardware may actually be lower in comparison. If you follow the advice and guidance in this redbook, you will know and understand the available options and how to measure and justify your decisions. 6.2 Automatic performance adjustment (QPFRADJ) The automatic performance adjustment in OS/400, activated and controlled through the system value QPFRADJ, can be a great boon and a great tool, but in very few situations, it may have a negative impact. In general terms, it is a fine tool to leave switched on; it balances workloads against resources and constantly strives to optimize the performance of all jobs and pools on the iSeries server. It is like having a fully skilled OS/400 expert constantly fiddling and re-balancing your system. Having said that, its very innate ability can also be its own downfall where Domino is concerned. The main issue involved in having the automatic performance adjuster turned on is when the workload on the system changes over the course of a day. Here is a typical customer scenario that describes the situation: 1. The workload during the prime shift involves Domino applications that provide various services for the end users. The Domino server may route mail, perform workflow functions designed into the application, access DB2 UDB data, and serve the company’s Web site. While all of this is happening, the automatic performance adjustor looks at all of the work happening on the system and provides Domino with the adequate resources. This involves increasing the activity level of the memory pool where Domino is executing and allocating more memory to the jobs that support the Domino function. 2. In the evening, the workload on the system changes. Most of the Domino activity ceases, and other night time tasks fire up, such as backups, batch jobs, and other clean-up tasks. The automatic performance adjustor now moves memory and activity levels to these tasks so they have the appropriate resources. Important: When making changes, apply only one change at a time and measure the effect of that change on system performance. 164 Domino for iSeries Sizing and Performance Tuning 3. The situation arises in the morning when all of the workers come into work and start using the Domino applications and services again. At this point, the automatic performance adjustor needs to start moving all of the memory from the night tasks back to the Domino server. This can mean poor performance until the automatic performance adjustment has caught up in rebalancing resources. Performance adjustment is relatively slow to react to such a sudden change (logically that is correct; it shouldn’t suddenly rush to allocate massive resources into the first spike of activity it detects). Because of this, it only re-establishes the resources over a period of some minutes. Meanwhile every Notes Client is attacking the Domino server with a connection request. In R5, if Domino doesn’t give a quick response, the client generates another request to the server, and then another, up to 5, in an attempt to establish a server session. As hundreds of users fire off these multiple connection attempts (“retries”), Domino will inevitably belatedly establish multiple connections for each user. Each Domino connection uses considerable server resources just to actually establish the session. In turn, this further ramps up the server load, slowing overall processing yet more, and therefore, causing other users to issue multiple connects in an ever increasing downward spiral. Domino itself also has a self-protection mechanism to limit maximum user connections with the Server_MaxUsers parameter in notes.ini. However Domino can easily handle its standard workload of hundreds or thousands of users, with resources to spare. Every Notes client is now pretending to be up to five users, before long Server_MaxUsers is reached, and these connections will stay engaged until Server_connection_timeout (four hours by default1) is reached! So now your early morning users are being refused access, and your phone starts to ring (and continues to ring Server_MaxUsers times for Server_Connection_Timeout minutes to be exact!). A simple solution to beat the “early morning rush” is to use the Work with Shared Pools (WRKSHRPOOL) command, press F11 (Display tuning data), and set a minimum size for the storage pool where your Domino servers are executing. Typically this is the *BASE pool. See 6.4, “Choosing which memory pool to use” on page 166, for information on using other pools. Both a minimum and maximum percentage of total main storage can be specified for each memory pool. These settings are used by the system if the automatic performance adjustment system value QPFRADJ is set to 2 or 3. This will guarantee a minimum amount of memory will remain allocated to Domino, therefore, avoiding the early morning ramp up problem. We recommend that you set QPFRADJ to 3 (Automatic adjustment). Option 2 (Adjustment at IPL and automatic adjustment) may also work to initially set the environment on a new system, but the values will be reset after each IPL and need to be re-adjusted after starting the servers. 6.2.1 The interactive iSeries tuning chart In many cases, we recommend that you leave performance tuning to the system with the system value QPFRADJ set to 3. However, in special cases, it may be beneficial to do some or all of the tuning manually. If you decide to do so, follow our road map shown in Figure 6-1. 1 The default to that is 4 hours, which later we recommend that you lower to 30 minutes. Chapter 6. Tuning the iSeries server for Lotus Domino 165 To follow the road map, follow steps 1 through 3 to ensure the machine pool has sufficient resources. If the machine pool is starved for resources, performance of the whole system may be impacted. After determining that the machine pool has adequate resources, repeat steps 4 through 7 in Figure 6-1 for all of the other pools in your iSeries server. Do this during periods of high system activity because it has no value to tune the system when there is only a relatively light workload on the system. Make sure that system value QPFRADJ is set to zero (to not adjust), before you follow the tuning road map. If you leave QPFRADJ set to value 2 or 3, it may undo the tuning you are attempting to perform. Figure 6-1 Interactive tuning roadmap 6.2.2 Summary Automatic performance adjustment is extremely useful. iSeries servers work and stay working at their optimum with it switched on (option 3) and left on. Only on very large sites where a dedicated team of operators and system programmers permanently monitor and perform the same functions “by hand” can it be safely left off. Be sure you have looked very carefully at all the implications of not using this important iSeries feature if you choose not to use it. 1. Enter the WRKSYSSTS command. Press PF21 to set the assistance level to Intermediate. 2. Wait 2 - 3 minutes and press PF5 to refresh. 3. Do the *MACHINE NDB faults meet the guidelines? Yes..... Press PF10 and go to step 4 No........Adjust QMCHPOOL -500K if fault rate = 0 +500K if fault rate >3.0 Press PF10 to reset and go to step 2 4. Is the DB fault + NDB fault > 20 in any pool? Yes ... Increase pool size by 500KB, press PF10 and repeat step 4 (repeat until all pools are less than 20) No ... Go to step 5 5. Wait 2 - 5 minutes, and press PF5. Press PF21 to set the Assistance level to Advanced. Is the Wait-to-Ineligible state = 0 ? Yes ... Reduce Activity level by 2, press PF10 to reset and repeat step 5 No .... Go to step 6 6. Is the Active to Wait state 10x the activity level? No ....System not heavily used or complex application mix; go to step 4 Yes ... Go to step 7 7. Is the sum of all fault rates for all pools within the guidelines? No .... Go to step 4 Yes ... Go to step 8 8. Activity levels and pool sizes probably OK. Continue monitoring the WRKSYSSTS display regularly. 166 Domino for iSeries Sizing and Performance Tuning 6.3 Determining Domino workload Domino by itself is not an application. Domino is an Application Architecture Framework onto which you can hang an almost unlimited type and style of application program/job/task written in LotusScript, Java, and the Lotus command language (known as @Functions or macro language). On the iSeries server, Domino is implemented in the subsystem architecture. Each Domino server then has its own set of tasks or jobs that comprise the function configured for a given Domino server. Each of the Domino tasks are presented as an iSeries job. For example, the SERVER task is implemented as a job called SERVER2 in the Domino subsystem. This one job will spawn multiple threads to handle simultaneous requests from multiple clients. Likewise, the HTTP Domino task is implemented as the HTTP job in the Domino server subsystem. The HTTP task will have multiple threads to handle the requests from various browser users. You can find more detailed information about each Domino tasks in Chapter 8, “Understanding the Domino server jobs” on page 229. With this understanding, it becomes relatively easy to figure out what your Domino workload is. The Domino work will execute in Domino subsystems and each of these subsystems will have Domino tasks or jobs that spawn multiple threads to handle the incoming requests. In the remainder of this chapter, you will find information about working with these various tasks or jobs. You’ll also find recommended settings for the various Domino parameters that you can tweak in your efforts to create the highest performing Domino environment on your iSeries server. 6.4 Choosing which memory pool to use A memory pool is a logical division of main memory or storage that is reserved for processing a job or group of jobs. In OS/400, all main storage can be divided into logical allocations called memory pools. By default, the system manages memory pools. The system manages the transfer of data and programs into memory pools if necessary. You can control how much work can be done in a subsystem by controlling the number and size of the memory pools. The greater the size of the memory pools in a subsystem, the more work can be done in the subsystem. Note: WRKSYSVAL QPFRADJ is not the only performance adjuster tool available for iSeries. Several IBM Business Partners have built similar and often more comprehensive toolkits that provide automatic performance monitoring and adjustment to provide a total and fully tailorable performance adjusting and tuning solution. 2 There is not always a one-to-one match between the name of server tasks and the job name. The INDEXER task appears as an UPDATE job, and all job names starting with the letter Q (for example QNNINSTS) do not appear in the Domino tasks list. Note: Prior to OS/400 V5R1, the pools were called storage pools. Now Operations Navigator, the graphical user interface for OS/400 administrators and operators, uses the term memory pool, while most of the CL commands still refer to “storage pool”. To be consistent with the latest version of the user interface, we use the term “memory pool” in this book. Chapter 6. Tuning the iSeries server for Lotus Domino 167 By default, all Domino servers will run in the *BASE memory pool. This is quite adequate in most scenarios, but there are times when you may want to place a Domino server in its own storage pool to allow you to specify specific priorities, amounts of memory, and so on. The scenario where this becomes most important is when Domino is running on the same server that is also hosting interactive, 5250-based applications written in RPG or other languages. Both the Domino tasks and the 5250 jobs will run at run priority 20. They will compete for the same CPU cycles. Since Domino tends to require much more processing power than traditional OS/400 applications, a heavy Domino application may severely impact the response times of other applications. If Domino and traditional interactive applications were left to run at the same priority level and in the same memory pool, neither set of applications would run well. In this scenario, it is much better to separate the memory pools for the Domino server and interactive jobs. The reason for doing so is this, as described above, is that the interactive jobs typically stay active for a very short time. If Domino tasks stay active for much longer time periods, the pages needed by the interactive jobs will be moved out of memory over time. Once the jobs become active again, those pages need to be moved back into memory. This costs valuable time and increases the response times. The purpose of separate memory pools is to guarantee a certain amount of memory to a group of jobs, which will not be paged out, because other jobs (not part of the same group) need memory. Typically this “group of jobs” belongs to the same application or run in the same subsystem, but that is not necessarily so. For example, you may use the predefined shared3 pool *INTERACT for all your interactive 5250 applications. Figure 6-2 shows an example of all active memory pools as they appear under Operations Navigator. Figure 6-2 Active memory pools Periodically checking the amount of memory your memory pools use is important. By monitoring these levels, you can tune your pools to run at maximum efficiency, which in turn, keeps the work cycle running smoothly. In Operations Navigator, you can easily monitor the amount of memory your pools are using. To check the memory usage, follow these steps: 1. In Operations Navigator, expand My AS/400 Connections. 2. Expand the connection for your iSeries server. 3. Expand Work Management. 3 “Shared” means this pool may be shared by jobs running in multiple subsystems, but not by all jobs in the system. 168 Domino for iSeries Sizing and Performance Tuning 4. Expand Memory Pools, and then click Active Pools or Shared Pools. 5. Right-click the memory pool you want to work (for example, Interactive) and select Properties. 6. Click the Configuration tab. The Current field, under Size, shows the amount of memory the pool currently has. The following sections describe the different types of memory pools and how to assign one to Domino servers. The machine pool (*MACHINE) This is the pool in which highly shared OS/400 jobs and microcode tasks (License Internal Code) run. The machine pool provides storage for jobs the system must run that do not require your attention. The size for this memory pool is specified in the system value QMCHPOOL. No user jobs run in this pool. While Operations Navigator refers to the pools by name (see Figure 6-2), the machine pool appears as system pool identifier 1 on the Work with System Status (WRKSYSSTS) display. Normally, this pool should have the rate of non-DB faults less than one per second. The base pool (*BASE) This is the memory pool where all Domino jobs run by default. It contains all memory that is not allocated by other (private or shared) memory pools. The base pool contains storage that can be shared by many subsystems. The system value QBASPOOL specifies the minimum size of the base pool. Its activity level is specified in the system value QBASACTLVL. The base pool is used for batch work and miscellaneous system functions. While the Operations Navigator refers to the pools by name (see Figure 6-2), the machine pool appears as system pool identifier 2 on the Work with System Status (WRKSYSSTS) display. We recommend that you keep all Domino jobs in the *BASE pool, unless you decide to isolate the storage for one or more Domino servers in one of the shared pools*SHRPOOL1 to *SHRPOOL60. General shared pools (*SPOOL, *INTERACT, *SHRPOOLxx) There can be up to 62 shared memory pools in OS/400. As the name implies, they can be shared by two or more subsystems. However, to enable automatic performance adjustment for a pool used by a Domino server, it may make sense to assign one of the shared pools to a single Domino subsystem. The size and activity level of a shared pool, as well as the expert cache behavior (paging option), can be changed with the Change Shared Pool (CHGSHRPOOL) or Work with Shared Storage Pools (WRKSHRPOOL) command. After you define the size and activity level for one of the shared pools, you use the Change Subsystem Description (CHGSBSD) command to assign the pool to the subsystem. For example, to assign *SHRPOOL3 to the subsystem DOMINO02, enter: CHGSBSD SBSD(QUSRNOTES/DOMINO02) POOLS((1 *SHRPOOL3)) To assign the *BASE pool to the subsystem again, you would use the following command: CHGSBSD SBSD(QUSRNOTES/DOMINO02) POOLS((1 *BASE)) Chapter 6. Tuning the iSeries server for Lotus Domino 169 The following rules apply to Domino subsystems:
The subsystem description for each Domino server resides in library QUSRNOTES.
The name of the subsystem description is, by default, DOMINOxx or the name you assigned to it when you configured the Domino server. You can find out about the existing subsystem names for Domino by using the Work with Domino Servers (WRKDOMSVR) command.
In general, each subsystem can have up to 10 memory pools assigned to it. However, the Domino tasks will only use the first pool in the subsystem description. Therefore don’t assign more than one pool to each subsystem.
The Change Subsystem Description (CHGSBSD) command can be used while the subsystem is active. However, if you assign a new pool, it will only be used when new jobs are started, which is normally only the case when the server is started. Using the Domino console command restart server is not sufficient, because the SERVER job remains active in the same pool as before and so do all other tasks submitted by SERVER in order to complete the restart.
The shared pools *SPOOL and *INTERACT should not be used for Domino subsystems, since they are predefined for the spool printer (QSPOOL) and the interactive subsystem (QINTER). QPFRADJ uses some special assumptions. Private memory pools Private pools can only be used by a single subsystem and are not automatically tuned by QPFRADJ. Therefore, we do not recommend using a private pool for a Domino server, unless you have a good reason for manually tuning a single Domino server. For more information about creating memory pools, see the iSeries Information Center (http://publib.boulder.ibm.com/pubs/html/as400/infocenter.htm) or OS/400 Work Management for Version 4 Release 4, SC41-5306. 6.4.1 Using expert cache You can set up the system to dynamically adjust a memory pool. Expert cache uses a storage management tuner, which runs independently of the system dynamic tuner, to examine overall paging characteristics and history of the pool. There is a setting for each memory pool that controls how aggressive OS/400 storage management is about paging data into memory. This setting is called expert cache. When expert cache is turned on for a memory pool, if objects are referred to sequentially, the system brings large blocks of data into memory and delays writing changes back to auxiliary storage. This reduces the number of I/O operations used by the job. It also reduces the contention for disk drives, which in turn, reduces the time that jobs wait on I/O requests. If objects are referred to randomly, the system does not bring in large blocks of data because if would not reduce the number of I/O operations. One advantage of using expert cache (*CALC) is that the system dynamically determines which objects should have larger blocks of data brought into main storage. This is based on how frequently the object is accessed. If the object is no longer accessed heavily, the system automatically makes the storage available for other objects that are accessed. If the newly accessed objects become heavily accessed, the objects have larger blocks of data placed in main storage. Expert cache was created because of the need for batch jobs to page large amounts of data into memory and then process it. It was not very efficient for the batch jobs to have to keep paging data into memory, so expert cache helped to alleviate this problem. 170 Domino for iSeries Sizing and Performance Tuning Domino can benefit from expert cache because of its batch-like processing characteristics. Because a Domino database contains the data and the design elements, it can be very helpful to use expert cache to keep the Domino database paged into memory. Setting expert cache There are two paging options associated with each shared memory pool, *FIXED and *CALC. By default, all pools have the option *FIXED specified. To turn on expert cache, change the paging option for the pool to *CALC using either the Work with System Status (WRKSYSSTS) or the Work with Shared Storage Pools (WRKSHRPOOL) command.
*FIXED: The system limits the amount of memory used by threads that are running in the memory pool. The system transfers data from auxiliary storage and frequently writes changed data back to auxiliary storage. When *FIXED is specified for a pool during performance data collection, the Storage Pool Activity section of the Performance Tools Component Report shows a value of 0 against the expert cache value for that pool.
*CALC: The system automatically determines the best approach for handling data in the memory pool. If many threads are running in a small memory pool, the system limits the amount of memory used by each thread. If the pool for those threads has enough memory, the system determines (object by object) how much data to bring into the pool. When *CALC is specified for a pool during a performance monitor data collection, the Storage Pool Activity section of the Performance Tools Component Report shows a value of 3 for the expert cache value for that pool. As with any performance advice, this does not mean that expert cache is recommended for every Domino environment. Expert cache can help in environments when a single Domino database is accessed very frequently and by many different users. A Domino help desk application is a good example. This database will be heavily used by the help desk staff, not only for the function the application provides, but also for the large quantities of data stored in the Domino database. 6.4.2 Activity levels of memory pools Memory pool activity levels allow for efficient use of system resources by allowing more than one thread to be active at the same time. The activity level of a pool is the maximum number of threads that can be active at the same time in a memory pool. The system manages the control of this level. Often during processing in a thread, a program waits for a system resource or a response from a workstation user. During such waits, a thread gives up its use of the memory pool so that another thread that is ready to be processed can take its place. When more threads are started than can run at the same time because of the activity level controls, the excess threads have to wait to use the processing unit (normally this wait is very short). The memory pool activity level lets you limit the amount of main storage contention in the various memory pools for your subsystems. For the memory pool activity level, the number of threads running (or active threads) refers to the number of threads that have an activity level. That is, the threads are actually running or they may be waiting for a disk I/O operation. In this sense, active threads do not refer to threads that are waiting for input, a message, a device to be allocated, or a file to be opened. Active threads do not refer to threads that are ineligible (threads that are ready to run, but the memory pool activity level is at its maximum). Note: If you tune using *CALC and the system ends abnormally, the recovery time could be longer than the recovery time would be with fixed paging. Chapter 6. Tuning the iSeries server for Lotus Domino 171 A rule of thumb for the activity level factor used for the Domino subsystem is to have a value large enough to support all the threads running on the subsystem. Use the WRKACTJOB command to find the total number of threads running in the Domino subsystem or better still use option 9 in the Work with Domino Servers (WRKDOMSVR) panel. Option 9 in WRKDOMSVR is preferable over WRKACTJOB, because it shows only a subset of all jobs and, therefore, causes less overhead. From there, you only have to press F11 twice to see the number of threads for each job (Domino task) or use option 12 to see all threads of an individual job. Most Domino tasks are multi-threaded. Examples of multi-threaded tasks include Server, Event, and AMgr. During previous testing, we found that an inadequate number of activity levels in the main memory pool can cause time-outs for the threads in the server job, which in turn, could cause the server application program/job itself, such as Domino, to crash. This is true for OS/400 V4R4 in particular since the threading model changed in V4R4, requiring more activity levels to be required. We also found that an insufficient number of activity levels causes the server task to end abnormally and then automatically start again, which causes a huge performance impact. These problems are reported in the QSYSOPR message queue with messages CPF0908 and CPF0909. 6.4.3 Faulting rates Keep the page faulting rate in machine pool, memory pool 1, as low as possible because this pool contains objects that are used system-wide. One way to affect the paging in the machine pool is to adjust the size of the pool. The automatic performance adjustor built into OS/400, QPFRADJ, calculates an acceptable level of faulting based on the number of active threads in a pool. Even if QPFRADJ = '0' (no tuning), you can use the same calculations to determine acceptable levels of faulting in your storage pools. QPFRADJ uses the faulting levels in Table 6-1. Table 6-1 QPFRADJ faulting levels QPFRADJ assumes:
*INTERACT has only interactive jobs.
*SPOOL has only spool jobs.
*SHRPOOL1-60 has only batch jobs. Here are some examples to illustrate this:
In an interactive pool, if there are 40 active threads, an acceptable faulting rate would be: 5 + (40 × .50) = 25 faults per second
If there are 20 batch threads active in a pool, an acceptable faulting rate would be: 10 + (20 × 2.00) = 50 faults per second Pool Minimum faults Faults per thread Maximum faults *MACHINE 10.00 .00 10.00 *BASE 10.00 2.00 100 Interactive 5.00 .50 200 Spool 5.00 1.00 100 Batch 10.00 2.00 100 172 Domino for iSeries Sizing and Performance Tuning
If the calculation is greater than Maximum Faults, then use the Maximum Faults value. Here are some general notes on sizing pools, activity rates, and faulting:
The rule for setting the machine pool size is to set it to two times the number shown in the Reserved Size column.
Notice the sum of Non Database (Non-DB) faults in all of the storage pools. The Non-DB faults include program code, thread work areas, and variables, among other structures. To affect the faulting rate in the pool (except the machine pool), change both the size and the activity level of the pool. You can use one of two methods to decrease the amount of page faults: – Reduce the activity levels per pool to provide more memory per activity level. – Increase the size of the storage pool to add more storage available per activity level. You can use one of two methods to increase the amount of faults: – Increase the number of activity levels per pool. – Decrease the size of the storage pool.
Notice the sum of the database faults in all of the storage pools. Remember that a system with no non-database faults is a dead system. Basically, a fault is an order to acquire data from a disk and transfer it to main storage to allow changes to the data. Technically speaking, a page fault is a program notification that occurs when a page that is marked as not being in main storage is referred to by an active program/job/task. To increase the number of activity levels, simply specify a new value for pool size on the WRKSYSSTS screen (see Figure 6-3 on page 175).
Notice the thread state transitions. When the pool size and activity level settings are in balance with each other, the ratio of columns (from left to right) should be ten to one (10:1). Usually, when the pool size and activity level settings are correct for the workload, the transition rates fall within the guidelines. Wait-to-Ineligible transitions need not be zero all the time. When there is a momentary period of heavy usage, it may be better to let the jobs become ineligible to avoid excessive page fault rates or thrashing.
Increase the number of activity levels to reduce the rate of Wait-to-Ineligible transitions. For pools including interactive like threads, the Wait->Ineligible should be small (less than 10% of Active->Wait). If you see any Active->Ineligible, increase the activity level (MAXACT) by 5 to 10 until the Active>Ineligible is 0. After you increase the MAXACT value, press PF10 to reset the statistics. Do not use PF5 to refresh. You should wait at least one minute between refreshes.
Keep the time frame of the observation period between five and 30 minutes. If the observation period is less than five minutes, the occasional peak loads tend to distract the rates of both faults and pages. If the time period is over 30 minutes, the important data may be lost because the counters holding the data may get wrapped. Notes:
The interactive guideline of 0.50 is based on the assumption that the user enters a transaction every 20 seconds and 10 faults per transaction is acceptable level. (10 faults per transaction per thread ÷ 20 seconds per transaction = 0.5 faults per thread.)
The batch guideline is based on the assumption that a page fault takes 0.010 seconds. For one job, the acceptable level is 12 faults per second. Therefore, the batch thread would spend 12% of its time page faulting (.010 × 12 × 100%). This should be an acceptable level for most systems. Chapter 6. Tuning the iSeries server for Lotus Domino 173 6.5 Choosing which processor priority to use The term “interactive” has a very specific meaning in OS/400. It refers to 5250 (or 3270) green-screen based processing with some strong performance implications. Each processor has an Interactive Commercial Processing Workload (Interactive CPW) rating that provides an indication of how much interactive work can be accommodated. Although we as Domino users would consider much of Domino’s Mail and Applications to be “interactive”, in OS/400 terms the tasks that support this Domino work are actually implemented as batch immediate jobs. When looking at changing or setting priorities, Domino has tasks with different behavior patterns; some such as the SERVER task, behave more like interactive jobs:
They become active for a relative short period of time, typically for some seconds or less.
Then they wait for the user’s next request, usually longer than a minute. ADMINP, AMGR, and UPDATE perform in a more batch like manner, similar to traditional batch jobs:
They run constantly for a long time (minutes or even hours).
They only pause when they have to wait for disk I/O or at time-slice end if another job/thread with higher priority is ready to run.
The number of jobs/threads with higher priority (and not in long wait) is higher than the activity level for the pool. The run priority of jobs on the iSeries is normally set based on the type of work that is being performed:
Interactive jobs started on a 5250 (or 3270) terminal run at priority 20.
Traditional batch jobs that are submitted to run in the background run at priority 50 by default. The higher priority of interactive jobs ensures that requests from a user are honored immediately by possibly interrupting a batch job running with lower priority (50). Since interactive usually only spend a short time actively processing, control will be soon returned to the batch job or jobs. All Domino jobs on the iSeries server run as batch immediate jobs and all have a run priority of 20. That is, even Domino tasks with a “batch like” behavior compete directly with other tasks or interactive jobs, which need to respond quickly. In most cases, this does not pose any kind of problem. However, we have seen situations where a huge portion of the total CPU utilization was consumed by background tasks like UPDATE or AMGR. The fact that overall CPU utilization raises very high does not necessarily mean it causes problems, but when the majority of CPU resources is consumed by high priority jobs, the response times may increase. So, if the response times for the Notes clients (and traditional interactive applications, if any) raises exceptionally high during times when you observe high CPU utilization caused by jobs like UPDATE, AMGR, it may help if you reduce the priority of those tasks. Note, however, you should not overestimate. Note: Remember that the purpose in tuning is to process the workload as smoothly as possible without overusing any of the system resources. 174 Domino for iSeries Sizing and Performance Tuning You may find out which jobs are consuming most CPU resources by using the Work with System Activity (WRKSYSACT) command, described in 4.10.4, “Work with System Activity (WRKSYSACT) command” on page 71, or the platform dependent statistics in statrep.nsf (see 5.3, “Combining performance data from Domino and iSeries” on page 152). 6.5.1 Changing the run priority of Domino tasks With Domino R5, you can change the priority (or other runtime characteristics like time slice) of Domino jobs permanently. Prior to R5, any changes to the runtime priority of a Domino job only lasted for the life of the job. If the Domino subsystem was stopped and then restarted, the priority of the individual jobs that had been changed were reset back to the original runtime priority. With R5, you can permanently enable a Domino job to run at a different priority than the default of priority 20. This is a two step process: 1. Create a class that defines the specified run priority. Use the Create Class (CRTCLS) command for this step. The class must be created in the library QUSRNOTES. Here is an example of creating a class named updatecls that will have a runtime priority of 30: crtcls cls(qusrnotes/updatecls) runpty(30) crtcls cls(qusrnotes/mailcls) runpty(28) crtslc cls(qusrnotes/amgrcls) runpty(25) 2. Tell the Domino server which tasks should use this class. Domino for iSeries looks for this information in the directory /qibm/userdata/lotus/notes/ domino_classes. You can use the Edit File Utility (EDTF) to create this file if it doesn't already exist. The contents of this file should be SERVER, the name of the Domino server; CLASS, the name of the class you created in step #1 above; and TASKS, a list of the Domino jobs (or tasks) that you want to use for the class you created to obtain a different runtime priority. An example follows: SERVER=Mail01 CLASS=updatecls TASKS=update CLASS=mailcls TASKS=router,smtp SERVER=App01 CLASS=amgrcls TASKS=Amgr Each time the update job is started for Domino server Mail01, it obtains the runtime priority that was defined in class updatecls, with a runtime priority of 30. Notice that the Domino tasks router and SMTP that which are also associated with Domino server Mail01 will have yet a different runtime priority, the priority associated with class mailcls (28). The /qibm/userdata/lotus/notes/domino_classes file is referenced for all Domino servers on a given iSeries server. To change the runtime priority of a task in a different Domino server, specify the server with the keyword SERVER and then reference the class and tasks that you want to use the new runtime priority. Here the AMgr task in the App01 Domino server will adopt the run priority of the amgrcls class, which is 25. Be careful when adjusting the run priority of Domino tasks. If you do not fully understand the impact of the changes you are making, you may cause other performance problems or cause the application to fail. For example, if you change the run priority of the router task to a lower run priority (30 as an example), mail may not be routed in a timely fashion and the server can get back logged. Chapter 6. Tuning the iSeries server for Lotus Domino 175 6.6 DASD: System ASP, user ASPs, and independent ASPs Typically on iSeries all disk (DASD) is shared by all jobs, using (system ASP 0). It is possible to limit disk space by dividing out some of the main DASD storage at a physical disk level into 1 to 32 auxiliary storage pools (ASP) composed of one or more disks in each user ASP. In V5R1, the independent ASP (IASP) allows the definition of up to 67 IASPs. The IASP allows a set of disks to be moved to a different iSeries server in the case of a planned or unplanned outage of one server. Using ASPs is less commonly done these days with the advent of RAID, mirrored disks, and storage area network (SAN) systems. But historically separating out some ASPs for journal files was essential, so that both the journal and data were not sharing the same physical unit for performance reasons and could be recovered in the event of a disk crash. This technique with using ASPs is particularly important to separate out the transaction logging files from the Domino data files and directories for performance reasons to minimize the movement of the disk access arm. Figure 6-3 shows the system status display, which includes the pool activity. Figure 6-3 Work with System Status display Notes:
All Domino jobs/tasks can have their runtime priority changed with the exception of the QNNINSTS job. That can be changed together with the default for all other task by changing the class with the same name as the domino subsystem (for example DOMINO01) in library QUSRNOTES.
OS/400 Version 4 Release 3 (V4R3) or higher is required for this support. 12/09/01 12:41:13 ACME01 % CPU used . . . . . . . : 26.2 System ASP . . . . . . . : 106.0 G Elapsed time . . . . . . : 00:00:05 % system ASP used . . . : 76.8542 Jobs in system . . . . . : 776 Total aux stg . . . . . : 106.0 G % perm addresses . . . . : .007 Current unprotect used . : 7345 M % temp addresses . . . . : .015 Maximum unprotect . . . : 7930 M Sys Pool Reserved Max ----DB----- --Non-DB--- Act- Wait- Act- Pool Size K Size K Act Fault Pages Fault Pages Wait Inel Inel 1 668800 280884 +++++ .0 .0 .0 .0 11.8 .0 .0 2 1741704 20 123 .0 .0 .0 .0 47.2 .0 .0 3 47184 0 9 .0 .0 .0 .0 .0 .0 .0 4 1260104 0 17 .0 .0 .9 1.9 23.6 .0 .0 5 1000800 316 1775 .0 .0 .0 .0 3139 .0 .0 Bottom ===> F21=Select assistance level 176 Domino for iSeries Sizing and Performance Tuning 6.7 Unleashing iSeries work management on Domino servers The way Domino is implemented on the iSeries is to have each Domino server be its own subsystem. There are various aspects to keep in mind when configuring a Domino server that can impact the performance. Let’s address some of these areas. 6.7.1 Configuring a Domino server When you configure a Domino server on your iSeries server, a subsystem description is automatically created for you. In Domino R5 for iSeries, you have the choice of accepting the default name of DOMINOxx or providing a more descriptive name of you own choice for the subsystem. You may want to consider creating a memory pool for every server instead of using the default values that point to a system pool *BASE as explained in 6.4, “Choosing which memory pool to use” on page 166. If you do, you should change the subsystem description in library QUSRNOTES, which was created during the configuration process. We found, however, that most Domino implementations are fine by leaving all Domino subsystems running out of the default *BASE memory pool. 6.7.2 Starting a Domino server By default (“out of the box”), all the jobs running in the Domino subsystem are started with the same job class, and same job class name as the default subsystem description for Domino in library QUSRNOTES. All jobs running in a Domino subsystem have exactly the same run attributes by default. This works well in the majority of cases, but is not ideal from an iSeries work management point of view when many Domino servers are competing for the same resources, or when a Domino server is competing with interactive tasks executing on the system. Some of the Domino jobs are for batch-like functions, such as router, updater, and indexer tasks, where some jobs are for highly responsive functions, typically the threads for users NRPC (client) requests and HTTP requests. If you start seeing a problem on an “untuned” Domino for iSeries, this is one of the first areas to look. You have two options to adjust the run parameters on a Domino subsystem:
Use Operations Navigator or option 2 in the Work with Active Jobs (WRKACTJOB) panel to change the parameters of one job (and all the threads initialized by that job) at a time on a real-time basis. This changes the run priority only for the duration of the job. Once the Domino server is ended and restarted, the run priority will change back to the default of run priority 20. For details on adjusting the run priority of individual jobs, see the discussion in the following section.
Create a partitioned server for separate function sets that enable you to create different memory pools for these servers. With this method, you can also specify different run attributes by using different class descriptions in various Domino subsystem descriptions. For example, create a separate Domino server just for mail routing purposes. We have customer requests to lower the priority of Domino R5 jobs, including Update, Sched, and AGMR, that are not used to service users directly. For details on changing the run priority of these and other Domino jobs/tasks, see 6.5.1, “Changing the run priority of Domino tasks” on page 174. Chapter 6. Tuning the iSeries server for Lotus Domino 177 6.7.3 Working with Domino jobs The Work with Active Jobs (WRKACTJOB) command measures system performance by measuring aspects such as the CPU usage and response time. As mentioned before, it may be better to use option 9 from Work with Domino Servers (WRKDOMSVR) panel to consume less CPU when looking at the resource consumption of Domino jobs. To view the Work with Active Jobs display, type the WRKACTJOB command on any command line and press the Enter key. By pressing F21, more jobs appear on the display as shown in Figure 6-4. Pressing F11 cycles the display between the basic status, elapsed data, and information about threads as shown in Figure 6-4. Figure 6-4 Work with Active Jobs display Place the cursor on any column and press F16 to arrange the display in a descending sequence. For example, you can look at CPU%, response time, and disk I/O. Answer the question: “Are any Domino jobs consuming a large proportion of CPU?” Press the F10 key. Wait at least one minute, and press the F10 key again. This ensures that you have a good sample of data for the time period during which you are experiencing decreased performance. Move the cursor to the CPU% column. Then, press the F16 key to sort the list and move those with the highest CPU usage to the top. If any individual Domino job/task is consuming large amounts of CPU, you need to determine what application functions are causing the workload for this task. It may be that too many users are active at one time for the CPU resources in demand. You can also use the Work with Active Jobs (WRKACTJOB) command to change the run attributes of a job. The following steps show how to change the run priority for one job: CPU %: 97.8 Elapsed time: 00:24:01 Active jobs: 260 Opt Subsystem/Job User Number Type CPU % Threads DOMINO06 QSYS 514292 SBS .0 1 ADMINP QNOTES 514325 BCI .0 1 AMGR QNOTES 514321 BCI .0 1 AMGR QNOTES 514324 BCI .0 3 CALCONN QNOTES 514331 BCI .0 1 DRT QNOTES 516888 BCI .0 1 HTTP QNOTES 515570 BCI .0 48 EVENT QNOTES 514334 BCI 1.1 8 QNNINSTS QNOTES 514293 BCH .0 1 REPLICA QNOTES 514307 BCI .0 1 REPORT QNOTES 514335 BCI .0 1 ROUTER QNOTES 514310 BCI 4.2 4 SCHED QNOTES 514328 BCI .0 1 SERVER QNOTES 514300 BCI 15.9 257 SERVER QNOTES 514343 BCI 45.6 256 SERVER QNOTES 514344 BCI 22.9 256 UPDATE QNOTES 514318 BCI 1.3 1 ===> F21=Display instructions/keys 178 Domino for iSeries Sizing and Performance Tuning 1. Press the F11 key to see the job priority. 2. Enter option 2 to change the job run attributes. Enter the desired run priority in the RUNPTY parameter on the command line to change the priority for a job. Valid run priority entries range from 1 (highest priority) through 99 (lowest priority). Priority ranges 1 to 10 are reserved for system tasks. You should not change the run priority of Domino tasks to a priority above run priority 20 (run priority values 1 -19). This could cause unexpected problems of your Domino environment or other applications. It is much more typical to lower the run priority of an individual Domino job by changing the run priority value to something above 20 (a good range is 21 to 35). Remember that changing the run priority of the job will affect the run priorities of all threads within the job. 3. Press Enter for the change to take affect. 6.8 Online backup with BRMS Online backup of Domino databases and now incremental online backup on iSeries are implemented with the IBM licensed product Backup Recovery and Media Services for AS/400 (BRMS), available for OS/400 V4R4 and V4R5 as 5769-BR1 and for V5R1 as 5722-BR1. Writing individual Domino databases to physical devices would have unacceptable performance. Therefore, the databases are backed up in groups (5 is the default). You can change this group value by adding the following entry in the notes.ini file for the server (WRKDOMSVR for Domino or WRKLQPSVR for QuickPlace - option 13): SAVDOMBRM_FILES_IN_GROUP=x SAVLQPBRM_FILES_IN_GROUP=x Here x is the number of files to group in one BRMS package. As you increase this number, all databases of your Domino server are backed up more quickly.However, all databases in the group (that is, the x number) will be logged during the backup with the changes backed up separately. Because all the databases in the group are logged until all databases in the group are backed up, the time during which changes to the databases can occur is increased and the size of the changes backed up will increase. When the databases are recovered, the changes to the database that occurred during the backup will be re-applied. This process takes longer as the number of changes increases. If your server is being backed up during heavy server usage, you will want to keep this group value relatively small (3 to 7), so that fewer changes occur to the database during the backup operation. As a result, recovery of the database can occur in a reasonable amount of time. If your server is being backed up during off hours when server use is low, you can set the group value higher (10 to 20 or even higher) to speed up the backup operation, while keeping the recovery time reasonable. The maximum group value is 120. If you are saving your data to a TSM server, then you should not set your group value higher than 90. When to take a full backup A complete backup must be the start point of your recovery. The more you take, the less log files need to be replayed in the event of a DB loss and subsequent recovery. The more full backups you take, the slower your backup cycle is (in time taken to backup), and the more data you need to store. BRMS has a limit of not more than 98 online incremental backups of a Domino database before you must perform a full backup. We recommend that you perform an online full backup of the entire Domino data directory at least once a week. Chapter 6. Tuning the iSeries server for Lotus Domino 179 Server backups using BRMS/400 and how QNNINBRM works When you use the Configure Domino Server (CFGDOMSVR) or the Change Domino Server (CHGDOMSVR) command to add incremental backup support, you specify ADLSVR(*BRMS) on the command. This causes the configure (or change) support to add an entry to the ServerTasks= line of QNNINBRM. This QNNINBRM adding task queries if a transaction log is full, copies it, and then tells Domino, via the API, that it is now available for re-use. There isn't a specific interval it checks for. If it finds that a transaction log is available, it checks more often to see if a new one becomes available to be copied. If a log isn't available, it gradually lengthens the time between checks. So if the system is busy and filling transaction logs at a fast rate, it checks more frequently to see if a log is full to be copied. If it slows down and logs aren't being filled as quickly, then the checks gradually become longer between checks until a log becomes full. Then it shortens the time again. The work it performs is saving the changes that have been made to the databases that are stored in the TxL logs (just keeping the changes is the “incremental” part of incremental backup). Server backups using BRMS/400 and performance recommendations BRMS/400 copies each completed log file to a sub-directory until the process of copying the TxL files to tape or another backup medium is completed. The safest and fastest place for these BRMS backup copies of the logs would be on another system, preferably in another building. For example, if the BRMS backup sub-directory were an NFS mount point, the files would immediately be held at a safe location. It would in any case also be preferable if that sub-directory were not on the same ASP as the open TxL logs, for performance reasons, and because if the physical disk failed (and there was no journaling, RAID mirroring or SAN available), the logs together with their recent backup copies would both be lost. Losing the TxL disks will crash Domino so all buffered Notes data will be lost and therefore data loss on restart will be high after fixup, compact, and updall. Faster backups, reduced volumes, increased recoverability IBM has shown from their own use of transaction logging that they can cut backup volume by 90% – huge cost savings. You can also recover a database to any point in time with this method – not just the 24 hour snapshot most companies have now. Section 13.4.1, “Incremental online backup with BRMS” on page 384, offers some considerations regarding Domino transaction logging, which is necessary for BRMS online backup. For more information about Domino online backup with BRMS, refer to: http://ibm.com/servers/eserver/iseries/service/brms 6.9 Network tuning Since all users are connected to either a local area network (LAN) or wide area network (WAN) to your Domino servers, the performance of the network is also crucial to provide decent response times to your users. The main items that are of concern here are the maximum transmission unit (MTU) size for the line description, send and receive buffer size, port filtering, and duplex settings. For the recommendation for each of these areas, read on. 180 Domino for iSeries Sizing and Performance Tuning 6.9.1 Maximum Transmission Unit (MTU) Size The Maximum Transmission Unit (MTU) Size parameter affects the actual size of the line flows. By increasing the value of this parameter, you can reduce the overall number of transmissions, and therefore, increase the potential capacity of the CPU and the IOP (input/output processor). Similar parameters also exist on the client. The negotiated value is the minimum of the server and client (and perhaps any bridges/routers), so increase them all. The recommended setting varies depending on the communications protocol that is being used: 4 MB Token Ring = 4060 16 MB Token Ring = 16388 Ethernet 802.3 = 1492 Ethernet version 2 = 1500 To change the MTU size, complete these steps: 1. Type CFGTPC on any command line. You see the Configure TCP/IP display. 2. From the Configure TCP/IP display, select option 2 (Work with TCP/IP Routes) and press Enter. You see the Work with TCP/IP Routes display. 3. From the Work with TCP/IP Routes display, type 2 in the Opt column next to the IP address used by your Domino server. You see the Change TCP/IP Route display. 4. On the Change TCP/IP Route display, type in the recommendation from above in the Maximum transmission unit (MTU) parameter. 6.9.2 TCP/IP buffer size Web serving performance can be increased by tuning the buffer size that is used by TCP/IP, especially when sending large amounts of data. If your network is very reliable, try increasing the buffer size from the default (8000) to 64000. If your network experiences a significant amount of collisions or congestion, you may be able to improve performance by decreasing the TCP/IP send and receive buffers. This is because it will take less time to detect a bad packet, and less data will need to be retransmitted. To change the buffer size, follow these steps: 1. From a command line, type CFGTCP. 2. From the Configure TCP/IP display, select option 3 (Change TCP/IP Attributes). 3. On the Change TCP/IP Attributes display screen, locate the TCP receive buffer size (TCPRCVBUF) and type a new value. 4. Locate the send buffer size (TCPSNDBUF) and type a new value. 5. Press Enter. 6.9.3 Port filtering Filtering happens on every communications line, so you want to make sure that you have the proper hardware configuration to enable this support. It has happened in the past that filtering has not been setup correctly on a system and the result was a communications performance problem. The primary concern has been with the 2838 Ethernet card. If you have this card, check the part number and ensure that filtering is optimally configured. Chapter 6. Tuning the iSeries server for Lotus Domino 181 If the part number is 21H9067 on the 2838 Ethernet card, then an external filter is required. The external filter cable with part number 97H7385 is recommended. If the part number on the 2838 Ethernet card is 21H5458, or anything else, it has built in filtering, so you should not have an external filter since this double filtering can cause problems. 6.9.4 Duplex Ethernet supports both half and full duplex. The best performance will be with FULL duplex. However, the duplex setting on the line description must match the setting on the port on the switch if the line is hooked up to a switch. Be especially careful if you set your Ethernet switch or your line description to *AUTO. In many cases, we have found where performance is severely degraded because the duplex setting did not auto-negotiate correctly. It is probably best to configure switches and line descriptions to either *FULL or *HALF explicitly. If the line is connected directly to a 8271 Ethernet switch, the switch's port is FULL duplex. Therefore, the duplex setting on the Ethernet line description should be set to FULL. If the line is connected to a “stackable” hub, then the duplex setting on the line should be set to HALF to match the hub setting. 6.9.5 TCPONLY for Ethernet TCP/IP performance can be even further improved in V4R4 by setting the TCPONLY parameter to *YES on an Ethernet line description. This decreases the TCP/IP code path length by limiting the code that is loaded in the IOP for the line description. Ethernet was a focus area in improving communications performance in V4R4. To get the best TCP/IP performance, upgrade to V4R4 and take advantage of the improvements and set up your network to use full duplex. 182 Domino for iSeries Sizing and Performance Tuning © Copyright IBM Corp. 1999, 2002 183 Chapter 7. Tuning Lotus Domino for better performance on iSeries A Domino server is no more (or less!) than a complex application running in an environment of its own. To effectively and successfully tune such an application, you obviously have to know what the application is doing, what all the smaller parts of the application are doing, and how it all ties in together. For example, consider a case where you reduce the priority on what appears an insignificant batch-like thread. You could kill the performance or even the entire application, if, for example this was the common and necessarily single-threaded lock handler! So you must know what you are tuning if you want consistent and predictable results. This chapter describes what parameters and configuration options within the Lotus Domino environment can be changed to influence the performance of the server. See Chapter 6, “Tuning the iSeries server for Lotus Domino” on page 161, for the influence of OS/400 configuration parameters on the performance of Lotus Domino. This chapter provides details on:
Tuning Domino using the notes.ini parameters
Special considerations for Domino tasks, for example UPDATE, that often have a high impact on overall resource utilization 7 184 Domino for iSeries Sizing and Performance Tuning 7.1 Basics of Domino tuning The following sections emphasize the possibilities a Domino administrator has to optimize the workload that a Domino for iSeries server can handle. Tuning for best performance, however, starts with a good application design during the development phase. More information on application development performance for Domino can be found in the redbook Performance Considerations for Domino Applications, SG24-5602. The best way to improve Domino performance is to: 1. Plan ahead. 2. Test the Domino application thoroughly on a test system and then on a pilot system. 3. Start performance monitoring from day one before the first user signs on. 4. Constantly analyze your performance data and adjust your sizing based on actual facts. 5. Implement the application on the production system. 6. Roll it out gradually, a few users at a time, constantly monitoring its performance. 7. Continue to analyze the performance data and report future trends and hardware needs. It has been considered best industry practice in recent years to use a centralized server or workstation to centrally capture, collect, and report the statistics of a group of servers. If you have this type of “background” processing on one of your iSeries machines, always set the run priority and time slice values relatively low on that subsystem description. This keeps these background activities in the background and does not disturb the productive work of the system. 7.2 Relating Domino tasks to iSeries jobs A Domino subsystem contains a number of jobs. Each supports a different basic Domino task. The following list explains the OS/400 job names and their corresponding tasks on the Domino side: ADMINP The major Domino scheduler job running batch-like updates AMGR Domino Agent Manager; takes care of pre-defined agents CALCONN Connects the Domino calendar to other types of calendars EVENT Domino Event Handler QNNINSTS Monitors and manages automatic recovery (No corresponding task) REPLICA Domino database Replicator Task REPORT Domino Report Generator (adding data to STATREP.NSF) ROUTER Domino Mail Routing Task SCHED Domino Scheduler Task SERVER The main Domino Server Task UPDATE UPDATE task for Domino database indexes and views For a fully detailed description of these functions, see Chapter 8, “Understanding the Domino server jobs” on page 229. After you determine what is using the excessive CPU or any other system resource, the Domino administrator and OS/400 operator should work together to reduce or remove the problem. Remember: A very obvious but important lesson about server tasks and functions is that if your organizational requirements have no need of a particular function, do not start and run it on your server! Chapter 7. Tuning Lotus Domino for better performance on iSeries 185 7.3 Domino parameters that affect performance The major tuning tool for Domino is the notes.ini parameters found both in the notes.ini file and in the configuration documents in the Domino Directory (names.nsf formerly called the Name and Address book or Public Address Book). Many of these settings drastically affect Domino performance. They are sorted by function in tables starting with Appendix C, “Important notes.ini parameters” on page 435. For more detailed information on notes.ini parameters, see:
Notes Administration help (installed on your Domino server as file help5_admin.nsf)
Search Notes.Net Ask Professor INI: http://www.notes.net/today.nsf/profini?OpenView
The Prominic.net site has a useful lookup and descriptive list: http://domino-8.prominic.com/
Chapter 42 and Appendix E in Lotus Domino 5—Administering the Domino System, which comes in two volumes with your Lotus Domino server. 7.4 Domino Database indexing: Controlling the UPDATE task One of the most common, preventable causes of poor response time and performance of Domino servers is excessive and unnecessary activity of the Domino UPDATE task. This task is responsible for updating and rebuilding the indexes of Domino database views. It is designed to run in the background and is intended to improve response time and performance by ensuring that when a user opens a database view, the user does not have to wait for it to be indexed. Ironically, it often turns out to have the opposite effect. This section explains how the UPDATE task is supposed to work. It looks at how Domino administrators and designers can monitor and control it to improve server performance. It also explains the potential impact to performance when the UPDATE task is running excessively. It provides a hint of the kind of improvements you might see by correcting the problem. 7.4.1 Components of the Domino database indexer Lotus Technote #167013 provides a great deal of information regarding the Domino Indexer and the UPDATE task. This section includes a few excerpts that are relevant to the topics in this book. However, you should read the entire Technote if you want to understand what happens under the covers. You can find the Technote by searching for 167013 on the Lotus Support Web site at: http://www.support.lotus.com The Indexer is composed of three components:
The UPDATE task
The UPDALL task
The Notes Indexing Facility (NIF) Subsystem The Indexer, as a whole, is responsible for keeping the active views and the full text indexes current, within the databases. It processes any requests for changes to documents within a database, so that the active view collections and the full text indexes display the most recent modifications and current information. The UPDATE task UPDATE is a Domino server task that should run at all times (under normal conditions; this is explained later). You can load the UPDATE task in one of two ways: 186 Domino for iSeries Sizing and Performance Tuning
Automatically, by specifying it in the “ServerTasks=” line in the notes.ini file. This is the default when you configure a Domino server. Here's an example of the “ServerTasks=” line: ServerTasks=Replica,Router,Update,Stats,AMgr,AdminP,Sched,....
Manually, from the Domino console, using this command: load update The UPDATE task works continuously from a queue called $UpdateQueue. When a change occurs in a database (such as deletions, additions, or edits), a corresponding request is entered into the UPDATE queue. UPDATE checks the queue every 5 seconds for any new requests that have been deposited and takes the requests from the queue on a first-come, first-served basis. $UpdateQueue is a hard-coded queue that has a maximum capacity of 500 requests. These requests include updates to view indexes as well as full text indexes. When a request is placed into the queue, the actual names of the databases and paths to the databases are stored in the queue, not the view names themselves. No published method is available to display the actual contents of the $Update queue. To see if UPDATE is running and what it is doing, you can use the show tasks console command. From the Domino console, it is displayed as the Indexer task as shown in Figure 7-1. Figure 7-1 The Indexer task at the Domino console However, when you use OS/400 functions (for example, the Work with Active Jobs (WRKACTJOB) command as shown in Figure 7-2 or using the Operations Navigator), it appears as an OS/400 job called UPDATE. Work with Domino Console Server: FMS170 Previous subcommands and messages: Statistic Collector Idle DECS Server Idle Event Monitor Idle Calendar Connector Idle Schedule Manager Idle Admin Process Idle Agent Manager Executive '1': Idle Agent Manager Idle Indexer Full text indexing documents in m_dir/MSS_TS_C.nsf Router Idle Replicator Idle Billing Idle QNNINBRM BRMS Log Manager Idle Enter a Domino subcommand. ===> F3=Exit F5=Refresh F6=Print F9=Retrieve F17=Top F18=Bottom F21=Command line Chapter 7. Tuning Lotus Domino for better performance on iSeries 187 Figure 7-2 The Indexer task on the Work with Active Jobs panel The UPDALL task The UPDALL task is a single instance of the UPDATE task. It does not operate off a queue like UPDATE, but accomplishes the same purpose in a different way. When the UPDALL task runs, it processes every Notes database, refreshing the views or updating the full text indexes for each database. The UPDALL task starts in one of two ways:
Automatically, at a scheduled time each day. By default, the UPDALL task is run at 2:00 AM. It is defined in the ServerTasksAt2 line in the notes.ini file as follows: ServerTasksAt2=Updall
Manually, from the Domino console. When you run the UPDALL task from the console command line, you can use various switches to control it. For example, you can invoke UPDALL to rebuild only the full text index on a single database. The UPDALL task performs additional functions beyond what the UPDATE task does. In the design of a view, you can specify, in View Properties, the frequency with which a view index is discarded:
Never
After each use
If inactive for a specified number of days It is the responsibility of the UPDALL task to remove the view index if a discard option is specified. In other words, if the Discard Index option is selected for a view, the index is not actually discarded immediately. Rather, the index is removed the next time the UPDALL tasks runs. For example, a view has a Discard View option of “After each use”. If you exit from a view at 1:00 p.m., the view will not be removed until UPDALL is run (usually done at 2:00 a.m.). Notes Indexing Facility The Notes Indexing Facility (NIF) is made up of a multitude of functions within Domino that allow a server to keep data notes ordered and current within a view. Specifically, the NIF performs these tasks: Work with Active Jobs FRA820 08.11.01 17:37:28 CPU %: 4,7 Elapsed time: 00:30:55 Active jobs: 209 Type options, press Enter. 2=Change 3=Hold 4=End 5=Work with 6=Release 7=Display message 8=Work with spooled files 13=Disconnect ... Opt Subsystem/Job User Type CPU % Function Status UPDATE QNOTES BCI 0,0 PGM-UPDATE SELW Bottom Parameters or command ===> F3=Exit F5=Refresh F7=Find F10=Restart statistics F11=Display elapsed data F12=Cancel F23=More options F24=More keys 188 Domino for iSeries Sizing and Performance Tuning
Updates Indexes
Opens and closes view collections/view indexes
Locates Index entries The majority of these requests are made by the server when users open and close databases. For example, the NIF subsystem – not the UPDATE task – forces the update of a view collection when a user switches between views. If a user makes a modification to a document within a view, and then switches quickly to another view and then back again to the original view, the new changes should be seen almost immediately by the user. The rest of this section focuses on the UPDATE task. 7.4.2 UPDATE task CPU utilization in a normal environment On a typical Domino server (one that is not experiencing performance problems due to the UPDATE task), UPDATE may run quite often, but usually for very short periods of time. Unless a database has just had several hundred or thousand documents added or changed, UPDATE does its work quickly, updates each view in the database it is indexing, and finishes in a few seconds. With small databases, even on older, slower hardware, doing an incremental update of a view, in which only a few or no documents have changed, takes less than a second. If you observe CPU utilization when this is taking place, you may see the utilization rise to 90% or more (on a single CPU), but only for a few seconds while UPDATE traverses all the views in a database. With very large databases (as long as the number of new or changed documents is small), the time to do an incremental update of a view still remains short, usually only a second or maybe a few seconds. In any case, with small or large databases, or fast or slow servers, the time it takes to do an incremental update of the views in a given database will be similar for all views in that database as long as the number of new or changed documents is small. This activity (incremental updates of view indexes) does not usually cause much of a performance problem for servers. Although the utilization is high, it is brief. Its effect is hardly noticeable. Also, although views are also “locked” and, therefore, inaccessible when they are being updated. No one usually notices because it normally takes, at most, a few seconds. You should also be aware that the time to do an incremental update of a view, in which few or no documents have changed, depends more on the number of documents in the database than the number of documents in the view. This may sound contrary to what you have read, but it is not. When the number of documents to be indexed is low or zero, the time to update is really mostly just overhead and depends on the database size. However, if you consider the time it takes to completely build a view, then it is the number of documents in the view that matters more than the actual number in the database. 7.4.3 The UPDATE task and CPU utilization in a problem environment Now, if you consider a server in which the UPDATE task is observed to be running almost all the time and in which the CPU utilization is consistently at or near 100%, usually something else is going on. This does not necessarily mean, the UPDATE task permanently consumes over 90% of all processor resources. There can also be cases where the overall utilization approaches 100% and the UPDATE task accounts for the major part of it. Chapter 7. Tuning Lotus Domino for better performance on iSeries 189 A high total CPU utilization does not necessarily cause problems on an iSeries server. However, if high priority jobs, such as Domino tasks or interactive (5250) jobs consume a major part of resources (including disk utilization), you can see the following results:
All jobs with same priority may see longer response times.
Jobs with lower priority, for example traditional batch jobs with lower priority (that is, the priority value is higher than 20), will drastically slow down or even stop working. Several reasons may cause the UPDATE task to use up too many processing cycles or even monopolize the CPU:
The total number of views in databases with many documents, to be maintained by the UPDATE task, is extremely high.
Full text indexes are created frequently.
Many views (for big databases) have to be completely rebuilt from scratch.
Many views contain time-dependent selection formulas. Too many views or full text indexes Since views and the capability of full text search represent the full power of many Domino applications, it is in most cases not an option to reduce any of them. If these two reasons turn out to be the cause of excessive resource consumption, probably the only choice is to invest in more and faster hardware. Views have to be completely rebuilt from scratch When the UPDATE task is called upon to completely rebuild a view from scratch, it is also very CPU-intensive. So far, this is the bad news. The good news is that this should normally not happen. UPDATE should almost never need to completely rebuild a view. Most of the time, this is redundant and a waste of CPU resources. The only circumstances in which UPDATE should rebuild views from scratch are:
All the documents in the view or database have changed or are new. This might occur because: – An agent or external program has updated them. – A Lotus Enterprise Integrator (LEI) Direct Transfer Activity deletes and refreshes all the documents in a database. This should be avoided whenever possible. Sometimes other techniques or tools can be used, such as using LEI Replication Activities, which have a smaller impact.
The database is new. If a new replica is created on a server, and some of the views have the proper settings, UPDATE builds the indexes for those views.
The design of the view has changed. If the view has the proper settings and previously existed, changing the design causes the whole view index to be rebuilt. As you might surmise, the circumstances in the second and third bullets should not be common occurrences. Time dependent views If the examples above are not the cause of views having to be rebuilt, then the likely cause is the existence of one or more views with time-dependent selection formulas in which the View Refresh Index options specify that the views should be automatically updated. The UPDATE task has no business rebuilding these kinds of views. Why? Because they are always out of date. 190 Domino for iSeries Sizing and Performance Tuning Whenever this kind of view is opened by a user, the view index is built from scratch, on the fly, for that user by the Notes Indexing Facility. It doesn't matter if UPDATE built it 5 minutes ago. It will be rebuilt again. And if another user opens it, it will be rebuilt again. Since it is always rebuilt when opened by a user (or by an application, for that matter), it makes no sense for UPDATE to rebuild it too. The wisdom of having views with time-dependent functions in their selection formulas is not discussed here, except to say that there are legitimate reasons to do this and that it is feasible, if managed correctly. The following section explains how. 7.4.4 Affecting the UPDATE task behavior through the view properties Before solutions to the CPU utilization problem are discussed, here is a review of the rules that the UPDATE task follows. The UPDATE task will update or rebuild a view index if and only if either the following are true:
The Refresh Index option for a view is set to “Automatic”
The view exists already You can use combinations of the Refresh Index and Discard Index options within the view properties (see Figure 7-10 on page 197) to control whether UPDATE rebuilds a view. For example, consider a view that is used rarely, say once a month, which displays only those records created in the last 30 days. You might specify the Refresh option as “Auto, after first use” so that the user always sees the most recent information, yet specify the Discard option as “After each use”. Therefore, according to the second rule above, UPDATE always ignores the view. You also save the disk space that the view index might otherwise have used. The solution to the CPU Utilization problem is to use these View Refresh Index and Discard Index Options properly. Setting the view properties to improve performance Here are some recommendations for how you might set these index options in the view properties, depending on what you want users to see and how often the view is used. Any view that has a view selection formula with a time-dependent function in it Time-dependent functions, such as @Now or @Today, are sometimes used in selection formulas for views for example to show all documents that have been created today or that are older than a certain number of days. For an example, see Figure 7-9 on page 197. These kind of views should have its Refresh Index option set to either:
Auto, after first use
Manual
Auto, at most every.... It should never be set to “Automatic.” This controls how often the view is updated by users (that is through the NIF; see “Notes Indexing Facility” on page 187) who open the database. However, it does not control how the UPDATE task handles it. It is important to understand that even though it makes no sense for UPDATE to rebuild a view that has a time-dependent selection formula, if the view exists, UPDATE will rebuild it. A suggestion to change this behavior has been submitted to development for consideration in the next major release of Domino. Chapter 7. Tuning Lotus Domino for better performance on iSeries 191 The index of a view of this type is always out of date. The setting of “Automatic” tells UPDATE to always rebuild the view, even if it does not exist. This setting would defeat the purpose of the Discard Index option, and since the view index will be obsolete as soon as UPDATE is finished, this should never be done. It is a waste of CPU. Use Auto, after first use if you want users to always see the latest information in the view and if the contents of the view might change between the times that UPDALL or UPDATE is run, in other words, nightly, or if all changes come as the result of replication. (UPDATE will run after a replication.) Select Auto, at most every nn hours if the contents of the view changes more frequently than nightly and if you need users to see the most recent data very soon after it changes. Select Manual when users do not need to see the most recent changes unless they specifically ask for it. In reality, UPDATE or UPDALL make sure that the view is no more stale than one day, assuming you run UPDALL on the server nightly. Time-dependent function and index refresh option set to ‘Auto, after first use’ Any view that has a view selection formula with a time-dependent function in it, such as @Now or @Today, and having its index refresh option set to “Auto, after first use”, should have its discard index option set to After each use (see the example in Figure 7-11 on page 198). This is necessary to ensure that the UPDATE tasks sees no index on the view and ignores it. Time-dependent function and index refresh option set to ‘Manual’ Any view that has a view selection formula with a time-dependent function in it, such as @Now or @Today, and having its index refresh option set to “Manual”, should have the discard index option set to Never. The refresh option of “Manual” allows the view index to be reused by other users. The view index is refreshed manually by pressing Shift+F9. This view always opens very quickly. However the user who presses Shift+F9 to refresh the view may have a long wait as it is being updated. Optionally, the discard option may be set to “If inactive for xx days”. But this would only be done if there was a need to conserve disk space used by the view. Seldom or never used views Views that are seldom or never used can have their discard index option set to “If inactive for xx days if there is a need to conserve disk space”. This can also shorten the time it takes UPDATE to refresh views, but the benefit depends on how many changed documents there are in the database. For databases with lots of documents (or all of them) changing every day, this setting is useful for controlling the indexes on views that aren't used. If used, the refresh index option should be either “Manual” or “Auto, after first use”. Setting it to “Automatic” negates the affect, since UPDATE always rebuilds the view (it will never be discarded). When should you use the ‘Automatic’ options? The refresh index option of “Automatic” should only be used on views that:
Do not have time-dependent selection formulas
Are frequently used
Have a Discard Index option of “Never” Generally, “Auto, after first use” is a better choice. The difference between these two settings would only be noticed: 192 Domino for iSeries Sizing and Performance Tuning
The first time the view is used
The first time it is used after being copied using the Notes client (File Database Copy)
After having all view indexes discarded on purpose using UPDALL (unlikely except perhaps when troubleshooting a bug) Now that you understand the problem and what to do about it, you might wonder if your server has this problem and how you might determine which views and databases, if any, might be the cause. 7.4.5 Detecting UPDATE task problems You might wonder if there is something that might be a tip-off that a Domino server is suffering from the problem described above. CPU utilization statistics alone may not be enough of an indicator. The real tip-off is an indication that the UPDATE task is running at high CPU utilization rates continuously. On OS/400, you can use Management Central (a component of Operations Navigator) or the Work with System Activity (WRKSYSACT)1 command to monitor utilization by job or by thread. If you set the refresh rate to around 15 seconds, and watch for a while, you can usually tell if UPDATE is the cause. If UPDATE is using 60 to 70% of a single CPU and the total CPU utilization is at 95 to 100% for more than a minute straight, that's a tip-off. If it only runs that high for a few seconds and then drops and rises again, it is not so likely to be the result of unnecessary view index rebuilds. If you want to know for sure, you need to analyze your Domino server's use of UPDATE. Analyzing your Domino server Determining whether your Domino servers have any databases with views that are being rebuilt unnecessarily is actually rather easy. You need access to the Domino Server log database (log.nsf) and add one statement to the notes.ini file on each server you want to monitor. The basic process is summarized below. Each step is described in more detail following the summary. 1. Add the line LOG_UPDATE=2 to notes.ini. 2. Restart the UPDATE task. 3. Collect data. 4. Filter the log file (using the Domino Administrator’s Analyze feature) to capture only the entries related to the UPDATE task. 5. Look for views in any database in which the time to update the view is considerably longer than to update most of the other views in the same database. 6. Examine the design and index options of the views you identify in step 5. Let's take it one step at a time: 1 The Work with System Activity (WRKSYSACT) command exists only on your iSeries server if the Performance Tools (5722-PT1 or 5769-PT1) are installed. As an alternative, the Work with Active Jobs (WRKACTJOB) can be used, but it has several disadvantages. Note: Another way to approach this is to look at the Database Sizes view in the log and look at the view sizes of the largest databases. The largest views in the largest databases are good candidates for being the culprit when UPDATE is running wild. But this process is trial and error, and you still have to examine the design of the suspected views. Chapter 7. Tuning Lotus Domino for better performance on iSeries 193 1. Add the line LOG_UPDATE=2 to notes.ini. This entry causes the server to log information whenever the UPDATE task updates views on the server. It records a time stamp, the database name, and the view name as shown in Figure 7-3. Figure 7-3 Miscellaneous events view in LOG.NSF The server also logs information when UPDALL runs. When you enable this logging on a server, it generates more information in the log, but the additional logging should not have a noticeable affect on server performance. It is safe to use on a production server. It is documented in the Domino 5 Administration Database (help/help5_admin.nsf) under Reference -> NOTES.INI Settings -> Log_Update. 2. Restart the UPDATE task. To restart the UPDATE task, issue the following command from the Domino Console: TELL UPDATE QUIT Wait for notification that it has ended, as shown in Figure 7-4. Figure 7-4 Ending the UPDATE task 3. When the UPDATE task has ended, issue the following command: LOAD UPDATE Normally, this does not affect any users because UPDATE runs in the background. The only possible side effect might be that someone encounters a long delay opening a view for the first time if that person is the first one to do so. But this would only happen if the database was new or all the documents were just added or changed. If this is a possibility, then you should schedule the restart off-shift. 194 Domino for iSeries Sizing and Performance Tuning 4. Collect data. It is best to collect a few hours worth of data during the busiest time of the day (or night), whenever you are experiencing the worst response times and highest CPU utilization. It is even better if you can be sure, ahead of time, that UPDATE (and not some other task or job) is responsible for the high utilization. With OS/400, you can do this easily using Management Central's performance monitor. The platform dependent statistics (see 5.1.4, “Platform-dependent statistics” on page 137) may also be very helpful. When you have collected enough data, you can remove the entry from the notes.ini file and restart the UPDATE task. 5. Filter the log. Use the Domino Administrator to select the log output you want to examine. From the Domino Administrator window, select the Server tab and then the Analysis tab as shown in Figure 7-5. Then click the Analyze button and select Log.... Figure 7-5 Analyzing LOG.NSF with the Domino Administrator 6. After you click Log..., you see the Server Log Analysis dialog window as shown in Figure 7-6. Enter the text Updating and set the other options as appropriate. As a result of Chapter 7. Tuning Lotus Domino for better performance on iSeries 195 the analysis, a new database can be created on your client or the Domino server, or the logging entries may be added to an existing database. Figure 7-6 Server Log Analysis dialog 7. After the analysis has been performed, the database containing the results opens automatically show a view similar to the example in Figure 7-7. Figure 7-7 Log Analysis results 8. Look for problem views. 196 Domino for iSeries Sizing and Performance Tuning A “problem view” meets this criterion: the time to update the view is more than twice the time to update most other views in the same database. With the information that you have in analysis results (Figure 7-7), you can calculate the time it takes to update each view by looking at the time stamps of the entry for that view and the subsequent entry. In the example shown in Figure 7-8, the “All Documents” view was updated beginning at 9:10:06 and ending at 9:10:07, taking about 1 second. In fact, most of the views took about 1 or 2 seconds to be updated. However, the “Time Log\Manager Reports\Not Submitted” view started at 9:10:26 and took more than 14 minutes to be updated. This is a clear indication of a problem. First of all, if all the documents in the database were new or changed, most or all of the other views would also take a long time to rebuild. You need to examine this view and the “Time Log\Payroll\HR No Authority” view (more than six minutes to be updated). Figure 7-8 Log analysis example 9. Examine the design. Now that you have identified the views that are causing UPDATE to run excessively, you should examine the design of the views to determine if the view has a time-sensitive function in its selection formula and to see what the index refresh and discard options are. You need designer access to the databases to do this. 7.4.6 Making the changes To illustrate how to make the changes, we assume that the view in the example above is the one we are going to examine. Follow these steps: 1. Using Domino Designer, open the database and view you want to check. Select the View Selection object as shown in Figure 7-9 and look at the formula. In this case, it contains the function @Today. This means that you need to change the Discard index view option. Chapter 7. Tuning Lotus Domino for better performance on iSeries 197 Figure 7-9 Example for a time-dependent selection formula 2. Select Design-> View Properties. Click the Advanced tab in the Properties box. You then see a display like the example in Figure 7-10. Figure 7-10 Default settings of the View properties 198 Domino for iSeries Sizing and Performance Tuning 3. If the Refresh index option is set to “Auto, after first use”, and the Discard index is set to “Never”, change it to After each use as shown in Figure 7-11. Figure 7-11 View Properties changed to ‘Discard index after each use’ Once you make the change, save the updated view. 4. To make the change take effect, the current view index needs to be discarded. You can wait for the nightly run of UPDALL to do this or you can do it manually by running the UPDALL command against the database from the Domino console2. Refer to the Domino 5 Administrators Help Database for details. Once that is complete, if you want to test the results, you can simply make some changes to the database and then close the database. You may have to wait several minutes for UPDATE to run. If you want to speed up the test, there is a notes.ini parameter you can use to control how UPDATE runs. Normally, although UPDATE looks for database changes every 5 seconds, it does not act on them for about 5 minutes. This is called the suppression time. UPDATE does this to wait for more updates and to “batch them up”. On a busy server, this reduces the frequency in which UPDATE runs. You can change the default from 5 minutes to 1 minute by adding Update_Suppression_Time=1 to the notes.ini file. Note: If the database you are changing is linked to a design template, the design change might be overwritten by the Designer Domino task each night. There may be other considerations as well. We keep it simple here to illustrate what needs to be changed. 2 You may also use the Submit Domino Command (SBMDOMCMD) or Run Domino Command (RUNDOMCMD) commands. Note: This should be a temporary change only. Chapter 7. Tuning Lotus Domino for better performance on iSeries 199 7.4.7 Conclusions The only way to predict what benefit you might achieve from making the changes described in this chapter is to actually measure the amount of time UPDATE is running and determine if there are any views that could be reconfigured. There have been some instances where CPU utilization has dropped as much as 45%. In others, the average CPU utilization was not affected much, but spikes in response time when UPDATE was rebuilding a view were greatly reduced. 7.5 TCP/IP between Domino servers on the same iSeries In previous releases of OS/400 and Lotus Domino, if you had multiple Domino servers (that is partitions) running within the same physical system, we recommended that you use the loopback interface (LOOPBACK_TcpIPAddress). This gives you more efficient TCP/IP communications between multiple servers on a single physical system. Starting with OS/400 V4R4, the TCP stack now redirects all communications to the loopback interface if the destination address is on the local system. However, to allow this new function to work properly, you should now remove all references to the LOOPBACK interface in the Domino server by following these steps: 1. Remove the PORT definition in the Server document. 2. Remove any Connection documents that reference the Loopback port, or modify the documents to reference a valid port such as TCP/IP. 3. Remove any reference to the loopback port from notes.ini (this should have been removed automatically after performing step 1). 7.6 GTR search engine version 3.4 Domino R5.03 now includes an alternate Global Text Retrieval (GTR) search engine version 3.4. GTR is a set of functions which enables Domino to perform high performance full text search to a large set of documents. High performance full text search is realized by searching an index file which is created from source documents in advance. 7.7 Using the Extended Directory Catalog The Extended Directory Catalog is a new type of directory catalog available in R5.0.5. For additional information on this feature and further details on setting it up, see the Release Notes, particularly the “Extended Directory Catalog” section in the “Things you need to know” section. Some early deployment work at IBM Research with an IBM Worldwide Extended DirCat (EDC) indicates substantial (approximately 20%) CPU reduction that would also improve overall server performance (including the router). 7.7.1 Comparing EDC to a standard directory catalog (DirCat) The improvement is relative to using a standard directory catalog, as the router can bog down in Full Text (FT) searches. The standard directory catalog is designed to be small, so it builds only one sorted view by last name or distinguished name at the Administrator's option, and uses FT search to find names that aren't matched in the view. For example, if the DirCat is sorted by last name, “Fred Bloggs/Accounts/MegaCorp" won't show up in the $Users view, 200 Domino for iSeries Sizing and Performance Tuning but FT search will find it. On the client, the difference between delay when finding a value in a sorted view and when searching for it in a full-text index is trivial, partly because we can't send messages that fast! Multiplied by thousands of recipients on a server, the delay becomes a bigger problem. The Extended Directory Catalog (EDC), which is designed to be fast instead of small, addresses this by aggregating all the source directories into a single superdirectory with standard views. The Router must still search the primary directory before searching any secondary directory, including an EDC. As with directory assistance, there's no necessity, and anyway you must not include the primary directory in the EDC. 7.7.2 Using Extended Directory Catalog to improve performance Using an Extended Directory Catalog on a server is particularly recommended if using the standard Server Directory Catalog (created from the DIRCAT50.NTF template) has been determined to be causing Mail Router backups. When the Mail Router uses the standard Server Directory Catalog for mail routing lookups, it uses full-text (FT) searching, a slower process than using views, to look up names that don't correspond to the “Sort by” directory catalog configuration selection, as detailed above. 7.8 Investigating mail backlogs and performance issues If mail routing or performance backlog issues develop, possibly as a result of using a standard or GTR Server Directory Catalog, try following these steps to troubleshoot the problem and implement the EDC: 1. To determine if the standard Server Directory Catalog is the cause of the problem, remove its file name from the Server document or from the Public directory profile. 2. If the standard Server Directory Catalog is the cause of the problem and the “Sort by” configuration selection is not Distinguished Name, change the selection to Distinguished Name and then re-enable the Directory Catalog to see if the change resolves the problem. 3. If using Distinguished Name as the “Sort by” selection does not solve the problem, use the Extended Directory Catalog instead, so that the Router uses view lookups rather than full-text searches to look up names. Watch for agents running on mail files that do not have a full text index on the mail files. searching through each new mail item (for example moving junk mail to a waste bin). If the mail file is not full text indexed, a temporary full index will be built for each new mail items that arrives. See “View_Rebuild_Dir” on page 222. 7.9 Domino console logging One concern with the Domino Console Log (buffer), when it comes to performance, is the actual size of the Domino console log. On the iSeries server, we recommend that the size of the Domino console log does not exceed 10,000 entries (the default). A Domino console log with more than 10,000 entries can result in poor response times for the Display Domino Console (DSPDOMCSL) and Work With Domino Console (WRKDOMCSL) commands. The size is set in the iSeries environment variable Notes_AS400_CONSOLE_ENTRIES. The Add Environment Variable (ADDENVVAR) and Change Environment Variable (CHGENVVAR) iSeries server commands are used to maintain the size of the Domino console log. You have to perform the ADDENVVAR or CHGENVVAR command in the job that starts the Domino server before it is actually started (unless you use system-level environment variable). Chapter 7. Tuning Lotus Domino for better performance on iSeries 201 For more information about iSeries environment variables, see 7.16, “iSeries environment variable settings” on page 224. 7.10 Number of users per Domino server What is the maximum number of users supported on a Domino for iSeries server? This is a question we often heard since publishing a guideline3 related to Domino for AS/400 R4.6 in the first edition of this book. There is no straight answer except for saying: “No limitation, neither for registered users nor for concurrently active users exists.” Was the recommendation wrong? No, it was not. With any version of Domino, there is a certain maximum workload that can be handled efficiently by each Domino server. With Domino R5, the implementation has been changed, so each server can support a much higher workload than it could with previous releases. Assume you observe the Domino response times while increasing workload on a powerful iSeries server. There is certain point, where hardware and operating system may still be able to support higher workload, but the response times begin to increase exceptionally because of contention within the Domino server. This makes it desirable for many administrators to limit the workload and add partitioned servers, when there are still hardware resources available. However, the trade-offs in running multiple partitions are different on different platforms. On other platforms, there is a reluctance to run more than one, or at most two to four partitions on production servers for reliability and manageability reasons. On iSeries, because of the way Domino takes advantage of OS/400 architecture, there is no such limitation. Rather, it is viewed as an advantage. By dividing large numbers of Domino users into smaller Domino partitions, higher reliability is achieved and greater overall performance and capacity are realized. Now, how do you limit the Domino workload? In a mail environment in a single organization, you may make two assumptions:
The workload produced by each single user normally does not deviate extremely from the average.
The percentage of concurrently active users may vary significantly for different times of the day, but not so much for a (randomly chosen) subset of the total number of users. Keeping the above in mind, the workload generated is proportional to the number of registered mail users. Therefore, in a mail environment, for a relatively homogenous organization, you may limit the workload by simply limiting the number mail databases, in other words the number of registered users per server. IBM Rochester, currently supports more than 10,000 Notes users on three iSeries servers and already started doing that in the early days based on a beta version of Domino 4.5. The recommendation for R4.6 mentioned above was a result of the experiences at IBM Rochester. After they migrated to Domino R5, they were able to raise the number of active mail users per Domino partition much higher. Therefore, the Workload Estimator now recommends that you distribute the workload over multiple Domino partitions, if you have more than 3000 concurrently active users (see 3.4.2, “Number of partitions” on page 26). Keep in mind though, the optimum number may vary in both directions, depending on the working habits of your users and the complexity of your applications. 3 At the time, we recommended 700 registered or 500 concurrently active users. 202 Domino for iSeries Sizing and Performance Tuning 7.11 Using a separate partition for the R5 SMTP MTA server The R5 SMTP MTA process can be considered to be a pure batch task working on inbound and outbound SMTP mail streams. Outbound it takes Notes mail, converting it as appropriate to MIME, and transforming it into an SMTP outbound stream to send to the SMTP relay or firewall. On the inbound side, it receives SMTP and MIME data streams from totally unknown origins anywhere on the Internet, and then attempts to parse it and convert it into Notes mail. In most major organizations and enterprises, we recommend that the SMTP MTA is run in a separate partition (or on a separate server) to avoid it interfering with the Notes mail servers. Processing SMTP MIME e-mail, large attachments, anti-virus checking, and anti-spam filtering can severely slow processing for “real” interactive users and jobs. Also, this type of “alien file and data” processing, is a high risk activity, much more prone to crashing and errors than the normal activity of a mail server because of the parsing and mixed file handling of SMTP data generated, basically, anywhere on the planet. If you do have a need to keep the R5 SMTP MTA on a mail server, we recommend that you change the job run-time priority and time slice. See 6.5, “Choosing which processor priority to use” on page 173, and for detailed information, see Chapter 11, “Internet and intranet performance tips” on page 301. 7.12 Excessive translations on iSeries and zSeries On the iSeries and zSeries servers, the whole operating system is based on EBCDIC, unlike the pSeries and xSeries servers, which are based on ASCII. Domino is based on Lotus Multi-Byte Character Set (LMBCS), a superset of ASCII, which works reasonably well translating with ASCII but sometimes is not quite as fast with EBCDIC. 7.12.1 Opening up any suspect job using the new Domino panel group Make sure to sign on to the iSeries with a user profile having *ALLOBJ and *JOBCTL special authorities and follow the instructions given in “Display Call Stack: Panel group update to improve readability” on page 131. This display shows whether you are spending a lot of time doing (excessive number of) “string translates” (StringXlate). Figure 7-12 shows a sample call stack with the string translate clearly shown in the sequence of procedures: “LCStreamConvert StreamTranslate StringXlate“ Chapter 7. Tuning Lotus Domino for better performance on iSeries 203 Figure 7-12 Sample call stack viewed with the new Domino panel group Note also “pthread_mutex_lock” and “lockSlowWithInitAllowSig__9Qp0wMutexFv”, which are where the delays and performance problems show up. If you determine from the call stack that Domino is spending a lot of time doing a high number of string translates, isolate the active databases (.NSFs) and template files (.NTFs) involved and display the attributes of the databases. Display Call Stack System: AS06 Job: SERVER User: QNOTES Number: 017486 Thread: 00000005 ILE ILE Procedure Statement Module program LE_Create_Thread2__FP12crtth_parm_t 0000000826 QLECRTTH QLESPI pthread_create_part2 0000001004 QP0WPTHR QP0WPTHR ThreadWrapper 0000000008 THREAD LIBNOTES Scheduler 0000000032 SCHED SERVER DbServer 0000000224 DBSERVER SERVER ServerNoteOpen 0000000029 SVNTOPEN SERVER NSFNoteOpenExtended 0000000044 NSFSEM3 LIBNOTES EMAfter 0000000011 EXTMGR LIBNOTES EMDispatch 0000000025 EXTMGR LIBNOTES NoteOpenHandler 0000000050 LNPEXT LIBDECSEXT FetchDocument 0000000006 LNPEXT LIBDECSEXT ExecProcedure 0000000026 LNPEXT LIBDECSEXT LCConnectionCall 0000000001 STATE LIBLCAPI retry_LCConnectionCall 0000000039 STATE LIBLCAPI LCXCall 0000000059 DB2 DB2 GetMetadata 0000000028 DB2 DB2 StoreData 0000000135 DB2 DB2 LCStreamConvert 0000000018 STREAM LIBLCAPI StreamTranslate 0000000058 STREAM LIBLCAPI StringXlate 0000000002 STREAM LIBLCAPI LCProcCritSecWait 0000000003 MLTPROC LIBLCAPI pthread_mutex_lock 0000000900 QP0WPMUT QP0WPTHR lockSlowWithInitAllowSig__9Qp0wMutexFv 0000000104 QP0WMTX QP0WMTX Bottom F3=Exit F10=Update stack F11=Display instruction F12=Cancel F16=Job menu F17=Top F18=Bottom F22=Display entire name 204 Domino for iSeries Sizing and Performance Tuning Figure 7-13 shows the alog.ntf attributes. Figure 7-13 Display Attributes of a Notes file alog4.ntf showing a CCSID of 819 If the database does not have the same Coded Character Set ID (CCSID) as your local iSeries, that is your local country or region’s character set (in the USA that would be 37), it may need switching. In the above example, the CCSID is set to 819 (that is the ASCII CCSID). This may be the cause of some of your high number of translates. Unnecessary translation efforts will cause performance bottlenecks. Shut down Domino, use the WRKLNK option 3 (Copy) on the database (.nsf or .ntf) in question, and alter the CCSID. Figure 7-14 shows the prompt of the Copy (CPY) command preparing to change the CCSID of the file to 37. Display Attributes Object . . . . . . : /domino/applsvr/alog4.ntf Type . . . . . . . . . . . . . . . . . : STMF Owner . . . . . . . . . . . . . . . . : QNOTES System object is on . . . . . . . . . : Local Auxiliary storage pool . . . . . . . . : 1 Object overflowed . . . . . . . . . : No Coded character set ID . . . . . . . . : 819 Hidden file . . . . . . . . . . . . . : No PC system file . . . . . . . . . . . . : No Read only . . . . . . . . . . . . . . : No Need to archive (PC) . . . . . . . . . : Yes Need to archive (AS/400) . . . . . . . : Yes More... Press Enter to continue. F3=Exit F12=Cancel F22=Display entire field Chapter 7. Tuning Lotus Domino for better performance on iSeries 205 Figure 7-14 Copy Object command preparing to switch from a CCSID of 819 to 37 Once you have checked to see that all your databases are now correctly set to the right CCSID, if you still have high string translate counts, start looking at the application coding itself, isolate which agents are causing the issue, and then focus in on the actual code. Is there a high use of simple STRINGs in the code rather than numeric, date, and binary fields as appropriate? Perhaps it is now time to review your coding standards and modify the code based on best industry practice rather than convenience. 7.13 Optimizing mail: White space, compact, and quotas This section looks at the two main compact styles, whether to recover white space, use quotas, and why you should have a mail archival policy. Using COMPACT -b to maintain white space and reach steady state Here are the latest recommendations on the issue of recovering the white space from Notes databases, particularly mail files to stop the server running out of space each week. Do not recover space with COMPACT We do not believe that our customers are actually doing the correct thing, in running COMPACT with white space deletion (COMPACT -B or -C). With TxL enabled, our very strong recommendation is to not remove white space for various reasons, not the least being the fact that the Database Instance ID (DBIID) will be changed forcing a full backup to be run immediately, and that the database must be shut down and offline to compact copy style. (For Copy Object (CPY) Type choices, press Enter. Object . . . . . . . . . . . . . mynotes.nsf To directory . . . . . . . . . . '.' To object . . . . . . . . . . . mynotes.nsf Symbolic link . . . . . . . . . *NO *NO, *YES From CCSID . . . . . . . . . . . 819 1-65533, *OBJ, *PCASCII... To CCSID . . . . . . . . . . . . 37 1-65533, *OBJ, *CALC... Data Format . . . . . . . . . . *BINARY *BINARY, *TEXT Directory subtree . . . . . . . *NODIR *NODIR, *NONE, *ALL Replace object . . . . . . . . . *YES *NO, *YES Owner . . . . . . . . . . . . . *KEEP *NEW, *KEEP Bottom F3=Exit F4=Prompt F5=Refresh F12=Cancel F13=How to use this display F24=More keys Note: Either use parameter OWNER(*KEEP) or, after copying the file, reset the Owner back to QNOTES. 206 Domino for iSeries Sizing and Performance Tuning more information on the DBIID, refer to 13.5.1, “What is the Database Instance ID (DBIID)” on page 385.) We believe that even on a busy mail server, it is not necessary to COMPACT -B (or -C) an active mail database to reduce the file size (with or without TxL enabled!) as long as an archiving policy is also in force. COMPACT and white space When COMPACT -b runs on a mail file, or indeed any database, rather than decrease the physical size of the database, we now recommend that you defragment the database from within, re-organizing all the records down to one end of the disk space, and shuffling the empty spaces (white space) up to the other end. COMPACT -b reorders data in a database so that the data is consecutive. The percent utilization goes up while space is compacted within the NSF structure. The file size remains the same. By defragmenting the database, the white spaces become a single large “white space”, and therefore, we now have room for larger incoming items, mail, attachments, etc. With this new incoming “data”, we can now accommodate within the existing database without having to “expand” the database by adding additional new blocks to the end. The database will gradually again become more fragmented through use. As random records (notes) are deleted, creating smaller white space holes, the database becomes increasingly fragmented. Consequently larger incoming notes and files, attachments, etc., will no longer fit within any spaces inside the database, and so the database must expand itself. Re-using white space For pure performance reasons, Notes always re-uses white space within a database before allocating additional disk blocks of storage. The rate of perceived growth of any database will slow down, and eventually stop, as the amount of white space created by COMPACT -b approaches the actual storage required for new messages (or notes etc.) delivered during the day or days between compactions. This is called steady state. COMPACT -b nightly reduces fragmentation If you ran COMPACT -b nightly for a week or two, you would be pleasantly surprised that, rather than seeing the disk reaching that “full” point on the Friday before your usual COMPACT -B, it would peak and then achieve “steady state”, probably below your previously noted high water marks. Massive fragmentation over a week can cause apparent size problems on your system. We recommend that you avoid this. The continual gentle online defragmentation achieves steady state below or close to your previous watermarks. COMPACT -b nightly can run online, speed up access, allow more users COMPACT -b can also be run online without affecting live mail or other database users (it might go slower for a minute or two, but no shutdown, etc. is required). Obviously you can choose to run it overnight or at quiet times. You may even want to run it more than once per day. Remember, the more efficient the database structure is, the faster the system can find and serve up the data, and the lower the overall system loading will be. Note: The parameters for the COMPACT command are case sensitive. That is, COMPACT -B and COMPACT -b perform differently. Important: Frequent inline COMPACT -b defragmentation is required to keep the white space in a usable, contiguous form for as much of the time as possible. Chapter 7. Tuning Lotus Domino for better performance on iSeries 207 Your users will also be pleasantly surprised that mail access feels “quicker” every day of the week, instead of just on Monday morning. That’s because the data has been re-sequenced within the database, and can now be accessed more quickly, so maybe you can even add a few more users! Why make work for yourself copy compacting your databases? If you really want, you could always run a COMPACT -B every month or two. But if you do, you must then run a set of full backups because all the DBIIDs and you are using Archival TxL for your backups. But by regularly running COMPACT -b, you have achieved steady state long before then, and the nightly COMPACT will keep things in order. Steady state nirvana Let’s say you follow the above recommendation, after you reach steady state, and then did an operating system level backup, delete, and restore of all databases and files (that is the old fashioned backup-restore defragmentation of the disk). In this case, each database would now have all its data and all its white space in sequentially allocated blocks directly associated with it, not scattered. Therefore, performance would be markedly improved in the I/O area. We suggest doing this after you reach steady state. We recommend doing the backup-restore defragmentation after a full backup and then use disk-to-disk copy for the backup/restore (with Domino down). Even without this drastic step of backup, delete, and restore, leaving white space alone to reach steady state would at least halt the march toward total fragmentation. This would occur by the totally unecessary freeing up and then immediately re-allocating of new blocks by "freeing up" white space to the operating system. Table 7-1 shows COMPACT command line options for compacting with and without white space recovery. Table 7-1 COMPACT command line options for COMPACT Option Command-line equivalent Description Domino Release Compaction style: In-place (recommended) -b Uses in-place compaction and recovers unused space without reducing the file size, unless there's a pending structural change to a database, in which case copy-style compaction occurs. This is the recommended method of compacting. New to R5 Compaction style: In-place with file size reduction -B Uses in-place compaction, recovers unused space, and reduces file size, unless there's a pending structural change in which case copy-style compacting occurs. If you use transaction logging, do full database backups after compacting completes. New to R5 Compaction style: Copy-style -C Upper or lower case “C” will work. Uses copy-style compaction. Use this option, for example, to solve database corruption problems with Release 5 format databases. New to R5 208 Domino for iSeries Sizing and Performance Tuning Implementing mail archiving Once an archival policy is implemented to restrict indefinite growth of users’ mailboxes, all users will settle into a pre-defined mail file size. Eventually all databases will reach steady state, freeing and re-using their own white-space, and taking the laborious task of managing and worrying about data growth away from the administrator. Mailbox quotas Setting mailbox quotas to control disk usage is like using a car crusher to keep a car park tidy. It makes little or no sense. In attempting to save fractional amounts of cash on disk space because you did not organize an archival policy, your company suffers for the following reasons:
You indiscriminately prevent your users from being able to do their productive job at your corporation, by stopping them from sending outbound mail, and possibly receiving inbound mail, until they administrate their own mailbox in peak busy corporate time.
Your e-mail rejects inform all your external suppliers, customers, and e-mail callers to these people that this person’s e-mail has been shutdown, causing outsiders to think the user has moved to another company and they should take their business elsewhere.
You are advertising on the Web that you have no control over your users or your IT budget or infrastructure by sending out unnecessary e-mail rejection notes almost at random.
You must employ additional IT staff to cope with the complaints, run additional backups, and restore mail hastily deleted to send out an urgent e-mail and manually “copy compact” the user’s inboxes after they have deleted mail that they may actually need in future. Where TxL is enabled, you then have to schedule a full backup for that mailbox immediately if recovery in the future is to be possible.
Potentially each user could spend two hours per week tidying their mail file, on an average wage of $30,000 per year ($15/hour), in a company with 1000 employees that is a cost of $30,000 per week, or $1.5m per year. Does that disk cost savings still look cheap to you? Would you like to estimate the cost of actual lost business during that time? In the above example, in person hours alone, it comes to 11,250 hours spent tidying the mailbox (that is work that IT should own), which is the equivalent of seven employees. Are these figures still looking good to you? Quotas can cause restore to totally fail If the option Obey database quotas during message delivery is enabled in the configuration document in the Domino Directory, the database can be recovered from backup to the size of the database quota. Once the quota is reached, however, any replayed transactions from the transactional log extents cannot be applied to the database past the limit of the quota. This results in lost transactions and lost data. Furthermore, the database quota cannot be increased until the database is fully restored. Since transactions cannot be applied over the database quota, a full database restoration is never achieved. Quotas add to the overhead and complexity of restores If Obey database quotas during message delivery is disabled, a database quota has been exceeded and the database must be recovered for any reason. During recovery, the transaction or transactions would be replayed into the database (for example, router delivering mail), so the database would be restored to its original state (which is over quota). Chapter 7. Tuning Lotus Domino for better performance on iSeries 209 To work around the issue, the database must be brought under quota and then compacted to reduce the size of the database. When the problem occurs, however, the default compaction style changes from copy style to in-place compaction (with no file reduction) for any databases that are transactionally logged. If users delete documents to bring the database under quota and compact it, the database will still remain over quota since in-place compaction is being used and the file size is not being reduced. The administrator must then compact the database using copy style compaction. When this is done, however, it changes the DBIID. As a result, a full backup must then be taken. Mail archival policy Use the standard R5 archival utility. Run automated archiving on all mail over 30, 60, or 90 days. Place all archived mail on an archive server (never the mail server). Add an icon to the user’s desktop, add an archive to the mailbox, and allow users to replicate their archive to local or access it on an archive server. Show users where “old mail” goes, since that is all they need to know. You only need a very modest system for an archive server because not many users will ever access it at any one time. It may, however, need quite a bit of disk space! If you really want, you can also archive the archives to CD for all mail that is over 12, 24, or 36 months old and give user a copy of the CD (keeping a copy in your off-site warehouse). Legal advantage of an archival policy Together with TxL keeping all the mail, a simple archival policy helps you when you are legally required to recover selected mail items from months or years ago by your legal department or other legal agency. Because you have a sensible mail backup and archival policy, this is very easy to do and everyone is happy with your IT department. If you use the quotas, the only one you will eventually find that you fulfill is part of the quota of staff to be made redundant in your department. For more information, see the following documents:
Why you need transaction logging white paper in the Technical Library: http://www.lotus.com/home.nsf/welcome/itcentral
Technotes: For more information, refer to http://www.support.lotus.com and search for the following Technote numbers: – Domino R5 Compact Not Reducing the Size of the Database #173311 – Transaction Logging in R5 #172508 – How Do You Check Your Transaction Log Statistics? #173421 – Commonly Asked Questions About Transactional Logging #177785 – How to Utilize Transactional Logging Statistics #177796 – “Circular” Versus “Archive” Transactional Logging in Domino 5.x #179363 – Transactional Logging and How it Operates #179858 7.13.1 Mail file size “So what’s wrong with my 800 MB mailbox? I might need all my old e-mails one day.” Let’s quickly go back to the basics and review how Domino databases react to change. Each addition, deletion, or change to a Notes record forces the active views to be rebuilt to reflect the new state of that index. With a mail file, every time a new mail item arrives, the view is rebuilt. The more items there are in the view, the more mailboxes there are on the system, and the more I/O and CPU are required to rebuild all those separate views. Imagine what an “All Staff” 1 MB memo does to the mail server. Multiply the amount of mail you expect to receive daily, by the average number of old e-mail in your inbox, by the number of mailboxes 210 Domino for iSeries Sizing and Performance Tuning on your system. That is the work factor you are imposing on Domino. Now reduce the mailbox size to a reasonable number and re-calculate. From those savings, you can either add another 20% more users or enjoy far better server performance. Very simple. Small well compacted mailboxes perform several hundred percent better than large ones do. 7.14 Domino memory management Domino is an application suite that has its own work and memory management functions that are a layer on top of the iSeries work and memory management. Figure 7-15 shows how Domino memory management is performed. A certain amount of main storage is set aside for:
NSF buffer (=indexes)
Database Cache (=document data)
Application code
Users Figure 7-15 How Domino divides the main storage All of these memory areas are dynamically adjusted by Domino. They both grow and shrink during a day. There are a number of variables that we recommend setting in the notes.ini file. Unless you explicitly add these variables in your notes.ini file, the default values for each of these variables are used, and in some cases, this is not what we want. Your Domino server needs to be restarted after these entries are added to the notes.ini file to use the changes. Use the show stat command on the Domino console to find out the resource utilization. Change the notes.ini setting accordingly. COMMAND SENT: sh stat database Database.BufferControlPool.Peak = 130,812 Database.BufferControlPool.Used = 48,740 Database.BufferPool.Maximum = 502000000 Database.BufferPool.Peak = 4780800 Database.BufferPool.PerCentReadsInBuffer = 98 Database.BufferPool.Reads = 702753 Database.BufferPool.Used = 4091136 Database.BufferPool.Writes = 30648 Database.DbCache.CurrentEntries = 0 Database.DbCache.HighWaterMark = 0 Database.DbCache.Hits = 0 Database.DbCache.InitialDbOpens = 121278 Database.DbCache.Lookups = 0 Database.DbCache.MaxEntries = 723 Database.DbCache.OvercrowdingRejections = 0 Hint: Use a more specific show stat command instead of showing all the statistics available every time. For example, show stat database produces the following output. NSF Buffer DB Cache Applications Users Chapter 7. Tuning Lotus Domino for better performance on iSeries 211 Database.NIFPool.Peak = 196,218 Database.NIFPool.Used = 128,438 Database.NSFPool.Peak = 130,812 Database.NSFPool.Used = 42,546 The following sections offer a few descriptions of the key notes.ini variables. They include a table of all the performance related variables and our recommendations on values that may need to be changed and suggestions on how to calculate a suitable and safe value for each. 7.14.1 NSF_Buffer_Pool_Size or Nsf_Buffer_Pool_Size_mb The NSF_Buffer_Pool_Size and NSF_Buffer_Pool_Size_MB parameters let you specify the maximum size of the NSF buffer pool. This pool is a section of memory dedicated to buffering I/O transfers between the Notes Index Facility (NIF) indexing functions and disk storage. We recommend that you set both NSF_Buffer_Pool_Size and PercentAvailSysResources (see “PercentAvailSysResources” on page 213) for all Domino R5 iSeries servers. This includes all non-partitioned and partitioned servers, because the Domino servers run out of their own storage pool, which is typically changed dynamically by the auto-performance adjuster (WRKSYSVAL QPFRADJ). Previously if you did not set NFS_Buffer_Pool_Size, Domino would calculate a value to use based on the total physical memory on your system (taking roughly 3/8). To prevent Domino from accidently stealing huge amounts of memory on very large iSeries servers (when none of the NFS_Buffer_Pool_Size parameters were set), the iSeries for Domino sets the notes.ini value to 300 MB during installation if it was not previously set. Since the iSeries uses single level (memory) store, the system does not really dedicate the NSF_Buffer_Pool_Size to the server. In reality, it just forces higher page faulting when using it. We have seen systems with multiple servers where the total of all NSF_Buffer_Pool_Size was well above the machine's main storage, and it still performed amazingly well. Performance was degraded somewhat (about 10%) due to the rather high faulting rate, but nothing failed and the customer seemed very happy with it. However, we much prefer to be more scientific and logical about memory allocation and usage, and control memory using these two parameters. There are a number of memory buffers used by Domino that are allocated based on one or the other of these parameters. Therefore, it is best to specify both to carefully worked out values. This default value of 300 MB may work fine for most servers, but a better value can and should be calculated. It is even more important to set this variable if your Domino applications use a lot of indexing. The recommended setting for R5 varies depending on the number of partitioned Domino servers that are being used. Here are easy steps to follow to calculate the NSF_Buffer_Pool_Size_MB on iSeries servers: 1. Find out from which storage pool the server is running. You can find this by using the WRKACTJOB command. The default is pool 2. 2. Find the size of the pool while your system is running in steady state (that is when QPFRADJ isn't changing the pool sizes). You can see this by running the WRKSYSSTS command. 3. If you multiple partitioned servers are using that pool, divide the pool size by the number of servers using the pool (excluding the *HTTPSETUP server) and multiply by 3/8. Set NSF_BUFFER_POOL_SIZE_MB in notes.ini to this number. Figure 7-16 shows the four partition Domino on iSeries server. 212 Domino for iSeries Sizing and Performance Tuning Figure 7-16 WRKACTJOB shows that all four Domino servers running out of pool 2 Since there are four servers, NSF_BUFFER_POOL_SIZE_MB should be: 3208 / 4 * 3/8 = 300 MB There are four things to check when fine tuning the NSF_Buffer_Pool_Size parameter: 1. Monitor how much NSF buffer pool your server uses by running the SHOW STAT DATABASE command. If BufferPoolPeak is routinely more than 95% of the maximum, then the NSF_Buffer_Pool_Size may need to be increased. If you have sufficient main memory, some memory may be freed from some other over-allocated task or job. Otherwise, you may have to consider buying additional memory or upgrading the system. Here are the statistics you will want to obtain: – Buffer pool maximum – (Database.BufferPool.Maximum) – Buffer pool used – (Database.BufferPool.Used) – Buffer pool peak – (Database.BufferPool.Peak) 2. Monitor Database.BufferPool.PerCentReadsInBuffer. When tested under a heavy workload, BufferPoolPeak was seen at 100%, while PerCentReadsInBuffer was under 95%. Under a light load, BufferPoolPeak was under 10%, while PerCentReadsInBuffer was over 97%. 3. Monitor non-database page faulting by using the WRKSYSSTS command or a performance monitor. High faulting rates indicate memory contention, and you may be able to actually improve performance by decreasing NSF_Buffer_Pool_Size. 4. Monitor the Mail Router’s database cache size (which defaults to NSF_Buffer_Pool_Size_MB * 3). You should monitor Mail.DBCacheEntries to see if your Opt Subsystem/Job Type Pool DOMINO01 SBS 2 QNNINSTS BCH 2 REPLICA BCI 2 ROUTER BCI 2 SERVER BCI 2 DOMINO02 SBS 2 QNNINSTS BCH 2 REPLICA BCI 2 ROUTER BCI 2 SERVER BCI 2 DOMINO03 SBS 2 HTTP BCI 2 QNNINSTS BCH 2 SERVER BCI 2 DOMINO04 SBS 2 HTTP BCI 2 QNNINSTS BCH 2 SERVER BCI 2 WRKSYSSTS at steady state shows pool 2 is 3207 M System Pool Reserved Max -----DB----- ---Non-DB--- Pool Size (M) Size (M) Active Fault Pages Fault Pages 1 340.52 167.89 +++++ .0 .0 .0 .0 2 3207.63 .17 400 .0 .0 4.5 6.3 3 35.83 .00 1 .0 .0 .0 .0 Chapter 7. Tuning Lotus Domino for better performance on iSeries 213 system is reaching the maximum available cache size. Then you can compare Mail.DBCacheHits to Mail.DBCacheReads to see how effectively your cache is being used For example, the following two parameters are equivalent: NSF_Buffer_Pool_Size=104857600 (100x1024x1024) NSF_Buffer_Pool_Size_MB=100 PercentAvailSysResources In Domino Release 5.0.4 or later, you can control the amount of memory allocated to each Domino server (subsystem) by percentage using the PercentAvailSysResources variable. Before the introduction of this variable in 5.0.4, each Domino server assumed it had 100 percent of system physical memory available to it. And, unless you set an actual value in NSF_Buffer_Pool_Size, Domino would take three eighths (3/8) of total physical memory. Other Domino memory allocation sizing parameters would take a percentage of the remaining five eighths (5/8) of all memory. On iSeries, this behavior was prevented by defaulting the notes.ini NSF_Buffer_Pool_Size parameter to 300 MB, where it was not set during installation. With the advent of the PercentAvailSysResources variable with Domino 5.0.4, you can now assign a portion of total system physical memory to each Domino server by specifying a value from 2% to 100%, which represents an absolute percentage of the system's total physical memory. On the iSeries server, memory calculations are based on the main storage size multiplied by PercentAvailSysResources if the Domino subsystem is running in *BASE pool or on the actual pool if running in a shared (for example *SHPOOL01) or private pool. NSFBuffer Pool defaults to approximately three eighths (3/8) of that calculated base for each server. Note also that if QPFRADJ is not set to zero, and you are using shared pools, your shared pool sizes will be adjusted on the fly. That may affect how PercentAvailSysResources works in cases where the Domino server is started or restarted when the subsystem is already active for a longer time and the pool size has been adjusted meanwhile. In that case, using NSF_Buffer_Pool_Size will be more predictable. The NSF_Buffer_Pool_Size (if omitted) is the time allocated three eighths (3/8) of the total amount of memory allocated to Domino by PercentAvailSysResources. If you need more memory in the NSF_Buffer_Pool_Size, you must adjust any one (or more) of the following areas:
Physical memory (so that the amount from the same percentage increases)
Proportion of memory that this partition acquires by increasing PercentAvailSysResources Note: The only difference between NSF_Buffer_Pool_Size and NSF_Buffer_Pool_Size_MB is that NSF_Buffer_Pool_Size is set in bytes. It also has a maximum possible value NSF_Buffer_Pool_Size_MB that is set in MB. Tip: You can create shared pools on the iSeries server and assign a Domino server instance to run in that pool. This allows performance adjustment to increase the shared pools storage when and if the base pool is not using the memory allocated. This way, a Domino server could be granted access to additional memory, when required when, other processes on the system do not require it. However, even when using performance adjustments, you must be certain that you have defined the pool to allow the server to function properly. Undersizing a shared pool can result in poor performance as the server becomes hindered from a lack of memory. 214 Domino for iSeries Sizing and Performance Tuning
Deliberately set NSF_Buffer_Pool_Size to a higher value than three eighths (3/8) of available memory In the first two cases, the remaining memory buffer allocation parameters (the 3/8 versus 5/8) will keep their correct relative values and balance. With the third case, you should carefully make certain that the NSF_Buffer_Pool_Size fits in with your actual requirements for this particular configuration by calculation if you set it to greater than 3/8 deliberately. Setting both variables will protect the overall system from Domino’s stealing habits, if, at some future point more physical memory is added with no (Domino NOTES.INI) parameter adjustments. You can find details on all the principal system performance parameters in Table C-3 on page 444. For example, if you want to dedicate 25 percent of total system memory to a Domino server (in its own dedicated partition, typically), add the following line to the notes.ini file: PercentAvailSysResources=25 This effectively leaves 75 percent of memory for other Domino partitions or applications. For example, a Domino server on a system with 10 Gb of memory and a Domino server with its notes.ini setting PercentAvailSysResources=25 would reserve 2.5 Gb for itself. The PercentAvailSysResources parameter has significant advantages over the old method of “hard coding” the memory using NSF_Buffer_Pool_Size parameters:
If you add additional memory, it is immediately proportionally shared between the Domino servers without any need to change or edit the notes.ini parameters.
In high availability clustering and failover scenarios, when a Domino server’s workload is failed-over to another partition, the percent of physical memory required by the receiving partition to cope with the increased workload is quite simply the sum of the two separate PercentAvailSysResources parameters from the failed and the newly joined partition. Example with four Domino partitions sharing with other applications For example, if you have a Domino server with four partitions and want to control the memory resources allocated to each partition, you could set the notes.ini of partition 1, 2, and 3 with the line PercentAvailSysResources=20 and the notes.ini of partition 4 with the line PercentAvailSysResources=40. Here you should make sure that the sum of the values you specify in each partition do not exceed 100 percent! If it does, the systems will simply steal from each other, which could cause unnecessary thrashing and paging. To reserve system memory for other applications, you can choose values that total less than 100 percent. Indexing and costs associated If your Domino applications use indexing, it is tempting to think that you can simply increase the size of the NSF buffer pool and dramatically enhance server performance. However, just as adding physical memory to a system does not always boost performance, neither does designating a bigger NSF buffer pool necessarily produce dramatic results. (See the article “Putting the right spin on Domino Performance” Parts 1 and 2, which is available from the Iris Today Web site at: http://www.notes.net/today.nsf). Chapter 7. Tuning Lotus Domino for better performance on iSeries 215 Using a buffer pool summary Domino does a good job by itself of allocating memory to the NSF buffer pool by calculating a default setting based on fixed overhead and total memory available. (Depending on the Domino version and other factors, the default usually amounts to between 25 percent and 33 percent, that is, the now famous 3/8 of system memory). What's more, changes to the NSF buffer pool allocation algorithm in R5 mean that the Domino server uses only the amount of memory it needs, up to the notes.ini NSF_Buffer_Pool_Size (default 300 MB on iSeries), and uses the amount allowed by PercentAvailSysResources. The R4 strategy pre-allocated and reserved the entire default or specified amount of memory. But in R5, when you specify a value of 600 MB for the buffer pool size and only 200 MB is needed, only 200 MB remains allocated. In Domino R4, if you specified a value of 600 MB, then all 600 MB was immediately treated as all allocated by Domino. To summarize, we recommend that you implement the PercentAvailSysResources notes.ini variable to control total and overall memory allocation. For releases prior to 5.0.4, where PercentAvailSysResources is not available, there are some scenarios where setting non-default values for the NSF_Buffer_Pool_Size variable can prove useful. After 5.0.4, we still recommend that you set NSF_Buffer_Pool_Size together with PercentAvailSysResources. If you employ any of these notes.ini variables, test your server performance to see if you get the results you expect. The number of users, size and number of views, and number of databases all factor in when determining what the settings should be. And remember to review the settings when the system configuration changes or after upgrading the system to a new Domino release, because it is likely that you will need to adjust them. You should always test and monitor with a realistic data set before you make any configuration changes to your production environment. To help you do so, we recommend the following actions:
If you are running 5.03 or higher, enable platform statistics by adding platform_statistics_enabled=1 to the notes.ini file.
Enable STATREP collection on the server.
While the server is under load, issue the sh stat command in 15 minute intervals. This dumps information about memory usage to the console.
Review the data. Table 7-2 lists the SH STAT database.* statistics and their utilization. Notes:
The NSF buffer pool setting in R5, up to and including 5.0.3, should not exceed 1920 MB (or 2,013,265,920 bytes).
If you have Domino Release 5.0.4 or later, you should use the PercentAvailSysResources variable to allocate memory on the iSeries. 216 Domino for iSeries Sizing and Performance Tuning Table 7-2 SH STAT output parameters and a description of their content and use Server_Max_Concurrent_Trans This parameter regulates the number of concurrent transactions that can occur in a server at one time. When the number of concurrent transactions is reached, the other transactions are put into a wait state until one of the active transactions completes. In R4, we recommended that you set this parameter to -1, but in R5 on iSeries, you must not use -1. Parameter Description Database.BufferControlPool.Peak Peak number of database control pools used. Database.BufferControlPool.Size Size of database buffer control pool. Database.BufferControlPool.Used Number of database control pools used. Database.BufferPool.Allocated Number of database control pools allocated. Database.BufferPool.Maximum Maximum size of database control pools. Database.BufferPool.Peak Peak number of database buffer pools. Database.BufferPool. PerCentReadsInBuffer Percentage of read requests satisfied from the buffer. In performance terms, this is “cache hits”. Note: This value will be low for quite a time after a server starts as the cache is being populated. This value should not be referred to if the server has not been running for at least two hours. Database.BufferPool.Reads Number of database buffer pool reads. Database.BufferPool.Used Number of used database buffer pool. Database.BufferPool.Writes Number of database buffer pool writes. DataBase.DbCache.CurrentEntries Number of entries in the database cache at this time. DataBase.DbCache.HighWaterMark High water mark of the database cache. DataBase.DbCache.Hits Number of hits to the database cache. DataBase.DbCache.InitialDbOpens Number of database opens done by the database cache. DataBase.DbCache.Lookups Number of lookups to the database cache. DataBase.DbCache.MaxEntries Maximum number of entries in the database cache DataBase.DbCache. OvercrowdingRejections Number of rejections due to the overcrowding of the database. Database.ExtMgrPool.Peak Peak number of external manager pools. Database.ExtMgrPool.Used Number of External Manager pools. Notes: Where statistics suggest that the NSF buffer pool size should be increased, the amount of the increase should not exceed 10%. Be sure to review the data again. Refer to the StatRep log to see if brief pairs or patterns of spikes are a “normal” daily trend at particular times and can therefore be safely ignored in the short term. Chapter 7. Tuning Lotus Domino for better performance on iSeries 217 In R5 Domino now fully utilizes thread pools. These pools are set by default to forty threads per port. However, if you set Server_Max_Concurrent_Trans in notes.ini (default is 20), be aware that IOCP threads default to two times this number (that is 40 if the default is set). If Server_Max_Concurrent_Trans is set to -1 (representing the maximum), Domino will create twice that number of IOCP threads, that is, thousands of threads for the server pool. You may then see:
Server job taking up a lot of CPU
Thousands of threads running under the Server job
High faulting and paging in Domino memory pool
Slow performance
Users unable to connect Server_MaxSessions This parameter defines the number of user sessions a Domino server can accommodate. When a new user attempts to log on, if the current number of sessions is greater than the value of Server_MaxSessions, Domino closes the least recently used session. For a session to be considered for closing, it must have been inactive for at least one minute. There is presently no way to prevent a user from opening a new session on a Domino server (for more information, see 10.2, “Clustering” on page 278). If there are too many concurrent sessions (active sessions) on the Domino server, errors may occur. Reaching this limit on iSeries can cause the SERVER task to start taking up a lot of CPU resources with a single thread, usually the one listening for connect requests on the TCPIP port, trying to get a lock on a semaphore. Users see the message “Remote system no longer responding”. Server_MaxUsers This parameter specifies the maximum number of active NRPC user sessions allowed on a server. This is only effective for NRPC (Notes Client) users and does not affect HTTP users or HTTP sessions. The HTTP user is accessing the server via an ICM server allowing HTTP users to continue to create new sessions even after this limit is reached. When this number is reached, the server state becomes “MaxUsers” and the server stops accepting new Database Open requests from NRPC users. This prevents new sessions from starting and stops existing NRPC users from opening another database. They also see the error message: Server Error: Access to the server is restricted due to maximum number of users. Servers that are not in a cluster reject the access requests. Important: Be absolutely certain this parameter is not set it to -1. We recommend that you do not set this parameter unless specifically advised by IBM Support. Important note: Reducing Server_MaxSessions to a low number will never prevent Domino from allowing more than that number of concurrent active users on the server. Reducing Server_MaxSessions will, however, cause sessions to start dropping very soon after they become inactive. This technically frees up resources, but becomes very heavy on the CPU and system in re-creating the sessions for all the same users slightly later. Conversely, Domino will not close any session that has been idle for less than a one minute regardless of the demand on the server. 218 Domino for iSeries Sizing and Performance Tuning When using the ICM for failover and workload balancing, here as in standard Domino clusters. Domino computes the server availability index based on all open sessions, whether they are from Notes clients, HTTP clients, or other Domino servers. To configure workload balancing and failover, use the same settings, that is Server_Restricted, Server_MaxUsers, and Server_Availability_Threshold. For stopping database availability, you would mark a database out of service or pending delete. We recommend that you set the default to 0 (unlimited access to server by user). Server_Session_Timeout The Server_Session_Timeout parameter from the notes.ini file causes Domino to terminate an NRPC network or mobile connection/session that has been idle for this amount of time. A session is considered idle if no activity detected. If it is set too low, the server may have to re-open database server sessions too often. The best setting for this parameter is determined by the server load, and numbers of concurrent users on the server, etc. The sesson timeout only applies to Notes clients and not to Web browsers. The Domino HTTP task accesses the server directly or via an ICM server so HTTP users are not affected by changes to this time-out value. R5 has much improved handling on concurrent user sessions over R4, so this parameter can be safely removed. It has no affect unless the server is busy. Be aware, creating a new session is very CPU expensive. Re-using sessions is much more efficient, which is why we have the parameter. The term “Session” also includes server “internal” sessions, such as agents, replicator tasks, server to server, and so on, so plan for it! Consider this example. If you have 60 agents running every 35 minutes, and you set this timeout to 30 minutes, you will unnecessarily create one session per minute because every agent will be timed out! The default for this parameter is 240 minutes (4 hours). We recommend that you reduce it to 30 to 45 minutes, but not less. iSeries parameter TCPKEEPALV The iSeries has a TCP parameter that specifies the time (minutes) that TCP waits before sending out an idle tickler probe to the other side of a connection to test for signs of activity. On the iSeries, this is the TCPKEEPALV parameter. The default value for this parameter is 120 minutes. Setting this value to a smaller number will cause sessions to be cleaned up sooner. If the tickler finds the “other half” of the conversation has “gone away”, that is, it does not respond at all, it is cleaned up. It should be noted that a TCP/IP conversation that responds at the tickler level, but not at application level will never be closed by TCP/IP, until either the application closes the conversation or the conversation is ended abnormally by OS/400. Use CFGTCP option 3. We recommend that you set this parameter to 15 minutes. Translog_MaxSize This parameter defines the maximum size in MB that the collection of 64 MB Circular TxL log files can grow to before being forced to start overwriting (re-using) themselves. This parameter and Translog_UseAll are both ignored when using archival style transaction logging. Chapter 7. Tuning Lotus Domino for better performance on iSeries 219 TxL logs are written in 64 KB chunks. The logs “segment” (next log file “extent”) at 64 MB. The logs are binary and created empty 64 MB long. When re-used, they are incrementally renamed and the contents cleared down. All of this helps performance and prevents disk fragmentation. Never routinely delete TxL log files! The only purpose of TxL on iSeries is fast recovery after a crash and incremental backups. Monitor the statistic Database.RM.SinceStartup.Critical.Log.Times. If this value is ever greater than 0, it suggests that at some point since server startup, the uncommitted transactions (those not hardened from virtual cached images to real disk images) in the (circular) log have nearly equalled the size of the log itself. Or it may indicate that new transactions are flowing through the log as fast, or faster, than the recovery manager can commit them. Since the recovery manager must not overwrite an uncommitted transaction in the logs, flushing the virtual database image to disk (hardening) gets a higher priority during these critical times. This then releases uncommitted transactions from the TxL logs to allow re-use of TxL log files. As a consequence of this, server performance suffers. To correct this, you can increase the log size, that is this parameter. 7.14.2 How queues relate to the NSF_Buffer_Pool_Size A message queue is a queue that Notes uses to store certain tasks that are queued up (indexing, design updates, full text, agents, open handles for lookups). The Indexer is a classic example of a Domino “task” that works from a queue. The following tasks access an Indexer queue:
The Indexer accesses this queue to pull requests from it.
Every “view update” request is added to this queue.
Chronos (the Domino scheduler) adds requests for Full Text updating to this queue. So, the Indexer queue contains various requests for databases to be indexed from various sources. The Indexer reads the next request from this queue, removes it from the queue, and performs the indexing functions. Therefore, a single Indexer task works on a single database indexing request that it pulled from the queue. If a second request comes into the queue, the next indexer then removes that request from the queue and starts working on it. If both of these requests are for the same database, then the two tasks will attempt to work on the same database, assuming it is on different indexes. Attention: The minimum size is 192 MB (it ignores all lower values). If you start Domino with physically less disk space for TxL logs than 192 MB, Domino will identify this as a severe error (panic). Note: Translog parameters are always overwritten by the values from the server document, so there is no point to changing them in the notes.ini directly. Domino has to create more than 65 MB of Archival TxL log before the Domino API reports the first complete archive log is complete and available for backup. Some third-party backup solutions use recent API additions allowing the active log to be flushed and a new log started to allow it to be backed up. Note: Queue access is on a first-come, first-served basis; no task has priority over another task, and the requests are added to the queue in First In, First Out (FIFO) form. 220 Domino for iSeries Sizing and Performance Tuning It is statistically more likely, however, the two tasks will work on different databases. If the request is for a full text index update, this is a single index and multiple UPDATE processes cannot update the same full text index at the same time. While multiple Indexer Tasks can update different view indexes within the same database at the same time, typically we don't recommend running large numbers of Indexers due to the fact that it may cause contention. When a message queue reaches its capacity, this is the error that occurs: "Server Error: Insufficient Memory - Message Queue Pool is Full" For the Indexer queue, you can check $UPDATEQUEUE and see if it is the Indexer Queue that has reached its capacity. “Message Queue Pool” is Full for the Indexer task or tasks may indicate:
We do not have sufficient indexers taking work from the Index Queue
We are not servicing the pool fast enough (that is insufficient resources)
The pool itself is full, causing the queue to reject further entries (pool too small) This type of error message is seen most frequently on mail servers on which mail databases are full text indexed. Here are some possible workarounds particularly for Mail Files:
Load another Indexer task on the server to service the queue more frequently.
Change the full-text update frequency from hourly to daily on the mail files.
Limit the number of simultaneous full text Indexers with FullTextMultiProcess=.
Prohibit all full text indexing (usually of user mail files) with Update_No_Fulltext=1. A display of the Notes statistics may reveal the buffer pools maxed out. In Figure 7-17, the sh stat database command was entered on the server console (or remote console). Tip: Loading multiple Indexer tasks on the server may or may not always be beneficial. It all depends on whether the multiple requests in the queue are for the SAME full text index on the SAME database. Chapter 7. Tuning Lotus Domino for better performance on iSeries 221 Figure 7-17 The sh stat command examining the database buffer cache and pool In a textbook example where there were major problems, the customer key values were:
Database.BufferPool.Maximum = 74693652
Database.BufferPool.Peak = 75110000
Database.NIFPool.Peak = 2223804
Database.NIFPool.Used = 4292080218
Database.NSFPool.Peak = 1765962
Database.NSFPool.Used = 4286027655 You should notice that:
Database.BufferPool.Maximum < Database.BufferPool.Peak
Database.NIFPool.Peak < Database.NIFPool.Used
Database.NSFPool.Peak < Database.NSFPool.Used Increasing the NSF_Buffer_Pool_Size in the notes.ini file may help to resolve the issue. However, the server may need to be monitored to see if the Database.BufferPool.Maximum still shows less than the Database.BufferPool.Peak or the “Message Queue Pool is Full” error continues. If this consistently happens, the server may be simply overloaded and need more memory or another resource to be upgraded. NSF_DbCache_Maxentries Each server uses a database cache to minimize the time lags involved with opening and closing databases on a server. When a user opens a database, Domino puts the database in the cache, where it remains for a period of time. While a database is in the cache, users can open and close the database quickly. > SH STAT DATABASE Database.BufferControlPool.Peak = 130,812 Database.BufferControlPool.Used = 40,814 Database.BufferPool.Maximum = 7194660 Database.BufferPool.Peak = 1728000 Database.BufferPool.PerCentReadsInBuffer = 98 Database.BufferPool.Reads = 4099 Database.BufferPool.Used = 1717134 Database.BufferPool.Writes = 747 Database.DbCache.CurrentEntries = 7 Database.DbCache.HighWaterMark = 37 Database.DbCache.Hits = 2404 Database.DbCache.InitialDbOpens = 3270 Database.DbCache.Lookups = 3256 Database.DbCache.MaxEntries = 25 Database.DbCache.OvercrowdingRejections = 239 Database.NIFPool.Peak = 130,812 Database.NIFPool.Used = 69,263 Database.NSFPool.Peak = 130,812 Database.NSFPool.Used = 20,812 222 Domino for iSeries Sizing and Performance Tuning How databases are removed from the cache Databases are removed from the cache through the process described here. An ager thread automatically “pushes” databases toward closure (by performing necessary writes, deallocating memory, and so forth) over a period of 15 to 20 minutes. Ideally, databases leave the cache in time to allow room for new entries, which are added when a database is closed. If not (the current number of entries exceeds NSF_DbCache_Maxentries), then one of the following actions occurs:
If the current number of entries is less than NSF_DbCache_Maxentries*1.5, Domino adds a database entry to the cache and the ager accelerates to quickly get down to the NSF_DbCache_Maxentries setting.
If the current number of entries is greater than or equal to NSF_DbCache_Maxentries*1.5, the database is closed normally instead of being put in the cache. View_Rebuild_Dir The View_Rebuild_Dir parameter changes the temporary work file directory where the UPDATE task keeps all of its work files: View_Rebuild_Dir= The presence of this parameter also causes all views including permuted or categorized views (for Domino Release 5.0.3 or later), to be rebuilt using the much faster R5 optimized view rebuild. This places all the temporary files (.DTF files) in the directory specified, instead of using the R4 default directory (which was the Domino data directory). If insufficient space is available or it’s an R4 database, the following warning appears and is logged: “Warning: unable to use optimized view rebuild for due to insufficient disk space at