Adobe LiveCycle ES2 Technical Guide Joel Lucuik, Senior Computer Scientist

Adobe® LiveCycle® ES2 Performance Tuning Guide This document provides guidance to application architects and developers to help them monitor and measure the performance characteristics of a LiveCycle application. This includes the use of built-in tools and mechanisms as well as the use of third party tools to analyze performance.

Introduction Download Adobe® LiveCycle® ES2

In this guide we describe LiveCycle parameters that can be used when adjusting LiveCycle to meet specific operational needs. This document is intended to be used by individuals with good technical knowledge of all aspects of a LiveCycle environment.

devnet/livecycle/trial.html

Each customer situation is unique with specific tuning requirements. LiveCycle’s Configuration Manager configures LiveCycle with the default settings, which may be good for general purposes, but should be revisited periodically.

Adobe® LiveCycle® Data Services Home Page

Many LiveCycle Services can be individually tuned. Tuning LiveCycle typically involves monitoring the JVM(s), operating system, and database, then deciding what adjustments are necessary to improve system performance. Monitoring tools such as the LiveCycle Health Monitor and JConsole can provide information to help with required decisions.

http://www.adobe.com/

http://www.adobe.com/ products/livecycle/

LiveCycle’s “Bedrock layer” concepts are introduced within this guide to help tuning of the C/C++ components outside of the JVM. Adobe® LiveCycle® ES Developer Center

In order to tune third party software such as application servers, and in some cases, the JVM and database, you may be referred you to 3rd party documentation.

http://www.adobe.com/ devnet/livecycle/

Performance factors There are many factors to consider when tuning a LiveCycle ES2 application. Consider each of the following when approaching a tuning exercise: • The application design, the volume of form data, quantity of attachments, as well as signatures or annotations in the forms that are used in the process will affect the user-interactive experience and the database growth over time. • The presence of LiveCycle ES2 solution components, such as LiveCycle PDF Generator, or other components that are incorporated into processes will affect the performance of the servers. • Some LiveCycle Services make heavy use of the underlying database. The choice of database server, CPU, and disk I/O bandwidth available on the database server will greatly influence performance. • Other database factors, such as where the database runs (on the same server as LiveCycle ES2 or on a remote server), if there is a parallel database load from other applications, and the available bandwidth between LiveCycle ES2 and the database server. Questions or suggestions about this Tech Guide? Contact us directly at [email protected]

• The amount of memory available on all servers. More complex or larger applications require more memory to service any single transaction. • The number and speed of the processors available on each server as well as the architecture of the processors. For example, Sun™ SPARC® and IBM® Power processors of nominally equal capacity do not perform equally for all LiveCycle applications. • The infrastructure architecture affects performance, including factors such as whether the server is single-tiered or multi-tiered, and if the architecture implements clustering or high availability. • The network bandwidth available between the application server and the desktop, or between the web server and the desktop. Network latency can affect the time it takes to transfer forms or data from the server to the desktop and vice versa, lengthening the overall response time. • Scalability factors, both over time and during peak hours. Significant factors affecting scalability include growth and variation of content, customers, users, and volume. • LiveCycle ES2 configuration settings for individual document services and core platform components. • The use of security systems such as firewalls, intrusion detection systems, content filters, disk encryption, and antivirus software on the server or the desktop. On servers in particular, antivirus software and the choice of antivirus settings or policies can cause significant degradation in performance. • Desktop infrastructure affects the performance of client-side components, in particular, the deployment of Adobe Reader and Workspace, which will affect the performance of a LiveCycle application from an end-user perspective. Desktop infrastructure factors include the speed of desktop systems, amount of memory available for the desktop, other desktop programs in use, as well as the version of Adobe Flash®, Adobe Reader®, or Adobe Acrobat® available on the desktop.

Monitoring LiveCycle LiveCycle can be monitored by the Health Monitor or external tools such as JConsole or JVM logging.

LCES Health Monitor The LCES Health Monitor is an interface delivered with LiveCycle that presents a health check of the internal details of the Java container. The Health Monitor can be found in the Administration Console at the top right on the menu bar.

The Health Monitor provides two views: System, and WorkManager. System This view presents memory utilization, CPU utilization and other low level details. • OSName: Name of the Operating system where LiveCycle is deployed. • Up Time: Time in hours and minutes for which LiveCycle has been up. • Total Physical Memory: Physical memory of the system in MB. • Free Physical Memory: Available physical memory of the system in MB. • Free swap space size: Free swap space in MB. • Latest Heap: Currently used heap in MB of the system. • Latest Committed Heap: Current heap allocated to LiveCycle. • Latest Non Heap: Currently used non heap in MB of the system. • Latest Committed Non Heap: Allocated non heap to the system in MB. • Latest Number of Thread: Total number of threads currently present in the system.

2

Also included in the health monitor are historical charts.

WorkManager This view presents the status of the work queues within LiveCycle, including successful, failed, total jobs, workflows, and events.

JConsole The second method of monitoring the JVM is with JConsole. This is a JMX Console shipped with the Oracle JDK, the IBM JDK, and Oracle JRockit. JConsole connects to the JVM to provide real time heap information, Garbage Collection (GC) facilities, active thread and class counts among other details. Each JDK has its own heap layout and GC strategy. For the sake of simplicity, the authentication and encryption security mechanisms in the examples in this document are disabled. However, you should implement these security mechanisms when implementing remote management in real-world environments. JBoss, WebLogic, and WebSphere require configuration changes to allow JConsole to connect to them.

3

JBoss and WebLogic JBoss and WebLogic require the following configuration change to the startup script to permit JConsole to connect: -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote

WebSphere To connect to WebSphere, the following steps are required: 1.

Create a file containing the following property definitions: com.sun.management.jmxremote.port=8999 com.sun.management.jmxremote.authenticate=false com.sun.management.jmxremote.ssl=false // The port can be any available port (above 1024).

2.

Modify the server JVM startup arguments by adding: -Djavax.management.builder.initial= -Dcom.sun.management.jmxremote -Dcom.sun.management.config. file=

Heap Memory Usage JConsole provides low level memory statistics and basic graphing facilities to help you understand the behavior of the JVM heap and Operating System memory usage. One can easily profile memory during load testing of LiveCycle to determine behavior under load. The JVM can be monitored in real time during any kind of performance testing. This will not affect performance statistics. JConsole permits you to trigger a full GC clean-up to see the “net” state of the heap. The GC response may not be immediate. To learn more about JConsole, visit: http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html

More detailed Memory usage documentation can be found on the Sun website: http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html#LowMemoryDetection

4

Memory Charting Memory charts are available in JConsole to help monitor activity in the JVM.

Memory Charting Exporting Chart data can be exported in csv format for use in various reporting/charting tools.

5

Heap Dump JConsole provides heap dumping facility to help analyze the current state of the heap.

Any tools that understands the “.hprof” format can be used to analyze the heap dump. One option is MAT, a plugin for Eclipse. See http://www.eclipse.org/mat/ for details. Eclipse facilities for analyzing a Heap Dump MAT is a free Memory Analyzer plugin for eclipse that is a fast and feature-rich Java heap analyzer that helps find memory leaks and reduce memory consumption. MAT can import: • .hprof: JConsole heap dump • .phd: output of “kill -3 [processId]” from a Solaris/WebSphere JDK with the help of the following plugin: http://www.ibm.com/developerworks/java/jdk/tools/dtfj.html For larger heap dumps consider adjusting the heap to -Xmx4096m in MemoryAnalyzer.ini.

6

JMX Facilities for Timing Processes LiveCycle stores execution times of core services and customer built processes to help analyze where time is being spent. The collected data can be seen at a granular level using JConsole. Look for the tree of data under “adobe.com”. Each custom service created within LiveCycle Workbench has JMX facilities associated with it automatically.

JVM Logging The Health Monitor and JConsole provide a snapshot of JVM behavior. For a data-over-time approach to assist analysis, consider the following JVM options for Sun, IBM or JRockit.

Sun JDK The following options are taken from http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp: • verboseGC •

Syntax: -verbosegc (See http://java.sun.com/docs/hotspot/gc1.4.2/example.html)

•

Dumps GC behavior to System.out

• PrintGCDetails •

Syntax: -XX:+PrintGCDetails

•

Prints additional information about collections “[GC [DefNew: 64575K, 959K(64576K), 0.0457646 secs] 196016K, 133633K(261184K), 0.0459067 secs]]”

• PrintGCTimestamps •

Syntax: -XX:+PrintGCTimeStamps

•

Prints the times at which the GCs happen relative to the start of the application: “111.042: [GC 111.042: [DefNew: 8128K, 8128K(8128K), 0.0000505 secs]111.042: [Tenured: 18154K, 2311K(24576K), 0.1290354 secs] 26282K, 2311K(32704K), 0.1293306 secs]”

• HeapDumpOnOutOfMemoryError •

Syntax: -XX:+HeapDumpOnOutOfMemoryError

•

Dumps the heap when an OutOfMemory error occurs. Heap is dumped to %JAVA_HOME%\ bin (where the Java process was launched from)

7

IBM JDK Verbose Logging 1.

Enabling verbose logging is done through admin console. Go to http://:/ibm/console

2.

Select Servers, Server Types, WebSphere application servers

3.

For each server, Select , Java and Process Management, Process definition, Java Virtual Machine

4.

Check the Verbose garbage collection check box

Note that verbose information will be output to the server’s native_stderr.log HeapDumpOnOutOfMemoryError • Syntax: -XX:+HeapDumpOnOutOfMemoryError • Dumps heap to [WebSphere’s JavaHome]\bin

Oracle JRockit The following details are based on: http://www.oracle.com/technology/products/jrockit/index.html. verbose • Syntax: -Xverbose • With -Xverbose, the JRockit JVM will output detailed information about the system. The output by default is saved to the standard file for error messages (stderr) but can be redirected to a separate file by using the -XverboseLog command line option. The information displayed depends on the parameter specified with the option. For example, specifying the parameter “cpuinfo” displays information about the CPU and indicates whether or not the JVM can determine if hyper-threading is enabled. verbose:memdbg • Syntax: -Xverbose:mdmdbg • Turns on memory output and adds new special memdbg details. For example: [memory ] 12.875: nursery GC 89648K, 89716K (89716K), 3.296 ms [memdbg ] nursery GC 291: promoted 1510 objects (69744 bytes) in 3.296 ms [memdbg ] Page faults before GC: 36784, page faults after GC: 36800, pages in heap: 22429 [finalizer] (YC) Pending finalizers 0, 0 [memdbg ] old collection 7 started [memdbg ] Compacting 8 heap parts at index 112 (type 2) (exceptional 0) [memdbg ] starting parallel marking phase [memdbg ] ending marking phase [memdbg ] current generational GC work score: 0.142956 [memdbg ] last single generational GC work score: 0.081486 [memdbg ] current error: -0.042956 [memdbg ] previous nursery size: 736760 [memdbg ] requested nursery size: 711984 [memdbg ] starting parallel sweeping phase [memdbg ] ending sweeping phase [memory ] 11.841-12.025: GC 89716K, 67088K (89716K), 184.000 ms [memdbg ] Page faults before GC: 36827, page faults after GC: 37036, pages in heap: 22429 [finalizer] (OC) Pending finalizers 0, 0

8

verbose:gc (or verbose:memory) • Syntax: -Xverbose:gc • Information about the memory management system, including: •

Start time of collection (seconds since JVM start)

•

End time of collection (seconds since JVM start)

•

Memory used by objects before collection (KB)

•

Memory used by objects after collection (KB)

•

Size of heap after collection (KB)

•

Total time of collection (seconds or milliseconds)

•

Total pause time during collection (milliseconds)

The information displayed by -Xverbose:memory or -Xverbose:gc will vary depending upon the type of garbage collector you are using. gcReport • Syntax: -XgcReport • The -XgcReport option generates an end-of-run report that shows garbage collection statistics. You can use this report to determine if you’re using the most effective garbage collector for your application. The report divides the statistics into “young collections” and “old collections”. For each of the types, the following information is provided: •

Number of collections: The total number of garbage collections of this type during the run.

•

Total promoted: The total number of objects and amount of bytes promoted from young space to old space by this type of garbage collection during the run.

•

Max promoted: The maximum number of objects and amount of bytes promoted by any single garbage collection of this type during the run.

•

Total GC time: The total time spent in this type of garbage collections during the run. For concurrent garbage collections, the total garbage collection time and the total garbage collection pause time will differ.

•

Mean GC time: The average time spent in a single garbage collection of this type during the run. For concurrent garbage collections the garbage collection time and the garbage collection pause time will differ.

•

Maximum GC pauses: The three longest garbage collection pauses caused by this type of garbage collection during the run.

• The effect of this option is identical to -Xverbose:gcreport. DumpFullState • Syntax: -XXdumpFullState • Usually, when the JRockit JVM crashes, it saves the state of the process (called a core dump), but the heap is removed from this state since it is large and would take up a great deal of disk space. With this option we can save all the process states including the heap. More disk space will be used, but it makes it much easier to use the core dump to find out what the problem was that caused the crash. This option saves a significant amount of information to disk. If you don’t want to save all the information that -XXdumpFullState saves, use -XXdumpSize:normal.

9

Managing the Heap One aspect of a tuning exercise is understanding heap behavior and making adjustments if required. JConsole is one tool which can help monitor the heap. If there are many very large objects in the heap, the Document object can store its bytes in a file backend. The file backend is only created when the size of the bytes is larger than the Max Inline file size. The benefit of this behavior is that it reduces the overall memory requirement of the JVM. For more information refer to Document Manager and the GDS (Global Document Storage) or MaxInLine Size. 64bit systems support large heaps which allow for a large Max InLine size if required. This may become useful if you require a much larger form cache for high speed rendering in the Forms DSC or faster output via the Output DSC. It should be noted that garbage collection in a large heap can be lengthy. It is important to find the heap size that you need, and not specify a much larger heap “just to be safe”. The best case scenario is a heap 20% larger than the maximum heap from test results. The size should be determined by running a test using the anticipated production system collateral and volume.

Overhead penalty and compressed pointers A 64bit JVM has object pointers with 64bit lengths. Heap usage increases as a result. It does not mean that it takes up twice the amount of space, since primitive types do not change size. A String of 100 characters would not require twice the space on a 64bit JVM, but would require more space overall. The average increase for byte differences is approximately 30%. Using compressed pointers is a benefit and will not require a heavy overhead penalty. This defaults to ON for IBM JDK with a heap of less than 25GB, JRockit with a heap of less than 4GB, and has to be explicitly turned on in HotSpot V14 or later, with up to a 32GB heap. Use -XX:+UseCompressedOops to enable compressed pointers in Hotspot. See: http://wikis.sun.com/display/HotSpotInternals/CompressedOops or more information. Compressed pointers is the default setting with the LiveCycle Turnkey (JBoss) installation.

10

Processes Recording and Logging LiveCycle Workbench provides tools to monitor the progression of a process. This includes the ability to record a process and play it back visually or instrumenting a process and reviewing the resulting log files.

Recording a Process Recording a process is useful to understand the performance of a process flow. Before invoking a process, it can be recorded from within workbench. All recordings should be turned off when not in use as recording can be CPU and disk intensive.

11

Instrumenting a Process Adding logging within a process flow can help determine which sections of a flow require the most attention and tuning. Running a process will not tell you how much time each step in the process takes. Consider the following process:

Instrumentation has been added at the Write String node to log the current time to c:\loggingTimes.txt. This technique can help detail exactly where time is being spent from node to node, and where there is opportunity for optimization. Logging would need to be done before and after the critical nodes to provide relevant data.

Document Manager and the Global Document Storage (GDS) LiveCycle ES2 uses an abstract object type, the Document Object, for a variety of purposes. These objects may be temporary and removed automatically after a short time or persist throughout the life of a process. The Document Manager decides where a document will reside. Temporary, versus persistent. Persistent documents must remain accessible for long periods and must be accessible equally from every node in a cluster. The physical storage area for these documents is referred to as the Global Document Storage area (GDS). In LiveCycle ES2 the GDS may be stored on a shared file system, such as a network attached storage (NAS) server, or configured to reside within the LiveCycle database where the document data is stored in the table TB_DM_CHUNK. This selection is made during system installation. When the ES2 User Action stores form data as a LiveCycle Document Object, the global LiveCycle Max Inline Size parameter will influence whether the bytes stored in the Document object are stored “in-line” that is, directly in the tables, or in the GDS which may be in a file system or database table of its own. The default Max Inline Size setting is 64 K. Note the above behavior also applies to Document type workflow variables.

12

MaxInLine Size The Default document max inline size (bytes) setting can be found in the LiveCycle Administration console under Home, Settings, Core System, Core Configurations. Performance intensive applications that keep very large documents (>= 64K) on the heap are potentially subject to a heavy garbage collection penalty. These objects can quickly take up a large part of the heap and subsequently trigger frequent garbage collections. 64K is a typical file size boundary at which most applications perform well. Files larger than 64K are stored in the GDS, and files smaller are stored on the heap. Given the fact that each application will have different requirements, a good starting point is to choose a Max Inline Size representing the smallest files used more frequently. Consider an application with files of size 30K (used frequently), 40K (used frequently) and 1.7Meg (used less frequently). Max InLine Size of 64K causes 1.7 Meg files to stay in the GDS, and smaller files to be stored directly in the heap.

Document disposal time-out The period of time before a non-persistent Document object will time out and be automatically invalidated. This time is measured from the creation of the Document object. A default value for all newly created Document objects can be set from within the LiveCycle Administration console . This setting can be used to control disk space usage. A mandatory setting, the default value is 600 seconds and can be adjusted under “Home, Settings, Core System Settings, Configuration Settings” - “Default document disposal timeout (seconds)”.

Tuning Services LiveCycle provides some facilities to tune services at a general level. More extensive options are available specific to each service.

General Service Tuning Options Allow anonymous access to LCES services Authentication of transactions can be an overhead to optimal performance, but keep in mind allowing anonymous access also means that you are allowing access to some LiveCycle services without authentication, which may or may not be suitable. To allow anonymous access: 1.

Log in to LiveCycle Administration Console.

2.

Services, Applications and Services, Service Management

3.

Click on the [Service] (e.g. Forms Service / Output Service)

4.

Click the Security tab.

5.

In the Require callers to authenticate list, select No (the default is Yes).

6.

Save.

Creating a Service Endpoint Pool Every time a service is accessed, an instance of the service is created to respond to the service request. This applies to both services provided as part of LiveCycle (such as BarCodedForms), or custom services. If there are large number of CPU intense services, performance can be enhanced by creating a pool of instances to respond to requests more efficiently. This is generally not recommended for services shipped with LiveCycle because once the pool size is reached subsequent threads will block until a service becomes available.

13

For custom services, the approach is to perform testing to obtain the maximum throughput and calculate highest number of services that will be in use at any one time. A pool can be setup for those services to potentially improve performance. Configuration Pools may be configured via the LiveCycle Administration console under ‘services, Applications and Services, Service Management, [New Service], Pooling, Pooled Instances”. For additional help, refer to http://help.adobe.com/en_US/livecycle/9.0/adminHelp/000142.html#1586591

Specific Service Tuning Options Tuning Content Services Ehcache Caches in Content Services (ehcache) under load can use up to 512 MB of heap under the default configuration. The cache setting can be optimized based on the use cases and memory available. Tracing for ehcache can be enabled to identify which caches are less effective and could use additional heap. 1.

Stop LiveCycle server

2.

Go to C:\Adobe\Adobe LiveCycle ES2\deploy (or installation directory)

3.

Open adobe-contentservices.ear

4.

Open contentservices.war\WEB-INF\classes\log4j.properties

5.

log4j.logger.org.alfresco.repo.cache.EhCacheTracerJob=debug

6.

Repackage and redeploy adobe-contentservices.ear.

7.

Start LiveCycle server

Using these logs, the highly used caches can be identified and then tuned. This tuning can be done in the following file: adobe-contentservices.ear\contentservices.war\WEB-INF\classes\alfresco\extension\ehcache-custom.xml

A cache element looks like :