Stressing Linux with Real-World Workloads
Ottawa Linux Symposium 2003 Open Source Development Labs Mark Wong Member of the Technical Staff
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
1
Overview ● ● ● ●
Problems with currently available test workloads Solution: OSDL Database Test Suite Ways to use the workloads Future work
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
2
Problems ●
●
●
●
●
More open source tests are micro benchmarks focused on individual components, not overall system Not many open workloads available that utilize relational databases TPC and SPEC benchmarks costs money or require auditing Impractical to acquire equipment for TPC and SPEC benchmarks Some commercial RDBMS's restrict publication of performance results
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
3
Goals ●
To create database workloads that compare performance between kernels and not between hardware/software solutions ● ● ●
●
Utilizing open source databases Simulating real-world usage Configurable to equipment at hand
To make this available to kernel community
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
4
Solution ●
Database Test Suite ● ● ●
●
●
Database Test 1, Web Commerce Database Test 2, OLTP Database Test 3, Ad Hoc Decision Support
Patch Lifecycle Manager (PLM), repository for kernels and patches Scalable Test Platform (STP), automated test harness
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
5
Database Test Suite ●
● ●
Currently consists of three real-world database workloads using open-source databases Designed to meet the testing needs of Linux developers Exercises an entire system but configurable to available equipment and desired workload characteristics ● ● ●
●
I/O sub systems Memory utilization Processor utilization
Supports SAP DB and (soon) PostgreSQL
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
6
Database Test Suite Overview ● ● ●
Database Test 1 Database Test 2 Database Test 3
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
7
Database Test 1 (DBT-1) ●
●
●
Web commerce database workload simulating on-line bookseller Focus is on database system characteristics while simulating the effects of Web servers and Web caches Consists of: ● ● ● ●
●
Driver emulating Web browser requests Application Server External database cache Database server
Primary metrics, bogotransactions (BT) per second
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
8
DBT-1 Component Diagram
Database Server
Pool of database connections
Pool of database connections
Application Server
TPC/IP Connection per Emulated User
Driver
TCP/IP Connections per thread in Application Server pool
Database Cache
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
9
DBT-1 Options ● ● ● ● ●
Database scale Database parameters Number of emulated users Number of database connections Rate of interaction requests
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
10
Database Test Suite Overview ● ● ●
Database Test 1 Database Test 2 Database Test 3
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
11
Database Test 2 (DBT-2) ●
Simulates activities of supplier managing warehouse orders and stock (OLTP): ● ● ● ● ●
●
Workload generated by: ● ● ●
●
Creating an order (new-order) Paying for an order (payment) Displaying an order (order-display) Delivering an order (delivery) Checking warehouse stock levels (stock-level) Driver emulates terminals Connection manager (Client) Database server
Primary metrics, number of new-order transactions per minute (NOTPM)
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
12
DBT-2 Component Diagram
Database Server
Pool of database connections
Copyright 2003 © Open Source Development Labs, Inc.
Client
TCP/IP Connection per Terminal Emulated
Driver
July 23 - 26, 2003
13
DBT-2 Options ● ● ● ● ● ●
Database scale Database parameters Area of database touched by the driver Number of database transactions Rate of transactions Mix of transactions
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
14
Database Test Suite Overview ● ● ●
Database Test 1 Database Test 2 Database Test 3
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
15
Database Test 3 (DBT-3) ●
●
●
Ad hoc decision support workload simulating wholesale supplier managing world-wide sales and distribution Workload generated by scripts that execute a series of predefined queries and updates to the database Workload consists of three phases ● ● ●
●
Consists of: ● ●
●
Loading the database Power test with single stream of queries Throughput test with multiple streams of queries Scripts Database server
Primary metric, geometric mean of power and throughput test
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
16
DBT-3 Component Digram Database Connection
Database Server
Load Script
Data Generator Database Connection per Stream
Query Generator
Query Templates
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
17
DBT-3 Options (map to prev pic) ● ● ● ●
Database scale Database parameters Queries can be modified Number of query streams
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
18
Database Test Usage Examples Database Test DBT-1 (Web Commerse)
DBT-2 (OLTP)
DBT-3 (Ad Hoc DSS )
Parameters
RDBMS
Database Size
I/O Read Activity
I/O Write Activity Memory Activity
Processor Utilization
10K items, 10K users, 1-tier
SAP DB
5 GB
Light
Intermittently heavy for data, consistently moderate for db log
Intense 1 GB, no Near 100% paging or swapping
10K items, 10K users, 3-tier
SAP DB
5 GB
Light
Intermittently heavy for data, consistently moderate for db log
Intense 1 GB, no Near 100% paging or swapping
100 warehouses, cached, 1-tier
SAP DB
11 GB
Light
Light for data, very Intense 4 GB, no Medium to high, heavy for db log paging or proportional to I/O swapping throughput
100 warehouses, non-cached, 1tier
SAP DB
11 GB
Heavy
Intermittently heavy for data, consistently moderate for db log
Intense 4 GB, no Low to high, paging or proportional to I/O swapping throughput
S cale Factor 10
SAP DB
16 GB
Light to Heavy
Light to none.
Intense 4 GB, no Highly variable paging or swapping
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
19
Database Considerations ● ●
Different database can affect workload characteristics SAP DB ● ● ● ●
●
Uses a thread model Supports raw devices Manages its own cache Database data and logs can be assigned to multiple devices
PostgreSQL (in progress) ● ● ● ●
Uses a process model Does not support raw devices Relies on operating system for cache benefits Database logs can be redirected using symlinks
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
20
Database Test Suite Benefits ●
●
Provides real-world workloads that can be used on available equipment Exercises overall system based on available hardware: ● ● ●
●
Memory I/O Processors
Helps Linux get ready for the enterprise
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
21
How-to Use the Database Test Suite 1. Download the source code ●
●
Releases made on SourceForge.net, http://www.sourceforge.net/projects/osdldbt BitKeeper ● ● ●
bk://developer.osdl.org/dbt1 bk://developer.osdl.org/dbt2 bk://developer.osdl.org/dbt3
2. Use the Patch Lifecycle Manager (PLM) and the Scalable Test Platform (STP) ● ●
PLM stores Linux kernels and patches STP executes the Database Test Suite against Linux kernels in PLM
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
22
Setting up the Database Test Suite ● ●
Can be intimidating! Ask us for help:
[email protected]
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
23
PLM/STP Flow Diagram Load Distro
kernel.org PLM identifies Linux releases
STP pulls Linux Kernel and patches
PLM User Submits Linux Patch User
Build kernel Load Test Execute Test
E-mail
Results STP
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
24
Patch Lifecycle Manager ●
●
●
●
Submit Linux kernel patches to be tested on the Scalable Test Platform Patches can be applied to any stable or beta kernel release Patches can be applied to any previously submitted patch http://www.osdl.org/cgi-bin/plm/
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
25
Scalable Test Platform ● ●
Run against any Linux kernel in PLM Linux distrobutions currently avaiable ●
●
Run any test in the Database Test Suite ● ● ●
●
Redhat 7.1, 9.0 DBT-1 currently runs on 2-, 4-, 8-way systems DBT-2 currently runs on 4-way systems DBT-3 currently runs on 4-way systems
http://www.osdl.org/stp/
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
26
Future Work ●
● ●
Identify other database workloads that would be helpful to stress Linux with Porting of existing tests to other databases Applications of the Database Test Suite ● ● ● ●
●
High memory DMA I/O Scheduler Hyper-threading on x86 Modifying open source databases to take advantage of latest kernel features
Recruiting members of the community
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
27
Getting involved ● ●
Project home page, http://www.osdl.org/projects/performance/ Mailing lists ● Development topics,
[email protected] ● General performance topics,
[email protected]
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
28
Summary ●
Database Test Suite ● ● ● ●
●
Model after real-world workloads Designed to stress an entire Linux system in full or in part Download and use on available hardware Use PLM and STP
Looking for people to ● ●
Get involved in using the Database Test Suite Aid in porting to other open-source databases
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
29
Questions?
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
30
This slide intentionally left blank
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
31
extra slides
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
32
STP Output Examples • Examples of the web output for each database test from
• •
STP, hopefully the aacraid high dma will be fixed so i can show profile data without people pointing out that bounce buffers are killing me Update web output should be done before OLS too, so I'll remake them eventually Maybe I'll put in e-mail examples too
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
33
DBT-1 STP Output 1
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
34
DBT-1 STP Output 2
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
35
DBT-1 STP Output 3
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
36
DBT-1 STP Output 4
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
37
DBT-1 STP Output 5
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
38
DBT-1 STP Output 6
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
39
DBT-1 STP Output 7
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
40
DBT-1 STP Output 8
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
41
DBT-1 STP Output 9
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
42
DBT-2 STP Output 1
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
43
DBT-2 STP Output 2
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
44
DBT-3 STP Output
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
45
DBT-3 STP Output 2
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
46
DBT-3 STP Output 3
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
47
DBT-1 Workload Description (hide) • Fourteen simulated Web interactions involve: • • • • • •
Searching Listing best sellers, new products Ordering, order checking Selecting and purchasing items Updating inventory information Customer registration
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
48
DBT-2 Workload Description • Five transactions: • • • • •
Creating an order Paying for an order Displaying an order Delivering an order Checking warehouse stock levels
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
49
DBT-3 Workload Description • Twenty-two predefined queries run sequentially (in a •
stream) that provide business analyses for pricing and promotions, and supply and demand management Two functions, which can run in parallel to the 22 queries, load new sales information into the database
Copyright 2003 © Open Source Development Labs, Inc.
July 23 - 26, 2003
50
