Dtrace – Dynamic System Tracing

● ● ●

David Cole

Sun Microsystems Jan 31st 2008

Overview • Tracing of the kernel, user level programs and libraries • Perfect for performance investigation and troubleshooting • Over 50,000 points in the kernel & limitless user level points can be traced • System is dynamically modified to enable trace points • Safe to use on production systems • Zero performance hit when disabled and when enabled you only take a hit on the trace points you have enabled.

2

Probes and Providers • Probes are tracing points of interest. • Probes are kernel activities or locations within the kernel, user programs or libraries. Each probe has a name. • Probes are made available by providers, and are arranged into modules and functions. Some of the standard providers include: syscall, io, vminfo, & pid • Probe naming: provider:module:function:name • Probe examples: > The tick-n probe fires every fixed interval. > syscall::read:entry > syscall::open:return 3

D Language • Programming language used to define probe usage and the actions to take. • D language is similar to C. The definitions of actions are similar to awk. $ cat open.d ●#!/usr/sbin/dtrace -qs ●

●

syscall::open:entry { ● printf(“%s filename=%s\n”, execname, copyinstr(arg0)) ●} ● ●

●

$ ./open.d in.tftpd filename=CoronaP2.parms ●metacity filename=/home/ben/.icons/default/cursors/bottom_side ●metacity filename=/home/ben/.icons/default/index.theme ●evolution filename=/home/jeff/.evolution/mail/imap/jeff/folders/INBOX/160988 ●jre filename=/home/lewis/.icons/default/cursors/watch ●jre filename=/home/lewis/.icons/default/index.theme ●metacity filename=/usr/share/icons/default/cursors/bottom_side ● ●

● ● ●

4

Representationof data • Raw data may not be useful without post-processing. • Data can be aggregated (where single pieces of data are not useful) to allow patterns to be identified. • In-built aggregation functions – count, sum, average etc. $ cat open_aggr.d ●#!/usr/sbin/dtrace -qs ●

●

syscall::open:entry ●{ ● @open_aggr[execname] = count(); ●} ●

●

$ ./open_aggr.d ^C

● ● ● ● ● ● ● ● ● ● ●

java ls nscd pidgin firefox-bin metacity automountd

2 6 10 12 37 72 78

5

And the rest.. • Dtrace has actions to record user stack traces within the kernel, user space programs, Java and Python. • Additional providers out there: > > > >

Java Python Ruby Perl

• As Solaris is Open Source, Dtrace has been implemented on Mac OS X 10.5 and FreeBSD (ongoing). • To learn more: > The Solaris Dynamic Tracing (DTrace) Guide -

http://docs.sun.com/app/docs/doc/817-6223 > Dtrace Portal : http://www.sun.com/bigadmin/content/dtrace

6

David Cole [email protected]

Sub Project: Geographic Integration Sponsors: Keith Milan, John Sprague

David Cole

[email protected]