1
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
5
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
6
<meta name="generator" content="Roller Weblogger 4.0.0.14u1 (BSC)">
7
<link rel="EditURI" type="application/rsd+xml" title="RSD" href="http://blogs.sun.com/ahl/rsd">
9
<link rel="alternate" type="application/atom+xml" title="Recent Entries (Atom)" href="http://blogs.sun.com/ahl/feed/entries/atom">
10
<link rel="alternate" type="application/rss+xml" title="Recent Entries (RSS)" href="http://blogs.sun.com/ahl/feed/entries/rss">
11
<link rel="alternate" type="application/atom+xml" title="Recent Comments (Atom)" href="http://blogs.sun.com/ahl/feed/comments/atom">
12
<link rel="alternate" type="application/rss+xml" title="Recent Comments (RSS)" href="http://blogs.sun.com/ahl/feed/comments/rss"><title>DTrace for Linux : Adam Leventhal's Weblog</title>
16
<link rel="stylesheet" type="text/css" media="all" href="dtrace_for_linux_files/pacifica-custom.css"></head><body bgcolor="#3c464d">
17
<div class="pagewrap">
18
<div class="innerpagewrap">
23
<table class="layout" cellpadding="0" cellspacing="0">
26
<!--Masthead - spans all table columns -->
27
<tr><td colspan="3"><div class="masthead">
28
<span class="blogdesc">inside the sausage factory</span>
29
<span class="blogname">Adam Leventhal's Weblog</span>
33
<tr><td class="subnav_left"></td>
35
<ul class="categories">
37
<a href="http://blogs.sun.com/ahl/feed/entries/atom">
38
<img src="dtrace_for_linux_files/feed-12x.gif">
42
<li><a href="http://blogs.sun.com/ahl/category/DTrace">DTrace</a></li>
43
<li><a href="http://blogs.sun.com/ahl/category/Fishworks">Fishworks</a></li>
44
<li><a href="http://blogs.sun.com/ahl/category/OpenSolaris">OpenSolaris</a></li>
45
<li><a href="http://blogs.sun.com/ahl/category/Other">Other</a></li>
46
<li><a href="http://blogs.sun.com/ahl/category/Software">Software</a></li>
47
<li><a href="http://blogs.sun.com/ahl/category/ZFS">ZFS</a></li>
50
<td class="subnav_right"></td></tr>
54
<td class="content_left"></td>
57
<div class="container aboutheader">
58
<div class="blogabout">
59
Adam Leventhal, Fishworks engineer
61
<div class="next-previous">
63
« <a href="http://blogs.sun.com/ahl/entry/opensolaris_on_lugradio">OpenSolaris on LugRa...</a> |
64
<a href="http://blogs.sun.com/ahl/">Main</a>
65
| <a href="http://blogs.sun.com/ahl/entry/dtrace_in_acm_queue">DTrace in ACM Queue</a> »
70
<div class="container">
74
<div class="dayheader">
75
<h2>Tuesday Dec 13, 2005</h2>
79
<div class="entryheader">
80
<h3 id="dtrace_for_linux"><a href="http://blogs.sun.com/ahl/entry/dtrace_for_linux">DTrace for Linux</a></h3>
82
<div class="entrybody">
85
With <a href="http://opensolaris.org/os/community/brandz">BrandZ</a>, it's now possible to use <a href="http://opensolaris.org/os/community/dtrace">DTrace</a> on
86
Linux applications. For the uninitiated, DTrace is the dynamic tracing facility
87
in <a href="http://opensolaris.org/">OpenSolaris</a>; it allows for <b>systemic</b> analysis of a scope and precision unequalled in the industry. With DTrace, administrators and
88
developers can trace low
89
level services like I/O and scheduling, up the system stack through
90
kernel functions calls, system calls, and system library calls, and into
91
applications written in C and C++ or any of a host of dynamic languages like
92
Java, Ruby, Perl or php. One of my contributions to BrandZ was to extend
93
DTrace support for Linux binaries executed in a branded Zone.
99
DTrace has several different <b>instrumentation providers</b> that
100
know how to instrument a particular part of the system and provide relevant
101
probes for that component. The <a href="http://docs.sun.com/app/docs/doc/817-6223/6mlkidllc?a=view">io provider</a> lets you trace disk
102
I/O, the <a href="http://docs.sun.com/app/docs/doc/817-6223/6mlkidljt?a=view">fbt (function boundary tracing) provider</a>
104
any kernel function call, etc. A typical system will start with more
105
than 30,000 probes but providers can create probes dynamically to trace
107
or user-land processes. When strictly focused on a user-land
109
most useful providers are typically the <a href="http://docs.sun.com/app/docs/doc/817-6223/6mlkidlk8?a=view">syscall provider</a> to
110
examine <a href="http://en.wikipedia.org/wiki/System_call">system calls</a> and
111
the <a href="http://docs.sun.com/app/docs/doc/817-6223/6mlkidlls?a=view">pid provider</a> that can trace any instruction in a any process
112
executing on the system.
118
For Linux processes, the pid provider just worked (well, once <a href="http://blogs.sun.com/rab">Russ</a> built a library to understand the Linux run-time linker),
119
and we introduced a new
120
provider -- the <b>lx-syscall provider</b>
121
-- to trace entry and return for
122
emulated Linux system calls. With these providers it's possible to
124
every facet of a Linux application's behavior and with the other DTrace
126
it's possible to reason about an application's use of system resources.
127
In other words, you can take that sluggish Linux application, stick it
129
Zone, dissect it using Solaris tools, and then bring it back to a
131
system with the fruits of your DTrace investigation<a name="ymmv_ref" href="#ymmv">[1]</a>.
137
To give an example of using DTrace on Linux applications, I needed an
138
application to examine. I wanted a well known program that either didn't run
139
on Solaris or operated sufficiently differently such examining the Linux
140
version rather than the Solaris port made sense. I decided on
141
<tt>/usr/bin/top</tt> partly because of the dramatic differences between how it
142
operates on Linux vs. Solaris (due to the differences in /proc), but mostly
143
because of what I've heard my colleague, <a href="http://blogs.sun.com/bmc">Bryan</a>, refer to as the
144
"top problem": your system is slow, so you run <tt>top</tt>. What's the top
151
Running <tt>top</tt> in the Linux branded zone, I opened a shell in the
152
global (Solaris) zone to use DTrace.
153
I started as I do on Solaris applications: I looked at system calls. I was
154
interested to see which system calls were being executed most frequently
155
which is easily expressed in DTrace:
161
</p><pre>bash-3.00# dtrace -n lx-syscall:::entry'/execname == "top"/{ @[probefunc] = count(); }'
162
dtrace: description 'lx-syscall:::entry' matched 272 probes
188
Note the use of the <a href="http://docs.sun.com/app/docs/doc/817-6223/6mlkidlh7?a=view">aggregation</a> denoted with the '@'.
189
Aggregations are the mechanism by which DTrace allows users to examine
190
<b>patterns</b> of system behavior rather than examining each individual datum
191
-- each system call for example.
193
(In case you also noticed the strange discrepancy between the number of open
194
and close system calls, many of those opens are failing so it makes sense that
195
they would have no corresponding close. I used the lx-syscall provider to suss
196
this out, but I omitted that investigation in a vain appeal for brevity.)
201
There may well be something fishy about this output, but nothing struck
202
me as so compellingly fishy to explore immediately. Instead, I fired up
204
short D script to see which system calls were taking the most time:
209
</p><pre>#!/usr/sbin/dtrace -s
214
self->ts = vtimestamp;
220
@[probefunc] = sum(vtimestamp - self->ts);
229
This script creates a table of system calls and the time spent in them (in
230
nanoseconds). The results were fairly interesting.
236
</p><pre>bash-3.00# ./lx-sys.d
237
dtrace: script './lx-sys.d' matched 550 probes
250
rt_sigaction 197646176
265
That seems like a lot of time spent in <b>munmap</b> for top. In fact, I'm
266
rather surprised that there's any mapping and unmapping going on at all (I
267
guess that should have raised an eyebrow after my initial system call count).
268
Unmapping memory is a pretty expensive operation that gets more expensive
269
on bigger systems as it requires the kernel to do some work on <b>every</b> CPU
270
to completely wipe out the mapping.
276
I then modified lx-sys.d to record the total amount of time top spent on the
277
CPU and the total amount of time spent in system calls to see how large a chunk
278
of time these seemingly expensive unmap operations were taking:
285
</p><pre>#!/usr/sbin/dtrace -s
290
self->ts = vtimestamp;
296
@[probefunc] = sum(vtimestamp - self->ts);
297
@["- all syscalls -"] = sum(vtimestamp - self->ts);
304
self->on = timestamp;
310
@["- total -"] = sum(timestamp - self->on);
318
I used the <a href="http://docs.sun.com/app/docs/doc/817-6223/6mlkidll6?a=view">sched provider</a> to see when top was going on and off
319
of the CPU, and I added a row to record the <b>total</b> time spent in all
320
system call. Here were the results (keep in mind I was just hitting ^C to
321
end the experiment after a few seconds so it's expected that these numbers
322
would be different from those above; there are ways to have more accurately
327
</p><pre>bash-3.00# ./lx-sys2.d
328
dtrace: script './lx-sys2.d' matched 550 probes
341
rt_sigaction 33207341
349
- all syscalls - 1589236337
357
So system calls are consuming roughly half of top's time on the CPU and
358
the munmap syscall is consuming roughly a quarter of that. This was enough
359
to convince me that there was probably room for improvement and further
360
investigation might bear fruit.
366
Next, I wanted to understand what this mapped memory was being used for so
367
I wrote a little script that traces all the functions called in the process
368
between when memory is mapped using the mmap2(2) system call and when it's
369
unmapped and returned to the system through the munmap(2) system call:
376
</p><pre>#!/usr/sbin/dtrace -s
378
#pragma D option quiet
380
lx-syscall::mmap2:return
386
printf("%*.s <- %s syscall\n", self->depth, "", probefunc);
393
printf("%*.s -> %s`%s\n", self->depth, "", probemod, probefunc);
399
printf("%*.s <- %s`%s\n", self->depth, "", probemod, probefunc);
403
lx-syscall::munmap:entry
404
/arg0 == self->ptr/
407
printf("%*.s -> %s syscall\n", self->depth, "", probefunc);
418
This script uses the $target variable which means that we need to run it with
419
the -p <pid> option where <pid> is the process ID of top. When mmap2 returns,
420
we set a thread local variable, 'ptr', which stores the address at the base
421
of the mapped region; for every function entry and return in the process we
423
call printf() if <tt>self->ptr</tt> is set; finally, we exit DTrace when
424
munmap is called with that same address. Here are the results:
430
</p><pre>bash-3.00# ./map.d -p `pgrep top`
432
<- LM2`libc.so.1`syscall
433
<- LM2`lx_brand.so.1`lx_emulate
434
<- LM2`lx_brand.so.1`lx_handler
436
<- libc.so.6`malloc
437
-> libc.so.6`memset
438
<- libc.so.6`memset
439
-> libc.so.6`cfree
440
<- libc.so.6`cfree
441
-> libc.so.6`munmap
442
<- LM2`lx_brand.so.1`lx_handler_table
443
-> LM2`lx_brand.so.1`lx_handler
444
-> LM2`lx_brand.so.1`lx_emulate
445
-> LM2`libc.so.1`syscall
452
I traced the probemod (shared object name) in addition to probefunc (function
453
name) so that we'd be able to differentiate proper Linux functions (in this
454
case all in libc.so.6) from functions in the emulation library
455
(LM2`lx_brand.so.1). What we can glean from this is that the mmap call is a
456
result of a call to malloc() and the unmap is due to a call to free(). What's
457
unfortunate is that we're not seeing any function calls in top itself. For some
458
reason, the top developer chose to strip this binary (presumably to save
459
precious 2k the symbol table would have used on disk) so we're stuck with no symbolic information for top's
460
functions and no ability to trace the individual function calls<a name="dash_ref" href="#dash">[2]</a>, but we can
461
still reason about this a bit more.
465
A little analysis in <a href="http://opensolaris.org/os/community/mdb/">mdb</a> revealed that cfree (an alias for free)
466
makes a tail-call to a function that calls munmap. It seems strange to me that
467
freeing memory would immediately results in an unmap operation (since it would
468
cause exactly the high overhead we're seeing here. To explore this, I wrote
469
another script which looks at what proportion of calls to malloc result in a
476
</p><pre>#!/usr/sbin/dtrace -s
478
pid$target::malloc:entry
480
self->follow = arg0;
483
lx-syscall::mmap2:entry
486
@["mapped"] = count();
490
pid$target::malloc:return
493
@["no map"] = count();
501
Here are the results:
506
</p><pre>bash-3.00# ./malloc.d -p `pgrep top`
507
dtrace: script './malloc.d' matched 11 probes
519
So a bunch of allocations result in a mmap, but not a huge number. Next I
520
decided to explore if there might be a correlation between the size of the
521
allocation and whether or not it resulted in a call to mmap using the following
529
</p><pre>#!/usr/sbin/dtrace -s
531
pid$target::malloc:entry
533
self->size = arg0;
536
lx-syscall::mmap2:entry
539
@["mapped"] = quantize(self->size);
543
pid$target::malloc:return
546
@["no map"] = quantize(self->size);
555
Rather than just counting the frequency, I used the <a href="">quantize
556
aggregating action</a> to built a power-of-two histogram on the number of bytes
557
being allocated (<tt>self->size</tt>). The output was quite illustrative:
562
</p><pre>bash-3.00# ./malloc2.d -p `pgrep top`
563
dtrace: script './malloc2.d' matched 11 probes
567
value ------------- Distribution ------------- count
570
8 |@@@@@@@@@@@@@@@ 852
580
value ------------- Distribution ------------- count
582
262144 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 213
590
All the allocations that required a mmap were <b>huge</b> -- between 256k and
591
512k. Now it makes sense why the Linux libc allocator would treat these
592
allocations a little differently than reasonably sized allocations. And this is
593
clearly a smoking gun for top performance: it would do much better to
594
preallocate a huge buffer and grow it as needed (assuming it actually needs it
595
at all) than to malloc it each time. Tracking down the offending line of code
596
would just require a non-stripped binary and a little DTrace invocation like
602
</p><pre># dtrace -n pid`pgrep top`::malloc:entry'/arg0 >= 262144/{@[ustack()] = count()}'
608
From symptoms to root cause on a Linux application in a few DTrace scripts --
609
and it took me approximately 1000 times longer to cobble together some vaguely
610
coherent prose describing the scripts than it did for me to actually perform
611
the investigation. BrandZ opens up some pretty interesting new vistas for
612
DTrace. I look forward to seeing Linux applications being brought in for
613
tune-ups on BrandZ and then reaping those benefits either back on their
614
mother Linux or sticking around to enjoy the <a href="http://opensolaris.org/os/community/fm/">fault management</a>, <a href="http://opensolaris.org/os/community/zfs/">ZFS</a>,
615
scalability, and, of course, continued access to DTrace in BrandZ.
622
<a name="ymmv" href="#ymmv_ref">[1]</a> Of course, results may vary since the guts of the Linux kernel differ
623
significantly from those of the Solaris kernel, but they're often fairly
624
similar. I/O or scheduling problems will be slightly different, but often
625
not <i>so</i> different that the conclusions lack applicability.
627
<a name="dash" href="#dash_ref">[2]</a> Actually, we <b>can</b> can
628
still trace function calls -- in fact, we can trace any instruction --
629
but it takes something of a heroic effort. We could disassemble parts
630
of top to identify calls sites and then use esoteric <tt>pid123::-:<i>address</i></tt> probe format to trace the stripped function. I said we could do it; I never said it would be pretty.
636
<a href="http://technorati.com/tag/BrandZ" rel="tag">BrandZ</a>
637
<a href="http://technorati.com/tag/DTrace" rel="tag">DTrace</a>
638
<a href="http://technorati.com/tag/Solaris" rel="tag">Solaris</a>
639
<a href="http://technorati.com/tag/OpenSolaris" rel="tag">OpenSolaris</a>
642
<div class="entryfooter">
644
Posted at 01:00PM Dec 13, 2005
645
by ahl in <span class="category">DTrace</span>
646
| <a href="http://blogs.sun.com/ahl/entry/dtrace_for_linux#comments" class="commentsLink">Comments[9]</a>
647
| <a href="http://blogs.sun.com/ahl/entry/dtrace_for_linux">Permalink</a>
655
<!-- Begin SiteCatalyst code version: G.5. -->
656
<script type="text/javascript" language="JavaScript">
658
if(typeof s_channel=='undefined'){var s_channel="ahl";}
660
<script type="text/javascript" language="JavaScript" src="dtrace_for_linux_files/s_code_remote.js"></script><script type="text/javascript" src="dtrace_for_linux_files/metrics_group1.js"></script>
661
<!-- End SiteCatalyst code version: G.5. -->
663
<div class="comments">
664
<a name="comments"></a>
665
<div class="comments" id="comments">
667
<div class="comments-head">Comments:</div>
670
<a name="comment-1134576732000" id="comment-1134576732000"></a>
671
<div class="comment odd" id="comment1">
673
Awsome! I've been waiting to try out dtrace for a long time now. Thanks!
675
<p class="comment-details">
679
on December 14, 2005 at 08:12 AM PST
681
<a href="http://blogs.sun.com/ahl/entry/dtrace_for_linux#comment-1134576732000" class="entrypermalink" title="comment permalink">#</a>
686
<a name="comment-1134622184000" id="comment-1134622184000"></a>
687
<div class="comment even" id="comment2">
689
Rather than jump to profiling an application I'd
690
much perfer to see the results of a three
691
way comparison of the libMicro benchmark running
692
on the same machine, Solaris/Linux/Linux_translated.
693
At least this will give us an idea of how much
694
overhead the translator has on system calls
695
in a controlled condition. If there is a large
696
anomaly a dtrace session would be interesting.
701
<p class="comment-details">
705
on December 14, 2005 at 08:49 PM PST
707
<a href="http://blogs.sun.com/ahl/entry/dtrace_for_linux#comment-1134622184000" class="entrypermalink" title="comment permalink">#</a>
712
<a name="comment-1134631985000" id="comment-1134631985000"></a>
713
<div class="comment odd" id="comment3">
716
That would be a different investigation entirely -- one I've started to
717
do and will continue to do, no doubt -- but the point was to
718
demonstrate how performance of a Linux application could be improved
719
not that of the emulation environment. <p class="comment-details">
721
<a rel="nofollow" href="http://blogs.sun.com/ahl"><b>Adam Leventhal</b></a>
723
on December 14, 2005 at 11:33 PM PST
725
<a href="http://blogs.sun.com/ahl/entry/dtrace_for_linux#comment-1134631985000" class="entrypermalink" title="comment permalink">#</a>
730
<a name="comment-1134653721000" id="comment-1134653721000"></a>
731
<div class="comment even" id="comment4">
732
Agreed :) Its nice to see dtrace coming to linux. My only problem
733
currently, is that the documentation for installing Brandz on a linux
734
box is not very clear and i am somewhat confused on what to do with the
735
required 2 packages on a Fedora Core server. <p class="comment-details">
737
<a rel="nofollow" href="http://hunter.userfriendly.net/"><b>Michael Weiner</b></a>
739
on December 15, 2005 at 05:35 AM PST
741
<a href="http://blogs.sun.com/ahl/entry/dtrace_for_linux#comment-1134653721000" class="entrypermalink" title="comment permalink">#</a>
746
<a name="comment-1134760917000" id="comment-1134760917000"></a>
747
<div class="comment odd" id="comment5">
748
Adam, This is great work, thanks.
749
Couple of observations: Looks like Linux is going to have a tool to do
750
the similar analysis natively. Here is the link to their version of the
752
http://sourceware.org/ml/systemtap/2005-q4/msg00437.html
753
Looking at the Linux native analysis, the problem you have experienced
754
doesn't seem to be there when ran natively. That brings to my mind a
755
question, is it really true that Solaris virtually environment and
756
native Linux are so close that one can do performance problem analysis
757
in one environment and apply in the other. Your thoughts on this and if
758
possibly explanation of the difference in results would be appreciated.
759
(I understand that the details of the machine configuration and
760
software versions are missing hence that might be the difference in the
761
results.) <p class="comment-details">
765
on December 16, 2005 at 11:21 AM PST
767
<a href="http://blogs.sun.com/ahl/entry/dtrace_for_linux#comment-1134760917000" class="entrypermalink" title="comment permalink">#</a>
772
<a name="comment-1134978306000" id="comment-1134978306000"></a>
773
<div class="comment even" id="comment6">
776
If there's something in particular that needs clarification, could you
777
post a question to brandz-discuss@opensolaris.org? We're looking for
778
feedback from the community.
781
Johnd,<br>As you point out, differences in the hardware and
782
software configurations could account for the difference in the
783
results. It's possible that the performance problem could be exagerated
784
in the BrandZ environment, but this result is still valid in that
785
mapping and unmapping memory is surely not the most efficient way for
786
the allocator to behave in this situation. We don't have much data to
787
support this claim, but I do think analysis under BrandZ will yeild
788
results which are applicable on a native Linux system. The results will
789
tend to be more accurate for investigations that deal strictly with
790
application code rather than interactions with the kernel. What I
791
didn't see in that email thread you referenced was any sort of analysis
792
of the user-land flow control (map.d and malloc.d above). Being able to
793
follow your investigation accross the user/kernel boundary and into
794
applications (in a bunch of different languages) is one of the key
795
differences between DTrace and every other tracing framework. Without
796
user-land tracing even if their analysis of the system call timing had
797
matched mine, it's not clear that they would have been able to drive to
798
the root cause. <p class="comment-details">
800
<a rel="nofollow" href="http://blogs.sun.com/ahl"><b>Adam Leventhal</b></a>
802
on December 18, 2005 at 11:45 PM PST
804
<a href="http://blogs.sun.com/ahl/entry/dtrace_for_linux#comment-1134978306000" class="entrypermalink" title="comment permalink">#</a>
809
<a name="comment-1134982710000" id="comment-1134982710000"></a>
810
<div class="comment odd" id="comment7">
811
I havnt seen anything here that wouldnt have been possible with a
812
combination of strace and ltrace. And I wouldnt have had to write any
813
scripts to use them either. Congratulations on writing a cool utility,
814
but this stuff is nothing new, and certainly not only possible on
815
solaris. <p class="comment-details">
819
on December 19, 2005 at 12:58 AM PST
821
<a href="http://blogs.sun.com/ahl/entry/dtrace_for_linux#comment-1134982710000" class="entrypermalink" title="comment permalink">#</a>
826
<a name="comment-1134997198000" id="comment-1134997198000"></a>
827
<div class="comment even" id="comment8">
830
Perhaps you're right about these specific examples, but DTrace lets you
831
do far more than strace and ltrace. For example, you can mix data from
832
the kernel, system libraries and Java in a single stream as I describe <a href="http://blogs.sun.com/roller/page/ahl?entry=open_sourcing_the_javaone_keynote" rel="nofollow">here</a>.
833
You can't do that in Linux or on any other operating system for that
834
matter. And that's just one example. You may want to understand a bit
835
more about what DTrace is capable of before you discount it. <p class="comment-details">
837
<a rel="nofollow" href="http://blogs.sun.com/"><b>Adam Leventhal</b></a>
839
on December 19, 2005 at 04:59 AM PST
841
<a href="http://blogs.sun.com/ahl/entry/dtrace_for_linux#comment-1134997198000" class="entrypermalink" title="comment permalink">#</a>
846
<a name="comment-1135140694000" id="comment-1135140694000"></a>
847
<div class="comment odd" id="comment9">
849
To avoid a stripped binary, build and install
852
<tt>make CFLAGS=-g3 install</tt>
854
I think I have a fairly good handle on problems
855
actually seen on Linux. The munmap() problem does
856
not exist; the name "Slowaris" exists for a reason.
857
(be thankful, because performance problems give
858
Sun a reason to pay your salary)
860
Every system has it's cruddy part though, and /proc is surely where Linux collects cruft.
862
<b>What top is required to do is quite ugly:</b>
864
First, it must support up to 4 million tasks.
865
Probably top can only do this at a reduced
866
refresh rate, but anyway, this means that top
867
can't leave the file descriptors open.
869
The per-task info is about 400 to 600 bytes.
870
This is not nice to the cache or TLB. At only
871
1000 tasks, that's half a meg of RAM to cache
872
and a full 128 TLB entries. Ouch. That's not
873
anywhere near 4 million tasks! Just playing around with the order of fields in a struct can make a huge difference for top.
875
Then there is the problem of doing lots of
876
64-bit division on a register-starved 32-bit
877
processor. This happens on both sides of the
878
user/kernel boundry, creating and parsing
879
numbers as ASCII decimal. Timestamps also
880
require 64-bit division.
882
A custom stack-like allocator would help.
883
The glibc obstacks stuff would work great.
884
(hey, add that to Solaris and send it off
885
to POSIX to be set in stone as a standard)
887
Smaller data structures would sure help,
888
in the short run at least, but top may be
889
getting support for more data columns in
890
the future. This also reduces code sharing
891
with the ps program, making maintenence hard.
894
</p><p class="comment-details">
896
<a rel="nofollow" href="http://procps.sf.net/"><b>Albert Cahalan</b></a>
898
on December 20, 2005 at 08:51 PM PST
900
<a href="http://blogs.sun.com/ahl/entry/dtrace_for_linux#comment-1135140694000" class="entrypermalink" title="comment permalink">#</a>
907
<div class="comments-form">
908
<div class="comments-head">Post a Comment:</div>
909
<a name="comment-form"></a>
911
<span class="status">Comments are closed for this entry.</span>
917
<div class="sidebar">
920
<div class="navsect">
921
<div class="navhead">
924
<div class="navbody">
925
<div class="hCalendarTable"><table summary="Blog Archive Calendar" class="hCalendarTable" border="0" cellspacing="0"><tbody><tr><td colspan="7" class="hCalendarMonthYearRow" align="center"><a href="http://blogs.sun.com/ahl/date/200902" title="Prev" class="hCalendarNavBar">«</a> March 2009</td></tr><tr><th class="hCalendarDayNameRow" align="center">Sun</th><th class="hCalendarDayNameRow" align="center">Mon</th><th class="hCalendarDayNameRow" align="center">Tue</th><th class="hCalendarDayNameRow" align="center">Wed</th><th class="hCalendarDayNameRow" align="center">Thu</th><th class="hCalendarDayNameRow" align="center">Fri</th><th class="hCalendarDayNameRow" align="center">Sat</th></tr><tr><td class="hCalendarDay"><div class="hCalendarDayTitle">1</div></td><td class="hCalendarDayLinked"><div class="hCalendarDayTitle"><a href="http://blogs.sun.com/ahl/date/20090302">2</a></div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">3</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">4</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">5</div></td><td class="hCalendarDayLinked"><div class="hCalendarDayTitle"><a href="http://blogs.sun.com/ahl/date/20090306">6</a></div></td><td class="hCalendarDayLinked"><div class="hCalendarDayTitle"><a href="http://blogs.sun.com/ahl/date/20090307">7</a></div></td></tr><tr><td class="hCalendarDay"><div class="hCalendarDayTitle">8</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">9</div></td><td class="hCalendarDayLinked"><div class="hCalendarDayTitle"><a href="http://blogs.sun.com/ahl/date/20090310">10</a></div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">11</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">12</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">13</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">14</div></td></tr><tr><td class="hCalendarDay"><div class="hCalendarDayTitle">15</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">16</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">17</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">18</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">19</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">20</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">21</div></td></tr><tr><td class="hCalendarDay"><div class="hCalendarDayTitle">22</div></td><td class="hCalendarDayCurrent"><div class="hCalendarDayTitle">23</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">24</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">25</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">26</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">27</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">28</div></td></tr><tr><td class="hCalendarDay"><div class="hCalendarDayTitle">29</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">30</div></td><td class="hCalendarDay"><div class="hCalendarDayTitle">31</div></td><td class="hCalendarDayNotInMonth"> </td><td class="hCalendarDayNotInMonth"> </td><td class="hCalendarDayNotInMonth"> </td><td class="hCalendarDayNotInMonth"> </td></tr><tr><td class="hCalendarDayNotInMonth"> </td><td class="hCalendarDayNotInMonth"> </td><td class="hCalendarDayNotInMonth"> </td><td class="hCalendarDayNotInMonth"> </td><td class="hCalendarDayNotInMonth"> </td><td class="hCalendarDayNotInMonth"> </td><td class="hCalendarDayNotInMonth"> </td></tr><tr class="hCalendarNextPrev"><td colspan="7" align="center"><a href="http://blogs.sun.com/ahl/" class="hCalendarNavBar">Today</a></td></tr></tbody></table>
932
<div class="navsect">
933
<div class="navbody">
934
<form id="searchForm" method="get" action="http://search.sun.com/blogs/index.jsp" style="margin: 0pt; padding: 0pt;">
935
<input name="weblog" value="ahl" type="hidden">
937
<input name="qt" size="10" maxlength="255" type="text">
938
<input value="Search" type="submit">
946
<div class="navsect">
947
<div class="navhead">
948
<h3>The DTrace Three</h3>
950
<div class="navbody">
952
<li class="rFolderItem">
953
<a href="http://blogs.sun.com/ahl" title="Adam Leventhal" class="rBookmark0">Adam Leventhal</a>
955
<li class="rFolderItem">
956
<a href="http://blogs.sun.com/bmc" title="Bryan Cantrill" class="rBookmark0">Bryan Cantrill</a>
958
<li class="rFolderItem">
959
<a href="http://blogs.sun.com/mws" title="Mike Shapiro" class="rBookmark0">Mike Shapiro</a>
964
<div class="navsect">
965
<div class="navhead">
966
<h3>DTrace Links</h3>
968
<div class="navbody">
970
<li class="rFolderItem">
971
<a href="http://www.infoworld.com/article/05/08/01/31FEinnovator10_1.html" title="InfoWorld Innovators 2005" class="rBookmark0">InfoWorld Innovators 2005</a>
973
<li class="rFolderItem">
974
<a href="http://www.opensolaris.org/os/community/dtrace/" title="DTrace Home" class="rBookmark0">DTrace Home</a>
976
<li class="rFolderItem">
977
<a href="http://blogs.sun.com/ahl/dtracesched" title="DTrace Schedule" class="rBookmark0">DTrace Schedule</a>
979
<li class="rFolderItem">
980
<a href="http://www.opensolaris.org/jive/forum.jspa?forumID=7" title="Discussion Forum" class="rBookmark0">Discussion Forum</a>
982
<li class="rFolderItem">
983
<a href="http://see.sun.com/Apps/DCS/mcp?r=703SsI4297t0006304e4m32f403gpD04299i0" title="Expert Exchange" class="rBookmark0">Expert Exchange</a>
985
<li class="rFolderItem">
986
<a href="http://www.linuxinsider.com/story/The-Importance-of-Solaris-10-37993.html" title="LinuxInsider story" class="rBookmark0">LinuxInsider story</a>
988
<li class="rFolderItem">
989
<a href="http://news.com.com/2100-7344_3-5374412.html" title="NY Times Solaris story" class="rBookmark0">NY Times Solaris story</a>
991
<li class="rFolderItem">
992
<a href="http://www.theregister.co.uk/2004/07/08/dtrace_user_take/" title="Register story" class="rBookmark0">Register story</a>
994
<li class="rFolderItem">
995
<a href="http://www.samag.com/documents/s=9171/sam0406h/0406h.htm" title="SysAdmin story" class="rBookmark0">SysAdmin story</a>
997
<li class="rFolderItem">
998
<a href="http://www.eweek.com/article2/0,1759,1626474,00.asp?kc=EWRSS03129TX1K0000612" title="eWEEK story" class="rBookmark0">eWEEK story</a>
1000
<li class="rFolderItem">
1001
<a href="http://www.sun.com/2004-0518/feature/index.html" title="sun.com story" class="rBookmark0">sun.com story</a>
1006
<div class="navsect">
1007
<div class="navhead">
1008
<h3>Fishworks Engineering</h3>
1010
<div class="navbody">
1011
<ul class="rFolder">
1012
<li class="rFolderItem">
1013
<a href="http://blogs.sun.com/wdp" title="" class="rBookmark0">Bill Pijewski</a>
1015
<li class="rFolderItem">
1016
<a href="http://blogs.sun.com/brendan" title="" class="rBookmark0">Brendan Gregg</a>
1018
<li class="rFolderItem">
1019
<a href="http://blogs.sun.com/bmc" title="" class="rBookmark0">Bryan Cantrill</a>
1021
<li class="rFolderItem">
1022
<a href="http://blogs.sun.com/cindi" title="" class="rBookmark0">Cindi McGuire</a>
1024
<li class="rFolderItem">
1025
<a href="http://blogs.sun.com/dap" title="" class="rBookmark0">Dave Pacheco</a>
1027
<li class="rFolderItem">
1028
<a href="http://blogs.sun.com/eschrock" title="" class="rBookmark0">Eric Schrock</a>
1030
<li class="rFolderItem">
1031
<a href="http://blogs.sun.com/greg" title="" class="rBookmark0">Greg Price</a>
1033
<li class="rFolderItem">
1034
<a href="http://blogs.sun.com/wesolows" title="" class="rBookmark0">Keith Wesolowski</a>
1036
<li class="rFolderItem">
1037
<a href="http://blogs.sun.com/mws" title="" class="rBookmark0">Mike Shapiro</a>
1039
<li class="rFolderItem">
1040
<a href="http://blogs.sun.com/tmp" title="" class="rBookmark0">Todd Patrick</a>
1048
<div class="navsect">
1049
<div class="navhead">
1050
<h3>Recent Posts</h3>
1052
<div class="navbody">
1053
<ul class="rEntriesList">
1054
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/ssd_announcement">SSDs for HSPs</a></li>
1055
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/hsp_wish_2009">Presentation: Hybrid Storage Pools and SSDs</a></li>
1056
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/fishworks_vm">Fishworks VM: the 7000 series on your laptop</a></li>
1057
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/more_anarchy">More from the storage anarchist</a></li>
1058
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dancing_with_the_anarchist">Dancing with the Anarchist</a></li>
1059
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/hsp_talk_at_the_opensolaris">HSP talk at the OpenSolaris Storage Summit</a></li>
1060
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/flash_workshop_at_asplos">Flash workshop at ASPLOS</a></li>
1061
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/shadow_of_hsp">Casting the shadow of the Hybrid Storage Pool</a></li>
1062
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/sun_storage_7410_space_calculator">Sun Storage 7410 space calculator</a></li>
1063
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/fishworks_launch">Hybrid Storage Pools in the 7410</a></li>
1064
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/hsp_goes_glossy">Hybrid Storage Pool goes glossy</a></li>
1065
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/apple_updates_dtrace_again">Apple updates DTrace... again</a></li>
1066
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/glimpse_into_netapp_s_flash">A glimpse into Netapp's flash future</a></li>
1067
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/hybrid_storage_pools_the_l2arc">Hybrid Storage Pools: The L2ARC</a></li>
1068
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/hybrid_storage_pools_in_cacm">Hybrid Storage Pools in CACM</a></li>
1069
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/flash_hybrid_pools_and_future">Flash, Hybrid Pools, and Future Storage</a></li>
1070
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/apple_updates_dtrace">Apple updates DTrace</a></li>
1071
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_conf_post_post_mortem">dtrace.conf post-post-mortem</a></li>
1072
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_at_javaone_no_more">DTrace and JavaOne: The End of the Beginning</a></li>
1073
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/expand_o_matic_raid_z">Expand-O-Matic RAID-Z</a></li>
1074
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/pid2proc_for_dtrace">pid2proc for DTrace</a></li>
1075
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/mac_os_x_and_the">Mac OS X and the missing probes</a></li>
1076
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_firefox_leopard">DTrace/Firefox/Leopard</a></li>
1077
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/what_if_machine_dtrace_port">What-If Machine: DTrace Port</a></li>
1078
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_knockoffs">DTrace Knockoffs</a></li>
1079
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_for_ruby_at_oscon">DTrace for Ruby at OSCON 2007</a></li>
1080
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_scobleized">DTrace "Scobleized"</a></li>
1081
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/iscsi_dtrace_provider_and_other">iSCSI DTrace provider and more to come</a></li>
1082
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_javaone_2007">DTrace @ JavaOne 2007</a></li>
1083
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/java_dtrace_article">Java/DTrace article</a></li>
1084
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/linux_defection">Linux Defection</a></li>
1085
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/gzip_for_zfs_update">gzip for ZFS update</a></li>
1086
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/a_little_zfs_hack">a small ZFS hack</a></li>
1087
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/on_testing">It's tested or it's broken</a></li>
1088
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace%3A_a_history">DTrace: a history</a></li>
1089
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_is_a_web_developer">DTrace is a web developer's best friend</a></li>
1090
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_user_number_one">DTrace user number one</a></li>
1091
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_on_mac_os_x">Apple's DTrace team</a></li>
1092
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/double_parity_raid_z">Double-Parity RAID-Z</a></li>
1093
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_on_geek_muse">DTrace on Geek Muse</a></li>
1094
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/javaone_dtrace_session_2006">JavaOne DTrace Session 2006</a></li>
1095
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_caption_contest">DTrace caption contest</a></li>
1096
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_at_javaone_2006">DTrace at JavaOne 2006</a></li>
1097
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/user_land_tracing_gets_better">User-land tracing gets better and better</a></li>
1098
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_at_scale_4x">DTrace at SCALE 4x</a></li>
1099
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_in_acm_queue">DTrace in ACM Queue</a></li>
1100
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/dtrace_for_linux">DTrace for Linux</a></li>
1101
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/opensolaris_on_lugradio">OpenSolaris on LugRadio</a></li>
1102
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/fall_of_code">Fall of Code?</a></li>
1103
<li class="recentposts"><a href="http://blogs.sun.com/ahl/entry/opensolaris_and_svk">OpenSolaris and svk</a></li>
1108
<div class="navsect">
1109
<div class="navhead">
1110
<h3>Other Blog Links</h3>
1112
<div class="navbody">
1113
<ul class="rNavigationBar">
1114
<li class="rNavItem">
1115
<a href="http://blogs.sun.com/"><span>blogs.sun.com</span></a>
1117
<li class="rNavItem">
1118
<a href="http://blogs.sun.com/ahl/"><span>Weblog</span></a>
1120
<li class="rNavItem"><a href="http://blogs.sun.com/ahl/page/dtracesched"><span>DTrace Schedule</span></a></li>
1121
<li class="rNavItem">
1122
<a href="http://blogs.sun.com/roller-ui/login-redirect.rol"><span>Login</span></a>
1131
<div class="affinity"></div>
1135
<div class="navsect">
1136
<div class="navbody">
1137
<div class="rFolderitem">
1138
<p>Today's Page Hits: 1571</p>
1143
<script type="text/javascript">
1144
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
1145
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
1146
</script><script src="dtrace_for_linux_files/ga.js" type="text/javascript"></script>
1147
<script type="text/javascript">
1148
var pageTracker = _gat._getTracker("UA-1092717-1");
1149
pageTracker._trackPageview();
1155
<td class="content_right"></td></tr>
1157
<!-- Footer - spans all columns -->
1158
<tr><td colspan="3" class="footer"></td></tr>
b'\\ No newline at end of file'
added
removed
Development/debugging/profiling/dtrace/dtrace_for_linux.html
