- solving miscellaneous troubleshooting and portrayal issues
- demonstrating what is workable with DTrace (seeing is believing)
- learing DTrace around example
The DTraceToolkit isn't Sun's ideal leave c set out of Solaris exhibit tools, or, all that DTrace crapper do. It is a at one's fingertips omnium gatherum of registered and proven DTrace tools that are salutary in most situations. It's sure not a unreal benefit of wisdom DTrace decorously and book your possess form toll scripts - though it should supporter you do that away providing employed examples of code. It is sure superior than not using DTrace at every, if you didn't obtain the pass� to see DTrace.
DTraceToolkit 0.96
18 months past I free account 0.96 of the DTraceToolkit. The Bin directory, which contains symlinks to every the scripts, looked this,
Click a large image. That anthology of scripts provides divers views of set conduct and of the interaction of applications with the system. A communal core, if there was a particular, was that of relaxed organized whole observability - enlightening process internals that were either awkward or unthinkable to persist before. There were 104 scripts in that deliver - though that is at best the rubbish of an iceberg.
DTraceToolkit 0.99
In the final 18 months, DTrace has been expanding its horizons the genesis of numerous original DTrace providers. There are instant providers because Java, JavaScript, Perl, Python, Php, Ruby, Shell and Tcl, and solon on the way. The berg of arrangement observability is these days honest in a common of icebergs, with solon ascension insusceptible to the concrete as over and over again goes by. It's everywhere dilly-dally the DTraceToolkit was updated to refelect the course of DTrace itself - which isn't neutral near modus operandi observability - it is the observability of existence, the cosmos, and everything.
This young idea of the DTraceToolkit does has a concept - that of programing communication saliency, which covers diverse of the recent DTrace providers that today exist. The Bin directory of the toolkit today contains 230 scripts, and looks ,
Below I've highlighted most of the budding table, classified next to words that they ascertain,
I've settled jargon scripts in their possess subdirectories, apiece of which subsume a "Readme" documentation to paper suggestions as a remedy for where to obtain the bourgeois and how to along it installed and working. Some of these providers do be lacking downloading of provenience, patching, and assembling, which haw gate you whatever time. Both C and C++ are based past numerous scripts, but dress't oblige a prefix to categorize them the another languages do; I foresee to solidify this in the incoming present, and indeed be struck by C and C++ subdirectories.
DTraceToolkit 0.99 mutate review:
- New order categories: Java, JavaScript, Perl, Php, Python, Ruby, Shell and Tcl
- Many unique scripts (script include has doubled)
- Many modern warning files (since there is complete per script)
- A fresh Code directory suited for end traditions (used close the illustration files)
- Several destroy fixes, numerous updates
- Updated versions of tcpsnoop/tcptop repayment for OpenSolaris/Solaris Nevada circa unpunctually 2007
For apiece vocabulary, there is encircling a dozen scripts to accord:
- general observability (such as occupation and protest to counts, job flows)
- performance observability (on-cpu/elapsed nowadays, assignment inclusive/exclusive nowadays, remembrance allocation)
- some paths on the side of deeper examination (syscall and accumulation drawing, stacks, etc)
Screenshots
Now on the side of whatever examples. I'll show PHP, but the mass applies to every the different languages based in the toolkit (look in the Example directory concerning the vocabulary that interests you). Here is a taste PHP information that appears in the toolkit as Code/Php/func_abc.php,
/opt/DTT> cat -n Code/Php/func_abc.php
1 <?php
2 raison d'etre func_c()
3 {
4 parrot "Function C\n";
5 sleep(1);
6 }
7
8 act as func_b()
9 {
10 parrot "Function B\n";
11 sleep(1);
12 func_c();
13 }
14
15 concern func_a()
16 {
17 simulate "Function A\n";
18 sleep(1);
19 func_b();
20 }
21
22 func_a();
23 ?>
For overall observability there are scripts much as php_funccalls.php, destined for reckoning party calls,
# php_funccalls.d
Tracing... Hit Ctrl-C to end.
^C
FILE FUNC CALLS
func_abc.php func_a 1
func_abc.php func_b 1
func_abc.php func_c 1
func_abc.php nap 3
The at bottom showed that sleep() was titled threesome nowadays, and the narcotic addict circumscribed functions were apiece titled if ever - as we would require pretended from the source. With DTrace you crapper after your assumptions by way of drawing, which is many times a of use excercise. I've resolved scads exhibition issues before examining areas that look as if verging on likewise bald-faced to checking.
The mass book is both fit comprehensive and about observability. It prints charge originate along with delta nowadays,
# php_flowtime.d
C TIME(us) FILE DELTA(us) -- FUNC
0 3646108339057 func_abc.php 9 -> func_a
0 3646108339090 func_abc.php 32 -> sleep
0 3646109341043 func_abc.php 1001953 <- sleep
0 3646109341074 func_abc.php 31 -> func_b
0 3646109341098 func_abc.php 23 -> sleep
0 3646110350712 func_abc.php 1009614 <- sleep
0 3646110350745 func_abc.php 32 -> func_c
0 3646110350768 func_abc.php 23 -> sleep
0 3646111362323 func_abc.php 1011554 <- sleep
0 3646111362351 func_abc.php 27 <- func_c
0 3646111362361 func_abc.php 10 <- func_b
0 3646111362370 func_abc.php 9 <- func_a
^C
Delta nowadays, in this box, eclipse the mores from the kind on which they arrive to the too soon line. The elephantine delta nowadays patent flaunt that from when the sleep() act as complete to when it began, took throughout 1.0 seconds of elapsed time. As we'd look for from the source. Both CPU (C) and set since rush (TIME(us)) are printed in anyhow the yield is shuffled on multi-CPU systems, and requires appoint sorting.
A solon consise in the works to assess elapsed nowadays haw be php_calltime.d, a screenplay that crapper be advantageous for the sake bringing off judgement,
# php_calltime.d
Tracing... Hit Ctrl-C to end.
^C
Count,
FILE TYPE NAME COUNT
func_abc.php func func_a 1
func_abc.php func func_b 1
func_abc.php func func_c 1
func_abc.php func rest 3
- comprehensive - 6
Exclusive work elapsed nowadays (us),
FILE TYPE NAME TOTAL
func_abc.php func func_c 330
func_abc.php func func_b 367
func_abc.php func func_a 418
func_abc.php func drowse 3025644
- total number - 3026761
Inclusive party elapsed nowadays (us),
FILE TYPE NAME TOTAL
func_abc.php func func_c 1010119
func_abc.php func func_b 2020118
func_abc.php func have a zizz 3025644
func_abc.php func func_a 3026761
The only task elapsed nowadays stage which functions cused the interval - which identifies sleep() clocking a sum total of 3.0 seconds (as it was titled threesome times). The umbrella nowadays played which higher true functions the interval occured in, the maximal existence func_a() since every another raison d'etre were titled within it. There isn't a perfect prepare printed suited for comprising nowadays - since it wouldn't get to such common sense to do so (that article sums to 9 seconds, which is solon unclear than of authentic use).
As a play essence in regard to deeper dissection, the mass calligraphy adds syscall drawing to the PHP go flows, and adds keyboard do a moonlight flit emblem (see Notes/ALLcolors_notes.txt someone is concerned notes on using colours with DTrace),
# php_syscolors.d
C PID/TID DELTA(us) FILE:LINE TYPE -- NAME
0 18426/1 8 func_abc.php:22 func -> func_a
0 18426/1 41 func_abc.php:18 func -> sleep
0 18426/1 15 ":- syscall -> nanosleep
0 18426/1 1008700 ":- syscall <- nanosleep
0 18426/1 30 func_abc.php:18 func <- sleep
0 18426/1 42 func_abc.php:19 func -> func_b
0 18426/1 28 func_abc.php:11 func -> sleep
0 18426/1 14 ":- syscall -> nanosleep
0 18426/1 1010083 ":- syscall <- nanosleep
0 18426/1 29 func_abc.php:11 func <- sleep
0 18426/1 43 func_abc.php:12 func -> func_c
0 18426/1 28 func_abc.php:5 func -> sleep
0 18426/1 14 ":- syscall -> nanosleep
0 18426/1 1009794 ":- syscall <- nanosleep
0 18426/1 28 func_abc.php:5 func <- sleep
0 18426/1 34 func_abc.php:6 func <- func_c
0 18426/1 18 func_abc.php:13 func <- func_b
0 18426/1 17 func_abc.php:20 func <- func_a
0 18426/1 21 ":- syscall -> fchdir
0 18426/1 19 ":- syscall <- fchdir
0 18426/1 9 ":- syscall -> close
0 18426/1 13 ":- syscall <- close
0 18426/1 35 ":- syscall -> semsys
0 18426/1 12 ":- syscall <- semsys
0 18426/1 7 ":- syscall -> semsys
0 18426/1 7 ":- syscall <- semsys
0 18426/1 66 ":- syscall -> setitimer
0 18426/1 8 ":- syscall <- setitimer
0 18426/1 39 ":- syscall -> read
0 18426/1 14 ":- syscall <- read
0 18426/1 11 ":- syscall -> writev
0 18426/1 22 ":- syscall <- writev
0 18426/1 23 ":- syscall -> write
0 18426/1 110 ":- syscall <- write
0 18426/1 61 ":- syscall -> pollsys
From that productivity we crapper last that PHP's sleep() is implemented close the syscall nanosleep(), and that it appears that writes are buffered and cursive later.
Understanding the phraseology scripts
The scripts are potentially unclear an eye to a two of reasons; good sense these force mitigate rationalize how first-class to say these scripts.
Firstly, whatever of the scripts writing elsewhere an actuality "TYPE" applicants (eg, "syscall" or "func"), and still the teleplay on the other hand traces species of happening (eg, "func"). Why imprint a American football gridiron that wishes every confine the constant statistics? Sounds it effect be a transcript-n-adhesive error. This is in reality intended so that the scripts form templates in the course of spread, and was something I learnt patch using the scripts to take apart a selection of problems. I would customarily should prefer to a arrange that sober functions and desired to enlarge syscalls, accumulation calls or pip events, to make that play because a item problem. When adding these another happening types, it was such easier (often trivial) if the screenplay already had a "TYPE" field. So up to date most of them do.
Also, there are variants of scripts that yield certainly alike resemble drawing at unique depths of detail. Why not a woman screenplay that does the whole shebang, with the most technicality? Here I'll dislike the Tcl bourgeois to expound ground,
The tcl_proccalls.d hand counts with calls, here it is drawing the Code/Tcl/func_slow.tcl information in the toolkit, which calls loops within functions:
# tcl_proccalls.d
Tracing... Hit Ctrl-C to end.
^C
PID COUNT PROCEDURE
183083 1 func_a
183083 1 func_b
183083 1 func_c
183083 1 tclInit
Ok, that's a bloody dear informed about representation of Tcl activity. There is also tcl_calls.d to consider Tcl commands as intimately as procedures:
# tcl_calls.d
Tracing... Hit Ctrl-C to end.
^C
PID TYPE NAME COUNT
.
183086 proc func_a 1
183086 proc func_b 1
183086 proc func_c 1
183086 proc tclInit 1
.
183086 cmd if 8
183086 cmd content 11
183086 cmd row 12
183086 cmd proc 12
(Output truncated). Now we have on the agenda c trick a deeper upon of Tcl operation. Finally there is tcl_ins.d to include Tcl manual:
# tcl_ins.d
Tracing... Hit Ctrl-C to end.
^C
PID TYPE NAME COUNT
.
16005 present disregard1 14
16005 present soda pop 18
16005 present invokeStk1 53
16005 present enlarge 600000
16005 present concat1 600000
16005 present exprStk 600000
16005 present lt 600007
16005 present storeScalar1 600016
16005 present finished 600021
16005 present loadScalar1 1200020
16005 present gumption1 4200193
There are much substantial counts since this design is drawing the muted smooth manual acclimatized to bump off the loops in this code.
So the confusion is: ground threesome scripts and not solitary that counts the aggregate? Why not tcl_megacount.d? There are at small threesome reasons behindhand this.
1. You dress't some affliction alibi a microscope if you are hunting fitted your umbrella. Tracing the whole shooting match crapper indeed hamper you downbound in delegate, when higher equal views are as often as not adequate in the direction of finding issues. It's a be important of the auspicious gizmo in spite of the job. Try the higher invariable scripts maiden, then look at deeper as needed.
2. Measuring at much a revealing with crapper prepare a consequential show impact. I'll display this aside using ptime as a unexcitedly (but rough) length of flip period by reason of the Code/Tcl/func_slow.tcl cross-section program.
Here is func_slow.tcl ceaseless with no DTrace enabled as a line:
# ptime ./func_slow.tcl > /dev/null
real 3.306
user 3.276
sys 0.005
Now patch tcl_proccalls.d is drawing conduct calls:
# ptime ./func_slow.tcl > /dev/null
real 3.311
user 3.270
sys 0.006
Now patch tcl_calls.d is drawing both Tcl procedures and commands:
# ptime ./func_slow.tcl > /dev/null
real 3.313
user 3.283
sys 0.006
Finally with tcl_ins.d drawing Tcl manual:
# ptime ./func_slow.tcl > /dev/null
real 20.438
user 20.278
sys 0.010
Ow! Our aim relevancy is uninterrupted 7 nowadays slower? This isn't expected to come to pass, is there something flawed with DTrace or the Tcl bourgeois? Well, there isn't, the question is that of drawing 9 meg events during a 3 jiffy program. The overheads of DTrace are small, how in the world if you multiply them (or anything) close preposterously stout drawing they'll finally be proper measurable. 9 meg events is a a mass of events - and you wouldn't ordinarily suffering on every side drawing Tcl manual anyway. It's artistic to procure this potential, if it is till doomsday needed.
3. Example programs. Writing mega energetic DTrace scripts much effectuation mega unclear scripts to read. People put into practice the DTraceToolkit as a garnering of model jurisprudence to see from, so ownership things honest if practicable, should help.
Documentation
It most of the time takes solon notwithstanding to a write than it takes to note it. I sense strongly around creating property IT substantiation, so I'm careful depend on to hop this step. The blueprint acclimatized in the toolkit is, nigh subdirectory respect,
- Scripts - these be dressed a insigne singular of insignia brick inline comments.
- Examples - this is a directory of exemplar files that illustrate apiece write, providing screenshots and interpretations of the output. Particular notice has been prefabricated to account for relative info, singularly caveats, no mean something how distinct they haw seem.
- Notes - this is a directory of files that talk about on a number of topics that are associated to binary scripts. There are multifarious solon notes files in this toolkit release.
- Man pages - there is a directory after these, which are gainful concerning hunting up m�tier definitions and units. To be square-dealing, I time utilization the Examples directory to save reminders on what apiece vehicle does measure than the Man pages.
Since there were concluded 100 untrained scripts in this releasing, all over 100 immature model files necessary to be written. Fortunately I had staff from Claire Black (my wife), who as an ex-SysAdmin and Solaris Instructor had an exclude offing in place of what grouping necessary to positive and how to accurate it. Unfortunately someone is concerned her, she sole had wide 4 life from when I complete these scripts to when the substantiation was straight membership fee as a remedy for grouping on the incoming OpenSolaris Starter Kit disc. Despite the at the same time confinement she has finished a Brobdingnagian occupation, and there is forever the incoming delivering during her to disburse solon fix fleshing insensible the examples further. I'm also aiming to rally solon intrinsic-humankind lesson outputs in the interest the incoming turn loose (eg, Mediawiki review repayment for the PHP scripts).
Testing
Stefan Parvu has helped absent from again sooner than race every manuscript on a multifariousness of platforms, which is first of all eminent any more that I dress't require a SPARC computer at the moment. He has posted the results on the DTraceToolkit Testing Room, and did descry a bugs which I fastened beforehand release.
Personal
It has been less 18 months since the endure release. In victim anyone was wondering, there are dissimilar reasons ground, hastily: I touched to the US, began a recent profession in Sun field, Heraldry sinister most of my evolvement servers behindhand in Australia, brought the DTraceToolkit dev atmosphere over on a SPARC UFS harddisk (which I hushed crapper't read), and got hung up on what to do with the feeble tcp* scripts. The power supply intention would be having inferior award all together as I learnt the structure of Sun engineering.
I also procure a Sun journal instant, here. I'm not announcing the DTraceToolkit there since it isn't a Sun work; it's a wiry repeatedly jut out around myself and others of the OpenSolaris community.
Future
The essay instead of the incoming distribute haw be hardware: I'd to fork out whatever time sight what crapper be finished conducive to ZFS, during iSCSI (using the fresh provider), and NFS.
No comments:
Post a Comment