James.Bottomley at HansenPartnership.com
Fri Jun 27 13:05:35 PDT 2008
On Fri, 2008-06-27 at 14:27 -0400, Theodore Tso wrote:
> On Fri, Jun 27, 2008 at 10:23:33AM -0500, James Bottomley wrote:
> > DTrace is more a piece of sun marketing coolaid which they use to beat
> > us up at every opportunity.
> > We actually have a reasonably functional equivalent piece of technology
> > called systemtap.
> Actually, we don't. The big thing that are missing are the tapsets
> --- the macro libraries that allow a system administrator to use it to
> find and solve performance problems without being a kernel developer,
> and more importantly, the documentation for said macro libraries so a
> system administrator can actually use it.
> For example, when some of the examples use constructs like this:
> probe kernel.function ("vfs_write"),
> kernel.function ("vfs_read")
> dev_nr = $file->f_dentry->d_inode->i_sb->s_dev
> inode_nr = $file->f_dentry->d_inode->i_ino
> if (dev_nr == ($1 << 20 | $2) # major/minor device
> && inode_nr == $3)
> printf ("%s(%d) %s 0x%x/%u\n",
> execname(), pid(), probefunc(), dev_nr, inode_nr)
> Taken from: http://sourceware.org/systemtap/wiki/WSFileMonitor?highlight=((WarStories))
> Do you really expect system administrators to use this tool?
No, but I expect most people doing reasonable debugging of the kernel to
know what it means ... certainly anyone trying to tune their kernel to
The basic problem here is that systemtap is adding debugging to kernel
functions ... if you don't know what the function is doing in a
reasonable amount of detail, it's not going to return much useful
information for you.
Adding a framework on to systemtap to help those with less intimate
knowledge of the kernel get something out of it is a different problem,
> When I've talked to System Tap developers about this problem, they
> shrug and say, "It's not our job to write tapset scripts, it's up to
> kernel developers who are subsystem experts". There are two massive
> problems with this. (1) Subsystem experts need help with suggestions
> what things system administrators might be most interested in
> monitoring to help solve performance problems. (2) Systemtap is a
> massive pain the !@#!@ to set up if you aren't using an Enterprise
> Distribution, requiring patched elfutil libraries, and debugging
> information. (Actually, given the latter, it can be painful even for
> Enterprise distro customers to use, since most distro's don't ship
> them by default, and the download URLs for the debuginfo RPM's aren't
> always easy to find.)
Well ... the type of tapsets you're talking about: ones an administrator
can usefully use, I don't see being written by kernel developers.
For SCSI, I just need a library of accessors and things, I'm not going
to write translation scripts between what the kernel is saying and some
notional high level debugging information that might be useful to an
administrator ... on the other hand, most of the major consumers of this
information do seem to be forking over wads of cash to the distros for
it, so I can see it becoming a distro support/consulting item.
More information about the Ksummit-2008-discuss