[Ksummit-2008-discuss] DTrace

[Theodore Tso]

On Fri, Jun 27, 2008 at 10:23:33AM -0500, James Bottomley wrote:
> 
> DTrace is more a piece of sun marketing coolaid which they use to beat
> us up at every opportunity.
> 
> We actually have a reasonably functional equivalent piece of technology
> called systemtap.

Actually, we don't.  The big thing that are missing are the tapsets
--- the macro libraries that allow a system administrator to use it to
find and solve performance problems without being a kernel developer,
and more importantly, the documentation for said macro libraries so a
system administrator can actually use it.

For example, when some of the examples use constructs like this:

probe kernel.function ("vfs_write"),
      kernel.function ("vfs_read")
{
  dev_nr = $file->f_dentry->d_inode->i_sb->s_dev
  inode_nr = $file->f_dentry->d_inode->i_ino

  if (dev_nr == ($1 << 20 | $2) # major/minor device
      && inode_nr == $3)
    printf ("%s(%d) %s 0x%x/%u\n",
      execname(), pid(), probefunc(), dev_nr, inode_nr)
}

Taken from: http://sourceware.org/systemtap/wiki/WSFileMonitor?highlight=((WarStories))

Do you really expect system administrators to use this tool? 

When I've talked to System Tap developers about this problem, they
shrug and say, "It's not our job to write tapset scripts, it's up to
kernel developers who are subsystem experts".  There are two massive
problems with this.  (1) Subsystem experts need help with suggestions
what things system administrators might be most interested in
monitoring to help solve performance problems.  (2) Systemtap is a
massive pain the !@#!@ to set up if you aren't using an Enterprise
Distribution, requiring patched elfutil libraries, and debugging
information.  (Actually, given the latter, it can be painful even for
Enterprise distro customers to use, since most distro's don't ship
them by default, and the download URLs for the debuginfo RPM's aren't
always easy to find.)

> I finally (after an unhappy week spend with dwarf) worked out how to fix
> this particular annoyance, but my general impression is that systemtap
> is klunky and prone to break in unexpected ways.  Although we could have
> a topic about systemtap at KS, I think a better way to improve it might
> be if more kernel developers started actually using it (and fixing the
> bugs they find).

Unfortunately, I have to agree; the real problem isn't as much kernel
developers, it's that (a) it's too hard for many kernel developers to
use (and so many kernel developers are using it), and (b) there aren't
enough tapsets.  The latter is something that kernel developers can
help solve, but unfortunately I'm not sure discussing it at the Kernel
Summit will necessarily lead to making forwarwd progress.

Although if someone on the debian kernel team is interested in making
make-kpkg work with modern 2.6 kernels (since the x86 architecture
merge, make-kpkg isn't including certain files that are necessary for
the kernel-headers deb to be useful, which is mandatory for systemtap)
and making make-kpkg understand how to create debuginfo packages so
that Debian/Ubuntu systems can usefully use systemtap, contact me
off-line.  I've done a small amount of work on this, but I don't
really have the time for this project, and I'd love it if someone else
were interested in trying to solve these issus.

					- Ted