[Ksummit-2012-discuss] [ATTEND] kernel debugging, tracing, backtracing

[Jason Wessel]

On 06/19/2012 09:59 PM, Roland Dreier wrote:
> As someone who has to look at kernel logs exfiltrated from systems
> running at remote sites, I think this looks very useful.  I certainly spend
> a fair bit of time staring at disassembly to figure out where in a function
> a particular offset is.
> 
> Of course we would have to make sure that we don't make stack trace
> printing less reliable when we do print this...
> 

It shouldn't be any less reliable because in the RFC the interface is
grafted onto the same mechanism that kallsyms uses today for the
kernel core.  For kernel modules the data is stored a bit differently
in the lookup tables, but the API plumbing uses the same kallsyms
interface.

You pay for this feature in several ways, where it was not clear to me
if developers and system integrators would be willing to "pay the
toll".  Having used the patch set to develop the patch for the kernel
core to develop the additional parts for the kernel modules, it was
definitely really nice to look at the oops and not even have to decode
(granted I do have a tool for that).

Since there does seem to be interest in this patch set, when I have a
chance I will put together some statistics on just what the incremental
cost of the feature looks like.  Here are the elements of the cost:

  * The vmlinux file must be built with debuginfo
    ooo This is used to get the lines for locations elf section

  * Binary size cost:
    o The kallsysms elf section is loaded with lines for locations
    o Kernel modules on the disk have to have the .debug_lineinfo
      section

  * Runtime cost:
    o The kallsyms table is bigger
    o The kernel module data section is slightly bigger
    o Kernel modules take a few more cpu cycles to load
      - There is an elf reader integrated into the module.c.
        It should probably be separated, but that means
        exposing more module internals, so for now it is all
        in module.c.
      - The elf reader decodes the .debug_line section and
        builds a table in memory which is similar to the
        symtab.
    o The kallsyms printk hook will take every so slightly
      more time to return the additional information.
    o The trace back takes up a bit more screen real estate
      but this one is totally worth the cost. :-)

It would be really nice if we had a place we could reliably fetch the
debug line tables from without keeping them in memory all the time,
but the only way I could find to reliably get a good back trace was to
keep the tables in memory.

Certainly I am open to further suggestions as to how to carry this
forward.

Thanks for the input,
Jason.