sp-rtrace
---------

Contents:
  1. Introduction
  2. Components
  2.1 Tracing modules
  2.1.1 Main module (sp-rtrace-main.so)
  2.1.2 Tracing modules
  2.2 Data pre-processor (rtrace)
  2.3 Data post-processor (rtrace-postproc)
  2.3.1 Internal data storage
  2.3.2 Post-processing
  2.3.2.1 leak filtering (--filter-leaks)
  2.3.2.2 backtrace compressing (--compress)
  2.3.3.3 future post-processing options
  3 Tracing Configuration
  4 Trace data flow
  5 Visualization tools
  6 Notes
  6.1 Limitations
  6.1 Scratchbox (Maemo)
  6.2 Debugging


1. Introduction

The sp-rtrace resource consumption tracking suite contains a set of
utilities for different resource types allocation / deallocation call
tracing, trace post-processing and visualization.


2. Components

+----------------------+    +---------------+      +----------------+
|     main module      | -> | pre-processor |  ,-> | post-processor |
+----------------------+    +---------------+  |   +----------------+
    | tracing module 1 |            |          |       |   |   ^
    +------------------+            v          |       v   v   |
    |       ...        |      binary output ---+      text output
    +------------------+                             /     |   ^
    | tracing module N |                            /      v   |
    +------------------+     +------------+ <------'    +----------+
                             | visualizer | <---------- | resolver |
                             +------------+             +----------+

2.1 Tracing modules

The tracing modules are the primary data source. They feed function
trace data to the pre-processor via named (or simple in managed mode)
pipe. 

2.1.1 Main module (sp-rtrace-main.so)

The main module is always preloaded for the target process when it
is started with rtrace. The main module provides the following
functionality:
  1) pre-processor connection management,
  2) data writing API used by tracing modules to report trace events,
  3) tracking of new libraries.

The main module is configured by environment variables. rtrace
automatically sets the necessary variables before it launches 
the target process. It is possible to set the variables and
launch the target process manually, however it is not recommended.

By default when tracing is enabled the main module attempts to open
a named pipe /tmp/rtrace-<pid> where pid is the process id of the 
current (traced) process. But in managed mode the main module
starts the rtrace process by itself and uses its standard input to
pipe trace data (see sp-rtrace manual page --manage-preproc option
description for when to use managed mode).

By default the main module buffers (KB) trace data for performance
reasons. However it's possible to disable the data buffering with
an environment variable.

The main module ensures that memory mapping information is being
kept up to date by tracking dlopen() calls and sending new library
(NL) packet whenever a new library is opened. This packet instructs
pre-processor to rescan /proc/<pid>/maps file for the target process.

The main module keeps registry of loaded tracing modules to be able
disable tracing when necessary.

2.1.2 Tracing modules

The tracing modules are either preloaded with LD_PRELOAD or LD_AUDIT.
They provide tracing of resource alloaction/deallocation calls. 
Usually one module covers single resource type (memory, file etc).
For example memory tracing module monitors memory allocation/deallocation
calls - malloc, alloc, calloc, posix_memalign, free.

Tracing modules must register themselves to the main module during
initialization to allow the main module enable/disable tracing when
necessary.

Tracing modules uses data writing API provided by the main module to 
report resource allocation/deallocation calls.


2.2 Data pre-processor (rtrace)

The data pre-processor provides the following functionality:
1) launching of a binary file with enabled resource tracing,
2) pre-processing data sent by tracing modules,
3) trace enabling/disabling for already launched process. 

If tracing is enabled (with the "-s" option) at start, after launching
the given process pre-processor creates a named pipe /tmp/rtrace-<pid>
and listens it for the trace data.

If tracing wasn't enabled at start, sp-rtrace exits after launching
the process-to-trace onto background.  This way one can start large
number of processes e.g. at device boot to wait for being traced,
without redundant sp-rtrace instances.

When managed mode is used, sp-rtrace uses exec to replace itself with
the target process.  A new pre-processor is started by the sp-rtrace
LD_PRELOAD main module itself for data pre-processing when tracing is
started.

The post-processor keeps track of the module memory mapping and
rescans /proc/<pid>/maps file whenever new library (NL) packet is
received. All executable module changes are reported by generating
memory mapping (MM) packets.

If post-processing options are specified, the pre-processor spawns
post-processor (rtrace-postproc) process and pipes the trace data
to it. Otherwise the data is stored in new file in the output
directory (home directory by default) - rtrace-raw-<pid>-<number>
Pre-processor never overwrites the existing data files, it increases
the file number until a new file can be created.

The pre-processor also uses internal buffer (4KB) for buffering 
pipe data which can be also disabled (by the same option as 
main module trace data buffer).

When --toggle option is specified pre-processor sends the toggle
signal to the target process. Then it either opens a named pipe and
starts listening for trace data (non-managed mode) or simply exits
(managed mode).


2.3 Data post-processor (rtrace-postproc)

The post-processor provides binary data conversion to human readable text format
and additional data post-processing. It also can parse the text format files, 
allowing it to process its own generated output.

If the --resolve option is specified, the post-processor spawns the resolver
(rtrace-resolve) process and pipes the output data to it for name resolving.

The data processing steps are:
1) parse input file and create internal data structures,
2) apply post-processing filters,
3) generate output data.

2.3.1 Internal data storage

The memory mapping, context registry, comments, function calls
are stored into double linked lists.

The function traces are stored into hash table and shared between
similar calls. Every function trace record holds double linked 
list of the function call references.

2.3.2 Post-processing

Currently two post-processing options are implemented - leak filtering
and backtrace compressing. 

2.3.2.1 leak filtering (--filter-leaks)

This option removes allocation records of resources that was later 
deallocated. The deallocation records also are removed, leaving only
allocations of the leaked resources. To speed up the resource lookup
the resource identifiers are indexed in hash table during freed 
allocation removal. 

2.3.2.2 backtrace compressing (--compress)

This option groups function calls by their backtraces and for all 
functions with the same backtrace only one is printed.

2.3.3.3 future post-processing options

A plugin system could be added to post-processor allowing users to
write their own custom plugins. 


2.4 Name resolver (rtrace-resolve)

The resolver processes the input file by replacing backtrace
addresses with resolved names. To resolve an address a memory
mapping record covering it must be located before the address.


3 Tracing configuration

The tracing module configuration is done with environment
variables (usually set by the rtrace launcher and other helper
scripts).

* SP_RTRACE_OUTPUT_DIR
  Specifies the output directory for rtrace logs. By default
  the home directory is used.

* SP_RTRACE_TOGGLE_SIGNAL
  Specifies the trace toggling signal number. By default SIGUSR1 
  is used.

* SP_RTRACE_START
  Defines that trace should be started automatically when the target
  process is spawned.

* SP_RTRACE_MANAGE_PREPROC
  Defines that the main trace module must spawn pre-processor
  process when necessary. By default sp-rtrace is spawned externally
  and the main module just connects to its named pipe.

* SP_RTRACE_BACKTRACE_DEPTH
  Specifies the maximum number of frames listed in the backtraces.

* SP_RTRACE_DISABLE_TIMESTAMP
  Disables timestamps reported by the tracing modules.

* SP_RTRACE_ENABLE_ARGUMENTS
  Enables sending function argument (FA) packets.

* SP_RTRACE_POSTPROC
  Specifies post-processing options. 

* SP_RTRACE_DISABLE_EVENT_BUFFERING
  Disables data buffering in main module and pre-processor.


4 Trace data flow

The usual data flow is:
  libsp-rtrace-<module>.so
    -> libsp-rtrace-main.so
       -> sp-rtrace
          -> sp-rtrace-postproc
	     -> sp-rtrace-resolve

Depending on sp-rtrace options used to invoke the traced process,
the sp-rtrace instance that actually collects the data can be:
* that sp-rtrace instance
  (-s option given without -m)
* sp-rtrace instance run later by user to toggle tracing
  (when neither -s nor -m options were given)
* sp-rtrace instance invoked by the main module
  (-m option given, regardless of whether -s was given)


5 Visualization tools

There are several resource trace visualization tools.

Binaries:

* sp-rtrace-pagemap
  ASCII graphs for all the writable memory mappings the traced process
  has, what kind of memory page types they have and how much of these
  pages are utilized by the traced (non-freed) allocations.  Trace
  data needs to contain both pagemap and memory modules output.

* sp-rtrace-timeline
  o allocation timelines:
    + non-freed allocation total + alloc overhead
    + alloc & free counts
    + resource lifetimes 
  o histograms:
  + allocation size vs. freed & non-freed allocation count
  + allocation size vs. freed & non-freed allocation amount 

Scripts:

* rtrace-calltree
  generate .dot callgraphs from the trace data, used by
  rtrace-graph-* scripts

* rtrace-graphs-overview
  o overview alloc/dealloc timeline and allocation count/size
    histogram charts (using sp-rtrace-timeline)
  o callgraphs to resolved functions which are allocating
    non-freed resources (using rtrace-calltree) with (hopefully)
    irrelevant functions removed from the graph to make it readable 

* rtrace-graphs-function
  create callgraph(s) of allocations going through given function

* rtrace-from-function
  list amount of allocations from the given function


6 Notes
	
6.1 Limitations

Binary data is portable only between machines having same endianess
and pointer size.  If you want to process e.g. 32-bit ARM trace data
on a 64-bit x86 machine, convert the trace data first to ASCII format.

6.1 Scratchbox (Maemo)

In scratchbox (arm) to launch process with enabled tracing the managed mode
must be used. Otherwise the pre-processor will create named pipe with
emulation process id while the main module will try to connect to a named
pipe created with its (target) process id.

6.2 Debugging

It's possible to enable additional debug output (mainly used for rtrace 
debugging) with --enable-debug-info configure option. Note that log functions
don't use async signal safe functions and therefore toggling trace could
lead to corruption.