Linux Tracing Technologies

• Function Tracer Design
  • Introduction
  • Prerequisites
  • HAVE_FUNCTION_TRACER
  • HAVE_FUNCTION_GRAPH_TRACER
  • HAVE_FUNCTION_GRAPH_FP_TEST
  • HAVE_FUNCTION_GRAPH_RET_ADDR_PTR
  • HAVE_SYSCALL_TRACEPOINTS
  • HAVE_FTRACE_MCOUNT_RECORD
  • HAVE_DYNAMIC_FTRACE
  • HAVE_DYNAMIC_FTRACE + HAVE_FUNCTION_GRAPH_TRACER
• Notes on Analysing Behaviour Using Events and Tracepoints
  • 1. Introduction
  • 2. Listing Available Events
  • 3. Enabling Events
  • 4. Event Filtering
  • 5. Analysing Event Variances with PCL
  • 6. Higher-Level Analysis with Helper Scripts
  • 7. Lower-Level Analysis with PCL
• ftrace - Function Tracer
  • Introduction
  • Implementation Details
  • The File System
  • The Tracers
  • Error conditions
  • Examples of using the tracer
  • Output format:
  • Latency trace format
  • trace_options
  • irqsoff
  • preemptoff
  • preemptirqsoff
  • wakeup
  • wakeup_rt
  • Latency tracing and events
  • Hardware Latency Detector
  • function
  • Single thread tracing
  • function graph tracer
  • dynamic ftrace
  • Selecting function filters via index
  • Dynamic ftrace with the function graph tracer
  • ftrace_enabled
  • Filter commands
  • trace_pipe
  • trace entries
  • Snapshot
  • Instances
  • Stack trace
  • More
• Using ftrace to hook to functions
  • Introduction
  • The ftrace context
  • The ftrace_ops structure
  • The callback function
  • Protect your callback
  • The ftrace FLAGS
  • Filtering which functions to trace
• Fprobe - Function entry/exit probe
  • Introduction
  • The usage of fprobe
  • The fprobe entry/exit handler
  • Share the callbacks with kprobes
  • The missed counter
  • Functions and structures
• Kernel Probes (Kprobes)
  • Concepts: Kprobes and Return Probes
  • Architectures Supported
  • Configuring Kprobes
  • API Reference
  • Kprobes Features and Limitations
  • Probe Overhead
  • TODO
  • Kprobes Example
  • Kretprobes Example
  • Deprecated Features
  • The kprobes debugfs interface
  • The kprobes sysctl interface
  • References
• Kprobe-based Event Tracing
  • Overview
  • Synopsis of kprobe_events
  • Types
  • User Memory Access
  • Per-Probe Event Filtering
  • Event Profiling
  • Kernel Boot Parameter
  • Usage examples
• Uprobe-tracer: Uprobe-based Event Tracing
  • Overview
  • Synopsis of uprobe_tracer
  • Types
  • Event Profiling
  • Usage examples
• Fprobe-based Event Tracing
  • Overview
  • Synopsis of fprobe-events
  • BTF arguments
  • Usage examples
• Using the Linux Kernel Tracepoints
  • Purpose of tracepoints
  • Usage
• Event Tracing
  • 1. Introduction
  • 2. Using Event Tracing
  • 3. Defining an event-enabled tracepoint
  • 4. Event formats
  • 5. Event filtering
  • 6. Event triggers
  • 7. In-kernel trace event API
• Subsystem Trace Points: kmem
  • 1. Slab allocation of small objects of unknown type
  • 2. Slab allocation of small objects of known type
  • 3. Page allocation
  • 4. Per-CPU Allocator Activity
  • 5. External Fragmentation
• Subsystem Trace Points: power
  • 1. Power state switch events
  • 2. Clocks events
  • 3. Power domains events
  • 4. PM QoS events
• NMI Trace Events
  • nmi_handler
• MSR Trace Events
• In-kernel memory-mapped I/O tracing
  • Preparation
  • Usage Quick Reference
  • Usage
  • How Mmiotrace Works
  • Trace Log Format
  • Explanation Keyword Space-separated arguments
  • Tools for Developers
• Event Histograms
  • 1. Introduction
  • 2. Histogram Trigger Command
• Histogram Design Notes
  • 'hist_debug' trace event files
  • Basic histograms
  • Variables
  • Actions and Handlers
  • A couple special cases
• Boot-time tracing
  • Overview
  • Options in the Boot Config
  • When to Start
  • Examples
• Hardware Latency Detector
  • Introduction
  • Usage
• OSNOISE Tracer
  • Usage
  • Tracer Configuration
  • Tracer Options
  • Additional Tracing
  • Running osnoise tracer without workload
• Timerlat tracer
  • Usage
  • Tracer options
  • timerlat and osnoise
  • IRQ stacktrace
  • User-space interface
• Intel(R) Trace Hub (TH)
  • Overview
  • Bus and Subdevices
  • Quick example
  • Host Debugger Mode
  • Software Sinks
• Lockless Ring Buffer Design
  • Terminology used in this Document
  • The Generic Ring Buffer
  • Making the Ring Buffer Lockless:
  • Nested writes
• System Trace Module
  • stm_source
  • stm_console
  • stm_ftrace
• MIPI SyS-T over STP
• CoreSight - ARM Hardware Trace
  • Coresight - HW Assisted Tracing on ARM
  • CoreSight System Configuration Manager
  • Coresight CPU Debug Module
  • Coresight Dummy Trace Module
  • CoreSight Embedded Cross Trigger (CTI & CTM).
  • ETMv4 sysfs linux driver programming reference.
  • CoreSight - Perf
  • The trace performance monitoring and diagnostics aggregator(TPDA)
  • Trace performance monitoring and diagnostics monitor(TPDM)
  • Trace Buffer Extension (TRBE).
  • UltraSoc - HW Assisted Tracing on SoC
• user_events: User-based Event Tracing
  • Overview
  • Registering
  • Deleting
  • Unregistering
  • Status
  • Writing Data
  • Example Code
• Runtime Verification
  • Runtime Verification
  • Deterministic Automata
  • Deterministic Automata Monitor Synthesis
  • Deterministic Automata Instrumentation
  • Monitor wip
  • Monitor wwnr
• HiSilicon PCIe Tune and Trace device
  • Introduction
  • Tune
  • Trace