@@ -0,0 +1,64 @@ +Introduction: +- + +The module hwlatdetector is a special purpose kernel module that is used to +detect large system latencies induced by the behavior of certain underlying +hardware or firmware, independent of Linux itself. The code was developed +originally to detect SMIs (System Management Interrupts) on x86 systems, +however there is nothing x86 specific about this patchset. It was +originally written for use by the 'RT' patch since the Real Time +kernel is highly latency sensitive. + +SMIs are usually not serviced by the Linux kernel, which typically does not +even know that they are occuring. SMIs are instead are set up by BIOS code +and are serviced by BIOS code, usually for 'critical' events such as +management of thermal sensors and fans. Sometimes though, SMIs are used for +other tasks and those tasks can spend an inordinate amount of time in the +handler (sometimes measured in milliseconds). Obviously this is a problem if +you are trying to keep event service latencies down in the microsecond range. Shivprasad koirala 7th edition pdf.

• Rt Preempt
• Config_preempt_rt_full

+ +The hardware latency detector works by hogging all of the cpus for configurable +amounts of time (by calling stopmachine), polling the CPU Time Stamp Counter +for some period, then looking for gaps in the TSC data. Any gap indicates a +time when the polling was interrupted and since the machine is stopped and +interrupts turned off the only thing that could do that would be an SMI. + +Note that the SMI detector should.NEVER. be used in a production environment. +It is intended to be run manually to determine if the hardware platform has a +problem with long system firmware service routines.

Start with the stock.config. Enable PREEMPT_RT and any other RT features you want. CONFIG_PREEMPT_RT on Raspberry Pi.

+ +Usage: +- + +Loading the module hwlatdetector passing the parameter 'enabled=1' (or by +setting the 'enable' entry in 'hwlatdetector' debugfs toggled on) is the only +step required to start the hwlatdetector. It is possible to redefine the +threshold in microseconds (us) above which latency spikes will be taken +into account (parameter 'threshold='). + +Example: + +# modprobe hwlatdetector enabled=1 threshold=100 + +After the module is loaded, it creates a directory named 'hwlatdetector' under +the debugfs mountpoint, '/debug/hwlatdetector' for this text. It is necessary +to have debugfs mounted, which might be on /sys/debug on your system.

Rt Preempt

@@ -1640,6 +1640,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted.

Config_preempt_rt_full

Ip= IPPNP See Documentation/filesystems/nfs/nfsroot.txt. +irqaffinity=SMP Set the default irq affinity mask + Format: +., + or + - + (must be a positive range in ascending order) + or a mixture +.,- + irqfixupHW When an interrupt is not handled search all handlers for it. Intended to get systems with badly broken. @@ -59,10 +59,17 @@ On PowerPC - Press 'ALT - Print Screen (or F13) -, On other - If you know of the key combos for other architectures, please let me know so I can add them to this section.On all - write a character to /proc/sysrq-trigger. E.g.: - +On all - write a character to /proc/sysrq-trigger, e.g.: echo t /proc/sysrq-trigger +On all - Enable network SysRq by writing a cookie to icmpechosysrq, e.g. + echo 0x01020304 /proc/sys/net/ipv4/icmpechosysrq + Send an ICMP echo request with this pattern plus the particular + SysRq command key.

Example: + # ping -c1 -s57 -p + will trigger the SysRq-H (help) command. + +. What are the 'command' keys? 'b' - Will immediately reboot the system without syncing or unmounting.

@@ -0,0 +1,186 @@ + Using the Linux Kernel Latency Histograms + + +This document gives a short explanation how to enable, configure and use +latency histograms. Latency histograms are primarily relevant in the +context of real-time enabled kernels (CONFIGPREEMPT/CONFIGPREEMPTRT) +and are used in the quality management of the Linux real-time +capabilities. + + +. Purpose of latency histograms + +A latency histogram continuously accumulates the frequencies of latency +data. There are two types of histograms +- potential sources of latencies +- effective latencies + + +.

Potential sources of latencies + +Potential sources of latencies are code segments where interrupts, +preemption or both are disabled (aka critical sections). To create +histograms of potential sources of latency, the kernel stores the time +stamp at the start of a critical section, determines the time elapsed +when the end of the section is reached, and increments the frequency +counter of that latency value - irrespective of whether any concurrently +running process is affected by latency or not. +- Configuration items (in the Kernel hacking/Tracers submenu) + CONFIGINTERRUPTOFFLATENCY + CONFIGPREEMPTOFFLATENCY + + +.

Effective latencies + +Effective latencies are actually occuring during wakeup of a process. To +determine effective latencies, the kernel stores the time stamp when a +process is scheduled to be woken up, and determines the duration of the +wakeup time shortly before control is passed over to this process. Note +that the apparent latency in user space may be somewhat longer, since the +process may be interrupted after control is passed over to it but before +the execution in user space takes place. Simply measuring the interval +between enqueuing and wakeup may also not appropriate in cases when a +process is scheduled as a result of a timer expiration.

The timer may have +missed its deadline, e.g. Due to disabled interrupts, but this latency +would not be registered.

Therefore, the offsets of missed timers are +recorded in a separate histogram. If both wakeup latency and missed timer +offsets are configured and enabled, a third histogram may be enabled that +records the overall latency as a sum of the timer latency, if any, and the +wakeup latency. This histogram is called 'timerandwakeup'.

+- Configuration items (in the Kernel hacking/Tracers submenu) + CONFIGWAKEUPLATENCY + CONFIGMISSEDTIMEROFSETS + + +. Usage + +The interface to the administration of the latency histograms is located +in the debugfs file system.

To mount it, either enter + +mount -t sysfs nodev /sys +mount -t debugfs nodev /sys/kernel/debug + +from shell command line level, or add + +nodev/sys sysfsdefaults0 0 +nodev/sys/kernel/debugdebugfsdefaults0 0 + +to the file /etc/fstab. All latency histogram related files are then +available in the directory /sys/kernel/debug/tracing/latencyhist. A +particular histogram type is enabled by writing non-zero to the related +variable in the /sys/kernel/debug/tracing/latencyhist/enable directory. +Select 'preemptirqsoff' for the histograms of potential sources of +latencies and 'wakeup' for histograms of effective latencies etc. The +histogram data - one per CPU - are available in the files + +/sys/kernel/debug/tracing/latencyhist/preemptoff/CPUx +/sys/kernel/debug/tracing/latencyhist/irqsoff/CPUx +/sys/kernel/debug/tracing/latencyhist/preemptirqsoff/CPUx +/sys/kernel/debug/tracing/latencyhist/wakeup/CPUx +/sys/kernel/debug/tracing/latencyhist/wakeup/sharedprio/CPUx +/sys/kernel/debug/tracing/latencyhist/missedtimeroffsets/CPUx +/sys/kernel/debug/tracing/latencyhist/timerandwakeup/CPUx + +The histograms are reset by writing non-zero to the file 'reset' in a +particular latency directory. To reset all latency data, use + +#!/bin/sh + +TRACINGDIR=/sys/kernel/debug/tracing +HISTDIR=$TRACINGDIR/latencyhist + +if test -d $HISTDIR +then + cd $HISTDIR + for i in `find.

grep /reset$` + do + echo 1 $i + done +fi + + +. Data format + +Latency data are stored with a resolution of one microsecond. The +maximum latency is 10,240 microseconds. The data are only valid, if the +overflow register is empty. Every output line contains the latency in +microseconds in the first row and the number of samples in the second +row. To display only lines with a positive latency count, use, for +example, + +grep -v ' 0$' /sys/kernel/debug/tracing/latencyhist/preemptoff/CPU0 + +#Minimum latency: 0 microseconds.

+#Average latency: 0 microseconds. +#Maximum latency: 25 microseconds. +#Total samples: +#There are 0 samples greater or equal than 10240 microseconds +#usecs samples + 0 + 1 49843506 + 2 58219047 + 3 5348126 + 4 2187960 + 5 3388262 + 6 959289 + 7 208294 + 8 40420 + 9 4485 + 10 14918 + 11 18340 + 12 25052 + 13 19455 + 14 5602 + 15 969 + 16 47 + 17 18 + 18 14 + 19 1 + 20 3 + 21 2 + 22 5 + 23 2 + 25 1 + + +. Wakeup latency of a selected process + +To only collect wakeup latency data of a particular process, write the +PID of the requested process to + +/sys/kernel/debug/tracing/latencyhist/wakeup/pid + +PIDs are not considered, if this variable is set to 0. + + +.

Details of the process with the highest wakeup latency so far + +Selected data of the process that suffered from the highest wakeup +latency that occurred in a particular CPU are available in the file + +/sys/kernel/debug/tracing/latencyhist/wakeup/maxlatency-CPUx. + +In addition, other relevant system data at the time when the +latency occurred are given. + +The format of the data is (all in one line): + + + +The value of is only relevant in the combined timer +and wakeup latency recording. In the wakeup recording, it is +always 0, in the missedtimeroffsets recording, it is the same +as. + +When retrospectively searching for the origin of a latency and +tracing was not enabled, it may be helpful to know the name and +some basic data of the task that (finally) was switching to the +late real-tlme task. In addition to the victim's data, also the +data of the possible culprit are therefore displayed after the +'. After the most recent system boot +is provided.

+ +These data are also reset when the wakeup histogram is reset. @@ -9,6 +9,7 @@ config OPROFILE tristate 'OProfile system profiling' depends on PROFILING depends on HAVEOPROFILE +depends on!PREEMPTRTFULL select RINGBUFFER select RINGBUFFERALLOWSWAP help @@ -52,6 +53,7 @@ config KPROBES config JUMPLABEL bool 'Optimize very unlikely/likely branches' depends on HAVEARCHJUMPLABEL + depends on (!INTERRUPTOFFHIST &&!PREEMPTOFFHIST &&!WAKEUPLATENCYHIST &&!MISSEDTIMEROFFSETSHIST) help This option enables a transparent branch optimization that makes certain almost-always-true or almost-always-false branch.