Commit ade56da1 authored by Tomas Krizek's avatar Tomas Krizek

doc: statcmp

parent 6f94d514
......@@ -58,6 +58,7 @@ Respdiff is conceptually chain of independent tools:
6. `histogram <doc/histogram.rst>`__: plot graph of answer latencies
7. `sumcmp <doc/sumcmp.rst>`__: compare a new diffsum report to a reference one
8. `sumstat <doc/sumstat.rst>`__: generate statistics from multiple diffsum reports
9. `statcmp <doc/statcmp.rst>`__: plot and compare statistics and reports
Changelog
......
Statcmp
=======
Usage
-----
.. code-block:: console
$ ./statcmp.py -s stats.json # plot statistics
$ ./statcmp.py -s stats.json -c respdiff.cfg report.json # plot report comparison
$ ./statcmp.py --help # for more info
Description
-----------
Plots statistics from ``stats.json`` and optionally compares given reports to
those statistics.
Multiple reports can be compared against the statistical data at the same time.
If any upper threshold is exceeded during comparison, the resulting exit code
will be non-zero.
Violin graph
------------
The generated plots contain so called violin graphs. The "body" (gray) of these
graphs is basically a flipped histogram. It shows how frequently the displayed
values appear in the given statistical sample.
There are 3 gray boundaries on each graph - minimum, median and maximum. The
orange bondary represents the selected upper boundary against which reports are
compared.
.. image:: example_violin_noreport.png
:alt: (violin graph of statistical distribution: see example_violin_noreport.png)
If a report is given, there are three possible outcomes. If the field falls
within minimum and selected upper boundary, it is drawn blue.
.. image:: example_violin_passed.png
:alt: (violin graph with a passing sample: see example_violin_passed.png)
If the field value exceeds the selected upper boundary, it is drawn red and
highlighted. These cases should be examined and usually represent a detected
issue.
.. image:: example_violin_failed.png
:alt: (violin graph with a failing sample: see example_violin_failed.png)
If the field value falls below the known minumum, it is drawn green and
highlighted. These signify a possible improvement.
.. image:: example_violin_improvement.png
:alt: (violin graph with an improved sample: see example_violin_improvement.png)
When multiple reports are given for comparison, a line is drawn for each sample.
.. image:: example_violin_multi.png
:alt: (violin graph with multiple sample: see example_violin_multi.png)
Utility scripts
---------------
TODO
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment