.. _tools:

#####
Tools
#####

This section describes the set of utilities included with the Vectorscan library.

********************
Quick Check: hscheck
********************

The ``hscheck`` tool allows the user to quickly check whether Vectorscan supports
a group of patterns. If a pattern is rejected by Vectorscan's compiler, the
compile error is provided on standard output.

For example, given the following three patterns (the last of which contains a
syntax error) in a file called ``/tmp/test``::

    1:/foo.*bar/
    2:/abc|def|ghi/
    3:/((foo|bar)/

... the ``hscheck`` tool will produce the following output::

    $ bin/hscheck -e /tmp/test

    OK: 1:/foo.*bar/
    OK: 2:/abc|def|ghi/
    FAIL (compile): 3:/((foo|bar)/: Missing close parenthesis for group started at index 0.
    SUMMARY: 1 of 3 failed.

********************
Benchmarker: hsbench
********************

The ``hsbench`` tool provides an easy way to measure Vectorscan's performance
for a particular set of patterns and corpus of data to be scanned.

Patterns are supplied in the format described below in
:ref:`tools_pattern_format`, while the corpus must be provided in the form of a
`corpus database`: this is a simple SQLite database format intended to allow for
easy control of how a corpus is broken into blocks and streams.

.. note:: A group of Python scripts for constructing corpora databases from
   various input types, such as PCAP network traffic captures or text files, can
   be found in the Vectorscan source tree in ``tools/hsbench/scripts``.

Running hsbench
===============

Given a file full of patterns specified with ``-e`` and a corpus database
specified with ``-c``, ``hsbench`` will perform a single-threaded benchmark and
produce output like this::

    $ hsbench -e /tmp/patterns -c /tmp/corpus.db

    Signatures:        /tmp/patterns
    Vectorscan info:    Version: 5.4.11 Features:  AVX2 Mode: STREAM
    Expression count:  200
    Bytecode size:     342,540 bytes
    Database CRC:      0x6cd6b67c
    Stream state size: 252 bytes
    Scratch size:      18,406 bytes
    Compile time:      0.153 seconds
    Peak heap usage:   78,073,856 bytes

    Time spent scanning:     0.600 seconds
    Corpus size:             72,138,183 bytes (63,946 blocks in 8,891 streams)
    Scan matches:            81 (0.001 matches/kilobyte)
    Overall block rate:      2,132,004.45 blocks/sec
    Overall throughput:      19,241.10 Mbit/sec

By default, the corpus is scanned twenty times, and the overall performance
reported is computed based the total number of bytes scanned in the time it
takes to perform all twenty scans. The number of repeats can be changed with the
``-n`` argument, and the results of each scan will be displayed if the
``--per-scan`` argument is specified.

To benchmark Vectorscan on more than one core, you can supply a list of cores
with the ``-T`` argument, which will instruct ``hsbench`` to start one
benchmark thread per core given and compute the throughput from the time taken
to complete all of them.

.. tip:: For single-threaded benchmarks on multi-processor systems, we recommend
   using a utility like ``taskset`` to lock the hsbench process to one core and
   minimize jitter due to the operating system's scheduler.

*******************************
Correctness Testing: hscollider
*******************************

The ``hscollider`` tool, or Pattern Collider, provides a way to verify
Vectorscan's matching behaviour. It does this by compiling and scanning patterns
(either singly or in groups) against known corpora and comparing the results
against another engine (the "ground truth"). Two sources of ground truth for
comparison are available:

 * The PCRE library (http://pcre.org/).
 * An NFA simulation run on Vectorscan's compile-time graph representation. This
   is used if PCRE cannot support the pattern or if PCRE execution fails due to
   a resource limit.

Much of Vectorscan's testing infrastructure is built on ``hscollider``, and the
tool is designed to take advantage of multiple cores and provide considerable
flexibility in controlling the test. These options are described in the help
(``hscollider -h``) and include:

 * Testing in streaming, block or vectored mode.
 * Testing corpora at different alignments in memory.
 * Testing patterns in groups of varying size.
 * Manipulating stream state or scratch space between tests.
 * Cross-compilation and serialization/deserialization of databases.
 * Synthetic generation of corpora given a pattern set.

Using hscollider to debug a pattern
===================================

One common use-case for ``hscollider`` is to determine whether Vectorscan will
match a pattern in the expected location, and whether this accords with PCRE's
behaviour for the same case.

Here is an example. We put our pattern in a file in Vectorscan's pattern
format::

    $ cat /tmp/pat
    1:/hatstand.*badgerbrush/

We put the corpus to be scanned in another file, with the same numeric
identifier at the start to indicate that it should match pattern 1::

    $ cat /tmp/corpus
    1:__hatstand__hatstand__badgerbrush_badgerbrush

Then we can run ``hscollider`` with its verbosity turned up (``-vv``) so that
individual matches are displayed in the output::

    $ bin/ue2collider -e /tmp/pat -c /tmp/corpus -Z 0 -T 1 -vv
    ue2collider: The Pattern Collider Mark II

    Number of threads:  1 (1 scanner, 1 generator)
    Expression path:    /tmp/pat
    Signature files:    none
    Mode of operation:  block mode
    UE2 scan alignment: 0
    Corpora read from file: /tmp/corpus

    Running single-pattern/single-compile test for 1 expressions.

    PCRE Match @ (2,45)
    PCRE Match @ (2,33)
    PCRE Match @ (12,45)
    PCRE Match @ (12,33)
    UE2 Match @ (0,33) for 1
    UE2 Match @ (0,45) for 1
    Scan call returned 0
    PASSED: id 1, alignment 0, corpus 0 (matched pcre:2, ue2:2)
    Thread 0 processed 1 units.

    Summary:
    Mode:                           Single/Block
    =========
    Expressions processed:          1
    Corpora processed:              1
    Expressions with failures:      0
      Corpora generation failures:  0
      Compilation failures:         pcre:0, ng:0, ue2:0
      Matching failures:            pcre:0, ng:0, ue2:0
      Match differences:            0
      No ground truth:              0
    Total match differences:        0

    Total elapsed time: 0.00522815 secs.

We can see from this output that both PCRE and Vectorscan find matches ending at
offset 33 and 45, and so ``hscollider`` considers this test case to have
passed.

(In the example command line above, ``-Z 0`` instructs us to only test at
corpus alignment 0, and ``-T 1`` instructs us to only use one thread.)

.. note:: In default operation, PCRE produces only one match for a scan, unlike
  Vectorscan's automata semantics. The ``hscollider`` tool uses libpcre's
  "callout" functionality to match Vectorscan's semantics.

Running a larger scan test
==========================

A set of patterns for testing purposes are distributed with Vectorscan, and these
can be tested via ``hscollider`` on an in-tree build. Two CMake targets are
provided to do this easily:

================================= =====================================
Make Target                       Description
================================= =====================================
``make collide_quick_test``       Tests all patterns in streaming mode.
``make collide_quick_test_block`` Tests all patterns in block mode.
================================= =====================================

*****************
Debugging: hsdump
*****************

When built in debug mode (using the CMake directive ``CMAKE_BUILD_TYPE`` set to
``Debug``), Vectorscan includes support for dumping information about its
internals during pattern compilation with the ``hsdump`` tool.

This information is mostly of use to Vectorscan developers familiar with the
library's internal structure, but can be used to diagnose issues with patterns
and provide more information in bug reports.

.. _tools_pattern_format:

**************
Pattern Format
**************

All of the Vectorscan tools accept patterns in the same format, read from plain
text files with one pattern per line. Each line looks like this:

* ``<integer id>:/<regex>/<flags>``

For example::

    1:/hatstand.*teakettle/s
    2:/(hatstand|teakettle)/iH
    3:/^.{10,20}hatstand/m

The integer ID is the value that will be reported when a match is found by
Vectorscan and must be unique.

The pattern itself is a regular expression in PCRE syntax; see
:ref:`compilation` for more information on supported features.

The flags are single characters that map to Vectorscan flags as follows:

=========   =================================    ===========
Character   API Flag                             Description
=========   =================================    ===========
``i``       :c:member:`HS_FLAG_CASELESS`         Case-insensitive matching
``s``       :c:member:`HS_FLAG_DOTALL`           Dot (``.``) will match newlines
``m``       :c:member:`HS_FLAG_MULTILINE`        Multi-line anchoring
``H``       :c:member:`HS_FLAG_SINGLEMATCH`      Report match ID at most once
``V``       :c:member:`HS_FLAG_ALLOWEMPTY`       Allow patterns that can match against empty buffers
``8``       :c:member:`HS_FLAG_UTF8`             UTF-8 mode
``W``       :c:member:`HS_FLAG_UCP`              Unicode property support
``P``       :c:member:`HS_FLAG_PREFILTER`        Prefiltering mode
``L``       :c:member:`HS_FLAG_SOM_LEFTMOST`     Leftmost start of match reporting
``C``       :c:member:`HS_FLAG_COMBINATION`      Logical combination of patterns
``Q``       :c:member:`HS_FLAG_QUIET`            Quiet at matching
=========   =================================    ===========

In addition to the set of flags above, :ref:`extparam` can be supplied
for each pattern. These are supplied after the flags as ``key=value`` pairs
between braces, separated by commas. For example::

    1:/hatstand.*teakettle/s{min_offset=50,max_offset=100}

All Vectorscan tools will accept a pattern file (or a directory containing
pattern files) with the ``-e`` argument. If no further arguments constraining
the pattern set are given, all patterns in those files are used.

To select a subset of the patterns, a single ID can be supplied with the ``-z``
argument, or a file containing a set of IDs can be supplied with the ``-s``
argument.
