keep debian folder in a separate branch

CHANGELOG.md

@@ -1,336 +0,0 @@
-# Hyperscan Change Log
-
-This is a list of notable changes to Hyperscan, in reverse chronological order.
-
-## [5.4.0] 2020-12-31
-- Improvement on literal matcher "Fat Teddy" performance, including
-  support for Intel(R) AVX-512 Vector Byte Manipulation Instructions (Intel(R)
-  AVX-512 VBMI).
-- Introduce a new 32-state shuffle-based DFA engine ("Sheng32"). This improves
-  scanning performance by leveraging AVX-512 VBMI.
-- Introduce a new 64-state shuffle-based DFA engine ("Sheng64"). This improves
-  scanning performance by leveraging AVX-512 VBMI.
-- Introduce a new shuffle-based hybrid DFA engine ("McSheng64"). This improves
-  scanning performance by leveraging AVX-512 VBMI.
-- Improvement on exceptional state handling performance for LimEx NFA, including
-  support for AVX-512 VBMI.
-- Improvement on lookaround performance with new models, including support for
-  AVX-512.
-- Improvement on DFA state space efficiency.
-- Optimization on decision of NFA/DFA generation.
-- hsbench: add CSV dump support for hsbench.
-- Bugfix for cmake error on Icelake under release mode.
-- Bugfix in find_vertices_in_cycles() to avoid self-loop checking in SCC.
-- Bugfix for issue #270: fix return value handling in chimera.
-- Bugfix for issue #284: use correct free function in logical combination.
-- Add BUILD_EXAMPLES cmake option to enable example code compilation. (#260)
-- Some typo fixing. (#242, #259)
-
-## [5.3.0] 2020-05-15
-- Improvement on literal matcher "Teddy" performance, including support for
-  Intel(R) AVX-512 Vector Byte Manipulation Instructions (Intel(R) AVX-512
-  VBMI).
-- Improvement on single-byte/two-byte matching performance, including support
-  for Intel(R) Advanced Vector Extensions 512 (Intel(R) AVX-512).
-- hsbench: add hyphen support for -T option.
-- tools/fuzz: add test scripts for synthetic pattern generation.
-- Bugfix for acceleration path analysis in LimEx NFA.
-- Bugfix for duplicate matches for Small-write engine.
-- Bugfix for UTF8 checking problem for hscollider.
-- Bugfix for issue #205: avoid crash of `hs_compile_lit_multi()` with clang and
-  ASAN.
-- Bugfix for issue #211: fix error in `db_check_platform()` function.
-- Bugfix for issue #217: fix cmake parsing issue of CPU arch for non-English
-  locale.
-- Bugfix for issue #228: avoid undefined behavior when calling `close()` after
-  `fdopendir()` in `loadExpressions()`.
-- Bugfix for issue #239: fix hyperscan compile issue under gcc-10.
-- Add VLAN packets processing capability in pcap analysis script. (#214)
-- Avoid extra convert instruction for "Noodle". (#221)
-- Add Hyperscan version marcro in `hs.h`. (#222)
-
-## [5.2.1] 2019-10-13
-- Bugfix for issue #186: fix compile issue when `BUILD_SHARED_LIBS` is on in
-  release mode.
-- Disable redundant move check for older compiler versions.
-
-## [5.2.0] 2019-07-12
-- Literal API: add new API `hs_compile_lit()` and `hs_compile_lit_multi()` to
-  process pure literal rule sets. The 2 literal APIs treat each expression text
-  in a literal sense without recognizing any regular grammers.
-- Logical combination: add support for purely negative combinations, which
-  report match at EOD in case of no sub-expressions matched.
-- Windows porting: support shared library (DLL) on Windows with available tools
-  hscheck, hsbench and hsdump.
-- Bugfix for issue #148: fix uninitialized use of `scatter_unit_uX` due to
-  padding.
-- Bugfix for issue #155: fix numerical result out of range error.
-- Bugfix for issue #165: avoid corruption of pending combination report in
-  streaming mode.
-- Bugfix for issue #174: fix scratch free issue when memory allocation fails.
-
-## [5.1.1] 2019-04-03
-- Add extra detection and handling when invalid rose programs are triggered.
-- Bugfix for issue #136: fix CMake parsing of CPU architecure for GCC-9.
-- Bugfix for issue #137: avoid file path impact on fat runtime build.
-- Bugfix for issue #141: fix rose literal programs for multi-pattern
-  matching when no pattern ids are provided.
-- Bugfix for issue #144: fix library install path in pkg-config files.
-
-## [5.1.0] 2019-01-17
-- Improve DFA state compression by wide-state optimization to reduce bytecode
-  size.
-- Create specific interpreter runtime handling to boost the performance of pure
-  literal matching.
-- Optimize original presentation of interpreter (the "Rose" engine ) to
-  increase overall performance.
-- Bugfix for logical combinations: fix error reporting combination's match in
-  case of sub-expression has EOD match under streaming mode.
-- Bugfix for logical combinations: fix miss reporting combination's match under
-  vacuous input.
-- Bugfix for issue #104: fix compile error with Boost 1.68.0.
-- Bugfix for issue #127: avoid pcre error for hscollider with installed PCRE
-  package.
-- Update version of PCRE used by testing tools as a syntax and semantic
-  reference to PCRE 8.41 or above.
-- Fix github repo address in doc.
-
-## [5.0.0] 2018-07-09
-- Introduce chimera hybrid engine of Hyperscan and PCRE, to fully support
-  PCRE syntax as well as to take advantage of the high performance nature of
-  Hyperscan.
-- New API feature: logical combinations (AND, OR and NOT) of patterns in a
-  given pattern set.
-- Windows porting: hsbench, hscheck, hscollider and hsdump tools now available
-  on Windows 8 or newer.
-- Improve undirected graph implementation to avoid graph copy and reduce
-  compile time.
-- Bugfix for issue #86: enable hscollider for installed PCRE package.
-
-## [4.7.0] 2018-01-24
-- Introduced hscollider pattern testing tool, for validating Hyperscan match
-  behaviour against PCRE.
-- Introduced hscheck pattern compilation tool.
-- Introduced hsdump development tool for producing information about Hyperscan
-  pattern compilation.
-- New API feature: extended approximate matching support for Hamming distance.
-- Bugfix for issue #69: Force C++ linkage in Xcode.
-- Bugfix for issue #73: More documentation for `hs_close_stream()`.
-- Bugfix for issue #78: Fix for fat runtime initialisation when used as a
-  shared library.
-
-## [4.6.0] 2017-09-22
-- New API feature: stream state compression. This allows the user to compress
-  and restore state for streams to reduce memory usage.
-- Many improvements to literal matching performance, including more support
-  for Intel(R) Advanced Vector Extensions 512 (Intel(R) AVX-512).
-- Compile time improvements, mainly reducing compiler memory allocation.
-  Also results in reduced compile time for some pattern sets.
-- Bugfix for issue #62: fix error building Hyperscan using older versions of
-  Boost.
-- Small updates to fix warnings identified by Coverity.
-
-## [4.5.2] 2017-07-26
-- Bugfix for issue #57: Treat characters between `\Q.\E` as codepoints in
-  UTF8 mode.
-- Bugfix for issue #60: Use a portable flag for mktemp for fat runtime builds.
-- Bugfix for fat runtime builds on AVX-512 capable machines with Hyperscan's
-  AVX-512 support disabled.
-
-## [4.5.1] 2017-06-16
-- Bugfix for issue #56: workaround for gcc-4.8 C++11 defect.
-- Bugfix for literal matching table generation, reversing a regression in
-  performance for some literal matching cases.
-- Bugfixes for hsbench, related to multicore benchmarking, portability fixes
-  for FreeBSD, and clarifying output results.
-- CMake: removed a duplicate else branch that causes very recent (v3.9) builds
-  of CMake to fail.
-
-## [4.5.0] 2017-06-09
-- New API feature: approximate matching using the "edit distance" extended
-  parameter. This allows the user to request all matches that are a given edit
-  distance from an exact match for a pattern.
-- Initial support for Intel(R) Advanced Vector Extensions 512 (Intel(R)
-  AVX-512), disabled by default. To enable it, pass `-DBUILD_AVX512=1` to
-  `cmake`.
-- Major compile time improvements in many subsystems, reducing compile time
-  significantly for many large pattern sets.
-- Internal reworking of literal matchers to operate on literals of at
-  most eight characters, with subsequent confirmation done in the Rose
-  interpreter. This reduces complexity and bytecode size and improves
-  performance for many pattern sets.
-- Improve performance of the FDR literal matcher front end.
-- Improve bucket assignment and other heuristics governing the FDR literal
-  matcher.
-- Improve optimisation passes that take advantage of extended parameter
-  constraints (`min_offset`, etc).
-- Introduce further lookaround specialisations to improve scanning performance.
-- Optimise Rose interpreter construction to reduce the length of programs
-  generated in some situations.
-- Remove the old "Rose" pattern decomposition analysis pass in favour of the
-  new "Violet" pass introduced in Hyperscan 4.3.0.
-- In streaming mode, allow exhaustion (where the stream can no longer produce
-  matchers) to be detected in more situations, improving scanning performance.
-- Improve parsing of control verbs (such as `(*UTF8)`) that can only occur at
-  the beginning of the pattern. Combinations of supported verbs in any order
-  are now permitted.
-- Update version of PCRE used by testing tools as a syntax and semantic
-  reference to PCRE 8.40.
-- Tuning support for Intel(R) microarchitecture code names Skylake, Skylake
-  Server, Goldmont.
-- CMake: when building a native build with a version of GCC that doesn't
-  recognise the host compiler, tune for the microarch selected by
-  `-march=native`.
-- CMake: don't fail if SQLite (which is only required to build the `hsbench`
-  tool) is not present.
-- CMake: detect libc++ directly and use that to inform the Boost version
-  requirement.
-- Bugfix for issue #51: make the fat runtime build wrapper less fragile.
-- Bugfix for issues #46, #52: use `sqlite3_errmsg()` to allow SQLite 3.6.x to
-  be used. Thanks to @EaseTheWorld for the PR.
-
-## [4.4.1] 2017-02-28
-- Bugfixes to fix issues where stale data was being referenced in scratch
-  memory. In particular this may have resulted in `hs_close_stream()`
-  referencing data from other previously scanned streams. This may result in
-  incorrect matches being been reported.
-
-## [4.4.0] 2017-01-20
-- Introduce the "fat runtime" build. This will build several variants of the
-  Hyperscan scanning engine specialised for different processor feature sets,
-  and use the appropriate one for the host at runtime. This uses the "ifunc"
-  indirect function attribute provided by GCC and is currently available on
-  Linux only, where it is the default for release builds.
-- New API function: add the `hs_valid_platform()` function. This function tests
-  whether the host provides the SSSE3 instruction set required by Hyperscan.
-- Introduce a new standard benchmarking tool, "hsbench". This provides an easy
-  way to measure Hyperscan's performance for a particular set of patterns and
-  corpus of data to be scanned.
-- Introduce a 64-bit GPR LimEx NFA model, which uses 64-bit GPRs on 64-bit
-  hosts and SSE registers on 32-bit hosts.
-- Introduce a new DFA model ("McSheng") which is a hybrid of the existing
-  McClellan and Sheng models. This improves scanning performance for some
-  cases.
-- Introduce lookaround specialisations to improve scanning performance.
-- Improve the handling of long literals by moving confirmation to the Rose
-  interpreter and simplifying the hash table used to track them in streaming
-  mode.
-- Improve compile time optimisation for removing redundant paths from
-  expression graphs.
-- Build: improve support for building with MSVC toolchain.
-- Reduce the size of small write DFAs used for small scans in block mode.
-- Introduce a custom graph type (`ue2_graph`) used in place of the Boost Graph
-  Library's `adjacency_list` type. Improves compile time performance and type
-  safety.
-- Improve scanning performance of the McClellan DFA.
-- Bugfix for a very unusual SOM case where the incorrect start offset was
-  reported for a match.
-- Bugfix for issue #37, removing execute permissions from some source files.
-- Bugfix for issue #41, handle Windows line endings in pattern files.
-
-## [4.3.2] 2016-11-15
-- Bugfix for issue #39. This small change is a workaround for an issue in
-  Boost 1.62. The fix has been submitted to Boost for inclusion in a future
-  release.
-
-## [4.3.1] 2016-08-29
-- Bugfix for issue #30. In recent versions of Clang, a write to a variable was
-  being elided, resulting in corrupted stream state after calling
-  `hs_reset_stream()`.
-
-## [4.3.0] 2016-08-24
-- Introduce a new analysis pass ("Violet") used for decomposition of patterns
-  into literals and smaller engines.
-- Introduce a new container engine ("Tamarama") for infix and suffix engines
-  that can be proven to run exclusively of one another. This reduces stream
-  state for pattern sets with many such engines.
-- Introduce a new shuffle-based DFA engine ("Sheng"). This improves scanning
-  performance for pattern sets where small engines are generated.
-- Improve the analysis used to extract extra mask information from short
-  literals.
-- Reduced compile time spent in equivalence class analysis.
-- Build: frame pointers are now only omitted for 32-bit release builds.
-- Build: Workaround for C++ issues reported on FreeBSD/libc++ platforms.
-  (github issue #27)
-- Simplify the LimEx NFA with a unified "variable shift" model, which reduces
-  the number of different NFA code paths to one per model size.
-- Allow some anchored prefixes that may squash the literal to which they are
-  attached to run eagerly. This improves scanning performance for some
-  patterns.
-- Simplify and improve EOD ("end of data") matching, using the interpreter for
-  all operations.
-- Elide unnecessary instructions in the Rose interpreter at compile time.
-- Reduce the number of inlined instantiations of the Rose interpreter in order
-  to reduce instruction cache pressure.
-- Small improvements to literal matcher acceleration.
-- Parser: ignore `\E` metacharacters that are not preceded by `\Q`. This
-  conforms to PCRE's behaviour, rather than returning a compile error.
-- Check for misaligned memory when allocating an error structure in Hyperscan's
-  compile path and return an appropriate error if detected.
-
-## [4.2.0] 2016-05-31
-- Introduce an interpreter for many complex actions to replace the use of
-  internal reports within the core of Hyperscan (the "Rose" engine). This
-  improves scanning performance and reduces database size for many pattern
-  sets.
-- Many enhancements to the acceleration framework used by NFA and DFA engines,
-  including more flexible multibyte implementations and more AVX2 support. This
-  improves scanning performance for many pattern sets.
-- Improved prefiltering support for complex patterns containing very large
-  bounded repeats (`R{M,N}` with large `N`).
-- Improve scanning performance of pattern sets with a very large number of
-  EOD-anchored patterns.
-- Improve scanning performance of large pattern sets that use the
-  `HS_FLAG_SINGLEMATCH` flag.
-- Improve scanning performance of pattern sets that contain a single literal by
-  improving the "Noodle" literal matcher.
-- Small reductions in total stream state for many pattern sets.
-- Improve runtime detection of AVX2 support.
-- Disable -Werror for release builds, in order to behave better for packagers
-  and users with different compiler combinations than those that we test.
-- Improve support for building on Windows with MSVC 2015 (github issue #14).
-  Support for Hyperscan on Windows is still experimental.
-- Small updates to fix warnings identified by Coverity.
-- Remove Python codegen for the "FDR" and "Teddy" literal matchers. These are
-  now implemented directly in C code.
-- Remove the specialist "Sidecar" engine in favour of using our more general
-  repeat engines.
-- New API function: add the `hs_expression_ext_info()` function. This is a
-  variant of `hs_expression_info()` that can accept patterns with extended
-  parameters.
-- New API error value: add the `HS_SCRATCH_IN_USE` error, which is returned
-  when Hyperscan detects that a scratch region is already in use on entry to an
-  API function.
-
-## [4.1.0] 2015-12-18
-- Update version of PCRE used by testing tools as a syntax and semantic
-  reference to PCRE 8.38.
-- Small updates to fix warnings identified by Coverity.
-- Clean up and unify exception handling behaviour across GPR and SIMD NFA
-  models.
-- Fix bug in handling of bounded repeat triggers with large gaps between them
-  for sparse repeat model.
-- Correctly reject POSIX collating elements (`[.ch.]`, `[=ch=]`) in the parser.
-  These are not supported by Hyperscan.
-- Add support for quoted sequences (`\Q...\E`) inside character classes.
-- Simplify FDR literal matcher runtime by removing some static specialization.
-- Fix handling of the POSIX `[:graph:]`, `[:print:]` and `[:punct:]` character
-  classes to match the behaviour of PCRE 8.38 in both standard operation and
-  with the UCP flag set. (Note: some bugs were fixed in this area in PCRE
-  8.38.) Previously Hyperscan's behaviour was the same as versions of PCRE
-  before 8.34.
-- Improve performance when compiling pattern sets that include a large number
-  of similar bounded repeat constructs. (github issue #9)
-
-## [4.0.1] 2015-10-30
-- Minor cleanups to test code.
-- CMake and other build system improvements.
-- API update: allow `hs_reset_stream()` and `hs_reset_and_copy_stream()` to be
-  supplied with a NULL scratch pointer if no matches are required. This is in
-  line with the behaviour of `hs_close_stream()`.
-- Disallow bounded repeats with a very large minimum repeat but no maximum,
-  i.e. {N,} for very large N.
-- Reduce compile memory usage in literal set explansion for some large cases.
-
-## [4.0.0] 2015-10-20
-- Original release of Hyperscan as open-source software.

COPYING

@@ -1,27 +0,0 @@
-Copyright (c) 2015, Intel Corporation
-Copyright (c) 2019-20, VectorCamp PC
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
- * Redistributions of source code must retain the above copyright notice,
-   this list of conditions and the following disclaimer.
- * Redistributions in binary form must reproduce the above copyright
-   notice, this list of conditions and the following disclaimer in the
-   documentation and/or other materials provided with the distribution.
- * Neither the name of Intel Corporation nor the names of its contributors
-   may be used to endorse or promote products derived from this software
-   without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
-FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
-SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
-CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
-OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-

Jenkinsfile

@@ -1,590 +0,0 @@
-pipeline {
-    agent none
-    stages {
-        stage("Build") {
-            failFast true
-            parallel {
-                stage("Release/SSE") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-release-SSE', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=no -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-release-SSE/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-release-SSE/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Release/AVX2") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-release-AVX2', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-release-AVX2/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-release-AVX2/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Release/AVX512") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-release-AVX512', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-release-AVX512/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-release-AVX512/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Release/FAT") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-release-fat', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=yes', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-release-fat/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Debug/SSE") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-debug-SSE', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=no -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-debug-SSE/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-debug-SSE/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Debug/AVX2") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-debug-AVX2', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-debug-AVX2/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-debug-AVX2/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Debug/AVX512") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-debug-AVX512', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-debug-AVX512/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-debug-AVX512/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Debug/FAT") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-debug-fat', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=yes', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-debug-fat/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Release/ARM") {
-                    agent { label "arm" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-release-arm', buildType: 'Release', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-release-arm/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-release-arm/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Debug/ARM") {
-                    agent { label "arm" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-debug-arm', buildType: 'Debug', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-debug-arm/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-debug-arm/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Release/Power") {
-                    agent { label "power" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-release-power', buildType: 'Release', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-release-power/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-release-power/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Debug/Power") {
-                    agent { label "power" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-debug-power', buildType: 'Debug', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-debug-power/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-debug-power/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Clang-Release/SSE") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-clang-release-SSE', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=no -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-clang-release-SSE/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-clang-release-SSE/bin/unit-hyperscan'
-                            }
-                        }
-                    }
-                }
-                stage("Clang-Release/AVX2") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-clang-release-AVX2', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-clang-release-AVX2/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-clang-release-AVX2/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Clang-Release/AVX512") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-clang-release-AVX512', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-clang-release-AVX512/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-clang-release-AVX512/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Clang-Release/FAT") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-clang-release-fat', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=yes', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-clang-release-fat/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Clang-Debug/SSE") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-clang-debug-SSE', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=no -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-clang-debug-SSE/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-clang-debug-SSE/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Clang-Debug/AVX2") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-clang-debug-AVX2', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-clang-debug-AVX2/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-clang-debug-AVX2/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Clang-Debug/AVX512") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-clang-debug-AVX512', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-clang-debug-AVX512/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-clang-debug-AVX512/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Clang-Debug/FAT") {
-                    agent { label "x86" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-clang-debug-fat', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=yes', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-clang-debug-fat/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Clang-Release/ARM") {
-                    agent { label "arm" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-clang-release-arm', buildType: 'Release', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-clang-release-arm/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-clang-release-arm/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Clang-Debug/ARM") {
-                    agent { label "arm" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-debug-arm', buildType: 'Debug', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-clang-debug-arm/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-clang-debug-arm/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Clang-Release/Power") {
-                    agent { label "power" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-clang-release-power', buildType: 'Release', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-clang-release-power/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-clang-release-power/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-                stage("Clang-Debug/Power") {
-                    agent { label "power" }
-                    stages {
-                        stage("Git checkout") {
-                            steps {
-                                checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
-                            }
-                        } 
-                        stage("Build") {
-                            steps {
-                                cmakeBuild buildDir: 'build-clang-debug-power', buildType: 'Debug', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
-                            }
-                        }
-                        stage("Unit Test") {
-                            steps {
-                                sh 'build-clang-debug-power/bin/unit-internal'
-                            }
-                        }
-                        stage("Test") {
-                            steps {
-                                sh 'build-clang-debug-power/bin/unit-hyperscan'
-                            }
-                        }
-                    } 
-                }
-            }
-        }
-    }
-}

LICENSE

@@ -1,123 +0,0 @@
-Hyperscan is licensed under the BSD License.
-
-Copyright (c) 2015, Intel Corporation
-
-Vectorscan is licensed under the BSD License.
-
-Copyright (c) 2020, VectorCamp PC
-Copyright (c) 2021, Arm Limited
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-  * Redistributions of source code must retain the above copyright notice,
-    this list of conditions and the following disclaimer.
-  * Redistributions in binary form must reproduce the above copyright
-    notice, this list of conditions and the following disclaimer in the
-    documentation and/or other materials provided with the distribution.
-  * Neither the name of Intel Corporation nor the names of its contributors
-    may be used to endorse or promote products derived from this software
-    without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
-FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
-SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
-CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
-OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-
---------------------------------------------------------------------------------
-
-This product also contains code from third parties, under the following
-licenses:
-
-Intel's Slicing-by-8 CRC32 implementation
------------------------------------------
-
-Copyright (c) 2004-2006, Intel Corporation
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-  * Redistributions of source code must retain the above copyright notice,
-    this list of conditions and the following disclaimer.
-  * Redistributions in binary form must reproduce the above copyright notice,
-    this list of conditions and the following disclaimer in the documentation
-    and/or other materials provided with the distribution.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
-FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
-SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
-CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
-OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-
-Boost C++ Headers Library
--------------------------
-
-Boost Software License - Version 1.0 - August 17th, 2003
-
-Permission is hereby granted, free of charge, to any person or organization
-obtaining a copy of the software and accompanying documentation covered by
-this license (the "Software") to use, reproduce, display, distribute,
-execute, and transmit the Software, and to prepare derivative works of the
-Software, and to permit third-parties to whom the Software is furnished to
-do so, all subject to the following:
-
-The copyright notices in the Software and this entire statement, including
-the above license grant, this restriction and the following disclaimer,
-must be included in all copies of the Software, in whole or in part, and
-all derivative works of the Software, unless such copies or derivative
-works are solely in the form of machine-executable object code generated by
-a source language processor.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT
-SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE
-FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,
-ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
-DEALINGS IN THE SOFTWARE.
-
-
-The Google C++ Testing Framework (Google Test)
-----------------------------------------------
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are
-met:
-
-    * Redistributions of source code must retain the above copyright
-notice, this list of conditions and the following disclaimer.
-    * Redistributions in binary form must reproduce the above
-copyright notice, this list of conditions and the following disclaimer
-in the documentation and/or other materials provided with the
-distribution.
-    * Neither the name of Google Inc. nor the names of its
-contributors may be used to endorse or promote products derived from
-this software without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-

README.md

@@ -1,96 +0,0 @@
-# Vectorscan?
-
-A fork of Intel's Hyperscan, modified to run on more platforms. Currently ARM NEON/ASIMD
-is 100% functional, and Power VSX are in development. ARM SVE2 will be implemented when
-harwdare becomes accessible to the developers. More platforms will follow in the future,
-on demand/request.
-
-Vectorscan will follow Intel's API and internal algorithms where possible, but will not
-hesitate to make code changes where it is thought of giving better performance or better
-portability. In addition, the code will be gradually simplified and made more uniform and
-all architecture specific -currently Intel- #ifdefs will be removed and abstracted away.
-
-# Why the fork?
-
-Originally, the ARM porting was supposed to be merged into Intel's own Hyperscan, and 2 
-Pull Requests had been made to the project for this reason ([1], [2]). Unfortunately, the
-PRs were rejected for now and the forseeable future, thus we have created Vectorscan for 
-our own multi-architectural and opensource collaborative needs.
-
-# What is Hyperscan?
-
-Hyperscan is a high-performance multiple regex matching library. It follows the
-regular expression syntax of the commonly-used libpcre library, but is a
-standalone library with its own C API.
-
-Hyperscan uses hybrid automata techniques to allow simultaneous matching of
-large numbers (up to tens of thousands) of regular expressions and for the
-matching of regular expressions across streams of data.
-
-Vectorscan is typically used in a DPI library stack, just like Hyperscan.
-
-# Cross Compiling for AArch64
-
-- To cross compile for AArch64, first adjust the variables set in cmake/setenv-arm64-cross.sh.
-  - `export CROSS=<arm-cross-compiler-dir>/bin/aarch64-linux-gnu-`
-  - `export CROSS_SYS=<arm-cross-compiler-system-dir>`
-  - `export BOOST_PATH=<boost-source-dir>`
-- Set the environment variables:
-  - `source cmake/setenv-arm64-cross.sh`
-- Configure Vectorscan:
-  - `mkdir <build-dir-name>`
-  - `cd <build-dir>`
-  - `cmake -DCROSS_COMPILE_AARCH64=1 <hyperscan-source-dir> -DCMAKE_TOOLCHAIN_FILE=<hyperscan-source-dir>/cmake/arm64-cross.cmake`
-- Build Vectorscan:
-  - `make -jT` where T is the number of threads used to compile.
-  - `cmake --build . -- -j T` can also be used instead of make.
-
-# Compiling for SVE
-
-The following cmake variables can be set in order to target Arm's Scalable
-Vector Extension. They are listed in ascending order of strength, with cmake
-detecting whether the feature is available in the compiler and falling back to
-a weaker version if not. Only one of these variables needs to be set as weaker
-variables will be implied as set.
-
-- `BUILD_SVE`
-- `BUILD_SVE2`
-- `BUILD_SVE2_BITPERM`
-
-# Documentation
-
-Information on building the Hyperscan library and using its API is available in
-the [Developer Reference Guide](http://intel.github.io/hyperscan/dev-reference/).
-
-# License
-
-Vectorscan, like Hyperscan is licensed under the BSD License. See the LICENSE file in the
-project repository.
-
-# Versioning
-
-The `master` branch on Github will always contain the most recent release of
-Hyperscan. Each version released to `master` goes through QA and testing before
-it is released; if you're a user, rather than a developer, this is the version
-you should be using.
-
-Further development towards the next release takes place on the `develop`
-branch.
-
-# Get Involved
-
-The official homepage for Vectorscan is at [www.github.com/VectorCamp/vectorscan](https://www.github.com/VectorCamp/vectorscan).
-
-# Original Hyperscan links
-The official homepage for Hyperscan is at [www.hyperscan.io](https://www.hyperscan.io).
-
-If you have questions or comments, we encourage you to [join the mailing
-list](https://lists.01.org/mailman/listinfo/hyperscan). Bugs can be filed by
-sending email to the list, or by creating an issue on Github.
-
-If you wish to contact the Hyperscan team at Intel directly, without posting
-publicly to the mailing list, send email to
-[hyperscan@intel.com](mailto:hyperscan@intel.com).
-
-[1]: https://github.com/intel/hyperscan/pull/272
-[2]: https://github.com/intel/hyperscan/pull/287

benchmarks/CMakeLists.txt

@@ -1,6 +0,0 @@
-if (NOT FAT_RUNTIME AND (BUILD_STATIC_AND_SHARED OR BUILD_STATIC_LIBS))
-  add_executable(benchmarks benchmarks.cpp)
-  set_source_files_properties(benchmarks.cpp PROPERTIES COMPILE_FLAGS
-      "-Wall -Wno-unused-variable")
-  target_link_libraries(benchmarks hs)
-endif()

benchmarks/benchmarks.cpp

@@ -1,253 +0,0 @@
-/*
- * Copyright (c) 2020, 2021, VectorCamp PC
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include <iostream>
-#include <chrono>
-#include <cstring>
-#include <ctime>
-#include <cstdlib>
-#include <memory>
-#include <functional>
-
-#include "benchmarks.hpp"
-
-#define MAX_LOOPS    1000000000
-#define MAX_MATCHES  5
-#define N            8
-
-struct hlmMatchEntry {
-    size_t to;
-    u32 id;
-    hlmMatchEntry(size_t end, u32 identifier) :
-            to(end), id(identifier) {}
-};
-
-std::vector<hlmMatchEntry> ctxt;
-
-static
-hwlmcb_rv_t hlmSimpleCallback(size_t to, u32 id,
-                              UNUSED struct hs_scratch *scratch) {
-    DEBUG_PRINTF("match @%zu = %u\n", to, id);
-
-    ctxt.push_back(hlmMatchEntry(to, id));
-
-    return HWLM_CONTINUE_MATCHING;
-}
-
-template<typename InitFunc, typename BenchFunc>
-static void run_benchmarks(int size, int loops, int max_matches, bool is_reverse, MicroBenchmark &bench, InitFunc &&init, BenchFunc &&func) {
-    init(bench);
-    double total_sec = 0.0;            
-    u64a total_size = 0;
-    double bw = 0.0;
-    double avg_bw = 0.0;
-    double max_bw = 0.0;
-    double avg_time = 0.0;
-    if (max_matches) {
-        int pos = 0;
-        for(int j = 0; j < max_matches - 1; j++) {
-            bench.buf[pos] = 'b';
-            pos = (j+1) *size / max_matches ;
-            bench.buf[pos] = 'a';
-            u64a actual_size = 0;
-            auto start = std::chrono::steady_clock::now();
-            for(int i = 0; i < loops; i++) { 
-                const u8 *res = func(bench);
-		if (is_reverse)
-		   actual_size += bench.buf.data() + size - res;
-		else
-                   actual_size += res - bench.buf.data();
-            }
-            auto end = std::chrono::steady_clock::now();
-            double dt = std::chrono::duration_cast<std::chrono::microseconds>(end - start).count();
-            total_sec += dt;
-            /*convert microseconds to seconds*/
-            /*calculate bandwidth*/
-            bw  = (actual_size / dt) * 1000000.0 / 1048576.0;
-	    /*std::cout << "act_size = " << act_size << std::endl;
-	    std::cout << "dt = " << dt << std::endl;
-	    std::cout << "bw = " << bw << std::endl;*/
-	    avg_bw += bw;
-            /*convert to MB/s*/
-            max_bw = std::max(bw, max_bw);
-            /*calculate average time*/
-            avg_time += total_sec / loops;
-        }
-        avg_time /= max_matches;
-        avg_bw /= max_matches;
-	total_sec /= 1000000.0;
-        /*convert average time to us*/
-        printf(KMAG "%s: %u matches, %u * %u iterations," KBLU " total elapsed time =" RST " %.3f s, " 
-               KBLU "average time per call =" RST " %.3f μs," KBLU " max bandwidth = " RST " %.3f MB/s," KBLU " average bandwidth =" RST " %.3f MB/s \n",
-               bench.label, max_matches, size ,loops, total_sec, avg_time, max_bw, avg_bw);
-    } else {
-        auto start = std::chrono::steady_clock::now();
-        for (int i = 0; i < loops; i++) {
-            const u8 *res = func(bench);
-        }
-        auto end = std::chrono::steady_clock::now();
-        total_sec += std::chrono::duration_cast<std::chrono::microseconds>(end - start).count();
-        /*calculate transferred size*/
-        total_size = size * loops;
-        /*calculate average time*/
-        avg_time = total_sec / loops;
-        /*convert microseconds to seconds*/
-        total_sec /= 1000000.0;
-        /*calculate maximum bandwidth*/
-        max_bw = total_size / total_sec;
-        /*convert to MB/s*/
-        max_bw /= 1048576.0;
-        printf(KMAG "%s: no matches, %u * %u iterations," KBLU " total elapsed time =" RST " %.3f s, " 
-               KBLU "average time per call =" RST " %.3f μs ," KBLU " bandwidth = " RST " %.3f MB/s \n",
-               bench.label, size ,loops, total_sec, avg_time, max_bw );
-    }
-}
-
-int main(){
-    int matches[] = {0, MAX_MATCHES};
-    std::vector<size_t> sizes;
-    for (size_t i = 0; i < N; i++) sizes.push_back(16000 << i*2);
-    const char charset[] = "aAaAaAaAAAaaaaAAAAaaaaAAAAAAaaaAAaaa"; 
-  
-    for (int m = 0; m < 2; m++) {
-        for (size_t i = 0; i < std::size(sizes); i++) {
-            MicroBenchmark bench("Shufti", sizes[i]);
-            run_benchmarks(sizes[i], MAX_LOOPS / sizes[i], matches[m], false, bench,
-                [&](MicroBenchmark &b) {
-                    b.chars.set('a');
-                    ue2::shuftiBuildMasks(b.chars, (u8 *)&b.lo, (u8 *)&b.hi);
-                    memset(b.buf.data(), 'b', b.size);
-                },
-                [&](MicroBenchmark &b) {
-                    return shuftiExec(b.lo, b.hi, b.buf.data(), b.buf.data() + b.size);
-                }
-            );
-        }
-
-        for (size_t i = 0; i < std::size(sizes); i++) {
-            MicroBenchmark bench("Reverse Shufti", sizes[i]);
-            run_benchmarks(sizes[i], MAX_LOOPS / sizes[i], matches[m], true, bench,
-                [&](MicroBenchmark &b) {
-                    b.chars.set('a');
-                    ue2::shuftiBuildMasks(b.chars, (u8 *)&b.lo, (u8 *)&b.hi);
-                    memset(b.buf.data(), 'b', b.size);
-                },
-                [&](MicroBenchmark &b) {
-                    return rshuftiExec(b.lo, b.hi, b.buf.data(), b.buf.data() + b.size);
-                }
-            );
-        }
-
-        for (size_t i = 0; i < std::size(sizes); i++) {
-            MicroBenchmark bench("Truffle", sizes[i]);
-            run_benchmarks(sizes[i], MAX_LOOPS / sizes[i], matches[m], false, bench,
-                [&](MicroBenchmark &b) {
-                    b.chars.set('a');
-                    ue2::truffleBuildMasks(b.chars, (u8 *)&b.lo, (u8 *)&b.hi);
-                    memset(b.buf.data(), 'b', b.size);
-                },
-                [&](MicroBenchmark &b) {
-                    return truffleExec(b.lo, b.hi, b.buf.data(), b.buf.data() + b.size);
-                }
-            );
-        }
-
-        for (size_t i = 0; i < std::size(sizes); i++) {
-            MicroBenchmark bench("Reverse Truffle", sizes[i]);
-            run_benchmarks(sizes[i], MAX_LOOPS / sizes[i], matches[m], true, bench,
-                [&](MicroBenchmark &b) {
-                    b.chars.set('a');
-                    ue2::truffleBuildMasks(b.chars, (u8 *)&b.lo, (u8 *)&b.hi);
-                    memset(b.buf.data(), 'b', b.size);
-                },
-                [&](MicroBenchmark &b) {
-                    return rtruffleExec(b.lo, b.hi, b.buf.data(), b.buf.data() + b.size);
-                }
-            );
-        }
-
-        for (size_t i = 0; i < std::size(sizes); i++) {
-            MicroBenchmark bench("Vermicelli", sizes[i]);
-            run_benchmarks(sizes[i], MAX_LOOPS / sizes[i], matches[m], false, bench,
-                [&](MicroBenchmark &b) {
-                    b.chars.set('a');
-                    ue2::truffleBuildMasks(b.chars, (u8 *)&b.lo, (u8 *)&b.hi);
-                    memset(b.buf.data(), 'b', b.size);
-                },
-                [&](MicroBenchmark &b) {
-                    return vermicelliExec('a', 'b', b.buf.data(), b.buf.data() + b.size);
-                }
-            );
-        }
-
-        for (size_t i = 0; i < std::size(sizes); i++) {
-            MicroBenchmark bench("Reverse Vermicelli", sizes[i]);
-            run_benchmarks(sizes[i], MAX_LOOPS / sizes[i], matches[m], true, bench,
-                [&](MicroBenchmark &b) {
-                    b.chars.set('a');
-                    ue2::truffleBuildMasks(b.chars, (u8 *)&b.lo, (u8 *)&b.hi);
-                    memset(b.buf.data(), 'b', b.size);
-                },
-                [&](MicroBenchmark &b) {
-                    return rvermicelliExec('a', 'b', b.buf.data(), b.buf.data() + b.size);
-                }
-            );
-        }
-
-        for (size_t i = 0; i < std::size(sizes); i++) {
-            //we imitate the noodle unit tests
-            std::string str;
-            const size_t char_len = 5;
-            str.resize(char_len + 1);
-            for (size_t j=0; j < char_len; j++) {
-                srand (time(NULL));
-                int key = rand() % + 36 ;
-                str[char_len] = charset[key];
-                str[char_len + 1] = '\0';
-            }
-
-            MicroBenchmark bench("Noodle", sizes[i]);
-            run_benchmarks(sizes[i], MAX_LOOPS / sizes[i], matches[m], false, bench,
-                [&](MicroBenchmark &b) {
-                    ctxt.clear();
-                    memset(b.buf.data(), 'a', b.size);
-                    u32 id = 1000;
-                    ue2::hwlmLiteral lit(str, true, id);
-                    b.nt = ue2::noodBuildTable(lit);
-                    assert(b.nt != nullptr);
-                },
-                [&](MicroBenchmark &b) {
-                    noodExec(b.nt.get(), b.buf.data(), b.size, 0, hlmSimpleCallback, &b.scratch);
-                    return b.buf.data() + b.size;
-                }
-           );
-        }
-    }
-
-    return 0;
-}

benchmarks/benchmarks.hpp

@@ -1,69 +0,0 @@
-/*
- * Copyright (c) 2020, 2021, VectorCamp PC
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include "nfa/shufti.h"
-#include "nfa/shufticompile.h"
-#include "nfa/truffle.h"
-#include "nfa/trufflecompile.h"
-#include "nfa/vermicelli.hpp"
-#include "hwlm/noodle_build.h"
-#include "hwlm/noodle_engine.h"
-#include "hwlm/noodle_internal.h"
-#include "hwlm/hwlm_literal.h"
-#include "util/bytecode_ptr.h"
-#include "scratch.h"
-
-/*define colour control characters*/
-#define RST  "\x1B[0m"
-#define KRED  "\x1B[31m"
-#define KGRN  "\x1B[32m"
-#define KYEL  "\x1B[33m"
-#define KBLU  "\x1B[34m"
-#define KMAG  "\x1B[35m"
-#define KCYN  "\x1B[36m"
-#define KWHT  "\x1B[37m"
-
-class MicroBenchmark
-{
-public:
-  char const *label;
-  size_t size;
-
-  // Shufti/Truffle
-  m128 lo, hi;
-  ue2::CharReach chars;
-  std::vector<u8> buf;
-
-  // Noodle
-  struct hs_scratch scratch;
-  ue2::bytecode_ptr<noodTable> nt;
-
-  MicroBenchmark(char const *label_, size_t size_)
-  :label(label_), size(size_), buf(size_) {
-  };
-};

chimera/CMakeLists.txt

@@ -1,47 +0,0 @@
-# Chimera lib
-
-include_directories(${PCRE_INCLUDE_DIRS})
-
-# only set these after all tests are done
-set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${EXTRA_C_FLAGS}")
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${EXTRA_CXX_FLAGS}")
-
-SET(chimera_HEADERS
-    ch.h
-    ch_common.h
-    ch_compile.h
-    ch_runtime.h
-)
-install(FILES ${chimera_HEADERS} DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}/hs")
-
-SET(chimera_SRCS
-    ${chimera_HEADERS}
-    ch_alloc.c
-    ch_alloc.h
-    ch_compile.cpp
-    ch_database.c
-    ch_database.h
-    ch_internal.h
-    ch_runtime.c
-    ch_scratch.h
-    ch_scratch.c
-)
-
-add_library(chimera STATIC ${chimera_SRCS})
-add_dependencies(chimera hs pcre)
-target_link_libraries(chimera hs pcre)
-
-install(TARGETS chimera DESTINATION ${CMAKE_INSTALL_LIBDIR})
-
-# expand out library names for pkgconfig static link info
-foreach (LIB ${CMAKE_CXX_IMPLICIT_LINK_LIBRARIES})
-    # this is fragile, but protects us from toolchain specific files
-    if (NOT EXISTS ${LIB})
-        set(PRIVATE_LIBS "${PRIVATE_LIBS} -l${LIB}")
-    endif()
-endforeach()
-set(PRIVATE_LIBS "${PRIVATE_LIBS} -L${LIBDIR} -lpcre")
-
-configure_file(libch.pc.in libch.pc @ONLY) # only replace @ quoted vars
-install(FILES ${CMAKE_BINARY_DIR}/chimera/libch.pc
-    DESTINATION "${CMAKE_INSTALL_LIBDIR}/pkgconfig")

chimera/ch.h

@@ -1,45 +0,0 @@
-/*
- * Copyright (c) 2018, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef CH_H_
-#define CH_H_
-
-/**
- * @file
- * @brief The complete Chimera API definition.
- *
- * Chimera is a hybrid solution of Hyperscan and PCRE.
- *
- * This header includes both the Chimera compiler and runtime components. See
- * the individual component headers for documentation.
- */
-
-#include "ch_compile.h"
-#include "ch_runtime.h"
-
-#endif /* CH_H_ */

chimera/ch_alloc.c

@@ -1,109 +0,0 @@
-/*
- * Copyright (c) 2018, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Runtime functions for setting custom allocators.
- */
-
-#include "ch.h"
-#include "ch_common.h"
-#include "ch_internal.h"
-#include "hs.h"
-#include "ue2common.h"
-
-#define default_malloc malloc
-#define default_free free
-
-ch_alloc_t ch_database_alloc = default_malloc;
-ch_alloc_t ch_misc_alloc = default_malloc;
-ch_alloc_t ch_scratch_alloc = default_malloc;
-
-ch_free_t ch_database_free = default_free;
-ch_free_t ch_misc_free = default_free;
-ch_free_t ch_scratch_free = default_free;
-
-static
-ch_alloc_t normalise_alloc(ch_alloc_t a) {
-    if (!a) {
-        return default_malloc;
-    } else {
-        return a;
-    }
-}
-
-static
-ch_free_t normalise_free(ch_free_t f) {
-    if (!f) {
-        return default_free;
-    } else {
-        return f;
-    }
-}
-
-HS_PUBLIC_API
-ch_error_t HS_CDECL ch_set_allocator(ch_alloc_t allocfunc,
-                                     ch_free_t freefunc) {
-    ch_set_database_allocator(allocfunc, freefunc);
-    ch_set_misc_allocator(allocfunc, freefunc);
-    ch_set_scratch_allocator(allocfunc, freefunc);
-
-    // Set core Hyperscan alloc/free.
-    hs_error_t ret = hs_set_allocator(allocfunc, freefunc);
-
-    return ret;
-}
-
-HS_PUBLIC_API
-ch_error_t HS_CDECL ch_set_database_allocator(ch_alloc_t allocfunc,
-                                              ch_free_t freefunc) {
-    ch_database_alloc = normalise_alloc(allocfunc);
-    ch_database_free = normalise_free(freefunc);
-
-    // Set Hyperscan database alloc/free.
-    return hs_set_database_allocator(allocfunc, freefunc);
-}
-
-HS_PUBLIC_API
-ch_error_t HS_CDECL ch_set_misc_allocator(ch_alloc_t allocfunc,
-                                          ch_free_t freefunc) {
-    ch_misc_alloc = normalise_alloc(allocfunc);
-    ch_misc_free = normalise_free(freefunc);
-
-    // Set Hyperscan misc alloc/free.
-    return hs_set_misc_allocator(allocfunc, freefunc);
-}
-
-HS_PUBLIC_API
-ch_error_t HS_CDECL ch_set_scratch_allocator(ch_alloc_t allocfunc,
-                                             ch_free_t freefunc) {
-    ch_scratch_alloc = normalise_alloc(allocfunc);
-    ch_scratch_free = normalise_free(freefunc);
-
-    // Set Hyperscan scratch alloc/free.
-    return hs_set_scratch_allocator(allocfunc, freefunc);
-}

chimera/ch_alloc.h

@@ -1,65 +0,0 @@
-/*
- * Copyright (c) 2018, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef CH_ALLOC_H
-#define CH_ALLOC_H
-
-#include "hs_common.h"
-#include "ue2common.h"
-#include "ch_common.h"
-
-#ifdef __cplusplus
-extern "C"
-{
-#endif
-extern hs_alloc_t ch_database_alloc;
-extern hs_alloc_t ch_misc_alloc;
-extern hs_alloc_t ch_scratch_alloc;
-
-extern hs_free_t ch_database_free;
-extern hs_free_t ch_misc_free;
-extern hs_free_t ch_scratch_free;
-#ifdef __cplusplus
-} /* extern C */
-#endif
-/** \brief Check the results of an alloc done with hs_alloc for alignment.
- *
- * If we have incorrect alignment, return an error. Caller should free the
- * offending block. */
-static really_inline
-ch_error_t ch_check_alloc(const void *mem) {
-    ch_error_t ret = CH_SUCCESS;
-    if (!mem) {
-        ret = CH_NOMEM;
-    } else if (!ISALIGNED_N(mem, alignof(unsigned long long))) {
-        ret = CH_BAD_ALLOC;
-    }
-    return ret;
-}
-
-#endif

chimera/ch_common.h

@@ -1,370 +0,0 @@
-/*
- * Copyright (c) 2018-2020, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef CH_COMMON_H_
-#define CH_COMMON_H_
-
-#include "hs_common.h"
-
-#include <stdlib.h>
-
-/**
- * @file
- * @brief The Chimera common API definition.
- *
- * Chimera is a hybrid of Hyperscan and PCRE.
- *
- * This header contains functions available to both the Chimera compiler and
- * runtime.
- */
-
-#ifdef __cplusplus
-extern "C"
-{
-#endif
-
-struct ch_database;
-
-/**
- * A Chimera pattern database.
- *
- * Generated by one of the Chimera compiler functions:
- *  - @ref ch_compile()
- *  - @ref ch_compile_multi()
- *  - @ref ch_compile_ext_multi()
- */
-typedef struct ch_database ch_database_t;
-
-/**
- * A type for errors returned by Chimera functions.
- */
-typedef int ch_error_t;
-
-/**
- * Free a compiled pattern database.
- *
- * The free callback set by @ref ch_set_allocator()) will be used by this
- * function.
- *
- * @param db
- *      A compiled pattern database. NULL may also be safely provided, in which
- *      case the function does nothing.
- *
- * @return
- *      @ref CH_SUCCESS on success, other values on failure.
- */
-ch_error_t HS_CDECL ch_free_database(ch_database_t *db);
-
-/**
- * Utility function for identifying this release version.
- *
- * @return
- *      A string containing the version number of this release build and the
- *      date of the build. It is allocated statically, so it does not need to
- *      be freed by the caller.
- */
-const char * HS_CDECL ch_version(void);
-
-/**
- * Returns the size of the given database.
- *
- * @param database
- *      Pointer to compiled expression database.
- *
- * @param database_size
- *      On success, the size of the compiled database in bytes is placed in this
- *      parameter.
- *
- * @return
- *      @ref CH_SUCCESS on success, other values on failure.
- */
-ch_error_t HS_CDECL ch_database_size(const ch_database_t *database,
-                                     size_t *database_size);
-
-/**
- * Utility function providing information about a database.
- *
- * @param database
- *      Pointer to a compiled database.
- *
- * @param info
- *      On success, a string containing the version and platform information for
- *      the supplied database is placed in the parameter. The string is
- *      allocated using the allocator supplied in @ref hs_set_allocator()
- *      (or malloc() if no allocator was set) and should be freed by the caller.
- *
- * @return
- *      @ref CH_SUCCESS on success, other values on failure.
- */
-ch_error_t HS_CDECL ch_database_info(const ch_database_t *database,
-                                     char **info);
-
-/**
- * The type of the callback function that will be used by Chimera to allocate
- * more memory at runtime as required.
- *
- * If Chimera is to be used in a multi-threaded, or similarly concurrent
- * environment, the allocation function will need to be re-entrant, or
- * similarly safe for concurrent use.
- *
- * @param size
- *      The number of bytes to allocate.
- * @return
- *      A pointer to the region of memory allocated, or NULL on error.
- */
-typedef void *(HS_CDECL *ch_alloc_t)(size_t size);
-
-/**
- * The type of the callback function that will be used by Chimera to free
- * memory regions previously allocated using the @ref ch_alloc_t function.
- *
- * @param ptr
- *      The region of memory to be freed.
- */
-typedef void (HS_CDECL *ch_free_t)(void *ptr);
-
-/**
- * Set the allocate and free functions used by Chimera for allocating
- * memory at runtime for stream state, scratch space, database bytecode,
- * and various other data structure returned by the Chimera API.
- *
- * The function is equivalent to calling @ref ch_set_scratch_allocator(),
- * @ref ch_set_database_allocator() and
- * @ref ch_set_misc_allocator() with the provided parameters.
- *
- * This call will override any previous allocators that have been set.
- *
- * Note: there is no way to change the allocator used for temporary objects
- * created during the various compile calls (@ref ch_compile() and @ref
- * ch_compile_multi()).
- *
- * @param alloc_func
- *      A callback function pointer that allocates memory. This function must
- *      return memory suitably aligned for the largest representable data type
- *      on this platform.
- *
- * @param free_func
- *      A callback function pointer that frees allocated memory.
- *
- * @return
- *      @ref CH_SUCCESS on success, other values on failure.
- */
-ch_error_t HS_CDECL ch_set_allocator(ch_alloc_t alloc_func,
-                                     ch_free_t free_func);
-
-/**
- * Set the allocate and free functions used by Chimera for allocating memory
- * for database bytecode produced by the compile calls (@ref ch_compile() and @ref
- * ch_compile_multi()).
- *
- * If no database allocation functions are set, or if NULL is used in place of
- * both parameters, then memory allocation will default to standard methods
- * (such as the system malloc() and free() calls).
- *
- * This call will override any previous database allocators that have been set.
- *
- * Note: the database allocator may also be set by calling @ref
- * ch_set_allocator().
- *
- * Note: there is no way to change how temporary objects created during the
- * various compile calls (@ref ch_compile() and @ref ch_compile_multi()) are
- * allocated.
- *
- * @param alloc_func
- *      A callback function pointer that allocates memory. This function must
- *      return memory suitably aligned for the largest representable data type
- *      on this platform.
- *
- * @param free_func
- *      A callback function pointer that frees allocated memory.
- *
- * @return
- *      @ref HS_SUCCESS on success, other values on failure.
- */
-ch_error_t HS_CDECL ch_set_database_allocator(ch_alloc_t alloc_func,
-                                              ch_free_t free_func);
-
-/**
- * Set the allocate and free functions used by Chimera for allocating memory
- * for items returned by the Chimera API such as @ref ch_compile_error_t.
- *
- * If no misc allocation functions are set, or if NULL is used in place of both
- * parameters, then memory allocation will default to standard methods (such as
- * the system malloc() and free() calls).
- *
- * This call will override any previous misc allocators that have been set.
- *
- * Note: the misc allocator may also be set by calling @ref ch_set_allocator().
- *
- * @param alloc_func
- *      A callback function pointer that allocates memory. This function must
- *      return memory suitably aligned for the largest representable data type
- *      on this platform.
- *
- * @param free_func
- *      A callback function pointer that frees allocated memory.
- *
- * @return
- *      @ref CH_SUCCESS on success, other values on failure.
- */
-ch_error_t HS_CDECL ch_set_misc_allocator(ch_alloc_t alloc_func,
-                                          ch_free_t free_func);
-
-/**
- * Set the allocate and free functions used by Chimera for allocating memory
- * for scratch space by @ref ch_alloc_scratch() and @ref ch_clone_scratch().
- *
- * If no scratch allocation functions are set, or if NULL is used in place of
- * both parameters, then memory allocation will default to standard methods
- * (such as the system malloc() and free() calls).
- *
- * This call will override any previous scratch allocators that have been set.
- *
- * Note: the scratch allocator may also be set by calling @ref
- * ch_set_allocator().
- *
- * @param alloc_func
- *      A callback function pointer that allocates memory. This function must
- *      return memory suitably aligned for the largest representable data type
- *      on this platform.
- *
- * @param free_func
- *      A callback function pointer that frees allocated memory.
- *
- * @return
- *      @ref CH_SUCCESS on success, other values on failure.
- */
-ch_error_t HS_CDECL ch_set_scratch_allocator(ch_alloc_t alloc_func,
-                                             ch_free_t free_func);
-
-/**
- * @defgroup CH_ERROR ch_error_t values
- *
- * @{
- */
-
-/**
- * The engine completed normally.
- */
-#define CH_SUCCESS              0
-
-/**
- * A parameter passed to this function was invalid.
- */
-#define CH_INVALID              (-1)
-
-/**
- * A memory allocation failed.
- */
-#define CH_NOMEM                (-2)
-
-/**
- * The engine was terminated by callback.
- *
- * This return value indicates that the target buffer was partially scanned,
- * but that the callback function requested that scanning cease after a match
- * was located.
- */
-#define CH_SCAN_TERMINATED      (-3)
-
-/**
- * The pattern compiler failed, and the @ref ch_compile_error_t should be
- * inspected for more detail.
- */
-#define CH_COMPILER_ERROR       (-4)
-
-/**
- * The given database was built for a different version of the Chimera matcher.
- */
-#define CH_DB_VERSION_ERROR     (-5)
-
-/**
- * The given database was built for a different platform (i.e., CPU type).
- */
-#define CH_DB_PLATFORM_ERROR    (-6)
-
-/**
- * The given database was built for a different mode of operation.  This error
- * is returned when streaming calls are used with a non-streaming database and
- * vice versa.
- */
-#define CH_DB_MODE_ERROR        (-7)
-
-/**
- * A parameter passed to this function was not correctly aligned.
- */
-#define CH_BAD_ALIGN            (-8)
-
-/**
- * The memory allocator did not correctly return memory suitably aligned for
- * the largest representable data type on this platform.
- */
-#define CH_BAD_ALLOC            (-9)
-
-/**
- * The scratch region was already in use.
- *
- * This error is returned when Chimera is able to detect that the scratch
- * region given is already in use by another Chimera API call.
- *
- * A separate scratch region, allocated with @ref ch_alloc_scratch() or @ref
- * ch_clone_scratch(), is required for every concurrent caller of the Chimera
- * API.
- *
- * For example, this error might be returned when @ref ch_scan() has been
- * called inside a callback delivered by a currently-executing @ref ch_scan()
- * call using the same scratch region.
- *
- * Note: Not all concurrent uses of scratch regions may be detected. This error
- * is intended as a best-effort debugging tool, not a guarantee.
- */
-#define CH_SCRATCH_IN_USE       (-10)
-
-/**
- * Unexpected internal error from Hyperscan.
- *
- * This error indicates that there was unexpected matching behaviors from
- * Hyperscan. This could be related to invalid usage of scratch space or
- * invalid memory operations by users.
- *
- */
-#define CH_UNKNOWN_HS_ERROR     (-13)
-
-/**
- * Returned when pcre_exec (called for some expressions internally from @ref
- * ch_scan) failed due to a fatal error.
- */
-#define CH_FAIL_INTERNAL        (-32)
-
-/** @} */
-
-#ifdef __cplusplus
-} /* extern "C" */
-#endif
-
-#endif /* CH_COMMON_H_ */

chimera/ch_compile.cpp

@@ -1,877 +0,0 @@
-/*
- * Copyright (c) 2018, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Compiler front-end, including public API calls for compilation.
- */
-
-#include "ch_compile.h"
-#include "ch_alloc.h"
-#include "ch_internal.h"
-#include "ch_database.h"
-#include "grey.h"
-#include "hs_common.h"
-#include "hs_internal.h"
-#include "ue2common.h"
-#include "util/compile_error.h"
-#include "util/multibit_build.h"
-#include "util/target_info.h"
-
-#include <algorithm>
-#include <cassert>
-#include <cstddef>
-#include <cstring>
-#include <memory>
-#include <ostream>
-#include <sstream>
-#include <limits.h>
-#include <string>
-#include <vector>
-
-#include <boost/core/noncopyable.hpp>
-
-#define PCRE_ERROR_MSG "Internal error building PCRE pattern."
-
-using namespace std;
-using namespace ue2;
-
-static const char failureNoMemory[] = "Unable to allocate memory.";
-static const char failureInternal[] = "Internal error.";
-static const char failureBadAlloc[] = "Allocator returned misaligned memory.";
-
-static const ch_compile_error_t ch_enomem
-    = { const_cast<char *>(failureNoMemory), 0 };
-static const ch_compile_error_t ch_einternal
-    = { const_cast<char *>(failureInternal), 0 };
-static const ch_compile_error_t ch_badalloc
-    = { const_cast<char *>(failureBadAlloc), 0 };
-
-static
-ch_compile_error_t *generateChimeraCompileError(const string &err,
-                                                int expression) {
-    ch_compile_error_t *ret =
-        (struct ch_compile_error *)ch_misc_alloc(sizeof(ch_compile_error_t));
-    if (ret) {
-        ch_error_t e = ch_check_alloc(ret);
-        if (e != CH_SUCCESS) {
-            ch_misc_free(ret);
-            return const_cast<ch_compile_error_t *>(&ch_badalloc);
-        }
-        char *msg = (char *)ch_misc_alloc(err.size() + 1);
-        if (msg) {
-            e = ch_check_alloc(msg);
-            if (e != HS_SUCCESS) {
-                ch_misc_free(msg);
-                return const_cast<ch_compile_error_t *>(&ch_badalloc);
-            }
-            memcpy(msg, err.c_str(), err.size() + 1);
-            ret->message = msg;
-        } else {
-            ch_misc_free(ret);
-            ret = nullptr;
-        }
-    }
-
-    if (!ret || !ret->message) {
-        return const_cast<ch_compile_error_t *>(&ch_enomem);
-    }
-
-    ret->expression = expression;
-
-    return ret;
-}
-
-static
-void freeChimeraCompileError(ch_compile_error_t *error) {
-    if (!error) {
-        return;
-    }
-    if (error == &ch_enomem || error == &ch_einternal ||
-        error == &ch_badalloc) {
-        // These are not allocated.
-        return;
-    }
-
-    ch_misc_free(error->message);
-    ch_misc_free(error);
-}
-
-static
-bool checkMode(unsigned int mode, ch_compile_error_t **comp_error) {
-    static const unsigned int supported = CH_MODE_GROUPS;
-
-    if (mode & ~supported) {
-        *comp_error =
-            generateChimeraCompileError("Invalid mode flag supplied.", -1);
-        return false;
-    }
-    return true;
-}
-
-/** \brief Throw a compile error if we're passed some unsupported flags. */
-static
-void checkFlags(const unsigned int flags) {
-    static const unsigned int supported = HS_FLAG_DOTALL
-                                        | HS_FLAG_MULTILINE
-                                        | HS_FLAG_CASELESS
-                                        | HS_FLAG_SINGLEMATCH
-                                        | HS_FLAG_UCP
-                                        | HS_FLAG_UTF8;
-
-    if (flags & ~supported) {
-        throw CompileError("Unrecognized flag used.");
-    }
-}
-
-static
-bool isHyperscanSupported(const char *expression, unsigned int flags,
-                          const hs_platform_info *platform) {
-    hs_database_t *db = nullptr;
-    hs_compile_error *comp_error = nullptr;
-
-    unsigned int id = 0;
-    hs_error_t err = hs_compile_multi(&expression, &flags, &id,
-                                      1, HS_MODE_BLOCK, platform, &db,
-                                      &comp_error);
-    if (err != HS_SUCCESS) {
-        assert(!db);
-        assert(comp_error);
-        DEBUG_PRINTF("unsupported: %s\n", comp_error->message);
-        hs_free_compile_error(comp_error);
-
-        return false;
-    }
-
-    assert(db);
-    assert(!comp_error);
-    hs_free_database(db);
-    return true;
-}
-
-static
-bool writeHyperscanDatabase(char *ptr, hs_database_t *db) {
-    // Note: we must use our serialization calls to re-home the database.
-    char *serialized = nullptr;
-    size_t slen = 0;
-    hs_error_t err = hs_serialize_database(db, &serialized, &slen);
-    if (err != HS_SUCCESS) {
-        DEBUG_PRINTF("hs_serialize_database returned %d\n", err);
-        assert(0);
-        return false;
-    }
-
-    DEBUG_PRINTF("writing database to ptr %p\n", ptr);
-
-    // deserialize_at without the platform tests.
-    err = hs_deserialize_database_at(serialized, slen, (hs_database_t *)ptr);
-    if (err != HS_SUCCESS) {
-        DEBUG_PRINTF("hs_deserialize_database_at returned %d\n", err);
-        assert(0);
-        ch_misc_free(serialized);
-        return false;
-    }
-
-    ch_misc_free(serialized);
-    return true;
-}
-
-static
-bool writeHyperscanDatabase(ch_bytecode *db, hs_database_t *hs_db) {
-    db->databaseOffset = ROUNDUP_CL(sizeof(*db));
-    char *ptr = (char *)db + db->databaseOffset;
-    return writeHyperscanDatabase(ptr, hs_db);
-}
-
-static
-int convertFlagsToPcreOptions(unsigned int flags) {
-    int options = 0;
-    if (flags & HS_FLAG_CASELESS) {
-        options |= PCRE_CASELESS;
-    }
-    if (flags & HS_FLAG_DOTALL) {
-        options |= PCRE_DOTALL;
-    }
-    if (flags & HS_FLAG_MULTILINE) {
-        options |= PCRE_MULTILINE;
-    }
-    if (flags & HS_FLAG_UTF8) {
-        options |= PCRE_UTF8;
-    }
-    if (flags & HS_FLAG_UCP) {
-        options |= PCRE_UCP;
-    }
-
-    // All other flags are meaningless to PCRE.
-
-    return options;
-}
-
-namespace {
-
-/** \brief Data about a single pattern. */
-struct PatternData : boost::noncopyable {
-    PatternData(const char *pattern, u32 flags, u32 idx, u32 id_in,
-                unsigned mode, unsigned long int match_limit,
-                unsigned long int match_limit_recursion,
-                const hs_platform_info *platform);
-    ~PatternData() {
-        pcre_free(compiled);
-        pcre_free(extra);
-    }
-
-    void buildPcre(const char *pattern, u32 flags);
-
-    size_t patternSize() const;
-
-    void writePattern(ch_pattern *pattern) const;
-
-    pcre *compiled;                     //!< pcre_compile output
-    pcre_extra *extra;                  //!< pcre_study output
-    size_t compiled_size;
-    int study_size;
-    int capture_cnt;
-    bool utf8;
-    u32 id;                             //!< ID from the user
-    u32 expr_index;                     //!< index in the expression array
-    bool singlematch;                   //!< pattern is in highlander mode
-    bool guard;       //!< this pattern should be guarded by the multimatcher
-    u32 minWidth;     //!< min match width
-    u32 maxWidth;     //!< max match width
-    u32 fixedWidth;   //!< fixed pattern width
-    unsigned long int matchLimit; //! pcre match limit
-    unsigned long int matchLimitRecursion; //! pcre match_limit_recursion
-};
-
-PatternData::PatternData(const char *pattern, u32 flags, u32 idx, u32 id_in,
-                         unsigned mode, unsigned long int match_limit,
-                         unsigned long int match_limit_recursion,
-                         const hs_platform_info *platform)
-    : compiled(nullptr), extra(nullptr), id(id_in), expr_index(idx),
-      singlematch(flags & HS_FLAG_SINGLEMATCH),
-      guard(false), minWidth(0), maxWidth(UINT_MAX),
-      fixedWidth(UINT_MAX), matchLimit(match_limit),
-      matchLimitRecursion(match_limit_recursion) {
-    assert(pattern);
-
-    flags |= HS_FLAG_ALLOWEMPTY; /* don't hand things off to pcre for no
-                                    reason */
-
-    buildPcre(pattern, flags);
-
-    // Fetch the expression info for a prefiltering, non-singlematch version of
-    // this pattern, if possible.
-    hs_expr_info *info = nullptr;
-    hs_compile_error_t *error = nullptr;
-    u32 infoflags = (flags | HS_FLAG_PREFILTER) & ~HS_FLAG_SINGLEMATCH;
-    u32 rawflags = (flags | HS_FLAG_SOM_LEFTMOST) & ~HS_FLAG_SINGLEMATCH;
-    hs_error_t err = hs_expression_info(pattern, infoflags, &info, &error);
-    if (err == HS_SUCCESS) {
-        assert(info);
-        hs_expr_info *i = (hs_expr_info *)info;
-        minWidth = i->min_width;
-        maxWidth = i->max_width;
-        bool ordered = i->unordered_matches ? false : true;
-
-        // Only enable capturing if required
-        u32 captureCnt = 0;
-        if (mode & CH_MODE_GROUPS) {
-            captureCnt = capture_cnt;
-        }
-
-        // No need to confirm with PCRE if:
-        // 1) pattern is fixed width
-        // 2) pattern isn't vacuous as it can't combine with start of match
-        // 3) no capturing in this pattern
-        // 4) no offset adjust in this pattern as hyperscan match callback
-        //    will arrive without order, i.e. [^a]\z has offset adjust
-        // 5) hyperscan compile succeeds without prefiltering
-        if (minWidth == maxWidth && minWidth && maxWidth != UINT_MAX &&
-            !captureCnt && ordered &&
-            isHyperscanSupported(pattern, rawflags, platform)) {
-            fixedWidth = maxWidth;
-        }
-
-        DEBUG_PRINTF("gathered info: widths=[%u,%u]\n", minWidth, maxWidth);
-
-        ch_misc_free(info);
-
-        u32 guardflags;
-        guardflags = flags | HS_FLAG_PREFILTER;
-        guard = isHyperscanSupported(pattern, guardflags, platform);
-    } else {
-        // We can't even prefilter this pattern, so we're dependent on Big Dumb
-        // Pcre Scans.
-        DEBUG_PRINTF("hs_expression_info failed, falling back to pcre\n");
-        hs_free_compile_error(error);
-    }
-}
-
-void PatternData::buildPcre(const char *pattern, u32 flags) {
-    int options = convertFlagsToPcreOptions(flags);
-    const char *errptr = nullptr;
-    int erroffset = 0;
-
-    compiled = pcre_compile(pattern, options, &errptr, &erroffset, nullptr);
-    if (!compiled) {
-        DEBUG_PRINTF("PCRE failed to compile: %s\n", pattern);
-        string err("PCRE compilation failed: ");
-        err += string(errptr);
-        err += ".";
-        throw CompileError(expr_index, err);
-    }
-
-    extra = pcre_study(compiled, PCRE_STUDY_JIT_COMPILE, &errptr);
-    // Note that it's OK for pcre_study to return NULL if there's nothing
-    // to be found, but a non-NULL error is always bad.
-    if (errptr) {
-        DEBUG_PRINTF("PCRE could not be studied: %s\n", errptr);
-        string err("PCRE compilation failed: ");
-        err += string(errptr);
-        err += ".";
-        throw CompileError(expr_index, err);
-    }
-
-    if (pcre_fullinfo(compiled, extra, PCRE_INFO_SIZE, &compiled_size)) {
-        throw CompileError(PCRE_ERROR_MSG);
-    }
-
-    if (!extra) {
-        study_size = 0;
-    } else {
-        if (pcre_fullinfo(compiled, extra, PCRE_INFO_STUDYSIZE, &study_size)) {
-            throw CompileError(PCRE_ERROR_MSG);
-        }
-    }
-
-    if (pcre_fullinfo(compiled, extra, PCRE_INFO_CAPTURECOUNT, &capture_cnt)) {
-        throw CompileError(PCRE_ERROR_MSG);
-    }
-
-    /* We use the pcre rather than hs to get this information as we may need it
-     * even in the pure unguarded pcre mode where there is no hs available. We
-     * can not use the compile flags due to (*UTF8) verb */
-    unsigned long int opts = 0; // PCRE_INFO_OPTIONS demands an unsigned long
-    if (pcre_fullinfo(compiled, extra, PCRE_INFO_OPTIONS, &opts)) {
-        throw CompileError(PCRE_ERROR_MSG);
-    }
-    utf8 = opts & PCRE_UTF8;
-}
-
-size_t PatternData::patternSize() const {
-    size_t len = 0;
-
-    // ch_pattern header.
-    len += sizeof(ch_pattern);
-
-    len = ROUNDUP_N(len, 8);
-    DEBUG_PRINTF("compiled pcre at %zu\n", len);
-    len += compiled_size;
-
-    // PCRE study data, which may be zero.
-    if (study_size) {
-        len = ROUNDUP_N(len, 8);
-        DEBUG_PRINTF("study at %zu\n", len);
-        len += (size_t)study_size;
-    }
-
-    DEBUG_PRINTF("pattern size %zu\n", len);
-    return len;
-}
-
-/** \brief Write out an ch_pattern structure, which should already be sized
- * correctly according to PatternData::patternSize. */
-void PatternData::writePattern(ch_pattern *pattern) const {
-    assert(pattern);
-    assert(ISALIGNED_CL(pattern));
-
-    pattern->id = id;
-
-    u32 flags = 0;
-    if (singlematch) {
-        flags |= CHIMERA_PATTERN_FLAG_SINGLEMATCH;
-    }
-    if (utf8) {
-        flags |= CHIMERA_PATTERN_FLAG_UTF8;
-    }
-
-    pattern->flags = flags;
-    pattern->maxWidth = maxWidth;
-    pattern->minWidth = minWidth == UINT_MAX ? 0 : minWidth;
-    pattern->fixedWidth = fixedWidth;
-
-    // Compiled PCRE pattern.
-    char *ptr = (char *)pattern;
-    ptr += ROUNDUP_N(sizeof(*pattern), 8);
-    DEBUG_PRINTF("compiled pcre at %zu\n", (size_t)(ptr - (char *)pattern));
-    memcpy(ptr, compiled, compiled_size);
-    ptr += compiled_size;
-
-    // PCRE match limits
-    pattern->extra.flags = PCRE_EXTRA_MATCH_LIMIT |
-                           PCRE_EXTRA_MATCH_LIMIT_RECURSION;
-    pattern->extra.match_limit = matchLimit ? matchLimit : 10000000;
-    // Set to avoid segment fault
-    pattern->extra.match_limit_recursion =
-        matchLimitRecursion ? matchLimitRecursion : 1500;
-
-    // PCRE study_data.
-    u32 studyOffset = 0;
-    if (extra) {
-        assert(extra->study_data);
-        ptr = ROUNDUP_PTR(ptr, 8);
-        DEBUG_PRINTF("study at %zu\n", (size_t)(ptr - (char *)pattern));
-        memcpy(ptr, extra->study_data, study_size);
-        studyOffset = (size_t)(ptr - (char *)pattern);
-
-        pattern->extra.flags |= PCRE_EXTRA_STUDY_DATA;
-        pattern->extra.study_data = ptr;
-
-        ptr += study_size;
-    } else {
-        pattern->extra.flags &= ~PCRE_EXTRA_STUDY_DATA;
-    }
-    pattern->studyOffset = studyOffset;
-
-    size_t pcreLen = (ptr - (char *)pattern);
-    assert(pcreLen <= patternSize());
-    pattern->length = (u32)pcreLen;
-
-    // We shouldn't overrun the space we've allocated for this pattern.
-    assert(patternSize() >= (size_t)(ptr - (char *)pattern));
-}
-
-} // namespace
-
-namespace ch {
-
-static
-void ch_compile_multi_int(const char *const *expressions, const unsigned *flags,
-                          const unsigned *ids, unsigned elements,
-                          unsigned mode, unsigned long int match_limit,
-                          unsigned long int match_limit_recursion,
-                          const hs_platform_info_t *platform,
-                          ch_database_t **out) {
-    vector<unique_ptr<PatternData>> pcres;
-    pcres.reserve(elements);
-    vector<u32> unguarded; // indices of unguarded PCREs.
-    vector<const char *> multiExpr;
-    vector<unsigned int> multiFlags;
-    vector<unsigned int> multiIds;
-    bool allConfirm = true;
-    bool allSingleMatch = true;
-    for (unsigned int i = 0; i < elements; i++) {
-        const char *myExpr = expressions[i];
-        unsigned int myFlags = flags ? flags[i] : 0;
-        unsigned int myId = ids ? ids[i] : 0;
-
-        checkFlags(myFlags);
-
-        // First, build with libpcre. A build failure from libpcre will throw
-        // an exception up to the caller.
-        auto patternData =
-            std::make_unique<PatternData>(myExpr, myFlags, i, myId, mode, match_limit,
-                                          match_limit_recursion, platform);
-        pcres.push_back(move(patternData));
-        PatternData &curr = *pcres.back();
-
-        if (!(myFlags & HS_FLAG_SINGLEMATCH)) {
-            allSingleMatch = false;
-        }
-
-        // in the multimatch, we always run in prefilter mode and accept vacuous
-        // patterns.
-        myFlags |=
-            HS_FLAG_ALLOWEMPTY | HS_FLAG_PREFILTER;
-
-        if (curr.fixedWidth != UINT_MAX) {
-            myFlags |= HS_FLAG_SOM_LEFTMOST;
-            DEBUG_PRINTF("fixed width, turn off prefiltering\n");
-            myFlags &= ~HS_FLAG_PREFILTER;
-            allConfirm = false;
-
-            // Single match can't coexist with SOM.
-            myFlags &= ~HS_FLAG_SINGLEMATCH;
-        }
-
-        if (curr.guard) {
-            // We use the index into the PCREs array as the Hyperscan idx.
-            multiExpr.push_back(myExpr);
-            multiFlags.push_back(myFlags);
-            multiIds.push_back(i);
-        } else {
-            // No Hyperscan support, PCRE is unguarded.
-            unguarded.push_back(i);
-        }
-    }
-
-    DEBUG_PRINTF("built %zu PCREs, %zu of which are unguarded\n",
-                 pcres.size(), unguarded.size());
-
-    // Work out our sizing for the output database.
-    size_t patternSize = 0;
-    for (unsigned int i = 0; i < elements; i++) {
-        size_t len = pcres[i]->patternSize();
-        patternSize += ROUNDUP_CL(len);
-    }
-    DEBUG_PRINTF("pcre bytecode takes %zu bytes\n", patternSize);
-
-    bool noMulti = multiExpr.empty();
-    size_t multiSize = 0;
-    hs_database *multidb = nullptr;
-    if (!noMulti) {
-        hs_compile_error_t *hs_comp_error = nullptr;
-        hs_error_t err = hs_compile_multi(&multiExpr[0], &multiFlags[0],
-                                          &multiIds[0], multiExpr.size(),
-                                          HS_MODE_BLOCK, platform, &multidb,
-                                          &hs_comp_error);
-
-        if (err != HS_SUCCESS) {
-            assert(hs_comp_error);
-            DEBUG_PRINTF("hs_compile_multi returned error: %s\n",
-                         hs_comp_error->message);
-            assert(0);
-            hs_free_compile_error(hs_comp_error);
-            throw CompileError("Internal error.");
-        }
-
-        assert(multidb);
-        err = hs_database_size(multidb, &multiSize);
-        if (err != HS_SUCCESS) {
-            assert(0);
-            throw CompileError("Internal error.");
-        }
-        DEBUG_PRINTF("built hyperscan database with len %zu bytes\n", multiSize);
-    }
-
-    size_t bytecodeLen = sizeof(ch_bytecode) +
-                         multiSize + alignof(u32) +
-                         (sizeof(u32) * unguarded.size()) +
-                         (sizeof(u32) * elements) +
-                         patternSize +
-                         128; // padding for alignment
-    size_t totalSize = sizeof(ch_database) + bytecodeLen;
-
-    DEBUG_PRINTF("allocating %zu bytes for database\n", totalSize);
-    char *ptr = (char *)ch_database_alloc(totalSize);
-    if (ch_check_alloc(ptr) != CH_SUCCESS) {
-        ch_database_free(ptr);
-        throw std::bad_alloc();
-    }
-
-    memset(ptr, 0, totalSize);
-
-    // First, the header.
-    ch_database *hydb = (ch_database *)ptr;
-    hydb->magic = CH_DB_MAGIC;
-    hydb->version = HS_VERSION_32BIT;
-    hydb->length = bytecodeLen;
-
-    // Then, the bytecode.
-    size_t shift = (size_t)hydb->bytes & 0x3f;
-    hydb->bytecode = offsetof(struct ch_database, bytes) - shift;
-    ch_bytecode *db = (ch_bytecode *)((char *)hydb + hydb->bytecode);
-    db->patternCount = elements;
-    db->activeSize = mmbit_size(elements);
-    db->flags = 0;
-    db->length = bytecodeLen;
-
-    if (noMulti) {
-        db->flags |= CHIMERA_FLAG_NO_MULTIMATCH;
-    }
-    if (mode & CH_MODE_GROUPS) {
-        db->flags |= CHIMERA_FLAG_GROUPS;
-    }
-    if (allConfirm) {
-        db->flags |= CHIMERA_FLAG_ALL_CONFIRM;
-    }
-    if (allSingleMatch) {
-        db->flags |= CHIMERA_FLAG_ALL_SINGLE;
-    }
-
-
-    // Find and set the max ovector size by looking at the capture count for
-    // each pcre.
-    u32 maxCaptureGroups = 0;
-    for (unsigned int i = 0; i < elements; i++) {
-        maxCaptureGroups = max(maxCaptureGroups, (u32)pcres[i]->capture_cnt);
-    }
-    db->maxCaptureGroups = maxCaptureGroups;
-    DEBUG_PRINTF("max capture groups is %u\n", maxCaptureGroups);
-
-    if (!noMulti) {
-        DEBUG_PRINTF("write hyperscan database\n");
-        // Write Hyperscan database directly after the header struct, then free it.
-        if (!writeHyperscanDatabase(db, multidb)) {
-            ch_database_free(hydb);
-            hs_free_database(multidb);
-            throw CompileError("Internal error.");
-        }
-        hs_free_database(multidb);
-    } else {
-        db->databaseOffset = ROUNDUP_CL(sizeof(*db));
-    }
-
-    // Then, write our unguarded PCRE list.
-    db->unguardedCount = unguarded.size();
-    db->unguardedOffset = ROUNDUP_N(db->databaseOffset + multiSize, 4);
-    ptr = (char *)db + db->unguardedOffset;
-    copy(unguarded.begin(), unguarded.end(), (u32 *)ptr);
-
-    // Then, write all our compiled PCRE patterns and the lookup table for
-    // them.
-    db->patternOffset = db->unguardedOffset + unguarded.size() * sizeof(u32);
-    u32 *patternOffset = (u32 *)((char *)db + db->patternOffset);
-    u32 offset = ROUNDUP_CL(db->patternOffset + elements * sizeof(u32));
-    for (unsigned int i = 0; i < elements; i++) {
-        *patternOffset = offset;
-        size_t len = pcres[i]->patternSize();
-        ptr = (char *)db + offset;
-        struct ch_pattern *pattern = (struct ch_pattern *)ptr;
-        pcres[i]->writePattern(pattern);
-        DEBUG_PRINTF("wrote pcre %u into offset %u, len %zu\n", i, offset, len);
-        offset += ROUNDUP_CL(len);
-        patternOffset++;
-    }
-
-    assert(offset <= totalSize);
-    assert(hydb->magic == CH_DB_MAGIC);
-    DEBUG_PRINTF("built hybrid database, size %zu bytes\n", totalSize);
-    DEBUG_PRINTF("offset=%u\n", offset);
-    *out = hydb;
-}
-
-} // namespace ch
-
-extern "C" HS_PUBLIC_API
-ch_error_t HS_CDECL ch_compile(const char *expression, unsigned flags,
-                               unsigned mode,
-                               const hs_platform_info_t *platform,
-                               ch_database_t **db,
-                               ch_compile_error_t **comp_error) {
-    if (!comp_error) {
-        if (db) {
-            db = nullptr;
-        }
-        // nowhere to write the string, but we can still report an error code
-        return CH_COMPILER_ERROR;
-    }
-    if (!db) {
-        *comp_error =
-            generateChimeraCompileError("Invalid parameter: db is NULL", -1);
-        return CH_COMPILER_ERROR;
-    }
-    if (!expression) {
-        *db = nullptr;
-        *comp_error =
-            generateChimeraCompileError("Invalid parameter: expressions is\
-                                         NULL", -1);
-        return CH_COMPILER_ERROR;
-    }
-
-    if (!checkMode(mode, comp_error)) {
-        *db = nullptr;
-        assert(*comp_error); // set by checkMode
-        return CH_COMPILER_ERROR;
-    }
-
-    try {
-        unsigned id = 0; // single expressions get zero as an ID
-        // Internal function to do all the work, now that we've handled all the
-        // argument checking.
-        ch::ch_compile_multi_int(&expression, &flags, &id, 1, mode, 0, 0,
-                                 platform, db);
-    }
-    catch (const CompileError &e) {
-        // Compiler error occurred
-        *db = nullptr;
-        *comp_error = generateChimeraCompileError(e.reason, e.hasIndex ?
-                                                  (int)e.index : -1);
-        return CH_COMPILER_ERROR;
-    }
-    catch (std::bad_alloc &) {
-        *db = nullptr;
-        *comp_error = const_cast<ch_compile_error_t *>(&ch_enomem);
-        return CH_COMPILER_ERROR;
-    }
-    catch (...) {
-        assert(!"Internal error, unexpected exception");
-        *db = nullptr;
-        *comp_error = const_cast<ch_compile_error_t *>(&ch_einternal);
-        return CH_COMPILER_ERROR;
-    }
-
-    DEBUG_PRINTF("success!\n");
-    return CH_SUCCESS;
-}
-
-extern "C" HS_PUBLIC_API
-ch_error_t HS_CDECL ch_compile_multi(const char *const *expressions,
-                                     const unsigned *flags, const unsigned *ids,
-                                     unsigned elements, unsigned mode,
-                                     const hs_platform_info_t *platform,
-                                     ch_database_t **db,
-                                     ch_compile_error_t **comp_error) {
-    if (!comp_error) {
-        if (db) {
-            db = nullptr;
-        }
-        // nowhere to write the string, but we can still report an error code
-        return CH_COMPILER_ERROR;
-    }
-    if (!db) {
-        *comp_error =
-            generateChimeraCompileError("Invalid parameter: db is NULL", -1);
-        return CH_COMPILER_ERROR;
-    }
-    if (!expressions) {
-        *db = nullptr;
-        *comp_error =
-            generateChimeraCompileError("Invalid parameter: expressions is\
-                                         NULL", -1);
-        return CH_COMPILER_ERROR;
-    }
-    if (!elements) {
-        *db = nullptr;
-        *comp_error = generateChimeraCompileError("Invalid parameter:\
-                                                   elements is zero", -1);
-        return CH_COMPILER_ERROR;
-    }
-
-    if (!checkMode(mode, comp_error)) {
-        *db = nullptr;
-        assert(*comp_error); // set by checkMode
-        return CH_COMPILER_ERROR;
-    }
-
-    try {
-        // Internal function to do all the work, now that we've handled all the
-        // argument checking.
-        ch::ch_compile_multi_int(expressions, flags, ids, elements, mode, 0, 0,
-                                 platform, db);
-    }
-    catch (const CompileError &e) {
-        // Compiler error occurred
-        *db = nullptr;
-        *comp_error = generateChimeraCompileError(e.reason, e.hasIndex ?
-                                                  (int)e.index : -1);
-        return CH_COMPILER_ERROR;
-    }
-    catch (std::bad_alloc &) {
-        *db = nullptr;
-        *comp_error = const_cast<ch_compile_error_t *>(&ch_enomem);
-        return CH_COMPILER_ERROR;
-    }
-    catch (...) {
-        assert(!"Internal error, unexpected exception");
-        *db = nullptr;
-        *comp_error = const_cast<ch_compile_error_t *>(&ch_einternal);
-        return CH_COMPILER_ERROR;
-    }
-
-    DEBUG_PRINTF("success!\n");
-    return CH_SUCCESS;
-}
-
-extern "C" HS_PUBLIC_API
-ch_error_t HS_CDECL ch_compile_ext_multi(
-                                    const char *const *expressions,
-                                    const unsigned *flags,
-                                    const unsigned *ids,
-                                    unsigned elements, unsigned mode,
-                                    unsigned long int match_limit,
-                                    unsigned long int match_limit_recursion,
-                                    const hs_platform_info_t *platform,
-                                    ch_database_t **db,
-                                    ch_compile_error_t **comp_error) {
-    if (!comp_error) {
-        if (db) {
-            db = nullptr;
-        }
-        // nowhere to write the string, but we can still report an error code
-        return CH_COMPILER_ERROR;
-    }
-    if (!db) {
-        *comp_error =
-            generateChimeraCompileError("Invalid parameter: db is NULL", -1);
-        return CH_COMPILER_ERROR;
-    }
-    if (!expressions) {
-        *db = nullptr;
-        *comp_error =
-            generateChimeraCompileError("Invalid parameter: expressions is\
-                                         NULL", -1);
-        return CH_COMPILER_ERROR;
-    }
-    if (!elements) {
-        *db = nullptr;
-        *comp_error = generateChimeraCompileError("Invalid parameter:\
-                                                   elements is zero", -1);
-        return CH_COMPILER_ERROR;
-    }
-
-    if (!checkMode(mode, comp_error)) {
-        *db = nullptr;
-        assert(*comp_error); // set by checkMode
-        return CH_COMPILER_ERROR;
-    }
-
-    try {
-        // Internal function to do all the work, now that we've handled all the
-        // argument checking.
-        ch::ch_compile_multi_int(expressions, flags, ids, elements, mode,
-                                 match_limit, match_limit_recursion, platform,
-                                 db);
-    }
-    catch (const CompileError &e) {
-        // Compiler error occurred
-        *db = nullptr;
-        *comp_error = generateChimeraCompileError(e.reason, e.hasIndex ?
-                                                  (int)e.index : -1);
-        return CH_COMPILER_ERROR;
-    }
-    catch (std::bad_alloc &) {
-        *db = nullptr;
-        *comp_error = const_cast<ch_compile_error_t *>(&ch_enomem);
-        return CH_COMPILER_ERROR;
-    }
-    catch (...) {
-        assert(!"Internal error, unexpected exception");
-        *db = nullptr;
-        *comp_error = const_cast<ch_compile_error_t *>(&ch_einternal);
-        return CH_COMPILER_ERROR;
-    }
-
-    DEBUG_PRINTF("success!\n");
-    return CH_SUCCESS;
-}
-
-extern "C" HS_PUBLIC_API
-ch_error_t HS_CDECL ch_free_compile_error(ch_compile_error_t *error) {
-    freeChimeraCompileError(error);
-    return CH_SUCCESS;
-}

chimera/ch_compile.h

@@ -1,394 +0,0 @@
-/*
- * Copyright (c) 2018, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef CH_COMPILE_H_
-#define CH_COMPILE_H_
-
-/**
- * @file
- * @brief The Chimera compiler API definition.
- *
- * Chimera is a hybrid solution of Hyperscan and PCRE.
- *
- * This header contains functions for compiling regular expressions into
- * Chimera databases that can be used by the Chimera runtime.
- */
-
-#include "ch_common.h"
-#include "hs_compile.h"
-
-#ifdef __cplusplus
-extern "C"
-{
-#endif
-
-/**
- * A type containing error details that is returned by the compile calls (@ref
- * ch_compile() and @ref ch_compile_multi() on failure. The caller may inspect
- * the values returned in this type to determine the cause of failure.
- */
-typedef struct ch_compile_error {
-    /**
-     * A human-readable error message describing the error.
-     */
-    char *message;
-
-    /**
-     * The zero-based number of the expression that caused the error (if this
-     * can be determined). If the error is not specific to an expression, then
-     * this value will be less than zero.
-     */
-    int expression;
-} ch_compile_error_t;
-
-/**
- * The basic regular expression compiler.
- *
- * This is the function call with which an expression is compiled into a
- * Chimera database which can be passed to the runtime function (
- * @ref ch_scan())
- *
- * @param expression
- *      The NULL-terminated expression to parse. Note that this string must
- *      represent ONLY the pattern to be matched, with no delimiters or flags;
- *      any global flags should be specified with the @a flags argument. For
- *      example, the expression `/abc?def/i` should be compiled by providing
- *      `abc?def` as the @a expression, and @ref CH_FLAG_CASELESS as the @a
- *      flags.
- *
- * @param flags
- *      Flags which modify the behaviour of the expression. Multiple flags may
- *      be used by ORing them together. Valid values are:
- *       - CH_FLAG_CASELESS - Matching will be performed case-insensitively.
- *       - CH_FLAG_DOTALL - Matching a `.` will not exclude newlines.
- *       - CH_FLAG_MULTILINE - `^` and `$` anchors match any newlines in data.
- *       - CH_FLAG_SINGLEMATCH - Only one match will be generated for the
- *                               expression per stream.
- *       - CH_FLAG_UTF8 - Treat this pattern as a sequence of UTF-8 characters.
- *       - CH_FLAG_UCP - Use Unicode properties for character classes.
- *
- * @param mode
- *      Compiler mode flag that affect the database as a whole for capturing
- *      groups. One of  CH_MODE_NOGROUPS or  CH_MODE_GROUPS must be supplied.
- *      See @ref CH_MODE_FLAG for more details.
- *
- * @param platform
- *      If not NULL, the platform structure is used to determine the target
- *      platform for the database. If NULL, a database suitable for running
- *      on the current host platform is produced.
- *
- * @param db
- *      On success, a pointer to the generated database will be returned in
- *      this parameter, or NULL on failure. The caller is responsible for
- *      deallocating the buffer using the @ref ch_free_database() function.
- *
- * @param compile_error
- *      If the compile fails, a pointer to a @ref ch_compile_error_t will be
- *      returned, providing details of the error condition. The caller is
- *      responsible for deallocating the buffer using the @ref
- *      ch_free_compile_error() function.
- *
- * @return
- *      @ref CH_SUCCESS is returned on successful compilation; @ref
- *      CH_COMPILER_ERROR on failure, with details provided in the error
- *      parameter.
- */
-ch_error_t HS_CDECL ch_compile(const char *expression, unsigned int flags,
-                               unsigned int mode,
-                               const hs_platform_info_t *platform,
-                               ch_database_t **db,
-                               ch_compile_error_t **compile_error);
-
-/**
- * The multiple regular expression compiler.
- *
- * This is the function call with which a set of expressions is compiled into a
- * database which can be passed to the runtime function (@ref ch_scan()).
- * Each expression can be labelled with a unique integer which is passed into
- * the match callback to identify the pattern that has matched.
- *
- * @param expressions
- *      Array of NULL-terminated expressions to compile. Note that (as for @ref
- *      ch_compile()) these strings must contain only the pattern to be
- *      matched, with no delimiters or flags. For example, the expression
- *      `/abc?def/i` should be compiled by providing `abc?def` as the first
- *      string in the @a expressions array, and @ref CH_FLAG_CASELESS as the
- *      first value in the @a flags array.
- *
- * @param flags
- *      Array of flags which modify the behaviour of each expression. Multiple
- *      flags may be used by ORing them together.  Specifying the NULL pointer
- *      in place of an array will set the flags value for all patterns to zero.
- *      Valid values are:
- *       - CH_FLAG_CASELESS - Matching will be performed case-insensitively.
- *       - CH_FLAG_DOTALL - Matching a `.` will not exclude newlines.
- *       - CH_FLAG_MULTILINE - `^` and `$` anchors match any newlines in data.
- *       - CH_FLAG_SINGLEMATCH - Only one match will be generated by patterns
- *                               with this match id per stream.
- *       - CH_FLAG_UTF8 - Treat this pattern as a sequence of UTF-8 characters.
- *       - CH_FLAG_UCP - Use Unicode properties for character classes.
- *
- * @param ids
- *      An array of integers specifying the ID number to be associated with the
- *      corresponding pattern in the expressions array. Specifying the NULL
- *      pointer in place of an array will set the ID value for all patterns to
- *      zero.
- *
- * @param elements
- *      The number of elements in the input arrays.
- *
- * @param mode
- *      Compiler mode flag that affect the database as a whole for capturing
- *      groups. One of  CH_MODE_NOGROUPS or  CH_MODE_GROUPS must be supplied.
- *      See @ref CH_MODE_FLAG for more details.
- *
- * @param platform
- *      If not NULL, the platform structure is used to determine the target
- *      platform for the database. If NULL, a database suitable for running
- *      on the current host platform is produced.
- *
- * @param db
- *      On success, a pointer to the generated database will be returned in
- *      this parameter, or NULL on failure. The caller is responsible for
- *      deallocating the buffer using the @ref ch_free_database() function.
- *
- * @param compile_error
- *      If the compile fails, a pointer to a @ref ch_compile_error_t will be
- *      returned, providing details of the error condition. The caller is
- *      responsible for deallocating the buffer using the @ref
- *      ch_free_compile_error() function.
- *
- * @return
- *      @ref CH_SUCCESS is returned on successful compilation; @ref
- *      CH_COMPILER_ERROR on failure, with details provided in the @a error
- *      parameter.
- *
- */
-ch_error_t HS_CDECL ch_compile_multi(const char *const *expressions,
-                                     const unsigned int *flags,
-                                     const unsigned int *ids,
-                                     unsigned int elements, unsigned int mode,
-                                     const hs_platform_info_t *platform,
-                                     ch_database_t **db,
-                                     ch_compile_error_t **compile_error);
-
-/**
- * The multiple regular expression compiler with extended match limits support.
- *
- * This is the function call with which a set of expressions is compiled into a
- * database in the same way as @ref ch_compile_multi(), but allows additional
- * parameters to be specified via match_limit and match_limit_recursion to
- * define match limits for PCRE runtime.
- *
- * @param expressions
- *      Array of NULL-terminated expressions to compile. Note that (as for @ref
- *      ch_compile()) these strings must contain only the pattern to be
- *      matched, with no delimiters or flags. For example, the expression
- *      `/abc?def/i` should be compiled by providing `abc?def` as the first
- *      string in the @a expressions array, and @ref CH_FLAG_CASELESS as the
- *      first value in the @a flags array.
- *
- * @param flags
- *      Array of flags which modify the behaviour of each expression. Multiple
- *      flags may be used by ORing them together.  Specifying the NULL pointer
- *      in place of an array will set the flags value for all patterns to zero.
- *      Valid values are:
- *       - CH_FLAG_CASELESS - Matching will be performed case-insensitively.
- *       - CH_FLAG_DOTALL - Matching a `.` will not exclude newlines.
- *       - CH_FLAG_MULTILINE - `^` and `$` anchors match any newlines in data.
- *       - CH_FLAG_SINGLEMATCH - Only one match will be generated by patterns
- *                               with this match id per stream.
- *       - CH_FLAG_UTF8 - Treat this pattern as a sequence of UTF-8 characters.
- *       - CH_FLAG_UCP - Use Unicode properties for character classes.
- *
- * @param ids
- *      An array of integers specifying the ID number to be associated with the
- *      corresponding pattern in the expressions array. Specifying the NULL
- *      pointer in place of an array will set the ID value for all patterns to
- *      zero.
- *
- * @param elements
- *      The number of elements in the input arrays.
- *
- * @param mode
- *      Compiler mode flag that affect the database as a whole for capturing
- *      groups. One of  CH_MODE_NOGROUPS or  CH_MODE_GROUPS must be supplied.
- *      See @ref CH_MODE_FLAG for more details.
- *
- * @param match_limit
- *      A limit from pcre_extra on the amount of match function called in PCRE
- *      to limit backtracking that can take place.
- *
- * @param match_limit_recursion
- *      A limit from pcre_extra on the recursion depth of match function
- *      in PCRE.
- *
- * @param platform
- *      If not NULL, the platform structure is used to determine the target
- *      platform for the database. If NULL, a database suitable for running
- *      on the current host platform is produced.
- *
- * @param db
- *      On success, a pointer to the generated database will be returned in
- *      this parameter, or NULL on failure. The caller is responsible for
- *      deallocating the buffer using the @ref ch_free_database() function.
- *
- * @param compile_error
- *      If the compile fails, a pointer to a @ref ch_compile_error_t will be
- *      returned, providing details of the error condition. The caller is
- *      responsible for deallocating the buffer using the @ref
- *      ch_free_compile_error() function.
- *
- * @return
- *      @ref CH_SUCCESS is returned on successful compilation; @ref
- *      CH_COMPILER_ERROR on failure, with details provided in the @a error
- *      parameter.
- *
- */
-ch_error_t HS_CDECL ch_compile_ext_multi(const char *const *expressions,
-                                         const unsigned int *flags,
-                                         const unsigned int *ids,
-                                         unsigned int elements,
-                                         unsigned int mode,
-                                         unsigned long int match_limit,
-                                         unsigned long int match_limit_recursion,
-                                         const hs_platform_info_t *platform,
-                                         ch_database_t **db,
-                                         ch_compile_error_t **compile_error);
-
-/**
- * Free an error structure generated by @ref ch_compile(), @ref
- * ch_compile_multi().
- *
- * @param error
- *      The @ref ch_compile_error_t to be freed. NULL may also be safely
- *      provided.
- *
- * @return
- *      @ref CH_SUCCESS on success, other values on failure.
- */
-ch_error_t HS_CDECL ch_free_compile_error(ch_compile_error_t *error);
-
-/**
- * @defgroup CH_PATTERN_FLAG Pattern flags
- *
- * @{
- */
-
-/**
- * Compile flag: Set case-insensitive matching.
- *
- * This flag sets the expression to be matched case-insensitively by default.
- * The expression may still use PCRE tokens (notably `(?i)` and
- * `(?-i)`) to switch case-insensitive matching on and off.
- */
-#define CH_FLAG_CASELESS        1
-
-/**
- * Compile flag: Matching a `.` will not exclude newlines.
- *
- * This flag sets any instances of the `.` token to match newline characters as
- * well as all other characters. The PCRE specification states that the `.`
- * token does not match newline characters by default, so without this flag the
- * `.` token will not cross line boundaries.
- */
-#define CH_FLAG_DOTALL          2
-
-/**
- * Compile flag: Set multi-line anchoring.
- *
- * This flag instructs the expression to make the `^` and `$` tokens match
- * newline characters as well as the start and end of the stream. If this flag
- * is not specified, the `^` token will only ever match at the start of a
- * stream, and the `$` token will only ever match at the end of a stream within
- * the guidelines of the PCRE specification.
- */
-#define CH_FLAG_MULTILINE       4
-
-/**
- * Compile flag: Set single-match only mode.
- *
- * This flag sets the expression's match ID to match at most once, only the
- * first match for each invocation of @ref ch_scan() will be returned.
- *
- */
-#define CH_FLAG_SINGLEMATCH     8
-
-/**
- * Compile flag: Enable UTF-8 mode for this expression.
- *
- * This flag instructs Chimera to treat the pattern as a sequence of UTF-8
- * characters. The results of scanning invalid UTF-8 sequences with a Chimera
- * library that has been compiled with one or more patterns using this flag are
- * undefined.
- */
-#define CH_FLAG_UTF8            32
-
-/**
- * Compile flag: Enable Unicode property support for this expression.
- *
- * This flag instructs Chimera to use Unicode properties, rather than the
- * default ASCII interpretations, for character mnemonics like `\w` and `\s` as
- * well as the POSIX character classes. It is only meaningful in conjunction
- * with @ref CH_FLAG_UTF8.
- */
-#define CH_FLAG_UCP             64
-
-/** @} */
-
-/**
- * @defgroup CH_MODE_FLAG Compile mode flags
- *
- * The mode flags are used as values for the mode parameter of the various
- * compile calls (@ref ch_compile(), @ref ch_compile_multi().
- *
- * By default, the matcher will only supply the start and end offsets of the
- * match when the match callback is called. Using mode flag @ref CH_MODE_GROUPS
- * will also fill the `captured' array with the start and end offsets of all
- * the capturing groups specified by the pattern that has matched.
- *
- * @{
- */
-
-/**
- * Compiler mode flag: Disable capturing groups.
- */
-#define CH_MODE_NOGROUPS        0
-
-/**
- * Compiler mode flag: Enable capturing groups.
- */
-#define CH_MODE_GROUPS          1048576
-
-/** @} */
-
-#ifdef __cplusplus
-} /* extern "C" */
-#endif
-
-#endif /* CH_COMPILE_H_ */

chimera/ch_database.c

@@ -1,126 +0,0 @@
-/*
- * Copyright (c) 2018, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Chimera: database construction, etc.
- */
-
-#include <limits.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-#include "allocator.h"
-#include "database.h"
-#include "hs.h"
-#include "ch.h"
-#include "hs_internal.h"
-#include "ch_common.h"
-#include "ch_alloc.h"
-#include "ch_database.h"
-#include "ch_internal.h"
-
-static really_inline
-int db_correctly_aligned(const void *db) {
-    return ISALIGNED_N(db, alignof(unsigned long long));
-}
-
-HS_PUBLIC_API
-ch_error_t HS_CDECL ch_free_database(ch_database_t *hydb) {
-    if (hydb && hydb->magic != CH_DB_MAGIC) {
-        return CH_INVALID;
-    }
-    ch_database_free(hydb);
-
-    return CH_SUCCESS;
-}
-
-HS_PUBLIC_API
-ch_error_t HS_CDECL ch_database_size(const ch_database_t *hydb, size_t *size) {
-    if (!size) {
-        return CH_INVALID;
-    }
-
-    ch_error_t ret = hydbIsValid(hydb);
-    if (unlikely(ret != CH_SUCCESS)) {
-        return ret;
-    }
-
-    *size = sizeof(struct ch_database) + hydb->length;
-    return CH_SUCCESS;
-}
-
-/** \brief Identifier prepended to database info. */
-static const char CHIMERA_IDENT[] = "Chimera ";
-
-HS_PUBLIC_API
-ch_error_t HS_CDECL ch_database_info(const ch_database_t *hydb, char **info) {
-    if (!info) {
-        return CH_INVALID;
-    }
-    *info = NULL;
-
-    if (!hydb || !db_correctly_aligned(hydb) || hydb->magic != CH_DB_MAGIC) {
-        return HS_INVALID;
-    }
-
-    const struct ch_bytecode *bytecode = ch_get_bytecode(hydb);
-    char noMulti = (bytecode->flags & CHIMERA_FLAG_NO_MULTIMATCH);
-    if (noMulti) {
-        size_t len = strlen(CHIMERA_IDENT);
-        *info = ch_misc_alloc(len + 1);
-        if (!(*info)) {
-            return CH_INVALID;
-        }
-        memcpy((*info), CHIMERA_IDENT, len);
-        (*info)[len] = '\0';
-        return CH_SUCCESS;
-    }
-
-    char *hsinfo = NULL;
-    hs_error_t ret = hs_database_info(getHyperscanDatabase(bytecode), &hsinfo);
-    if (ret != HS_SUCCESS) {
-        assert(!hsinfo);
-        return ret;
-    }
-
-    size_t hybridlen = strlen(CHIMERA_IDENT);
-    size_t hslen = strlen(hsinfo);
-    *info = ch_misc_alloc(hybridlen + hslen + 1);
-    if (!(*info)) {
-        ch_misc_free(hsinfo);
-        return CH_INVALID;
-    }
-
-    memcpy((*info), CHIMERA_IDENT, hybridlen);
-    memcpy((*info) + hybridlen, hsinfo, hslen);
-    (*info)[hybridlen + hslen] = '\0';
-    ch_misc_free(hsinfo);
-
-    return CH_SUCCESS;
-}

chimera/ch_database.h

@@ -1,158 +0,0 @@
-/*
- * Copyright (c) 2018, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Runtime code for ch_database manipulation.
- */
-
-#ifndef CH_DATABASE_H_
-#define CH_DATABASE_H_
-
-#ifdef __cplusplus
-extern "C"
-{
-#endif
-
-#define PCRE_STATIC
-#include <pcre.h>
-
-#include "ch_compile.h" // for CH_MODE_ flags
-#include "ue2common.h"
-#include "hs_version.h"
-#include "hs.h"
-
-#define CH_DB_MAGIC 0xdedededeU //!< Magic number stored in \ref ch_database
-
-/** \brief Main Chimera database header. */
-struct ch_database {
-    u32 magic;     //!< must be \ref CH_DB_MAGIC
-    u32 version;   //!< release version
-    u32 length;    //!< total allocated length in bytes
-    u32 reserved0; //!< unused
-    u32 reserved1; //!< unused
-    u32 bytecode;  //!< offset relative to db start
-    u32 padding[16];  //!< padding for alignment of rest of bytecode
-    char bytes[];
-};
-
-/** \brief Chimera bytecode header, which follows the \ref ch_database and is
- * always 64-byte aligned. */
-struct ch_bytecode {
-    u32 length; //!< length of bytecode including this header struct
-    u32 flags;          //!< whole-database flags (CHIMERA_FLAG_NO_MULTIMATCH,
-                        // CHIMERA_FLAG_GROUPS)
-    u32 patternCount;   //!< total number of patterns
-    u32 activeSize;     //!< size of mmbit to store active pattern ids
-    u32 databaseOffset; //!< offset for database following \ref ch_bytecode
-                        // header
-    u32 patternOffset;  //!< points to an array of u32 offsets, each pointing to
-                        // a \ref ch_pattern
-    u32 unguardedOffset;  //!< pointer to a list of unguarded pattern indices
-    u32 unguardedCount;   //!< number of unguarded patterns
-    u32 maxCaptureGroups; //!< max number of capture groups used by any pattern
-};
-
-/** \brief Per-pattern header.
- *
- * struct is followed in bytecode by:
- * 1. pcre bytecode (always present)
- * 2. pcre study data (sometimes)
- */
-struct ch_pattern {
-    u32 id;        //!< pattern ID to report to the user
-    u32 flags;     //!< per-pattern flags (e.g. \ref CHIMERA_PATTERN_FLAG_UTF8)
-    u32 maxWidth;  //!< maximum width of a match, or UINT_MAX for inf.
-    u32 minWidth;  //!< minimum width of a match.
-    u32 fixedWidth;//!< pattern has fixed width.
-    u32 studyOffset;     //!< offset relative to struct start of study data,
-                         // or zero if there is none
-    u32 length; //!< length of struct plus pcre bytecode and study data
-    pcre_extra extra; //!< pcre_extra struct, used to store study data ptr for
-                      // the currently-running pcre at runtime.
-};
-
-static really_inline
-const void *ch_get_bytecode(const struct ch_database *db) {
-    assert(db);
-    const void *bytecode = (const char *)db + db->bytecode;
-    assert(ISALIGNED_16(bytecode));
-    return bytecode;
-}
-
-struct hs_database;
-
-static really_inline
-const struct hs_database *getHyperscanDatabase(const struct ch_bytecode *db) {
-    assert(db);
-    const char *ptr = (const char *)db;
-    const struct hs_database *hs_db;
-    hs_db = (const struct hs_database *)(ptr + db->databaseOffset);
-    assert(ISALIGNED_CL(hs_db));
-    return hs_db;
-}
-
-static really_inline
-const u32 *getUnguarded(const struct ch_bytecode *db) {
-    assert(db);
-    const char *ptr = (const char *)db;
-    const u32 *unguarded = (const u32 *)(ptr + db->unguardedOffset);
-    assert(ISALIGNED_N(unguarded, sizeof(u32)));
-    return unguarded;
-}
-
-static really_inline
-const struct ch_pattern *getPattern(const struct ch_bytecode *db, u32 i) {
-    assert(db);
-    assert(i < db->patternCount);
-    const char *ptr = (const char *)db;
-    const u32 *patternOffset = (const u32 *)(ptr + db->patternOffset);
-    assert(patternOffset[i] < db->length);
-    return (const struct ch_pattern *)(ptr + patternOffset[i]);
-}
-
-static really_inline
-ch_error_t hydbIsValid(const struct ch_database *hydb) {
-    if (!hydb || hydb->magic != CH_DB_MAGIC) {
-        DEBUG_PRINTF("bad magic (%u != %u)\n", hydb->magic, CH_DB_MAGIC);
-        return CH_INVALID;
-    }
-
-    if (hydb->version != HS_VERSION_32BIT) {
-        DEBUG_PRINTF("bad version\n");
-        return CH_DB_VERSION_ERROR;
-    }
-
-    return CH_SUCCESS;
-}
-
-#ifdef __cplusplus
-} /* extern "C" */
-#endif
-
-#endif /* CH_DATABASE_H_ */
-

chimera/ch_internal.h

@@ -1,44 +0,0 @@
-/*
- * Copyright (c) 2018, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Chimera: data structures and internals.
- */
-
-#ifndef CH_INTERNAL_H
-#define CH_INTERNAL_H
-
-#define CHIMERA_FLAG_NO_MULTIMATCH 1 //!< Don't run a multimatch scan
-#define CHIMERA_FLAG_GROUPS        2 //!< Return capturing groups
-#define CHIMERA_FLAG_ALL_CONFIRM   4 //!< All patterns need confirm
-#define CHIMERA_FLAG_ALL_SINGLE    8 //!< All patterns need only one match
-
-#define CHIMERA_PATTERN_FLAG_SINGLEMATCH    1 //!< only report the first match
-#define CHIMERA_PATTERN_FLAG_UTF8           2 //!< pattern is in UTF-8 mode
-
-#endif

chimera/ch_runtime.c

@@ -1,643 +0,0 @@
-/*
- * Copyright (c) 2018-2020, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Chimera: main runtime.
- */
-
-#include <limits.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-#include "ch.h"
-#include "hs.h"
-#include "hs_internal.h"
-#include "ue2common.h"
-#include "ch_database.h"
-#include "ch_internal.h"
-#include "ch_scratch.h"
-#include "util/multibit.h"
-#include "util/unicode_def.h"
-
-typedef struct queue_item PQ_T;
-
-static
-char PQ_COMP(PQ_T *pqc_items, int a, int b) {
-    if ((pqc_items)[a].to != (pqc_items)[b].to) {
-        return (pqc_items)[a].to < (pqc_items)[b].to;
-    } else if ((pqc_items)[a].from != (pqc_items)[b].from) {
-        return (pqc_items)[a].from < (pqc_items)[b].from;
-    } else {
-        return (pqc_items)[a].id < (pqc_items)[b].id;
-    }
-}
-
-static
-char PQ_COMP_B(PQ_T *pqc_items, int a, PQ_T b_fixed) {
-    if ((pqc_items)[a].to != (b_fixed).to) {
-        return (pqc_items)[a].to < (b_fixed).to;
-    } else if ((pqc_items)[a].from != (b_fixed).from) {
-        return (pqc_items)[a].from < (b_fixed).from;
-    } else {
-        return (pqc_items)[a].id < b_fixed.id;
-    }
-}
-
-#include "util/pqueue.h"
-
-static really_inline
-void pq_insert_with(struct match_pq *pq, int from, int to, u32 id) {
-    DEBUG_PRINTF("inserting pattern%u in pq at %u\n", id, to);
-    struct queue_item temp = {
-        .from = from,
-        .to = to,
-        .id = id,
-    };
-
-    pq_insert(pq->item, pq->size, temp);
-    ++pq->size;
-}
-
-static really_inline
-void pq_pop_nice(struct match_pq *pq) {
-    pq_pop(pq->item, pq->size);
-    pq->size--;
-}
-
-/** dummy event handler for use when user does not provide one */
-static
-int HS_CDECL null_onEvent(UNUSED unsigned id, UNUSED unsigned long long from,
-                 UNUSED unsigned long long to, UNUSED unsigned flags,
-                 UNUSED unsigned size, UNUSED const ch_capture_t *captured,
-                 UNUSED void *ctxt) {
-    return 0;
-}
-
-/** \brief Chimera runtime context. */
-struct HybridContext {
-    const char *data; //!< buffer being scanned
-    u32 length; //!< length of data buffer
-    u32 valid_utf8_highwater; //!< UTF-8 has been validated up to here.
-    const struct ch_bytecode *db;
-    struct ch_scratch *scratch;
-    struct match_pq *pq;
-    /** \brief user-supplied match callback */
-    int (HS_CDECL *match_callback)(unsigned int id, unsigned long long from,
-                          unsigned long long to, unsigned int flags,
-                          unsigned int size, const ch_capture_t *capture,
-                          void *ctx);
-    /** \brief user-supplied error callback */
-    int (HS_CDECL *error_callback)(ch_error_event_t error_type, unsigned int id,
-                          void *info, void *ctx);
-    /** \brief user-supplied context */
-    void *context;
-};
-
-// Internal PCRE func.
-extern int _pcre_valid_utf(const unsigned char *, int, int *);
-
-/** UTF-8 validity check. Returns >0 if the given region of the data is valid
- * UTF-8, 0 otherwise. */
-static
-char isValidUTF8(struct HybridContext *hyctx, u32 end) {
-    assert(hyctx);
-
-    if (hyctx->valid_utf8_highwater >= end) {
-        return 1; // Already validated.
-    }
-
-    const unsigned char *data =
-        (const unsigned char *)hyctx->data + hyctx->valid_utf8_highwater;
-    int validate_len = end - hyctx->valid_utf8_highwater;
-
-    DEBUG_PRINTF("validating %d bytes\n", validate_len);
-
-    int erroroffset = 0;
-    if (_pcre_valid_utf(data, validate_len, &erroroffset)) {
-        DEBUG_PRINTF("UTF8 invalid at offset %d\n", erroroffset);
-        return 0;
-    }
-
-    hyctx->valid_utf8_highwater = end;
-    return 1;
-}
-
-static
-const pcre *getPcre(const struct ch_pattern *pattern) {
-    const char *ptr = (const char *)pattern;
-    const pcre *p = (const pcre *)(ptr + ROUNDUP_N(sizeof(*pattern), 8));
-    assert(ISALIGNED_N(p, 8));
-    return p;
-}
-
-/** \brief Fill the Chimera groups array from a pcre_exec ovector. */
-static
-void fillGroupsFromOvector(ch_capture_t *groups, int numPairs, int *ovector) {
-    assert(groups);
-    assert(ISALIGNED_N(groups, alignof(ch_capture_t)));
-
-    DEBUG_PRINTF("filling %d groups (@ %p) from pcre ovector\n",
-                 numPairs, groups);
-
-    for (int i = 0; i < numPairs * 2; i += 2) {
-        if (ovector[i] == -1) {
-            groups->flags = CH_CAPTURE_FLAG_INACTIVE;
-        } else {
-            groups->flags = CH_CAPTURE_FLAG_ACTIVE;
-            assert(ovector[i] <= ovector[i + 1]);
-            groups->from = ovector[i];
-            groups->to = ovector[i + 1];
-        }
-        ++groups;
-    }
-}
-
-static
-ch_error_t handlePcreNonMatch(const struct ch_pattern *pattern, int rv,
-                              ch_error_event_handler onError,
-                              void *userContext) {
-    assert(rv < 0);
-
-    if (rv == PCRE_ERROR_NOMATCH) {
-        DEBUG_PRINTF("no match found by libpcre\n");
-        return CH_SUCCESS;
-    } else if (rv == PCRE_ERROR_MATCHLIMIT) {
-        DEBUG_PRINTF("pcre hit match limit\n");
-        if (onError) {
-            return onError(CH_ERROR_MATCHLIMIT, pattern->id, NULL,
-                           userContext);
-        }
-        return CH_SUCCESS;
-    } else if (rv == PCRE_ERROR_RECURSIONLIMIT) {
-        DEBUG_PRINTF("pcre hit recursion limit\n");
-        if (onError) {
-            return onError(CH_ERROR_RECURSIONLIMIT, pattern->id, NULL,
-                           userContext);
-        }
-        return CH_SUCCESS;
-    }
-
-    // All other errors not handled above are fatal.
-    return CH_FAIL_INTERNAL;
-}
-
-static
-ch_error_t scanPcre(struct HybridContext *hyctx, UNUSED unsigned int length,
-                    unsigned int offset, u32 id) {
-    const char *data = hyctx->data;
-    unsigned int full_length = hyctx->length;
-    ch_error_event_handler onError = hyctx->error_callback;
-    void *userContext = hyctx->context;
-
-    const struct ch_pattern *pattern = getPattern(hyctx->db, id);
-    const pcre *p = getPcre(pattern);
-
-    // Set up the PCRE extra block.
-    const pcre_extra *extra = &pattern->extra;
-
-    int startoffset = offset;
-
-    int *ovector = hyctx->scratch->ovector;
-    int ovectorSize = (hyctx->scratch->maxCaptureGroups + 1) * 3;
-    assert(ovectorSize >= 2);
-
-    DEBUG_PRINTF("scanning %u bytes, pattern %u, startoffset %d\n",
-                 length, id, startoffset);
-
-    int options = 0;
-    if (pattern->flags & CHIMERA_PATTERN_FLAG_UTF8) {
-        // We do our own UTF-8 validation.
-        options |= PCRE_NO_UTF8_CHECK;
-        if (!isValidUTF8(hyctx, full_length)) {
-            return handlePcreNonMatch(pattern, PCRE_ERROR_BADUTF8, onError,
-                                      userContext);
-        }
-    }
-
-    int rv = pcre_exec(p, extra, data, full_length, startoffset, options,
-                       ovector, ovectorSize);
-
-    DEBUG_PRINTF("pcre return code is %d\n", rv);
-
-    // Handle all non-match or error cases, all of which involve us
-    // terminating the loop.
-    if (rv < 0) {
-        return handlePcreNonMatch(pattern, rv, onError, userContext);
-    }
-
-    // We've found a match, and we should always have room for at least the
-    // start and end offsets in our ovector. Pass this info to the user.
-    assert(rv >= 1);
-    assert(rv < ovectorSize);
-    int from = ovector[0];
-    int to = ovector[1];
-    DEBUG_PRINTF("match %d -> %d\n", from, to);
-
-    struct ch_patterndata *pd = hyctx->scratch->patternData + id;
-
-    if (hyctx->db->flags & CHIMERA_FLAG_GROUPS) {
-        fillGroupsFromOvector(pd->match, rv, ovector);
-    } else {
-        rv = 0;
-    }
-    pd->groupCount = (u32)rv;
-
-    // Insert new matched item to the queue
-    pq_insert_with(hyctx->pq, from, to, id);
-
-    // Next scan starts at the first codepoint after the match.  It's
-    // possible that we have a vacuous match, in which case we must step
-    // past it to ensure that we always progress.
-    if (from != to) {
-        startoffset = to;
-    } else if (pattern->flags & CHIMERA_PATTERN_FLAG_UTF8) {
-        startoffset = to + 1;
-        while (startoffset < (int)full_length &&
-               ((data[startoffset] & 0xc0) == UTF_CONT_BYTE_HEADER)) {
-            ++startoffset;
-        }
-    } else {
-        startoffset = to + 1;
-    }
-
-    pd->scanStart = startoffset;
-    DEBUG_PRINTF("new offset %u\n", pd->scanStart);
-
-    return CH_SUCCESS;
-}
-
-static
-ch_error_t catchupPcre(struct HybridContext *hyctx, unsigned int id,
-                       unsigned long long from, unsigned long long to) {
-    ch_match_event_handler onEvent = hyctx->match_callback;
-    void *userContext = hyctx->context;
-    DEBUG_PRINTF("priority queue size %u\n", hyctx->pq->size);
-    while (hyctx->pq->size) {
-        u32 num_item = hyctx->pq->size;
-        struct queue_item *item = pq_top(hyctx->pq->item);
-        size_t top_from = item->from;
-        size_t top_to = item->to;
-        u32 top_id = item->id;
-
-        if (top_to > to) {
-            pq_insert_with(hyctx->pq, from, to, id);
-            break;
-        }
-        pq_pop_nice(hyctx->pq);
-
-        const struct ch_pattern *pattern = getPattern(hyctx->db, top_id);
-        struct ch_patterndata *pd = hyctx->scratch->patternData + top_id;
-
-        // Report match for pattern
-        DEBUG_PRINTF("trigger match@%zu\n", top_to);
-        ch_callback_t cbrv =
-            onEvent(pattern->id, top_from, top_to, 0 /* flags */,
-                    pd->groupCount, pd->match, userContext);
-
-        if (cbrv == CH_CALLBACK_TERMINATE) {
-            DEBUG_PRINTF("user callback told us to terminate scanning\n");
-            return CH_SCAN_TERMINATED;
-        } else if (cbrv == CH_CALLBACK_SKIP_PATTERN) {
-            DEBUG_PRINTF("user callback told us to skip this pattern\n");
-            pd->scanStart = hyctx->length;
-        }
-
-        if (top_id == id) {
-            break;
-        }
-
-        // Push a new match to replace the old one
-        unsigned int start = pd->scanStart;
-        unsigned int len = hyctx->length - pd->scanStart;
-        if (hyctx->length >= pd->scanStart &&
-            !(pattern->flags & CHIMERA_PATTERN_FLAG_SINGLEMATCH)) {
-            DEBUG_PRINTF("get a new match item\n");
-            int ret = scanPcre(hyctx, len, start, top_id);
-
-            if (ret == CH_CALLBACK_TERMINATE) {
-                DEBUG_PRINTF("user callback told us to terminate scanning\n");
-                return CH_SCAN_TERMINATED;
-            } else if (ret == CH_CALLBACK_SKIP_PATTERN) {
-                DEBUG_PRINTF("user callback told us to skip this pattern\n");
-                pd->scanStart = hyctx->length;
-                ret = CH_SUCCESS;
-            } else if (ret == CH_FAIL_INTERNAL) {
-                return ret;
-            }
-
-            // No further match is found
-            if (hyctx->pq->size == num_item - 1) {
-                pd->scanStart = hyctx->length;
-            }
-        }
-    }
-
-    return CH_SUCCESS;
-}
-
-/** \brief Callback used for internal Hyperscan multi-matcher. */
-static
-int HS_CDECL multiCallback(unsigned int id, unsigned long long from,
-                  unsigned long long to, UNUSED unsigned int flags,
-                  void *ctx) {
-    assert(ctx);
-    struct HybridContext *hyctx = ctx;
-
-    DEBUG_PRINTF("match for ID %u at offset %llu\n", id, to);
-    assert(id < hyctx->db->patternCount);
-
-    const struct ch_pattern *pattern = getPattern(hyctx->db, id);
-    struct ch_patterndata *pd = hyctx->scratch->patternData + id;
-    char needConfirm = pattern->fixedWidth == ~0U;
-
-    if (needConfirm &&
-        mmbit_isset(hyctx->scratch->active, hyctx->db->patternCount, id)) {
-        if ((hyctx->db->flags & CHIMERA_FLAG_ALL_CONFIRM) &&
-             mmbit_all(hyctx->scratch->active, hyctx->db->patternCount)) {
-            return 1;
-        }
-        return 0;
-    }
-    // Store the fact that we've seen this bit.
-    char already = mmbit_set(hyctx->scratch->active,
-                             hyctx->db->patternCount, id);
-    DEBUG_PRINTF("match from %u to %llu\n", pd->scanStart, to);
-
-    if (!already) {
-        pd->scanStart = 0;
-    } else if (to < pd->scanStart + pattern->minWidth) {
-        return 0;
-    } else if (pattern->flags & CHIMERA_PATTERN_FLAG_SINGLEMATCH) {
-        if ((hyctx->db->flags & CHIMERA_FLAG_ALL_SINGLE) &&
-             mmbit_all(hyctx->scratch->active, hyctx->db->patternCount)) {
-            return 1;
-        }
-        // Note: we may have unordered match from Hyperscan,
-        // thus possibly get to < pd->scanStart.
-        return 0;
-    }
-
-    int ret = HS_SUCCESS;
-    unsigned int start = pd->scanStart;
-    unsigned int len = hyctx->length - pd->scanStart;
-    assert(hyctx->length >= pd->scanStart);
-    const char *data = hyctx->data;
-    if (needConfirm) {
-        DEBUG_PRINTF("run confirm for the first time\n");
-        ret = scanPcre(hyctx, len, start, id);
-        hyctx->scratch->ret = ret;
-        if (ret == CH_CALLBACK_TERMINATE) {
-            DEBUG_PRINTF("user callback told us to terminate scanning\n");
-            return HS_SCAN_TERMINATED;
-        } else if (ret == CH_CALLBACK_SKIP_PATTERN) {
-            DEBUG_PRINTF("user callback told us to skip this pattern\n");
-            pd->scanStart = hyctx->length;
-            ret = HS_SUCCESS;
-            hyctx->scratch->ret = ret;
-        } else if (ret == CH_FAIL_INTERNAL) {
-            return ret;
-        }
-    } else {
-        if (already) {
-            DEBUG_PRINTF("catch up with new matches\n");
-            ret = catchupPcre(hyctx, id, from, to);
-
-            hyctx->scratch->ret = ret;
-            if (pd->scanStart >= hyctx->length) {
-                return ret;
-            }
-        }
-        int startoffset = 0;
-        // Next scan starts at the first codepoint after the match.  It's
-        // possible that we have a vacuous match, in which case we must step
-        // past it to ensure that we always progress.
-        if (from != to) {
-            startoffset = to;
-        } else if (pattern->flags & CHIMERA_PATTERN_FLAG_UTF8) {
-            startoffset = to + 1;
-            while (startoffset < (int)hyctx->length &&
-                   ((data[startoffset] & 0xc0) == UTF_CONT_BYTE_HEADER)) {
-                ++startoffset;
-            }
-        } else {
-            startoffset = to + 1;
-        }
-        pd->scanStart = startoffset;
-        int rv = 0;
-        if (hyctx->db->flags & CHIMERA_FLAG_GROUPS) {
-            ch_capture_t *groups = pd->match;
-            groups->flags = CH_CAPTURE_FLAG_ACTIVE;
-            groups->from = from;
-            groups->to = to;
-            rv = 1;
-        }
-        pd->groupCount = (u32)rv;
-        pq_insert_with(hyctx->pq, from, to, id);
-    }
-
-    return ret;
-}
-
-static
-hs_error_t scanHyperscan(struct HybridContext *hyctx, const char *data,
-                         unsigned int length) {
-    DEBUG_PRINTF("scanning %u bytes with Hyperscan\n", length);
-    const struct ch_bytecode *hydb = hyctx->db;
-    const hs_database_t *db = getHyperscanDatabase(hydb);
-    hs_scratch_t *scratch = hyctx->scratch->multi_scratch;
-
-    hs_error_t err = hs_scan(db, data, length, 0, scratch, multiCallback,
-                             hyctx);
-
-    return err;
-}
-
-/** \brief Init match priority queue.
- *
- * Add a first match offset for each pattern that is not supported by Hyperscan
- * with prefiltering.
- */
-static really_inline
-ch_error_t initQueue(struct HybridContext *hyctx, struct match_pq *pq) {
-    const struct ch_bytecode *db = hyctx->db;
-
-    u8 *active = hyctx->scratch->active;
-    mmbit_clear(active, db->patternCount);
-
-    // Init match queue size
-    pq->size = 0;
-
-    unsigned int length = hyctx->length;
-    const u32 *unguarded = getUnguarded(db);
-    for (u32 i = 0; i < db->unguardedCount; i++) {
-        u32 patternId = unguarded[i];
-        DEBUG_PRINTF("switch on unguarded pcre %u\n", patternId);
-        mmbit_set(active, db->patternCount, patternId);
-
-        DEBUG_PRINTF("get a new match item\n");
-        int ret = scanPcre(hyctx, length, 0, patternId);
-
-        struct ch_patterndata *pd = hyctx->scratch->patternData + patternId;
-        if (ret == CH_CALLBACK_TERMINATE) {
-            DEBUG_PRINTF("user callback told us to terminate scanning\n");
-            return CH_SCAN_TERMINATED;
-        } else if (ret == CH_CALLBACK_SKIP_PATTERN) {
-            DEBUG_PRINTF("user callback told us to skip this pattern\n");
-            pd->scanStart = length;
-            ret = CH_SUCCESS;
-        } else if (ret == CH_FAIL_INTERNAL) {
-            return ret;
-        }
-    }
-
-    return CH_SUCCESS;
-}
-
-static really_inline
-ch_error_t ch_scan_i(const ch_database_t *hydb,
-                     const char *data, unsigned int length,
-                     UNUSED unsigned int flags,
-                     ch_scratch_t *scratch,
-                     ch_match_event_handler onEvent,
-                     ch_error_event_handler onError,
-                     void *userContext) {
-    if (unlikely(!hydb || !scratch || !data)) {
-        DEBUG_PRINTF("args invalid\n");
-        return CH_INVALID;
-    }
-    ch_error_t ret = hydbIsValid(hydb);
-    if (ret != CH_SUCCESS) {
-        DEBUG_PRINTF("database invalid\n");
-        return ret;
-    }
-
-    if (!ISALIGNED_CL(scratch)) {
-        DEBUG_PRINTF("bad alignment %p\n", scratch);
-        return CH_INVALID;
-    }
-
-    if (scratch->magic != CH_SCRATCH_MAGIC) {
-        DEBUG_PRINTF("scratch invalid\n");
-        return CH_INVALID;
-    }
-
-    if (unlikely(markScratchInUse(scratch))) {
-        return CH_SCRATCH_IN_USE;
-    }
-
-    // Hyperscan underlying scratch and database validity will be checked by
-    // the hs_scan() call, so no need to do it here.
-
-    // PCRE takes the data region length in as an int, so this limits our block
-    // size to INT_MAX.
-    if (length > INT_MAX) {
-        DEBUG_PRINTF("length invalid\n");
-        unmarkScratchInUse(scratch);
-        return CH_INVALID;
-    }
-
-    const struct ch_bytecode *db = ch_get_bytecode(hydb);
-
-    scratch->pq.size = 0;
-    scratch->ret = CH_SUCCESS;
-
-    // Firstly, we run Hyperscan in block mode and add its matches into the
-    // active list for subsequent confirmation with pcre.
-    struct HybridContext hyctx = {
-        .data = data,
-        .length = length,
-        .valid_utf8_highwater = 0,
-        .db = db,
-        .scratch = scratch,
-        .pq = &scratch->pq,
-        .match_callback = onEvent ? onEvent : null_onEvent,
-        .error_callback = onError,
-        .context = userContext
-    };
-
-    // Init priority queue.
-    ret = initQueue(&hyctx, &scratch->pq);
-    if (ret != CH_SUCCESS) {
-        DEBUG_PRINTF("Chimera returned error %d\n", ret);
-        unmarkScratchInUse(scratch);
-        return ret;
-    }
-
-    if (!(db->flags & CHIMERA_FLAG_NO_MULTIMATCH)) {
-        ret = scanHyperscan(&hyctx, data, length);
-        // Errors from pcre scan.
-        if (scratch->ret == CH_CALLBACK_TERMINATE) {
-            DEBUG_PRINTF("Pcre terminates scan\n");
-            unmarkScratchInUse(scratch);
-            return CH_SCAN_TERMINATED;
-        } else if (scratch->ret != CH_SUCCESS) {
-            DEBUG_PRINTF("Pcre internal error\n");
-            unmarkScratchInUse(scratch);
-            return scratch->ret;
-        }
-        // Errors from Hyperscan scan. Note Chimera could terminate
-        // Hyperscan callback on purpose so this is not counted as an error.
-        if (ret != HS_SUCCESS && ret != HS_SCAN_TERMINATED) {
-            assert(scratch->ret == CH_SUCCESS);
-            DEBUG_PRINTF("Hyperscan returned error %d\n", ret);
-            unmarkScratchInUse(scratch);
-            return ret;
-        }
-    }
-
-    DEBUG_PRINTF("Flush priority queue\n");
-    // Catch up with PCRE and make up id and offsets as we don't really care
-    // about their values
-    ret = catchupPcre(&hyctx, ~0U, length, length);
-    if (ret != CH_SUCCESS) {
-        DEBUG_PRINTF("PCRE catch up returned error %d\n", ret);
-        unmarkScratchInUse(scratch);
-        return ret;
-    }
-
-    unmarkScratchInUse(scratch);
-    return CH_SUCCESS;
-}
-
-HS_PUBLIC_API
-ch_error_t HS_CDECL ch_scan(const ch_database_t *hydb, const char *data,
-                            unsigned int length, unsigned int flags,
-                            ch_scratch_t *scratch,
-                            ch_match_event_handler onEvent,
-                            ch_error_event_handler onError, void *userContext) {
-    ch_error_t ret = ch_scan_i(hydb, data, length, flags, scratch, onEvent,
-                               onError, userContext);
-
-    return ret;
-}
-
-HS_PUBLIC_API
-const char * HS_CDECL ch_version(void) {
-    return HS_VERSION_STRING;
-}

chimera/ch_runtime.h

@@ -1,378 +0,0 @@
-/*
- * Copyright (c) 2018, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef CH_RUNTIME_H_
-#define CH_RUNTIME_H_
-
-#include <stdlib.h>
-
-/**
- * @file
- * @brief The Chimera runtime API definition.
- *
- * Chimera is a hybrid of Hyperscan and PCRE regular expression engine.
- *
- * This header contains functions for using compiled Chimera databases for
- * scanning data at runtime.
- */
-
-#include "hs_common.h"
-
-#ifdef __cplusplus
-extern "C"
-{
-#endif
-
-struct ch_scratch;
-
-/**
- * A Chimera scratch space.
- */
-typedef struct ch_scratch ch_scratch_t;
-
-/**
- * Callback return value used to tell the Chimera matcher what to do after
- * processing this match.
- */
-typedef int ch_callback_t;
-
-/**
- * @defgroup CH_CALLBACK ch_callback_t values
- *
- * @{
- */
-
-/**
- * Continue matching.
- */
-#define CH_CALLBACK_CONTINUE     0
-
-/**
- * Terminate matching.
- */
-#define CH_CALLBACK_TERMINATE    1
-
-/**
- * Skip remaining matches for this ID and continue.
- */
-#define CH_CALLBACK_SKIP_PATTERN 2
-
-
-/** @} */
-
-
-/**
- * Type used to differentiate the errors raised with the @ref
- * ch_error_event_handler callback.
- */
-typedef int ch_error_event_t;
-
-/**
- * @defgroup CH_ERROR_EVENT ch_error_event_t values
- *
- * @{
- */
-
-/**
- * PCRE hits its match limit and reports PCRE_ERROR_MATCHLIMIT.
- */
-#define CH_ERROR_MATCHLIMIT      1
-
-/**
- * PCRE hits its recursion limit and reports PCRE_ERROR_RECURSIONLIMIT.
- */
-#define CH_ERROR_RECURSIONLIMIT  2
-
-/** @} */
-
-/**
- * Structure representing a captured subexpression within a match. An array of
- * these structures corresponding to capture groups in order is passed to the
- * callback on match, with active structures identified by the
- * CH_CAPTURE_FLAG_ACTIVE flag.
- */
-typedef struct ch_capture {
-    /**
-     * The flags indicating if this structure is active.
-     */
-    unsigned int flags;
-
-    /**
-     * offset at which this capture group begins.
-     */
-    unsigned long long from; /*< offset at which this capture group begins. */
-
-    /**
-     * offset at which this capture group ends.
-     */
-    unsigned long long to;
-} ch_capture_t;
-
-/**
- * @defgroup CH_CAPTURE ch_capture_t flags
- *
- * These flags are used in @ref ch_capture_t::flags to indicate if this
- * structure is active.
- *
- * @{
- */
-
-/**
- * Flag indicating that a particular capture group is inactive, used in @ref
- * ch_capture_t::flags.
- */
-#define CH_CAPTURE_FLAG_INACTIVE      0
-
-/**
- * Flag indicating that a particular capture group is active, used in @ref
- * ch_capture_t::flags.
- */
-#define CH_CAPTURE_FLAG_ACTIVE      1
-
-/** @} */
-
-/**
- * Definition of the match event callback function type.
- *
- * A callback function matching the defined type must be provided by the
- * application calling the @ref ch_scan()
- *
- * This callback function will be invoked whenever a match is located in the
- * target data during the execution of a scan. The details of the match are
- * passed in as parameters to the callback function, and the callback function
- * should return a value indicating whether or not matching should continue on
- * the target data. If no callbacks are desired from a scan call, NULL may be
- * provided in order to suppress match production.
- *
- * @param id
- *      The ID number of the expression that matched. If the expression was a
- *      single expression compiled with @ref ch_compile(), this value will be
- *      zero.
- *
- * @param from
- *      The offset of the first byte that matches the expression.
- *
- * @param to
- *      The offset after the last byte that matches the expression.
- *
- * @param flags
- *      This is provided for future use and is unused at present.
- *
- * @param size
- *      The number of valid entries pointed to by the captured parameter.
- *
- * @param captured
- *      A pointer to an array of @ref ch_capture_t structures that
- *      contain the start and end offsets of entire pattern match and
- *      each captured subexpression.
- *
- * @param ctx
- *      The pointer supplied by the user to the @ref ch_scan() function.
- *
- * @return
- *      The callback can return @ref CH_CALLBACK_TERMINATE to stop matching.
- *      Otherwise, a return value of @ref CH_CALLBACK_CONTINUE will continue,
- *      with the current pattern if configured to produce multiple matches per
- *      pattern, while a return value of @ref CH_CALLBACK_SKIP_PATTERN will
- *      cease matching this pattern but continue matching the next pattern.
- */
-typedef ch_callback_t (HS_CDECL *ch_match_event_handler)(unsigned int id,
-                                                unsigned long long from,
-                                                unsigned long long to,
-                                                unsigned int flags,
-                                                unsigned int size,
-                                                const ch_capture_t *captured,
-                                                void *ctx);
-
-/**
- * Definition of the Chimera error event callback function type.
- *
- * A callback function matching the defined type may be provided by the
- * application calling the @ref ch_scan function. This callback function
- * will be invoked when an error event occurs during matching; this indicates
- * that some matches for a given expression may not be reported.
- *
- * @param error_type
- *      The type of error event that occurred. Currently these errors
- *      correspond to resource limits on PCRE backtracking
- *      @ref CH_ERROR_MATCHLIMIT and @ref CH_ERROR_RECURSIONLIMIT.
- *
- * @param id
- *      The ID number of the expression that matched.
- *
- * @param info
- *      Event-specific data, for future use. Currently unused.
- *
- * @param ctx
- *      The context pointer supplied by the user to the @ref ch_scan
- *      function.
- *
- * @return
- *      The callback can return @ref CH_CALLBACK_SKIP_PATTERN to cease matching
- *      this pattern but continue matching the next pattern. Otherwise, we stop
- *      matching for all patterns with @ref CH_CALLBACK_TERMINATE.
- */
- typedef ch_callback_t (HS_CDECL *ch_error_event_handler)(
-                                                 ch_error_event_t error_type,
-                                                 unsigned int id, void *info,
-                                                 void *ctx);
-
-/**
- * The block regular expression scanner.
- *
- * This is the function call in which the actual pattern matching takes place
- * for block-mode pattern databases.
- *
- * @param db
- *      A compiled pattern database.
- *
- * @param data
- *      Pointer to the data to be scanned.
- *
- * @param length
- *      The number of bytes to scan.
- *
- * @param flags
- *      Flags modifying the behaviour of this function. This parameter is
- *      provided for future use and is unused at present.
- *
- * @param scratch
- *      A per-thread scratch space allocated by @ref ch_alloc_scratch() for this
- *      database.
- *
- * @param onEvent
- *      Pointer to a match event callback function. If a NULL pointer is given,
- *      no matches will be returned.
- *
- * @param onError
- *      Pointer to a error event callback function. If a NULL pointer is given,
- *      @ref CH_ERROR_MATCHLIMIT and @ref CH_ERROR_RECURSIONLIMIT errors will
- *      be ignored and match will continue.
- *
- * @param context
- *      The user defined pointer which will be passed to the callback function.
- *
- * @return
- *      Returns @ref CH_SUCCESS on success; @ref CH_SCAN_TERMINATED if the
- *      match callback indicated that scanning should stop; other values on
- *      error.
- */
-ch_error_t HS_CDECL ch_scan(const ch_database_t *db, const char *data,
-                            unsigned int length, unsigned int flags,
-                            ch_scratch_t *scratch,
-                            ch_match_event_handler onEvent,
-                            ch_error_event_handler onError,
-                            void *context);
-
-/**
- * Allocate a "scratch" space for use by Chimera.
- *
- * This is required for runtime use, and one scratch space per thread, or
- * concurrent caller, is required. Any allocator callback set by @ref
- * ch_set_scratch_allocator() or @ref ch_set_allocator() will be used by this
- * function.
- *
- * @param db
- *      The database, as produced by @ref ch_compile().
- *
- * @param scratch
- *      On first allocation, a pointer to NULL should be provided so a new
- *      scratch can be allocated. If a scratch block has been previously
- *      allocated, then a pointer to it should be passed back in to see if it
- *      is valid for this database block. If a new scratch block is required,
- *      the original will be freed and the new one returned, otherwise the
- *      previous scratch block will be returned. On success, the scratch block
- *      will be suitable for use with the provided database in addition to any
- *      databases that original scratch space was suitable for.
- *
- * @return
- *      @ref CH_SUCCESS on successful allocation; @ref CH_NOMEM if the
- *      allocation fails.  Other errors may be returned if invalid parameters
- *      are specified.
- */
-ch_error_t HS_CDECL ch_alloc_scratch(const ch_database_t *db,
-                                     ch_scratch_t **scratch);
-
-/**
- * Allocate a scratch space that is a clone of an existing scratch space.
- *
- * This is useful when multiple concurrent threads will be using the same set
- * of compiled databases, and another scratch space is required. Any allocator
- * callback set by @ref ch_set_scratch_allocator() or @ref ch_set_allocator()
- * will be used by this function.
- *
- * @param src
- *      The existing @ref ch_scratch_t to be cloned.
- *
- * @param dest
- *      A pointer to the new scratch space will be returned here.
- *
- * @return
- *      @ref CH_SUCCESS on success; @ref CH_NOMEM if the allocation fails.
- *      Other errors may be returned if invalid parameters are specified.
- */
-ch_error_t HS_CDECL ch_clone_scratch(const ch_scratch_t *src,
-                                     ch_scratch_t **dest);
-
-/**
- * Provides the size of the given scratch space.
- *
- * @param scratch
- *      A per-thread scratch space allocated by @ref ch_alloc_scratch() or @ref
- *      ch_clone_scratch().
- *
- * @param scratch_size
- *      On success, the size of the scratch space in bytes is placed in this
- *      parameter.
- *
- * @return
- *      @ref CH_SUCCESS on success, other values on failure.
- */
-ch_error_t HS_CDECL ch_scratch_size(const ch_scratch_t *scratch,
-                                    size_t *scratch_size);
-
-/**
- * Free a scratch block previously allocated by @ref ch_alloc_scratch() or @ref
- * ch_clone_scratch().
- *
- * The free callback set by @ref ch_set_scratch_allocator() or @ref
- * ch_set_allocator() will be used by this function.
- *
- * @param scratch
- *      The scratch block to be freed. NULL may also be safely provided.
- *
- * @return
- *      @ref CH_SUCCESS on success, other values on failure.
- */
-ch_error_t HS_CDECL ch_free_scratch(ch_scratch_t *scratch);
-
-#ifdef __cplusplus
-} /* extern "C" */
-#endif
-
-#endif /* CH_RUNTIME_H_ */

chimera/ch_scratch.c

@@ -1,316 +0,0 @@
-/*
- * Copyright (c) 2018, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Chimera: scratch space alloc.
- */
-
-#include <string.h>
-
-#include "allocator.h"
-#include "ch.h"
-#include "hs.h"
-#include "hs_internal.h"
-#include "ue2common.h"
-#include "ch_alloc.h"
-#include "ch_internal.h"
-#include "ch_scratch.h"
-#include "ch_database.h"
-
-static
-size_t getPatternDataSize(const ch_scratch_t *s) {
-    size_t numCapturingStructs =
-        s->patternCount * (s->maxCaptureGroups + 1);
-    return (sizeof(struct ch_patterndata) * s->patternCount) +
-           alignof(struct ch_capture) + // padding
-           (sizeof(struct ch_capture) * numCapturingStructs);
-}
-
-static
-void initPatternData(const ch_scratch_t *s) {
-    // ch_capture array is aligned, directly after the patterndata array.
-    char *ptr = (char *)s->patternData +
-            (sizeof(struct ch_patterndata) * s->patternCount);
-    struct ch_capture *cap = (struct ch_capture *)
-            (ROUNDUP_PTR(ptr, alignof(struct ch_capture)));
-
-    for (u32 i = 0; i < s->patternCount; i++) {
-        struct ch_patterndata *pd = &s->patternData[i];
-        pd->match = cap;
-        DEBUG_PRINTF("pattern %u: pd=%p, match=%p\n", i, pd, pd->match);
-        cap += (s->maxCaptureGroups + 1);
-    }
-}
-
-static
-ch_error_t alloc_scratch(const ch_scratch_t *proto, ch_scratch_t **scratch) {
-    size_t ovectorSize = (proto->maxCaptureGroups + 1) * sizeof(int) * 3;
-    size_t capturedSize =
-        sizeof(struct ch_capture) * (proto->maxCaptureGroups + 1);
-    size_t patternDataSize = getPatternDataSize(proto);
-    size_t activeSize = proto->activeSize;
-    size_t queueSize = proto->patternCount * sizeof(struct queue_item);
-
-    // max padding for alignment below.
-    size_t padding = alignof(int) + alignof(struct ch_capture) +
-                     alignof(struct ch_patterndata) +
-                     alignof(struct queue_item);
-
-    size_t allocSize = sizeof(ch_scratch_t) + ovectorSize + capturedSize +
-                       patternDataSize + activeSize + queueSize + padding
-                       + 256; /* padding for cacheline alignment */
-    ch_scratch_t *s;
-    ch_scratch_t *s_tmp = ch_scratch_alloc(allocSize);
-    ch_error_t err = ch_check_alloc(s_tmp);
-    if (err != CH_SUCCESS) {
-        ch_scratch_free(s_tmp);
-        *scratch = NULL;
-        return err;
-    }
-
-    memset(s_tmp, 0, allocSize);
-    s = ROUNDUP_PTR(s_tmp, 64);
-    // Set ordinary members.
-    *s = *proto;
-
-    s->magic = CH_SCRATCH_MAGIC;
-    s->in_use = 0;
-    s->scratch_alloc = (char *)s_tmp;
-
-    // Set pointers internal to allocation.
-
-    char *ptr = (char *)s + sizeof(*s);
-    ptr = ROUNDUP_PTR(ptr, alignof(int));
-    s->ovector = (int *)ptr;
-    ptr += ovectorSize;
-
-    ptr = ROUNDUP_PTR(ptr, alignof(struct ch_capture));
-    s->captured = (struct ch_capture *)ptr;
-    ptr += capturedSize;
-
-    ptr = ROUNDUP_PTR(ptr, alignof(struct ch_patterndata));
-    s->patternData = (struct ch_patterndata *)ptr;
-    ptr += patternDataSize;
-
-    // Pre-fill pattern data, setting captureOffsets
-    initPatternData(s);
-
-    ptr = ROUNDUP_PTR(ptr, alignof(struct queue_item));
-    s->pq.item = (struct queue_item *)ptr;
-    ptr += queueSize;
-
-    s->active = (u8 *)ptr;
-
-    // Store size.
-    s->scratchSize = allocSize;
-
-    // We should never overrun our allocation.
-    assert((ptr + activeSize) - (char *)s <= (ptrdiff_t)allocSize);
-
-    *scratch = s;
-    return CH_SUCCESS;
-}
-
-HS_PUBLIC_API
-ch_error_t HS_CDECL ch_alloc_scratch(const ch_database_t *hydb,
-                                     ch_scratch_t **scratch) {
-    if (!hydb || !scratch) {
-        DEBUG_PRINTF("invalid args\n");
-        return CH_INVALID;
-    }
-
-    DEBUG_PRINTF("hydb=%p, &scratch=%p\n", hydb, scratch);
-    ch_error_t rv = hydbIsValid(hydb);
-    if (rv != CH_SUCCESS) {
-        DEBUG_PRINTF("invalid database\n");
-        return rv;
-    }
-
-    if (*scratch != NULL) {
-        /* has to be aligned before we can do anything with it */
-        if (!ISALIGNED_CL(*scratch)) {
-            return CH_INVALID;
-        }
-        if ((*scratch)->magic != CH_SCRATCH_MAGIC) {
-            return CH_INVALID;
-        }
-        if (markScratchInUse(*scratch)) {
-            return CH_SCRATCH_IN_USE;
-        }
-    }
-
-    // We allocate a prototype of the scratch header to do our sizing with.
-    ch_scratch_t *proto;
-    ch_scratch_t *proto_tmp = ch_scratch_alloc(sizeof(ch_scratch_t) + 256);
-    ch_error_t proto_ret = ch_check_alloc(proto_tmp);
-    if (proto_ret != CH_SUCCESS) {
-        ch_scratch_free(proto_tmp);
-        ch_scratch_free(*scratch);
-        *scratch = NULL;
-        return proto_ret;
-    }
-
-    proto = ROUNDUP_PTR(proto_tmp, 64);
-
-    int resize = 0;
-    if (*scratch) {
-        *proto = **scratch;
-    } else {
-        memset(proto, 0, sizeof(*proto));
-        resize = 1;
-    }
-    proto->scratch_alloc = (char *)proto_tmp;
-
-    const struct ch_bytecode *db = ch_get_bytecode(hydb);
-
-    if (db->maxCaptureGroups > proto->maxCaptureGroups) {
-        proto->maxCaptureGroups = db->maxCaptureGroups;
-        resize = 1;
-    }
-
-    if (db->patternCount > proto->patternCount) {
-        proto->patternCount = db->patternCount;
-        proto->activeSize = db->activeSize;
-        resize = 1;
-    }
-
-    if (resize) {
-        if (*scratch) {
-            ch_scratch_free((*scratch)->scratch_alloc);
-        }
-
-        ch_error_t alloc_ret = alloc_scratch(proto, scratch);
-        ch_scratch_free(proto_tmp);
-        if (alloc_ret != CH_SUCCESS) {
-            *scratch = NULL;
-            return alloc_ret;
-        }
-    } else {
-        ch_scratch_free(proto_tmp);
-        unmarkScratchInUse(*scratch);
-    }
-
-    if (db->flags & CHIMERA_FLAG_NO_MULTIMATCH) {
-        return CH_SUCCESS;
-    }
-
-    // We may still have to realloc the underlying Hyperscan scratch.
-    rv = hs_alloc_scratch(getHyperscanDatabase(db),
-                           &(*scratch)->multi_scratch);
-    if (rv != HS_SUCCESS) {
-        DEBUG_PRINTF("hs_alloc_scratch for multi_scratch failed\n");
-        hs_free_scratch((*scratch)->multi_scratch);
-        ch_scratch_free((*scratch)->scratch_alloc);
-        *scratch = NULL;
-        return rv;
-    }
-
-    return CH_SUCCESS;
-}
-
-HS_PUBLIC_API
-ch_error_t HS_CDECL ch_clone_scratch(const ch_scratch_t *src,
-                                     ch_scratch_t **dest) {
-    if (!dest || !src || !ISALIGNED_CL(src) ||
-        src->magic != CH_SCRATCH_MAGIC) {
-        DEBUG_PRINTF("scratch invalid\n");
-        return CH_INVALID;
-    }
-
-    ch_error_t ret = alloc_scratch(src, dest);
-    if (ret != CH_SUCCESS) {
-        DEBUG_PRINTF("alloc_scratch failed\n");
-        *dest = NULL;
-        return ret;
-    }
-
-    if (src->multi_scratch) {
-        (*dest)->multi_scratch = NULL;
-        ret = hs_clone_scratch(src->multi_scratch, &(*dest)->multi_scratch);
-        if (ret != HS_SUCCESS) {
-            DEBUG_PRINTF("hs_clone_scratch(multi_scratch,...) failed\n");
-            ch_scratch_free(*dest);
-            return ret;
-        }
-    }
-
-    return CH_SUCCESS;
-}
-
-HS_PUBLIC_API
-ch_error_t HS_CDECL ch_free_scratch(ch_scratch_t *scratch) {
-    ch_error_t ret = CH_SUCCESS;
-    if (scratch) {
-        /* has to be aligned before we can do anything with it */
-        if (!ISALIGNED_CL(scratch)) {
-            return CH_INVALID;
-        }
-        if (scratch->magic != CH_SCRATCH_MAGIC) {
-            return CH_INVALID;
-        }
-        if (markScratchInUse(scratch)) {
-            return CH_SCRATCH_IN_USE;
-        }
-
-        if (scratch->multi_scratch) {
-            ret = hs_free_scratch(scratch->multi_scratch);
-        }
-
-        scratch->magic = 0;
-        assert(scratch->scratch_alloc);
-        DEBUG_PRINTF("scratch %p is really at %p : freeing\n", scratch,
-                     scratch->scratch_alloc);
-        ch_scratch_free(scratch->scratch_alloc);
-    }
-
-    return ret;
-}
-
-/** Not public, but used for info from our internal tools. Note that in the
- * hybrid matcher the scratch is definitely not a contiguous memory region. */
-HS_PUBLIC_API
-ch_error_t HS_CDECL ch_scratch_size(const ch_scratch_t *scratch, size_t *size) {
-    ch_error_t ret = CH_SUCCESS;
-    if (!size || !scratch || !ISALIGNED_CL(scratch) ||
-        scratch->magic != CH_SCRATCH_MAGIC) {
-        return CH_INVALID;
-    } else {
-        size_t multi_size = 0;
-
-        if (scratch->multi_scratch) {
-            ret = hs_scratch_size(scratch->multi_scratch, &multi_size);
-        }
-        if (ret) {
-            multi_size = 0;
-        }
-
-        *size = scratch->scratchSize + multi_size;
-    }
-
-    return ret;
-}

chimera/ch_scratch.h

@@ -1,119 +0,0 @@
-/*
- * Copyright (c) 2018, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Scratch and associated data structures.
- *
- * This header gets pulled into many places (many deep, slow to compile
- * places). Try to keep the included headers under control.
- */
-
-#ifndef CH_SCRATCH_H_
-#define CH_SCRATCH_H_
-
-#include "ch_common.h"
-#include "ch_runtime.h"
-
-#ifdef __cplusplus
-extern "C"
-{
-#endif
-
-#define CH_SCRATCH_MAGIC 0x554F4259 //!< Magic number stored in \ref ch_scratch
-
-struct queue_item {
-    int from; /** \brief used to store the start location. */
-    int to; /** \brief used to store the current location. */
-    u32 id; /**< pattern index. */
-};
-
-struct match_pq {
-    struct queue_item *item;
-    u32 size; /**< current size of the priority queue */
-};
-
-/** \brief Information about a pattern stored at runtime when a match is
- * encountered. */
-struct ch_patterndata {
-    struct ch_capture *match; //!< buffered group info
-    u32 groupCount; //!< number of capturing groups
-    u32 scanStart; //!< start of match window (still to be single-scanned).
-};
-
-/** \brief Scratch space header for Chimera. */
-struct ch_scratch {
-    u32 magic; //!< must be \ref CH_SCRATCH_MAGIC
-    u8 in_use; /**< non-zero when being used by an API call. */
-    struct hs_scratch *multi_scratch; //!< for hyperscan scatch.
-    int *ovector; //!< maximally-sized ovector for PCRE usage.
-    struct ch_capture *captured; //!< max-sized capture group struct.
-    u8 *active; //!< active multibit.
-    struct ch_patterndata *patternData; //!< per-pattern match data, indexed by
-                                        // pattern ID.
-    struct match_pq pq; //!< priority queue to ensure matching ordering
-    u32 patternCount; //!< number of patterns, used to size active multibit
-    u32 activeSize;   //!< size of active multibit
-    u32 maxCaptureGroups; //!< largest num of capturing groups required
-    u32 scratchSize; //!< size of allocation
-    int ret;  //!< return value in Hyperscan callback
-    char *scratch_alloc; /* user allocated scratch object */
-};
-
-/**
- * \brief Mark scratch as in use.
- *
- * Returns non-zero if it was already in use, zero otherwise.
- */
-static really_inline
-char markScratchInUse(struct ch_scratch *scratch) {
-    DEBUG_PRINTF("marking scratch as in use\n");
-    assert(scratch && scratch->magic == CH_SCRATCH_MAGIC);
-    if (scratch->in_use) {
-        DEBUG_PRINTF("scratch already in use!\n");
-        return 1;
-    }
-    scratch->in_use = 1;
-    return 0;
-}
-
-/**
- * \brief Mark scratch as no longer in use.
- */
-static really_inline
-void unmarkScratchInUse(struct ch_scratch *scratch) {
-    DEBUG_PRINTF("marking scratch as not in use\n");
-    assert(scratch && scratch->magic == CH_SCRATCH_MAGIC);
-    assert(scratch->in_use == 1);
-    scratch->in_use = 0;
-}
-
-#ifdef __cplusplus
-} /* extern "C" */
-#endif
-
-#endif /* CH_SCRATCH_H_ */

chimera/libch.pc.in

@@ -1,12 +0,0 @@
-prefix=@CMAKE_INSTALL_PREFIX@
-exec_prefix=@CMAKE_INSTALL_PREFIX@
-libdir=@CMAKE_INSTALL_PREFIX@/@CMAKE_INSTALL_LIBDIR@
-includedir=@CMAKE_INSTALL_PREFIX@/@CMAKE_INSTALL_INCLUDEDIR@
-
-Name: libch
-Description: Intel(R) Chimera Library
-Version: @HS_VERSION@
-Requires.private: libhs
-Libs: -L${libdir} -lchimera
-Libs.private: @PRIVATE_LIBS@
-Cflags: -I${includedir}/hs

cmake/arch.cmake

@@ -1,196 +0,0 @@
-# detect architecture features
-#
-# must be called after determining where compiler intrinsics are defined
-
-if (HAVE_C_X86INTRIN_H)
-    set (INTRIN_INC_H "x86intrin.h")
-elseif (HAVE_C_INTRIN_H)
-    set (INTRIN_INC_H "intrin.h")
-elseif (HAVE_C_ARM_NEON_H)
-    set (INTRIN_INC_H "arm_neon.h")
-    set (FAT_RUNTIME OFF)
-elseif (HAVE_C_PPC64EL_ALTIVEC_H)
-    set (INTRIN_INC_H "altivec.h")
-    set (FAT_RUNTIME OFF)
-else()
-    message (FATAL_ERROR "No intrinsics header found")
-endif ()
-
-if (ARCH_ARM32 OR ARCH_AARCH64)
-    CHECK_C_SOURCE_COMPILES("#include <${INTRIN_INC_H}>
-int main() {
-    int32x4_t a = vdupq_n_s32(1);
-    (void)a;
-}" HAVE_NEON)
-endif ()
-
-if (ARCH_AARCH64)
-    set(PREV_FLAGS "${CMAKE_C_FLAGS}")
-    if (BUILD_SVE2_BITPERM)
-        set(CMAKE_C_FLAGS "-march=${GNUCC_ARCH} ${CMAKE_C_FLAGS}")
-        CHECK_C_SOURCE_COMPILES("#include <arm_sve.h>
-        int main() {
-            svuint8_t a = svbext(svdup_u8(1), svdup_u8(2));
-            (void)a;
-        }" HAVE_SVE2_BITPERM)
-        if (HAVE_SVE2_BITPERM)
-            add_definitions(-DHAVE_SVE2_BITPERM)
-        endif ()
-    endif()
-    if (BUILD_SVE2)
-        set(CMAKE_C_FLAGS "-march=${GNUCC_ARCH} ${CMAKE_C_FLAGS}")
-        CHECK_C_SOURCE_COMPILES("#include <arm_sve.h>
-        int main() {
-            svuint8_t a = svbsl(svdup_u8(1), svdup_u8(2), svdup_u8(3));
-            (void)a;
-        }" HAVE_SVE2)
-    endif()
-    if (HAVE_SVE2 OR HAVE_SVE2_BITPERM)
-        add_definitions(-DHAVE_SVE2)
-    endif ()
-    if (BUILD_SVE)
-        set(CMAKE_C_FLAGS "-march=${GNUCC_ARCH} ${CMAKE_C_FLAGS}")
-        CHECK_C_SOURCE_COMPILES("#include <arm_sve.h>
-        int main() {
-            svuint8_t a = svdup_u8(1);
-            (void)a;
-        }" HAVE_SVE)
-    endif ()
-    if (HAVE_SVE OR HAVE_SVE2 OR HAVE_SVE2_BITPERM)
-        add_definitions(-DHAVE_SVE)
-    endif ()
-    set(CMAKE_C_FLAGS "${PREV_FLAGS}")
-endif()
-
-if (BUILD_AVX512)
-    CHECK_C_COMPILER_FLAG(${SKYLAKE_FLAG} HAS_ARCH_SKYLAKE)
-    if (NOT HAS_ARCH_SKYLAKE)
-        message (FATAL_ERROR "AVX512 not supported by compiler")
-    endif ()
-endif ()
-
-if (BUILD_AVX512VBMI)
-    CHECK_C_COMPILER_FLAG(${ICELAKE_FLAG} HAS_ARCH_ICELAKE)
-    if (NOT HAS_ARCH_ICELAKE)
-        message (FATAL_ERROR "AVX512VBMI not supported by compiler")
-    endif ()
-endif ()
-
-if (FAT_RUNTIME)
-    if (NOT DEFINED(BUILD_AVX2))
-        set(BUILD_AVX2 TRUE)
-    endif ()
-    # test the highest level microarch to make sure everything works
-    if (BUILD_AVX512)
-        if (BUILD_AVX512VBMI)
-            set (CMAKE_REQUIRED_FLAGS "${CMAKE_C_FLAGS} ${EXTRA_C_FLAGS} ${ICELAKE_FLAG}")
-        else ()
-            set (CMAKE_REQUIRED_FLAGS "${CMAKE_C_FLAGS} ${EXTRA_C_FLAGS} ${SKYLAKE_FLAG}")
-        endif (BUILD_AVX512VBMI)
-    elseif (BUILD_AVX2)
-        set (CMAKE_REQUIRED_FLAGS "${CMAKE_C_FLAGS} ${EXTRA_C_FLAGS} -march=core-avx2 -mavx2")
-    elseif ()
-        set (CMAKE_REQUIRED_FLAGS "${CMAKE_C_FLAGS} ${EXTRA_C_FLAGS} -march=core-i7 -mssse3")
-    endif ()
-else (NOT FAT_RUNTIME)
-    # if not fat runtime, then test given cflags
-    set (CMAKE_REQUIRED_FLAGS "${CMAKE_C_FLAGS} ${EXTRA_C_FLAGS} ${ARCH_C_FLAGS}")
-endif ()
-
-if (ARCH_IA32 OR ARCH_X86_64)
-    # ensure we have the minimum of SSE4.2 - call a SSE4.2 intrinsic
-    CHECK_C_SOURCE_COMPILES("#include <${INTRIN_INC_H}>
-int main() {
-    __m128i a = _mm_set1_epi8(1);
-    (void)_mm_shuffle_epi8(a, a);
-}" HAVE_SSE42)
-
-    # now look for AVX2
-    CHECK_C_SOURCE_COMPILES("#include <${INTRIN_INC_H}>
-#if !defined(__AVX2__)
-#error no avx2
-#endif
-
-int main(){
-    __m256i z = _mm256_setzero_si256();
-    (void)_mm256_xor_si256(z, z);
-}" HAVE_AVX2)
-
-    # and now for AVX512
-    CHECK_C_SOURCE_COMPILES("#include <${INTRIN_INC_H}>
-#if !defined(__AVX512BW__)
-#error no avx512bw
-#endif
-
-int main(){
-    __m512i z = _mm512_setzero_si512();
-    (void)_mm512_abs_epi8(z);
-}" HAVE_AVX512)
-
-    # and now for AVX512VBMI
-    CHECK_C_SOURCE_COMPILES("#include <${INTRIN_INC_H}>
-#if !defined(__AVX512VBMI__)
-#error no avx512vbmi
-#endif
-
-int main(){
-    __m512i a = _mm512_set1_epi8(0xFF);
-    __m512i idx = _mm512_set_epi64(3ULL, 2ULL, 1ULL, 0ULL, 7ULL, 6ULL, 5ULL, 4ULL);
-    (void)_mm512_permutexvar_epi8(idx, a);
-}" HAVE_AVX512VBMI)
-
-
-elseif (ARCH_ARM32 OR ARCH_AARCH64)
-    CHECK_C_SOURCE_COMPILES("#include <${INTRIN_INC_H}>
-int main() {
-    int32x4_t a = vdupq_n_s32(1);
-    (void)a;
-}" HAVE_NEON)
-elseif (ARCH_PPC64EL)
-    CHECK_C_SOURCE_COMPILES("#include <${INTRIN_INC_H}>
-int main() {
-    vector int a = vec_splat_s32(1);
-    (void)a;
-}" HAVE_VSX)
-else ()
-    message (FATAL_ERROR "Unsupported architecture")
-endif ()
-
-if (FAT_RUNTIME)
-    if ((ARCH_IA32 OR ARCH_X86_64) AND NOT HAVE_SSE42)
-        message(FATAL_ERROR "SSE4.2 support required to build fat runtime")
-    endif ()
-    if ((ARCH_IA32 OR ARCH_X86_64) AND BUILD_AVX2 AND NOT HAVE_AVX2)
-        message(FATAL_ERROR "AVX2 support required to build fat runtime")
-    endif ()
-    if ((ARCH_IA32 OR ARCH_X86_64) AND BUILD_AVX512 AND NOT HAVE_AVX512)
-        message(FATAL_ERROR "AVX512 support requested but not supported")
-    endif ()
-    if ((ARCH_IA32 OR ARCH_X86_64) AND BUILD_AVX512VBMI AND NOT HAVE_AVX512VBMI)
-        message(FATAL_ERROR "AVX512VBMI support requested but not supported")
-    endif ()
-else (NOT FAT_RUNTIME)
-    if ((ARCH_IA32 OR ARCH_X86_64) AND NOT BUILD_AVX2)
-        message(STATUS "Building without AVX2 support")
-    endif ()
-    if ((ARCH_IA32 OR ARCH_X86_64) AND NOT HAVE_AVX512)
-        message(STATUS "Building without AVX512 support")
-    endif ()
-    if ((ARCH_IA32 OR ARCH_X86_64) AND NOT HAVE_AVX512VBMI)
-        message(STATUS "Building without AVX512VBMI support")
-    endif ()
-    if ((ARCH_IA32 OR ARCH_X86_64) AND NOT HAVE_SSE42)
-        message(FATAL_ERROR "A minimum of SSE4.2 compiler support is required")
-    endif ()
-    if ((ARCH_ARM32 OR ARCH_AARCH64) AND NOT HAVE_NEON)
-        message(FATAL_ERROR "NEON support required for ARM support")
-    endif ()
-    if (ARCH_PPPC64EL AND NOT HAVE_VSX)
-        message(FATAL_ERROR "VSX support required for Power support")
-    endif ()
-
-endif ()
-
-unset (PREV_FLAGS)
-unset (CMAKE_REQUIRED_FLAGS)
-unset (INTRIN_INC_H)

cmake/arm64-cross.cmake

@@ -1,22 +0,0 @@
-set(CMAKE_SYSTEM_NAME "Linux")
-set(CMAKE_SYSTEM_PROCESSOR "aarch64")
-
-# specify the cross compiler
-set(CMAKE_C_COMPILER "$ENV{CROSS}gcc")
-set(CMAKE_CXX_COMPILER "$ENV{CROSS}g++")
-# where is the target environment
-set(CMAKE_SYSROOT $ENV{CROSS_SYS})
-
-set(Boost_INCLUDE_DIR $ENV{BOOST_PATH})
-
-# for libraries and headers in the target directories
-set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
-set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
-set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
-
-set(THREADS_PTHREAD_ARG "2" CACHE STRING "Result from TRY_RUN" FORCE)
-
-set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -falign-functions=16 -falign-jumps=16 -falign-labels=16 -falign-loops=16" CACHE STRING "" FORCE)
-
-set(GNUCC_ARCH "armv8.2-a+fp16+simd+rcpc+dotprod+crypto")
-set(TUNE_FLAG "neoverse-n1")

cmake/attrib.cmake

@@ -1,13 +0,0 @@
-# tests for compiler properties
-
-# set -Werror so we can't ignore unused attribute warnings
-set (CMAKE_REQUIRED_FLAGS "-Werror")
-
-CHECK_C_SOURCE_COMPILES("
-    int foo(int) __attribute__ ((ifunc(\"foo_i\")));
-    int f1(int i) { return i; }
-    void (*foo_i()) { return f1; }
-    int main(void) { return 0; }
-    " HAS_C_ATTR_IFUNC)
-
-unset(CMAKE_REQUIRED_FLAGS)

cmake/backtrace.cmake

@@ -1,56 +0,0 @@
-# The `backtrace' function is available on Linux via glibc, and on FreeBSD if
-# the 'libexecinfo' package is installed.
-
-CHECK_C_SOURCE_COMPILES(
-    "#include <stdlib.h>\n#include <execinfo.h>\nint main () { backtrace(NULL, 0); }"
-    BACKTRACE_LIBC)
-
-if(BACKTRACE_LIBC)
-    set(HAVE_BACKTRACE TRUE)
-    set(BACKTRACE_CFLAGS "")
-    set(BACKTRACE_LDFLAGS "")
-endif()
-
-if(NOT BACKTRACE_LIBC)
-    # FreeBSD 10 has backtrace but requires libexecinfo
-    list(INSERT CMAKE_REQUIRED_LIBRARIES 0 "-lexecinfo")
-    CHECK_C_SOURCE_COMPILES(
-        "#include <stdlib.h>\n#include <execinfo.h>\nint main () { backtrace(NULL, 0); }"
-        BACKTRACE_LIBEXECINFO)
-    list(REMOVE_ITEM CMAKE_REQUIRED_LIBRARIES "-lexecinfo")
-
-    if(BACKTRACE_LIBEXECINFO)
-        set(HAVE_BACKTRACE TRUE)
-        set(BACKTRACE_CFLAGS "")
-        set(BACKTRACE_LDFLAGS "-lexecinfo")
-    else()
-        # older FreeBSD requires it from ports
-        list(INSERT CMAKE_REQUIRED_INCLUDES 0 "/usr/local/include")
-        list(INSERT CMAKE_REQUIRED_LIBRARIES 0 "-L/usr/local/lib -lexecinfo")
-        CHECK_C_SOURCE_COMPILES(
-            "#include <stdlib.h>\n#include <execinfo.h>\nint main () { backtrace(NULL, 0); }"
-            BACKTRACE_LIBEXECINFO_LOCAL)
-        list(REMOVE_ITEM CMAKE_REQUIRED_INCLUDES 0 "/usr/local/include")
-        list(REMOVE_ITEM CMAKE_REQUIRED_LIBRARIES "-L/usr/local/lib -lexecinfo")
-        if(BACKTRACE_LIBEXECINFO_LOCAL)
-            set(HAVE_BACKTRACE TRUE)
-            set(BACKTRACE_CFLAGS "-I/usr/local/include")
-            set(BACKTRACE_LDFLAGS "-L/usr/local/lib -lexecinfo")
-        endif()
-    endif()
-endif()
-
-if(HAVE_BACKTRACE)
-    CHECK_C_COMPILER_FLAG(-rdynamic HAS_RDYNAMIC)
-    if(HAS_RDYNAMIC)
-        list(INSERT BACKTRACE_LDFLAGS 0 -rdynamic)
-    endif()
-else()
-    set(BACKTRACE_CFLAGS "")
-    set(BACKTRACE_LDFLAGS "")
-endif()
-
-# cmake scope fun
-set(HAVE_BACKTRACE ${HAVE_BACKTRACE} CACHE BOOL INTERNAL)
-set(BACKTRACE_CFLAGS ${BACKTRACE_CFLAGS} CACHE STRING INTERNAL)
-set(BACKTRACE_LDFLAGS ${BACKTRACE_LDFLAGS} CACHE STRING INTERNAL)

cmake/boost.cmake

@@ -1,72 +0,0 @@
-# Various checks related to Boost
-
-set(BOOST_USE_STATIC_LIBS OFF)
-set(BOOST_USE_MULTITHREADED OFF)
-set(BOOST_USE_STATIC_RUNTIME OFF)
-if (HAVE_LIBCPP)
-    # we need a more recent boost for libc++
-    set(BOOST_MINVERSION 1.61.0)
-else ()
-    set(BOOST_MINVERSION 1.57.0)
-endif ()
-set(BOOST_NO_BOOST_CMAKE ON)
-
-unset(Boost_INCLUDE_DIR CACHE)
-# we might have boost in tree, so provide a hint and try again
-set(BOOST_INCLUDEDIR "${PROJECT_SOURCE_DIR}/include")
-find_package(Boost ${BOOST_MINVERSION} QUIET)
-if(NOT Boost_FOUND)
-    # otherwise check for Boost installed on the system
-    unset(BOOST_INCLUDEDIR)
-    find_package(Boost ${BOOST_MINVERSION} QUIET)
-    if(NOT Boost_FOUND)
-        message(FATAL_ERROR "Boost ${BOOST_MINVERSION} or later not found. Either install system packages if available, extract Boost headers to ${CMAKE_SOURCE_DIR}/include, or set the CMake BOOST_ROOT variable.")
-    endif()
-endif()
-
-message(STATUS "Boost version: ${Boost_MAJOR_VERSION}.${Boost_MINOR_VERSION}.${Boost_SUBMINOR_VERSION}")
-
-# Boost 1.62 has a bug that we've patched around, check if it is required
-if (Boost_VERSION EQUAL 106200)
-    set (CMAKE_REQUIRED_INCLUDES ${BOOST_INCLUDEDIR} "${PROJECT_SOURCE_DIR}/include")
-    set (BOOST_REV_TEST "
-#include <boost/graph/graph_concepts.hpp>
-#include <boost/graph/adjacency_list.hpp>
-#include <boost/graph/reverse_graph.hpp>
-#include <boost/concept/assert.hpp>
-
-int main(int,char*[])
-{
-  using namespace boost;
-  // Check const reverse_graph
-  {
-    typedef adjacency_list< vecS, vecS, bidirectionalS,
-      property<vertex_color_t, int>,
-      property<edge_weight_t, int>,
-      property<graph_name_t, std::string>
-    > AdjList;
-    typedef reverse_graph<AdjList> Graph;
-    BOOST_CONCEPT_ASSERT(( BidirectionalGraphConcept<Graph> ));
-  }
-  return 0;
-}
-")
-
-    CHECK_CXX_SOURCE_COMPILES("${BOOST_REV_TEST}" BOOST_REVGRAPH_OK)
-
-    if (NOT BOOST_REVGRAPH_OK)
-        message(STATUS "trying patched")
-        CHECK_CXX_SOURCE_COMPILES("
-#include <boost-patched/graph/reverse_graph.hpp>
-${BOOST_REV_TEST}" BOOST_REVGRAPH_PATCH)
-    endif()
-
-    if (NOT BOOST_REVGRAPH_OK AND NOT BOOST_REVGRAPH_PATCH)
-        message(FATAL_ERROR "Something is wrong with this copy of boost::reverse_graph")
-    endif()
-
-    unset (CMAKE_REQUIRED_INCLUDES)
-else ()
-    unset(BOOST_REVGRAPH_OK CACHE)
-    unset(BOOST_REVGRAPH_PATCH CACHE)
-endif () # Boost 1.62.0

cmake/build_wrapper.sh

@@ -1,28 +0,0 @@
-#!/bin/sh -e
-# This is used for renaming symbols for the fat runtime, don't call directly
-# TODO: make this a lot less fragile!
-cleanup () {
-    rm -f ${SYMSFILE} ${KEEPSYMS}
-}
-
-PREFIX=$1
-KEEPSYMS_IN=$2
-shift 2
-# $@ contains the actual build command
-OUT=$(echo "$@" | rev | cut -d ' ' -f 2- | rev | sed 's/.* -o \(.*\.o\).*/\1/')
-trap cleanup INT QUIT EXIT
-SYMSFILE=$(mktemp -p /tmp ${PREFIX}_rename.syms.XXXXX)
-KEEPSYMS=$(mktemp -p /tmp keep.syms.XXXXX)
-# find the libc used by gcc
-LIBC_SO=$("$@" --print-file-name=libc.so.6)
-cp ${KEEPSYMS_IN} ${KEEPSYMS}
-# get all symbols from libc and turn them into patterns
-nm -f p -g -D ${LIBC_SO} | sed -s 's/\([^ @]*\).*/^\1$/' >> ${KEEPSYMS}
-# build the object
-"$@"
-# rename the symbols in the object
-nm -f p -g ${OUT} | cut -f1 -d' ' | grep -v -f ${KEEPSYMS} | sed -e "s/\(.*\)/\1\ ${PREFIX}_\1/" >> ${SYMSFILE}
-if test -s ${SYMSFILE}
-then
-    objcopy --redefine-syms=${SYMSFILE} ${OUT}
-fi

cmake/config.h.in

@@ -1,142 +0,0 @@
-/* used by cmake */
-
-#ifndef CONFIG_H_
-#define CONFIG_H_
-
-/* "Define if the build is 32 bit" */
-#cmakedefine ARCH_32_BIT
-
-/* "Define if the build is 64 bit" */
-#cmakedefine ARCH_64_BIT
-
-/* "Define if building for IA32" */
-#cmakedefine ARCH_IA32
-
-/* "Define if building for EM64T" */
-#cmakedefine ARCH_X86_64
-
-/* "Define if building for ARM32" */
-#cmakedefine ARCH_ARM32
-
-/* "Define if building for AARCH64" */
-#cmakedefine ARCH_AARCH64
-
-/* "Define if building for PPC64EL" */
-#cmakedefine ARCH_PPC64EL
-
-/* "Define if cross compiling for AARCH64" */
-#cmakedefine CROSS_COMPILE_AARCH64
-
-/* Define if building SVE for AARCH64. */
-#cmakedefine BUILD_SVE
-
-/* Define if building SVE2 for AARCH64. */
-#cmakedefine BUILD_SVE2
-
-/* Define if building SVE2+BITPERM for AARCH64. */
-#cmakedefine BUILD_SVE2_BITPERM
-
-/* internal build, switch on dump support. */
-#cmakedefine DUMP_SUPPORT
-
-/* Define if building "fat" runtime. */
-#cmakedefine FAT_RUNTIME
-
-/* Define if building AVX2 in the fat runtime. */
-#cmakedefine BUILD_AVX2
-
-/* Define if building AVX-512 in the fat runtime. */
-#cmakedefine BUILD_AVX512
-
-/* Define if building AVX512VBMI in the fat runtime. */
-#cmakedefine BUILD_AVX512VBMI
-
-/* Define to 1 if `backtrace' works. */
-#cmakedefine HAVE_BACKTRACE
-
-/* C compiler has __builtin_assume_aligned */
-#cmakedefine HAVE_CC_BUILTIN_ASSUME_ALIGNED
-
-/* C++ compiler has __builtin_assume_aligned */
-#cmakedefine HAVE_CXX_BUILTIN_ASSUME_ALIGNED
-
-/* C++ compiler has x86intrin.h */
-#cmakedefine HAVE_CXX_X86INTRIN_H
-
-/* C compiler has x86intrin.h */
-#cmakedefine HAVE_C_X86INTRIN_H
-
-/* C++ compiler has intrin.h */
-#cmakedefine HAVE_CXX_INTRIN_H
-
-/* C compiler has intrin.h */
-#cmakedefine HAVE_C_INTRIN_H
-
-/* C compiler has arm_neon.h */
-#cmakedefine HAVE_C_ARM_NEON_H
-
-/* C compiler has arm_sve.h */
-#cmakedefine HAVE_C_ARM_SVE_H
-
-/* C compiler has arm_neon.h */
-#cmakedefine HAVE_C_PPC64EL_ALTIVEC_H
-
-/* Define to 1 if you have the declaration of `pthread_setaffinity_np', and to
-   0 if you don't. */
-#cmakedefine HAVE_DECL_PTHREAD_SETAFFINITY_NP
-
-#cmakedefine HAVE_PTHREAD_NP_H
-
-/* Define to 1 if you have the `malloc_info' function. */
-#cmakedefine HAVE_MALLOC_INFO
-
-/* Define to 1 if you have the `memmem' function. */
-#cmakedefine HAVE_MEMMEM
-
-/* Define to 1 if you have a working `mmap' system call. */
-#cmakedefine HAVE_MMAP
-
-/* Define to 1 if `posix_memalign' works. */
-#cmakedefine HAVE_POSIX_MEMALIGN
-
-/* Define to 1 if you have the `setrlimit' function. */
-#cmakedefine HAVE_SETRLIMIT
-
-/* Define to 1 if you have the `shmget' function. */
-#cmakedefine HAVE_SHMGET
-
-/* Define to 1 if you have the `sigaction' function. */
-#cmakedefine HAVE_SIGACTION
-
-/* Define to 1 if you have the `sigaltstack' function. */
-#cmakedefine HAVE_SIGALTSTACK
-
-/* Define if the sqlite3_open_v2 call is available */
-#cmakedefine HAVE_SQLITE3_OPEN_V2
-
-/* Define to 1 if you have the <unistd.h> header file. */
-#cmakedefine HAVE_UNISTD_H
-
-/* Define to 1 if you have the `_aligned_malloc' function. */
-#cmakedefine HAVE__ALIGNED_MALLOC
-
-/* Define if compiler has __builtin_constant_p */
-#cmakedefine HAVE__BUILTIN_CONSTANT_P
-
-/* Optimize, inline critical functions */
-#cmakedefine HS_OPTIMIZE
-
-#cmakedefine HS_VERSION
-#cmakedefine HS_MAJOR_VERSION
-#cmakedefine HS_MINOR_VERSION
-#cmakedefine HS_PATCH_VERSION
-
-#cmakedefine BUILD_DATE
-
-/* define if this is a release build. */
-#cmakedefine RELEASE_BUILD
-
-/* define if reverse_graph requires patch for boost 1.62.0 */
-#cmakedefine BOOST_REVGRAPH_PATCH
-
-#endif /* CONFIG_H_ */

cmake/formatdate.py

@@ -1,18 +0,0 @@
-#!/usr/bin/env python
-from __future__ import print_function
-import os
-import sys
-import datetime
-
-def usage():
-    print("Usage:", os.path.basename(sys.argv[0]), "<seconds from epoch>")
-
-if len(sys.argv) != 2:
-    usage()
-    sys.exit(1)
-
-ts = sys.argv[1]
-
-build_date = datetime.datetime.utcfromtimestamp(int(ts))
-
-print(build_date.strftime("%Y-%m-%d"))

cmake/keep.syms.in

@@ -1,11 +0,0 @@
-# names to exclude
-hs_misc_alloc
-hs_misc_free
-hs_free_scratch
-hs_stream_alloc
-hs_stream_free
-hs_scratch_alloc
-hs_scratch_free
-hs_database_alloc
-hs_database_free
-^_

cmake/pcre.cmake

@@ -1,63 +0,0 @@
-# first look in pcre-$version or pcre subdirs
-if (PCRE_SOURCE)
-    # either provided on cmdline or we've seen it already
-    set (PCRE_BUILD_SOURCE TRUE)
-elseif (EXISTS ${PROJECT_SOURCE_DIR}/pcre-${PCRE_REQUIRED_VERSION})
-    set (PCRE_SOURCE ${PROJECT_SOURCE_DIR}/pcre-${PCRE_REQUIRED_VERSION})
-    set (PCRE_BUILD_SOURCE TRUE)
-elseif (EXISTS ${PROJECT_SOURCE_DIR}/pcre)
-    set (PCRE_SOURCE ${PROJECT_SOURCE_DIR}/pcre)
-    set (PCRE_BUILD_SOURCE TRUE)
-endif()
-
-if (PCRE_BUILD_SOURCE)
-    if (NOT IS_ABSOLUTE ${PCRE_SOURCE})
-        set(PCRE_SOURCE "${CMAKE_BINARY_DIR}/${PCRE_SOURCE}")
-    endif ()
-    set (saved_INCLUDES "${CMAKE_REQUIRED_INCLUDES}")
-    set (CMAKE_REQUIRED_INCLUDES "${CMAKE_REQUIRED_INCLUDES} ${PCRE_SOURCE}")
-
-    if (PCRE_CHECKED)
-        set(PCRE_INCLUDE_DIRS ${PCRE_SOURCE} ${PROJECT_BINARY_DIR}/pcre)
-        set(PCRE_LDFLAGS -L"${LIBDIR}" -lpcre)
-
-        # already processed this file and set up pcre building
-        return()
-    endif ()
-
-    # first, check version number
-    CHECK_C_SOURCE_COMPILES("#include <pcre.h.generic>
-    #if PCRE_MAJOR != ${PCRE_REQUIRED_MAJOR_VERSION} || PCRE_MINOR < ${PCRE_REQUIRED_MINOR_VERSION}
-    #error Incorrect pcre version
-    #endif
-    main() {}" CORRECT_PCRE_VERSION)
-    set (CMAKE_REQUIRED_INCLUDES "${saved_INCLUDES}")
-
-    if (NOT CORRECT_PCRE_VERSION)
-        unset(CORRECT_PCRE_VERSION CACHE)
-        message(STATUS "Incorrect version of pcre - version ${PCRE_REQUIRED_VERSION} or above is required")
-        return ()
-    else()
-        message(STATUS "PCRE version ${PCRE_REQUIRED_VERSION} or above - building from source.")
-    endif()
-
-    # PCRE compile options
-    option(PCRE_BUILD_PCRECPP OFF)
-    option(PCRE_BUILD_PCREGREP OFF)
-    option(PCRE_SHOW_REPORT OFF)
-    set(PCRE_SUPPORT_UNICODE_PROPERTIES ON CACHE BOOL "Build pcre with unicode")
-    add_subdirectory(${PCRE_SOURCE} ${PROJECT_BINARY_DIR}/pcre EXCLUDE_FROM_ALL)
-    set(PCRE_INCLUDE_DIRS ${PCRE_SOURCE} ${PROJECT_BINARY_DIR}/pcre)
-    set(PCRE_LDFLAGS -L"${LIBDIR}" -lpcre)
-else ()
-    # pkgconf should save us
-    find_package(PkgConfig)
-    pkg_check_modules(PCRE libpcre>=${PCRE_REQUIRED_VERSION})
-    if (PCRE_FOUND)
-        set(CORRECT_PCRE_VERSION TRUE)
-        message(STATUS "PCRE version ${PCRE_REQUIRED_VERSION} or above")
-    else ()
-        message(STATUS "PCRE version ${PCRE_REQUIRED_VERSION} or above not found")
-        return ()
-    endif ()
-endif (PCRE_BUILD_SOURCE)

cmake/platform.cmake

@@ -1,24 +0,0 @@
-# determine compiler
-if (CMAKE_CXX_COMPILER_ID MATCHES "Clang")
-  set(CMAKE_COMPILER_IS_CLANG TRUE)
-endif()
-
-# determine the target arch
-
-if (CROSS_COMPILE_AARCH64)
-  set(ARCH_AARCH64 TRUE)
-  set(ARCH_64_BIT TRUE)
-  message(STATUS "Cross compiling for aarch64")
-else()
-  # really only interested in the preprocessor here
-  CHECK_C_SOURCE_COMPILES("#if !(defined(__x86_64__) || defined(_M_X64))\n#error not 64bit\n#endif\nint main(void) { return 0; }" ARCH_X86_64)
-  CHECK_C_SOURCE_COMPILES("#if !(defined(__i386__) || defined(_M_IX86))\n#error not 32bit\n#endif\nint main(void) { return 0; }" ARCH_IA32)
-  CHECK_C_SOURCE_COMPILES("#if !defined(__ARM_ARCH_ISA_A64)\n#error not 64bit\n#endif\nint main(void) { return 0; }" ARCH_AARCH64)
-  CHECK_C_SOURCE_COMPILES("#if !defined(__ARM_ARCH_ISA_ARM)\n#error not 32bit\n#endif\nint main(void) { return 0; }" ARCH_ARM32)
-  CHECK_C_SOURCE_COMPILES("#if !defined(__PPC64__) && !(defined(__LITTLE_ENDIAN__) && defined(__VSX__))\n#error not ppc64el\n#endif\nint main(void) { return 0; }" ARCH_PPC64EL)
-  if (ARCH_X86_64 OR ARCH_AARCH64 OR ARCH_PPC64EL)
-    set(ARCH_64_BIT TRUE)
-  else()
-    set(ARCH_32_BIT TRUE)
-  endif()
-endif()

cmake/ragel.cmake

@@ -1,16 +0,0 @@
-# function for doing all the dirty work in turning a .rl into C++
-
-function(ragelmaker src_rl)
-    get_filename_component(src_dir ${src_rl} PATH) # old cmake needs PATH
-    get_filename_component(src_file ${src_rl} NAME_WE)
-    set(rl_out ${CMAKE_CURRENT_BINARY_DIR}/${src_dir}/${src_file}.cpp)
-    add_custom_command(
-        OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${src_dir}/${src_file}.cpp
-        COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_CURRENT_BINARY_DIR}/${src_dir}
-        COMMAND ${RAGEL} ${CMAKE_CURRENT_SOURCE_DIR}/${src_rl} -o ${rl_out}
-        DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/${src_rl}
-        )
-    add_custom_target(ragel_${src_file} DEPENDS ${rl_out})
-    set_source_files_properties(${rl_out} PROPERTIES GENERATED TRUE)
-endfunction(ragelmaker)
-

cmake/setenv-arm64-cross.sh

@@ -1,19 +0,0 @@
-#!/bin/bash
-export BOOST_VERSION=1_57_0
-export BOOST_DOT_VERSION=${BOOST_VERSION//_/.}
-export CROSS=<arm-cross-compiler-dir>/bin/aarch64-linux-gnu-
-export CROSS_SYS=<arm-cross-compiler-system-dir>
-
-# if [ ! -d "boost_$BOOST_VERSION" ];
-# then
-# 	wget -O boost_$BOOST_VERSION.tar.gz https://sourceforge.net/projects/boost/files/boost/$BOOST_DOT_VERSION/boost_$BOOST_VERSION.tar.gz/download
-# 	tar xf boost_$BOOST_VERSION.tar.gz
-# fi
-if [ ! -d "pcre-8.41" ];
-then
-	wget -O pcre-8.41.tar.bz2 https://ftp.pcre.org/pub/pcre/pcre-8.41.tar.bz2
-	tar xf pcre-8.41.tar.bz2
-	export PCRE_SOURCE=1
-fi
-
-export BOOST_PATH=<boost-source-dir>

cmake/sqlite3.cmake

@@ -1,51 +0,0 @@
-#
-# a lot of noise to find sqlite
-#
-
-option(SQLITE_PREFER_STATIC "Build sqlite3 statically instead of using an installed lib" OFF)
-
-if(NOT SQLITE_PREFER_STATIC)
-find_package(PkgConfig QUIET)
-
-# first check for sqlite on the system
-pkg_check_modules(SQLITE3 sqlite3)
-endif()
-
-if (NOT SQLITE3_FOUND)
-    message(STATUS "looking for sqlite3 in source tree")
-    # look in the source tree
-    if (EXISTS "${PROJECT_SOURCE_DIR}/sqlite3/sqlite3.h" AND
-            EXISTS "${PROJECT_SOURCE_DIR}/sqlite3/sqlite3.c")
-        message(STATUS "  found sqlite3 in source tree")
-        set(SQLITE3_FOUND TRUE)
-        set(SQLITE3_BUILD_SOURCE TRUE)
-        set(SQLITE3_INCLUDE_DIRS "${PROJECT_SOURCE_DIR}/sqlite3")
-        set(SQLITE3_LDFLAGS sqlite3_static)
-    else()
-        message(STATUS "  no sqlite3 in source tree")
-    endif()
-endif()
-
-# now do version checks
-if (SQLITE3_FOUND)
-    list(INSERT CMAKE_REQUIRED_INCLUDES 0 "${SQLITE3_INCLUDE_DIRS}")
-    CHECK_C_SOURCE_COMPILES("#include <sqlite3.h>\n#if SQLITE_VERSION_NUMBER >= 3008007 && SQLITE_VERSION_NUMBER < 3008010\n#error broken sqlite\n#endif\nint main() {return 0;}" SQLITE_VERSION_OK)
-    if (NOT SQLITE_VERSION_OK)
-        message(FATAL_ERROR "sqlite3 is broken from 3.8.7 to 3.8.10 - please find a working version")
-    endif()
-if (NOT SQLITE3_BUILD_SOURCE)
-    set(_SAVED_FLAGS ${CMAKE_REQUIRED_FLAGS})
-    list(INSERT CMAKE_REQUIRED_LIBRARIES 0 ${SQLITE3_LDFLAGS})
-    CHECK_SYMBOL_EXISTS(sqlite3_open_v2 sqlite3.h HAVE_SQLITE3_OPEN_V2)
-    list(REMOVE_ITEM CMAKE_REQUIRED_INCLUDES "${SQLITE3_INCLUDE_DIRS}")
-    list(REMOVE_ITEM CMAKE_REQUIRED_LIBRARIES ${SQLITE3_LDFLAGS})
-else()
-    if (NOT TARGET sqlite3_static)
-    # build sqlite as a static lib to compile into our test programs
-    add_library(sqlite3_static STATIC "${PROJECT_SOURCE_DIR}/sqlite3/sqlite3.c")
-    set_target_properties(sqlite3_static PROPERTIES COMPILE_FLAGS "-Wno-error -Wno-extra -Wno-unused -Wno-cast-qual -DSQLITE_OMIT_LOAD_EXTENSION")
-    endif()
-endif()
-endif()
-
-# that's enough about sqlite

doc/dev-reference/CMakeLists.txt

@@ -1,35 +0,0 @@
-find_program(DOXYGEN doxygen)
-
-if (DOXYGEN STREQUAL DOXYGEN-NOTFOUND)
-    message(STATUS "Doxygen not found, unable to generate API reference")
-else()
-configure_file("${CMAKE_CURRENT_SOURCE_DIR}/hyperscan.doxyfile.in"
-    "${CMAKE_CURRENT_BINARY_DIR}/hyperscan.doxyfile" @ONLY)
-
-add_custom_target(dev-reference-doxygen
-    ${DOXYGEN} ${CMAKE_CURRENT_BINARY_DIR}/hyperscan.doxyfile
-    COMMENT "Building doxygen XML for API reference")
-endif()
-
-find_program(SPHINX_BUILD sphinx-build)
-
-if (SPHINX_BUILD STREQUAL SPHINX_BUILD-NOTFOUND)
-    message(STATUS "Sphinx not found, unable to generate developer reference")
-else()
-set(SPHINX_BUILD_DIR "${CMAKE_CURRENT_BINARY_DIR}/_build")
-set(SPHINX_CACHE_DIR "${CMAKE_CURRENT_BINARY_DIR}/_doctrees")
-set(SPHINX_HTML_DIR "${CMAKE_CURRENT_BINARY_DIR}/html")
-
-configure_file("${CMAKE_CURRENT_SOURCE_DIR}/conf.py.in"
-    "${CMAKE_CURRENT_BINARY_DIR}/conf.py" @ONLY)
-
-add_custom_target(dev-reference
-    ${SPHINX_BUILD}
-        -b html
-        -c "${CMAKE_CURRENT_BINARY_DIR}"
-        -d "${SPHINX_CACHE_DIR}"
-        "${CMAKE_CURRENT_SOURCE_DIR}"
-        "${SPHINX_HTML_DIR}"
-    DEPENDS dev-reference-doxygen
-    COMMENT "Building HTML dev reference with Sphinx")
-endif()

doc/dev-reference/_static/hyperscan.css

@@ -1,10 +0,0 @@
-/* Differentiate the way we display regex fragments. */
-.regexp {
-  color: darkred !important;
-}
-
-/* Avoid (the alabaster theme default) Goudy Old Style, which renders in
- * italics on some Mac/Safari systems. */
-body {
-    font-family: 'minion pro', 'bell mt', Georgia, 'Hiragino Mincho Pro', serif;
-}

doc/dev-reference/api_constants.rst

@@ -1,53 +0,0 @@
-.. _api_constants:
-
-########################
-API Reference: Constants
-########################
-
-***********
-Error Codes
-***********
-
-.. doxygengroup:: HS_ERROR
-   :content-only:
-   :no-link:
-
-*****************
-hs_expr_ext flags
-*****************
-
-.. doxygengroup:: HS_EXT_FLAG
-   :content-only:
-   :no-link:
-
-*************
-Pattern flags
-*************
-
-.. doxygengroup:: HS_PATTERN_FLAG
-   :content-only:
-   :no-link:
-
-*************************
-CPU feature support flags
-*************************
-
-.. doxygengroup:: HS_CPU_FEATURES_FLAG
-   :content-only:
-   :no-link:
-
-****************
-CPU tuning flags
-****************
-
-.. doxygengroup:: HS_TUNE_FLAG
-   :content-only:
-   :no-link:
-
-******************
-Compile mode flags
-******************
-
-.. doxygengroup:: HS_MODE_FLAG
-   :content-only:
-   :no-link:

doc/dev-reference/api_files.rst

@@ -1,29 +0,0 @@
-.. _api_files:
-
-####################
-API Reference: Files
-####################
-
-**********
-File: hs.h
-**********
-
-.. doxygenfile:: hs.h
-
-*****************
-File: hs_common.h
-*****************
-
-.. doxygenfile:: hs_common.h
-
-******************
-File: hs_compile.h
-******************
-
-.. doxygenfile:: hs_compile.h
-
-******************
-File: hs_runtime.h
-******************
-
-.. doxygenfile:: hs_runtime.h

doc/dev-reference/chimera.rst

@@ -1,333 +0,0 @@
-.. _chimera:
-
-#######
-Chimera
-#######
-
-This section describes Chimera library.
-
-************
-Introduction
-************
-
-Chimera is a software regular expression matching engine that is a hybrid of
-Hyperscan and PCRE. The design goals of Chimera are to fully support PCRE
-syntax as well as to take advantage of the high performance nature of Hyperscan.
-
-Chimera inherits the design guideline of Hyperscan with C APIs for compilation
-and scanning.
-
-The Chimera API itself is composed of two major components:
-
-===========
-Compilation
-===========
-
-These functions take a group of regular expressions, along with identifiers and
-option flags, and compile them into an immutable database that can be used by
-the Chimera scanning API. This compilation process performs considerable
-analysis and optimization work in order to build a database that will match
-the given expressions efficiently.
-
-See :ref:`chcompile` for more details
-
-========
-Scanning
-========
-
-Once a Chimera database has been created, it can be used to scan data in memory.
-Chimera only supports block mode in which we scan a single contiguous block in
-memory.
-
-Matches are delivered to the application via a user-supplied callback function
-that is called synchronously for each match.
-
-For a given database, Chimera provides several guarantees:
-
-* No memory allocations occur at runtime with the exception of scratch space
-  allocation, it should be done ahead of time for performance-critical
-  applications:
-
-  - **Scratch space**: temporary memory used for internal data at scan time.
-    Structures in scratch space do not persist beyond the end of a single scan
-    call.
-
-* The size of the scratch space required for a given database is fixed and
-  determined at database compile time. This means that the memory requirement
-  of the application are known ahead of time, and the scratch space can be
-  pre-allocated if required for performance reasons.
-
-* Any pattern that has successfully been compiled by the Chimera compiler can
-  be scanned against any input. There could be internal resource limits or
-  other limitations caused by PCRE at runtime that could cause a scan call to
-  return an error.
-
-.. note:: Chimera is designed to have the same matching behavior as PCRE,
-   including greedy/ungreedy, capturing, etc. Chimera reports both
-   **start offset** and **end offset** for each match like PCRE. Different
-   from the fashion of reporting all matches in Hyperscan, Chimera only reports
-   non-overlapping matches. For example, the pattern :regexp:`/foofoo/` will
-   match ``foofoofoofoo`` at offsets (0, 6) and (6, 12).
-
-.. note:: Since Chimera is a hybrid of Hyperscan and PCRE in order to support
-   full PCRE syntax, there will be extra performance overhead compared to
-   Hyperscan-only solution. Please always use Hyperscan for better performance
-   unless you must need full PCRE syntax support.
-
-See :ref:`chruntime` for more details
-
-************
-Requirements
-************
-
-The PCRE library (http://pcre.org/) version 8.41 is required for Chimera.
-
-.. note:: Since Chimera needs to reference PCRE internal function, please place PCRE source
-   directory under Hyperscan root directory in order to build Chimera.
-
-Beside this, both hardware and software requirements of Chimera are the same to Hyperscan.
-See :ref:`hardware` and :ref:`software` for more details.
-
-.. note:: Building Hyperscan will automatically generate Chimera library.
-   Currently only static library is supported for Chimera, so please
-   use static build type when configure CMake build options.
-
-.. _chcompile:
-
-******************
-Compiling Patterns
-******************
-
-===================
-Building a Database
-===================
-
-The Chimera compiler API accepts regular expressions and converts them into a
-compiled pattern database that can then be used to scan data.
-
-The API provides two functions that compile regular expressions into
-databases:
-
-#. :c:func:`ch_compile`: compiles a single expression into a pattern database.
-
-#. :c:func:`ch_compile_multi`: compiles an array of expressions into a pattern
-   database. All of the supplied patterns will be scanned for concurrently at
-   scan time, with user-supplied identifiers returned when they match.
-
-#. :c:func:`ch_compile_ext_multi`: compiles an array of expressions as above,
-   but allows PCRE match limits to be specified for each expression.
-
-Compilation allows the Chimera library to analyze the given pattern(s) and
-pre-determine how to scan for these patterns in an optimized fashion using
-Hyperscan and PCRE.
-
-===============
-Pattern Support
-===============
-
-Chimera fully supports the pattern syntax used by the PCRE library ("libpcre"),
-described at <http://www.pcre.org/>.The version of PCRE used to validate
-Chimera's interpretation of this syntax is 8.41.
-
-=========
-Semantics
-=========
-
-Chimera supports the exact same semantics of PCRE library. Moreover, it supports
-multiple simultaneous pattern matching like Hyperscan and the multiple matches
-will be reported in order by end offset.
-
-.. _chruntime:
-
-*********************
-Scanning for Patterns
-*********************
-
-Chimera provides scan function with ``ch_scan``.
-
-================
-Handling Matches
-================
-
-``ch_scan`` will call a user-supplied callback function when a match
-is found. This function has the following signature:
-
-  .. doxygentypedef:: ch_match_event_handler
-       :outline:
-       :no-link:
-
-The *id* argument will be set to the identifier for the matching expression
-provided at compile time, and the *from* argument will be set to the
-start-offset of the match the *to* argument will be set to the end-offset
-of the match. The *captured* stores offsets of entire pattern match as well as
-captured subexpressions. The *size* will be set to the number of valid entries in
-the *captured*.
-
-The match callback function has the capability to continue or halt scanning
-by returning different values.
-
-See :c:type:`ch_match_event_handler` for more information.
-
-=======================
-Handling Runtime Errors
-=======================
-
-``ch_scan`` will call a user-supplied callback function when a runtime error
-occurs in libpcre. This function has the following signature:
-
-  .. doxygentypedef:: ch_error_event_handler
-       :outline:
-       :no-link:
-
-The *id* argument will be set to the identifier for the matching expression
-provided at compile time.
-
-The match callback function has the capability to either halt scanning or
-continue scanning for the next pattern.
-
-See :c:type:`ch_error_event_handler` for more information.
-
-=============
-Scratch Space
-=============
-
-While scanning data, Chimera needs a small amount of temporary memory to store
-on-the-fly internal data. This amount is unfortunately too large to fit on the
-stack, particularly for embedded applications, and allocating memory dynamically
-is too expensive, so a pre-allocated "scratch" space must be provided to the
-scanning functions.
-
-The function :c:func:`ch_alloc_scratch` allocates a large enough region of
-scratch space to support a given database. If the application uses multiple
-databases, only a single scratch region is necessary: in this case, calling
-:c:func:`ch_alloc_scratch` on each database (with the same ``scratch`` pointer)
-will ensure that the scratch space is large enough to support scanning against
-any of the given databases.
-
-While the Chimera library is re-entrant, the use of scratch spaces is not.
-For example, if by design it is deemed necessary to run recursive or nested
-scanning (say, from the match callback function), then an additional scratch
-space is required for that context.
-
-In the absence of recursive scanning, only one such space is required per thread
-and can (and indeed should) be allocated before data scanning is to commence.
-
-In a scenario where a set of expressions are compiled by a single "main"
-thread and data will be scanned by multiple "worker" threads, the convenience
-function :c:func:`ch_clone_scratch` allows multiple copies of an existing
-scratch space to be made for each thread (rather than forcing the caller to pass
-all the compiled databases through :c:func:`ch_alloc_scratch` multiple times).
-
-For example:
-
-.. code-block:: c
-
-    ch_error_t err;
-    ch_scratch_t *scratch_prototype = NULL;
-    err = ch_alloc_scratch(db, &scratch_prototype);
-    if (err != CH_SUCCESS) {
-        printf("ch_alloc_scratch failed!");
-        exit(1);
-    }
-
-    ch_scratch_t *scratch_thread1 = NULL;
-    ch_scratch_t *scratch_thread2 = NULL;
-
-    err = ch_clone_scratch(scratch_prototype, &scratch_thread1);
-    if (err != CH_SUCCESS) {
-        printf("ch_clone_scratch failed!");
-        exit(1);
-    }
-    err = ch_clone_scratch(scratch_prototype, &scratch_thread2);
-    if (err != CH_SUCCESS) {
-        printf("ch_clone_scratch failed!");
-        exit(1);
-    }
-
-    ch_free_scratch(scratch_prototype);
-
-    /* Now two threads can both scan against database db,
-       each with its own scratch space. */
-
-
-=================
-Custom Allocators
-=================
-
-By default, structures used by Chimera at runtime (scratch space, etc) are
-allocated with the default system allocators, usually
-``malloc()`` and ``free()``.
-
-The Chimera API provides a facility for changing this behaviour to support
-applications that use custom memory allocators.
-
-These functions are:
-
-- :c:func:`ch_set_database_allocator`, which sets the allocate and free functions
-  used for compiled pattern databases.
-- :c:func:`ch_set_scratch_allocator`, which sets the allocate and free
-  functions used for scratch space.
-- :c:func:`ch_set_misc_allocator`, which sets the allocate and free functions
-  used for miscellaneous data, such as compile error structures and
-  informational strings.
-
-The :c:func:`ch_set_allocator` function can be used to set all of the custom
-allocators to the same allocate/free pair.
-
-
-************************
-API Reference: Constants
-************************
-
-===========
-Error Codes
-===========
-
-.. doxygengroup:: CH_ERROR
-   :content-only:
-   :no-link:
-
-=============
-Pattern flags
-=============
-
-.. doxygengroup:: CH_PATTERN_FLAG
-   :content-only:
-   :no-link:
-
-==================
-Compile mode flags
-==================
-
-.. doxygengroup:: CH_MODE_FLAG
-   :content-only:
-   :no-link:
-
-
-********************
-API Reference: Files
-********************
-
-==========
-File: ch.h
-==========
-
-.. doxygenfile:: ch.h
-
-=================
-File: ch_common.h
-=================
-
-.. doxygenfile:: ch_common.h
-
-==================
-File: ch_compile.h
-==================
-
-.. doxygenfile:: ch_compile.h
-
-==================
-File: ch_runtime.h
-==================
-
-.. doxygenfile:: ch_runtime.h

doc/dev-reference/compilation.rst

@@ -1,633 +0,0 @@
-.. include:: <isonum.txt>
-.. _compilation:
-
-##################
-Compiling Patterns
-##################
-
-*******************
-Building a Database
-*******************
-
-The Hyperscan compiler API accepts regular expressions and converts them into a
-compiled pattern database that can then be used to scan data.
-
-The API provides three functions that compile regular expressions into
-databases:
-
-#. :c:func:`hs_compile`: compiles a single expression into a pattern database.
-
-#. :c:func:`hs_compile_multi`: compiles an array of expressions into a pattern
-   database. All of the supplied patterns will be scanned for concurrently at
-   scan time, with user-supplied identifiers returned when they match.
-
-#. :c:func:`hs_compile_ext_multi`: compiles an array of expressions as above,
-   but allows :ref:`extparam` to be specified for each expression.
-
-Compilation allows the Hyperscan library to analyze the given pattern(s) and
-pre-determine how to scan for these patterns in an optimized fashion that would
-be far too expensive to compute at run-time.
-
-When compiling expressions, a decision needs to be made whether the resulting
-compiled patterns are to be used in a streaming, block or vectored mode:
-
-- **Streaming mode**: the target data to be scanned is a continuous stream, not
-  all of which is available at once; blocks of data are scanned in sequence and
-  matches may span multiple blocks in a stream. In streaming mode, each stream
-  requires a block of memory to store its state between scan calls.
-
-- **Block mode**: the target data is a discrete, contiguous block which can be
-  scanned in one call and does not require state to be retained.
-
-- **Vectored mode**: the target data consists of a list of non-contiguous
-  blocks that are available all at once. As for block mode, no retention of
-  state is required.
-
-To compile patterns to be used in streaming mode, the ``mode`` parameter of
-:c:func:`hs_compile` must be set to :c:member:`HS_MODE_STREAM`; similarly,
-block mode requires the use of :c:member:`HS_MODE_BLOCK` and vectored mode
-requires the use of :c:member:`HS_MODE_VECTORED`. A pattern database compiled
-for one mode (streaming, block or vectored) can only be used in that mode. The
-version of Hyperscan used to produce a compiled pattern database must match the
-version of Hyperscan used to scan with it.
-
-Hyperscan provides support for targeting a database at a particular CPU
-platform; see :ref:`instr_specialization` for details.
-
-=====================
-Compile Pure Literals
-=====================
-
-Pure literal is a special case of regular expression. A character sequence is
-regarded as a pure literal if and only if each character is read and
-interpreted independently. No syntax association happens between any adjacent
-characters.
-
-For example, given an expression written as :regexp:`/bc?/`. We could say it is
-a regular expression, with the meaning that character ``b`` followed by nothing
-or by one character ``c``. On the other view, we could also say it is a pure
-literal expression, with the meaning that this is a character sequence of 3-byte
-length, containing characters ``b``, ``c`` and ``?``. In regular case, the
-question mark character ``?`` has a particular syntax role called 0-1 quantifier,
-which has a syntax association with the character ahead of it. Similar
-characters exist in regular grammar like ``[``, ``]``, ``(``, ``)``, ``{``,
-``}``, ``-``, ``*``, ``+``, ``\``, ``|``, ``/``, ``:``, ``^``, ``.``, ``$``.
-While in pure literal case, all these meta characters lost extra meanings
-expect for that they are just common ASCII codes.
-
-Hyperscan is initially designed to process common regular expressions. It is
-hence embedded with a complex parser to do comprehensive regular grammar
-interpretation. Particularly, the identification of above meta characters is the
-basic step for the interpretation of far more complex regular grammars.
-
-However in real cases, patterns may not always be regular expressions. They
-could just be pure literals. Problem will come if the pure literals contain
-regular meta characters. Supposing fed directly into traditional Hyperscan
-compile API, all these meta characters will be interpreted in predefined ways,
-which is unnecessary and the result is totally out of expectation. To avoid
-such misunderstanding by traditional API, users have to preprocess these
-literal patterns by converting the meta characters into some other formats:
-either by adding a backslash ``\`` before certain meta characters, or by
-converting all the characters into a hexadecimal representation.
-
-In ``v5.2.0``, Hyperscan introduces 2 new compile APIs for pure literal patterns:
-
-#. :c:func:`hs_compile_lit`: compiles a single pure literal into a pattern
-   database.
-
-#. :c:func:`hs_compile_lit_multi`: compiles an array of pure literals into a
-   pattern database. All of the supplied patterns will be scanned for
-   concurrently at scan time, with user-supplied identifiers returned when they
-   match.
-
-These 2 APIs are designed for use cases where all patterns contained in the
-target rule set are pure literals. Users can pass the initial pure literal
-content directly into these APIs without worrying about writing regular meta
-characters in their patterns. No preprocessing work is needed any more.
-
-For new APIs, the ``length`` of each literal pattern is a newly added parameter.
-Hyperscan needs to locate the end position of the input expression via clearly
-knowing each literal's length, not by simply identifying character ``\0`` of a
-string.
-
-Supported flags: :c:member:`HS_FLAG_CASELESS`, :c:member:`HS_FLAG_SINGLEMATCH`,
-:c:member:`HS_FLAG_SOM_LEFTMOST`.
-
-.. note:: We don't support literal compilation API with :ref:`extparam`. And
-          for runtime implementation, traditional runtime APIs can still be
-          used to match pure literal patterns.
-
-.. note:: If the target rule set contains at least one regular expression,
-          please use traditional compile APIs :c:func:`hs_compile`,
-          :c:func:`hs_compile_multi` and :c:func:`hs_compile_ext_multi`.
-          The new literal APIs introduced here are designed for rule sets
-          containing only pure literal expressions.
-
-***************
-Pattern Support
-***************
-
-Hyperscan supports the pattern syntax used by the PCRE library ("libpcre"),
-described at <http://www.pcre.org/>. However, not all constructs available in
-libpcre are supported. The use of unsupported constructs will result in
-compilation errors.
-
-The version of PCRE used to validate Hyperscan's interpretation of this syntax
-is 8.41 or above.
-
-====================
-Supported Constructs
-====================
-
-The following regex constructs are supported by Hyperscan:
-
-* Literal characters and strings, with all libpcre quoting and character
-  escapes.
-
-* Character classes such as :regexp:`.` (dot), :regexp:`[abc]`, and
-  :regexp:`[^abc]`, as well as the predefined character classes :regexp:`\\s`,
-  :regexp:`\\d`, :regexp:`\\w`, :regexp:`\\v`, and :regexp:`\\h` and their
-  negated counterparts (:regexp:`\\S`, :regexp:`\\D`, :regexp:`\\W`,
-  :regexp:`\\V`, and :regexp:`\\H`).
-
-* The POSIX named character classes :regexp:`[[:xxx:]]` and negated named
-  character classes :regexp:`[[:^xxx:]]`.
-
-* Unicode character properties, such as :regexp:`\\p{L}`, :regexp:`\\P{Sc}`,
-  :regexp:`\\p{Greek}`.
-
-* Quantifiers:
-
-  * Quantifiers such as :regexp:`?`, :regexp:`*` and :regexp:`+` are supported
-    when applied to arbitrary supported sub-expressions.
-
-  * Bounded repeat qualifiers such as :regexp:`{n}`, :regexp:`{m,n}`,
-    :regexp:`{n,}` are supported with limitations.
-
-    * For arbitrary repeated sub-patterns: *n* and *m* should be either small
-      or infinite, e.g. :regexp:`(a|b){4}`, :regexp:`(ab?c?d){4,10}` or
-      :regexp:`(ab(cd)*){6,}`.
-
-    * For single-character width sub-patterns such as :regexp:`[^\\a]` or
-      :regexp:`.` or :regexp:`x`, nearly all repeat counts are supported, except
-      where repeats are extremely large (maximum bound greater than 32767).
-      Stream states may be very large for large bounded repeats, e.g.
-      :regexp:`a.{2000}b`. Note: such sub-patterns may be considerably
-      cheaper if at the beginning or end of patterns and especially if the
-      :c:member:`HS_FLAG_SINGLEMATCH` flag is on for that pattern.
-
-  * Lazy modifiers (:regexp:`?` appended to another quantifier, e.g.
-    :regexp:`\\w+?`) are supported but ignored (as Hyperscan reports all
-    matches).
-
-* Parenthesization, including the named and unnamed capturing and
-  non-capturing forms. However, capturing is ignored.
-
-* Alternation with the :regexp:`|` symbol, as in :regexp:`foo|bar`.
-
-* The anchors :regexp:`^`, :regexp:`$`, :regexp:`\\A`, :regexp:`\\Z` and
-  :regexp:`\\z`.
-
-* Option modifiers:
-
-  These allow behaviour to be switched on (with :regexp:`(?<option>)`) and off
-  (with :regexp:`(?-<option>)`) for a sub-pattern. The supported options are:
-
-    * :regexp:`i`: Case-insensitive matching, as per
-      :c:member:`HS_FLAG_CASELESS`.
-    * :regexp:`m`: Multi-line matching, as per :c:member:`HS_FLAG_MULTILINE`.
-    * :regexp:`s`: Interpret ``.`` as "any character", as per
-      :c:member:`HS_FLAG_DOTALL`.
-    * :regexp:`x`: Extended syntax, which will ignore most whitespace in the
-      pattern for compatibility with libpcre's ``PCRE_EXTENDED`` option.
-
-  For example, the expression :regexp:`foo(?i)bar(?-i)baz` will switch on
-  case-insensitive matching *only* for the ``bar`` portion of the match.
-
-* The :regexp:`\\b` and :regexp:`\\B` zero-width assertions (word boundary and
-  'not word boundary', respectively).
-
-* Comments in :regexp:`(?# comment)` syntax.
-
-* The :regexp:`(*UTF8)` and :regexp:`(*UCP)` control verbs at the beginning of a
-  pattern, used to enable UTF-8 and UCP mode.
-
-.. note:: Bounded-repeat quantifiers with large repeat counts of arbitrary
-   expressions (e.g. :regexp:`([a-z]|bc*d|xy?z){1000,5000}`) will result in a
-   "Pattern too large" error at pattern compile time.
-
-.. note:: At this time, not all patterns can be successfully compiled with the
-  :c:member:`HS_FLAG_SOM_LEFTMOST` flag, which enables per-pattern support for
-  :ref:`som`. The patterns that support this flag are a subset of patterns that
-  can be successfully compiled with Hyperscan; notably, many bounded repeat
-  forms that can be compiled with Hyperscan without the Start of Match flag
-  enabled cannot be compiled with the flag enabled.
-
-======================
-Unsupported Constructs
-======================
-
-The following regex constructs are not supported by Hyperscan:
-
-* Backreferences and capturing sub-expressions.
-* Arbitrary zero-width assertions.
-* Subroutine references and recursive patterns.
-* Conditional patterns.
-* Backtracking control verbs.
-* The :regexp:`\\C` "single-byte" directive (which breaks UTF-8 sequences).
-* The :regexp:`\\R` newline match.
-* The :regexp:`\\K` start of match reset directive.
-* Callouts and embedded code.
-* Atomic grouping and possessive quantifiers.
-
-.. _semantics:
-
-*********
-Semantics
-*********
-
-While Hyperscan follows libpcre syntax, it provides different semantics. The
-major departures from libpcre semantics are motivated by the requirements of
-streaming and multiple simultaneous pattern matching.
-
-The major departures from libpcre semantics are:
-
-#. **Multiple pattern matching**: Hyperscan allows matches to be reported for
-   several patterns simultaneously. This is not equivalent to separating the
-   patterns by :regexp:`|` in libpcre, which evaluates alternations
-   left-to-right.
-
-#. **Lack of ordering**: the multiple matches that Hyperscan produces are not
-   guaranteed to be ordered, although they will always fall within the bounds of
-   the current scan.
-
-#. **End offsets only**: Hyperscan's default behaviour is only to report the end
-   offset of a match. Reporting of the start offset can be enabled with
-   per-expression flags at pattern compile time. See :ref:`som` for details.
-
-#. **"All matches" reported**: scanning :regexp:`/foo.*bar/` against
-   ``fooxyzbarbar`` will return two matches from Hyperscan -- at the points
-   corresponding to the ends of ``fooxyzbar`` and ``fooxyzbarbar``. In contrast,
-   libpcre semantics by default would report only one match at ``fooxyzbarbar``
-   (greedy semantics) or, if non-greedy semantics were switched on, one match at
-   ``fooxyzbar``. This means that switching between greedy and non-greedy
-   semantics is a no-op in Hyperscan.
-
-To support libpcre quantifier semantics while accurately reporting streaming
-matches at the time they occur is impossible. For example, consider the pattern
-above, :regexp:`/foo.*bar/`, in streaming mode, against the following
-stream (three blocks scanned in sequence):
-
-    =============   =======     ========
-    block 1         block 2     block 3
-    =============   =======     ========
-    ``fooxyzbar``   ``baz``     ``qbar``
-    =============   =======     ========
-
-Since the :regexp:`.*` repeat in the pattern is a *greedy* repeat in libpcre, it
-must match as much as possible without causing the rest of the pattern to fail.
-However, in streaming mode, this would require knowledge of data in the stream
-beyond the current block being scanned.
-
-In this example, the match at offset 9 in the first block is only the correct
-match (under libpcre semantics) if there is no ``bar`` in a subsequent block --
-as in block 3 -- which would constitute a better match for the pattern.
-
-.. _som:
-
-==============
-Start of Match
-==============
-
-In standard operation, Hyperscan will only provide the end offset of a match
-when the match callback is called. If the :c:member:`HS_FLAG_SOM_LEFTMOST` flag
-is specified for a particular pattern, then the same set of matches is
-returned, but each match will also provide the leftmost possible start offset
-corresponding to its end offset.
-
-Using the SOM flag entails a number of trade-offs and limitations:
-
-* Reduced pattern support: For many patterns, tracking SOM is complex and can
-  result in Hyperscan failing to compile a pattern with a "Pattern too
-  large" error, even if the pattern is supported in normal operation.
-* Increased stream state: At scan time, state space is required to track
-  potential SOM offsets, and this must be stored in persistent stream state in
-  streaming mode. Accordingly, SOM will generally increase the stream state
-  required to match a pattern.
-* Performance overhead: Similarly, there is generally a performance cost
-  associated with tracking SOM.
-* Incompatible features: Some other Hyperscan pattern flags (such as
-  :c:member:`HS_FLAG_SINGLEMATCH` and :c:member:`HS_FLAG_PREFILTER`) can not be
-  used in combination with SOM. Specifying them together with
-  :c:member:`HS_FLAG_SOM_LEFTMOST` will result in a compilation error.
-
-In streaming mode, the amount of precision delivered by SOM can be controlled
-with the SOM horizon flags. These instruct Hyperscan to deliver accurate SOM
-information within a certain distance of the end offset, and return a special
-start offset of :c:member:`HS_OFFSET_PAST_HORIZON` otherwise. Specifying a
-small or medium SOM horizon will usually reduce the stream state required for a
-given database.
-
-.. note:: In streaming mode, the start offset returned for a match may refer to
-   a point in the stream *before* the current block being scanned. Hyperscan
-   provides no facility for accessing earlier blocks; if the calling application
-   needs to inspect historical data, then it must store it itself.
-
-.. _extparam:
-
-===================
-Extended Parameters
-===================
-
-In some circumstances, more control over the matching behaviour of a pattern is
-required than can be specified easily using regular expression syntax. For
-these scenarios, Hyperscan provides the :c:func:`hs_compile_ext_multi` function
-that allows a set of "extended parameters" to be set on a per-pattern basis.
-
-Extended parameters are specified using an :c:type:`hs_expr_ext_t` structure,
-which provides the following fields:
-
-* ``flags``: Flags governing which of the other fields in the structure are
-  used.
-* ``min_offset``: The minimum end offset in the data stream at which this
-  expression should match successfully.
-* ``max_offset``: The maximum end offset in the data stream at which this
-  expression should match successfully.
-* ``min_length``: The minimum match length (from start to end) required to
-  successfully match this expression.
-* ``edit_distance``: Match this expression within a given Levenshtein distance.
-* ``hamming_distance``: Match this expression within a given Hamming distance.
-
-These parameters either allow the set of matches produced by a pattern to be
-constrained at compile time (rather than relying on the application to process
-unwanted matches at runtime), or allow matching a pattern approximately (within
-a given edit distance) to produce more matches.
-
-For example, the pattern :regexp:`/foo.*bar/` when given a ``min_offset`` of 10
-and a ``max_offset`` of 15 will not produce matches when scanned against
-``foobar`` or ``foo0123456789bar`` but will produce a match against the data
-streams ``foo0123bar`` or ``foo0123456bar``.
-
-Similarly, the pattern :regexp:`/foobar/` when given an ``edit_distance`` of 2
-will produce matches when scanned against ``foobar``, ``f00bar``, ``fooba``,
-``fobr``, ``fo_baz``, ``foooobar``, and anything else that lies within edit
-distance of 2 (as defined by Levenshtein distance).
-
-When the same pattern :regexp:`/foobar/` is given a ``hamming_distance`` of 2,
-it will produce matches when scanned against ``foobar``, ``boofar``,
-``f00bar``, and anything else with at most two characters substituted from the
-original pattern. For more details, see the :ref:`approximate_matching`
-section.
-
-=================
-Prefiltering Mode
-=================
-
-Hyperscan provides a per-pattern flag, :c:member:`HS_FLAG_PREFILTER`, which can
-be used to implement a prefilter for a pattern than Hyperscan would not
-ordinarily support.
-
-This flag instructs Hyperscan to compile an "approximate" version of this
-pattern for use in a prefiltering application, even if Hyperscan does not
-support the pattern in normal operation.
-
-The set of matches returned when this flag is used is guaranteed to be a
-superset of the matches specified by the non-prefiltering expression.
-
-If the pattern contains pattern constructs not supported by Hyperscan (such as
-zero-width assertions, back-references or conditional references) these
-constructs will be replaced internally with broader constructs that may match
-more often.
-
-For example, the pattern :regexp:`/(\\w+) again \\1/` contains the
-back-reference :regexp:`\\1`. In prefiltering mode, this pattern might be
-approximated by having its back-reference replaced with its referent, forming
-:regexp:`/\\w+ again \\w+/`.
-
-Furthermore, in prefiltering mode Hyperscan may simplify a pattern that would
-otherwise return a "Pattern too large" error at compile time, or for performance
-reasons (subject to the matching guarantee above).
-
-It is generally expected that the application will subsequently confirm
-prefilter matches with another regular expression matcher that can provide exact
-matches for the pattern.
-
-.. note:: The use of this flag in combination with Start of Match mode (using
-   the :c:member:`HS_FLAG_SOM_LEFTMOST` flag) is not currently supported and
-   will result in a pattern compilation error.
-
-.. _instr_specialization:
-
-******************************
-Instruction Set Specialization
-******************************
-
-Hyperscan is able to make use of several modern instruction set features found
-on x86 processors to provide improvements in scanning performance.
-
-Some of these features are selected when the library is built; for example,
-Hyperscan will use the native ``POPCNT`` instruction on processors where it is
-available and the library has been optimized for the host architecture.
-
-.. note:: By default, the Hyperscan runtime is built with the ``-march=native``
-   compiler flag and (where possible) will make use of all instructions known by
-   the host's C compiler.
-
-To use some instruction set features, however, Hyperscan must build a
-specialized database to support them. This means that the target platform must
-be specified at pattern compile time.
-
-The Hyperscan compiler API functions all accept an optional
-:c:type:`hs_platform_info_t` argument, which describes the target platform
-for the database to be built. If this argument is NULL, the database will be
-targeted at the current host platform.
-
-The :c:type:`hs_platform_info_t` structure has two fields:
-
-#. ``tune``: This allows the application to specify information about the target
-   platform which may be used to guide the optimisation process of the compile.
-   Use of this field does not limit the processors that the resulting database
-   can run on, but may impact the performance of the resulting database.
-
-#. ``cpu_features``: This allows the application to specify a mask of CPU
-   features that may be used on the target platform. For example,
-   :c:member:`HS_CPU_FEATURES_AVX2` can be specified for Intel\ |reg| Advanced
-   Vector Extensions 2 (Intel\ |reg| AVX2) instruction set support. If a flag
-   for a particular CPU feature is specified, the database will not be usable on
-   a CPU without that feature.
-
-An :c:type:`hs_platform_info_t` structure targeted at the current host can be
-built with the :c:func:`hs_populate_platform` function.
-
-See :ref:`api_constants` for the full list of CPU tuning and feature flags.
-
-.. _approximate_matching:
-
-********************
-Approximate matching
-********************
-
-Hyperscan provides an experimental approximate matching mode, which will match
-patterns within a given edit distance. The exact matching behavior is defined as
-follows:
-
-#. **Edit distance** is defined as Levenshtein distance. That is, there are
-   three possible edit types considered: insertion, removal and substitution.
-   A more formal description can be found on
-   `Wikipedia <https://en.wikipedia.org/wiki/Levenshtein_distance>`__.
-
-#. **Hamming distance** is the number of positions by which two strings of
-   equal length differ. That is, it is the number of substitutions required to
-   convert one string to the other. There are no insertions or removals when
-   approximate matching using a Hamming distance. A more formal description can
-   be found on
-   `Wikipedia <https://en.wikipedia.org/wiki/Hamming_distance>`__.
-
-#. **Approximate matching** will match all *corpora* within a given edit or
-   Hamming distance. That is, given a pattern, approximate matching will match
-   anything that can be edited to arrive at a corpus that exactly matches the
-   original pattern.
-
-#. **Matching semantics** are exactly the same as described in :ref:`semantics`.
-
-Here are a few examples of approximate matching:
-
-* Pattern :regexp:`/foo/` can match ``foo`` when using regular Hyperscan
-  matching behavior. With approximate matching within edit distance 2, the
-  pattern will produce matches when scanned against ``foo``, ``foooo``, ``f00``,
-  ``f``, and anything else that lies within edit distance 2 of matching corpora
-  for the original pattern (``foo`` in this case).
-
-* Pattern :regexp:`/foo(bar)+/` with edit distance 1 will match ``foobarbar``,
-  ``foobarb0r``, ``fooarbar``, ``foobarba``, ``f0obarbar``, ``fobarbar`` and
-  anything else that lies within edit distance 1 of matching corpora for the
-  original pattern (``foobarbar`` in this case).
-
-* Pattern :regexp:`/foob?ar/` with edit distance 2 will match ``fooar``,
-  ``foo``, ``fabar``, ``oar`` and anything else that lies within edit distance 2
-  of matching corpora for the original pattern (``fooar`` in this case).
-
-Currently, there are trade-offs and limitations that come with approximate
-matching support. Here they are, in a nutshell:
-
-* Reduced pattern support:
-
-  * For many patterns, approximate matching is complex and can result in
-    Hyperscan failing to compile a pattern with a "Pattern too large" error,
-    even if the pattern is supported in normal operation.
-  * Additionally, some patterns cannot be approximately matched because they
-    reduce to so-called "vacuous" patterns (patterns that match everything). For
-    example, pattern :regexp:`/foo/` with edit distance 3, if implemented,
-    would reduce to matching zero-length buffers. Such patterns will result in a
-    "Pattern cannot be approximately matched" compile error. Approximate
-    matching within a Hamming distance does not remove symbols, so will not
-    reduce to a vacuous pattern.
-  * Finally, due to the inherent complexities of defining matching behavior,
-    approximate matching implements a reduced subset of regular expression
-    syntax. Approximate matching does not support UTF-8 (and other
-    multibyte character encodings), and word boundaries (that is, ``\b``, ``\B``
-    and other equivalent constructs). Patterns containing unsupported constructs
-    will result in "Pattern cannot be approximately matched" compile error.
-  * When using approximate matching in conjunction with SOM, all of the
-    restrictions of SOM also apply. See :ref:`som` for more
-    details.
-* Increased stream state/byte code size requirements: due to approximate
-  matching byte code being inherently larger and more complex than exact
-  matching, the corresponding requirements also increase.
-* Performance overhead: similarly, there is generally a performance cost
-  associated with approximate matching, both due to increased matching
-  complexity, and due to the fact that it will produce more matches.
-
-Approximate matching is always disabled by default, and can be enabled on a
-per-pattern basis by using an extended parameter described in :ref:`extparam`.
-
-.. _logical_combinations:
-
-********************
-Logical Combinations
-********************
-
-For situations when a user requires behaviour that depends on the presence or
-absence of matches from groups of patterns, Hyperscan provides support for the
-logical combination of patterns in a given pattern set, with three operators:
-``NOT``, ``AND`` and ``OR``.
-
-The logical value of such a combination is based on each expression's matching
-status at a given offset. The matching status of any expression has a boolean
-value: *false* if the expression has not yet matched or *true* if the expression
-has already matched. In particular, the value of a ``NOT`` operation at a given
-offset is *true* if the expression it refers to is *false* at this offset.
-
-For example, ``NOT 101`` means that expression 101 has not yet matched at this
-offset.
-
-A logical combination is passed to Hyperscan at compile time as an expression.
-This combination expression will raise matches at every offset where one of its
-sub-expressions matches and the logical value of the whole expression is *true*.
-
-To illustrate, here is an example combination expression: ::
-
-    ((301 OR 302) AND 303) AND (304 OR NOT 305)
-
-If expression 301 matches at offset 10, the logical value of 301 is *true*
-while the other patterns' values are *false*. Hence, the whole combination's value is
-*false*.
-
-Then expression 303 matches at offset 20. Now the values of 301 and 303 are
-*true* while the other patterns' values are still *false*. In this case, the
-combination's value is *true*, so the combination expression raises a match at
-offset 20.
-
-Finally, expression 305 has matches at offset 30. Now the values of 301, 303 and 305
-are *true* while the other patterns' values are still *false*. In this case, the
-combination's value is *false* and no match is raised.
-
-**Using Logical Combinations**
-
-In logical combination syntax, an expression is written as infix notation, it
-consists of operands, operators and parentheses. The operands are expression
-IDs, and operators are ``!`` (NOT), ``&`` (AND) or ``|`` (OR). For example, the
-combination described in the previous section would be written as: ::
-
-    ((301 | 302) & 303) & (304 | !305)
-
-In a logical combination expression:
-
- * The priority of operators are ``!`` > ``&`` > ``|``. For example:
-    - ``A&B|C`` is treated as ``(A&B)|C``,
-    - ``A|B&C`` is treated as ``A|(B&C)``,
-    - ``A&!B`` is treated as ``A&(!B)``.
- * Extra parentheses are allowed. For example:
-    - ``(A)&!(B)`` is the same as ``A&!B``,
-    - ``(A&B)|C`` is the same as ``A&B|C``.
- * Whitespace is ignored.
-
-To use a logical combination expression, it must be passed to one of the
-Hyperscan compile functions (:c:func:`hs_compile_multi`,
-:c:func:`hs_compile_ext_multi`) along with the :c:member:`HS_FLAG_COMBINATION` flag,
-which identifies the pattern as a logical combination expression. The patterns
-referred to in the logical combination expression must be compiled together in
-the same pattern set as the combination expression.
-
-When an expression has the :c:member:`HS_FLAG_COMBINATION` flag set, it ignores
-all other flags except the :c:member:`HS_FLAG_SINGLEMATCH` flag and the
-:c:member:`HS_FLAG_QUIET` flag.
-
-Hyperscan will accept logical combination expressions at compile time that
-evaluate to *true* when no patterns have matched, and report the match for
-combination at end of data if no patterns have matched; for example: ::
-
-    !101
-    !101|102
-    !101&!102
-    !(101&102)
-
-Patterns that are referred to as operands within a logical combination (for
-example, 301 through 305 in the examples above) may also use the
-:c:member:`HS_FLAG_QUIET` flag to silence the reporting of individual matches
-for those patterns. In the absence of this flag, all matches (for
-both individual patterns and their logical combinations) will be reported.
-
-When an expression has both the :c:member:`HS_FLAG_COMBINATION` flag and the
-:c:member:`HS_FLAG_QUIET` flag set, no matches for this logical combination
-will be reported.

doc/dev-reference/conf.py.in

@@ -1,275 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Hyperscan documentation build configuration file, created by
-# sphinx-quickstart on Tue Sep 29 15:59:19 2015.
-#
-# This file is execfile()d with the current directory set to its
-# containing dir.
-#
-# Note that not all possible configuration values are present in this
-# autogenerated file.
-#
-# All configuration values have a default; values that are commented out
-# serve to show the default.
-
-import sys
-import os
-
-# If extensions (or modules to document with autodoc) are in another directory,
-# add these directories to sys.path here. If the directory is relative to the
-# documentation root, use os.path.abspath to make it absolute, like shown here.
-#sys.path.insert(0, os.path.abspath('.'))
-
-# -- General configuration ------------------------------------------------
-
-# If your documentation needs a minimal Sphinx version, state it here.
-#needs_sphinx = '1.0'
-
-# Add any Sphinx extension module names here, as strings. They can be
-# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
-# ones.
-extensions = ['breathe']
-
-# Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
-
-# The suffix of source filenames.
-source_suffix = '.rst'
-
-# The encoding of source files.
-#source_encoding = 'utf-8-sig'
-
-# The master toctree document.
-master_doc = 'index'
-
-# General information about the project.
-project = u'Hyperscan'
-copyright = u'2015-2018, Intel Corporation'
-
-# The version info for the project you're documenting, acts as replacement for
-# |version| and |release|, also used in various other places throughout the
-# built documents.
-#
-# The short X.Y version.
-version = '@HS_MAJOR_VERSION@.@HS_MINOR_VERSION@'
-# The full version, including alpha/beta/rc tags.
-release = '@HS_VERSION@'
-
-# The language for content autogenerated by Sphinx. Refer to documentation
-# for a list of supported languages.
-#language = None
-
-# There are two options for replacing |today|: either, you set today to some
-# non-false value, then it is used:
-#today = ''
-# Else, today_fmt is used as the format for a strftime call.
-#today_fmt = '%B %d, %Y'
-
-# List of patterns, relative to source directory, that match files and
-# directories to ignore when looking for source files.
-exclude_patterns = ['_build']
-
-# The reST default role (used for this markup: `text`) to use for all
-# documents.
-#default_role = None
-
-# If true, '()' will be appended to :func: etc. cross-reference text.
-#add_function_parentheses = True
-
-# If true, the current module name will be prepended to all description
-# unit titles (such as .. function::).
-#add_module_names = True
-
-# If true, sectionauthor and moduleauthor directives will be shown in the
-# output. They are ignored by default.
-#show_authors = False
-
-# The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'sphinx'
-
-# A list of ignored prefixes for module index sorting.
-#modindex_common_prefix = []
-
-# If true, keep warnings as "system message" paragraphs in the built documents.
-#keep_warnings = False
-
-
-# -- Options for HTML output ----------------------------------------------
-
-# The theme to use for HTML and HTML Help pages.  See the documentation for
-# a list of builtin themes.
-html_theme = 'alabaster'
-
-# Theme options are theme-specific and customize the look and feel of a theme
-# further.  For a list of options available for each theme, see the
-# documentation.
-html_theme_options = {
-    # Change some style colors; these are used for admonitions
-    'pink_1' : '#e0f8ff',
-    'pink_2' : '#e0f8ff'
-}
-
-# Add any paths that contain custom themes here, relative to this directory.
-#html_theme_path = []
-
-# The name for this set of Sphinx documents.  If None, it defaults to
-# "<project> v<release> documentation".
-#html_title = None
-
-# A shorter title for the navigation bar.  Default is the same as html_title.
-#html_short_title = None
-
-# The name of an image file (relative to this directory) to place at the top
-# of the sidebar.
-#html_logo = None
-
-# The name of an image file (within the static path) to use as favicon of the
-# docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
-# pixels large.
-#html_favicon = None
-
-# Add any paths that contain custom static files (such as style sheets) here,
-# relative to this directory. They are copied after the builtin static files,
-# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['@CMAKE_CURRENT_SOURCE_DIR@/_static']
-
-# Add any extra paths that contain custom files (such as robots.txt or
-# .htaccess) here, relative to this directory. These files are copied
-# directly to the root of the documentation.
-#html_extra_path = []
-
-# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
-# using the given strftime format.
-#html_last_updated_fmt = '%b %d, %Y'
-
-# If true, SmartyPants will be used to convert quotes and dashes to
-# typographically correct entities.
-#html_use_smartypants = True
-
-# Custom sidebar templates, maps document names to template names.
-html_sidebars = {
-    '**': ['globaltoc.html', 'searchbox.html']
-}
-
-# Additional templates that should be rendered to pages, maps page names to
-# template names.
-#html_additional_pages = {}
-
-# If false, no module index is generated.
-#html_domain_indices = True
-
-# If false, no index is generated.
-#html_use_index = True
-
-# If true, the index is split into individual pages for each letter.
-#html_split_index = False
-
-# If true, links to the reST sources are added to the pages.
-html_show_sourcelink = False
-
-# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
-#html_show_sphinx = True
-
-# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
-#html_show_copyright = True
-
-# If true, an OpenSearch description file will be output, and all pages will
-# contain a <link> tag referring to it.  The value of this option must be the
-# base URL from which the finished HTML is served.
-#html_use_opensearch = ''
-
-# This is the file name suffix for HTML files (e.g. ".xhtml").
-#html_file_suffix = None
-
-# Output file base name for HTML help builder.
-htmlhelp_basename = 'Hyperscandoc'
-
-
-# -- Options for LaTeX output ---------------------------------------------
-
-latex_elements = {
-# The paper size ('letterpaper' or 'a4paper').
-#'papersize': 'letterpaper',
-
-# The font size ('10pt', '11pt' or '12pt').
-#'pointsize': '10pt',
-
-# Additional stuff for the LaTeX preamble.
-#'preamble': '',
-}
-
-# Grouping the document tree into LaTeX files. List of tuples
-# (source start file, target name, title,
-#  author, documentclass [howto, manual, or own class]).
-latex_documents = [
-  ('index', 'Hyperscan.tex', u'Hyperscan Documentation',
-   u'Intel Corporation', 'manual'),
-]
-
-# The name of an image file (relative to this directory) to place at the top of
-# the title page.
-#latex_logo = None
-
-# For "manual" documents, if this is true, then toplevel headings are parts,
-# not chapters.
-#latex_use_parts = False
-
-# If true, show page references after internal links.
-#latex_show_pagerefs = False
-
-# If true, show URL addresses after external links.
-#latex_show_urls = False
-
-# Documents to append as an appendix to all manuals.
-#latex_appendices = []
-
-# If false, no module index is generated.
-#latex_domain_indices = True
-
-
-# -- Options for manual page output ---------------------------------------
-
-# One entry per manual page. List of tuples
-# (source start file, name, description, authors, manual section).
-man_pages = [
-    ('index', 'hyperscan', u'Hyperscan Documentation',
-     [u'Intel Corporation'], 1)
-]
-
-# If true, show URL addresses after external links.
-#man_show_urls = False
-
-
-# -- Options for Texinfo output -------------------------------------------
-
-# Grouping the document tree into Texinfo files. List of tuples
-# (source start file, target name, title, author,
-#  dir menu entry, description, category)
-texinfo_documents = [
-  ('index', 'Hyperscan', u'Hyperscan Documentation',
-   u'Intel Corporation', 'Hyperscan', 'High-performance regular expression matcher.',
-   'Miscellaneous'),
-]
-
-# Documents to append as an appendix to all manuals.
-#texinfo_appendices = []
-
-# If false, no module index is generated.
-#texinfo_domain_indices = True
-
-# How to display URL addresses: 'footnote', 'no', or 'inline'.
-#texinfo_show_urls = 'footnote'
-
-# If true, do not generate a @detailmenu in the "Top" node's menu.
-#texinfo_no_detailmenu = False
-
-# -- Options for Breathe doxygen import -----------------------------------
-
-breathe_projects = { "hyperscan": "doxygen_xml" }
-breathe_default_project = "hyperscan"
-breathe_domain_by_extension = {"h" : "c"}
-
-# -- Add some customisation -----------------------------------------------
-
-def setup(app):
-    app.add_stylesheet("hyperscan.css") # Custom stylesheet for e.g. :regex:

doc/dev-reference/copyright.rst

@@ -1,33 +0,0 @@
-.. include:: <isonum.txt>
-
-#########
-Copyright
-#########
-
-No license (express or implied, by estoppel or otherwise) to any intellectual
-property rights is granted by this document.
-
-Intel disclaims all express and implied warranties, including without
-limitation, the implied warranties of merchantability, fitness for a particular
-purpose, and non-infringement, as well as any warranty arising from course of
-performance, course of dealing, or usage in trade.
-
-This document contains information on products, services and/or processes in
-development.  All information provided here is subject to change without
-notice. Contact your Intel representative to obtain the latest forecast,
-schedule, specifications and roadmaps.
-
-The products and services described may contain defects or errors known as
-errata which may cause deviations from published specifications. Current
-characterized errata are available on request.
-
-Copies of documents which have an order number and are referenced in this
-document, or other Intel literature, may be obtained by calling 1-800-548-4725,
-or go to: <http://www.intel.com/design/literature.htm>.
-
-Intel, and the Intel logo, are trademarks of Intel Corporation in the U.S.
-and/or other countries.
-
-\*Other names and brands may be claimed as the property of others.
-
-Copyright |copy| 2015-2018, Intel Corporation. All rights reserved.

doc/dev-reference/getting_started.rst

@@ -1,309 +0,0 @@
-.. include:: <isonum.txt>
-
-###############
-Getting Started
-###############
-
-Very Quick Start
-****************
-
-#. Clone Hyperscan ::
-
-     cd <where-you-want-hyperscan-source>
-     git clone git://github.com/intel/hyperscan
-
-#. Configure Hyperscan
-
-   Ensure that you have the correct :ref:`dependencies <software>` present,
-   and then:
-
-   ::
-
-     cd <where-you-want-to-build-hyperscan>
-     mkdir <build-dir>
-     cd <build-dir>
-     cmake [-G <generator>] [options] <hyperscan-source-path>
-
-   Known working generators:
-      * ``Unix Makefiles`` --- make-compatible makefiles (default on Linux/FreeBSD/Mac OS X)
-      * ``Ninja`` --- `Ninja <http://martine.github.io/ninja/>`_ build files.
-      * ``Visual Studio 15 2017`` --- Visual Studio projects
-
-   Generators that might work include:
-      * ``Xcode`` --- OS X Xcode projects.
-
-#. Build Hyperscan
-
-   Depending on the generator used:
-     * ``cmake --build .`` --- will build everything
-     * ``make -j<jobs>`` --- use makefiles in parallel
-     * ``ninja`` --- use Ninja build
-     * ``MsBuild.exe`` --- use Visual Studio MsBuild
-     * etc.
-
-#. Check Hyperscan
-
-   Run the Hyperscan unit tests: ::
-
-     bin/unit-hyperscan
-
-Requirements
-************
-
-.. _hardware:
-
-Hardware
-========
-
-Hyperscan will run on x86 processors in 64-bit (Intel\ |reg| 64 Architecture) and
-32-bit (IA-32 Architecture) modes.
-
-Hyperscan is a high performance software library that takes advantage of recent
-Intel architecture advances. At a minimum, support for Supplemental Streaming
-SIMD Extensions 3 (SSSE3) is required, which should be available on any modern
-x86 processor.
-
-Additionally, Hyperscan can make use of:
-
-    * Intel Streaming SIMD Extensions 4.2 (SSE4.2)
-    * the POPCNT instruction
-    * Bit Manipulation Instructions (BMI, BMI2)
-    * Intel Advanced Vector Extensions 2 (Intel AVX2)
-
-if present.
-
-These can be determined at library compile time, see :ref:`target_arch`.
-
-.. _software:
-
-Software
-========
-
-As a software library, Hyperscan doesn't impose any particular runtime
-software requirements, however to build the Hyperscan library we require a
-modern C and C++ compiler -- in particular, Hyperscan requires C99 and C++11
-compiler support. The supported compilers are:
-
-    * GCC, v4.8.1 or higher
-    * Clang, v3.4 or higher (with libstdc++ or libc++)
-    * Intel C++ Compiler v15 or higher
-    * Visual C++ 2017 Build Tools
-
-Examples of operating systems that Hyperscan is known to work on include:
-
-Linux:
-
-* Ubuntu 14.04 LTS or newer
-* RedHat/CentOS 7 or newer
-
-FreeBSD:
-
-* 10.0 or newer
-
-Windows:
-
-* 8 or newer
-
-Mac OS X:
-
-* 10.8 or newer, using XCode/Clang
-
-Hyperscan *may* compile and run on other platforms, but there is no guarantee.
-We currently have experimental support for Windows using Intel C++ Compiler
-or Visual Studio 2017.
-
-In addition, the following software is required for compiling the Hyperscan library:
-
-======================================================= =========== ======================================
-Dependency                                              Version     Notes
-======================================================= =========== ======================================
-`CMake <http://www.cmake.org/>`_                        >=2.8.11
-`Ragel <http://www.colm.net/open-source/ragel/>`_       6.9
-`Python <http://www.python.org/>`_                      2.7
-`Boost <http://boost.org/>`_                            >=1.57      Boost headers required
-`Pcap <http://tcpdump.org>`_                            >=0.8       Optional: needed for example code only
-======================================================= =========== ======================================
-
-Most of these dependencies can be provided by the package manager on the build
-system (e.g. Debian/Ubuntu/RedHat packages, FreeBSD ports, etc). However,
-ensure that the correct version is present. As for Windows, in order to have
-Ragel, you may use Cygwin to build it from source.
-
-Boost Headers
--------------
-
-Compiling Hyperscan depends on a recent version of the Boost C++ header
-library. If the Boost libraries are installed on the build machine in the
-usual paths, CMake will find them. If the Boost libraries are not installed,
-the location of the Boost source tree can be specified during the CMake
-configuration step using the ``BOOST_ROOT`` variable (described below).
-
-Another alternative is to put a copy of (or a symlink to) the boost
-subdirectory in ``<hyperscan-source-path>/include/boost``.
-
-For example: for the Boost-1.59.0 release: ::
-
-    ln -s boost_1_59_0/boost <hyperscan-source-path>/include/boost
-
-As Hyperscan uses the header-only parts of Boost, it is not necessary to
-compile the Boost libraries.
-
-CMake Configuration
-===================
-
-When CMake is invoked, it generates build files using the given options.
-Options are passed to CMake in the form ``-D<variable name>=<value>``.
-Common options for CMake include:
-
-+------------------------+----------------------------------------------------+
-| Variable               | Description                                        |
-+========================+====================================================+
-| CMAKE_C_COMPILER       | C compiler to use. Default is /usr/bin/cc.         |
-+------------------------+----------------------------------------------------+
-| CMAKE_CXX_COMPILER     | C++ compiler to use. Default is /usr/bin/c++.      |
-+------------------------+----------------------------------------------------+
-| CMAKE_INSTALL_PREFIX   | Install directory for ``install`` target           |
-+------------------------+----------------------------------------------------+
-| CMAKE_BUILD_TYPE       | Define which kind of build to generate.            |
-|                        | Valid options are Debug, Release, RelWithDebInfo,  |
-|                        | and MinSizeRel. Default is RelWithDebInfo.         |
-+------------------------+----------------------------------------------------+
-| BUILD_SHARED_LIBS      | Build Hyperscan as a shared library instead of     |
-|                        | the default static library.                        |
-+------------------------+----------------------------------------------------+
-| BUILD_STATIC_AND_SHARED| Build both static and shared Hyperscan libs.       |
-|                        | Default off.                                       |
-+------------------------+----------------------------------------------------+
-| BOOST_ROOT             | Location of Boost source tree.                     |
-+------------------------+----------------------------------------------------+
-| DEBUG_OUTPUT           | Enable very verbose debug output. Default off.     |
-+------------------------+----------------------------------------------------+
-| FAT_RUNTIME            | Build the :ref:`fat runtime<fat_runtime>`. Default |
-|                        | true on Linux, not available elsewhere.            |
-+------------------------+----------------------------------------------------+
-
-For example, to generate a ``Debug`` build: ::
-
-    cd <build-dir>
-    cmake -DCMAKE_BUILD_TYPE=Debug <hyperscan-source-path>
-
-
-
-Build Type
-----------
-
-CMake determines a number of features for a build based on the Build Type.
-Hyperscan defaults to ``RelWithDebInfo``, i.e. "release with debugging
-information". This is a performance optimized build without runtime assertions
-but with debug symbols enabled.
-
-The other types of builds are:
-
- * ``Release``: as above, but without debug symbols
- * ``MinSizeRel``: a stripped release build
- * ``Debug``: used when developing Hyperscan. Includes runtime assertions
-   (which has a large impact on runtime performance), and will also enable
-   some other build features like building internal unit
-   tests.
-
-.. _target_arch:
-
-Target Architecture
--------------------
-
-Unless using the :ref:`fat runtime<fat_runtime>`, by default Hyperscan will be
-compiled to target the instruction set of the processor of the machine that
-being used for compilation. This is done via the use of ``-march=native``. The
-result of this means that a library built on one machine may not work on a
-different machine if they differ in supported instruction subsets.
-
-To override the use of ``-march=native``, set appropriate flags for the
-compiler in ``CFLAGS`` and ``CXXFLAGS`` environment variables before invoking
-CMake, or ``CMAKE_C_FLAGS`` and ``CMAKE_CXX_FLAGS`` on the CMake command line. For
-example, to set the instruction subsets up to ``SSE4.2`` using GCC 4.8: ::
-
-    cmake -DCMAKE_C_FLAGS="-march=corei7" \
-      -DCMAKE_CXX_FLAGS="-march=corei7" <hyperscan-source-path>
-
-For more information, refer to :ref:`instr_specialization`.
-
-.. _fat_runtime:
-
-Fat Runtime
------------
-
-A feature introduced in Hyperscan v4.4 is the ability for the Hyperscan
-library to dispatch the most appropriate runtime code for the host processor.
-This feature is called the "fat runtime", as a single Hyperscan library
-contains multiple copies of the runtime code for different instruction sets.
-
-.. note::
-
-    The fat runtime feature is only available on Linux. Release builds of
-    Hyperscan will default to having the fat runtime enabled where supported.
-
-When building the library with the fat runtime, the Hyperscan runtime code
-will be compiled multiple times for these different instruction sets, and
-these compiled objects are combined into one library. There are no changes to
-how user applications are built against this library.
-
-When applications are executed, the correct version of the runtime is selected
-for the machine that it is running on. This is done using a ``CPUID`` check
-for the presence of the instruction set, and then an indirect function is
-resolved so that the right version of each API function is used. There is no
-impact on function call performance, as this check and resolution is performed
-by the ELF loader once when the binary is loaded.
-
-If the Hyperscan library is used on x86 systems without ``SSSE3``, the runtime
-API functions will resolve to functions that return :c:member:`HS_ARCH_ERROR`
-instead of potentially executing illegal instructions. The API function
-:c:func:`hs_valid_platform` can be used by application writers to determine if
-the current platform is supported by Hyperscan.
-
-As of this release, the variants of the runtime that are built, and the CPU
-capability that is required, are the following:
-
-+--------------+---------------------------------+---------------------------+
-| Variant      | CPU Feature Flag(s) Required    | gcc arch flag             |
-+==============+=================================+===========================+
-| Core 2       | ``SSSE3``                       | ``-march=core2``          |
-+--------------+---------------------------------+---------------------------+
-| Core i7      | ``SSE4_2`` and ``POPCNT``       | ``-march=corei7``         |
-+--------------+---------------------------------+---------------------------+
-| AVX 2        | ``AVX2``                        | ``-march=core-avx2``      |
-+--------------+---------------------------------+---------------------------+
-| AVX 512      | ``AVX512BW`` (see note below)   | ``-march=skylake-avx512`` |
-+--------------+---------------------------------+---------------------------+
-| AVX 512 VBMI | ``AVX512VBMI`` (see note below) | ``-march=icelake-server`` |
-+--------------+---------------------------------+---------------------------+
-
-.. note::
-
-    Hyperscan v4.5 adds support for AVX-512 instructions - in particular the
-    ``AVX-512BW`` instruction set that was introduced on Intel "Skylake" Xeon
-    processors - however the AVX-512 runtime variant is **not** enabled by
-    default in fat runtime builds as not all toolchains support AVX-512
-    instruction sets. To build an AVX-512 runtime, the CMake variable
-    ``BUILD_AVX512`` must be enabled manually during configuration. For
-    example: ::
-
-        cmake -DBUILD_AVX512=on <...>
-
-    Hyperscan v5.3 adds support for AVX512VBMI instructions - in particular the
-    ``AVX512VBMI`` instruction set that was introduced on Intel "Icelake" Xeon
-    processors - however the AVX512VBMI runtime variant is **not** enabled by
-    default in fat runtime builds as not all toolchains support AVX512VBMI
-    instruction sets. To build an AVX512VBMI runtime, the CMake variable
-    ``BUILD_AVX512VBMI`` must be enabled manually during configuration. For
-    example: ::
-
-        cmake -DBUILD_AVX512VBMI=on <...>
-
-As the fat runtime requires compiler, libc, and binutils support, at this time
-it will only be enabled for Linux builds where the compiler supports the
-`indirect function "ifunc" function attribute
-<https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-indirect-functions-3321>`_.
-
-This attribute should be available on all supported versions of GCC, and
-recent versions of Clang and ICC. There is currently no operating system
-support for this feature on non-Linux systems.

doc/dev-reference/index.rst

@@ -1,23 +0,0 @@
-###############################################
-Hyperscan |version| Developer's Reference Guide
-###############################################
-
--------
-|today|
--------
-
-.. toctree::
-   :maxdepth: 2
-
-   copyright
-   preface
-   intro
-   getting_started
-   compilation
-   runtime
-   serialization
-   performance
-   tools
-   api_constants
-   api_files
-   chimera

doc/dev-reference/intro.rst

@@ -1,85 +0,0 @@
-.. include:: <isonum.txt>
-.. _intro:
-
-############
-Introduction
-############
-
-Hyperscan is a software regular expression matching engine designed with
-high performance and flexibility in mind. It is implemented as a library that
-exposes a straightforward C API.
-
-The Hyperscan API itself is composed of two major components:
-
-***********
-Compilation
-***********
-
-These functions take a group of regular expressions, along with identifiers and
-option flags, and compile them into an immutable database that can be used by
-the Hyperscan scanning API. This compilation process performs considerable
-analysis and optimization work in order to build a database that will match the
-given expressions efficiently.
-
-If a pattern cannot be built into a database for any reason (such as the use of
-an unsupported expression construct, or the overflowing of a resource limit),
-an error will be returned by the pattern compiler.  
-
-Compiled databases can be serialized and relocated, so that they can be stored
-to disk or moved between hosts. They can also be targeted to particular
-platform features (for example, the use of Intel\ |reg| Advanced Vector Extensions
-2 (Intel\ |reg| AVX2) instructions).
-
-See :ref:`compilation` for more detail.
-
-********
-Scanning
-********
-
-Once a Hyperscan database has been created, it can be used to scan data in
-memory. Hyperscan provides several scanning modes, depending on whether the
-data to be scanned is available as a single contiguous block, whether it is
-distributed amongst several blocks in memory at the same time, or whether it is
-to be scanned as a sequence of blocks in a stream.
-
-Matches are delivered to the application via a user-supplied callback function
-that is called synchronously for each match.
-
-For a given database, Hyperscan provides several guarantees:
-
-* No memory allocations occur at runtime with the exception of two
-  fixed-size allocations, both of which should be done ahead of time for
-  performance-critical applications:
-
-  - **Scratch space**: temporary memory used for internal data at scan time.
-    Structures in scratch space do not persist beyond the end of a single scan
-    call.
-  - **Stream state**: in streaming mode only, some state space is required to
-    store data that persists between scan calls for each stream. This allows
-    Hyperscan to track matches that span multiple blocks of data.
-
-* The sizes of the scratch space and stream state (in streaming mode) required
-  for a given database are fixed and determined at database compile time. This
-  means that the memory requirements of the application are known ahead of
-  time, and these structures can be pre-allocated if required for performance
-  reasons.
-
-* Any pattern that has successfully been compiled by the Hyperscan compiler can
-  be scanned against any input. There are no internal resource limits or other
-  limitations at runtime that could cause a scan call to return an error.
-
-See :ref:`runtime` for more detail.
-
-*****
-Tools
-*****
-
-Some utilities for testing and benchmarking Hyperscan are included with the
-library. See :ref:`tools` for more information.
-
-************
-Example Code
-************
-
-Some simple example code demonstrating the use of the Hyperscan API is
-available in the ``examples/`` subdirectory of the Hyperscan distribution.

doc/dev-reference/performance.rst

@@ -1,345 +0,0 @@
-.. _perf:
-
-##########################
-Performance Considerations
-##########################
-
-Hyperscan supports a wide range of patterns in all three scanning modes. It is
-capable of extremely high levels of performance, but certain patterns can
-reduce performance markedly.
-
-The following guidelines will help construct patterns and pattern sets that
-will perform better:
-
-*****************************
-Regular expression constructs
-*****************************
-
-.. tip:: Do not hand-optimize regular expression constructs.
-
-Quite a large number of regular expressions can be written in multiple ways.
-For example, caseless matching of :regexp:`/abc/` can be written as:
-
-* :regexp:`/[Aa][Bb][Cc]/`
-* :regexp:`/(A|a)(B|b)(C|c)/`
-* :regexp:`/(?i)abc(?-i)/`
-* :regexp:`/abc/i`
-
-Hyperscan is capable of handling all these constructs. Unless there is a
-specific reason otherwise, do not rewrite patterns from one form to another.
-
-As another example, matching of :regexp:`/foo(bar|baz)(frotz)?/` can be
-equivalently written as:
-
-* :regexp:`/foobarfrotz|foobazfrotz|foobar|foobaz/`
-
-This change will not improve performance or reduce overheads.
-
-*************
-Library usage
-*************
-
-.. tip:: Do not hand-optimize library usage.
-
-The Hyperscan library is capable of dealing with small writes, unusually large
-and small pattern sets, etc. Unless there is a specific performance problem
-with some usage of the library, it is best to use Hyperscan in a simple and
-direct fashion. For example, it is unlikely for there to be much benefit in
-buffering input to the library into larger blocks unless streaming writes are
-tiny (say, 1-2 bytes at a time).
-
-Unlike many other pattern matching products, Hyperscan will run faster with
-small numbers of patterns and slower with large numbers of patterns in a smooth
-fashion (as opposed to, typically, running at a moderate speed up to some fixed
-limit then either breaking or running half as fast).
-
-Hyperscan also provides high-throughput matching with a single thread of
-control per core; if a database runs at 3.0 Gbps in Hyperscan it means that a
-3000-bit block of data will be scanned in 1 microsecond in a single thread of
-control, not that it is required to scan 22 3000-bit blocks of data in 22
-microseconds. Thus, it is not usually necessary to buffer data to supply
-Hyperscan with available parallelism.
-
-********************
-Block-based matching
-********************
-
-.. tip:: Prefer block-based matching to streaming matching where possible.
-
-Whenever input data appears in discrete records, or already requires some sort
-of transformation (e.g. URI normalization) that requires all the data to be
-accumulated before processing, it should be scanned in block rather than in
-streaming mode.
-
-Unnecessary use of streaming mode reduces the number of optimizations that can
-be applied in Hyperscan and may make some patterns run slower.
-
-If there is a mixture of 'block' and 'streaming' mode patterns, these should be
-scanned in separate databases except in the case that the streaming patterns
-vastly outnumber the block mode patterns.
-
-*********************
-Unnecessary databases
-*********************
-
-.. tip:: Avoid unnecessary 'union' databases.
-
-If there are 5 different types of network traffic T1 through T5 that must
-be scanned against 5 different signature sets, it will be far more efficient to
-construct 5 separate databases and scan traffic against the appropriate one
-than it will be to merge all 5 signature sets and remove inappropriate matches
-after the fact.
-
-This will be true even in the case where there is substantial overlap among the
-signatures. Only if the common subset of the signatures is overwhelmingly large
-(say, 90% of the signatures appear in all 5 traffic types) should a database
-that merges all 5 signature sets be considered, and only then if there are no
-performance issues with specific patterns that appear outside the common
-subset.
-
-******************************
-Allocate scratch ahead of time
-******************************
-
-.. tip:: Do not allocate scratch space for your pattern database just before
-   calling a scan function. Instead, do it just after the pattern database is
-   compiled or deserialized.
-
-Scratch allocation is not necessarily a cheap operation. Since it is the first
-time (after compilation or deserialization) that a pattern database is used,
-Hyperscan performs some validation checks inside :c:func:`hs_alloc_scratch` and
-must also allocate memory.
-
-Therefore, it is important to ensure that :c:func:`hs_alloc_scratch` is not
-called in the application's scanning path just before :c:func:`hs_scan` (for
-example).
-
-Instead, scratch should be allocated immediately after a pattern database is
-compiled or deserialized, then retained for later scanning operations.
-
-***********************************************
-Allocate one scratch space per scanning context
-***********************************************
-
-.. tip:: A scratch space can be allocated so that it can be used with any one of
-   a number of databases. Each concurrent scan operation (such as a thread)
-   needs its own scratch space.
-
-The :c:func:`hs_alloc_scratch` function can accept an existing scratch space and
-"grow" it to support scanning with another pattern database. This means that
-instead of allocating one scratch space for every database used by an
-application, one can call :c:func:`hs_alloc_scratch` with a pointer to the same
-:c:type:`hs_scratch_t` and it will be sized appropriately for use with any of
-the given databases. For example:
-
-.. code-block:: c
-
-    hs_database_t *db1 = buildDatabaseOne();
-    hs_database_t *db2 = buildDatabaseTwo();
-    hs_database_t *db3 = buildDatabaseThree();
-
-    hs_error_t err;
-    hs_scratch_t *scratch = NULL;
-    err = hs_alloc_scratch(db1, &scratch);
-    if (err != HS_SUCCESS) {
-        printf("hs_alloc_scratch failed!");
-        exit(1);
-    }
-    err = hs_alloc_scratch(db2, &scratch);
-    if (err != HS_SUCCESS) {
-        printf("hs_alloc_scratch failed!");
-        exit(1);
-    }
-    err = hs_alloc_scratch(db3, &scratch);
-    if (err != HS_SUCCESS) {
-        printf("hs_alloc_scratch failed!");
-        exit(1);
-    }
-
-    /* scratch may now be used to scan against any of
-       the databases db1, db2, db3. */
-
-*****************
-Anchored patterns
-*****************
-
-.. tip:: If a pattern is meant to appear at the start of data, be sure to
-   anchor it.
-
-Anchored patterns (:regexp:`/^.../`) are far simpler to match than other
-patterns, especially patterns anchored to the start of the buffer (or stream, in
-streaming mode). Anchoring patterns to the end of the buffer results in less of
-a performance gain, especially in streaming mode.
-
-There are a variety of ways to anchor a pattern to a particular offset:
-
-- The :regexp:`^` and :regexp:`\\A` constructs anchor the pattern to the start
-  of the buffer. For example, :regexp:`/^foo/` can *only* match at offset 3.
-
-- The :regexp:`$`, :regexp:`\\z` and :regexp:`\\Z` constructs anchor the pattern
-  to the end of the buffer. For example, :regexp:`/foo\\z/` can only match when
-  the data buffer being scanned ends in ``foo``. (It should be noted that
-  :regexp:`$` and :regexp:`\\Z` will also match before a newline at the end of
-  the buffer, so :regexp:`/foo\\z/` would match against either ``abc foo`` or
-  ``abc foo\n``.)
-
-- The ``min_offset`` and ``max_offset`` extended parameters may also be used to
-  constrain where a pattern could match. For example, the pattern
-  :regexp:`/foo/` with a ``max_offset`` of 10 will only match at offsets less
-  than or equal to 10 in the buffer. (This pattern could also be written as
-  :regexp:`/^.{0,7}foo/`, compiled with the :c:member:`HS_FLAG_DOTALL` flag).
-
-
-*******************
-Matching everywhere
-*******************
-
-.. tip:: Avoid patterns that match everywhere, and remember that our semantics
-   are 'match everywhere, end of match only'.
-
-Pattern that match everywhere will run slowly due to the sheer number of
-matches that they return.
-
-Patterns like :regexp:`/.*/` in an automata-based matcher will match before and
-after every single character position, so a buffer with 100 characters will
-return 101 matches. Greedy pattern matchers such as libpcre will return a
-single match in this case, but our semantics is to return all matches. This is
-likely to be very expensive for our code and for the client code of the
-library.
-
-Another result of our semantics ("match everywhere") is that patterns that have
-optional start or ending sections -- for example :regexp:`/x?abcd*/` -- may not
-perform as expected.
-
-Firstly, the :regexp:`x?` portion of the pattern is unnecessary, as it will not
-affect the match results.
-
-Secondly, the above pattern will match 'more' than :regexp:`/abc/` but
-:regexp:`/abc/` will always detect any input data that will be matched by
-:regexp:`/x?abcd*/` -- it will just produce fewer matches.
-
-For example, input data ``0123abcdddd`` will match :regexp:`/abc/` once but
-:regexp:`/abcd*/` five times (at ``abc``, ``abcd``, ``abcdd``, ``abcddd``, and
-``abcdddd``).
-
-*********************************
-Bounded repeats in streaming mode
-*********************************
-
-.. tip:: Bounded repeats are expensive in streaming mode.
-
-A bounded repeat construction such as :regexp:`/X.{1000,1001}abcd/` is extremely
-expensive in streaming mode, of necessity. It requires us to take action on
-each ``X`` character (itself expensive, relative to searching for longer strings)
-and potentially record a history of hundreds of offsets where ``X`` occurred in
-case the ``X`` and ``abcd`` characters are separated by a stream boundary.
-
-Heavy and unnecessary use of bounded repeats should be avoided, especially
-where other parts of a signature are quite specific. For example, a virus
-signature that matches a virus payload may be sufficient without including a
-prefix that includes, for example, a 2-character Windows executable prefix and
-a bounded repeat beforehand.
-
-***************
-Prefer literals
-***************
-
-.. tip:: Where possible, prefer patterns which 'require' literals, especially
-   longer literals, and in streaming mode, prefer signatures that 'require'
-   literals earlier in the pattern.
-
-Patterns which must match on a literal will run faster than patterns that do
-not. For example:
-
-- :regexp:`/\\wab\\d*\\w\\w\\w/` will run faster than
-- :regexp:`/\\w\\w\\d*\\w\\w/`, or, for that matter
-- :regexp:`/\\w(abc)?\\d*\\w\\w\\w/` (this contains a literal but it need
-  not appear in the input).
-
-Even implicit literals are better than none: :regexp:`/[0-2][3-5].*\\w\\w/`
-still effectively contains 9 2-character literals. No hand-optimization of this
-case is required; this pattern will not run faster if rewritten as:
-:regexp:`/(03|04|05|13|14|15|23|24|25).*\\w\\w/`.
-
-Under all circumstances it is better to use longer literals than shorter ones.
-A database consisting of 100 14-character literals will scan considerably
-faster than one consisting of 100 4-character literals and return fewer
-positives.
-
-Additionally, in streaming mode, a signature that contains a longer literal
-early in the pattern is preferred to one that does not.
-
-For example: :regexp:`/b\\w*foobar/` is not as good a pattern as
-:regexp:`/blah\\w*foobar/`.
-
-The disparity between these patterns is much smaller in block mode.
-
-Longer literals anywhere in the pattern are still preferred in streaming mode.
-For example, both of the above patterns are stronger and will scan faster than
-:regexp:`/b\\w*fo/` even in streaming mode.
-
-**************
-"Dot all" mode
-**************
-
-.. tip:: Use "dot all" mode where possible.
-
-Not using the :c:member:`HS_FLAG_DOTALL` pattern flag can be expensive, as
-implicitly, it means that patterns of the form :regexp:`/A.*B/` become
-:regexp:`/A[^\\n]*B/`.
-
-It is likely that scanning tasks without the DOTALL flag are better done 'line
-at a time', with the newline sequences marking the beginning and end of each
-block.
-
-This will be true in most use-cases (an exception being where the DOTALL flag
-is off but the pattern contains either explicit newlines or constructs such as
-:regexp:`\\s` that implicitly match a newline character).
-
-*****************
-Single-match flag
-*****************
-
-.. tip:: Consider using the single-match flag to limit matches to one match per
-   pattern only if possible.
-
-If only one match per pattern is required, use the flag provided to indicate
-this (:c:member:`HS_FLAG_SINGLEMATCH`). This flag can allow a number of
-optimizations to be applied, allowing both performance improvements and state
-space reductions when streaming.
-
-However, there is some overhead associated with tracking whether each pattern in
-the pattern set has matched, and some applications with infrequent matches may
-see reduced performance when the single-match flag is used.
-
-********************
-Start of Match flag
-********************
-
-.. tip:: Do not request Start of Match information if it is not not needed.
-
-Start of Match (SOM) information can be expensive to gather and can require
-large amounts of stream state to store in streaming mode. As such, SOM
-information should only be requested with the :c:member:`HS_FLAG_SOM_LEFTMOST`
-flag for patterns that require it.
-
-SOM information is not generally expected to be cheaper (in either performance
-terms or in stream state overhead) than the use of bounded repeats.
-Consequently, :regexp:`/foo.*bar/L` with a check on start of match values after
-the callback is considerably more expensive and general than
-:regexp:`/foo.{300}bar/`.
-
-Similarly, the :c:member:`hs_expr_ext::min_length` extended parameter can be
-used to specify a lower bound on the length of the matches for a pattern. Using
-this facility may be more lightweight in some circumstances than using the SOM
-flag and post-confirming match length in the calling application.
-
-********************
-Approximate matching
-********************
-
-.. tip:: Approximate matching is an experimental feature.
-
-There is generally a performance impact associated with approximate matching due
-to the reduced specificity of the matches. This impact may vary significantly
-depending on the pattern and edit distance.

doc/dev-reference/preface.rst

@@ -1,47 +0,0 @@
-#######
-Preface
-#######
-
-********
-Overview
-********
-
-Hyperscan is a regular expression engine designed to offer high performance, the
-ability to match multiple expressions simultaneously and flexibility in
-scanning operation.
-
-Patterns are provided to a compilation interface which generates an immutable
-pattern database. The scan interface then can be used to scan a target data
-buffer for the given patterns, returning any matching results from that data
-buffer. Hyperscan also provides a streaming mode, in which matches that span
-several blocks in a stream are detected.
-
-This document is designed to facilitate code-level integration of the Hyperscan
-library with existing or new applications.
-
-:ref:`intro` is a short overview of the Hyperscan library, with more detail on
-the Hyperscan API provided in the subsequent sections: :ref:`compilation` and
-:ref:`runtime`.
-
-:ref:`perf` provides details on various factors which may impact the
-performance of a Hyperscan integration.
-
-:ref:`api_constants` and :ref:`api_files` provides a detailed summary of the
-Hyperscan Application Programming Interface (API).
-
-********
-Audience
-********
-
-This guide is aimed at developers interested in integrating Hyperscan into an
-application. For information on building the Hyperscan library, see the Quick
-Start Guide.
-
-***********
-Conventions
-***********
-
-* Text in a ``fixed-width font`` refers to a code element, e.g. type name;
-  function or method name.
-* Text in a :regexp:`coloured fixed-width font` refers to a regular
-  expression or a part of a regular expression.

doc/dev-reference/runtime.rst

@@ -1,242 +0,0 @@
-.. _runtime:
-
-#####################
-Scanning for Patterns
-#####################
-
-Hyperscan provides three different scanning modes, each with its own scan
-function beginning with ``hs_scan``. In addition, streaming mode has a number
-of other API functions for managing stream state.
-
-****************
-Handling Matches
-****************
-
-All of these functions will call a user-supplied callback function when a match
-is found. This function has the following signature:
-
-  .. doxygentypedef:: match_event_handler
-     :outline:
-     :no-link:
-
-The *id* argument will be set to the identifier for the matching expression
-provided at compile time, and the *to* argument will be set to the end-offset
-of the match. If SOM was requested for the pattern (see :ref:`som`), the
-*from* argument will be set to the leftmost possible start-offset for the match.
-
-The match callback function has the capability to halt scanning
-by returning a non-zero value.
-
-See :c:type:`match_event_handler` for more information.
-
-**************
-Streaming Mode
-**************
-
-The core of the Hyperscan streaming runtime API consists of functions to open,
-scan, and close Hyperscan data streams:
-
-* :c:func:`hs_open_stream`: allocates and initializes a new stream for scanning.
-
-* :c:func:`hs_scan_stream`: scans a block of data in a given stream, raising
-  matches as they are detected.
-
-* :c:func:`hs_close_stream`: completes scanning of a given stream (raising any
-  matches that occur at the end of the stream) and frees the stream state. After
-  a call to :c:func:`hs_close_stream`, the stream handle is invalid and should
-  not be used again for any purpose.
-
-Any matches detected in the data as it is scanned are returned to the calling
-application via a function pointer callback.
-
-The match callback function has the capability to halt scanning of the current
-data stream by returning a non-zero value. In streaming mode, the result of
-this is that the stream is then left in a state where no more data can be
-scanned, and any subsequent calls to :c:func:`hs_scan_stream` for that stream
-will return immediately with :c:member:`HS_SCAN_TERMINATED`. The caller must
-still call :c:func:`hs_close_stream` to complete the clean-up process for that
-stream.
-
-Streams exist in the Hyperscan library so that pattern matching state can be
-maintained across multiple blocks of target data -- without maintaining this
-state, it would not be possible to detect patterns that span these blocks of
-data. This, however, does come at the cost of requiring an amount of storage
-per-stream (the size of this storage is fixed at compile time), and a slight
-performance penalty in some cases to manage the state.
-
-While Hyperscan does always support a strict ordering of multiple matches,
-streaming matches will not be delivered at offsets before the current stream
-write, with the exception of zero-width asserts, where constructs such as
-:regexp:`\\b` and :regexp:`$` can cause a match on the final character of a
-stream write to be delayed until the next stream write or stream close
-operation.
-
-=================
-Stream Management
-=================
-
-In addition to :c:func:`hs_open_stream`, :c:func:`hs_scan_stream`, and
-:c:func:`hs_close_stream`, the Hyperscan API provides a number of other
-functions for the management of streams:
-
-* :c:func:`hs_reset_stream`: resets a stream to its initial state; this is
-  equivalent to calling :c:func:`hs_close_stream` but will not free the memory
-  used for stream state.
-
-* :c:func:`hs_copy_stream`: constructs a (newly allocated) duplicate of a
-  stream.
-
-* :c:func:`hs_reset_and_copy_stream`: constructs a duplicate of a stream into
-  another, resetting the destination stream first. This call avoids the
-  allocation done by :c:func:`hs_copy_stream`.
-
-==================
-Stream Compression
-==================
-
-A stream object is allocated as a fixed size region of memory which has been
-sized to ensure that no memory allocations are required during scan
-operations. When the system is under memory pressure, it may be useful to reduce
-the memory consumed by streams that are not expected to be used soon. The
-Hyperscan API provides calls for translating a stream to and from a compressed
-representation for this purpose. The compressed representation differs from the
-full stream object as it does not reserve space for components which are not
-required given the current stream state. The Hyperscan API functions for this
-functionality are:
-
-* :c:func:`hs_compress_stream`: fills the provided buffer with a compressed
-  representation of the stream and returns the number of bytes consumed by the
-  compressed representation. If the buffer is not large enough to hold the
-  compressed representation, :c:member:`HS_INSUFFICIENT_SPACE` is returned along
-  with the required size. This call does not modify the original stream in any
-  way: it may still be written to with :c:func:`hs_scan_stream`, used as part of
-  the various reset calls to reinitialise its state, or
-  :c:func:`hs_close_stream` may be called to free its resources.
-
-* :c:func:`hs_expand_stream`: creates a new stream based on a buffer containing
-  a compressed representation.
-
-* :c:func:`hs_reset_and_expand_stream`: constructs a stream based on a buffer
-  containing a compressed representation on top of an existing stream, resetting
-  the existing stream first. This call avoids the allocation done by
-  :c:func:`hs_expand_stream`.
-
-Note: it is not recommended to use stream compression between every call to scan
-for performance reasons as it takes time to convert between the compressed
-representation and a standard stream.
-
-
-**********
-Block Mode
-**********
-
-The block mode runtime API consists of a single function: :c:func:`hs_scan`. Using
-the compiled patterns this function identifies matches in the target data,
-using a function pointer callback to communicate with the application.
-
-This single :c:func:`hs_scan` function is essentially equivalent to calling
-:c:func:`hs_open_stream`, making a single call to :c:func:`hs_scan_stream`, and
-then :c:func:`hs_close_stream`, except that block mode operation does not
-incur all the stream related overhead.
-
-*************
-Vectored Mode
-*************
-
-The vectored mode runtime API, like the block mode API, consists of a single
-function: :c:func:`hs_scan_vector`. This function accepts an array of data
-pointers and lengths, facilitating the scanning in sequence of a set of data
-blocks that are not contiguous in memory.
-
-From the caller's perspective, this mode will produce the same matches as if
-the set of data blocks were (a) scanned in sequence with a series of streaming
-mode scans, or (b) copied in sequence into a single block of memory and then
-scanned in block mode.
-
-*************
-Scratch Space
-*************
-
-While scanning data, Hyperscan needs a small amount of temporary memory to store
-on-the-fly internal data. This amount is unfortunately too large to fit on the
-stack, particularly for embedded applications, and allocating memory dynamically
-is too expensive, so a pre-allocated "scratch" space must be provided to the
-scanning functions.
-
-The function :c:func:`hs_alloc_scratch` allocates a large enough region of
-scratch space to support a given database. If the application uses multiple
-databases, only a single scratch region is necessary: in this case, calling
-:c:func:`hs_alloc_scratch` on each database (with the same ``scratch`` pointer)
-will ensure that the scratch space is large enough to support scanning against
-any of the given databases.
-
-While the Hyperscan library is re-entrant, the use of scratch spaces is not.
-For example, if by design it is deemed necessary to run recursive or nested
-scanning (say, from the match callback function), then an additional scratch
-space is required for that context.
-
-In the absence of recursive scanning, only one such space is required per thread
-and can (and indeed should) be allocated before data scanning is to commence.
-
-In a scenario where a set of expressions are compiled by a single "main"
-thread and data will be scanned by multiple "worker" threads, the convenience
-function :c:func:`hs_clone_scratch` allows multiple copies of an existing
-scratch space to be made for each thread (rather than forcing the caller to pass
-all the compiled databases through :c:func:`hs_alloc_scratch` multiple times).
-
-For example:
-
-.. code-block:: c
-
-    hs_error_t err;
-    hs_scratch_t *scratch_prototype = NULL;
-    err = hs_alloc_scratch(db, &scratch_prototype);
-    if (err != HS_SUCCESS) {
-        printf("hs_alloc_scratch failed!");
-        exit(1);
-    }
-
-    hs_scratch_t *scratch_thread1 = NULL;
-    hs_scratch_t *scratch_thread2 = NULL;
-
-    err = hs_clone_scratch(scratch_prototype, &scratch_thread1);
-    if (err != HS_SUCCESS) {
-        printf("hs_clone_scratch failed!");
-        exit(1);
-    }
-    err = hs_clone_scratch(scratch_prototype, &scratch_thread2);
-    if (err != HS_SUCCESS) {
-        printf("hs_clone_scratch failed!");
-        exit(1);
-    }
-
-    hs_free_scratch(scratch_prototype);
-
-    /* Now two threads can both scan against database db,
-       each with its own scratch space. */
-
-*****************
-Custom Allocators
-*****************
-
-By default, structures used by Hyperscan at runtime (scratch space, stream
-state, etc) are allocated with the default system allocators, usually
-``malloc()`` and ``free()``.
-
-The Hyperscan API provides a facility for changing this behaviour to support
-applications that use custom memory allocators.
-
-These functions are:
-
-- :c:func:`hs_set_database_allocator`, which sets the allocate and free functions
-  used for compiled pattern databases.
-- :c:func:`hs_set_scratch_allocator`, which sets the allocate and free
-  functions used for scratch space.
-- :c:func:`hs_set_stream_allocator`, which sets the allocate and free functions
-  used for stream state in streaming mode.
-- :c:func:`hs_set_misc_allocator`, which sets the allocate and free functions
-  used for miscellaneous data, such as compile error structures and
-  informational strings.
-
-The :c:func:`hs_set_allocator` function can be used to set all of the custom
-allocators to the same allocate/free pair.

doc/dev-reference/serialization.rst

@@ -1,67 +0,0 @@
-.. _serialization:
-
-#############
-Serialization
-#############
-
-For some applications, compiling Hyperscan pattern databases immediately prior
-to use is not an appropriate design. Some users may wish to:
-
-* Compile pattern databases on a different host;
-
-* Persist compiled databases to storage and only re-compile pattern databases
-  when the patterns change;
-
-* Control the region of memory in which the compiled database is located.
-
-Hyperscan pattern databases are not completely flat in memory: they contain
-pointers and have specific alignment requirements. Therefore, they cannot be
-copied (or otherwise relocated) directly. To enable these use cases, Hyperscan
-provides functionality for serializing and deserializing compiled pattern
-databases.
-
-The API provides the following functions:
-
-#. :c:func:`hs_serialize_database`: serializes a pattern database into a
-   flat relocatable buffer of bytes.
-
-#. :c:func:`hs_deserialize_database`: reconstructs a newly allocated pattern
-   database from the output of :c:func:`hs_serialize_database`.
-
-#. :c:func:`hs_deserialize_database_at`: reconstructs a pattern
-   database at a given memory location from the output of
-   :c:func:`hs_serialize_database`.
-
-#. :c:func:`hs_serialized_database_size`: given a serialized pattern database,
-   returns the size of the memory block required by the database when
-   deserialized.
-
-#. :c:func:`hs_serialized_database_info`: given a serialized pattern database,
-   returns a string containing information about the database. This call is
-   analogous to :c:func:`hs_database_info`.
-
-.. note:: Hyperscan performs both version and platform compatibility checks
-   upon deserialization. The :c:func:`hs_deserialize_database` and
-   :c:func:`hs_deserialize_database_at` functions will only permit the
-   deserialization of databases compiled with (a) the same version of Hyperscan
-   and (b) platform features supported by the current host platform. See
-   :ref:`instr_specialization` for more information on platform specialization.
-
-===================
-The Runtime Library
-===================
-
-The main Hyperscan library (``libhs``) contains both the compiler and runtime
-portions of the library. This means that in order to support the Hyperscan
-compiler, which is written in C++, it requires C++ linkage and has a
-dependency on the C++ standard library.
-
-Many embedded applications require only the scanning ("runtime") portion of the
-Hyperscan library. In these cases, pattern compilation generally takes place on
-another host, and serialized pattern databases are delivered to the application
-for use.
-
-To support these applications without requiring the C++ dependency, a
-runtime-only version of the Hyperscan library, called ``libhs_runtime``, is also
-distributed. This library does not depend on the C++ standard library and
-provides all Hyperscan functions other that those used to compile databases.

doc/dev-reference/tools.rst

@@ -1,265 +0,0 @@
-.. _tools:
-
-#####
-Tools
-#####
-
-This section describes the set of utilities included with the Hyperscan library.
-
-********************
-Quick Check: hscheck
-********************
-
-The ``hscheck`` tool allows the user to quickly check whether Hyperscan supports
-a group of patterns. If a pattern is rejected by Hyperscan's compiler, the
-compile error is provided on standard output.
-
-For example, given the following three patterns (the last of which contains a
-syntax error) in a file called ``/tmp/test``::
-
-    1:/foo.*bar/
-    2:/abc|def|ghi/
-    3:/((foo|bar)/
-
-... the ``hscheck`` tool will produce the following output::
-
-    $ bin/hscheck -e /tmp/test
-
-    OK: 1:/foo.*bar/
-    OK: 2:/abc|def|ghi/
-    FAIL (compile): 3:/((foo|bar)/: Missing close parenthesis for group started at index 0.
-    SUMMARY: 1 of 3 failed.
-
-********************
-Benchmarker: hsbench
-********************
-
-The ``hsbench`` tool provides an easy way to measure Hyperscan's performance
-for a particular set of patterns and corpus of data to be scanned.
-
-Patterns are supplied in the format described below in
-:ref:`tools_pattern_format`, while the corpus must be provided in the form of a
-`corpus database`: this is a simple SQLite database format intended to allow for
-easy control of how a corpus is broken into blocks and streams.
-
-.. note:: A group of Python scripts for constructing corpora databases from
-   various input types, such as PCAP network traffic captures or text files, can
-   be found in the Hyperscan source tree in ``tools/hsbench/scripts``.
-
-Running hsbench
-===============
-
-Given a file full of patterns specified with ``-e`` and a corpus database
-specified with ``-c``, ``hsbench`` will perform a single-threaded benchmark and
-produce output like this::
-
-    $ hsbench -e /tmp/patterns -c /tmp/corpus.db
-
-    Signatures:        /tmp/patterns
-    Hyperscan info:    Version: 4.3.1 Features:  AVX2 Mode: STREAM
-    Expression count:  200
-    Bytecode size:     342,540 bytes
-    Database CRC:      0x6cd6b67c
-    Stream state size: 252 bytes
-    Scratch size:      18,406 bytes
-    Compile time:      0.153 seconds
-    Peak heap usage:   78,073,856 bytes
-
-    Time spent scanning:     0.600 seconds
-    Corpus size:             72,138,183 bytes (63,946 blocks in 8,891 streams)
-    Scan matches:            81 (0.001 matches/kilobyte)
-    Overall block rate:      2,132,004.45 blocks/sec
-    Overall throughput:      19,241.10 Mbit/sec
-
-By default, the corpus is scanned twenty times, and the overall performance
-reported is computed based the total number of bytes scanned in the time it
-takes to perform all twenty scans. The number of repeats can be changed with the
-``-n`` argument, and the results of each scan will be displayed if the
-``--per-scan`` argument is specified.
-
-To benchmark Hyperscan on more than one core, you can supply a list of cores
-with the ``-T`` argument, which will instruct ``hsbench`` to start one
-benchmark thread per core given and compute the throughput from the time taken
-to complete all of them.
-
-.. tip:: For single-threaded benchmarks on multi-processor systems, we recommend
-   using a utility like ``taskset`` to lock the hsbench process to one core and
-   minimize jitter due to the operating system's scheduler.
-
-*******************************
-Correctness Testing: hscollider
-*******************************
-
-The ``hscollider`` tool, or Pattern Collider, provides a way to verify
-Hyperscan's matching behaviour. It does this by compiling and scanning patterns
-(either singly or in groups) against known corpora and comparing the results
-against another engine (the "ground truth"). Two sources of ground truth for
-comparison are available:
-
- * The PCRE library (http://pcre.org/).
- * An NFA simulation run on Hyperscan's compile-time graph representation. This
-   is used if PCRE cannot support the pattern or if PCRE execution fails due to
-   a resource limit.
-
-Much of Hyperscan's testing infrastructure is built on ``hscollider``, and the
-tool is designed to take advantage of multiple cores and provide considerable
-flexibility in controlling the test. These options are described in the help
-(``hscollider -h``) and include:
-
- * Testing in streaming, block or vectored mode.
- * Testing corpora at different alignments in memory.
- * Testing patterns in groups of varying size.
- * Manipulating stream state or scratch space between tests.
- * Cross-compilation and serialization/deserialization of databases.
- * Synthetic generation of corpora given a pattern set.
-
-Using hscollider to debug a pattern
-===================================
-
-One common use-case for ``hscollider`` is to determine whether Hyperscan will
-match a pattern in the expected location, and whether this accords with PCRE's
-behaviour for the same case.
-
-Here is an example. We put our pattern in a file in Hyperscan's pattern
-format::
-
-    $ cat /tmp/pat
-    1:/hatstand.*badgerbrush/
-
-We put the corpus to be scanned in another file, with the same numeric
-identifier at the start to indicate that it should match pattern 1::
-
-    $ cat /tmp/corpus
-    1:__hatstand__hatstand__badgerbrush_badgerbrush
-
-Then we can run ``hscollider`` with its verbosity turned up (``-vv``) so that
-individual matches are displayed in the output::
-
-    $ bin/ue2collider -e /tmp/pat -c /tmp/corpus -Z 0 -T 1 -vv
-    ue2collider: The Pattern Collider Mark II
-
-    Number of threads:  1 (1 scanner, 1 generator)
-    Expression path:    /tmp/pat
-    Signature files:    none
-    Mode of operation:  block mode
-    UE2 scan alignment: 0
-    Corpora read from file: /tmp/corpus
-
-    Running single-pattern/single-compile test for 1 expressions.
-
-    PCRE Match @ (2,45)
-    PCRE Match @ (2,33)
-    PCRE Match @ (12,45)
-    PCRE Match @ (12,33)
-    UE2 Match @ (0,33) for 1
-    UE2 Match @ (0,45) for 1
-    Scan call returned 0
-    PASSED: id 1, alignment 0, corpus 0 (matched pcre:2, ue2:2)
-    Thread 0 processed 1 units.
-
-    Summary:
-    Mode:                           Single/Block
-    =========
-    Expressions processed:          1
-    Corpora processed:              1
-    Expressions with failures:      0
-      Corpora generation failures:  0
-      Compilation failures:         pcre:0, ng:0, ue2:0
-      Matching failures:            pcre:0, ng:0, ue2:0
-      Match differences:            0
-      No ground truth:              0
-    Total match differences:        0
-
-    Total elapsed time: 0.00522815 secs.
-
-We can see from this output that both PCRE and Hyperscan find matches ending at
-offset 33 and 45, and so ``hscollider`` considers this test case to have
-passed.
-
-(In the example command line above, ``-Z 0`` instructs us to only test at
-corpus alignment 0, and ``-T 1`` instructs us to only use one thread.)
-
-.. note:: In default operation, PCRE produces only one match for a scan, unlike
-  Hyperscan's automata semantics. The ``hscollider`` tool uses libpcre's
-  "callout" functionality to match Hyperscan's semantics.
-
-Running a larger scan test
-==========================
-
-A set of patterns for testing purposes are distributed with Hyperscan, and these
-can be tested via ``hscollider`` on an in-tree build. Two CMake targets are
-provided to do this easily:
-
-================================= =====================================
-Make Target                       Description
-================================= =====================================
-``make collide_quick_test``       Tests all patterns in streaming mode.
-``make collide_quick_test_block`` Tests all patterns in block mode.
-================================= =====================================
-
-*****************
-Debugging: hsdump
-*****************
-
-When built in debug mode (using the CMake directive ``CMAKE_BUILD_TYPE`` set to
-``Debug``), Hyperscan includes support for dumping information about its
-internals during pattern compilation with the ``hsdump`` tool.
-
-This information is mostly of use to Hyperscan developers familiar with the
-library's internal structure, but can be used to diagnose issues with patterns
-and provide more information in bug reports.
-
-.. _tools_pattern_format:
-
-**************
-Pattern Format
-**************
-
-All of the Hyperscan tools accept patterns in the same format, read from plain
-text files with one pattern per line. Each line looks like this:
-
-* ``<integer id>:/<regex>/<flags>``
-
-For example::
-
-    1:/hatstand.*teakettle/s
-    2:/(hatstand|teakettle)/iH
-    3:/^.{10,20}hatstand/m
-
-The integer ID is the value that will be reported when a match is found by
-Hyperscan and must be unique.
-
-The pattern itself is a regular expression in PCRE syntax; see
-:ref:`compilation` for more information on supported features.
-
-The flags are single characters that map to Hyperscan flags as follows:
-
-=========   =================================    ===========
-Character   API Flag                             Description
-=========   =================================    ===========
-``i``       :c:member:`HS_FLAG_CASELESS`         Case-insensitive matching
-``s``       :c:member:`HS_FLAG_DOTALL`           Dot (``.``) will match newlines
-``m``       :c:member:`HS_FLAG_MULTILINE`        Multi-line anchoring
-``H``       :c:member:`HS_FLAG_SINGLEMATCH`      Report match ID at most once
-``V``       :c:member:`HS_FLAG_ALLOWEMPTY`       Allow patterns that can match against empty buffers
-``8``       :c:member:`HS_FLAG_UTF8`             UTF-8 mode
-``W``       :c:member:`HS_FLAG_UCP`              Unicode property support
-``P``       :c:member:`HS_FLAG_PREFILTER`        Prefiltering mode
-``L``       :c:member:`HS_FLAG_SOM_LEFTMOST`     Leftmost start of match reporting
-``C``       :c:member:`HS_FLAG_COMBINATION`      Logical combination of patterns
-``Q``       :c:member:`HS_FLAG_QUIET`            Quiet at matching
-=========   =================================    ===========
-
-In addition to the set of flags above, :ref:`extparam` can be supplied
-for each pattern. These are supplied after the flags as ``key=value`` pairs
-between braces, separated by commas. For example::
-
-    1:/hatstand.*teakettle/s{min_offset=50,max_offset=100}
-
-All Hyperscan tools will accept a pattern file (or a directory containing
-pattern files) with the ``-e`` argument. If no further arguments constraining
-the pattern set are given, all patterns in those files are used.
-
-To select a subset of the patterns, a single ID can be supplied with the ``-z``
-argument, or a file containing a set of IDs can be supplied with the ``-s``
-argument.

examples/CMakeLists.txt

@@ -1,27 +0,0 @@
-find_library(PCAP_LIBRARY pcap)
-
-if (NOT PCAP_LIBRARY)
-    message(STATUS "Could not find libpcap - some examples will not be built")
-endif()
-
-add_executable(simplegrep simplegrep.c)
-set_source_files_properties(simplegrep.c PROPERTIES COMPILE_FLAGS
-    "-Wall -Wno-unused-parameter")
-target_link_libraries(simplegrep hs)
-
-if (PCAP_LIBRARY)
-add_executable(pcapscan pcapscan.cc)
-set_source_files_properties(pcapscan.cc PROPERTIES COMPILE_FLAGS
-    "-Wall -Wno-unused-parameter")
-target_link_libraries(pcapscan hs pcap)
-endif()
-
-if (PCAP_LIBRARY)
-add_executable(patbench patbench.cc)
-set_source_files_properties(patbench.cc PROPERTIES COMPILE_FLAGS
-    "-Wall -Wno-unused-parameter")
-target_link_libraries(patbench hs pcap)
-endif()
-
-install(FILES simplegrep.c pcapscan.cc patbench.cc README.md
-        DESTINATION ${CMAKE_INSTALL_DOCDIR}/examples)

examples/README.md

@@ -1,155 +0,0 @@
-Hyperscan Example Code
-======================
-
-Copyright (C) 2015 Intel Corporation. All rights reserved.
-
-The files in this directory contain example code demonstrating the use of the
-Hyperscan regular expression matching library. The examples have been
-constructed to be useful utility programs, but they have been simplified
-somewhat, so generally contain "shortcuts" that one would not take if building
-a "real" system.
-
-The examples each contain a short description in a comment at the top of the
-file, including build instructions.
-
----
-
-
-Example 1: simplegrep
----------------------
-
-The first example program (`simplegrep.c`) is modelled on the ubiquitous grep
-tool to search a file for a single regular expression. 'simplegrep' does the
-same, but eschews a lot of grep's complexity: it is unable to read data from
-`stdin`, and doesn't support grep's plethora of command-line arguments.
-
-This code is intended to be simple portable C99.
-
-simplegrep demonstrates the following Hyperscan concepts:
-
-- Single pattern compilation: As simplegrep can scan for one pattern only, it
-  uses the `hs_compile` function instead of the multi-pattern variant:
-  `hs_compile_multi`.
-
-- Block mode pattern-matching: simplegrep will search a single data buffer
-  for the given pattern, so it has no need to set up and tear down streams.
-  (See the next section for a streaming mode example)
-
-- Scratch space allocation and use: Hyperscan requires a small amount of
-  temporary memory that is used in the `hs_scan` call. The caller needs to
-  guarantee that only one instance of `hs_scan` is using the scratch space at a
-  time, but there is no requirement that the same scratch area be used on
-  consecutive calls to `hs_scan`. Given that it is expensive to allocate the
-  scratch space, one would typically allocate all necessary scratch space at
-  system startup and reuse it throughout execution of the program.
-
-
-Example 2: pcapscan
--------------------
-
-The second example program (`pcapscan.cc`) is a very simple packet scanning
-benchmark. It scans a given PCAP file full of network traffic against a group
-of regular expressions and returns some coarse performance measurements.  This
-example provides a quick way to examine the performance achievable on a
-particular combination of platform, pattern set and input data.
-
-In block mode, pcapscan scans each packet individually against a Hyperscan
-database. In streaming mode, pcapscan assigns packets to flows using a
-rudimentary connection tracker, then scans the packets in each flow with
-Hyperscan's streaming mode interface. This demonstrates the use of streaming
-mode operation to detect matches that straddle packet boundaries.
-
-**Note**: the flow assignment implemented here is intended as a simple demo; it
-merely ensures that packets with the same 5-tuple are written to the same
-stream in the order in which they appear in the PCAP file.  No packet
-re-ordering or connection state tracking (as you would expect to find in a real
-network scanning application) is done.
-
-pcapscan introduces the following Hyperscan concepts:
-
-- Multi-pattern compilation: Unlike simplegrep, pcapscan requires a file of
-  expressions as input instead of a single pattern. pcapscan will read this
-  file in, one pattern per line, and use it as input to the `hs_compile_multi`
-  function. This function generates a pattern database that will match all the
-  input patterns in parallel.
-
-- Streamed pattern-matching: pcapscan uses the `hs_scan_stream` function
-  (instead of the block-mode `hs_scan` call) to allow it to identify matches
-  that occur in a stream of data, even if they straddle the boundaries between blocks.
-  Streaming mode operation has a number of unique properties:
-
-  - Stream state that persists for the lifetime of the stream must be allocated
-    with the `hs_open_stream` function before scanning can take place.
-    Similarly, it must be freed with `hs_close_stream` after it is no longer
-    needed. Each stream being scanned concurrently requires its own stream
-    state.
-
-  - In streaming mode, a non-zero return from the user-specified event-handler
-    function has consequences for the rest of that stream's lifetime: when a
-    non-zero return occurs, it signals that no more of the stream should be
-    scanned. Consequently if the user makes a subsequent call to
-    `hs_scan_stream` on a stream whose processing was terminated in this way,
-    hs_scan_stream will return `HS_SCAN_TERMINATED`. This case has not been
-    demonstrated in pcapscan, as its callback always returns 0.
-
-  - Match handling during stream shutdown: As matches may occur when the
-    `hs_close_stream` function is called, it too must be provided with scratch
-    space in order to perform this match processing. Similarly, the user must
-    be prepared to be issued match event callbacks during the `hs_close_stream`
-    call. For this reason, we advise that stream shutdown be an integral part
-    of the system design.
-
-
-Example 3: patbench
--------------------
-
-This program allows users to detect which signatures may be the most expensive
-in a set of patterns. It is designed for use with small to medium pattern set
-sizes (e.g. 5-500). If used with very large pattern sets it may take a very
-long time - the number of recompiles done is `g * O(lg2(n))` where `g` is the
-number of generations and `n` is the number of patterns (assuming that `n >>
-g`).
-
-This utility will return a cumulative series of removed patterns. The first
-generation will find and remove a single pattern. The second generation will
-begin with the first pattern removed and find another pattern to remove, etc.
-So if we have 100 patterns and 15 generations, the final generation's score
-will be a run over 85 patterns.
-
-This utility is probabilistic. It is possible that the pattern removed in a
-generation is not a particularly expensive pattern. To reduce noise in the
-results use 'taskset' and set the number of repeats to a level that still
-completes in reasonable time (this will reduce the effect of random measurement
-noise).
-
-The criterion for performance can be altered by use of the `-C<x>` flag where
-`<x>` can be `t,r,s,c,b`, selecting pattern matching throughput, scratch size,
-stream state size (only available in streaming mode), compile time and bytecode
-size respectively.
-
-This utility will also not produce good results if all the patterns are roughly
-equally expensive.
-
-### Factor Group Size:
-
-If there are multiple expensive patterns that are very similar on the
-left-hand-side or identical, this utility will typically not find these groups
-unless the `-F` flag is used to search for a group size that is equal to or
-larger than the size of the group of similar patterns.
-
-Otherwise, removing a portion of the similar patterns will have no or almost no
-effect, and the search procedure used relies on the ability to remove all of
-the similar patterns in at least one search case, something which will only
-happen if the `factor_group_size` is large enough.
-
-This alters the operation of the tool so that instead of trying to find the
-single pattern whose removal has the most effect by binary search (the default
-with `factor_group_size == 1`), we attempt to find the N patterns whose removal
-has the most effect by searching over `N + 1` evenly sized groups, removing
-only `1/(N + 1)` of the search signatures per iteration.
-
-Note that the number of recompiles done greatly increases with increased factor
-group size.  For example, with `factor_group_size = 1`, we do `g * 2 * lg2(n)`
-recompiles, while with `factor_group_size = 4`, we do `g * 4 * log(5/4)(n)`.
-Informally the number of generations we require goes up as we eliminate a
-smaller number of signatures and the we have to do more work per generation.

examples/patbench.cc

@@ -1,905 +0,0 @@
-/*
- * Copyright (c) 2015-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/*
- * Hyperscan pattern benchmarker.
- *
- * This program allows users to detect which signatures may be the most
- * expensive in a set of patterns. It is designed for use with small to medium
- * pattern set sizes (e.g. 5-500). If used with very large pattern sets it may
- * take a very long time - the number of recompiles done is g * O(lg2(n)) where
- * g is the number of generations and n is the number of patterns (assuming
- * that n >> g).
- *
- * This utility will return a cumulative series of removed patterns. The first
- * generation will find and remove a single pattern. The second generation will
- * begin with the first pattern removed and find another pattern to remove,
- * etc. So if we have 100 patterns and 15 generations, the final generation's
- * score will be a run over 85 patterns.
- *
- * This utility is probabilistic. It is possible that the pattern removed in a
- * generation is not a particularly expensive pattern. To reduce noise in the
- * results use 'taskset' and set the number of repeats to a level that still
- * completes in reasonable time (this will reduce the effect of random
- * measurement noise).
- *
- * The criterion for performance can be altered by use of the -C<x> flag where
- * <x> can be t,r,s,c,b, selecting pattern matching throughput, scratch size,
- * stream state size (only available in streaming mode), compile time and
- * bytecode size respectively.
- *
- * This utility will also not produce good results if all the patterns are
- * roughly equally expensive.
- *
- * Factor Group Size:
- *
- * If there are multiple expensive patterns that are very similar on the
- * left-hand-side or identical, this utility will typically not find these
- * groups unless the -F flag is used to search for a group size that is equal
- * to or larger than the size of the group of similar patterns.
- *
- * Otherwise, removing a portion of the similar patterns will have no or almost
- * no effect, and the search procedure used relies on the ability to remove all
- * of the similar patterns in at least one search case, something which will
- * only happen if the factor_group_size is large enough.
- *
- * This alters the operation of our tool so that instead of trying to find the
- * single pattern whose removal has the most effect by binary search (the
- * default with factor_group_size == 1), we attempt to find the N patterns
- * whose removal has the most effect by searching over N+1 evenly sized groups,
- * removing only 1/(N+1) of the search signatures per iteration.
- *
- * Note that the number of recompiles done greatly increases with increased
- * factor group size.  For example, with factor_group_size = 1, we do g * 2 *
- * lg2(n) recompiles, while with factor_group_size = 4, we do g * 4 *
- * log(5/4)(n). Informally the number of generations we require goes up as we
- * eliminate a smaller number of signatures and the we have to do more work per
- * generation.
- *
- *
- * Build instructions:
- *
- *     g++ -o patbench patbench.cc $(pkg-config --cflags --libs libhs) -lpcap
- *
- * Usage:
- *
- *     ./patbench [ -n repeats] [ -G generations] [ -C criterion ]
- *             [ -F factor_group_size ] [ -N | -S ] <pattern file> <pcap file>
- *
- *     -n repeats sets the number of times the PCAP is repeatedly scanned
- *        with the pattern
- *     -G generations sets the number of generations that the algorithm is
- *        run for
- *     -N sets non-streaming mode, -S sets streaming mode (default)
- *     -F sets the factor group size (must be >0); this allows the detection
- *        of multiple interacting factors
- *
- *     -C sets the "criterion", which can be either:
- *          t  throughput (the default) - this requires a pcap file
- *          r  scratch size
- *          s  stream state size
- *          c  compile time
- *          b  bytecode size
- *
- * We recommend the use of a utility like 'taskset' on multiprocessor hosts to
- * lock execution to a single processor: this will remove processor migration
- * by the scheduler as a source of noise in the results.
- *
- */
-
-#include <random>
-#include <algorithm>
-#include <cstring>
-#include <chrono>
-#include <fstream>
-#include <iomanip>
-#include <iostream>
-#include <set>
-#include <string>
-#include <vector>
-#include <unordered_map>
-
-#include <unistd.h>
-
-// We use the BSD primitives throughout as they exist on both BSD and Linux.
-#define __FAVOR_BSD
-#include <netinet/in.h>
-#include <netinet/in_systm.h>
-#include <netinet/ip.h>
-#include <netinet/tcp.h>
-#include <netinet/udp.h>
-#include <netinet/ip_icmp.h>
-#include <net/ethernet.h>
-#include <arpa/inet.h>
-
-#include <pcap.h>
-
-#include <hs.h>
-
-using std::cerr;
-using std::cout;
-using std::endl;
-using std::ifstream;
-using std::string;
-using std::unordered_map;
-using std::vector;
-using std::set;
-using std::min;
-using std::max;
-using std::copy;
-using std::random_device;
-using std::mt19937;
-
-enum Criterion {
-    CRITERION_THROUGHPUT,
-    CRITERION_BYTECODE_SIZE,
-    CRITERION_COMPILE_TIME,
-    CRITERION_STREAM_STATE,
-    CRITERION_SCRATCH_SIZE
-};
-
-static bool higher_is_better(Criterion c) {
-    return c == CRITERION_THROUGHPUT;
-}
-
-static void print_criterion(Criterion c, double val) {
-    std::ios::fmtflags f(cout.flags());
-    switch (c) {
-    case CRITERION_THROUGHPUT:
-        cout << std::fixed << std::setprecision(3) << val << " Megabits/s";
-        break;
-    case CRITERION_COMPILE_TIME:
-        cout << std::fixed << std::setprecision(3) << val << " seconds";
-        break;
-    case CRITERION_BYTECODE_SIZE:
-    case CRITERION_STREAM_STATE:
-    case CRITERION_SCRATCH_SIZE:
-    default:
-        cout << static_cast<size_t>(val) << " bytes";
-        break;
-    }
-    cout.flags(f);
-}
-
-// Key for identifying a stream in our pcap input data, using data from its IP
-// headers.
-struct FiveTuple {
-    unsigned int protocol;
-    unsigned int srcAddr;
-    unsigned int srcPort;
-    unsigned int dstAddr;
-    unsigned int dstPort;
-
-    // Construct a FiveTuple from a TCP or UDP packet.
-    FiveTuple(const struct ip *iphdr) {
-        // IP fields
-        protocol = iphdr->ip_p;
-        srcAddr = iphdr->ip_src.s_addr;
-        dstAddr = iphdr->ip_dst.s_addr;
-
-        // UDP/TCP ports
-        const struct udphdr *uh = (const struct udphdr *)
-                (((const char *)iphdr) + (iphdr->ip_hl * 4));
-        srcPort = uh->uh_sport;
-        dstPort = uh->uh_dport;
-    }
-
-    bool operator==(const FiveTuple &a) const {
-        return protocol == a.protocol && srcAddr == a.srcAddr &&
-               srcPort == a.srcPort && dstAddr == a.dstAddr &&
-               dstPort == a.dstPort;
-    }
-};
-
-// A *very* simple hash function, used when we create an unordered_map of
-// FiveTuple objects.
-struct FiveTupleHash {
-    size_t operator()(const FiveTuple &x) const {
-        return x.srcAddr ^ x.dstAddr ^ x.protocol ^ x.srcPort ^ x.dstPort;
-    }
-};
-
-// Helper function. See end of file.
-static bool payloadOffset(const unsigned char *pkt_data, unsigned int *offset,
-                          unsigned int *length);
-
-// Match event handler: called every time Hyperscan finds a match.
-static
-int onMatch(unsigned int id, unsigned long long from, unsigned long long to,
-            unsigned int flags, void *ctx) {
-    // Our context points to a size_t storing the match count
-    size_t *matches = (size_t *)ctx;
-    (*matches)++;
-    return 0; // continue matching
-}
-
-// Simple timing class
-class Clock {
-public:
-    void start() {
-        time_start = std::chrono::system_clock::now();
-    }
-
-    void stop() {
-        time_end = std::chrono::system_clock::now();
-    }
-
-    double seconds() const {
-        std::chrono::duration<double> delta = time_end - time_start;
-        return delta.count();
-    }
-private:
-    std::chrono::time_point<std::chrono::system_clock> time_start, time_end;
-};
-
-// Class wrapping all state associated with the benchmark
-class Benchmark {
-private:
-    // Packet data to be scanned
-    vector<string> packets;
-
-    // Stream ID for each packet
-    vector<size_t> stream_ids;
-
-    // Map used to construct stream_ids
-    unordered_map<FiveTuple, size_t, FiveTupleHash> stream_map;
-
-    // Hyperscan compiled database
-    hs_database_t *db = nullptr;
-
-    // Hyperscan temporary scratch space
-    hs_scratch_t *scratch = nullptr;
-
-    // Vector of Hyperscan stream state
-    vector<hs_stream_t *> streams;
-
-    // Count of matches found while scanning
-    size_t matchCount = 0;
-public:
-    ~Benchmark() {
-        hs_free_scratch(scratch);
-        hs_free_database(db);
-    }
-
-    // Initialisation; after this call, Benchmark owns the database and will
-    // ensure it is freed.
-    void setDatabase(hs_database_t *hs_db) {
-        hs_free_database(db); // Free previous database.
-        db = hs_db;
-        // (Re)allocate scratch to ensure that it is large enough to handle the
-        // database.
-        hs_error_t err = hs_alloc_scratch(db, &scratch);
-        if (err != HS_SUCCESS) {
-            cerr << "ERROR: could not allocate scratch space. Exiting." << endl;
-            exit(-1);
-        }
-    }
-    const hs_database_t *getDatabase() const {
-        return db;
-    }
-
-    size_t getScratchSize() const {
-        size_t scratch_size;
-        hs_error_t err = hs_scratch_size(scratch, &scratch_size);
-        if (err != HS_SUCCESS) {
-            cerr << "ERROR: could not query scratch space size. Exiting."
-                 << endl;
-            exit(-1);
-        }
-        return scratch_size;
-    }
-
-    // Read a set of streams from a pcap file
-    bool readStreams(const char *pcapFile) {
-        // Open PCAP file for input
-        char errbuf[PCAP_ERRBUF_SIZE];
-        pcap_t *pcapHandle = pcap_open_offline(pcapFile, errbuf);
-        if (pcapHandle == nullptr) {
-            cerr << "ERROR: Unable to open pcap file \"" << pcapFile
-                 << "\": " << errbuf << endl;
-            return false;
-        }
-
-        struct pcap_pkthdr pktHeader;
-        const unsigned char *pktData;
-        while ((pktData = pcap_next(pcapHandle, &pktHeader)) != nullptr) {
-            unsigned int offset = 0, length = 0;
-            if (!payloadOffset(pktData, &offset, &length)) {
-                continue;
-            }
-
-            // Valid TCP or UDP packet
-            const struct ip *iphdr = (const struct ip *)(pktData
-                    + sizeof(struct ether_header));
-            const char *payload = (const char *)pktData + offset;
-
-            size_t id = stream_map.insert(std::make_pair(FiveTuple(iphdr),
-                                          stream_map.size())).first->second;
-
-            packets.push_back(string(payload, length));
-            stream_ids.push_back(id);
-        }
-        pcap_close(pcapHandle);
-
-        return !packets.empty();
-    }
-
-    // Return the number of bytes scanned
-    size_t bytes() const {
-        size_t sum = 0;
-        for (const auto &packet : packets) {
-            sum += packet.size();
-        }
-        return sum;
-    }
-
-    // Return the number of matches found.
-    size_t matches() const {
-        return matchCount;
-    }
-
-    // Clear the number of matches found.
-    void clearMatches() {
-        matchCount = 0;
-    }
-
-    // Open a Hyperscan stream for each stream in stream_ids
-    void openStreams() {
-        streams.resize(stream_map.size());
-        for (auto &stream : streams) {
-            hs_error_t err = hs_open_stream(db, 0, &stream);
-            if (err != HS_SUCCESS) {
-                cerr << "ERROR: Unable to open stream. Exiting." << endl;
-                exit(-1);
-            }
-        }
-    }
-
-    // Close all open Hyperscan streams (potentially generating any
-    // end-anchored matches)
-    void closeStreams() {
-        for (auto &stream : streams) {
-            hs_error_t err =
-                hs_close_stream(stream, scratch, onMatch, &matchCount);
-            if (err != HS_SUCCESS) {
-                cerr << "ERROR: Unable to close stream. Exiting." << endl;
-                exit(-1);
-            }
-        }
-    }
-
-    // Scan each packet (in the ordering given in the PCAP file) through
-    // Hyperscan using the streaming interface.
-    void scanStreams() {
-        for (size_t i = 0; i != packets.size(); ++i) {
-            const std::string &pkt = packets[i];
-            hs_error_t err = hs_scan_stream(streams[stream_ids[i]],
-                                            pkt.c_str(), pkt.length(), 0,
-                                            scratch, onMatch, &matchCount);
-            if (err != HS_SUCCESS) {
-                cerr << "ERROR: Unable to scan packet. Exiting." << endl;
-                exit(-1);
-            }
-        }
-    }
-
-    // Scan each packet (in the ordering given in the PCAP file) through
-    // Hyperscan using the block-mode interface.
-    void scanBlock() {
-        for (size_t i = 0; i != packets.size(); ++i) {
-            const std::string &pkt = packets[i];
-            hs_error_t err = hs_scan(db, pkt.c_str(), pkt.length(), 0,
-                                     scratch, onMatch, &matchCount);
-            if (err != HS_SUCCESS) {
-                cerr << "ERROR: Unable to scan packet. Exiting." << endl;
-                exit(-1);
-            }
-        }
-    }
-};
-
-// helper function - see end of file
-static void parseFile(const char *filename, vector<string> &patterns,
-                      vector<unsigned> &flags, vector<unsigned> &ids,
-                      vector<string> &originals);
-
-class Sigdata {
-    vector<unsigned> flags;
-    vector<unsigned> ids;
-    vector<string> patterns;
-    vector<string> originals;
-
-public:
-    Sigdata() {}
-    Sigdata(const char *filename) {
-        parseFile(filename, patterns, flags, ids, originals);
-
-    }
-
-    const string &get_original(unsigned index) const {
-        return originals[index];
-    }
-
-    hs_database_t *compileDatabase(unsigned mode, double *compileTime) const {
-        hs_database_t *db = nullptr;
-        hs_compile_error_t *compileErr;
-
-        // Turn our vector of strings into a vector of char*'s to pass in to
-        // hs_compile_multi. (This is just using the vector of strings as
-        // dynamic storage.)
-        vector<const char *> cstrPatterns;
-        cstrPatterns.reserve(patterns.size());
-        for (const auto &pattern : patterns) {
-            cstrPatterns.push_back(pattern.c_str());
-        }
-
-        Clock clock;
-        clock.start();
-        hs_error_t err = hs_compile_multi(cstrPatterns.data(), flags.data(),
-                                          ids.data(), cstrPatterns.size(), mode,
-                                          nullptr, &db, &compileErr);
-        clock.stop();
-        if (err != HS_SUCCESS) {
-            if (compileErr->expression < 0) {
-                // The error does not refer to a particular expression.
-                cerr << "ERROR: " << compileErr->message << endl;
-            } else {
-                cerr << "ERROR: Pattern '"
-                     << patterns[compileErr->expression]
-                     << "' failed with error '" << compileErr->message << "'"
-                     << endl;
-            }
-            // As the compileErr pointer points to dynamically allocated memory,
-            // if we get an error, we must be sure to release it. This is not
-            // necessary when no error is detected.
-            hs_free_compile_error(compileErr);
-            exit(-1);
-        }
-
-        *compileTime = clock.seconds();
-        return db;
-    }
-
-    unsigned size() const {
-        return patterns.size();
-    }
-
-    Sigdata cloneExclude(const set<unsigned> &excludeIndexSet) const {
-        Sigdata c;
-        for (unsigned i = 0, e = size(); i != e; ++i) {
-            if (excludeIndexSet.find(i) == excludeIndexSet.end()) {
-                c.flags.push_back(flags[i]);
-                c.ids.push_back(ids[i]);
-                c.patterns.push_back(patterns[i]);
-                c.originals.push_back(originals[i]);
-            }
-        }
-        return c;
-    }
-};
-
-static
-void usage(const char *) {
-    cerr << "Usage:" << endl << endl;
-    cerr << "  patbench [-n repeats] [ -G generations] [ -C criterion ]" << endl
-         << "           [ -F factor_group_size ] [ -N | -S ] "
-         << "<pattern file> <pcap file>" << endl << endl
-         << "    -n repeats sets the number of times the PCAP is repeatedly "
-            "scanned" << endl << "       with the pattern." << endl
-         << "    -G generations sets the number of generations that the "
-            "algorithm is" << endl << "       run for." << endl
-         << "    -N sets non-streaming mode, -S sets streaming mode (default)."
-         << endl << "    -F sets the factor group size (must be >0); this "
-                    "allows the detection" << endl
-         << "       of multiple interacting factors." << endl << "" << endl
-         << "    -C sets the 'criterion', which can be either:" << endl
-         << "         t  throughput (the default) - this requires a pcap file"
-         << endl << "         r  scratch size" << endl
-         << "         s  stream state size" << endl
-         << "         c  compile time" << endl << "         b  bytecode size"
-         << endl << endl
-         << "We recommend the use of a utility like 'taskset' on "
-            "multiprocessor hosts to" << endl
-         << "lock execution to a single processor: this will remove processor "
-            "migration" << endl
-         << "by the scheduler as a source of noise in the results." << endl;
-}
-
-static
-double measure_stream_time(Benchmark &bench, unsigned int repeatCount) {
-    Clock clock;
-    bench.clearMatches();
-    clock.start();
-    for (unsigned int i = 0; i < repeatCount; i++) {
-        bench.openStreams();
-        bench.scanStreams();
-        bench.closeStreams();
-    }
-    clock.stop();
-    double secsScan = clock.seconds();
-    return secsScan;
-}
-
-static
-double measure_block_time(Benchmark &bench, unsigned int repeatCount) {
-    Clock clock;
-    bench.clearMatches();
-    clock.start();
-    for (unsigned int i = 0; i < repeatCount; i++) {
-        bench.scanBlock();
-    }
-    clock.stop();
-    double secsScan = clock.seconds();
-    return secsScan;
-}
-
-static
-double eval_set(Benchmark &bench, Sigdata &sigs, unsigned int mode,
-                unsigned repeatCount, Criterion criterion,
-                bool diagnose = true) {
-    double compileTime = 0;
-    bench.setDatabase(sigs.compileDatabase(mode, &compileTime));
-
-    switch (criterion) {
-    case CRITERION_BYTECODE_SIZE: {
-        size_t dbSize;
-        hs_error_t err = hs_database_size(bench.getDatabase(), &dbSize);
-        if (err != HS_SUCCESS) {
-            cerr << "ERROR: could not retrieve bytecode size" << endl;
-            exit(1);
-        }
-        return dbSize;
-    }
-    case CRITERION_COMPILE_TIME:
-        return compileTime;
-    case CRITERION_STREAM_STATE: {
-        size_t streamStateSize;
-        hs_error_t err = hs_stream_size(bench.getDatabase(), &streamStateSize);
-        if (err != HS_SUCCESS) {
-            cerr << "ERROR: could not retrieve stream state size" << endl;
-            exit(1);
-        }
-        return streamStateSize;
-    }
-    case CRITERION_SCRATCH_SIZE:
-        return bench.getScratchSize();
-    case CRITERION_THROUGHPUT:
-    default:
-        break; // do nothing - we are THROUGHPUT
-    }
-    double scan_time;
-    if (mode == HS_MODE_NOSTREAM) {
-        scan_time = measure_block_time(bench, repeatCount);
-    } else {
-        scan_time = measure_stream_time(bench, repeatCount);
-    }
-    size_t bytes = bench.bytes();
-    size_t matches = bench.matches();
-    if (diagnose) {
-        std::ios::fmtflags f(cout.flags());
-        cout << "Scan time " << std::fixed << std::setprecision(3) << scan_time
-             << " sec, Scanned " << bytes * repeatCount << " bytes, Throughput "
-             << std::fixed << std::setprecision(3)
-             << (bytes * 8 * repeatCount) / (scan_time * 1000000)
-             << " Mbps, Matches " << matches << endl;
-        cout.flags(f);
-    }
-    return (bytes * 8 * repeatCount) / (scan_time * 1000000);
-}
-
-// Main entry point.
-int main(int argc, char **argv) {
-    unsigned int repeatCount = 1;
-    unsigned int mode = HS_MODE_STREAM;
-    Criterion criterion = CRITERION_THROUGHPUT;
-    unsigned int gen_max = 10;
-    unsigned int factor_max = 1;
-    // Process command line arguments.
-    int opt;
-    while ((opt = getopt(argc, argv, "SNn:G:F:C:")) != -1) {
-        switch (opt) {
-        case 'F':
-            factor_max = atoi(optarg);
-            break;
-        case 'G':
-            gen_max = atoi(optarg);
-            break;
-        case 'S':
-            mode = HS_MODE_STREAM;
-            break;
-        case 'N':
-            mode = HS_MODE_NOSTREAM;
-            break;
-        case 'C':
-            switch (optarg[0]) {
-            case 't':
-                criterion = CRITERION_THROUGHPUT;
-                break;
-            case 'b':
-                criterion = CRITERION_BYTECODE_SIZE;
-                break;
-            case 'c':
-                criterion = CRITERION_COMPILE_TIME;
-                break;
-            case 's':
-                criterion = CRITERION_STREAM_STATE;
-                break;
-            case 'r':
-                criterion = CRITERION_SCRATCH_SIZE;
-                break;
-            default:
-                cerr << "Unrecognised criterion: " << optarg[0] << endl;
-                usage(argv[0]);
-                exit(-1);
-            }
-            break;
-        case 'n':
-            repeatCount = atoi(optarg);
-            break;
-        default:
-            usage(argv[0]);
-            exit(-1);
-        }
-    }
-
-    if (argc - optind != ((criterion == CRITERION_THROUGHPUT) ? 2 : 1)) {
-        usage(argv[0]);
-        exit(-1);
-    }
-
-    const char *patternFile = argv[optind];
-    const char *pcapFile = argv[optind + 1];
-
-    // Read our input PCAP file in
-    Benchmark bench;
-    if (criterion == CRITERION_THROUGHPUT) {
-        if (!bench.readStreams(pcapFile)) {
-            cerr << "Unable to read packets from PCAP file. Exiting." << endl;
-            exit(-1);
-        }
-    }
-
-    if ((criterion == CRITERION_STREAM_STATE) && (mode != HS_MODE_STREAM)) {
-        cerr << "Cannot evaluate stream state for block mode compile. Exiting."
-             << endl;
-        exit(-1);
-    }
-
-    cout << "Base signatures: " << patternFile;
-    if (pcapFile) {
-        cout << "\tPCAP input file: " << pcapFile
-             << "\tRepeat count: " << repeatCount;
-    }
-    if (mode == HS_MODE_STREAM) {
-        cout << "\tMode: streaming";
-    } else {
-        cout << "\tMode: block";
-    }
-    cout << endl;
-
-    Sigdata sigs(patternFile);
-
-    // calculate and show a baseline
-    eval_set(bench, sigs, mode, repeatCount, criterion);
-
-    set<unsigned> work_sigs, exclude;
-
-    for (unsigned i = 0; i < sigs.size(); ++i) {
-        work_sigs.insert(i);
-    }
-
-    double score_base =
-        eval_set(bench, sigs, mode, repeatCount, criterion, false);
-    bool maximize = higher_is_better(criterion);
-    cout << "Number of signatures: " << sigs.size() << endl;
-    cout << "Base performance: ";
-    print_criterion(criterion, score_base);
-    cout << endl;
-
-    unsigned generations = min(gen_max, (sigs.size() - 1) / factor_max);
-
-    cout << "Cutting signatures cumulatively for " << generations
-         << " generations" << endl;
-    for (unsigned gen = 0; gen < generations; ++gen) {
-        cout << "Generation " << gen << " ";
-        set<unsigned> s(work_sigs.begin(), work_sigs.end());
-        double best = maximize ? 0 : 1000000000000.0;
-        unsigned count = 0;
-        while (s.size() > factor_max) {
-            count++;
-            cout << "." << std::flush;
-            vector<unsigned> sv(s.begin(), s.end());
-            random_device rng;
-            mt19937 urng(rng());
-            shuffle(sv.begin(), sv.end(), urng);
-            unsigned groups = factor_max + 1;
-            for (unsigned current_group = 0; current_group < groups;
-                 current_group++) {
-                unsigned sz = sv.size();
-                unsigned lo = (current_group * sz) / groups;
-                unsigned hi = ((current_group + 1) * sz) / groups;
-
-                set<unsigned> s_part1(sv.begin(), sv.begin() + lo);
-                set<unsigned> s_part2(sv.begin() + hi, sv.end());
-                set<unsigned> s_tmp = s_part1;
-                s_tmp.insert(s_part2.begin(), s_part2.end());
-                set<unsigned> tmp = s_tmp;
-                tmp.insert(exclude.begin(), exclude.end());
-                Sigdata sigs_tmp = sigs.cloneExclude(tmp);
-                double score = eval_set(bench, sigs_tmp, mode, repeatCount,
-                                        criterion, false);
-
-                if ((current_group == 0) ||
-                    (!maximize ? (score < best) : (score > best))) {
-                    s = s_tmp;
-                    best = score;
-                }
-            }
-        }
-        for (unsigned i = count; i < 16; i++) {
-            cout << " ";
-        }
-        std::ios::fmtflags out_f(cout.flags());
-        cout << "Performance: ";
-        print_criterion(criterion, best);
-        cout << " (" << std::fixed << std::setprecision(3) << (best / score_base)
-             << "x) after cutting:" << endl;
-        cout.flags(out_f);
-
-        // s now has factor_max signatures
-        for (const auto &found : s) {
-            exclude.insert(found);
-            work_sigs.erase(found);
-            cout << sigs.get_original(found) << endl;
-        }
-
-        cout << endl;
-    }
-    return 0;
-}
-
-/**
- * Helper function to locate the offset of the first byte of the payload in the
- * given ethernet frame. Offset into the packet, and the length of the payload
- * are returned in the arguments @a offset and @a length.
- */
-static
-bool payloadOffset(const unsigned char *pkt_data, unsigned int *offset,
-                   unsigned int *length) {
-    const ip *iph = (const ip *)(pkt_data + sizeof(ether_header));
-    const tcphdr *th = nullptr;
-
-    // Ignore packets that aren't IPv4
-    if (iph->ip_v != 4) {
-        return false;
-    }
-
-    // Ignore fragmented packets.
-    if (iph->ip_off & htons(IP_MF | IP_OFFMASK)) {
-        return false;
-    }
-
-    // IP header length, and transport header length.
-    unsigned int ihlen = iph->ip_hl * 4;
-    unsigned int thlen = 0;
-
-    switch (iph->ip_p) {
-    case IPPROTO_TCP:
-        th = (const tcphdr *)((const char *)iph + ihlen);
-        thlen = th->th_off * 4;
-        break;
-    case IPPROTO_UDP:
-        thlen = sizeof(udphdr);
-        break;
-    default:
-        return false;
-    }
-
-    *offset = sizeof(ether_header) + ihlen + thlen;
-    *length = sizeof(ether_header) + ntohs(iph->ip_len) - *offset;
-
-    return *length != 0;
-}
-
-static unsigned parseFlags(const string &flagsStr) {
-    unsigned flags = 0;
-    for (const auto &c : flagsStr) {
-        switch (c) {
-        case 'i':
-            flags |= HS_FLAG_CASELESS; break;
-        case 'm':
-            flags |= HS_FLAG_MULTILINE; break;
-        case 's':
-            flags |= HS_FLAG_DOTALL; break;
-        case 'H':
-            flags |= HS_FLAG_SINGLEMATCH; break;
-        case 'V':
-            flags |= HS_FLAG_ALLOWEMPTY; break;
-        case '8':
-            flags |= HS_FLAG_UTF8; break;
-        case 'W':
-            flags |= HS_FLAG_UCP; break;
-        case '\r': // stray carriage-return
-            break;
-        default:
-            cerr << "Unsupported flag \'" << c << "\'" << endl;
-            exit(-1);
-        }
-    }
-    return flags;
-}
-
-static void parseFile(const char *filename, vector<string> &patterns,
-                      vector<unsigned> &flags, vector<unsigned> &ids,
-                      vector<string> &originals) {
-    ifstream inFile(filename);
-    if (!inFile.good()) {
-        cerr << "ERROR: Can't open pattern file \"" << filename << "\"" << endl;
-        exit(-1);
-    }
-
-    for (unsigned i = 1; !inFile.eof(); ++i) {
-        string line;
-        getline(inFile, line);
-
-        // if line is empty, or a comment, we can skip it
-        if (line.empty() || line[0] == '#') {
-            continue;
-        }
-
-        // otherwise, it should be ID:PCRE, e.g.
-        //  10001:/foobar/is
-
-        size_t colonIdx = line.find_first_of(':');
-        if (colonIdx == string::npos) {
-            cerr << "ERROR: Could not parse line " << i << endl;
-            exit(-1);
-        }
-
-        // we should have an unsigned int as an ID, before the colon
-        unsigned id = std::stoi(line.substr(0, colonIdx).c_str());
-
-        // rest of the expression is the PCRE
-        const string expr(line.substr(colonIdx + 1));
-
-        size_t flagsStart = expr.find_last_of('/');
-        if (flagsStart == string::npos) {
-            cerr << "ERROR: no trailing '/' char" << endl;
-            exit(-1);
-        }
-
-        string pcre(expr.substr(1, flagsStart - 1));
-        string flagsStr(expr.substr(flagsStart + 1, expr.size() - flagsStart));
-        unsigned flag = parseFlags(flagsStr);
-
-        originals.push_back(line);
-        patterns.push_back(pcre);
-        flags.push_back(flag);
-        ids.push_back(id);
-    }
-}

examples/pcapscan.cc

@@ -1,681 +0,0 @@
-/*
- * Copyright (c) 2015-2016, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/*
- * Hyperscan example program 2: pcapscan
- *
- * This example is a very simple packet scanning benchmark. It scans a given
- * PCAP file full of network traffic against a group of regular expressions and
- * returns some coarse performance measurements.  This example provides a quick
- * way to examine the performance achievable on a particular combination of
- * platform, pattern set and input data.
- *
- * Build instructions:
- *
- *     g++ -std=c++11 -O2 -o pcapscan pcapscan.cc $(pkg-config --cflags --libs libhs) -lpcap
- *
- * Usage:
- *
- *     ./pcapscan [-n repeats] <pattern file> <pcap file>
- *
- * We recommend the use of a utility like 'taskset' on multiprocessor hosts to
- * pin execution to a single processor: this will remove processor migration
- * by the scheduler as a source of noise in the results.
- *
- */
-
-#include <cstring>
-#include <chrono>
-#include <fstream>
-#include <iomanip>
-#include <iostream>
-#include <string>
-#include <unordered_map>
-#include <vector>
-
-#include <unistd.h>
-
-// We use the BSD primitives throughout as they exist on both BSD and Linux.
-#define __FAVOR_BSD
-#include <netinet/in.h>
-#include <netinet/in_systm.h>
-#include <netinet/ip.h>
-#include <netinet/tcp.h>
-#include <netinet/udp.h>
-#include <netinet/ip_icmp.h>
-#include <net/ethernet.h>
-#include <arpa/inet.h>
-
-#include <pcap.h>
-
-#include <hs.h>
-
-using std::cerr;
-using std::cout;
-using std::endl;
-using std::ifstream;
-using std::string;
-using std::unordered_map;
-using std::vector;
-
-// Key for identifying a stream in our pcap input data, using data from its IP
-// headers.
-struct FiveTuple {
-    unsigned int protocol;
-    unsigned int srcAddr;
-    unsigned int srcPort;
-    unsigned int dstAddr;
-    unsigned int dstPort;
-
-    // Construct a FiveTuple from a TCP or UDP packet.
-    FiveTuple(const struct ip *iphdr) {
-        // IP fields
-        protocol = iphdr->ip_p;
-        srcAddr = iphdr->ip_src.s_addr;
-        dstAddr = iphdr->ip_dst.s_addr;
-
-        // UDP/TCP ports
-        const struct udphdr *uh =
-            (const struct udphdr *)(((const char *)iphdr) + (iphdr->ip_hl * 4));
-        srcPort = uh->uh_sport;
-        dstPort = uh->uh_dport;
-    }
-
-    bool operator==(const FiveTuple &a) const {
-        return protocol == a.protocol && srcAddr == a.srcAddr &&
-               srcPort == a.srcPort && dstAddr == a.dstAddr &&
-               dstPort == a.dstPort;
-    }
-};
-
-// A *very* simple hash function, used when we create an unordered_map of
-// FiveTuple objects.
-struct FiveTupleHash {
-    size_t operator()(const FiveTuple &x) const {
-        return x.srcAddr ^ x.dstAddr ^ x.protocol ^ x.srcPort ^ x.dstPort;
-    }
-};
-
-// Helper function. See end of file.
-static bool payloadOffset(const unsigned char *pkt_data, unsigned int *offset,
-                          unsigned int *length);
-
-// Match event handler: called every time Hyperscan finds a match.
-static
-int onMatch(unsigned int id, unsigned long long from, unsigned long long to,
-            unsigned int flags, void *ctx) {
-    // Our context points to a size_t storing the match count
-    size_t *matches = (size_t *)ctx;
-    (*matches)++;
-    return 0; // continue matching
-}
-
-// Simple timing class
-class Clock {
-public:
-    void start() {
-        time_start = std::chrono::system_clock::now();
-    }
-
-    void stop() {
-        time_end = std::chrono::system_clock::now();
-    }
-
-    double seconds() const {
-        std::chrono::duration<double> delta = time_end - time_start;
-        return delta.count();
-    }
-private:
-    std::chrono::time_point<std::chrono::system_clock> time_start, time_end;
-};
-
-// Class wrapping all state associated with the benchmark
-class Benchmark {
-private:
-    // Packet data to be scanned.
-    vector<string> packets;
-
-    // The stream ID to which each packet belongs
-    vector<size_t> stream_ids;
-
-    // Map used to construct stream_ids
-    unordered_map<FiveTuple, size_t, FiveTupleHash> stream_map;
-
-    // Hyperscan compiled database (streaming mode)
-    const hs_database_t *db_streaming;
-
-    // Hyperscan compiled database (block mode)
-    const hs_database_t *db_block;
-
-    // Hyperscan temporary scratch space (used in both modes)
-    hs_scratch_t *scratch;
-
-    // Vector of Hyperscan stream state (used in streaming mode)
-    vector<hs_stream_t *> streams;
-
-    // Count of matches found during scanning
-    size_t matchCount;
-
-public:
-    Benchmark(const hs_database_t *streaming, const hs_database_t *block)
-        : db_streaming(streaming), db_block(block), scratch(nullptr),
-          matchCount(0) {
-        // Allocate enough scratch space to handle either streaming or block
-        // mode, so we only need the one scratch region.
-        hs_error_t err = hs_alloc_scratch(db_streaming, &scratch);
-        if (err != HS_SUCCESS) {
-            cerr << "ERROR: could not allocate scratch space. Exiting." << endl;
-            exit(-1);
-        }
-        // This second call will increase the scratch size if more is required
-        // for block mode.
-        err = hs_alloc_scratch(db_block, &scratch);
-        if (err != HS_SUCCESS) {
-            cerr << "ERROR: could not allocate scratch space. Exiting." << endl;
-            exit(-1);
-        }
-    }
-
-    ~Benchmark() {
-        // Free scratch region
-        hs_free_scratch(scratch);
-    }
-
-    // Read a set of streams from a pcap file
-    bool readStreams(const char *pcapFile) {
-        // Open PCAP file for input
-        char errbuf[PCAP_ERRBUF_SIZE];
-        pcap_t *pcapHandle = pcap_open_offline(pcapFile, errbuf);
-        if (pcapHandle == nullptr) {
-            cerr << "ERROR: Unable to open pcap file \"" << pcapFile
-                << "\": " << errbuf << endl;
-            return false;
-        }
-
-        struct pcap_pkthdr pktHeader;
-        const unsigned char *pktData;
-        while ((pktData = pcap_next(pcapHandle, &pktHeader)) != nullptr) {
-            unsigned int offset = 0, length = 0;
-            if (!payloadOffset(pktData, &offset, &length)) {
-                continue;
-            }
-
-            // Valid TCP or UDP packet
-            const struct ip *iphdr = (const struct ip *)(pktData
-                    + sizeof(struct ether_header));
-            const char *payload = (const char *)pktData + offset;
-
-            size_t id = stream_map.insert(std::make_pair(FiveTuple(iphdr),
-                                          stream_map.size())).first->second;
-
-            packets.push_back(string(payload, length));
-            stream_ids.push_back(id);
-        }
-        pcap_close(pcapHandle);
-
-        return !packets.empty();
-    }
-
-    // Return the number of bytes scanned
-    size_t bytes() const {
-        size_t sum = 0;
-        for (const auto &packet : packets) {
-            sum += packet.size();
-        }
-        return sum;
-    }
-
-    // Return the number of matches found.
-    size_t matches() const {
-        return matchCount;
-    }
-
-    // Clear the number of matches found.
-    void clearMatches() {
-        matchCount = 0;
-    }
-
-    // Open a Hyperscan stream for each stream in stream_ids
-    void openStreams() {
-        streams.resize(stream_map.size());
-        for (auto &stream : streams) {
-            hs_error_t err = hs_open_stream(db_streaming, 0, &stream);
-            if (err != HS_SUCCESS) {
-                cerr << "ERROR: Unable to open stream. Exiting." << endl;
-                exit(-1);
-            }
-        }
-    }
-
-    // Close all open Hyperscan streams (potentially generating any
-    // end-anchored matches)
-    void closeStreams() {
-        for (auto &stream : streams) {
-            hs_error_t err = hs_close_stream(stream, scratch, onMatch,
-                                             &matchCount);
-            if (err != HS_SUCCESS) {
-                cerr << "ERROR: Unable to close stream. Exiting." << endl;
-                exit(-1);
-            }
-        }
-    }
-
-    // Scan each packet (in the ordering given in the PCAP file) through
-    // Hyperscan using the streaming interface.
-    void scanStreams() {
-        for (size_t i = 0; i != packets.size(); ++i) {
-            const std::string &pkt = packets[i];
-            hs_error_t err = hs_scan_stream(streams[stream_ids[i]],
-                                            pkt.c_str(), pkt.length(), 0,
-                                            scratch, onMatch, &matchCount);
-            if (err != HS_SUCCESS) {
-                cerr << "ERROR: Unable to scan packet. Exiting." << endl;
-                exit(-1);
-            }
-        }
-    }
-
-    // Scan each packet (in the ordering given in the PCAP file) through
-    // Hyperscan using the block-mode interface.
-    void scanBlock() {
-        for (size_t i = 0; i != packets.size(); ++i) {
-            const std::string &pkt = packets[i];
-            hs_error_t err = hs_scan(db_block, pkt.c_str(), pkt.length(), 0,
-                                     scratch, onMatch, &matchCount);
-            if (err != HS_SUCCESS) {
-                cerr << "ERROR: Unable to scan packet. Exiting." << endl;
-                exit(-1);
-            }
-        }
-    }
-
-    // Display some information about the compiled database and scanned data.
-    void displayStats() {
-        size_t numPackets = packets.size();
-        size_t numStreams = stream_map.size();
-        size_t numBytes = bytes();
-        hs_error_t err;
-
-        cout << numPackets << " packets in " << numStreams
-             << " streams, totalling " << numBytes << " bytes." << endl;
-        cout << "Average packet length: " << numBytes / numPackets << " bytes."
-             << endl;
-        cout << "Average stream length: " << numBytes / numStreams << " bytes."
-             << endl;
-        cout << endl;
-
-        size_t dbStream_size = 0;
-        err = hs_database_size(db_streaming, &dbStream_size);
-        if (err == HS_SUCCESS) {
-            cout << "Streaming mode Hyperscan database size    : "
-                 << dbStream_size << " bytes." << endl;
-        } else {
-            cout << "Error getting streaming mode Hyperscan database size"
-                 << endl;
-        }
-
-        size_t dbBlock_size = 0;
-        err = hs_database_size(db_block, &dbBlock_size);
-        if (err == HS_SUCCESS) {
-            cout << "Block mode Hyperscan database size        : "
-                 << dbBlock_size << " bytes." << endl;
-        } else {
-            cout << "Error getting block mode Hyperscan database size"
-                 << endl;
-        }
-
-        size_t stream_size = 0;
-        err = hs_stream_size(db_streaming, &stream_size);
-        if (err == HS_SUCCESS) {
-            cout << "Streaming mode Hyperscan stream state size: "
-                 << stream_size << " bytes (per stream)." << endl;
-        } else {
-            cout << "Error getting stream state size" << endl;
-        }
-    }
-};
-
-// helper function - see end of file
-static void parseFile(const char *filename, vector<string> &patterns,
-                      vector<unsigned> &flags, vector<unsigned> &ids);
-
-static hs_database_t *buildDatabase(const vector<const char *> &expressions,
-                                    const vector<unsigned> flags,
-                                    const vector<unsigned> ids,
-                                    unsigned int mode) {
-    hs_database_t *db;
-    hs_compile_error_t *compileErr;
-    hs_error_t err;
-
-    Clock clock;
-    clock.start();
-
-    err = hs_compile_multi(expressions.data(), flags.data(), ids.data(),
-                           expressions.size(), mode, nullptr, &db, &compileErr);
-
-    clock.stop();
-
-    if (err != HS_SUCCESS) {
-        if (compileErr->expression < 0) {
-            // The error does not refer to a particular expression.
-            cerr << "ERROR: " << compileErr->message << endl;
-        } else {
-            cerr << "ERROR: Pattern '" << expressions[compileErr->expression]
-                 << "' failed compilation with error: " << compileErr->message
-                 << endl;
-        }
-        // As the compileErr pointer points to dynamically allocated memory, if
-        // we get an error, we must be sure to release it. This is not
-        // necessary when no error is detected.
-        hs_free_compile_error(compileErr);
-        exit(-1);
-    }
-
-    cout << "Hyperscan " << (mode == HS_MODE_STREAM ? "streaming" : "block")
-         << " mode database compiled in " << clock.seconds() << " seconds."
-         << endl;
-
-    return db;
-}
-
-/**
- * This function will read in the file with the specified name, with an
- * expression per line, ignoring lines starting with '#' and build a Hyperscan
- * database for it.
- */
-static void databasesFromFile(const char *filename,
-                              hs_database_t **db_streaming,
-                              hs_database_t **db_block) {
-    // hs_compile_multi requires three parallel arrays containing the patterns,
-    // flags and ids that we want to work with. To achieve this we use
-    // vectors and new entries onto each for each valid line of input from
-    // the pattern file.
-    vector<string> patterns;
-    vector<unsigned> flags;
-    vector<unsigned> ids;
-
-    // do the actual file reading and string handling
-    parseFile(filename, patterns, flags, ids);
-
-    // Turn our vector of strings into a vector of char*'s to pass in to
-    // hs_compile_multi. (This is just using the vector of strings as dynamic
-    // storage.)
-    vector<const char*> cstrPatterns;
-    for (const auto &pattern : patterns) {
-        cstrPatterns.push_back(pattern.c_str());
-    }
-
-    cout << "Compiling Hyperscan databases with " << patterns.size()
-         << " patterns." << endl;
-
-    *db_streaming = buildDatabase(cstrPatterns, flags, ids, HS_MODE_STREAM);
-    *db_block = buildDatabase(cstrPatterns, flags, ids, HS_MODE_BLOCK);
-}
-
-static void usage(const char *prog) {
-    cerr << "Usage: " << prog << " [-n repeats] <pattern file> <pcap file>" << endl;
-}
-
-// Main entry point.
-int main(int argc, char **argv) {
-    unsigned int repeatCount = 1;
-
-    // Process command line arguments.
-    int opt;
-    while ((opt = getopt(argc, argv, "n:")) != -1) {
-        switch (opt) {
-        case 'n':
-            repeatCount = atoi(optarg);
-            break;
-        default:
-            usage(argv[0]);
-            exit(-1);
-        }
-    }
-
-    if (argc - optind != 2) {
-        usage(argv[0]);
-        exit(-1);
-    }
-
-    const char *patternFile = argv[optind];
-    const char *pcapFile = argv[optind + 1];
-
-    // Read our pattern set in and build Hyperscan databases from it.
-    cout << "Pattern file: " << patternFile << endl;
-    hs_database_t *db_streaming, *db_block;
-    databasesFromFile(patternFile, &db_streaming, &db_block);
-
-    // Read our input PCAP file in
-    Benchmark bench(db_streaming, db_block);
-    cout << "PCAP input file: " << pcapFile << endl;
-    if (!bench.readStreams(pcapFile)) {
-        cerr << "Unable to read packets from PCAP file. Exiting." << endl;
-        exit(-1);
-    }
-
-    if (repeatCount != 1) {
-        cout << "Repeating PCAP scan " << repeatCount << " times." << endl;
-    }
-
-    bench.displayStats();
-
-    Clock clock;
-
-    // Streaming mode scans.
-    double secsStreamingScan = 0.0, secsStreamingOpenClose = 0.0;
-    for (unsigned int i = 0; i < repeatCount; i++) {
-        // Open streams.
-        clock.start();
-        bench.openStreams();
-        clock.stop();
-        secsStreamingOpenClose += clock.seconds();
-
-        // Scan all our packets in streaming mode.
-        clock.start();
-        bench.scanStreams();
-        clock.stop();
-        secsStreamingScan += clock.seconds();
-
-        // Close streams.
-        clock.start();
-        bench.closeStreams();
-        clock.stop();
-        secsStreamingOpenClose += clock.seconds();
-    }
-
-    // Collect data from streaming mode scans.
-    size_t bytes = bench.bytes();
-    double tputStreamScanning = (bytes * 8 * repeatCount) / secsStreamingScan;
-    double tputStreamOverhead = (bytes * 8 * repeatCount) / (secsStreamingScan + secsStreamingOpenClose);
-    size_t matchesStream = bench.matches();
-    double matchRateStream = matchesStream / ((bytes * repeatCount) / 1024.0); // matches per kilobyte
-
-    // Scan all our packets in block mode.
-    bench.clearMatches();
-    clock.start();
-    for (unsigned int i = 0; i < repeatCount; i++) {
-        bench.scanBlock();
-    }
-    clock.stop();
-    double secsScanBlock = clock.seconds();
-
-    // Collect data from block mode scans.
-    double tputBlockScanning = (bytes * 8 * repeatCount) / secsScanBlock;
-    size_t matchesBlock = bench.matches();
-    double matchRateBlock = matchesBlock / ((bytes * repeatCount) / 1024.0); // matches per kilobyte
-
-    cout << endl << "Streaming mode:" << endl << endl;
-    cout << "  Total matches: " << matchesStream << endl;
-    cout << std::fixed << std::setprecision(4);
-    cout << "  Match rate:    " << matchRateStream << " matches/kilobyte" << endl;
-    cout << std::fixed << std::setprecision(2);
-    cout << "  Throughput (with stream overhead): "
-              << tputStreamOverhead/1000000 << " megabits/sec" << endl;
-    cout << "  Throughput (no stream overhead):   "
-              << tputStreamScanning/1000000 << " megabits/sec" << endl;
-
-    cout << endl << "Block mode:" << endl << endl;
-    cout << "  Total matches: " << matchesBlock << endl;
-    cout << std::fixed << std::setprecision(4);
-    cout << "  Match rate:    " << matchRateBlock << " matches/kilobyte" << endl;
-    cout << std::fixed << std::setprecision(2);
-    cout << "  Throughput:    "
-              << tputBlockScanning/1000000 << " megabits/sec" << endl;
-
-    cout << endl;
-    if (bytes < (2*1024*1024)) {
-        cout << endl << "WARNING: Input PCAP file is less than 2MB in size." << endl
-                  << "This test may have been too short to calculate accurate results." << endl;
-    }
-
-    // Close Hyperscan databases
-    hs_free_database(db_streaming);
-    hs_free_database(db_block);
-
-    return 0;
-}
-
-/**
- * Helper function to locate the offset of the first byte of the payload in the
- * given ethernet frame. Offset into the packet, and the length of the payload
- * are returned in the arguments @a offset and @a length.
- */
-static bool payloadOffset(const unsigned char *pkt_data, unsigned int *offset,
-                          unsigned int *length) {
-    const ip *iph = (const ip *)(pkt_data + sizeof(ether_header));
-    const tcphdr *th = nullptr;
-
-    // Ignore packets that aren't IPv4
-    if (iph->ip_v != 4) {
-        return false;
-    }
-
-    // Ignore fragmented packets.
-    if (iph->ip_off & htons(IP_MF|IP_OFFMASK)) {
-        return false;
-    }
-
-    // IP header length, and transport header length.
-    unsigned int ihlen = iph->ip_hl * 4;
-    unsigned int thlen = 0;
-
-    switch (iph->ip_p) {
-    case IPPROTO_TCP:
-        th = (const tcphdr *)((const char *)iph + ihlen);
-        thlen = th->th_off * 4;
-        break;
-    case IPPROTO_UDP:
-        thlen = sizeof(udphdr);
-        break;
-    default:
-        return false;
-    }
-
-    *offset = sizeof(ether_header) + ihlen + thlen;
-    *length = sizeof(ether_header) + ntohs(iph->ip_len) - *offset;
-
-    return *length != 0;
-}
-
-static unsigned parseFlags(const string &flagsStr) {
-    unsigned flags = 0;
-    for (const auto &c : flagsStr) {
-        switch (c) {
-        case 'i':
-            flags |= HS_FLAG_CASELESS; break;
-        case 'm':
-            flags |= HS_FLAG_MULTILINE; break;
-        case 's':
-            flags |= HS_FLAG_DOTALL; break;
-        case 'H':
-            flags |= HS_FLAG_SINGLEMATCH; break;
-        case 'V':
-            flags |= HS_FLAG_ALLOWEMPTY; break;
-        case '8':
-            flags |= HS_FLAG_UTF8; break;
-        case 'W':
-            flags |= HS_FLAG_UCP; break;
-        case '\r': // stray carriage-return
-            break;
-        default:
-            cerr << "Unsupported flag \'" << c << "\'" << endl;
-            exit(-1);
-        }
-    }
-    return flags;
-}
-
-static void parseFile(const char *filename, vector<string> &patterns,
-                      vector<unsigned> &flags, vector<unsigned> &ids) {
-    ifstream inFile(filename);
-    if (!inFile.good()) {
-        cerr << "ERROR: Can't open pattern file \"" << filename << "\"" << endl;
-        exit(-1);
-    }
-
-    for (unsigned i = 1; !inFile.eof(); ++i) {
-        string line;
-        getline(inFile, line);
-
-        // if line is empty, or a comment, we can skip it
-        if (line.empty() || line[0] == '#') {
-            continue;
-        }
-
-        // otherwise, it should be ID:PCRE, e.g.
-        //  10001:/foobar/is
-
-        size_t colonIdx = line.find_first_of(':');
-        if (colonIdx == string::npos) {
-            cerr << "ERROR: Could not parse line " << i << endl;
-            exit(-1);
-        }
-
-        // we should have an unsigned int as an ID, before the colon
-        unsigned id = std::stoi(line.substr(0, colonIdx).c_str());
-
-        // rest of the expression is the PCRE
-        const string expr(line.substr(colonIdx + 1));
-
-        size_t flagsStart = expr.find_last_of('/');
-        if (flagsStart == string::npos) {
-            cerr << "ERROR: no trailing '/' char" << endl;
-            exit(-1);
-        }
-
-        string pcre(expr.substr(1, flagsStart - 1));
-        string flagsStr(expr.substr(flagsStart + 1, expr.size() - flagsStart));
-        unsigned flag = parseFlags(flagsStr);
-
-        patterns.push_back(pcre);
-        flags.push_back(flag);
-        ids.push_back(id);
-    }
-}
-

examples/simplegrep.c

@@ -1,221 +0,0 @@
-/*
- * Copyright (c) 2015, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/*
- * Hyperscan example program 1: simplegrep
- *
- * This is a simple example of Hyperscan's most basic functionality: it will
- * search a given input file for a pattern supplied as a command-line argument.
- * It is intended to demonstrate correct usage of the hs_compile and hs_scan
- * functions of Hyperscan.
- *
- * Patterns are scanned in 'DOTALL' mode, which is equivalent to PCRE's '/s'
- * modifier. This behaviour can be changed by modifying the "flags" argument to
- * hs_compile.
- *
- * Build instructions:
- *
- *     gcc -o simplegrep simplegrep.c $(pkg-config --cflags --libs libhs)
- *
- * Usage:
- *
- *     ./simplegrep <pattern> <input file>
- *
- * Example:
- *
- *     ./simplegrep int simplegrep.c
- *
- */
-
-#include <errno.h>
-#include <limits.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-#include <hs.h>
-
-/**
- * This is the function that will be called for each match that occurs. @a ctx
- * is to allow you to have some application-specific state that you will get
- * access to for each match. In our simple example we're just going to use it
- * to pass in the pattern that was being searched for so we can print it out.
- */
-static int eventHandler(unsigned int id, unsigned long long from,
-                        unsigned long long to, unsigned int flags, void *ctx) {
-    printf("Match for pattern \"%s\" at offset %llu\n", (char *)ctx, to);
-    return 0;
-}
-
-/**
- * Fill a data buffer from the given filename, returning it and filling @a
- * length with its length. Returns NULL on failure.
- */
-static char *readInputData(const char *inputFN, unsigned int *length) {
-    FILE *f = fopen(inputFN, "rb");
-    if (!f) {
-        fprintf(stderr, "ERROR: unable to open file \"%s\": %s\n", inputFN,
-                strerror(errno));
-        return NULL;
-    }
-
-    /* We use fseek/ftell to get our data length, in order to keep this example
-     * code as portable as possible. */
-    if (fseek(f, 0, SEEK_END) != 0) {
-        fprintf(stderr, "ERROR: unable to seek file \"%s\": %s\n", inputFN,
-                strerror(errno));
-        fclose(f);
-        return NULL;
-    }
-    long dataLen = ftell(f);
-    if (dataLen < 0) {
-        fprintf(stderr, "ERROR: ftell() failed: %s\n", strerror(errno));
-        fclose(f);
-        return NULL;
-    }
-    if (fseek(f, 0, SEEK_SET) != 0) {
-        fprintf(stderr, "ERROR: unable to seek file \"%s\": %s\n", inputFN,
-                strerror(errno));
-        fclose(f);
-        return NULL;
-    }
-
-    /* Hyperscan's hs_scan function accepts length as an unsigned int, so we
-     * limit the size of our buffer appropriately. */
-    if ((unsigned long)dataLen > UINT_MAX) {
-        dataLen = UINT_MAX;
-        printf("WARNING: clipping data to %ld bytes\n", dataLen);
-    } else if (dataLen == 0) {
-        fprintf(stderr, "ERROR: input file \"%s\" is empty\n", inputFN);
-        fclose(f);
-        return NULL;
-    }
-
-    char *inputData = malloc(dataLen);
-    if (!inputData) {
-        fprintf(stderr, "ERROR: unable to malloc %ld bytes\n", dataLen);
-        fclose(f);
-        return NULL;
-    }
-
-    char *p = inputData;
-    size_t bytesLeft = dataLen;
-    while (bytesLeft) {
-        size_t bytesRead = fread(p, 1, bytesLeft, f);
-        bytesLeft -= bytesRead;
-        p += bytesRead;
-        if (ferror(f) != 0) {
-            fprintf(stderr, "ERROR: fread() failed\n");
-            free(inputData);
-            fclose(f);
-            return NULL;
-        }
-    }
-
-    fclose(f);
-
-    *length = (unsigned int)dataLen;
-    return inputData;
-}
-
-int main(int argc, char *argv[]) {
-    if (argc != 3) {
-        fprintf(stderr, "Usage: %s <pattern> <input file>\n", argv[0]);
-        return -1;
-    }
-
-    char *pattern = argv[1];
-    char *inputFN = argv[2];
-
-    /* First, we attempt to compile the pattern provided on the command line.
-     * We assume 'DOTALL' semantics, meaning that the '.' meta-character will
-     * match newline characters. The compiler will analyse the given pattern and
-     * either return a compiled Hyperscan database, or an error message
-     * explaining why the pattern didn't compile.
-     */
-    hs_database_t *database;
-    hs_compile_error_t *compile_err;
-    if (hs_compile(pattern, HS_FLAG_DOTALL, HS_MODE_BLOCK, NULL, &database,
-                   &compile_err) != HS_SUCCESS) {
-        fprintf(stderr, "ERROR: Unable to compile pattern \"%s\": %s\n",
-                pattern, compile_err->message);
-        hs_free_compile_error(compile_err);
-        return -1;
-    }
-
-    /* Next, we read the input data file into a buffer. */
-    unsigned int length;
-    char *inputData = readInputData(inputFN, &length);
-    if (!inputData) {
-        hs_free_database(database);
-        return -1;
-    }
-
-    /* Finally, we issue a call to hs_scan, which will search the input buffer
-     * for the pattern represented in the bytecode. Note that in order to do
-     * this, scratch space needs to be allocated with the hs_alloc_scratch
-     * function. In typical usage, you would reuse this scratch space for many
-     * calls to hs_scan, but as we're only doing one, we'll be allocating it
-     * and deallocating it as soon as our matching is done.
-     *
-     * When matches occur, the specified callback function (eventHandler in
-     * this file) will be called. Note that although it is reminiscent of
-     * asynchronous APIs, Hyperscan operates synchronously: all matches will be
-     * found, and all callbacks issued, *before* hs_scan returns.
-     *
-     * In this example, we provide the input pattern as the context pointer so
-     * that the callback is able to print out the pattern that matched on each
-     * match event.
-     */
-    hs_scratch_t *scratch = NULL;
-    if (hs_alloc_scratch(database, &scratch) != HS_SUCCESS) {
-        fprintf(stderr, "ERROR: Unable to allocate scratch space. Exiting.\n");
-        free(inputData);
-        hs_free_database(database);
-        return -1;
-    }
-
-    printf("Scanning %u bytes with Hyperscan\n", length);
-
-    if (hs_scan(database, inputData, length, 0, scratch, eventHandler,
-                pattern) != HS_SUCCESS) {
-        fprintf(stderr, "ERROR: Unable to scan input buffer. Exiting.\n");
-        hs_free_scratch(scratch);
-        free(inputData);
-        hs_free_database(database);
-        return -1;
-    }
-
-    /* Scanning is complete, any matches have been handled, so now we just
-     * clean up and exit.
-     */
-    hs_free_scratch(scratch);
-    free(inputData);
-    hs_free_database(database);
-    return 0;
-}

hs.def

@@ -1,43 +0,0 @@
-; Hyperscan DLL export definitions
-
-LIBRARY hs
-
-EXPORTS
-   hs_alloc_scratch
-   hs_clone_scratch
-   hs_close_stream
-   hs_compile
-   hs_compile_ext_multi
-   hs_compile_multi
-   hs_compress_stream
-   hs_copy_stream
-   hs_database_info
-   hs_database_size
-   hs_deserialize_database
-   hs_deserialize_database_at
-   hs_expand_stream
-   hs_expression_ext_info
-   hs_expression_info
-   hs_free_compile_error
-   hs_free_database
-   hs_free_scratch
-   hs_open_stream
-   hs_populate_platform
-   hs_reset_and_copy_stream
-   hs_reset_and_expand_stream
-   hs_reset_stream
-   hs_scan
-   hs_scan_stream
-   hs_scan_vector
-   hs_scratch_size
-   hs_serialize_database
-   hs_serialized_database_info
-   hs_serialized_database_size
-   hs_set_allocator
-   hs_set_database_allocator
-   hs_set_misc_allocator
-   hs_set_scratch_allocator
-   hs_set_stream_allocator
-   hs_stream_size
-   hs_valid_platform
-   hs_version

hs_runtime.def

@@ -1,36 +0,0 @@
-; Hyperscan DLL export definitions
-
-LIBRARY hs_runtime
-
-EXPORTS
-   hs_alloc_scratch
-   hs_clone_scratch
-   hs_close_stream
-   hs_compress_stream
-   hs_copy_stream
-   hs_database_info
-   hs_database_size
-   hs_deserialize_database
-   hs_deserialize_database_at
-   hs_expand_stream
-   hs_free_database
-   hs_free_scratch
-   hs_open_stream
-   hs_reset_and_copy_stream
-   hs_reset_and_expand_stream
-   hs_reset_stream
-   hs_scan
-   hs_scan_stream
-   hs_scan_vector
-   hs_scratch_size
-   hs_serialize_database
-   hs_serialized_database_info
-   hs_serialized_database_size
-   hs_set_allocator
-   hs_set_database_allocator
-   hs_set_misc_allocator
-   hs_set_scratch_allocator
-   hs_set_stream_allocator
-   hs_stream_size
-   hs_valid_platform
-   hs_version

include/boost-patched/graph/dominator_tree.hpp

@@ -1,501 +0,0 @@
-//=======================================================================
-// Copyright (C) 2005-2009 Jongsoo Park <jongsoo.park -at- gmail.com>
-//
-// Distributed under the Boost Software License, Version 1.0.
-// (See accompanying file LICENSE_1_0.txt or copy at
-// http://www.boost.org/LICENSE_1_0.txt)
-//=======================================================================
-
-#ifndef BOOST_GRAPH_DOMINATOR_HPP
-#define BOOST_GRAPH_DOMINATOR_HPP
-
-#include <boost/config.hpp>
-#include <deque>
-#include <set>
-#include <boost/graph/depth_first_search.hpp>
-#include <boost/concept/assert.hpp>
-
-// Dominator tree computation
-
-// NOTE: This file contains modifications from the distributed Boost version to
-// correctly support supplying a vertex index map to the algorithm. To
-// differentiate it, it has been moved into the boost_ue2 namespace.
-
-namespace boost_ue2 {
-
-  using namespace boost;
-
-  namespace detail {
-    /**
-     * An extended time_stamper which also records vertices for each dfs number
-     */
-    template<class TimeMap, class VertexVector, class TimeT, class Tag>
-    class time_stamper_with_vertex_vector
-      : public base_visitor<
-      time_stamper_with_vertex_vector<TimeMap, VertexVector, TimeT, Tag> >
-    {
-    public :
-      typedef Tag event_filter;
-      time_stamper_with_vertex_vector(TimeMap timeMap, VertexVector& v,
-                                      TimeT& t)
-        : timeStamper_(timeMap, t), v_(v) { }
-
-      template<class Graph>
-      void
-      operator()(const typename property_traits<TimeMap>::key_type& v,
-                 const Graph& g)
-      {
-        timeStamper_(v, g);
-        v_[timeStamper_.m_time] = v;
-      }
-
-    private :
-      time_stamper<TimeMap, TimeT, Tag> timeStamper_;
-      VertexVector& v_;
-    };
-
-    /**
-     * A convenient way to create a time_stamper_with_vertex_vector
-     */
-    template<class TimeMap, class VertexVector, class TimeT, class Tag>
-    time_stamper_with_vertex_vector<TimeMap, VertexVector, TimeT, Tag>
-    stamp_times_with_vertex_vector(TimeMap timeMap, VertexVector& v, TimeT& t,
-                                   Tag)
-    {
-      return time_stamper_with_vertex_vector<TimeMap, VertexVector, TimeT,
-                                             Tag>(timeMap, v, t);
-    }
-
-    template<class Graph, class IndexMap, class TimeMap, class PredMap,
-             class DomTreePredMap>
-    class dominator_visitor
-    {
-      typedef typename graph_traits<Graph>::vertex_descriptor Vertex;
-      typedef typename graph_traits<Graph>::vertices_size_type VerticesSizeType;
-
-    public :
-      /**
-       * @param g [in] the target graph of the dominator tree
-       * @param entry [in] the entry node of g
-       * @param indexMap [in] the vertex index map for g
-       * @param domTreePredMap [out] the immediate dominator map
-       *                             (parent map in dominator tree)
-       */
-      dominator_visitor(const Graph& g, const Vertex& entry,
-                        const IndexMap& indexMap,
-                        DomTreePredMap domTreePredMap)
-        : semi_(num_vertices(g)),
-          ancestor_(num_vertices(g), graph_traits<Graph>::null_vertex()),
-          samedom_(ancestor_),
-          best_(semi_),
-          semiMap_(make_iterator_property_map(semi_.begin(),
-                                              indexMap)),
-          ancestorMap_(make_iterator_property_map(ancestor_.begin(),
-                                                  indexMap)),
-          bestMap_(make_iterator_property_map(best_.begin(),
-                                              indexMap)),
-          buckets_(num_vertices(g)),
-          bucketMap_(make_iterator_property_map(buckets_.begin(),
-                                                indexMap)),
-          entry_(entry),
-          domTreePredMap_(domTreePredMap),
-          numOfVertices_(num_vertices(g)),
-          samedomMap(make_iterator_property_map(samedom_.begin(),
-                                                indexMap))
-      {
-      }
-
-      void
-      operator()(const Vertex& n, const TimeMap& dfnumMap,
-                 const PredMap& parentMap, const Graph& g)
-      {
-        if (n == entry_) return;
-
-        const Vertex p(get(parentMap, n));
-        Vertex s(p);
-
-        // 1. Calculate the semidominator of n,
-        // based on the semidominator thm.
-        // * Semidominator thm. : To find the semidominator of a node n,
-        //   consider all predecessors v of n in the CFG (Control Flow Graph).
-        //  - If v is a proper ancestor of n in the spanning tree
-        //    (so dfnum(v) < dfnum(n)), then v is a candidate for semi(n)
-        //  - If v is a non-ancestor of n (so dfnum(v) > dfnum(n))
-        //    then for each u that is an ancestor of v (or u = v),
-        //    Let semi(u) be a candidate for semi(n)
-        //   of all these candidates, the one with lowest dfnum is
-        //   the semidominator of n.
-
-        // For each predecessor of n
-        typename graph_traits<Graph>::in_edge_iterator inItr, inEnd;
-        for (boost::tie(inItr, inEnd) = in_edges(n, g); inItr != inEnd; ++inItr)
-          {
-            const Vertex v = source(*inItr, g);
-            // To deal with unreachable nodes
-            if (get(dfnumMap, v) < 0 || get(dfnumMap, v) >= numOfVertices_)
-              continue;
-
-            Vertex s2;
-            if (get(dfnumMap, v) <= get(dfnumMap, n))
-              s2 = v;
-            else
-              s2 = get(semiMap_, ancestor_with_lowest_semi_(v, dfnumMap));
-
-            if (get(dfnumMap, s2) < get(dfnumMap, s))
-              s = s2;
-          }
-        put(semiMap_, n, s);
-
-        // 2. Calculation of n's dominator is deferred until
-        // the path from s to n has been linked into the forest
-        get(bucketMap_, s).push_back(n);
-        get(ancestorMap_, n) = p;
-        get(bestMap_, n) = n;
-
-        // 3. Now that the path from p to v has been linked into
-        // the spanning forest, these lines calculate the dominator of v,
-        // based on the dominator thm., or else defer the calculation
-        // until y's dominator is known
-        // * Dominator thm. : On the spanning-tree path below semi(n) and
-        //   above or including n, let y be the node
-        //   with the smallest-numbered semidominator. Then,
-        //
-        //  idom(n) = semi(n) if semi(y)=semi(n) or
-        //            idom(y) if semi(y) != semi(n)
-        typename std::deque<Vertex>::iterator buckItr;
-        for (buckItr = get(bucketMap_, p).begin();
-             buckItr != get(bucketMap_, p).end();
-             ++buckItr)
-          {
-            const Vertex v(*buckItr);
-            const Vertex y(ancestor_with_lowest_semi_(v, dfnumMap));
-            if (get(semiMap_, y) == get(semiMap_, v))
-              put(domTreePredMap_, v, p);
-            else
-              put(samedomMap, v, y);
-          }
-
-        get(bucketMap_, p).clear();
-      }
-
-    protected :
-
-      /**
-       * Evaluate function in Tarjan's path compression
-       */
-      const Vertex
-      ancestor_with_lowest_semi_(const Vertex& v, const TimeMap& dfnumMap)
-      {
-        const Vertex a(get(ancestorMap_, v));
-
-        if (get(ancestorMap_, a) != graph_traits<Graph>::null_vertex())
-          {
-            const Vertex b(ancestor_with_lowest_semi_(a, dfnumMap));
-
-            put(ancestorMap_, v, get(ancestorMap_, a));
-
-            if (get(dfnumMap, get(semiMap_, b)) <
-                get(dfnumMap, get(semiMap_, get(bestMap_, v))))
-              put(bestMap_, v, b);
-          }
-
-        return get(bestMap_, v);
-      }
-
-      std::vector<Vertex> semi_, ancestor_, samedom_, best_;
-      PredMap semiMap_, ancestorMap_, bestMap_;
-      std::vector< std::deque<Vertex> > buckets_;
-
-      iterator_property_map<typename std::vector<std::deque<Vertex> >::iterator,
-                            IndexMap> bucketMap_;
-
-      const Vertex& entry_;
-      DomTreePredMap domTreePredMap_;
-      const VerticesSizeType numOfVertices_;
-
-    public :
-
-      PredMap samedomMap;
-    };
-
-  } // namespace detail
-
-  /**
-   * @brief Build dominator tree using Lengauer-Tarjan algorithm.
-   *                It takes O((V+E)log(V+E)) time.
-   *
-   * @pre dfnumMap, parentMap and verticesByDFNum have dfs results corresponding
-   *      indexMap.
-   *      If dfs has already run before,
-   *      this function would be good for saving computations.
-   * @pre Unreachable nodes must be masked as
-   *      graph_traits<Graph>::null_vertex in parentMap.
-   * @pre Unreachable nodes must be masked as
-   *      (std::numeric_limits<VerticesSizeType>::max)() in dfnumMap.
-   *
-   * @param domTreePredMap [out] : immediate dominator map (parent map
-   * in dom. tree)
-   *
-   * @note reference Appel. p. 452~453. algorithm 19.9, 19.10.
-   *
-   * @todo : Optimization in Finding Dominators in Practice, Loukas Georgiadis
-   */
-  template<class Graph, class IndexMap, class TimeMap, class PredMap,
-           class VertexVector, class DomTreePredMap>
-  void
-  lengauer_tarjan_dominator_tree_without_dfs
-    (const Graph& g,
-     const typename graph_traits<Graph>::vertex_descriptor& entry,
-     const IndexMap& indexMap,
-     TimeMap dfnumMap, PredMap parentMap, VertexVector& verticesByDFNum,
-     DomTreePredMap domTreePredMap)
-  {
-    // Typedefs and concept check
-    typedef typename graph_traits<Graph>::vertex_descriptor Vertex;
-    typedef typename graph_traits<Graph>::vertices_size_type VerticesSizeType;
-
-    BOOST_CONCEPT_ASSERT(( BidirectionalGraphConcept<Graph> ));
-
-    const VerticesSizeType numOfVertices = num_vertices(g);
-    if (numOfVertices == 0) return;
-
-    // 1. Visit each vertex in reverse post order and calculate sdom.
-    detail::dominator_visitor<Graph, IndexMap, TimeMap, PredMap, DomTreePredMap>
-      visitor(g, entry, indexMap, domTreePredMap);
-
-    VerticesSizeType i;
-    for (i = 0; i < numOfVertices; ++i)
-      {
-        const Vertex u(verticesByDFNum[numOfVertices - 1 - i]);
-        if (u != graph_traits<Graph>::null_vertex())
-          visitor(u, dfnumMap, parentMap, g);
-      }
-
-    // 2. Now all the deferred dominator calculations,
-    // based on the second clause of the dominator thm., are performed
-    for (i = 0; i < numOfVertices; ++i)
-      {
-        const Vertex n(verticesByDFNum[i]);
-
-        if (n == entry || n == graph_traits<Graph>::null_vertex())
-          continue;
-
-        Vertex u = get(visitor.samedomMap, n);
-        if (u != graph_traits<Graph>::null_vertex())
-          {
-            put(domTreePredMap, n, get(domTreePredMap, u));
-          }
-      }
-  }
-
-  /**
-   * Unlike lengauer_tarjan_dominator_tree_without_dfs,
-   * dfs is run in this function and
-   * the result is written to dfnumMap, parentMap, vertices.
-   *
-   * If the result of dfs required after this algorithm,
-   * this function can eliminate the need of rerunning dfs.
-   */
-  template<class Graph, class IndexMap, class TimeMap, class PredMap,
-           class VertexVector, class DomTreePredMap>
-  void
-  lengauer_tarjan_dominator_tree
-    (const Graph& g,
-     const typename graph_traits<Graph>::vertex_descriptor& entry,
-     const IndexMap& indexMap,
-     TimeMap dfnumMap, PredMap parentMap, VertexVector& verticesByDFNum,
-     DomTreePredMap domTreePredMap)
-  {
-    // Typedefs and concept check
-    typedef typename graph_traits<Graph>::vertices_size_type VerticesSizeType;
-
-    BOOST_CONCEPT_ASSERT(( BidirectionalGraphConcept<Graph> ));
-
-    // 1. Depth first visit
-    const VerticesSizeType numOfVertices = num_vertices(g);
-    if (numOfVertices == 0) return;
-
-    VerticesSizeType time =
-      (std::numeric_limits<VerticesSizeType>::max)();
-    std::vector<default_color_type>
-      colors(numOfVertices, color_traits<default_color_type>::white());
-    depth_first_visit
-      (g, entry,
-       make_dfs_visitor
-         (make_pair(record_predecessors(parentMap, on_tree_edge()),
-                    detail::stamp_times_with_vertex_vector
-                      (dfnumMap, verticesByDFNum, time, on_discover_vertex()))),
-       make_iterator_property_map(colors.begin(), indexMap));
-
-    // 2. Run main algorithm.
-    lengauer_tarjan_dominator_tree_without_dfs(g, entry, indexMap, dfnumMap,
-                                               parentMap, verticesByDFNum,
-                                               domTreePredMap);
-  }
-
-  /**
-   * Use vertex_index as IndexMap and make dfnumMap, parentMap, verticesByDFNum
-   * internally.
-   * If we don't need the result of dfs (dfnumMap, parentMap, verticesByDFNum),
-   * this function would be more convenient one.
-   */
-  template<class Graph, class DomTreePredMap>
-  void
-  lengauer_tarjan_dominator_tree
-    (const Graph& g,
-     const typename graph_traits<Graph>::vertex_descriptor& entry,
-     DomTreePredMap domTreePredMap)
-  {
-    // typedefs
-    typedef typename graph_traits<Graph>::vertex_descriptor Vertex;
-    typedef typename graph_traits<Graph>::vertices_size_type VerticesSizeType;
-    typedef typename property_map<Graph, vertex_index_t>::const_type IndexMap;
-    typedef
-      iterator_property_map<typename std::vector<VerticesSizeType>::iterator,
-                            IndexMap> TimeMap;
-    typedef
-      iterator_property_map<typename std::vector<Vertex>::iterator, IndexMap>
-      PredMap;
-
-    // Make property maps
-    const VerticesSizeType numOfVertices = num_vertices(g);
-    if (numOfVertices == 0) return;
-
-    const IndexMap indexMap = get(vertex_index, g);
-
-    std::vector<VerticesSizeType> dfnum(numOfVertices, 0);
-    TimeMap dfnumMap(make_iterator_property_map(dfnum.begin(), indexMap));
-
-    std::vector<Vertex> parent(numOfVertices,
-                               graph_traits<Graph>::null_vertex());
-    PredMap parentMap(make_iterator_property_map(parent.begin(), indexMap));
-
-    std::vector<Vertex> verticesByDFNum(parent);
-
-    // Run main algorithm
-    lengauer_tarjan_dominator_tree(g, entry,
-                                   indexMap, dfnumMap, parentMap,
-                                   verticesByDFNum, domTreePredMap);
-  }
-
-  /**
-   * Muchnick. p. 182, 184
-   *
-   * using iterative bit vector analysis
-   */
-  template<class Graph, class IndexMap, class DomTreePredMap>
-  void
-  iterative_bit_vector_dominator_tree
-    (const Graph& g,
-     const typename graph_traits<Graph>::vertex_descriptor& entry,
-     const IndexMap& indexMap,
-     DomTreePredMap domTreePredMap)
-  {
-    typedef typename graph_traits<Graph>::vertex_descriptor Vertex;
-    typedef typename graph_traits<Graph>::vertex_iterator vertexItr;
-    typedef typename graph_traits<Graph>::vertices_size_type VerticesSizeType;
-    typedef
-      iterator_property_map<typename std::vector< std::set<Vertex> >::iterator,
-                            IndexMap> vertexSetMap;
-
-    BOOST_CONCEPT_ASSERT(( BidirectionalGraphConcept<Graph> ));
-
-    // 1. Finding dominator
-    // 1.1. Initialize
-    const VerticesSizeType numOfVertices = num_vertices(g);
-    if (numOfVertices == 0) return;
-
-    vertexItr vi, viend;
-    boost::tie(vi, viend) = vertices(g);
-    const std::set<Vertex> N(vi, viend);
-
-    bool change = true;
-
-    std::vector< std::set<Vertex> > dom(numOfVertices, N);
-    vertexSetMap domMap(make_iterator_property_map(dom.begin(), indexMap));
-    get(domMap, entry).clear();
-    get(domMap, entry).insert(entry);
-
-    while (change)
-      {
-        change = false;
-        for (boost::tie(vi, viend) = vertices(g); vi != viend; ++vi)
-          {
-            if (*vi == entry) continue;
-
-            std::set<Vertex> T(N);
-
-            typename graph_traits<Graph>::in_edge_iterator inItr, inEnd;
-            for (boost::tie(inItr, inEnd) = in_edges(*vi, g); inItr != inEnd; ++inItr)
-              {
-                const Vertex p = source(*inItr, g);
-
-                std::set<Vertex> tempSet;
-                std::set_intersection(T.begin(), T.end(),
-                                      get(domMap, p).begin(),
-                                      get(domMap, p).end(),
-                                      std::inserter(tempSet, tempSet.begin()));
-                T.swap(tempSet);
-              }
-
-            T.insert(*vi);
-            if (T != get(domMap, *vi))
-              {
-                change = true;
-                get(domMap, *vi).swap(T);
-              }
-          } // end of for (boost::tie(vi, viend) = vertices(g)
-      } // end of while(change)
-
-    // 2. Build dominator tree
-    for (boost::tie(vi, viend) = vertices(g); vi != viend; ++vi)
-      get(domMap, *vi).erase(*vi);
-
-    Graph domTree(numOfVertices);
-
-    for (boost::tie(vi, viend) = vertices(g); vi != viend; ++vi)
-      {
-        if (*vi == entry) continue;
-
-        // We have to iterate through copied dominator set
-        const std::set<Vertex> tempSet(get(domMap, *vi));
-        typename std::set<Vertex>::const_iterator s;
-        for (s = tempSet.begin(); s != tempSet.end(); ++s)
-          {
-            typename std::set<Vertex>::iterator t;
-            for (t = get(domMap, *vi).begin(); t != get(domMap, *vi).end(); )
-              {
-        typename std::set<Vertex>::iterator old_t = t;
-        ++t; // Done early because t may become invalid
-                if (*old_t == *s) continue;
-                if (get(domMap, *s).find(*old_t) != get(domMap, *s).end())
-                  get(domMap, *vi).erase(old_t);
-              }
-          }
-      }
-
-    for (boost::tie(vi, viend) = vertices(g); vi != viend; ++vi)
-      {
-        if (*vi != entry && get(domMap, *vi).size() == 1)
-          {
-            Vertex temp = *get(domMap, *vi).begin();
-            put(domTreePredMap, *vi, temp);
-          }
-      }
-  }
-
-  template<class Graph, class DomTreePredMap>
-  void
-  iterative_bit_vector_dominator_tree
-    (const Graph& g,
-     const typename graph_traits<Graph>::vertex_descriptor& entry,
-     DomTreePredMap domTreePredMap)
-  {
-    typename property_map<Graph, vertex_index_t>::const_type
-      indexMap = get(vertex_index, g);
-
-    iterative_bit_vector_dominator_tree(g, entry, indexMap, domTreePredMap);
-  }
-} // namespace boost
-
-#endif // BOOST_GRAPH_DOMINATOR_HPP

include/boost-patched/graph/reverse_graph.hpp

@@ -1,27 +0,0 @@
-#ifndef REVERSE_GRAPH_PATCHED_H_
-#define REVERSE_GRAPH_PATCHED_H_
-
-#include <boost/version.hpp>
-
-#include <boost/graph/reverse_graph.hpp>
-
-#if defined(BOOST_REVGRAPH_PATCH)
-
-// Boost 1.62.0 does not implement degree() in reverse_graph which is required
-// by BidirectionalGraph, so add it.
-
-namespace boost {
-
-template <class BidirectionalGraph, class GRef>
-inline typename graph_traits<BidirectionalGraph>::degree_size_type
-degree(const typename graph_traits<BidirectionalGraph>::vertex_descriptor u,
-       const reverse_graph<BidirectionalGraph,GRef>& g)
-{
-    return degree(u, g.m_g);
-}
-
-} // namespace boost
-
-#endif // Boost 1.62.0
-
-#endif

libhs.pc.in

@@ -1,10 +0,0 @@
-prefix=@CMAKE_INSTALL_PREFIX@
-exec_prefix=@CMAKE_INSTALL_PREFIX@
-libdir=@CMAKE_INSTALL_PREFIX@/@CMAKE_INSTALL_LIBDIR@
-includedir=@CMAKE_INSTALL_PREFIX@/@CMAKE_INSTALL_INCLUDEDIR@
-
-Name: libhs
-Description: Intel(R) Hyperscan Library
-Version: @HS_VERSION@
-Libs: -L${libdir} -lhs
-Cflags: -I${includedir}/hs

src/alloc.c

@@ -1,113 +0,0 @@
-/*
- * Copyright (c) 2015-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Runtime functions for setting custom allocators.
- */
-
-#include <stdlib.h>
-#include <string.h>
-
-#include "allocator.h"
-
-#define default_malloc malloc
-#define default_free free
-
-hs_alloc_t hs_database_alloc = default_malloc;
-hs_alloc_t hs_misc_alloc = default_malloc;
-hs_alloc_t hs_scratch_alloc = default_malloc;
-hs_alloc_t hs_stream_alloc = default_malloc;
-
-hs_free_t hs_database_free = default_free;
-hs_free_t hs_misc_free = default_free;
-hs_free_t hs_scratch_free = default_free;
-hs_free_t hs_stream_free = default_free;
-
-static
-hs_alloc_t normalise_alloc(hs_alloc_t a) {
-    if (!a) {
-        return default_malloc;
-    } else {
-        return a;
-    }
-}
-
-static
-hs_free_t normalise_free(hs_free_t f) {
-    if (!f) {
-        return default_free;
-    } else {
-        return f;
-    }
-}
-
-HS_PUBLIC_API
-hs_error_t HS_CDECL hs_set_allocator(hs_alloc_t allocfunc, hs_free_t freefunc) {
-    hs_set_database_allocator(allocfunc, freefunc);
-    hs_set_misc_allocator(allocfunc, freefunc);
-    hs_set_stream_allocator(allocfunc, freefunc);
-    hs_set_scratch_allocator(allocfunc, freefunc);
-
-    return HS_SUCCESS;
-}
-
-HS_PUBLIC_API
-hs_error_t HS_CDECL hs_set_database_allocator(hs_alloc_t allocfunc,
-                                              hs_free_t freefunc) {
-    hs_database_alloc = normalise_alloc(allocfunc);
-    hs_database_free = normalise_free(freefunc);
-
-    return HS_SUCCESS;
-}
-
-HS_PUBLIC_API
-hs_error_t HS_CDECL hs_set_misc_allocator(hs_alloc_t allocfunc,
-                                          hs_free_t freefunc) {
-    hs_misc_alloc = normalise_alloc(allocfunc);
-    hs_misc_free = normalise_free(freefunc);
-
-    return HS_SUCCESS;
-}
-
-HS_PUBLIC_API
-hs_error_t HS_CDECL hs_set_scratch_allocator(hs_alloc_t allocfunc,
-                                             hs_free_t freefunc) {
-    hs_scratch_alloc = normalise_alloc(allocfunc);
-    hs_scratch_free = normalise_free(freefunc);
-
-    return HS_SUCCESS;
-}
-
-HS_PUBLIC_API
-hs_error_t HS_CDECL hs_set_stream_allocator(hs_alloc_t allocfunc,
-                                            hs_free_t freefunc) {
-    hs_stream_alloc = normalise_alloc(allocfunc);
-    hs_stream_free = normalise_free(freefunc);
-
-    return HS_SUCCESS;
-}

src/allocator.h

@@ -1,66 +0,0 @@
-/*
- * Copyright (c) 2015, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef ALLOCATOR_H
-#define ALLOCATOR_H
-
-#include "hs_common.h"
-#include "ue2common.h"
-
-#ifdef __cplusplus
-extern "C"
-{
-#endif
-extern hs_alloc_t hs_database_alloc;
-extern hs_alloc_t hs_misc_alloc;
-extern hs_alloc_t hs_scratch_alloc;
-extern hs_alloc_t hs_stream_alloc;
-
-extern hs_free_t hs_database_free;
-extern hs_free_t hs_misc_free;
-extern hs_free_t hs_scratch_free;
-extern hs_free_t hs_stream_free;
-#ifdef __cplusplus
-} /* extern C */
-#endif
-/** \brief Check the results of an alloc done with hs_alloc for alignment.
- *
- * If we have incorrect alignment, return an error. Caller should free the
- * offending block. */
-static really_inline
-hs_error_t hs_check_alloc(const void *mem) {
-    hs_error_t ret = HS_SUCCESS;
-    if (!mem) {
-        ret = HS_NOMEM;
-    } else if (!ISALIGNED_N(mem, alignof(unsigned long long))) {
-        ret = HS_BAD_ALLOC;
-    }
-    return ret;
-}
-
-#endif

src/compiler/asserts.cpp

@@ -1,311 +0,0 @@
-/*
- * Copyright (c) 2015-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Convert temporary assert vertices (from construction method) to
- * edge-based flags.
- *
- * This pass converts the temporary assert vertices created by the Glushkov
- * construction process above (vertices with special assertions flags) into
- * edges between those vertices' neighbours in the graph.
- *
- * These edges have the appropriate flags applied to them -- a path (u,t,v)
- * through an assert vertex t will be replaced with the edge (u,v) with the
- * assertion flags from t.
- *
- * Edges with mutually incompatible flags (such as the conjunction of
- * word-to-word and word-to-nonword) are dropped.
- */
-#include "asserts.h"
-
-#include "compiler/compiler.h"
-#include "nfagraph/ng.h"
-#include "nfagraph/ng_prune.h"
-#include "nfagraph/ng_redundancy.h"
-#include "nfagraph/ng_util.h"
-#include "parser/position.h" // for POS flags
-#include "util/compile_error.h"
-#include "util/graph_range.h"
-
-#include <queue>
-#include <set>
-
-using namespace std;
-
-namespace ue2 {
-
-/** Hard limit on the maximum number of edges we'll clone before we throw up
- * our hands and report 'Pattern too large.' */
-static const size_t MAX_ASSERT_EDGES = 300000;
-
-/** Flags representing the word-boundary assertions, \\b or \\B. */
-static const int WORDBOUNDARY_FLAGS = POS_FLAG_ASSERT_WORD_TO_WORD
-                                    | POS_FLAG_ASSERT_WORD_TO_NONWORD
-                                    | POS_FLAG_ASSERT_NONWORD_TO_WORD
-                                    | POS_FLAG_ASSERT_NONWORD_TO_NONWORD
-                                    | POS_FLAG_ASSERT_WORD_TO_WORD_UCP
-                                    | POS_FLAG_ASSERT_WORD_TO_NONWORD_UCP
-                                    | POS_FLAG_ASSERT_NONWORD_TO_WORD_UCP
-                                    | POS_FLAG_ASSERT_NONWORD_TO_NONWORD_UCP;
-
-#define OPEN_EDGE 0U
-#define DEAD_EDGE (~0U)
-
-static
-u32 disjunct(u32 flags1, u32 flags2) {
-    /* from two asserts in parallel */
-    DEBUG_PRINTF("disjunct %x %x\n", flags1, flags2);
-    u32 rv;
-    if (flags1 == DEAD_EDGE) {
-        rv = flags2;
-    } else if (flags2 == DEAD_EDGE) {
-        rv = flags1;
-    } else if (flags1 == OPEN_EDGE || flags2 == OPEN_EDGE) {
-        rv = OPEN_EDGE;
-    } else {
-        rv = flags1 | flags2;
-    }
-    DEBUG_PRINTF("--> %x\n", rv);
-    return rv;
-}
-
-static
-u32 conjunct(u32 flags1, u32 flags2) {
-    /* from two asserts in series */
-    DEBUG_PRINTF("conjunct %x %x\n", flags1, flags2);
-    u32 rv;
-    if (flags1 == OPEN_EDGE) {
-        rv = flags2;
-    } else if (flags2 == OPEN_EDGE) {
-        rv = flags1;
-    } else if (flags1 & flags2) {
-        rv = flags1 & flags2;
-    } else {
-        rv = DEAD_EDGE; /* the conjunction of two different word boundary
-                         * assertion is impassable */
-    }
-
-    DEBUG_PRINTF("--> %x\n", rv);
-    return rv;
-}
-
-typedef map<pair<NFAVertex, NFAVertex>, NFAEdge> edge_cache_t;
-
-static
-void replaceAssertVertex(NGHolder &g, NFAVertex t, const ExpressionInfo &expr,
-                         edge_cache_t &edge_cache, u32 &assert_edge_count) {
-    DEBUG_PRINTF("replacing assert vertex %zu\n", g[t].index);
-
-    const u32 flags = g[t].assert_flags;
-    DEBUG_PRINTF("consider assert vertex %zu with flags %u\n", g[t].index,
-                 flags);
-
-    // Wire up all the predecessors to all the successors.
-
-    for (const auto &inEdge : in_edges_range(t, g)) {
-        NFAVertex u = source(inEdge, g);
-        if (u == t) {
-            continue; // ignore self-loops
-        }
-
-        const u32 flags_inc_in = conjunct(g[inEdge].assert_flags,
-                                          flags);
-        if (flags_inc_in == DEAD_EDGE) {
-            DEBUG_PRINTF("fail, in-edge has bad flags %d\n",
-                         g[inEdge].assert_flags);
-            continue;
-        }
-
-        for (const auto &outEdge : out_edges_range(t, g)) {
-            NFAVertex v = target(outEdge, g);
-
-            DEBUG_PRINTF("consider path [%zu,%zu,%zu]\n", g[u].index,
-                         g[t].index, g[v].index);
-
-            if (v == t) {
-                continue; // ignore self-loops
-            }
-
-            const u32 flags_final = conjunct(g[outEdge].assert_flags,
-                                             flags_inc_in);
-
-            if (flags_final == DEAD_EDGE) {
-                DEBUG_PRINTF("fail, out-edge has bad flags %d\n",
-                             g[outEdge].assert_flags);
-                continue;
-            }
-
-            if ((g[u].assert_flags & POS_FLAG_MULTILINE_START)
-                && v == g.acceptEod) {
-                DEBUG_PRINTF("fail, (?m)^ does not match \\n at eod\n");
-                continue;
-            }
-
-            /* Replace path (u,t,v) with direct edge (u,v), unless the edge
-             * already exists, in which case we just need to edit its
-             * properties.
-             *
-             * Use edge_cache to prevent us going O(N).
-             */
-            auto cache_key = make_pair(u, v);
-            auto ecit = edge_cache.find(cache_key);
-            if (ecit == edge_cache.end()) {
-                DEBUG_PRINTF("adding edge %zu %zu\n", g[u].index, g[v].index);
-                NFAEdge e = add_edge(u, v, g);
-                edge_cache.emplace(cache_key, e);
-                g[e].assert_flags = flags;
-                if (++assert_edge_count > MAX_ASSERT_EDGES) {
-                    throw CompileError(expr.index, "Pattern is too large.");
-                }
-            } else {
-                NFAEdge e = ecit->second;
-                DEBUG_PRINTF("updating edge %zu %zu [a %zu]\n", g[u].index,
-                             g[v].index, g[t].index);
-                // Edge already exists.
-                u32 &e_flags = g[e].assert_flags;
-                e_flags = disjunct(e_flags, flags_final);
-                assert(e_flags != DEAD_EDGE);
-            }
-        }
-    }
-
-    // Clear vertex t to remove all the old edges.
-    /* no need to clear the cache, as we will never look up its edge as it is
-     * unreachable */
-    clear_vertex(t, g);
-}
-
-static
-void setReportId(ReportManager &rm, NGHolder &g, const ExpressionInfo &expr,
-                 NFAVertex v, s32 adj) {
-    // Don't try and set the report ID of a special vertex.
-    assert(!is_special(v, g));
-
-    // There should be no reports set already.
-    assert(g[v].reports.empty());
-
-    Report r = rm.getBasicInternalReport(expr, adj);
-
-    g[v].reports.insert(rm.getInternalId(r));
-    DEBUG_PRINTF("set report id for vertex %zu, adj %d\n", g[v].index, adj);
-}
-
-static
-void checkForMultilineStart(ReportManager &rm, NGHolder &g,
-                            const ExpressionInfo &expr) {
-    vector<NFAEdge> dead;
-    for (auto v : adjacent_vertices_range(g.start, g)) {
-        if (!(g[v].assert_flags & POS_FLAG_MULTILINE_START)) {
-            continue;
-        }
-        DEBUG_PRINTF("mls %zu %08x\n", g[v].index, g[v].assert_flags);
-
-        /* we have found a multi-line start (maybe more than one) */
-
-        /* we need to interpose a dummy dot vertex between v and accept if
-         * required so that ^ doesn't match trailing \n */
-         for (const auto &e : out_edges_range(v, g)) {
-            if (target(e, g) == g.accept) {
-                dead.emplace_back(e);
-            }
-        }
-        /* assert has been resolved; clear flag */
-        g[v].assert_flags &= ~POS_FLAG_MULTILINE_START;
-    }
-
-    for (const auto &e : dead) {
-        NFAVertex dummy = add_vertex(g);
-        g[dummy].char_reach.setall();
-        setReportId(rm, g, expr, dummy, -1);
-        add_edge(source(e, g), dummy, g[e], g);
-        add_edge(dummy, g.accept, g);
-    }
-
-    remove_edges(dead, g);
-}
-
-static
-bool hasAssertVertices(const NGHolder &g) {
-    for (auto v : vertices_range(g)) {
-        int flags = g[v].assert_flags;
-        if (flags & WORDBOUNDARY_FLAGS) {
-            return true;
-        }
-    }
-    return false;
-}
-
-/** \brief Convert temporary assert vertices (from construction method) to
- * edge-based flags.
- *
- * Remove the horrors that are the temporary assert vertices which arise from
- * our construction method. Allows the rest of our code base to live in
- * blissful ignorance of their existence. */
-void removeAssertVertices(ReportManager &rm, NGHolder &g,
-                          const ExpressionInfo &expr) {
-    size_t num = 0;
-
-    DEBUG_PRINTF("before: graph has %zu vertices\n", num_vertices(g));
-
-    // Sweep over the graph and ascertain that we do actually have vertices
-    // with assertion flags set. Otherwise, we're done.
-    if (!hasAssertVertices(g)) {
-        DEBUG_PRINTF("no assert vertices, done\n");
-        return;
-    }
-
-    u32 assert_edge_count = 0;
-
-    // Build a cache of (u, v) vertex pairs to edge descriptors.
-    edge_cache_t edge_cache;
-    for (const auto &e : edges_range(g)) {
-        edge_cache[make_pair(source(e, g), target(e, g))] = e;
-    }
-
-    for (auto v : vertices_range(g)) {
-        if (g[v].assert_flags & WORDBOUNDARY_FLAGS) {
-            replaceAssertVertex(g, v, expr, edge_cache, assert_edge_count);
-            num++;
-        }
-    }
-
-    checkForMultilineStart(rm, g, expr);
-
-    if (num) {
-        DEBUG_PRINTF("resolved %zu assert vertices\n", num);
-        pruneUseless(g);
-        pruneEmptyVertices(g);
-        renumber_vertices(g);
-        renumber_edges(g);
-    }
-
-    DEBUG_PRINTF("after: graph has %zu vertices\n", num_vertices(g));
-    assert(!hasAssertVertices(g));
-}
-
-} // namespace ue2

src/compiler/asserts.h

@@ -1,53 +0,0 @@
-/*
- * Copyright (c) 2015-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Convert temporary assert vertices (from construction method) to
- * edge-based flags.
- */
-#ifndef ASSERTS_H
-#define ASSERTS_H
-
-namespace ue2 {
-
-class ExpressionInfo;
-class ReportManager;
-class NGHolder;
-
-/** \brief Convert temporary assert vertices (from construction method) to
- * edge-based flags.
- *
- * Remove the horrors that are the temporary assert vertices which arise from
- * our construction method. Allows the rest of our code base to live in
- * blissful ignorance of their existence. */
-void removeAssertVertices(ReportManager &rm, NGHolder &g,
-                          const ExpressionInfo &expr);
-
-} // namespace ue2
-
-#endif // ASSERTS_H

src/compiler/compiler.cpp

@@ -1,635 +0,0 @@
-/*
- * Copyright (c) 2015-2020, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Compiler front-end interface.
- */
-#include "allocator.h"
-#include "asserts.h"
-#include "compiler.h"
-#include "crc32.h"
-#include "database.h"
-#include "grey.h"
-#include "hs_internal.h"
-#include "hs_runtime.h"
-#include "ue2common.h"
-#include "nfagraph/ng_builder.h"
-#include "nfagraph/ng_dump.h"
-#include "nfagraph/ng.h"
-#include "nfagraph/ng_util.h"
-#include "parser/buildstate.h"
-#include "parser/dump.h"
-#include "parser/Component.h"
-#include "parser/logical_combination.h"
-#include "parser/parse_error.h"
-#include "parser/Parser.h"          // for flags
-#include "parser/position.h"
-#include "parser/position_dump.h"
-#include "parser/position_info.h"
-#include "parser/prefilter.h"
-#include "parser/shortcut_literal.h"
-#include "parser/unsupported.h"
-#include "parser/utf8_validate.h"
-#include "rose/rose_build.h"
-#include "rose/rose_internal.h"
-#include "som/slot_manager_dump.h"
-#include "util/bytecode_ptr.h"
-#include "util/compile_error.h"
-#include "util/target_info.h"
-#include "util/verify_types.h"
-#include "util/ue2string.h"
-
-#include <algorithm>
-#include <cassert>
-#include <cstdlib>
-#include <cstring>
-#include <fstream>
-#include <memory>
-#include <sstream>
-
-using namespace std;
-
-namespace ue2 {
-
-static
-void validateExt(const hs_expr_ext &ext) {
-    static const unsigned long long ALL_EXT_FLAGS = HS_EXT_FLAG_MIN_OFFSET |
-                                                    HS_EXT_FLAG_MAX_OFFSET |
-                                                    HS_EXT_FLAG_MIN_LENGTH |
-                                                    HS_EXT_FLAG_EDIT_DISTANCE |
-                                                    HS_EXT_FLAG_HAMMING_DISTANCE;
-    if (ext.flags & ~ALL_EXT_FLAGS) {
-        throw CompileError("Invalid hs_expr_ext flag set.");
-    }
-
-    if ((ext.flags & HS_EXT_FLAG_MIN_OFFSET) &&
-        (ext.flags & HS_EXT_FLAG_MAX_OFFSET) &&
-        (ext.min_offset > ext.max_offset)) {
-        throw CompileError("In hs_expr_ext, min_offset must be less than or "
-                           "equal to max_offset.");
-    }
-
-    if ((ext.flags & HS_EXT_FLAG_MIN_LENGTH) &&
-        (ext.flags & HS_EXT_FLAG_MAX_OFFSET) &&
-        (ext.min_length > ext.max_offset)) {
-        throw CompileError("In hs_expr_ext, min_length must be less than or "
-                           "equal to max_offset.");
-    }
-
-    if ((ext.flags & HS_EXT_FLAG_EDIT_DISTANCE) &&
-        (ext.flags & HS_EXT_FLAG_HAMMING_DISTANCE)) {
-        throw CompileError("In hs_expr_ext, cannot have both edit distance and "
-                           "Hamming distance.");
-    }
-
-}
-
-void ParsedLitExpression::parseLiteral(const char *expression, size_t len,
-                                       bool nocase) {
-    const char *c = expression;
-    for (size_t i = 0; i < len; i++) {
-        lit.push_back(*c, nocase);
-        c++;
-    }
-}
-
-ParsedLitExpression::ParsedLitExpression(unsigned index_in,
-                                         const char *expression,
-                                         size_t expLength, unsigned flags,
-                                         ReportID report)
-    : expr(index_in, false, flags & HS_FLAG_SINGLEMATCH, false, false,
-           SOM_NONE, report, 0, MAX_OFFSET, 0, 0, 0, false) {
-    // For pure literal expression, below 'HS_FLAG_'s are unuseful:
-    // DOTALL/ALLOWEMPTY/UTF8/UCP/PREFILTER/COMBINATION/QUIET/MULTILINE
-
-    if (flags & ~HS_FLAG_ALL) {
-        DEBUG_PRINTF("Unrecognised flag, flags=%u.\n", flags);
-        throw CompileError("Unrecognised flag.");
-    }
-
-    // FIXME: we disallow highlander + SOM, see UE-1850.
-    if ((flags & HS_FLAG_SINGLEMATCH) && (flags & HS_FLAG_SOM_LEFTMOST)) {
-        throw CompileError("HS_FLAG_SINGLEMATCH is not supported in "
-                           "combination with HS_FLAG_SOM_LEFTMOST.");
-    }
-
-    // Set SOM type.
-    if (flags & HS_FLAG_SOM_LEFTMOST) {
-        expr.som = SOM_LEFT;
-    }
-
-    // Transfer expression text into ue2_literal.
-    bool nocase = flags & HS_FLAG_CASELESS ? true : false;
-    parseLiteral(expression, expLength, nocase);
-
-}
-
-ParsedExpression::ParsedExpression(unsigned index_in, const char *expression,
-                                   unsigned flags, ReportID report,
-                                   const hs_expr_ext *ext)
-    : expr(index_in, flags & HS_FLAG_ALLOWEMPTY, flags & HS_FLAG_SINGLEMATCH,
-           false, flags & HS_FLAG_PREFILTER, SOM_NONE, report, 0, MAX_OFFSET,
-           0, 0, 0, flags & HS_FLAG_QUIET) {
-    // We disallow SOM + Quiet.
-    if ((flags & HS_FLAG_QUIET) && (flags & HS_FLAG_SOM_LEFTMOST)) {
-        throw CompileError("HS_FLAG_QUIET is not supported in "
-                           "combination with HS_FLAG_SOM_LEFTMOST.");
-    }
-    flags &= ~HS_FLAG_QUIET;
-    ParseMode mode(flags);
-
-    component = parse(expression, mode);
-
-    expr.utf8 = mode.utf8; /* utf8 may be set by parse() */
-
-    const size_t len = strlen(expression);
-    if (expr.utf8 && !isValidUtf8(expression, len)) {
-        throw ParseError("Expression is not valid UTF-8.");
-    }
-
-    if (!component) {
-        assert(0); // parse() should have thrown a ParseError.
-        throw ParseError("Parse error.");
-    }
-
-    if (flags & ~HS_FLAG_ALL) {
-        DEBUG_PRINTF("Unrecognised flag, flags=%u.\n", flags);
-        throw CompileError("Unrecognised flag.");
-    }
-
-    // FIXME: we disallow highlander + SOM, see UE-1850.
-    if ((flags & HS_FLAG_SINGLEMATCH) && (flags & HS_FLAG_SOM_LEFTMOST)) {
-        throw CompileError("HS_FLAG_SINGLEMATCH is not supported in "
-                           "combination with HS_FLAG_SOM_LEFTMOST.");
-    }
-
-    // FIXME: we disallow prefilter + SOM, see UE-1899.
-    if ((flags & HS_FLAG_PREFILTER) && (flags & HS_FLAG_SOM_LEFTMOST)) {
-        throw CompileError("HS_FLAG_PREFILTER is not supported in "
-                           "combination with HS_FLAG_SOM_LEFTMOST.");
-    }
-
-    // Set SOM type.
-    if (flags & HS_FLAG_SOM_LEFTMOST) {
-        expr.som = SOM_LEFT;
-    }
-
-    // Set extended parameters, if we have them.
-    if (ext) {
-        // Ensure that the given parameters make sense.
-        validateExt(*ext);
-
-        if (ext->flags & HS_EXT_FLAG_MIN_OFFSET) {
-            expr.min_offset = ext->min_offset;
-        }
-        if (ext->flags & HS_EXT_FLAG_MAX_OFFSET) {
-            expr.max_offset = ext->max_offset;
-        }
-        if (ext->flags & HS_EXT_FLAG_MIN_LENGTH) {
-            expr.min_length = ext->min_length;
-        }
-        if (ext->flags & HS_EXT_FLAG_EDIT_DISTANCE) {
-            expr.edit_distance = ext->edit_distance;
-        }
-        if (ext->flags & HS_EXT_FLAG_HAMMING_DISTANCE) {
-            expr.hamm_distance = ext->hamming_distance;
-        }
-    }
-
-    // These are validated in validateExt, so an error will already have been
-    // thrown if these conditions don't hold.
-    assert(expr.max_offset >= expr.min_offset);
-    assert(expr.max_offset >= expr.min_length);
-
-    // Since prefiltering and SOM aren't supported together, we must squash any
-    // min_length constraint as well.
-    if (flags & HS_FLAG_PREFILTER && expr.min_length) {
-        DEBUG_PRINTF("prefiltering mode: squashing min_length constraint\n");
-        expr.min_length = 0;
-    }
-}
-
-#if defined(DUMP_SUPPORT) || defined(DEBUG)
-/**
- * \brief Dumps the parse tree to screen in debug mode and to disk in dump
- * mode.
- */
-void dumpExpression(UNUSED const ParsedExpression &pe,
-                    UNUSED const char *stage, UNUSED const Grey &grey) {
-#if defined(DEBUG)
-    DEBUG_PRINTF("===== Rule ID: %u (expression index: %u) =====\n",
-                 pe.expr.report, pe.expr.index);
-    ostringstream debug_tree;
-    dumpTree(debug_tree, pe.component.get());
-    printf("%s\n", debug_tree.str().c_str());
-#endif // DEBUG
-
-#if defined(DUMP_SUPPORT)
-    if (grey.dumpFlags & Grey::DUMP_PARSE) {
-        stringstream ss;
-        ss << grey.dumpPath << "Expr_" << pe.expr.index << "_componenttree_"
-           << stage << ".txt";
-        ofstream out(ss.str().c_str());
-        out << "Component Tree for " << pe.expr.report << endl;
-        dumpTree(out, pe.component.get());
-        if (pe.expr.utf8) {
-            out << "UTF8 mode" << endl;
-        }
-    }
-#endif // DEBUG
-}
-#endif
-
-/** \brief Run Component tree optimisations on \a expr. */
-static
-void optimise(ParsedExpression &pe) {
-    if (pe.expr.min_length || pe.expr.som) {
-        return;
-    }
-
-    DEBUG_PRINTF("optimising\n");
-    pe.component->optimise(true /* root is connected to sds */);
-}
-
-void addExpression(NG &ng, unsigned index, const char *expression,
-                   unsigned flags, const hs_expr_ext *ext, ReportID id) {
-    assert(expression);
-    const CompileContext &cc = ng.cc;
-    DEBUG_PRINTF("index=%u, id=%u, flags=%u, expr='%s'\n", index, id, flags,
-                 expression);
-
-    if (flags & HS_FLAG_COMBINATION) {
-        if (flags & ~(HS_FLAG_COMBINATION | HS_FLAG_QUIET |
-                      HS_FLAG_SINGLEMATCH)) {
-            throw CompileError("only HS_FLAG_QUIET and HS_FLAG_SINGLEMATCH "
-                               "are supported in combination "
-                               "with HS_FLAG_COMBINATION.");
-        }
-        if (flags & HS_FLAG_QUIET) {
-            DEBUG_PRINTF("skip QUIET logical combination expression %u\n", id);
-        } else {
-            u32 ekey = INVALID_EKEY;
-            u64a min_offset = 0;
-            u64a max_offset = MAX_OFFSET;
-            if (flags & HS_FLAG_SINGLEMATCH) {
-                ekey = ng.rm.getExhaustibleKey(id);
-            }
-            if (ext) {
-                validateExt(*ext);
-                if (ext->flags & ~(HS_EXT_FLAG_MIN_OFFSET |
-                                   HS_EXT_FLAG_MAX_OFFSET)) {
-                    throw CompileError("only HS_EXT_FLAG_MIN_OFFSET and "
-                                       "HS_EXT_FLAG_MAX_OFFSET extra flags "
-                                       "are supported in combination "
-                                       "with HS_FLAG_COMBINATION.");
-                }
-                if (ext->flags & HS_EXT_FLAG_MIN_OFFSET) {
-                    min_offset = ext->min_offset;
-                }
-                if (ext->flags & HS_EXT_FLAG_MAX_OFFSET) {
-                    max_offset = ext->max_offset;
-                }
-            }
-            ng.rm.pl.parseLogicalCombination(id, expression, ekey, min_offset,
-                                             max_offset);
-            DEBUG_PRINTF("parsed logical combination expression %u\n", id);
-        }
-        return;
-    }
-
-    // Ensure that our pattern isn't too long (in characters).
-    if (strlen(expression) > cc.grey.limitPatternLength) {
-        throw CompileError("Pattern length exceeds limit.");
-    }
-
-    // Do per-expression processing: errors here will result in an exception
-    // being thrown up to our caller
-    ParsedExpression pe(index, expression, flags, id, ext);
-    dumpExpression(pe, "orig", cc.grey);
-
-    // Apply prefiltering transformations if desired.
-    if (pe.expr.prefilter) {
-        prefilterTree(pe.component, ParseMode(flags));
-        dumpExpression(pe, "prefiltered", cc.grey);
-    }
-
-    // Expressions containing zero-width assertions and other extended pcre
-    // types aren't supported yet. This call will throw a ParseError exception
-    // if the component tree contains such a construct.
-    checkUnsupported(*pe.component);
-
-    pe.component->checkEmbeddedStartAnchor(true);
-    pe.component->checkEmbeddedEndAnchor(true);
-
-    if (cc.grey.optimiseComponentTree) {
-        optimise(pe);
-        dumpExpression(pe, "opt", cc.grey);
-    }
-
-    DEBUG_PRINTF("component=%p, nfaId=%u, reportId=%u\n",
-                 pe.component.get(), pe.expr.index, pe.expr.report);
-
-    // You can only use the SOM flags if you've also specified an SOM
-    // precision mode.
-    if (pe.expr.som != SOM_NONE && cc.streaming && !ng.ssm.somPrecision()) {
-        throw CompileError("To use a SOM expression flag in streaming mode, "
-                           "an SOM precision mode (e.g. "
-                           "HS_MODE_SOM_HORIZON_LARGE) must be specified.");
-    }
-
-    // If this expression is a literal, we can feed it directly to Rose rather
-    // than building the NFA graph.
-    if (shortcutLiteral(ng, pe)) {
-        DEBUG_PRINTF("took literal short cut\n");
-        return;
-    }
-
-    auto built_expr = buildGraph(ng.rm, cc, pe);
-    if (!built_expr.g) {
-        DEBUG_PRINTF("NFA build failed on ID %u, but no exception was "
-                     "thrown.\n", pe.expr.report);
-        throw CompileError("Internal error.");
-    }
-
-    if (!pe.expr.allow_vacuous && matches_everywhere(*built_expr.g)) {
-        throw CompileError("Pattern matches empty buffer; use "
-                           "HS_FLAG_ALLOWEMPTY to enable support.");
-    }
-
-    if (!ng.addGraph(built_expr.expr, std::move(built_expr.g))) {
-        DEBUG_PRINTF("NFA addGraph failed on ID %u.\n", pe.expr.report);
-        throw CompileError("Error compiling expression.");
-    }
-}
-
-void addLitExpression(NG &ng, unsigned index, const char *expression,
-                      unsigned flags, const hs_expr_ext *ext, ReportID id,
-                      size_t expLength) {
-    assert(expression);
-    const CompileContext &cc = ng.cc;
-    DEBUG_PRINTF("index=%u, id=%u, flags=%u, expr='%s', len='%zu'\n", index,
-                 id, flags, expression, expLength);
-
-    // Extended parameters are not supported for pure literal patterns.
-    if (ext && ext->flags != 0LLU) {
-        throw CompileError("Extended parameters are not supported for pure "
-                           "literal matching API.");
-    }
-
-    // Ensure that our pattern isn't too long (in characters).
-    if (expLength > cc.grey.limitPatternLength) {
-        throw CompileError("Pattern length exceeds limit.");
-    }
-
-    // filter out flags not supported by pure literal API.
-    u64a not_supported = HS_FLAG_DOTALL | HS_FLAG_ALLOWEMPTY | HS_FLAG_UTF8 |
-                         HS_FLAG_UCP | HS_FLAG_PREFILTER | HS_FLAG_COMBINATION |
-                         HS_FLAG_QUIET | HS_FLAG_MULTILINE;
-
-    if (flags & not_supported) {
-        throw CompileError("Only HS_FLAG_CASELESS, HS_FLAG_SINGLEMATCH and "
-                           "HS_FLAG_SOM_LEFTMOST are supported in literal API.");
-    }
-
-    // This expression must be a pure literal, we can build ue2_literal
-    // directly based on expression text.
-    ParsedLitExpression ple(index, expression, expLength, flags, id);
-
-    // Feed the ue2_literal into Rose.
-    const auto &expr = ple.expr;
-    if (ng.addLiteral(ple.lit, expr.index, expr.report, expr.highlander,
-                      expr.som, expr.quiet)) {
-        DEBUG_PRINTF("took pure literal\n");
-        return;
-    }
-}
-
-static
-bytecode_ptr<RoseEngine> generateRoseEngine(NG &ng) {
-    const u32 minWidth =
-        ng.minWidth.is_finite() ? verify_u32(ng.minWidth) : ROSE_BOUND_INF;
-    auto rose = ng.rose->buildRose(minWidth);
-
-    if (!rose) {
-        DEBUG_PRINTF("error building rose\n");
-        assert(0);
-        return nullptr;
-    }
-
-    dumpReportManager(ng.rm, ng.cc.grey);
-    dumpSomSlotManager(ng.ssm, ng.cc.grey);
-    dumpSmallWrite(rose.get(), ng.cc.grey);
-
-    return rose;
-}
-
-platform_t target_to_platform(const target_t &target_info) {
-    platform_t p;
-    p = 0;
-
-    if (!target_info.has_avx2()) {
-        p |= HS_PLATFORM_NOAVX2;
-    }
-    if (!target_info.has_avx512()) {
-        p |= HS_PLATFORM_NOAVX512;
-    }
-    if (!target_info.has_avx512vbmi()) {
-        p |= HS_PLATFORM_NOAVX512VBMI;
-    }
-    return p;
-}
-
-/** \brief Encapsulate the given bytecode (RoseEngine) in a newly-allocated
- * \ref hs_database, ensuring that it is padded correctly to give cacheline
- * alignment.  */
-static
-hs_database_t *dbCreate(const char *in_bytecode, size_t len, u64a platform) {
-    size_t db_len = sizeof(struct hs_database) + len;
-    DEBUG_PRINTF("db size %zu\n", db_len);
-    DEBUG_PRINTF("db platform %llx\n", platform);
-
-    struct hs_database *db = (struct hs_database *)hs_database_alloc(db_len);
-    if (hs_check_alloc(db) != HS_SUCCESS) {
-        hs_database_free(db);
-        return nullptr;
-    }
-
-    // So that none of our database is uninitialized
-    memset(db, 0, db_len);
-
-    // we need to align things manually
-    size_t shift = (uintptr_t)db->bytes & 0x3f;
-    DEBUG_PRINTF("shift is %zu\n", shift);
-
-    db->bytecode = offsetof(struct hs_database, bytes) - shift;
-    char *bytecode = (char *)db + db->bytecode;
-    assert(ISALIGNED_CL(bytecode));
-
-    db->magic = HS_DB_MAGIC;
-    db->version = HS_DB_VERSION;
-    db->length = len;
-    db->platform = platform;
-
-    // Copy bytecode
-    memcpy(bytecode, in_bytecode, len);
-
-    db->crc32 = Crc32c_ComputeBuf(0, bytecode, db->length);
-    return db;
-}
-
-
-struct hs_database *build(NG &ng, unsigned int *length, u8 pureFlag) {
-    assert(length);
-
-    auto rose = generateRoseEngine(ng);
-    struct RoseEngine *roseHead = rose.get();
-    roseHead->pureLiteral = pureFlag;
-
-    if (!rose) {
-        throw CompileError("Unable to generate bytecode.");
-    }
-    *length = rose.size();
-    if (!*length) {
-        DEBUG_PRINTF("RoseEngine has zero length\n");
-        assert(0);
-        throw CompileError("Internal error.");
-    }
-
-    const char *bytecode = (const char *)(rose.get());
-    const platform_t p = target_to_platform(ng.cc.target_info);
-    struct hs_database *db = dbCreate(bytecode, *length, p);
-    if (!db) {
-        throw CompileError("Could not allocate memory for bytecode.");
-    }
-
-    return db;
-}
-
-static
-void stripFromPositions(vector<PositionInfo> &v, Position pos) {
-    auto removed = remove(v.begin(), v.end(), PositionInfo(pos));
-    v.erase(removed, v.end());
-}
-
-static
-void connectInitialStates(GlushkovBuildState &bs,
-                          const ParsedExpression &expr) {
-    vector<PositionInfo> initials = expr.component->first();
-    const NFABuilder &builder = bs.getBuilder();
-    const Position startState = builder.getStart();
-    const Position startDotStarState = builder.getStartDotStar();
-
-    DEBUG_PRINTF("wiring initials = %s\n",
-                 dumpPositions(initials.begin(), initials.end()).c_str());
-
-    vector<PositionInfo> starts = {startState, startDotStarState};
-
-    // strip start and startDs, which can be present due to boundaries
-    stripFromPositions(initials, startState);
-    stripFromPositions(initials, startDotStarState);
-
-    // replace epsilons with accepts
-    for (const auto &s : initials) {
-        if (s.pos != GlushkovBuildState::POS_EPSILON) {
-            continue;
-        }
-
-        assert(starts.size() == 2); /* start, startds */
-        vector<PositionInfo> starts_temp = starts;
-        starts_temp[0].flags = s.flags;
-        starts_temp[1].flags = s.flags;
-        bs.connectAccepts(starts_temp);
-    }
-
-    if (!initials.empty()) {
-        bs.connectRegions(starts, initials);
-    }
-}
-
-static
-void connectFinalStates(GlushkovBuildState &bs, const ParsedExpression &expr) {
-    vector<PositionInfo> finals = expr.component->last();
-
-    DEBUG_PRINTF("wiring finals = %s\n",
-                 dumpPositions(finals.begin(), finals.end()).c_str());
-
-    bs.connectAccepts(finals);
-}
-
-#ifndef NDEBUG
-static
-bool isSupported(const Component &c) {
-    try {
-        checkUnsupported(c);
-        return true;
-    }
-    catch (ParseError &) {
-        return false;
-    }
-}
-#endif
-
-BuiltExpression buildGraph(ReportManager &rm, const CompileContext &cc,
-                           const ParsedExpression &pe) {
-    assert(isSupported(*pe.component));
-
-    const auto builder = makeNFABuilder(rm, cc, pe);
-    assert(builder);
-
-    // Set up START and ACCEPT states; retrieve the special states
-    const auto bs = makeGlushkovBuildState(*builder, pe.expr.prefilter);
-
-    // Map position IDs to characters/components
-    pe.component->notePositions(*bs);
-
-    // Wire the start dotstar state to the firsts
-    connectInitialStates(*bs, pe);
-
-    DEBUG_PRINTF("wire up body of expr\n");
-    // Build the rest of the FOLLOW set
-    vector<PositionInfo> initials = {builder->getStartDotStar(),
-                                     builder->getStart()};
-    pe.component->buildFollowSet(*bs, initials);
-
-    // Wire the lasts to the accept state
-    connectFinalStates(*bs, pe);
-
-    // Create our edges
-    bs->buildEdges();
-
-    BuiltExpression built_expr = builder->getGraph();
-    assert(built_expr.g);
-
-    dumpDotWrapper(*built_expr.g, built_expr.expr, "00_before_asserts",
-                   cc.grey);
-    removeAssertVertices(rm, *built_expr.g, built_expr.expr);
-
-    return built_expr;
-}
-
-} // namespace ue2

src/compiler/compiler.h

@@ -1,172 +0,0 @@
-/*
- * Copyright (c) 2015-2019, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Compiler front-end interface
- */
-
-#ifndef COMPILER_H
-#define COMPILER_H
-
-#include "ue2common.h"
-#include "database.h"
-#include "compiler/expression_info.h"
-#include "parser/Component.h"
-#include "util/noncopyable.h"
-#include "util/ue2string.h"
-
-#include <memory>
-
-struct hs_database;
-struct hs_expr_ext;
-
-namespace ue2 {
-
-struct CompileContext;
-struct Grey;
-struct target_t;
-class NG;
-class NGHolder;
-class ReportManager;
-
-/** \brief Class gathering together the pieces of a parsed expression. */
-class ParsedExpression : noncopyable {
-public:
-    ParsedExpression(unsigned index, const char *expression, unsigned flags,
-                     ReportID report, const hs_expr_ext *ext = nullptr);
-
-    /** \brief Expression information (from flags, extparam etc) */
-    ExpressionInfo expr;
-
-    /** \brief Root node of parsed component tree. */
-    std::unique_ptr<Component> component;
-};
-
-
-/** \brief Class gathering together the pieces of a parsed lit-expression. */
-class ParsedLitExpression : noncopyable {
-public:
-    ParsedLitExpression(unsigned index, const char *expression,
-                        size_t expLength, unsigned flags, ReportID report);
-
-    void parseLiteral(const char *expression, size_t len, bool nocase);
-
-    /** \brief Expression information (from flags, extparam etc) */
-    ExpressionInfo expr;
-
-    /** \brief Format the lit-expression text into Hyperscan literal type. */
-    ue2_literal lit;
-};
-
-/**
- * \brief Class gathering together the pieces of an expression that has been
- * built into an NFA graph.
- */
-struct BuiltExpression {
-    /** \brief Expression information (from flags, extparam etc) */
-    ExpressionInfo expr;
-
-    /** \brief Built Glushkov NFA graph. */
-    std::unique_ptr<NGHolder> g;
-};
-
-/**
- * Add an expression to the compiler.
- *
- * @param ng
- *      The global NG object.
- * @param index
- *      The index of the expression (used for errors)
- * @param expression
- *      NULL-terminated PCRE expression
- * @param flags
- *      The full set of Hyperscan flags associated with this rule.
- * @param ext
- *      Struct containing extra parameters for this expression, or NULL if
- *      none.
- * @param report
- *      The identifier to associate with the expression; returned by engine on
- *      match.
- */
-void addExpression(NG &ng, unsigned index, const char *expression,
-                   unsigned flags, const hs_expr_ext *ext, ReportID report);
-
-void addLitExpression(NG &ng, unsigned index, const char *expression,
-                      unsigned flags, const hs_expr_ext *ext, ReportID id,
-                      size_t expLength);
-
-/**
- * Build a Hyperscan database out of the expressions we've been given. A
- * fatal error will result in an exception being thrown.
- *
- * @param ng
- *      The global NG object.
- * @param[out] length
- *      The number of bytes occupied by the compiled structure.
- * @param pureFlag
- *      The flag indicating invocation from literal API or not.
- * @return
- *      The compiled structure. Should be deallocated with the
- *      hs_database_free() function.
- */
-struct hs_database *build(NG &ng, unsigned int *length, u8 pureFlag);
-
-/**
- * Constructs an NFA graph from the given expression tree.
- *
- * @param rm
- *      Global ReportManager for this compile.
- * @param cc
- *      Global compile context for this compile.
- * @param expr
- *      ParsedExpression object.
- * @return
- *      nullptr on error.
- */
-BuiltExpression buildGraph(ReportManager &rm, const CompileContext &cc,
-                           const ParsedExpression &expr);
-
-/**
- * Build a platform_t out of a target_t.
- */
-platform_t target_to_platform(const target_t &target_info);
-
-#if defined(DUMP_SUPPORT) || defined(DEBUG)
-void dumpExpression(const ParsedExpression &expr, const char *stage,
-                    const Grey &grey);
-#else
-static really_inline
-void dumpExpression(UNUSED const ParsedExpression &expr,
-                    UNUSED const char *stage, UNUSED const Grey &grey) {
-}
-
-#endif
-
-} // namespace
-
-#endif // COMPILER_H

src/compiler/error.cpp

@@ -1,110 +0,0 @@
-/*
- * Copyright (c) 2015-2016, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Compile-time error utils.
- */
-#include "allocator.h"
-#include "error.h"
-#include "ue2common.h"
-#include "hs_compile.h"
-#include "util/compile_error.h"
-
-#include <cstring>
-#include <string>
-
-using std::string;
-
-static const char failureNoMemory[] = "Unable to allocate memory.";
-static const char failureInternal[] = "Internal error.";
-static const char failureBadAlloc[] = "Allocator returned misaligned memory.";
-
-extern const hs_compile_error_t hs_enomem = {
-    const_cast<char *>(failureNoMemory), 0
-};
-extern const hs_compile_error_t hs_einternal = {
-    const_cast<char *>(failureInternal), 0
-};
-extern const hs_compile_error_t hs_badalloc = {
-    const_cast<char *>(failureBadAlloc), 0
-};
-
-namespace ue2 {
-
-hs_compile_error_t *generateCompileError(const string &err, int expression) {
-    hs_compile_error_t *ret =
-        (struct hs_compile_error *)hs_misc_alloc(sizeof(hs_compile_error_t));
-    if (ret) {
-        hs_error_t e = hs_check_alloc(ret);
-        if (e != HS_SUCCESS) {
-            hs_misc_free(ret);
-            return const_cast<hs_compile_error_t *>(&hs_badalloc);
-        }
-        char *msg = (char *)hs_misc_alloc(err.size() + 1);
-        if (msg) {
-            e = hs_check_alloc(msg);
-            if (e != HS_SUCCESS) {
-                hs_misc_free(msg);
-                return const_cast<hs_compile_error_t *>(&hs_badalloc);
-            }
-            memcpy(msg, err.c_str(), err.size() + 1);
-            ret->message = msg;
-        } else {
-            hs_misc_free(ret);
-            ret = nullptr;
-        }
-    }
-
-    if (!ret || !ret->message) {
-        return const_cast<hs_compile_error_t *>(&hs_enomem);
-    }
-
-    ret->expression = expression;
-
-    return ret;
-}
-
-hs_compile_error_t *generateCompileError(const CompileError &e) {
-    return generateCompileError(e.reason, e.hasIndex ? (int)e.index : -1);
-}
-
-void freeCompileError(hs_compile_error_t *error) {
-    if (!error) {
-        return;
-    }
-    if (error == &hs_enomem || error == &hs_einternal ||
-        error == &hs_badalloc) {
-        // These are not allocated.
-        return;
-    }
-
-    hs_misc_free(error->message);
-    hs_misc_free(error);
-}
-
-} // namespace ue2

src/compiler/error.h

@@ -1,55 +0,0 @@
-/*
- * Copyright (c) 2015, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Compile-time error utils.
-  */
-
-#ifndef COMPILE_ERROR_H
-#define COMPILE_ERROR_H
-
-#include <string>
-
-struct hs_compile_error;
-
-// Special errors that aren't allocated with hs_alloc/hs_free.
-extern const hs_compile_error hs_enomem;
-extern const hs_compile_error hs_einternal;
-
-namespace ue2 {
-
-class CompileError;
-
-hs_compile_error *generateCompileError(const std::string &err, int expression);
-hs_compile_error *generateCompileError(const CompileError &e);
-
-void freeCompileError(hs_compile_error *error);
-
-} // namespace ue2
-
-#endif

src/compiler/expression_info.h

@@ -1,108 +0,0 @@
-/*
- * Copyright (c) 2017-2018, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/**
- * \file
- * \brief ExpressionInfo class for storing the properties of an expression.
- */
-
-#ifndef COMPILER_EXPRESSION_INFO_H
-#define COMPILER_EXPRESSION_INFO_H
-
-#include "ue2common.h"
-#include "som/som.h"
-
-namespace ue2 {
-
-/** \brief Properties of an expression. */
-class ExpressionInfo {
-public:
-    ExpressionInfo(unsigned int index_in, bool allow_vacuous_in,
-                   bool highlander_in, bool utf8_in, bool prefilter_in,
-                   som_type som_in, ReportID report_in, u64a min_offset_in,
-                   u64a max_offset_in, u64a min_length_in, u32 edit_distance_in,
-                   u32 hamm_distance_in, bool quiet_in)
-        : index(index_in), report(report_in), allow_vacuous(allow_vacuous_in),
-          highlander(highlander_in), utf8(utf8_in), prefilter(prefilter_in),
-          som(som_in), min_offset(min_offset_in), max_offset(max_offset_in),
-          min_length(min_length_in), edit_distance(edit_distance_in),
-          hamm_distance(hamm_distance_in), quiet(quiet_in) {}
-
-    /**
-     * \brief Index of the expression represented by this graph.
-     *
-     * Used:
-     * - down the track in error handling;
-     * - for identifying parts of an expression in highlander mode.
-     */
-    unsigned int index;
-
-    /** \brief Report ID specified by the user. */
-    ReportID report;
-
-    /** \brief Vacuous pattern is allowed. (HS_FLAG_ALLOWEMPTY) */
-    bool allow_vacuous;
-
-    /** \brief "Highlander" (single match) pattern. (HS_FLAG_SINGLEMATCH) */
-    bool highlander;
-
-    /** \brief UTF-8 pattern. (HS_FLAG_UTF8) */
-    bool utf8;
-
-    /** \brief Prefiltering pattern. (HS_FLAG_PREFILTER) */
-    bool prefilter;
-
-    /** \brief Start-of-match type requested, or SOM_NONE. */
-    som_type som;
-
-    /** \brief Minimum match offset extended parameter. 0 if not used. */
-    u64a min_offset;
-
-    /**
-     * \brief Maximum match offset extended parameter.
-     * MAX_OFFSET if not used.
-     */
-    u64a max_offset;
-
-    /** \brief Minimum match length extended parameter. 0 if not used. */
-    u64a min_length;
-
-    /**
-     * \brief Approximate matching edit distance extended parameter.
-     * 0 if not used.
-     */
-    u32 edit_distance;
-    u32 hamm_distance;
-
-    /** \brief Quiet on match. */
-    bool quiet;
-};
-
-}
-
-#endif // COMPILER_EXPRESSION_INFO_H

src/crc32.c

@@ -1,601 +0,0 @@
-/*
- * Copyright (c) 2015-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include "crc32.h"
-#include "config.h"
-#include "ue2common.h"
-#include "util/arch.h"
-
-#if !defined(HAVE_SSE42)
-
-/***
- *** What follows is derived from Intel's Slicing-by-8 CRC32 impl, which is BSD
- *** licensed and available from http://sourceforge.net/projects/slicing-by-8/
- ***/
-
-/*
- * Copyright (c) 2004-2006 Intel Corporation - All Rights Reserved
- *
- *
- * This software program is licensed subject to the BSD License, 
- * available at http://www.opensource.org/licenses/bsd-license.html.
- *
- * Abstract:
- *
- *  Tables for software CRC generation
- */
-
-/*
- * The following CRC lookup table was generated automagically
- * using the following model parameters: 
- *
- * Generator Polynomial = ................. 0x1EDC6F41
- * Generator Polynomial Length = .......... 32 bits
- * Reflected Bits = ....................... TRUE
- * Table Generation Offset = .............. 32 bits
- * Number of Slices = ..................... 8 slices
- * Slice Lengths = ........................ 8 8 8 8 8 8 8 8 
- * Directory Name = ....................... .\
- * File Name = ............................ 8x256_tables.c
- */
-
-static
-u32 crc_tableil8_o32[256] =
-{
- 0x00000000, 0xF26B8303, 0xE13B70F7, 0x1350F3F4, 0xC79A971F, 0x35F1141C, 0x26A1E7E8, 0xD4CA64EB,
- 0x8AD958CF, 0x78B2DBCC, 0x6BE22838, 0x9989AB3B, 0x4D43CFD0, 0xBF284CD3, 0xAC78BF27, 0x5E133C24,
- 0x105EC76F, 0xE235446C, 0xF165B798, 0x030E349B, 0xD7C45070, 0x25AFD373, 0x36FF2087, 0xC494A384,
- 0x9A879FA0, 0x68EC1CA3, 0x7BBCEF57, 0x89D76C54, 0x5D1D08BF, 0xAF768BBC, 0xBC267848, 0x4E4DFB4B,
- 0x20BD8EDE, 0xD2D60DDD, 0xC186FE29, 0x33ED7D2A, 0xE72719C1, 0x154C9AC2, 0x061C6936, 0xF477EA35,
- 0xAA64D611, 0x580F5512, 0x4B5FA6E6, 0xB93425E5, 0x6DFE410E, 0x9F95C20D, 0x8CC531F9, 0x7EAEB2FA,
- 0x30E349B1, 0xC288CAB2, 0xD1D83946, 0x23B3BA45, 0xF779DEAE, 0x05125DAD, 0x1642AE59, 0xE4292D5A,
- 0xBA3A117E, 0x4851927D, 0x5B016189, 0xA96AE28A, 0x7DA08661, 0x8FCB0562, 0x9C9BF696, 0x6EF07595,
- 0x417B1DBC, 0xB3109EBF, 0xA0406D4B, 0x522BEE48, 0x86E18AA3, 0x748A09A0, 0x67DAFA54, 0x95B17957,
- 0xCBA24573, 0x39C9C670, 0x2A993584, 0xD8F2B687, 0x0C38D26C, 0xFE53516F, 0xED03A29B, 0x1F682198,
- 0x5125DAD3, 0xA34E59D0, 0xB01EAA24, 0x42752927, 0x96BF4DCC, 0x64D4CECF, 0x77843D3B, 0x85EFBE38,
- 0xDBFC821C, 0x2997011F, 0x3AC7F2EB, 0xC8AC71E8, 0x1C661503, 0xEE0D9600, 0xFD5D65F4, 0x0F36E6F7,
- 0x61C69362, 0x93AD1061, 0x80FDE395, 0x72966096, 0xA65C047D, 0x5437877E, 0x4767748A, 0xB50CF789,
- 0xEB1FCBAD, 0x197448AE, 0x0A24BB5A, 0xF84F3859, 0x2C855CB2, 0xDEEEDFB1, 0xCDBE2C45, 0x3FD5AF46,
- 0x7198540D, 0x83F3D70E, 0x90A324FA, 0x62C8A7F9, 0xB602C312, 0x44694011, 0x5739B3E5, 0xA55230E6,
- 0xFB410CC2, 0x092A8FC1, 0x1A7A7C35, 0xE811FF36, 0x3CDB9BDD, 0xCEB018DE, 0xDDE0EB2A, 0x2F8B6829,
- 0x82F63B78, 0x709DB87B, 0x63CD4B8F, 0x91A6C88C, 0x456CAC67, 0xB7072F64, 0xA457DC90, 0x563C5F93,
- 0x082F63B7, 0xFA44E0B4, 0xE9141340, 0x1B7F9043, 0xCFB5F4A8, 0x3DDE77AB, 0x2E8E845F, 0xDCE5075C,
- 0x92A8FC17, 0x60C37F14, 0x73938CE0, 0x81F80FE3, 0x55326B08, 0xA759E80B, 0xB4091BFF, 0x466298FC,
- 0x1871A4D8, 0xEA1A27DB, 0xF94AD42F, 0x0B21572C, 0xDFEB33C7, 0x2D80B0C4, 0x3ED04330, 0xCCBBC033,
- 0xA24BB5A6, 0x502036A5, 0x4370C551, 0xB11B4652, 0x65D122B9, 0x97BAA1BA, 0x84EA524E, 0x7681D14D,
- 0x2892ED69, 0xDAF96E6A, 0xC9A99D9E, 0x3BC21E9D, 0xEF087A76, 0x1D63F975, 0x0E330A81, 0xFC588982,
- 0xB21572C9, 0x407EF1CA, 0x532E023E, 0xA145813D, 0x758FE5D6, 0x87E466D5, 0x94B49521, 0x66DF1622,
- 0x38CC2A06, 0xCAA7A905, 0xD9F75AF1, 0x2B9CD9F2, 0xFF56BD19, 0x0D3D3E1A, 0x1E6DCDEE, 0xEC064EED,
- 0xC38D26C4, 0x31E6A5C7, 0x22B65633, 0xD0DDD530, 0x0417B1DB, 0xF67C32D8, 0xE52CC12C, 0x1747422F,
- 0x49547E0B, 0xBB3FFD08, 0xA86F0EFC, 0x5A048DFF, 0x8ECEE914, 0x7CA56A17, 0x6FF599E3, 0x9D9E1AE0,
- 0xD3D3E1AB, 0x21B862A8, 0x32E8915C, 0xC083125F, 0x144976B4, 0xE622F5B7, 0xF5720643, 0x07198540,
- 0x590AB964, 0xAB613A67, 0xB831C993, 0x4A5A4A90, 0x9E902E7B, 0x6CFBAD78, 0x7FAB5E8C, 0x8DC0DD8F,
- 0xE330A81A, 0x115B2B19, 0x020BD8ED, 0xF0605BEE, 0x24AA3F05, 0xD6C1BC06, 0xC5914FF2, 0x37FACCF1,
- 0x69E9F0D5, 0x9B8273D6, 0x88D28022, 0x7AB90321, 0xAE7367CA, 0x5C18E4C9, 0x4F48173D, 0xBD23943E,
- 0xF36E6F75, 0x0105EC76, 0x12551F82, 0xE03E9C81, 0x34F4F86A, 0xC69F7B69, 0xD5CF889D, 0x27A40B9E,
- 0x79B737BA, 0x8BDCB4B9, 0x988C474D, 0x6AE7C44E, 0xBE2DA0A5, 0x4C4623A6, 0x5F16D052, 0xAD7D5351
-};
-
-/*
- * end of the CRC lookup table crc_tableil8_o32
- */
-
-
-
-/*
- * The following CRC lookup table was generated automagically
- * using the following model parameters: 
- *
- * Generator Polynomial = ................. 0x1EDC6F41
- * Generator Polynomial Length = .......... 32 bits
- * Reflected Bits = ....................... TRUE
- * Table Generation Offset = .............. 32 bits
- * Number of Slices = ..................... 8 slices
- * Slice Lengths = ........................ 8 8 8 8 8 8 8 8 
- * Directory Name = ....................... .\
- * File Name = ............................ 8x256_tables.c
- */
-
-static
-u32 crc_tableil8_o40[256] =
-{
- 0x00000000, 0x13A29877, 0x274530EE, 0x34E7A899, 0x4E8A61DC, 0x5D28F9AB, 0x69CF5132, 0x7A6DC945,
- 0x9D14C3B8, 0x8EB65BCF, 0xBA51F356, 0xA9F36B21, 0xD39EA264, 0xC03C3A13, 0xF4DB928A, 0xE7790AFD,
- 0x3FC5F181, 0x2C6769F6, 0x1880C16F, 0x0B225918, 0x714F905D, 0x62ED082A, 0x560AA0B3, 0x45A838C4,
- 0xA2D13239, 0xB173AA4E, 0x859402D7, 0x96369AA0, 0xEC5B53E5, 0xFFF9CB92, 0xCB1E630B, 0xD8BCFB7C,
- 0x7F8BE302, 0x6C297B75, 0x58CED3EC, 0x4B6C4B9B, 0x310182DE, 0x22A31AA9, 0x1644B230, 0x05E62A47,
- 0xE29F20BA, 0xF13DB8CD, 0xC5DA1054, 0xD6788823, 0xAC154166, 0xBFB7D911, 0x8B507188, 0x98F2E9FF,
- 0x404E1283, 0x53EC8AF4, 0x670B226D, 0x74A9BA1A, 0x0EC4735F, 0x1D66EB28, 0x298143B1, 0x3A23DBC6,
- 0xDD5AD13B, 0xCEF8494C, 0xFA1FE1D5, 0xE9BD79A2, 0x93D0B0E7, 0x80722890, 0xB4958009, 0xA737187E,
- 0xFF17C604, 0xECB55E73, 0xD852F6EA, 0xCBF06E9D, 0xB19DA7D8, 0xA23F3FAF, 0x96D89736, 0x857A0F41,
- 0x620305BC, 0x71A19DCB, 0x45463552, 0x56E4AD25, 0x2C896460, 0x3F2BFC17, 0x0BCC548E, 0x186ECCF9,
- 0xC0D23785, 0xD370AFF2, 0xE797076B, 0xF4359F1C, 0x8E585659, 0x9DFACE2E, 0xA91D66B7, 0xBABFFEC0,
- 0x5DC6F43D, 0x4E646C4A, 0x7A83C4D3, 0x69215CA4, 0x134C95E1, 0x00EE0D96, 0x3409A50F, 0x27AB3D78,
- 0x809C2506, 0x933EBD71, 0xA7D915E8, 0xB47B8D9F, 0xCE1644DA, 0xDDB4DCAD, 0xE9537434, 0xFAF1EC43,
- 0x1D88E6BE, 0x0E2A7EC9, 0x3ACDD650, 0x296F4E27, 0x53028762, 0x40A01F15, 0x7447B78C, 0x67E52FFB,
- 0xBF59D487, 0xACFB4CF0, 0x981CE469, 0x8BBE7C1E, 0xF1D3B55B, 0xE2712D2C, 0xD69685B5, 0xC5341DC2,
- 0x224D173F, 0x31EF8F48, 0x050827D1, 0x16AABFA6, 0x6CC776E3, 0x7F65EE94, 0x4B82460D, 0x5820DE7A,
- 0xFBC3FAF9, 0xE861628E, 0xDC86CA17, 0xCF245260, 0xB5499B25, 0xA6EB0352, 0x920CABCB, 0x81AE33BC,
- 0x66D73941, 0x7575A136, 0x419209AF, 0x523091D8, 0x285D589D, 0x3BFFC0EA, 0x0F186873, 0x1CBAF004,
- 0xC4060B78, 0xD7A4930F, 0xE3433B96, 0xF0E1A3E1, 0x8A8C6AA4, 0x992EF2D3, 0xADC95A4A, 0xBE6BC23D,
- 0x5912C8C0, 0x4AB050B7, 0x7E57F82E, 0x6DF56059, 0x1798A91C, 0x043A316B, 0x30DD99F2, 0x237F0185,
- 0x844819FB, 0x97EA818C, 0xA30D2915, 0xB0AFB162, 0xCAC27827, 0xD960E050, 0xED8748C9, 0xFE25D0BE,
- 0x195CDA43, 0x0AFE4234, 0x3E19EAAD, 0x2DBB72DA, 0x57D6BB9F, 0x447423E8, 0x70938B71, 0x63311306,
- 0xBB8DE87A, 0xA82F700D, 0x9CC8D894, 0x8F6A40E3, 0xF50789A6, 0xE6A511D1, 0xD242B948, 0xC1E0213F,
- 0x26992BC2, 0x353BB3B5, 0x01DC1B2C, 0x127E835B, 0x68134A1E, 0x7BB1D269, 0x4F567AF0, 0x5CF4E287,
- 0x04D43CFD, 0x1776A48A, 0x23910C13, 0x30339464, 0x4A5E5D21, 0x59FCC556, 0x6D1B6DCF, 0x7EB9F5B8,
- 0x99C0FF45, 0x8A626732, 0xBE85CFAB, 0xAD2757DC, 0xD74A9E99, 0xC4E806EE, 0xF00FAE77, 0xE3AD3600,
- 0x3B11CD7C, 0x28B3550B, 0x1C54FD92, 0x0FF665E5, 0x759BACA0, 0x663934D7, 0x52DE9C4E, 0x417C0439,
- 0xA6050EC4, 0xB5A796B3, 0x81403E2A, 0x92E2A65D, 0xE88F6F18, 0xFB2DF76F, 0xCFCA5FF6, 0xDC68C781,
- 0x7B5FDFFF, 0x68FD4788, 0x5C1AEF11, 0x4FB87766, 0x35D5BE23, 0x26772654, 0x12908ECD, 0x013216BA,
- 0xE64B1C47, 0xF5E98430, 0xC10E2CA9, 0xD2ACB4DE, 0xA8C17D9B, 0xBB63E5EC, 0x8F844D75, 0x9C26D502,
- 0x449A2E7E, 0x5738B609, 0x63DF1E90, 0x707D86E7, 0x0A104FA2, 0x19B2D7D5, 0x2D557F4C, 0x3EF7E73B,
- 0xD98EEDC6, 0xCA2C75B1, 0xFECBDD28, 0xED69455F, 0x97048C1A, 0x84A6146D, 0xB041BCF4, 0xA3E32483
-};
-
-/*
- * end of the CRC lookup table crc_tableil8_o40
- */
-
-
-
-/*
- * The following CRC lookup table was generated automagically
- * using the following model parameters: 
- *
- * Generator Polynomial = ................. 0x1EDC6F41
- * Generator Polynomial Length = .......... 32 bits
- * Reflected Bits = ....................... TRUE
- * Table Generation Offset = .............. 32 bits
- * Number of Slices = ..................... 8 slices
- * Slice Lengths = ........................ 8 8 8 8 8 8 8 8 
- * Directory Name = ....................... .\
- * File Name = ............................ 8x256_tables.c
- */
-
-static
-u32 crc_tableil8_o48[256] =
-{
- 0x00000000, 0xA541927E, 0x4F6F520D, 0xEA2EC073, 0x9EDEA41A, 0x3B9F3664, 0xD1B1F617, 0x74F06469,
- 0x38513EC5, 0x9D10ACBB, 0x773E6CC8, 0xD27FFEB6, 0xA68F9ADF, 0x03CE08A1, 0xE9E0C8D2, 0x4CA15AAC,
- 0x70A27D8A, 0xD5E3EFF4, 0x3FCD2F87, 0x9A8CBDF9, 0xEE7CD990, 0x4B3D4BEE, 0xA1138B9D, 0x045219E3,
- 0x48F3434F, 0xEDB2D131, 0x079C1142, 0xA2DD833C, 0xD62DE755, 0x736C752B, 0x9942B558, 0x3C032726,
- 0xE144FB14, 0x4405696A, 0xAE2BA919, 0x0B6A3B67, 0x7F9A5F0E, 0xDADBCD70, 0x30F50D03, 0x95B49F7D,
- 0xD915C5D1, 0x7C5457AF, 0x967A97DC, 0x333B05A2, 0x47CB61CB, 0xE28AF3B5, 0x08A433C6, 0xADE5A1B8,
- 0x91E6869E, 0x34A714E0, 0xDE89D493, 0x7BC846ED, 0x0F382284, 0xAA79B0FA, 0x40577089, 0xE516E2F7,
- 0xA9B7B85B, 0x0CF62A25, 0xE6D8EA56, 0x43997828, 0x37691C41, 0x92288E3F, 0x78064E4C, 0xDD47DC32,
- 0xC76580D9, 0x622412A7, 0x880AD2D4, 0x2D4B40AA, 0x59BB24C3, 0xFCFAB6BD, 0x16D476CE, 0xB395E4B0,
- 0xFF34BE1C, 0x5A752C62, 0xB05BEC11, 0x151A7E6F, 0x61EA1A06, 0xC4AB8878, 0x2E85480B, 0x8BC4DA75,
- 0xB7C7FD53, 0x12866F2D, 0xF8A8AF5E, 0x5DE93D20, 0x29195949, 0x8C58CB37, 0x66760B44, 0xC337993A,
- 0x8F96C396, 0x2AD751E8, 0xC0F9919B, 0x65B803E5, 0x1148678C, 0xB409F5F2, 0x5E273581, 0xFB66A7FF,
- 0x26217BCD, 0x8360E9B3, 0x694E29C0, 0xCC0FBBBE, 0xB8FFDFD7, 0x1DBE4DA9, 0xF7908DDA, 0x52D11FA4,
- 0x1E704508, 0xBB31D776, 0x511F1705, 0xF45E857B, 0x80AEE112, 0x25EF736C, 0xCFC1B31F, 0x6A802161,
- 0x56830647, 0xF3C29439, 0x19EC544A, 0xBCADC634, 0xC85DA25D, 0x6D1C3023, 0x8732F050, 0x2273622E,
- 0x6ED23882, 0xCB93AAFC, 0x21BD6A8F, 0x84FCF8F1, 0xF00C9C98, 0x554D0EE6, 0xBF63CE95, 0x1A225CEB,
- 0x8B277743, 0x2E66E53D, 0xC448254E, 0x6109B730, 0x15F9D359, 0xB0B84127, 0x5A968154, 0xFFD7132A,
- 0xB3764986, 0x1637DBF8, 0xFC191B8B, 0x595889F5, 0x2DA8ED9C, 0x88E97FE2, 0x62C7BF91, 0xC7862DEF,
- 0xFB850AC9, 0x5EC498B7, 0xB4EA58C4, 0x11ABCABA, 0x655BAED3, 0xC01A3CAD, 0x2A34FCDE, 0x8F756EA0,
- 0xC3D4340C, 0x6695A672, 0x8CBB6601, 0x29FAF47F, 0x5D0A9016, 0xF84B0268, 0x1265C21B, 0xB7245065,
- 0x6A638C57, 0xCF221E29, 0x250CDE5A, 0x804D4C24, 0xF4BD284D, 0x51FCBA33, 0xBBD27A40, 0x1E93E83E,
- 0x5232B292, 0xF77320EC, 0x1D5DE09F, 0xB81C72E1, 0xCCEC1688, 0x69AD84F6, 0x83834485, 0x26C2D6FB,
- 0x1AC1F1DD, 0xBF8063A3, 0x55AEA3D0, 0xF0EF31AE, 0x841F55C7, 0x215EC7B9, 0xCB7007CA, 0x6E3195B4,
- 0x2290CF18, 0x87D15D66, 0x6DFF9D15, 0xC8BE0F6B, 0xBC4E6B02, 0x190FF97C, 0xF321390F, 0x5660AB71,
- 0x4C42F79A, 0xE90365E4, 0x032DA597, 0xA66C37E9, 0xD29C5380, 0x77DDC1FE, 0x9DF3018D, 0x38B293F3,
- 0x7413C95F, 0xD1525B21, 0x3B7C9B52, 0x9E3D092C, 0xEACD6D45, 0x4F8CFF3B, 0xA5A23F48, 0x00E3AD36,
- 0x3CE08A10, 0x99A1186E, 0x738FD81D, 0xD6CE4A63, 0xA23E2E0A, 0x077FBC74, 0xED517C07, 0x4810EE79,
- 0x04B1B4D5, 0xA1F026AB, 0x4BDEE6D8, 0xEE9F74A6, 0x9A6F10CF, 0x3F2E82B1, 0xD50042C2, 0x7041D0BC,
- 0xAD060C8E, 0x08479EF0, 0xE2695E83, 0x4728CCFD, 0x33D8A894, 0x96993AEA, 0x7CB7FA99, 0xD9F668E7,
- 0x9557324B, 0x3016A035, 0xDA386046, 0x7F79F238, 0x0B899651, 0xAEC8042F, 0x44E6C45C, 0xE1A75622,
- 0xDDA47104, 0x78E5E37A, 0x92CB2309, 0x378AB177, 0x437AD51E, 0xE63B4760, 0x0C158713, 0xA954156D,
- 0xE5F54FC1, 0x40B4DDBF, 0xAA9A1DCC, 0x0FDB8FB2, 0x7B2BEBDB, 0xDE6A79A5, 0x3444B9D6, 0x91052BA8
-};
-
-/*
- * end of the CRC lookup table crc_tableil8_o48
- */
-
-
-
-/*
- * The following CRC lookup table was generated automagically
- * using the following model parameters: 
- *
- * Generator Polynomial = ................. 0x1EDC6F41
- * Generator Polynomial Length = .......... 32 bits
- * Reflected Bits = ....................... TRUE
- * Table Generation Offset = .............. 32 bits
- * Number of Slices = ..................... 8 slices
- * Slice Lengths = ........................ 8 8 8 8 8 8 8 8 
- * Directory Name = ....................... .\
- * File Name = ............................ 8x256_tables.c
- */
-
-static
-u32 crc_tableil8_o56[256] =
-{
- 0x00000000, 0xDD45AAB8, 0xBF672381, 0x62228939, 0x7B2231F3, 0xA6679B4B, 0xC4451272, 0x1900B8CA,
- 0xF64463E6, 0x2B01C95E, 0x49234067, 0x9466EADF, 0x8D665215, 0x5023F8AD, 0x32017194, 0xEF44DB2C,
- 0xE964B13D, 0x34211B85, 0x560392BC, 0x8B463804, 0x924680CE, 0x4F032A76, 0x2D21A34F, 0xF06409F7,
- 0x1F20D2DB, 0xC2657863, 0xA047F15A, 0x7D025BE2, 0x6402E328, 0xB9474990, 0xDB65C0A9, 0x06206A11,
- 0xD725148B, 0x0A60BE33, 0x6842370A, 0xB5079DB2, 0xAC072578, 0x71428FC0, 0x136006F9, 0xCE25AC41,
- 0x2161776D, 0xFC24DDD5, 0x9E0654EC, 0x4343FE54, 0x5A43469E, 0x8706EC26, 0xE524651F, 0x3861CFA7,
- 0x3E41A5B6, 0xE3040F0E, 0x81268637, 0x5C632C8F, 0x45639445, 0x98263EFD, 0xFA04B7C4, 0x27411D7C,
- 0xC805C650, 0x15406CE8, 0x7762E5D1, 0xAA274F69, 0xB327F7A3, 0x6E625D1B, 0x0C40D422, 0xD1057E9A,
- 0xABA65FE7, 0x76E3F55F, 0x14C17C66, 0xC984D6DE, 0xD0846E14, 0x0DC1C4AC, 0x6FE34D95, 0xB2A6E72D,
- 0x5DE23C01, 0x80A796B9, 0xE2851F80, 0x3FC0B538, 0x26C00DF2, 0xFB85A74A, 0x99A72E73, 0x44E284CB,
- 0x42C2EEDA, 0x9F874462, 0xFDA5CD5B, 0x20E067E3, 0x39E0DF29, 0xE4A57591, 0x8687FCA8, 0x5BC25610,
- 0xB4868D3C, 0x69C32784, 0x0BE1AEBD, 0xD6A40405, 0xCFA4BCCF, 0x12E11677, 0x70C39F4E, 0xAD8635F6,
- 0x7C834B6C, 0xA1C6E1D4, 0xC3E468ED, 0x1EA1C255, 0x07A17A9F, 0xDAE4D027, 0xB8C6591E, 0x6583F3A6,
- 0x8AC7288A, 0x57828232, 0x35A00B0B, 0xE8E5A1B3, 0xF1E51979, 0x2CA0B3C1, 0x4E823AF8, 0x93C79040,
- 0x95E7FA51, 0x48A250E9, 0x2A80D9D0, 0xF7C57368, 0xEEC5CBA2, 0x3380611A, 0x51A2E823, 0x8CE7429B,
- 0x63A399B7, 0xBEE6330F, 0xDCC4BA36, 0x0181108E, 0x1881A844, 0xC5C402FC, 0xA7E68BC5, 0x7AA3217D,
- 0x52A0C93F, 0x8FE56387, 0xEDC7EABE, 0x30824006, 0x2982F8CC, 0xF4C75274, 0x96E5DB4D, 0x4BA071F5,
- 0xA4E4AAD9, 0x79A10061, 0x1B838958, 0xC6C623E0, 0xDFC69B2A, 0x02833192, 0x60A1B8AB, 0xBDE41213,
- 0xBBC47802, 0x6681D2BA, 0x04A35B83, 0xD9E6F13B, 0xC0E649F1, 0x1DA3E349, 0x7F816A70, 0xA2C4C0C8,
- 0x4D801BE4, 0x90C5B15C, 0xF2E73865, 0x2FA292DD, 0x36A22A17, 0xEBE780AF, 0x89C50996, 0x5480A32E,
- 0x8585DDB4, 0x58C0770C, 0x3AE2FE35, 0xE7A7548D, 0xFEA7EC47, 0x23E246FF, 0x41C0CFC6, 0x9C85657E,
- 0x73C1BE52, 0xAE8414EA, 0xCCA69DD3, 0x11E3376B, 0x08E38FA1, 0xD5A62519, 0xB784AC20, 0x6AC10698,
- 0x6CE16C89, 0xB1A4C631, 0xD3864F08, 0x0EC3E5B0, 0x17C35D7A, 0xCA86F7C2, 0xA8A47EFB, 0x75E1D443,
- 0x9AA50F6F, 0x47E0A5D7, 0x25C22CEE, 0xF8878656, 0xE1873E9C, 0x3CC29424, 0x5EE01D1D, 0x83A5B7A5,
- 0xF90696D8, 0x24433C60, 0x4661B559, 0x9B241FE1, 0x8224A72B, 0x5F610D93, 0x3D4384AA, 0xE0062E12,
- 0x0F42F53E, 0xD2075F86, 0xB025D6BF, 0x6D607C07, 0x7460C4CD, 0xA9256E75, 0xCB07E74C, 0x16424DF4,
- 0x106227E5, 0xCD278D5D, 0xAF050464, 0x7240AEDC, 0x6B401616, 0xB605BCAE, 0xD4273597, 0x09629F2F,
- 0xE6264403, 0x3B63EEBB, 0x59416782, 0x8404CD3A, 0x9D0475F0, 0x4041DF48, 0x22635671, 0xFF26FCC9,
- 0x2E238253, 0xF36628EB, 0x9144A1D2, 0x4C010B6A, 0x5501B3A0, 0x88441918, 0xEA669021, 0x37233A99,
- 0xD867E1B5, 0x05224B0D, 0x6700C234, 0xBA45688C, 0xA345D046, 0x7E007AFE, 0x1C22F3C7, 0xC167597F,
- 0xC747336E, 0x1A0299D6, 0x782010EF, 0xA565BA57, 0xBC65029D, 0x6120A825, 0x0302211C, 0xDE478BA4,
- 0x31035088, 0xEC46FA30, 0x8E647309, 0x5321D9B1, 0x4A21617B, 0x9764CBC3, 0xF54642FA, 0x2803E842
-};
-
-/*
- * end of the CRC lookup table crc_tableil8_o56
- */
-
-
-
-/*
- * The following CRC lookup table was generated automagically
- * using the following model parameters: 
- *
- * Generator Polynomial = ................. 0x1EDC6F41
- * Generator Polynomial Length = .......... 32 bits
- * Reflected Bits = ....................... TRUE
- * Table Generation Offset = .............. 32 bits
- * Number of Slices = ..................... 8 slices
- * Slice Lengths = ........................ 8 8 8 8 8 8 8 8 
- * Directory Name = ....................... .\
- * File Name = ............................ 8x256_tables.c
- */
-
-static
-u32 crc_tableil8_o64[256] =
-{
- 0x00000000, 0x38116FAC, 0x7022DF58, 0x4833B0F4, 0xE045BEB0, 0xD854D11C, 0x906761E8, 0xA8760E44,
- 0xC5670B91, 0xFD76643D, 0xB545D4C9, 0x8D54BB65, 0x2522B521, 0x1D33DA8D, 0x55006A79, 0x6D1105D5,
- 0x8F2261D3, 0xB7330E7F, 0xFF00BE8B, 0xC711D127, 0x6F67DF63, 0x5776B0CF, 0x1F45003B, 0x27546F97,
- 0x4A456A42, 0x725405EE, 0x3A67B51A, 0x0276DAB6, 0xAA00D4F2, 0x9211BB5E, 0xDA220BAA, 0xE2336406,
- 0x1BA8B557, 0x23B9DAFB, 0x6B8A6A0F, 0x539B05A3, 0xFBED0BE7, 0xC3FC644B, 0x8BCFD4BF, 0xB3DEBB13,
- 0xDECFBEC6, 0xE6DED16A, 0xAEED619E, 0x96FC0E32, 0x3E8A0076, 0x069B6FDA, 0x4EA8DF2E, 0x76B9B082,
- 0x948AD484, 0xAC9BBB28, 0xE4A80BDC, 0xDCB96470, 0x74CF6A34, 0x4CDE0598, 0x04EDB56C, 0x3CFCDAC0,
- 0x51EDDF15, 0x69FCB0B9, 0x21CF004D, 0x19DE6FE1, 0xB1A861A5, 0x89B90E09, 0xC18ABEFD, 0xF99BD151,
- 0x37516AAE, 0x0F400502, 0x4773B5F6, 0x7F62DA5A, 0xD714D41E, 0xEF05BBB2, 0xA7360B46, 0x9F2764EA,
- 0xF236613F, 0xCA270E93, 0x8214BE67, 0xBA05D1CB, 0x1273DF8F, 0x2A62B023, 0x625100D7, 0x5A406F7B,
- 0xB8730B7D, 0x806264D1, 0xC851D425, 0xF040BB89, 0x5836B5CD, 0x6027DA61, 0x28146A95, 0x10050539,
- 0x7D1400EC, 0x45056F40, 0x0D36DFB4, 0x3527B018, 0x9D51BE5C, 0xA540D1F0, 0xED736104, 0xD5620EA8,
- 0x2CF9DFF9, 0x14E8B055, 0x5CDB00A1, 0x64CA6F0D, 0xCCBC6149, 0xF4AD0EE5, 0xBC9EBE11, 0x848FD1BD,
- 0xE99ED468, 0xD18FBBC4, 0x99BC0B30, 0xA1AD649C, 0x09DB6AD8, 0x31CA0574, 0x79F9B580, 0x41E8DA2C,
- 0xA3DBBE2A, 0x9BCAD186, 0xD3F96172, 0xEBE80EDE, 0x439E009A, 0x7B8F6F36, 0x33BCDFC2, 0x0BADB06E,
- 0x66BCB5BB, 0x5EADDA17, 0x169E6AE3, 0x2E8F054F, 0x86F90B0B, 0xBEE864A7, 0xF6DBD453, 0xCECABBFF,
- 0x6EA2D55C, 0x56B3BAF0, 0x1E800A04, 0x269165A8, 0x8EE76BEC, 0xB6F60440, 0xFEC5B4B4, 0xC6D4DB18,
- 0xABC5DECD, 0x93D4B161, 0xDBE70195, 0xE3F66E39, 0x4B80607D, 0x73910FD1, 0x3BA2BF25, 0x03B3D089,
- 0xE180B48F, 0xD991DB23, 0x91A26BD7, 0xA9B3047B, 0x01C50A3F, 0x39D46593, 0x71E7D567, 0x49F6BACB,
- 0x24E7BF1E, 0x1CF6D0B2, 0x54C56046, 0x6CD40FEA, 0xC4A201AE, 0xFCB36E02, 0xB480DEF6, 0x8C91B15A,
- 0x750A600B, 0x4D1B0FA7, 0x0528BF53, 0x3D39D0FF, 0x954FDEBB, 0xAD5EB117, 0xE56D01E3, 0xDD7C6E4F,
- 0xB06D6B9A, 0x887C0436, 0xC04FB4C2, 0xF85EDB6E, 0x5028D52A, 0x6839BA86, 0x200A0A72, 0x181B65DE,
- 0xFA2801D8, 0xC2396E74, 0x8A0ADE80, 0xB21BB12C, 0x1A6DBF68, 0x227CD0C4, 0x6A4F6030, 0x525E0F9C,
- 0x3F4F0A49, 0x075E65E5, 0x4F6DD511, 0x777CBABD, 0xDF0AB4F9, 0xE71BDB55, 0xAF286BA1, 0x9739040D,
- 0x59F3BFF2, 0x61E2D05E, 0x29D160AA, 0x11C00F06, 0xB9B60142, 0x81A76EEE, 0xC994DE1A, 0xF185B1B6,
- 0x9C94B463, 0xA485DBCF, 0xECB66B3B, 0xD4A70497, 0x7CD10AD3, 0x44C0657F, 0x0CF3D58B, 0x34E2BA27,
- 0xD6D1DE21, 0xEEC0B18D, 0xA6F30179, 0x9EE26ED5, 0x36946091, 0x0E850F3D, 0x46B6BFC9, 0x7EA7D065,
- 0x13B6D5B0, 0x2BA7BA1C, 0x63940AE8, 0x5B856544, 0xF3F36B00, 0xCBE204AC, 0x83D1B458, 0xBBC0DBF4,
- 0x425B0AA5, 0x7A4A6509, 0x3279D5FD, 0x0A68BA51, 0xA21EB415, 0x9A0FDBB9, 0xD23C6B4D, 0xEA2D04E1,
- 0x873C0134, 0xBF2D6E98, 0xF71EDE6C, 0xCF0FB1C0, 0x6779BF84, 0x5F68D028, 0x175B60DC, 0x2F4A0F70,
- 0xCD796B76, 0xF56804DA, 0xBD5BB42E, 0x854ADB82, 0x2D3CD5C6, 0x152DBA6A, 0x5D1E0A9E, 0x650F6532,
- 0x081E60E7, 0x300F0F4B, 0x783CBFBF, 0x402DD013, 0xE85BDE57, 0xD04AB1FB, 0x9879010F, 0xA0686EA3
-};
-
-/*
- * end of the CRC lookup table crc_tableil8_o64
- */
-
-
-
-/*
- * The following CRC lookup table was generated automagically
- * using the following model parameters: 
- *
- * Generator Polynomial = ................. 0x1EDC6F41
- * Generator Polynomial Length = .......... 32 bits
- * Reflected Bits = ....................... TRUE
- * Table Generation Offset = .............. 32 bits
- * Number of Slices = ..................... 8 slices
- * Slice Lengths = ........................ 8 8 8 8 8 8 8 8 
- * Directory Name = ....................... .\
- * File Name = ............................ 8x256_tables.c
- */
-
-static
-u32 crc_tableil8_o72[256] =
-{
- 0x00000000, 0xEF306B19, 0xDB8CA0C3, 0x34BCCBDA, 0xB2F53777, 0x5DC55C6E, 0x697997B4, 0x8649FCAD,
- 0x6006181F, 0x8F367306, 0xBB8AB8DC, 0x54BAD3C5, 0xD2F32F68, 0x3DC34471, 0x097F8FAB, 0xE64FE4B2,
- 0xC00C303E, 0x2F3C5B27, 0x1B8090FD, 0xF4B0FBE4, 0x72F90749, 0x9DC96C50, 0xA975A78A, 0x4645CC93,
- 0xA00A2821, 0x4F3A4338, 0x7B8688E2, 0x94B6E3FB, 0x12FF1F56, 0xFDCF744F, 0xC973BF95, 0x2643D48C,
- 0x85F4168D, 0x6AC47D94, 0x5E78B64E, 0xB148DD57, 0x370121FA, 0xD8314AE3, 0xEC8D8139, 0x03BDEA20,
- 0xE5F20E92, 0x0AC2658B, 0x3E7EAE51, 0xD14EC548, 0x570739E5, 0xB83752FC, 0x8C8B9926, 0x63BBF23F,
- 0x45F826B3, 0xAAC84DAA, 0x9E748670, 0x7144ED69, 0xF70D11C4, 0x183D7ADD, 0x2C81B107, 0xC3B1DA1E,
- 0x25FE3EAC, 0xCACE55B5, 0xFE729E6F, 0x1142F576, 0x970B09DB, 0x783B62C2, 0x4C87A918, 0xA3B7C201,
- 0x0E045BEB, 0xE13430F2, 0xD588FB28, 0x3AB89031, 0xBCF16C9C, 0x53C10785, 0x677DCC5F, 0x884DA746,
- 0x6E0243F4, 0x813228ED, 0xB58EE337, 0x5ABE882E, 0xDCF77483, 0x33C71F9A, 0x077BD440, 0xE84BBF59,
- 0xCE086BD5, 0x213800CC, 0x1584CB16, 0xFAB4A00F, 0x7CFD5CA2, 0x93CD37BB, 0xA771FC61, 0x48419778,
- 0xAE0E73CA, 0x413E18D3, 0x7582D309, 0x9AB2B810, 0x1CFB44BD, 0xF3CB2FA4, 0xC777E47E, 0x28478F67,
- 0x8BF04D66, 0x64C0267F, 0x507CEDA5, 0xBF4C86BC, 0x39057A11, 0xD6351108, 0xE289DAD2, 0x0DB9B1CB,
- 0xEBF65579, 0x04C63E60, 0x307AF5BA, 0xDF4A9EA3, 0x5903620E, 0xB6330917, 0x828FC2CD, 0x6DBFA9D4,
- 0x4BFC7D58, 0xA4CC1641, 0x9070DD9B, 0x7F40B682, 0xF9094A2F, 0x16392136, 0x2285EAEC, 0xCDB581F5,
- 0x2BFA6547, 0xC4CA0E5E, 0xF076C584, 0x1F46AE9D, 0x990F5230, 0x763F3929, 0x4283F2F3, 0xADB399EA,
- 0x1C08B7D6, 0xF338DCCF, 0xC7841715, 0x28B47C0C, 0xAEFD80A1, 0x41CDEBB8, 0x75712062, 0x9A414B7B,
- 0x7C0EAFC9, 0x933EC4D0, 0xA7820F0A, 0x48B26413, 0xCEFB98BE, 0x21CBF3A7, 0x1577387D, 0xFA475364,
- 0xDC0487E8, 0x3334ECF1, 0x0788272B, 0xE8B84C32, 0x6EF1B09F, 0x81C1DB86, 0xB57D105C, 0x5A4D7B45,
- 0xBC029FF7, 0x5332F4EE, 0x678E3F34, 0x88BE542D, 0x0EF7A880, 0xE1C7C399, 0xD57B0843, 0x3A4B635A,
- 0x99FCA15B, 0x76CCCA42, 0x42700198, 0xAD406A81, 0x2B09962C, 0xC439FD35, 0xF08536EF, 0x1FB55DF6,
- 0xF9FAB944, 0x16CAD25D, 0x22761987, 0xCD46729E, 0x4B0F8E33, 0xA43FE52A, 0x90832EF0, 0x7FB345E9,
- 0x59F09165, 0xB6C0FA7C, 0x827C31A6, 0x6D4C5ABF, 0xEB05A612, 0x0435CD0B, 0x308906D1, 0xDFB96DC8,
- 0x39F6897A, 0xD6C6E263, 0xE27A29B9, 0x0D4A42A0, 0x8B03BE0D, 0x6433D514, 0x508F1ECE, 0xBFBF75D7,
- 0x120CEC3D, 0xFD3C8724, 0xC9804CFE, 0x26B027E7, 0xA0F9DB4A, 0x4FC9B053, 0x7B757B89, 0x94451090,
- 0x720AF422, 0x9D3A9F3B, 0xA98654E1, 0x46B63FF8, 0xC0FFC355, 0x2FCFA84C, 0x1B736396, 0xF443088F,
- 0xD200DC03, 0x3D30B71A, 0x098C7CC0, 0xE6BC17D9, 0x60F5EB74, 0x8FC5806D, 0xBB794BB7, 0x544920AE,
- 0xB206C41C, 0x5D36AF05, 0x698A64DF, 0x86BA0FC6, 0x00F3F36B, 0xEFC39872, 0xDB7F53A8, 0x344F38B1,
- 0x97F8FAB0, 0x78C891A9, 0x4C745A73, 0xA344316A, 0x250DCDC7, 0xCA3DA6DE, 0xFE816D04, 0x11B1061D,
- 0xF7FEE2AF, 0x18CE89B6, 0x2C72426C, 0xC3422975, 0x450BD5D8, 0xAA3BBEC1, 0x9E87751B, 0x71B71E02,
- 0x57F4CA8E, 0xB8C4A197, 0x8C786A4D, 0x63480154, 0xE501FDF9, 0x0A3196E0, 0x3E8D5D3A, 0xD1BD3623,
- 0x37F2D291, 0xD8C2B988, 0xEC7E7252, 0x034E194B, 0x8507E5E6, 0x6A378EFF, 0x5E8B4525, 0xB1BB2E3C
-};
-
-/*
- * end of the CRC lookup table crc_tableil8_o72
- */
-
-
-
-/*
- * The following CRC lookup table was generated automagically
- * using the following model parameters: 
- *
- * Generator Polynomial = ................. 0x1EDC6F41
- * Generator Polynomial Length = .......... 32 bits
- * Reflected Bits = ....................... TRUE
- * Table Generation Offset = .............. 32 bits
- * Number of Slices = ..................... 8 slices
- * Slice Lengths = ........................ 8 8 8 8 8 8 8 8 
- * Directory Name = ....................... .\
- * File Name = ............................ 8x256_tables.c
- */
-
-static
-u32 crc_tableil8_o80[256] =
-{
- 0x00000000, 0x68032CC8, 0xD0065990, 0xB8057558, 0xA5E0C5D1, 0xCDE3E919, 0x75E69C41, 0x1DE5B089,
- 0x4E2DFD53, 0x262ED19B, 0x9E2BA4C3, 0xF628880B, 0xEBCD3882, 0x83CE144A, 0x3BCB6112, 0x53C84DDA,
- 0x9C5BFAA6, 0xF458D66E, 0x4C5DA336, 0x245E8FFE, 0x39BB3F77, 0x51B813BF, 0xE9BD66E7, 0x81BE4A2F,
- 0xD27607F5, 0xBA752B3D, 0x02705E65, 0x6A7372AD, 0x7796C224, 0x1F95EEEC, 0xA7909BB4, 0xCF93B77C,
- 0x3D5B83BD, 0x5558AF75, 0xED5DDA2D, 0x855EF6E5, 0x98BB466C, 0xF0B86AA4, 0x48BD1FFC, 0x20BE3334,
- 0x73767EEE, 0x1B755226, 0xA370277E, 0xCB730BB6, 0xD696BB3F, 0xBE9597F7, 0x0690E2AF, 0x6E93CE67,
- 0xA100791B, 0xC90355D3, 0x7106208B, 0x19050C43, 0x04E0BCCA, 0x6CE39002, 0xD4E6E55A, 0xBCE5C992,
- 0xEF2D8448, 0x872EA880, 0x3F2BDDD8, 0x5728F110, 0x4ACD4199, 0x22CE6D51, 0x9ACB1809, 0xF2C834C1,
- 0x7AB7077A, 0x12B42BB2, 0xAAB15EEA, 0xC2B27222, 0xDF57C2AB, 0xB754EE63, 0x0F519B3B, 0x6752B7F3,
- 0x349AFA29, 0x5C99D6E1, 0xE49CA3B9, 0x8C9F8F71, 0x917A3FF8, 0xF9791330, 0x417C6668, 0x297F4AA0,
- 0xE6ECFDDC, 0x8EEFD114, 0x36EAA44C, 0x5EE98884, 0x430C380D, 0x2B0F14C5, 0x930A619D, 0xFB094D55,
- 0xA8C1008F, 0xC0C22C47, 0x78C7591F, 0x10C475D7, 0x0D21C55E, 0x6522E996, 0xDD279CCE, 0xB524B006,
- 0x47EC84C7, 0x2FEFA80F, 0x97EADD57, 0xFFE9F19F, 0xE20C4116, 0x8A0F6DDE, 0x320A1886, 0x5A09344E,
- 0x09C17994, 0x61C2555C, 0xD9C72004, 0xB1C40CCC, 0xAC21BC45, 0xC422908D, 0x7C27E5D5, 0x1424C91D,
- 0xDBB77E61, 0xB3B452A9, 0x0BB127F1, 0x63B20B39, 0x7E57BBB0, 0x16549778, 0xAE51E220, 0xC652CEE8,
- 0x959A8332, 0xFD99AFFA, 0x459CDAA2, 0x2D9FF66A, 0x307A46E3, 0x58796A2B, 0xE07C1F73, 0x887F33BB,
- 0xF56E0EF4, 0x9D6D223C, 0x25685764, 0x4D6B7BAC, 0x508ECB25, 0x388DE7ED, 0x808892B5, 0xE88BBE7D,
- 0xBB43F3A7, 0xD340DF6F, 0x6B45AA37, 0x034686FF, 0x1EA33676, 0x76A01ABE, 0xCEA56FE6, 0xA6A6432E,
- 0x6935F452, 0x0136D89A, 0xB933ADC2, 0xD130810A, 0xCCD53183, 0xA4D61D4B, 0x1CD36813, 0x74D044DB,
- 0x27180901, 0x4F1B25C9, 0xF71E5091, 0x9F1D7C59, 0x82F8CCD0, 0xEAFBE018, 0x52FE9540, 0x3AFDB988,
- 0xC8358D49, 0xA036A181, 0x1833D4D9, 0x7030F811, 0x6DD54898, 0x05D66450, 0xBDD31108, 0xD5D03DC0,
- 0x8618701A, 0xEE1B5CD2, 0x561E298A, 0x3E1D0542, 0x23F8B5CB, 0x4BFB9903, 0xF3FEEC5B, 0x9BFDC093,
- 0x546E77EF, 0x3C6D5B27, 0x84682E7F, 0xEC6B02B7, 0xF18EB23E, 0x998D9EF6, 0x2188EBAE, 0x498BC766,
- 0x1A438ABC, 0x7240A674, 0xCA45D32C, 0xA246FFE4, 0xBFA34F6D, 0xD7A063A5, 0x6FA516FD, 0x07A63A35,
- 0x8FD9098E, 0xE7DA2546, 0x5FDF501E, 0x37DC7CD6, 0x2A39CC5F, 0x423AE097, 0xFA3F95CF, 0x923CB907,
- 0xC1F4F4DD, 0xA9F7D815, 0x11F2AD4D, 0x79F18185, 0x6414310C, 0x0C171DC4, 0xB412689C, 0xDC114454,
- 0x1382F328, 0x7B81DFE0, 0xC384AAB8, 0xAB878670, 0xB66236F9, 0xDE611A31, 0x66646F69, 0x0E6743A1,
- 0x5DAF0E7B, 0x35AC22B3, 0x8DA957EB, 0xE5AA7B23, 0xF84FCBAA, 0x904CE762, 0x2849923A, 0x404ABEF2,
- 0xB2828A33, 0xDA81A6FB, 0x6284D3A3, 0x0A87FF6B, 0x17624FE2, 0x7F61632A, 0xC7641672, 0xAF673ABA,
- 0xFCAF7760, 0x94AC5BA8, 0x2CA92EF0, 0x44AA0238, 0x594FB2B1, 0x314C9E79, 0x8949EB21, 0xE14AC7E9,
- 0x2ED97095, 0x46DA5C5D, 0xFEDF2905, 0x96DC05CD, 0x8B39B544, 0xE33A998C, 0x5B3FECD4, 0x333CC01C,
- 0x60F48DC6, 0x08F7A10E, 0xB0F2D456, 0xD8F1F89E, 0xC5144817, 0xAD1764DF, 0x15121187, 0x7D113D4F
-};
-
-/*
- * end of the CRC lookup table crc_tableil8_o80
- */
-
-
-
-/*
- * The following CRC lookup table was generated automagically
- * using the following model parameters: 
- *
- * Generator Polynomial = ................. 0x1EDC6F41
- * Generator Polynomial Length = .......... 32 bits
- * Reflected Bits = ....................... TRUE
- * Table Generation Offset = .............. 32 bits
- * Number of Slices = ..................... 8 slices
- * Slice Lengths = ........................ 8 8 8 8 8 8 8 8 
- * Directory Name = ....................... .\
- * File Name = ............................ 8x256_tables.c
- */
-
-static
-u32 crc_tableil8_o88[256] =
-{
- 0x00000000, 0x493C7D27, 0x9278FA4E, 0xDB448769, 0x211D826D, 0x6821FF4A, 0xB3657823, 0xFA590504,
- 0x423B04DA, 0x0B0779FD, 0xD043FE94, 0x997F83B3, 0x632686B7, 0x2A1AFB90, 0xF15E7CF9, 0xB86201DE,
- 0x847609B4, 0xCD4A7493, 0x160EF3FA, 0x5F328EDD, 0xA56B8BD9, 0xEC57F6FE, 0x37137197, 0x7E2F0CB0,
- 0xC64D0D6E, 0x8F717049, 0x5435F720, 0x1D098A07, 0xE7508F03, 0xAE6CF224, 0x7528754D, 0x3C14086A,
- 0x0D006599, 0x443C18BE, 0x9F789FD7, 0xD644E2F0, 0x2C1DE7F4, 0x65219AD3, 0xBE651DBA, 0xF759609D,
- 0x4F3B6143, 0x06071C64, 0xDD439B0D, 0x947FE62A, 0x6E26E32E, 0x271A9E09, 0xFC5E1960, 0xB5626447,
- 0x89766C2D, 0xC04A110A, 0x1B0E9663, 0x5232EB44, 0xA86BEE40, 0xE1579367, 0x3A13140E, 0x732F6929,
- 0xCB4D68F7, 0x827115D0, 0x593592B9, 0x1009EF9E, 0xEA50EA9A, 0xA36C97BD, 0x782810D4, 0x31146DF3,
- 0x1A00CB32, 0x533CB615, 0x8878317C, 0xC1444C5B, 0x3B1D495F, 0x72213478, 0xA965B311, 0xE059CE36,
- 0x583BCFE8, 0x1107B2CF, 0xCA4335A6, 0x837F4881, 0x79264D85, 0x301A30A2, 0xEB5EB7CB, 0xA262CAEC,
- 0x9E76C286, 0xD74ABFA1, 0x0C0E38C8, 0x453245EF, 0xBF6B40EB, 0xF6573DCC, 0x2D13BAA5, 0x642FC782,
- 0xDC4DC65C, 0x9571BB7B, 0x4E353C12, 0x07094135, 0xFD504431, 0xB46C3916, 0x6F28BE7F, 0x2614C358,
- 0x1700AEAB, 0x5E3CD38C, 0x857854E5, 0xCC4429C2, 0x361D2CC6, 0x7F2151E1, 0xA465D688, 0xED59ABAF,
- 0x553BAA71, 0x1C07D756, 0xC743503F, 0x8E7F2D18, 0x7426281C, 0x3D1A553B, 0xE65ED252, 0xAF62AF75,
- 0x9376A71F, 0xDA4ADA38, 0x010E5D51, 0x48322076, 0xB26B2572, 0xFB575855, 0x2013DF3C, 0x692FA21B,
- 0xD14DA3C5, 0x9871DEE2, 0x4335598B, 0x0A0924AC, 0xF05021A8, 0xB96C5C8F, 0x6228DBE6, 0x2B14A6C1,
- 0x34019664, 0x7D3DEB43, 0xA6796C2A, 0xEF45110D, 0x151C1409, 0x5C20692E, 0x8764EE47, 0xCE589360,
- 0x763A92BE, 0x3F06EF99, 0xE44268F0, 0xAD7E15D7, 0x572710D3, 0x1E1B6DF4, 0xC55FEA9D, 0x8C6397BA,
- 0xB0779FD0, 0xF94BE2F7, 0x220F659E, 0x6B3318B9, 0x916A1DBD, 0xD856609A, 0x0312E7F3, 0x4A2E9AD4,
- 0xF24C9B0A, 0xBB70E62D, 0x60346144, 0x29081C63, 0xD3511967, 0x9A6D6440, 0x4129E329, 0x08159E0E,
- 0x3901F3FD, 0x703D8EDA, 0xAB7909B3, 0xE2457494, 0x181C7190, 0x51200CB7, 0x8A648BDE, 0xC358F6F9,
- 0x7B3AF727, 0x32068A00, 0xE9420D69, 0xA07E704E, 0x5A27754A, 0x131B086D, 0xC85F8F04, 0x8163F223,
- 0xBD77FA49, 0xF44B876E, 0x2F0F0007, 0x66337D20, 0x9C6A7824, 0xD5560503, 0x0E12826A, 0x472EFF4D,
- 0xFF4CFE93, 0xB67083B4, 0x6D3404DD, 0x240879FA, 0xDE517CFE, 0x976D01D9, 0x4C2986B0, 0x0515FB97,
- 0x2E015D56, 0x673D2071, 0xBC79A718, 0xF545DA3F, 0x0F1CDF3B, 0x4620A21C, 0x9D642575, 0xD4585852,
- 0x6C3A598C, 0x250624AB, 0xFE42A3C2, 0xB77EDEE5, 0x4D27DBE1, 0x041BA6C6, 0xDF5F21AF, 0x96635C88,
- 0xAA7754E2, 0xE34B29C5, 0x380FAEAC, 0x7133D38B, 0x8B6AD68F, 0xC256ABA8, 0x19122CC1, 0x502E51E6,
- 0xE84C5038, 0xA1702D1F, 0x7A34AA76, 0x3308D751, 0xC951D255, 0x806DAF72, 0x5B29281B, 0x1215553C,
- 0x230138CF, 0x6A3D45E8, 0xB179C281, 0xF845BFA6, 0x021CBAA2, 0x4B20C785, 0x906440EC, 0xD9583DCB,
- 0x613A3C15, 0x28064132, 0xF342C65B, 0xBA7EBB7C, 0x4027BE78, 0x091BC35F, 0xD25F4436, 0x9B633911,
- 0xA777317B, 0xEE4B4C5C, 0x350FCB35, 0x7C33B612, 0x866AB316, 0xCF56CE31, 0x14124958, 0x5D2E347F,
- 0xE54C35A1, 0xAC704886, 0x7734CFEF, 0x3E08B2C8, 0xC451B7CC, 0x8D6DCAEB, 0x56294D82, 0x1F1530A5
-};
-
-/*
- * end of the CRC lookup table crc_tableil8_o88
- */
-
-//#define VERIFY_ASSERTION
-
-#ifdef VERIFY_ASSERTION
-
-// Trivial byte-by-byte version: you can switch on the assertion in the
-// Crc32_ComputeBuf function (by defining VERIFY_ASSERTION) to check this
-// against the slicing variant.
-static really_inline
-u32 crc32c(u32 running_crc, const unsigned char* p_buf, size_t length) {
-    u32 crc = running_crc;
-    while (length--) {
-        crc = crc_tableil8_o32[(crc ^ *p_buf++) & 0x000000FF] ^ (crc >> 8);
-    }
-    return crc;
-}
-
-#endif // VERIFY_ASSERTION
-
-// Slicing-by-8 approach, which is much faster. Derived from Intel's
-// BSD-licensed code, with additions to handled aligned case automatically.
-static really_inline
-u32 crc32c_sb8_64_bit(u32 running_crc, const unsigned char* p_buf,
-                      const size_t length) {
-    u32 crc = running_crc;
-
-    // Process byte-by-byte until p_buf is aligned
-
-    const unsigned char *aligned_buf = ROUNDUP_PTR(p_buf, 4);
-    size_t init_bytes = aligned_buf - p_buf;
-    size_t running_length = ((length - init_bytes)/8)*8;
-    size_t end_bytes = length - init_bytes - running_length;
-
-    while (p_buf < aligned_buf) {
-        crc = crc_tableil8_o32[(crc ^ *p_buf++) & 0x000000FF] ^ (crc >> 8);
-    }
-
-    // Main aligned loop, processes eight bytes at a time.
-
-    u32 term1, term2;
-    for (size_t li = 0; li < running_length/8; li++) {
-        u32 block = *(const u32 *)p_buf;
-        crc ^= block;
-        p_buf += 4;
-        term1 = crc_tableil8_o88[crc & 0x000000FF] ^
-                crc_tableil8_o80[(crc >> 8) & 0x000000FF];
-        term2 = crc >> 16;
-        crc = term1 ^
-              crc_tableil8_o72[term2 & 0x000000FF] ^
-              crc_tableil8_o64[(term2 >> 8) & 0x000000FF];
-
-
-        block = *(const u32 *)p_buf;
-
-        term1 = crc_tableil8_o56[block & 0x000000FF] ^
-                crc_tableil8_o48[(block >> 8) & 0x000000FF];
-
-        term2 = block >> 16;
-        crc =   crc ^
-                term1 ^
-                crc_tableil8_o40[term2 & 0x000000FF] ^
-                crc_tableil8_o32[(term2 >> 8) & 0x000000FF];
-        p_buf += 4;
-    }
-
-    // Remaining bytes
-
-    for(size_t li = 0; li < end_bytes; li++) {
-        crc = crc_tableil8_o32[(crc ^ *p_buf++) & 0x000000FF] ^ (crc >> 8);
-    }
-
-    return crc;
-}
-
-#else // HAVE_SSE42
-#include "util/arch/x86/crc32.h"
-#endif
-
-#ifdef VERIFY_ASSERTION
-#include <assert.h>
-#endif
-
-// Externally visible function
-u32 Crc32c_ComputeBuf(u32 inCrc32, const void *buf, size_t bufLen) {
-#if defined(HAVE_SSE42)
-    u32 crc = crc32c_sse42(inCrc32, (const unsigned char *)buf, bufLen);
-#else
-    u32 crc = crc32c_sb8_64_bit(inCrc32, (const unsigned char *)buf, bufLen);
-#endif
-
-#ifdef VERIFY_ASSERTION
-    assert(crc == crc32c(inCrc32, (const unsigned char *)buf, bufLen));
-#endif
-
-    return crc;
-}

src/crc32.h

@@ -1,46 +0,0 @@
-/*
- * Copyright (c) 2015, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef CRC32_H_36A5015B5840C1
-#define CRC32_H_36A5015B5840C1
-
-#include "ue2common.h"
-
-#ifdef __cplusplus
-extern "C"
-{
-#endif
-
-u32 Crc32c_ComputeBuf(u32 inCrc32, const void *buf, size_t bufLen);
-
-#ifdef __cplusplus
-}
-#endif
-
-#endif /* CRC32_H_36A5015B5840C1 */
-

src/database.c

@@ -1,454 +0,0 @@
-/*
- * Copyright (c) 2015-2020, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Runtime code for hs_database manipulation.
-  */
-
-#include <stdio.h>
-#include <string.h>
-
-#include "allocator.h"
-#include "hs_common.h"
-#include "hs_internal.h"
-#include "hs_version.h"
-#include "ue2common.h"
-#include "database.h"
-#include "crc32.h"
-#include "rose/rose_internal.h"
-#include "util/unaligned.h"
-
-static really_inline
-int db_correctly_aligned(const void *db) {
-    return ISALIGNED_N(db, alignof(unsigned long long));
-}
-
-HS_PUBLIC_API
-hs_error_t HS_CDECL hs_free_database(hs_database_t *db) {
-    if (db && db->magic != HS_DB_MAGIC) {
-        return HS_INVALID;
-    }
-    hs_database_free(db);
-
-    return HS_SUCCESS;
-}
-
-HS_PUBLIC_API
-hs_error_t HS_CDECL hs_serialize_database(const hs_database_t *db, char **bytes,
-                                          size_t *serialized_length) {
-    if (!db || !bytes || !serialized_length) {
-        return HS_INVALID;
-    }
-
-    if (!db_correctly_aligned(db)) {
-        return HS_BAD_ALIGN;
-    }
-
-    hs_error_t ret = validDatabase(db);
-    if (ret != HS_SUCCESS) {
-        return ret;
-    }
-
-    size_t length = sizeof(struct hs_database) + db->length;
-
-    char *out = hs_misc_alloc(length);
-    ret = hs_check_alloc(out);
-    if (ret != HS_SUCCESS) {
-        hs_misc_free(out);
-        return ret;
-    }
-
-    memset(out, 0, length);
-
-    u32 *buf = (u32 *)out;
-    *buf = db->magic;
-    buf++;
-    *buf = db->version;
-    buf++;
-    *buf = db->length;
-    buf++;
-    memcpy(buf, &db->platform, sizeof(u64a));
-    buf += 2;
-    *buf = db->crc32;
-    buf++;
-    *buf = db->reserved0;
-    buf++;
-    *buf = db->reserved1;
-    buf++;
-
-    const char *bytecode = hs_get_bytecode(db);
-    memcpy(buf, bytecode, db->length);
-
-    *bytes = out;
-    *serialized_length = length;
-    return HS_SUCCESS;
-}
-
-// check that the database header's platform is compatible with the current
-// runtime platform.
-static
-hs_error_t db_check_platform(const u64a p) {
-    if (p != hs_current_platform
-        && p != (hs_current_platform | hs_current_platform_no_avx2)
-        && p != (hs_current_platform | hs_current_platform_no_avx512)
-        && p != (hs_current_platform | hs_current_platform_no_avx512vbmi)) {
-        return HS_DB_PLATFORM_ERROR;
-    }
-    // passed all checks
-    return HS_SUCCESS;
-}
-
-// Decode and check the database header, returning appropriate errors or
-// HS_SUCCESS if it's OK. The header should be allocated on the stack
-// and later copied into the deserialized database.
-static
-hs_error_t db_decode_header(const char **bytes, const size_t length,
-                            struct hs_database *header) {
-    if (!*bytes) {
-        return HS_INVALID;
-    }
-
-    if (length < sizeof(struct hs_database)) {
-        return HS_INVALID;
-    }
-
-    // There's no requirement, really, that the serialized stream of bytes
-    // we've been given is 4-byte aligned, so we use unaligned loads here.
-
-    const u32 *buf = (const u32 *)*bytes;
-
-    // Zero header so that none of it (e.g. its padding) is uninitialized.
-    memset(header, 0, sizeof(struct hs_database));
-
-    header->magic = unaligned_load_u32(buf++);
-    if (header->magic != HS_DB_MAGIC) {
-        return HS_INVALID;
-    }
-
-    header->version = unaligned_load_u32(buf++);
-    if (header->version != HS_DB_VERSION) {
-        return HS_DB_VERSION_ERROR;
-    }
-
-    header->length = unaligned_load_u32(buf++);
-    if (length != sizeof(struct hs_database) + header->length) {
-        DEBUG_PRINTF("bad length %zu, expecting %zu\n", length,
-                     sizeof(struct hs_database) + header->length);
-        return HS_INVALID;
-    }
-
-    header->platform = unaligned_load_u64a(buf);
-    buf += 2;
-    header->crc32 = unaligned_load_u32(buf++);
-    header->reserved0 = unaligned_load_u32(buf++);
-    header->reserved1 = unaligned_load_u32(buf++);
-
-    *bytes = (const char *)buf;
-
-    return HS_SUCCESS; // Header checks out
-}
-
-// Check the CRC on a database
-static
-hs_error_t db_check_crc(const hs_database_t *db) {
-    const char *bytecode = hs_get_bytecode(db);
-    u32 crc = Crc32c_ComputeBuf(0, bytecode, db->length);
-    if (crc != db->crc32) {
-        DEBUG_PRINTF("crc mismatch! 0x%x != 0x%x\n", crc, db->crc32);
-        return HS_INVALID;
-    }
-    return HS_SUCCESS;
-}
-
-static
-void db_copy_bytecode(const char *serialized, hs_database_t *db) {
-    // we need to align things manually
-    uintptr_t shift = (uintptr_t)db->bytes & 0x3f;
-    db->bytecode = offsetof(struct hs_database, bytes) - shift;
-    char *bytecode = (char *)db + db->bytecode;
-
-    // Copy the bytecode into place
-    memcpy(bytecode, serialized, db->length);
-}
-
-HS_PUBLIC_API
-hs_error_t HS_CDECL hs_deserialize_database_at(const char *bytes,
-                                               const size_t length,
-                                               hs_database_t *db) {
-    if (!bytes || !db) {
-        return HS_INVALID;
-    }
-
-    // We require the user to deserialize into an 8-byte aligned region.
-    if (!ISALIGNED_N(db, 8)) {
-        return HS_BAD_ALIGN;
-    }
-
-    // Decode the header
-    hs_database_t header;
-    hs_error_t ret = db_decode_header(&bytes, length, &header);
-    if (ret != HS_SUCCESS) {
-        return ret;
-    }
-
-    // Make sure the serialized database is for our platform
-    ret = db_check_platform(header.platform);
-    if (ret != HS_SUCCESS) {
-        return ret;
-    }
-
-    // Zero new space for safety
-    size_t dblength = sizeof(struct hs_database) + header.length;
-    memset(db, 0, dblength);
-
-    // Copy the decoded header into place
-    memcpy(db, &header, sizeof(header));
-
-    // Copy the bytecode into the correctly-aligned location, set offsets
-    db_copy_bytecode(bytes, db);
-
-    if (db_check_crc(db) != HS_SUCCESS) {
-        return HS_INVALID;
-    }
-
-    return HS_SUCCESS;
-}
-
-HS_PUBLIC_API
-hs_error_t HS_CDECL hs_deserialize_database(const char *bytes,
-                                            const size_t length,
-                                            hs_database_t **db) {
-    if (!bytes || !db) {
-        return HS_INVALID;
-    }
-
-    *db = NULL;
-
-    // Decode and check the header
-    hs_database_t header;
-    hs_error_t ret = db_decode_header(&bytes, length, &header);
-    if (ret != HS_SUCCESS) {
-        return ret;
-    }
-
-    // Make sure the serialized database is for our platform
-    ret = db_check_platform(header.platform);
-    if (ret != HS_SUCCESS) {
-        return ret;
-    }
-
-    // Allocate space for new database
-    size_t dblength = sizeof(struct hs_database) + header.length;
-    struct hs_database *tempdb = hs_database_alloc(dblength);
-    ret = hs_check_alloc(tempdb);
-    if (ret != HS_SUCCESS) {
-        hs_database_free(tempdb);
-        return ret;
-    }
-
-    // Zero new space for safety
-    memset(tempdb, 0, dblength);
-
-    // Copy the decoded header into place
-    memcpy(tempdb, &header, sizeof(header));
-
-    // Copy the bytecode into the correctly-aligned location, set offsets
-    db_copy_bytecode(bytes, tempdb);
-
-    if (db_check_crc(tempdb) != HS_SUCCESS) {
-        hs_database_free(tempdb);
-        return HS_INVALID;
-    }
-
-    *db = tempdb;
-    return HS_SUCCESS;
-}
-
-HS_PUBLIC_API
-hs_error_t HS_CDECL hs_database_size(const hs_database_t *db, size_t *size) {
-    if (!size) {
-        return HS_INVALID;
-    }
-
-    hs_error_t ret = validDatabase(db);
-    if (unlikely(ret != HS_SUCCESS)) {
-        return ret;
-    }
-
-    *size = sizeof(struct hs_database) + db->length;
-    return HS_SUCCESS;
-}
-
-HS_PUBLIC_API
-hs_error_t HS_CDECL hs_serialized_database_size(const char *bytes,
-                                                const size_t length,
-                                                size_t *size) {
-    // Decode and check the header
-    hs_database_t header;
-    hs_error_t ret = db_decode_header(&bytes, length, &header);
-    if (ret != HS_SUCCESS) {
-        return ret;
-    }
-
-    if (!size) {
-        return HS_INVALID;
-    }
-
-    *size = sizeof(struct hs_database) + header.length;
-    return HS_SUCCESS;
-}
-
-hs_error_t dbIsValid(const hs_database_t *db) {
-    if (db->magic != HS_DB_MAGIC) {
-        DEBUG_PRINTF("bad magic\n");
-        return HS_INVALID;
-    }
-
-    if (db->version != HS_DB_VERSION) {
-        DEBUG_PRINTF("bad version\n");
-        return HS_DB_VERSION_ERROR;
-    }
-
-    if (db_check_platform(db->platform) != HS_SUCCESS) {
-        DEBUG_PRINTF("bad platform\n");
-        return HS_DB_PLATFORM_ERROR;
-    }
-
-    if (!ISALIGNED_16(hs_get_bytecode(db))) {
-        DEBUG_PRINTF("bad alignment\n");
-        return HS_INVALID;
-    }
-
-    hs_error_t rv = db_check_crc(db);
-    if (rv != HS_SUCCESS) {
-        DEBUG_PRINTF("bad crc\n");
-        return rv;
-    }
-
-    return HS_SUCCESS;
-}
-
-/** Allocate a buffer and prints the database info into it. Returns an
- * appropriate error code on failure, or HS_SUCCESS on success. */
-static
-hs_error_t print_database_string(char **s, u32 version, const platform_t plat,
-                                 u32 raw_mode) {
-    assert(s);
-    *s = NULL;
-
-    u8 release = (version >> 8) & 0xff;
-    u8 minor = (version >> 16) & 0xff;
-    u8 major = (version >> 24) & 0xff;
-
-    const char *features = (plat & HS_PLATFORM_NOAVX512VBMI)
-                               ? (plat & HS_PLATFORM_NOAVX512)
-                                   ? (plat & HS_PLATFORM_NOAVX2) ? "" : "AVX2"
-                                   : "AVX512"
-                               : "AVX512VBMI";
-
-    const char *mode = NULL;
-
-    if (raw_mode == HS_MODE_STREAM) {
-        mode = "STREAM";
-    } else if (raw_mode == HS_MODE_VECTORED) {
-        mode = "VECTORED";
-    } else {
-        assert(raw_mode == HS_MODE_BLOCK);
-        mode = "BLOCK";
-    }
-
-    // Initial allocation size, which should be large enough to print our info.
-    // If it isn't, snprintf will tell us and we can resize appropriately.
-    size_t len = 256;
-
-    while (1) {
-        char *buf = hs_misc_alloc(len);
-        hs_error_t ret = hs_check_alloc(buf);
-        if (ret != HS_SUCCESS) {
-            hs_misc_free(buf);
-            return ret;
-        }
-
-        int p_len = snprintf(
-            buf, len, "Version: %u.%u.%u Features: %s Mode: %s",
-            major, minor, release, features, mode);
-        if (p_len < 0) {
-            DEBUG_PRINTF("snprintf output error, returned %d\n", p_len);
-            hs_misc_free(buf);
-            break;
-        } else if ((size_t)p_len < len) { // output fit within buffer.
-            assert(buf[p_len] == '\0');
-            *s = buf;
-            return HS_SUCCESS;
-        } else { // output didn't fit: resize and reallocate.
-            len = (size_t)p_len + 1; // must add one for null terminator.
-            hs_misc_free(buf);
-        }
-    }
-
-    return HS_NOMEM;
-}
-
-HS_PUBLIC_API
-hs_error_t HS_CDECL hs_serialized_database_info(const char *bytes,
-                                                size_t length, char **info) {
-    if (!info) {
-        return HS_INVALID;
-    }
-    *info = NULL;
-
-    // Decode and check the header
-    hs_database_t header;
-    hs_error_t ret = db_decode_header(&bytes, length, &header);
-    if (ret != HS_SUCCESS) {
-        return ret;
-    }
-
-    u32 mode = unaligned_load_u32(bytes + offsetof(struct RoseEngine, mode));
-
-    return print_database_string(info, header.version, header.platform, mode);
-}
-
-HS_PUBLIC_API
-hs_error_t HS_CDECL hs_database_info(const hs_database_t *db, char **info) {
-    if (!info) {
-        return HS_INVALID;
-    }
-    *info = NULL;
-
-    if (!db || !db_correctly_aligned(db) || db->magic != HS_DB_MAGIC) {
-        return HS_INVALID;
-    }
-
-    platform_t plat;
-    plat = db->platform;
-
-    const struct RoseEngine *rose = hs_get_bytecode(db);
-
-    return print_database_string(info, db->version, plat, rose->mode);
-}

src/database.h

@@ -1,143 +0,0 @@
-/*
- * Copyright (c) 2015-2020, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Runtime code for hs_database manipulation.
- */
-
-#ifndef DATABASE_H_D467FD6F343DDE
-#define DATABASE_H_D467FD6F343DDE
-
-#ifdef __cplusplus
-extern "C"
-{
-#endif
-
-#include "hs_compile.h" // for HS_MODE_ flags
-#include "hs_version.h"
-#include "ue2common.h"
-#include "util/arch.h"
-
-#define HS_DB_VERSION HS_VERSION_32BIT
-#define HS_DB_MAGIC   (0xdbdbdbdbU)
-
-// Values in here cannot (easily) change - add new ones!
-
-// CPU type is the low 6 bits (we can't need more than 64, surely!)
-
-#define HS_PLATFORM_INTEL           1
-#define HS_PLATFORM_ARM             2
-#define HS_PLATFORM_CPU_MASK        0x3F
-
-#define HS_PLATFORM_NOAVX2          (4<<13)
-#define HS_PLATFORM_NOAVX512        (8<<13)
-#define HS_PLATFORM_NOAVX512VBMI    (0x10<<13)
-
-/** \brief Platform features bitmask. */
-typedef u64a platform_t;
-
-static UNUSED
-const platform_t hs_current_platform = {
-#if !defined(HAVE_AVX2)
-    HS_PLATFORM_NOAVX2 |
-#endif
-#if !defined(HAVE_AVX512)
-    HS_PLATFORM_NOAVX512 |
-#endif
-#if !defined(HAVE_AVX512VBMI)
-    HS_PLATFORM_NOAVX512VBMI |
-#endif
-    0,
-};
-
-static UNUSED
-const platform_t hs_current_platform_no_avx2 = {
-    HS_PLATFORM_NOAVX2 |
-    HS_PLATFORM_NOAVX512 |
-    HS_PLATFORM_NOAVX512VBMI |
-    0,
-};
-
-static UNUSED
-const platform_t hs_current_platform_no_avx512 = {
-    HS_PLATFORM_NOAVX512 |
-    HS_PLATFORM_NOAVX512VBMI |
-    0,
-};
-
-static UNUSED
-const platform_t hs_current_platform_no_avx512vbmi = {
-    HS_PLATFORM_NOAVX512VBMI |
-    0,
-};
-
-/*
- * a header to enclose the actual bytecode - useful for keeping info about the
- * compiled data.
- */
-struct hs_database {
-    u32 magic;
-    u32 version;
-    u32 length;
-    u64a platform;
-    u32 crc32;
-    u32 reserved0;
-    u32 reserved1;
-    u32 bytecode;    // offset relative to db start
-    u32 padding[16];
-    char bytes[];
-};
-
-static really_inline
-const void *hs_get_bytecode(const struct hs_database *db) {
-    return ((const char *)db + db->bytecode);
-}
-
-/**
- * Cheap database sanity checks used in block mode scan calls and streaming
- * mode open calls.
- */
-static really_inline
-hs_error_t validDatabase(const hs_database_t *db) {
-    if (!db || db->magic != HS_DB_MAGIC) {
-        return HS_INVALID;
-    }
-    if (db->version != HS_DB_VERSION) {
-        return HS_DB_VERSION_ERROR;
-    }
-
-    return HS_SUCCESS;
-}
-
-hs_error_t dbIsValid(const struct hs_database *db);
-
-#ifdef __cplusplus
-} /* extern "C" */
-#endif
-
-#endif /* DATABASE_H_D467FD6F343DDE */

src/dispatcher.c

@@ -1,153 +0,0 @@
-/*
- * Copyright (c) 2016-2020, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include "config.h"
-#include "hs_common.h"
-#include "hs_runtime.h"
-#include "ue2common.h"
-#if defined(ARCH_IA32) || defined(ARCH_X86_64)
-#include "util/arch/x86/cpuid_inline.h"
-#endif
-#include "util/join.h"
-
-#if defined(DISABLE_AVX512_DISPATCH)
-#define avx512_ disabled_
-#define check_avx512() (0)
-#endif
-
-#if defined(DISABLE_AVX512VBMI_DISPATCH)
-#define avx512vbmi_ disabled_
-#define check_avx512vbmi() (0)
-#endif
-
-#define CREATE_DISPATCH(RTYPE, NAME, ...)                                      \
-    /* create defns */                                                         \
-    RTYPE JOIN(avx512vbmi_, NAME)(__VA_ARGS__);                                \
-    RTYPE JOIN(avx512_, NAME)(__VA_ARGS__);                                    \
-    RTYPE JOIN(avx2_, NAME)(__VA_ARGS__);                                      \
-    RTYPE JOIN(corei7_, NAME)(__VA_ARGS__);                                    \
-    RTYPE JOIN(core2_, NAME)(__VA_ARGS__);                                     \
-                                                                               \
-    /* error func */                                                           \
-    static inline RTYPE JOIN(error_, NAME)(__VA_ARGS__) {                      \
-        return (RTYPE)HS_ARCH_ERROR;                                           \
-    }                                                                          \
-                                                                               \
-    /* resolver */                                                             \
-    static RTYPE (*JOIN(resolve_, NAME)(void))(__VA_ARGS__) {                  \
-        if (check_avx512vbmi()) {                                              \
-            return JOIN(avx512vbmi_, NAME);                                    \
-        }                                                                      \
-        if (check_avx512()) {                                                  \
-            return JOIN(avx512_, NAME);                                        \
-        }                                                                      \
-        if (check_avx2()) {                                                    \
-            return JOIN(avx2_, NAME);                                          \
-        }                                                                      \
-        if (check_sse42() && check_popcnt()) {                                 \
-            return JOIN(corei7_, NAME);                                        \
-        }                                                                      \
-        if (check_ssse3()) {                                                   \
-            return JOIN(core2_, NAME);                                         \
-        }                                                                      \
-        /* anything else is fail */                                            \
-        return JOIN(error_, NAME);                                             \
-    }                                                                          \
-                                                                               \
-    /* function */                                                             \
-    HS_PUBLIC_API                                                              \
-    RTYPE NAME(__VA_ARGS__) __attribute__((ifunc("resolve_" #NAME)))
-
-CREATE_DISPATCH(hs_error_t, hs_scan, const hs_database_t *db, const char *data,
-                unsigned length, unsigned flags, hs_scratch_t *scratch,
-                match_event_handler onEvent, void *userCtx);
-
-CREATE_DISPATCH(hs_error_t, hs_stream_size, const hs_database_t *database,
-                size_t *stream_size);
-
-CREATE_DISPATCH(hs_error_t, hs_database_size, const hs_database_t *db,
-                size_t *size);
-CREATE_DISPATCH(hs_error_t, dbIsValid, const hs_database_t *db);
-CREATE_DISPATCH(hs_error_t, hs_free_database, hs_database_t *db);
-
-CREATE_DISPATCH(hs_error_t, hs_open_stream, const hs_database_t *db,
-                unsigned int flags, hs_stream_t **stream);
-
-CREATE_DISPATCH(hs_error_t, hs_scan_stream, hs_stream_t *id, const char *data,
-                unsigned int length, unsigned int flags, hs_scratch_t *scratch,
-                match_event_handler onEvent, void *ctxt);
-
-CREATE_DISPATCH(hs_error_t, hs_close_stream, hs_stream_t *id,
-                hs_scratch_t *scratch, match_event_handler onEvent, void *ctxt);
-
-CREATE_DISPATCH(hs_error_t, hs_scan_vector, const hs_database_t *db,
-                const char *const *data, const unsigned int *length,
-                unsigned int count, unsigned int flags, hs_scratch_t *scratch,
-                match_event_handler onevent, void *context);
-
-CREATE_DISPATCH(hs_error_t, hs_database_info, const hs_database_t *db, char **info);
-
-CREATE_DISPATCH(hs_error_t, hs_copy_stream, hs_stream_t **to_id,
-                const hs_stream_t *from_id);
-
-CREATE_DISPATCH(hs_error_t, hs_reset_stream, hs_stream_t *id,
-                unsigned int flags, hs_scratch_t *scratch,
-                match_event_handler onEvent, void *context);
-
-CREATE_DISPATCH(hs_error_t, hs_reset_and_copy_stream, hs_stream_t *to_id,
-                const hs_stream_t *from_id, hs_scratch_t *scratch,
-                match_event_handler onEvent, void *context);
-
-CREATE_DISPATCH(hs_error_t, hs_serialize_database, const hs_database_t *db,
-                char **bytes, size_t *length);
-
-CREATE_DISPATCH(hs_error_t, hs_deserialize_database, const char *bytes,
-                const size_t length, hs_database_t **db);
-
-CREATE_DISPATCH(hs_error_t, hs_deserialize_database_at, const char *bytes,
-                const size_t length, hs_database_t *db);
-
-CREATE_DISPATCH(hs_error_t, hs_serialized_database_info, const char *bytes,
-                size_t length, char **info);
-
-CREATE_DISPATCH(hs_error_t, hs_serialized_database_size, const char *bytes,
-                const size_t length, size_t *deserialized_size);
-
-CREATE_DISPATCH(hs_error_t, hs_compress_stream, const hs_stream_t *stream,
-                char *buf, size_t buf_space, size_t *used_space);
-
-CREATE_DISPATCH(hs_error_t, hs_expand_stream, const hs_database_t *db,
-                hs_stream_t **stream, const char *buf,size_t buf_size);
-
-CREATE_DISPATCH(hs_error_t, hs_reset_and_expand_stream, hs_stream_t *to_stream,
-                const char *buf, size_t buf_size, hs_scratch_t *scratch,
-                match_event_handler onEvent, void *context);
-
-/** INTERNALS **/
-
-CREATE_DISPATCH(u32, Crc32c_ComputeBuf, u32 inCrc32, const void *buf, size_t bufLen);

src/fdr/engine_description.cpp

@@ -1,49 +0,0 @@
-/*
- * Copyright (c) 2015, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include "engine_description.h"
-#include "hs_compile.h" // for hs_platform_info
-#include "util/target_info.h"
-
-namespace ue2 {
-
-EngineDescription::~EngineDescription() {}
-
-bool EngineDescription::isValidOnTarget(const target_t &target_in) const {
-    return target_in.can_run_on_code_built_for(code_target);
-}
-
-target_t targetByArchFeatures(u64a cpu_features) {
-    hs_platform_info p;
-    p.tune = HS_TUNE_FAMILY_GENERIC;
-    p.cpu_features = cpu_features;
-
-    return target_t(p);
-}
-
-} // namespace ue2

src/fdr/engine_description.h

@@ -1,61 +0,0 @@
-/*
- * Copyright (c) 2015-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef ENGINE_DESCRIPTION_H
-#define ENGINE_DESCRIPTION_H
-
-#include "ue2common.h"
-#include "util/target_info.h"
-
-namespace ue2 {
-
-class EngineDescription {
-    u32 id;
-    target_t code_target; // the target that we built this code for
-    u32 numBuckets;
-
-public:
-    EngineDescription(u32 id_in, const target_t &code_target_in,
-                      u32 numBuckets_in)
-        : id(id_in), code_target(code_target_in), numBuckets(numBuckets_in) {}
-
-    virtual ~EngineDescription();
-
-    u32 getID() const { return id; }
-    u32 getNumBuckets() const { return numBuckets; }
-
-    bool isValidOnTarget(const target_t &target_in) const;
-    virtual u32 getDefaultFloodSuffixLength() const = 0;
-};
-
-/** Returns a target given a CPU feature set value. */
-target_t targetByArchFeatures(u64a cpu_features);
-
-} // namespace ue2
-
-#endif

src/fdr/fdr.c

@@ -1,855 +0,0 @@
-/*
- * Copyright (c) 2015-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include "fdr.h"
-#include "fdr_confirm.h"
-#include "fdr_confirm_runtime.h"
-#include "fdr_internal.h"
-#include "fdr_loadval.h"
-#include "flood_runtime.h"
-#include "scratch.h"
-#include "teddy.h"
-#include "teddy_internal.h"
-#include "util/arch.h"
-#include "util/bitutils.h"
-#include "util/simd_utils.h"
-#include "util/uniform_ops.h"
-
-/** \brief number of bytes processed in each iteration */
-#define ITER_BYTES          16
-
-/** \brief total zone buffer size */
-#define ZONE_TOTAL_SIZE     64
-
-/** \brief maximum number of allowed zones */
-#define ZONE_MAX            3
-
-/** \brief zone information.
- *
- * Zone represents a region of data to scan in FDR.
- *
- * The incoming buffer is to split in multiple zones to ensure two properties:
- * 1: that we can read 8? bytes behind to generate a hash safely
- * 2: that we can read the 3 byte after the current byte (domain > 8)
- */
-struct zone {
-    /** \brief copied buffer, used only when it is a boundary zone. */
-    u8 ALIGN_CL_DIRECTIVE buf[ZONE_TOTAL_SIZE];
-
-    /** \brief shift amount for fdr state to avoid unwanted match. */
-    u8 shift;
-
-    /** \brief if boundary zone, start points into the zone buffer after the
-     * pre-padding. Otherwise, points to the main buffer, appropriately. */
-    const u8 *start;
-
-    /** \brief if boundary zone, end points to the end of zone. Otherwise,
-     * pointer to the main buffer, appropriately. */
-    const u8 *end;
-
-    /** \brief the amount to adjust to go from a pointer in the zones region
-     * (between start and end) to a pointer in the original data buffer. */
-    ptrdiff_t zone_pointer_adjust;
-
-    /** \brief firstFloodDetect from FDR_Runtime_Args for non-boundary zones,
-     * otherwise end of the zone buf. floodPtr always points inside the same
-     * buffer as the start pointe. */
-    const u8 *floodPtr;
-};
-
-static
-const ALIGN_CL_DIRECTIVE u8 zone_or_mask[ITER_BYTES+1][ITER_BYTES] = {
-    { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-      0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
-    { 0xff, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-      0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
-    { 0xff, 0xff, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-      0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
-    { 0xff, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00, 0x00,
-      0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
-    { 0xff, 0xff, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00,
-      0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
-    { 0xff, 0xff, 0xff, 0xff, 0xff, 0x00, 0x00, 0x00,
-      0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
-    { 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x00, 0x00,
-      0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
-    { 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x00,
-      0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
-    { 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
-      0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
-    { 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
-      0xff, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
-    { 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
-      0xff, 0xff, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
-    { 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
-      0xff, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00, 0x00 },
-    { 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
-      0xff, 0xff, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00 },
-    { 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
-      0xff, 0xff, 0xff, 0xff, 0xff, 0x00, 0x00, 0x00 },
-    { 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
-      0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x00, 0x00 },
-    { 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
-      0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x00 },
-    { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-      0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }
-};
-
-/* generates an initial state mask based on the last byte-ish of history rather
- * than being all accepting. If there is no history to consider, the state is
- * generated based on the minimum length of each bucket in order to prevent
- * confirms.
- */
-static really_inline
-m128 getInitState(const struct FDR *fdr, u8 len_history, const u64a *ft,
-                  const struct zone *z) {
-    m128 s;
-    if (len_history) {
-        /* +1: the zones ensure that we can read the byte at z->end */
-        u32 tmp = lv_u16(z->start + z->shift - 1, z->buf, z->end + 1);
-        tmp &= fdr->domainMask;
-        s = load_m128_from_u64a(ft + tmp);
-        s = rshiftbyte_m128(s, 1);
-    } else {
-        s = fdr->start;
-    }
-    return s;
-}
-
-static really_inline
-void get_conf_stride_1(const u8 *itPtr, UNUSED const u8 *start_ptr,
-                       UNUSED const u8 *end_ptr, u32 domain_mask_flipped,
-                       const u64a *ft, u64a *conf0, u64a *conf8, m128 *s) {
-    /* +1: the zones ensure that we can read the byte at z->end */
-    assert(itPtr >= start_ptr && itPtr + ITER_BYTES <= end_ptr);
-    u64a domain_mask = ~domain_mask_flipped;
-
-    u64a it_hi = *(const u64a *)itPtr;
-    u64a it_lo = *(const u64a *)(itPtr + 8);
-    u64a reach0  = domain_mask & it_hi;
-    u64a reach1  = domain_mask & (it_hi >> 8);
-    u64a reach2  = domain_mask & (it_hi >> 16);
-    u64a reach3  = domain_mask & (it_hi >> 24);
-    u64a reach4  = domain_mask & (it_hi >> 32);
-    u64a reach5  = domain_mask & (it_hi >> 40);
-    u64a reach6  = domain_mask & (it_hi >> 48);
-    u64a reach7  = domain_mask & ((it_hi >> 56) | (it_lo << 8));
-    u64a reach8  = domain_mask & it_lo;
-    u64a reach9  = domain_mask & (it_lo >> 8);
-    u64a reach10 = domain_mask & (it_lo >> 16);
-    u64a reach11 = domain_mask & (it_lo >> 24);
-    u64a reach12 = domain_mask & (it_lo >> 32);
-    u64a reach13 = domain_mask & (it_lo >> 40);
-    u64a reach14 = domain_mask & (it_lo >> 48);
-    u64a reach15 = domain_mask & unaligned_load_u32(itPtr + 15);
-
-    m128 st0  = load_m128_from_u64a(ft + reach0);
-    m128 st1  = lshiftbyte_m128(load_m128_from_u64a(ft + reach1), 1);
-    m128 st2  = lshiftbyte_m128(load_m128_from_u64a(ft + reach2), 2);
-    m128 st3  = lshiftbyte_m128(load_m128_from_u64a(ft + reach3), 3);
-    m128 st4  = lshiftbyte_m128(load_m128_from_u64a(ft + reach4), 4);
-    m128 st5  = lshiftbyte_m128(load_m128_from_u64a(ft + reach5), 5);
-    m128 st6  = lshiftbyte_m128(load_m128_from_u64a(ft + reach6), 6);
-    m128 st7  = lshiftbyte_m128(load_m128_from_u64a(ft + reach7), 7);
-    m128 st8  = load_m128_from_u64a(ft + reach8);
-    m128 st9  = lshiftbyte_m128(load_m128_from_u64a(ft + reach9), 1);
-    m128 st10 = lshiftbyte_m128(load_m128_from_u64a(ft + reach10), 2);
-    m128 st11 = lshiftbyte_m128(load_m128_from_u64a(ft + reach11), 3);
-    m128 st12 = lshiftbyte_m128(load_m128_from_u64a(ft + reach12), 4);
-    m128 st13 = lshiftbyte_m128(load_m128_from_u64a(ft + reach13), 5);
-    m128 st14 = lshiftbyte_m128(load_m128_from_u64a(ft + reach14), 6);
-    m128 st15 = lshiftbyte_m128(load_m128_from_u64a(ft + reach15), 7);
-
-    st0 = or128(st0, st1);
-    st2 = or128(st2, st3);
-    st4 = or128(st4, st5);
-    st6 = or128(st6, st7);
-    st0 = or128(st0, st2);
-    st4 = or128(st4, st6);
-    st0 = or128(st0, st4);
-
-    st8 = or128(st8, st9);
-    st10 = or128(st10, st11);
-    st12 = or128(st12, st13);
-    st14 = or128(st14, st15);
-    st8 = or128(st8, st10);
-    st12 = or128(st12, st14);
-    st8 = or128(st8, st12);
-
-    m128 st = or128(*s, st0);
-    *conf0 = movq(st) ^ ~0ULL;
-    st = rshiftbyte_m128(st, 8);
-    st = or128(st, st8);
-
-    *conf8 = movq(st) ^ ~0ULL;
-    *s = rshiftbyte_m128(st, 8);
-}
-
-static really_inline
-void get_conf_stride_2(const u8 *itPtr, UNUSED const u8 *start_ptr,
-                       UNUSED const u8 *end_ptr, u32 domain_mask_flipped,
-                       const u64a *ft, u64a *conf0, u64a *conf8, m128 *s) {
-    assert(itPtr >= start_ptr && itPtr + ITER_BYTES <= end_ptr);
-
-    u64a reach0 = andn(domain_mask_flipped, itPtr);
-    u64a reach2 = andn(domain_mask_flipped, itPtr + 2);
-    u64a reach4 = andn(domain_mask_flipped, itPtr + 4);
-    u64a reach6 = andn(domain_mask_flipped, itPtr + 6);
-
-    m128 st0 = load_m128_from_u64a(ft + reach0);
-    m128 st2 = load_m128_from_u64a(ft + reach2);
-    m128 st4 = load_m128_from_u64a(ft + reach4);
-    m128 st6 = load_m128_from_u64a(ft + reach6);
-
-    u64a reach8 = andn(domain_mask_flipped, itPtr + 8);
-    u64a reach10 = andn(domain_mask_flipped, itPtr + 10);
-    u64a reach12 = andn(domain_mask_flipped, itPtr + 12);
-    u64a reach14 = andn(domain_mask_flipped, itPtr + 14);
-
-    m128 st8 = load_m128_from_u64a(ft + reach8);
-    m128 st10 = load_m128_from_u64a(ft + reach10);
-    m128 st12 = load_m128_from_u64a(ft + reach12);
-    m128 st14 = load_m128_from_u64a(ft + reach14);
-
-    st2  = lshiftbyte_m128(st2, 2);
-    st4  = lshiftbyte_m128(st4, 4);
-    st6  = lshiftbyte_m128(st6, 6);
-
-    *s = or128(*s, st0);
-    *s = or128(*s, st2);
-    *s = or128(*s, st4);
-    *s = or128(*s, st6);
-
-    *conf0 = movq(*s);
-    *s = rshiftbyte_m128(*s, 8);
-    *conf0 ^= ~0ULL;
-
-    st10 = lshiftbyte_m128(st10, 2);
-    st12 = lshiftbyte_m128(st12, 4);
-    st14 = lshiftbyte_m128(st14, 6);
-
-    *s = or128(*s, st8);
-    *s = or128(*s, st10);
-    *s = or128(*s, st12);
-    *s = or128(*s, st14);
-
-    *conf8 = movq(*s);
-    *s = rshiftbyte_m128(*s, 8);
-    *conf8 ^= ~0ULL;
-}
-
-static really_inline
-void get_conf_stride_4(const u8 *itPtr, UNUSED const u8 *start_ptr,
-                       UNUSED const u8 *end_ptr, u32 domain_mask_flipped,
-                       const u64a *ft, u64a *conf0, u64a *conf8, m128 *s) {
-    assert(itPtr >= start_ptr && itPtr + ITER_BYTES <= end_ptr);
-
-    u64a reach0 = andn(domain_mask_flipped, itPtr);
-    u64a reach4 = andn(domain_mask_flipped, itPtr + 4);
-    u64a reach8 = andn(domain_mask_flipped, itPtr + 8);
-    u64a reach12 = andn(domain_mask_flipped, itPtr + 12);
-
-    m128 st0 = load_m128_from_u64a(ft + reach0);
-    m128 st4 = load_m128_from_u64a(ft + reach4);
-    m128 st8 = load_m128_from_u64a(ft + reach8);
-    m128 st12 = load_m128_from_u64a(ft + reach12);
-
-    st4 = lshiftbyte_m128(st4, 4);
-    st12 = lshiftbyte_m128(st12, 4);
-
-    *s = or128(*s, st0);
-    *s = or128(*s, st4);
-    *conf0 = movq(*s);
-    *s = rshiftbyte_m128(*s, 8);
-    *conf0 ^= ~0ULL;
-
-    *s = or128(*s, st8);
-    *s = or128(*s, st12);
-    *conf8 = movq(*s);
-    *s = rshiftbyte_m128(*s, 8);
-    *conf8 ^= ~0ULL;
-}
-
-static really_inline
-void do_confirm_fdr(u64a *conf, u8 offset, hwlmcb_rv_t *control,
-                    const u32 *confBase, const struct FDR_Runtime_Args *a,
-                    const u8 *ptr, u32 *last_match_id, struct zone *z) {
-    const u8 bucket = 8;
-
-    if (likely(!*conf)) {
-        return;
-    }
-
-    /* ptr is currently referring to a location in the zone's buffer, we also
-     * need a pointer in the original, main buffer for the final string compare.
-     */
-    const u8 *ptr_main = (const u8 *)((uintptr_t)ptr + z->zone_pointer_adjust);
-
-    const u8 *confLoc = ptr;
-
-    do  {
-        u32 bit = findAndClearLSB_64(conf);
-        u32 byte = bit / bucket + offset;
-        u32 bitRem = bit % bucket;
-        u32 idx = bitRem;
-        u32 cf = confBase[idx];
-        if (!cf) {
-            continue;
-        }
-        const struct FDRConfirm *fdrc = (const struct FDRConfirm *)
-                                        ((const u8 *)confBase + cf);
-        if (!(fdrc->groups & *control)) {
-            continue;
-        }
-        u64a confVal = unaligned_load_u64a(confLoc + byte - sizeof(u64a) + 1);
-        confWithBit(fdrc, a, ptr_main - a->buf + byte, control,
-                    last_match_id, confVal, conf, bit);
-    } while (unlikely(!!*conf));
-}
-
-static really_inline
-void dumpZoneInfo(UNUSED struct zone *z, UNUSED size_t zone_id) {
-#ifdef DEBUG
-    DEBUG_PRINTF("zone: zone=%zu, bufPtr=%p\n", zone_id, z->buf);
-    DEBUG_PRINTF("zone: startPtr=%p, endPtr=%p, shift=%u\n",
-                 z->start, z->end, z->shift);
-    DEBUG_PRINTF("zone: zone_pointer_adjust=%zd, floodPtr=%p\n",
-                 z->zone_pointer_adjust, z->floodPtr);
-    DEBUG_PRINTF("zone buf:");
-    for (size_t i = 0; i < ZONE_TOTAL_SIZE; i++) {
-        if (i % 8 == 0) {
-            printf("_");
-        }
-        if (z->buf[i]) {
-            printf("%02x", z->buf[i]);
-        } else {
-            printf("..");
-        }
-    }
-    printf("\n");
-#endif
-};
-
-/**
- * \brief Updates attributes for non-boundary region zone.
- */
-static really_inline
-void createMainZone(const u8 *flood, const u8 *begin, const u8 *end,
-                    struct zone *z) {
-    z->zone_pointer_adjust = 0; /* zone buffer is the main buffer */
-    z->start = begin;
-    z->end = end;
-    z->floodPtr = flood;
-    z->shift = 0;
-}
-
-/**
- * \brief Create zone for short cases (<= ITER_BYTES).
- *
- * For this case we need to copy everything into the zone's internal buffer.
- *
- * We need to ensure that we run over real data if it exists (in history or
- * before zone begin). We also need to ensure 8 bytes before any data being
- * matched can be read (to perform a conf hash).
- *
- * We also need to ensure that the data at z->end can be read.
- *
- * Hence, the zone consists of:
- *     16 bytes of history,
- *     1 - 24 bytes of data form the buffer (ending at end),
- *     1 byte of final padding
- */
-static really_inline
-void createShortZone(const u8 *buf, const u8 *hend, const u8 *begin,
-                     const u8 *end, struct zone *z) {
-    /* the floodPtr for BOUNDARY zones are maximum of end of zone buf to avoid
-     * the checks in boundary zone. */
-    z->floodPtr = z->buf + ZONE_TOTAL_SIZE;
-
-    ptrdiff_t z_len = end - begin;
-    assert(z_len > 0);
-    assert(z_len <= ITER_BYTES);
-
-    z->shift = ITER_BYTES - z_len; /* ignore bytes outside region specified */
-
-    static const size_t ZONE_SHORT_DATA_OFFSET = 16; /* after history */
-
-    /* we are guaranteed to always have 16 initialised bytes at the end of
-     * the history buffer (they may be garbage coming from the stream state
-     * preceding hbuf, but bytes that don't correspond to actual history
-     * shouldn't affect computations). */
-    *(m128 *)z->buf = loadu128(hend - sizeof(m128));
-
-    /* The amount of data we have to copy from main buffer. */
-    size_t copy_len = MIN((size_t)(end - buf),
-                          ITER_BYTES + sizeof(CONF_TYPE));
-
-    u8 *zone_data = z->buf + ZONE_SHORT_DATA_OFFSET;
-    switch (copy_len) {
-    case 1:
-        *zone_data = *(end - 1);
-        break;
-    case 2:
-        *(u16 *)zone_data = unaligned_load_u16(end - 2);
-        break;
-    case 3:
-        *(u16 *)zone_data = unaligned_load_u16(end - 3);
-        *(zone_data + 2) = *(end - 1);
-        break;
-    case 4:
-        *(u32 *)zone_data = unaligned_load_u32(end - 4);
-        break;
-    case 5:
-    case 6:
-    case 7:
-        /* perform copy with 2 overlapping 4-byte chunks from buf. */
-        *(u32 *)zone_data = unaligned_load_u32(end - copy_len);
-        unaligned_store_u32(zone_data + copy_len - sizeof(u32),
-                            unaligned_load_u32(end - sizeof(u32)));
-        break;
-    case 8:
-        *(u64a *)zone_data = unaligned_load_u64a(end - 8);
-        break;
-    case 9:
-    case 10:
-    case 11:
-    case 12:
-    case 13:
-    case 14:
-    case 15:
-        /* perform copy with 2 overlapping 8-byte chunks from buf. */
-        *(u64a *)zone_data = unaligned_load_u64a(end - copy_len);
-        unaligned_store_u64a(zone_data + copy_len - sizeof(u64a),
-                             unaligned_load_u64a(end - sizeof(u64a)));
-        break;
-    case 16:
-        /* copy 16-bytes from buf. */
-        *(m128 *)zone_data = loadu128(end - 16);
-        break;
-    default:
-        assert(copy_len <= sizeof(m128) + sizeof(u64a));
-
-        /* perform copy with (potentially overlapping) 8-byte and 16-byte chunks.
-         */
-        *(u64a *)zone_data = unaligned_load_u64a(end - copy_len);
-        storeu128(zone_data + copy_len - sizeof(m128),
-                  loadu128(end - sizeof(m128)));
-        break;
-    }
-
-    /* set the start and end location of the zone buf
-     * to be scanned */
-    u8 *z_end = z->buf + ZONE_SHORT_DATA_OFFSET + copy_len;
-    assert(ZONE_SHORT_DATA_OFFSET + copy_len >= ITER_BYTES);
-
-    /* copy the post-padding byte; this is required for domain > 8 due to
-     * overhang */
-    assert(ZONE_SHORT_DATA_OFFSET + copy_len + 3 < 64);
-    *z_end = 0;
-
-    z->end = z_end;
-    z->start = z_end - ITER_BYTES;
-    z->zone_pointer_adjust = (ptrdiff_t)((uintptr_t)end - (uintptr_t)z_end);
-    assert(z->start + z->shift == z_end - z_len);
-}
-
-/**
- * \brief Create a zone for the start region.
- *
- * This function requires that there is > ITER_BYTES of data in the buffer to
- * scan. The start zone itself is always responsible for scanning exactly
- * ITER_BYTES of data - there are no warmup/junk bytes scanned.
- *
- * This zone ensures that the byte at z->end can be read and corresponds to
- * the next byte of data.
- *
- * 8 bytes of history data are provided before z->start to allow proper hash
- * generation in streaming mode. If buf != begin, upto 8 bytes of data
- * prior to begin is also provided.
- *
- * Although we are not interested in bare literals which start before begin
- * if buf != begin, lookarounds associated with the literal may require
- * the data prior to begin for hash purposes.
- */
-static really_inline
-void createStartZone(const u8 *buf, const u8 *hend, const u8 *begin,
-                     struct zone *z) {
-    assert(ITER_BYTES == sizeof(m128));
-    assert(sizeof(CONF_TYPE) == 8);
-    static const size_t ZONE_START_BEGIN = sizeof(CONF_TYPE);
-
-    const u8 *end = begin + ITER_BYTES;
-
-    /* set floodPtr to the end of zone buf to avoid checks in start zone */
-    z->floodPtr = z->buf + ZONE_TOTAL_SIZE;
-
-    z->shift = 0; /* we are processing ITER_BYTES of real data */
-
-    /* we are guaranteed to always have 16 initialised bytes at the end of the
-     * history buffer (they may be garbage coming from the stream state
-     * preceding hbuf, but bytes that don't correspond to actual history
-     * shouldn't affect computations). However, for start zones, history is only
-     * required for conf hash purposes so we only need 8 bytes */
-    unaligned_store_u64a(z->buf, unaligned_load_u64a(hend - sizeof(u64a)));
-
-    /* The amount of data we have to copy from main buffer. */
-    size_t copy_len = MIN((size_t)(end - buf),
-                          ITER_BYTES + sizeof(CONF_TYPE));
-    assert(copy_len >= 16);
-
-    /* copy the post-padding byte; this is required for domain > 8 due to
-     * overhang. The start requires that there is data after the zone so it
-     * it safe to dereference end */
-    z->buf[ZONE_START_BEGIN + copy_len] = *end;
-
-    /* set the start and end location of the zone buf to be scanned */
-    u8 *z_end = z->buf + ZONE_START_BEGIN + copy_len;
-    z->end = z_end;
-    z->start = z_end - ITER_BYTES;
-
-    /* copy the first 8 bytes of the valid region */
-    unaligned_store_u64a(z->buf + ZONE_START_BEGIN,
-                         unaligned_load_u64a(end - copy_len));
-
-    /* copy the last 16 bytes, may overlap with the previous 8 byte write */
-    storeu128(z_end - sizeof(m128), loadu128(end - sizeof(m128)));
-
-    z->zone_pointer_adjust = (ptrdiff_t)((uintptr_t)end - (uintptr_t)z_end);
-
-    assert(ZONE_START_BEGIN + copy_len + 3 < 64);
-}
-
-/**
- * \brief Create a zone for the end region.
- *
- * This function requires that there is > ITER_BYTES of data in the buffer to
- * scan. The end zone is responsible for a scanning the <= ITER_BYTES rump of
- * data and optional ITER_BYTES. The main zone cannot handle the last 3 bytes
- * of the buffer. The end zone is required to handle an optional full
- * ITER_BYTES from main zone when there are less than 3 bytes to scan. The
- * main zone size is reduced by ITER_BYTES in this case.
- *
- * This zone ensures that the byte at z->end can be read by filling it with a
- * padding character.
- *
- * Upto 8 bytes of data prior to begin is also provided for the purposes of
- * generating hashes. History is not copied, as all locations which require
- * history for generating a hash are the responsiblity of the start zone.
- */
-static really_inline
-void createEndZone(const u8 *buf, const u8 *begin, const u8 *end,
-                   struct zone *z) {
-    /* the floodPtr for BOUNDARY zones are maximum of end of zone buf to avoid
-     * the checks in boundary zone. */
-    z->floodPtr = z->buf + ZONE_TOTAL_SIZE;
-
-    ptrdiff_t z_len = end - begin;
-    assert(z_len > 0);
-    size_t iter_bytes_second = 0;
-    size_t z_len_first = z_len;
-    if (z_len > ITER_BYTES) {
-        z_len_first = z_len - ITER_BYTES;
-        iter_bytes_second = ITER_BYTES;
-    }
-    z->shift = ITER_BYTES - z_len_first;
-
-    const u8 *end_first = end - iter_bytes_second;
-    /* The amount of data we have to copy from main buffer for the
-     * first iteration. */
-    size_t copy_len_first = MIN((size_t)(end_first - buf),
-                                ITER_BYTES + sizeof(CONF_TYPE));
-    assert(copy_len_first >= 16);
-
-    size_t total_copy_len = copy_len_first + iter_bytes_second;
-    assert(total_copy_len + 3 < 64);
-
-    /* copy the post-padding byte; this is required for domain > 8 due to
-     * overhang */
-    z->buf[total_copy_len] = 0;
-
-    /* set the start and end location of the zone buf
-     * to be scanned */
-    u8 *z_end = z->buf + total_copy_len;
-    z->end = z_end;
-    z->start = z_end - ITER_BYTES - iter_bytes_second;
-    assert(z->start + z->shift == z_end - z_len);
-
-    u8 *z_end_first = z_end - iter_bytes_second;
-    /* copy the first 8 bytes of the valid region */
-    unaligned_store_u64a(z->buf,
-                         unaligned_load_u64a(end_first - copy_len_first));
-
-    /* copy the last 16 bytes, may overlap with the previous 8 byte write */
-    storeu128(z_end_first - sizeof(m128), loadu128(end_first - sizeof(m128)));
-    if (iter_bytes_second) {
-        storeu128(z_end - sizeof(m128), loadu128(end - sizeof(m128)));
-    }
-
-    z->zone_pointer_adjust = (ptrdiff_t)((uintptr_t)end - (uintptr_t)z_end);
-}
-
-/**
- * \brief Prepare zones.
- *
- * This function prepares zones with actual buffer and some padded bytes.
- * The actual ITER_BYTES bytes in zone is preceded by main buf and/or
- * history buf and succeeded by padded bytes possibly from main buf,
- * if available.
- */
-static really_inline
-size_t prepareZones(const u8 *buf, size_t len, const u8 *hend,
-                    size_t start, const u8 *flood, struct zone *zoneArr) {
-    const u8 *ptr = buf + start;
-    size_t remaining = len - start;
-
-    if (remaining <= ITER_BYTES) {
-        /* enough bytes to make only one zone */
-        createShortZone(buf, hend, ptr, buf + len, &zoneArr[0]);
-        return 1;
-    }
-
-    /* enough bytes to make more than one zone */
-
-    size_t numZone = 0;
-    createStartZone(buf, hend, ptr, &zoneArr[numZone++]);
-    ptr += ITER_BYTES;
-
-    assert(ptr < buf + len);
-
-    /* find maximum buffer location that the main zone can scan
-     * - must be a multiple of ITER_BYTES, and
-     * - cannot contain the last 3 bytes (due to 3 bytes read behind the
-         end of buffer in FDR main loop)
-     */
-    const u8 *main_end = buf + start + ROUNDDOWN_N(len - start - 3, ITER_BYTES);
-
-    /* create a zone if multiple of ITER_BYTES are found */
-    if (main_end > ptr) {
-        createMainZone(flood, ptr, main_end, &zoneArr[numZone++]);
-        ptr = main_end;
-    }
-    /* create a zone with rest of the data from the main buffer */
-    createEndZone(buf, ptr, buf + len, &zoneArr[numZone++]);
-    return numZone;
-}
-
-#define INVALID_MATCH_ID (~0U)
-
-#define FDR_MAIN_LOOP(zz, s, get_conf_fn)                                   \
-    do {                                                                    \
-        const u8 *tryFloodDetect = zz->floodPtr;                            \
-        const u8 *start_ptr = zz->start;                                    \
-        const u8 *end_ptr = zz->end;                                        \
-        for (const u8 *itPtr = ROUNDDOWN_PTR(start_ptr, 64); itPtr + 4*ITER_BYTES <= end_ptr;      \
-            itPtr += 4*ITER_BYTES) {                                        \
-            __builtin_prefetch(itPtr);                                      \
-        }                                                                   \
-                                                                            \
-        for (const u8 *itPtr = start_ptr; itPtr + ITER_BYTES <= end_ptr;    \
-            itPtr += ITER_BYTES) {                                          \
-            if (unlikely(itPtr > tryFloodDetect)) {                         \
-                tryFloodDetect = floodDetect(fdr, a, &itPtr, tryFloodDetect,\
-                                             &floodBackoff, &control,       \
-                                             ITER_BYTES);                   \
-                if (unlikely(control == HWLM_TERMINATE_MATCHING)) {         \
-                    return HWLM_TERMINATED;                                 \
-                }                                                           \
-            }                                                               \
-            __builtin_prefetch(itPtr + ITER_BYTES);                         \
-            u64a conf0;                                                     \
-            u64a conf8;                                                     \
-            get_conf_fn(itPtr, start_ptr, end_ptr, domain_mask_flipped,     \
-                        ft, &conf0, &conf8, &s);                            \
-            do_confirm_fdr(&conf0, 0, &control, confBase, a, itPtr,         \
-                           &last_match_id, zz);                             \
-            do_confirm_fdr(&conf8, 8, &control, confBase, a, itPtr,         \
-                           &last_match_id, zz);                             \
-            if (unlikely(control == HWLM_TERMINATE_MATCHING)) {             \
-                return HWLM_TERMINATED;                                     \
-            }                                                               \
-        } /* end for loop */                                                \
-    } while (0)                                                             \
-
-static never_inline
-hwlm_error_t fdr_engine_exec(const struct FDR *fdr,
-                             const struct FDR_Runtime_Args *a,
-                             hwlm_group_t control) {
-    assert(ISALIGNED_CL(fdr));
-
-    u32 floodBackoff = FLOOD_BACKOFF_START;
-    u32 last_match_id = INVALID_MATCH_ID;
-    u32 domain_mask_flipped = ~fdr->domainMask;
-    u8 stride = fdr->stride;
-    const u64a *ft =
-        (const u64a *)((const u8 *)fdr + ROUNDUP_CL(sizeof(struct FDR)));
-    assert(ISALIGNED_CL(ft));
-    const u32 *confBase = (const u32 *)((const u8 *)fdr + fdr->confOffset);
-    assert(ISALIGNED_CL(confBase));
-    struct zone zones[ZONE_MAX];
-    assert(fdr->domain > 8 && fdr->domain < 16);
-    memset(zones, 0, sizeof(zones));
-
-    size_t numZone = prepareZones(a->buf, a->len,
-                                  a->buf_history + a->len_history,
-                                  a->start_offset, a->firstFloodDetect, zones);
-    assert(numZone <= ZONE_MAX);
-    m128 state = getInitState(fdr, a->len_history, ft, &zones[0]);
-
-    for (size_t curZone = 0; curZone < numZone; curZone++) {
-        struct zone *z = &zones[curZone];
-        dumpZoneInfo(z, curZone);
-
-        /* When a zone contains less data than is processed in an iteration
-         * of FDR_MAIN_LOOP(), we need to scan over some extra data.
-         *
-         * We have chosen to scan this extra data at the start of the
-         * iteration. The extra data is either data we have already scanned or
-         * garbage (if it is earlier than offset 0),
-         *
-         * As a result we need to shift the incoming state back so that it will
-         * properly line up with the data being scanned.
-         *
-         * We also need to forbid reporting any matches in the data being
-         * rescanned as they have already been reported (or are over garbage but
-         * later stages should also provide that safety guarantee).
-         */
-
-        u8 shift = z->shift;
-
-        state = variable_byte_shift_m128(state, shift);
-
-        state = or128(state, load128(zone_or_mask[shift]));
-
-        switch (stride) {
-        case 1:
-            FDR_MAIN_LOOP(z, state, get_conf_stride_1);
-            break;
-        case 2:
-            FDR_MAIN_LOOP(z, state, get_conf_stride_2);
-            break;
-        case 4:
-            FDR_MAIN_LOOP(z, state, get_conf_stride_4);
-            break;
-        default:
-            break;
-        }
-    }
-
-    return HWLM_SUCCESS;
-}
-
-#if defined(HAVE_AVX2)
-#define ONLY_AVX2(func) func
-#else
-#define ONLY_AVX2(func) NULL
-#endif
-
-typedef hwlm_error_t (*FDRFUNCTYPE)(const struct FDR *fdr,
-                                    const struct FDR_Runtime_Args *a,
-                                    hwlm_group_t control);
-
-static const FDRFUNCTYPE funcs[] = {
-    fdr_engine_exec,
-    NULL, /* old: fast teddy */
-    NULL, /* old: fast teddy */
-    ONLY_AVX2(fdr_exec_fat_teddy_msks1),
-    ONLY_AVX2(fdr_exec_fat_teddy_msks1_pck),
-    ONLY_AVX2(fdr_exec_fat_teddy_msks2),
-    ONLY_AVX2(fdr_exec_fat_teddy_msks2_pck),
-    ONLY_AVX2(fdr_exec_fat_teddy_msks3),
-    ONLY_AVX2(fdr_exec_fat_teddy_msks3_pck),
-    ONLY_AVX2(fdr_exec_fat_teddy_msks4),
-    ONLY_AVX2(fdr_exec_fat_teddy_msks4_pck),
-    fdr_exec_teddy_msks1,
-    fdr_exec_teddy_msks1_pck,
-    fdr_exec_teddy_msks2,
-    fdr_exec_teddy_msks2_pck,
-    fdr_exec_teddy_msks3,
-    fdr_exec_teddy_msks3_pck,
-    fdr_exec_teddy_msks4,
-    fdr_exec_teddy_msks4_pck,
-};
-
-#define FAKE_HISTORY_SIZE 16
-static const u8 fake_history[FAKE_HISTORY_SIZE];
-
-hwlm_error_t fdrExec(const struct FDR *fdr, const u8 *buf, size_t len,
-                     size_t start, HWLMCallback cb,
-                     struct hs_scratch *scratch, hwlm_group_t groups) {
-    // We guarantee (for safezone construction) that it is safe to read 16
-    // bytes before the end of the history buffer.
-    const u8 *hbuf = fake_history + FAKE_HISTORY_SIZE;
-
-    const struct FDR_Runtime_Args a = {
-        buf,
-        len,
-        hbuf,
-        0,
-        start,
-        cb,
-        scratch,
-        nextFloodDetect(buf, len, FLOOD_BACKOFF_START),
-        0
-    };
-    if (unlikely(a.start_offset >= a.len)) {
-        return HWLM_SUCCESS;
-    } else {
-        assert(funcs[fdr->engineID]);
-        return funcs[fdr->engineID](fdr, &a, groups);
-    }
-}
-
-hwlm_error_t fdrExecStreaming(const struct FDR *fdr, const u8 *hbuf,
-                              size_t hlen, const u8 *buf, size_t len,
-                              size_t start, HWLMCallback cb,
-                              struct hs_scratch *scratch,
-                              hwlm_group_t groups) {
-    struct FDR_Runtime_Args a = {
-        buf,
-        len,
-        hbuf,
-        hlen,
-        start,
-        cb,
-        scratch,
-        nextFloodDetect(buf, len, FLOOD_BACKOFF_START),
-        /* we are guaranteed to always have 16 initialised bytes at the end of
-         * the history buffer (they may be garbage). */
-        hbuf ? unaligned_load_u64a(hbuf + hlen - sizeof(u64a)) : (u64a)0
-    };
-
-    hwlm_error_t ret;
-    if (unlikely(a.start_offset >= a.len)) {
-        ret = HWLM_SUCCESS;
-    } else {
-        assert(funcs[fdr->engineID]);
-        ret = funcs[fdr->engineID](fdr, &a, groups);
-    }
-
-    return ret;
-}

src/fdr/fdr.h

@@ -1,85 +0,0 @@
-/*
- * Copyright (c) 2015-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief FDR literal matcher: runtime API.
- */
-
-#ifndef FDR_H
-#define FDR_H
-
-#include "ue2common.h"
-#include "hwlm/hwlm.h"
-
-// C linkage in the API
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-struct FDR;
-struct hs_scratch;
-
-/**
- * \brief Block-mode scan.
- *
- * \param fdr FDR matcher engine.
- * \param buf Buffer to scan.
- * \param len Length of buffer to scan.
- * \param start First offset in buf at which a match may start.
- * \param cb Callback to call when a match is found.
- * \param scratch Scratch supplied to callback on match.
- * \param groups Initial groups mask.
- */
-hwlm_error_t fdrExec(const struct FDR *fdr, const u8 *buf, size_t len,
-                     size_t start, HWLMCallback cb, struct hs_scratch *scratch,
-                     hwlm_group_t groups);
-
-/**
- * \brief Streaming-mode scan.
- *
- * \param fdr FDR matcher engine.
- * \param hbuf History buffer.
- * \param hlen Length of history buffer (hbuf).
- * \param buf Buffer to scan.
- * \param len Length of buffer to scan (buf).
- * \param start First offset in buf at which a match may start.
- * \param cb Callback to call when a match is found.
- * \param scratch Scratch supplied to callback on match.
- * \param groups Initial groups mask.
- */
-hwlm_error_t fdrExecStreaming(const struct FDR *fdr, const u8 *hbuf,
-                              size_t hlen, const u8 *buf, size_t len,
-                              size_t start, HWLMCallback cb,
-                              struct hs_scratch *scratch,
-                              hwlm_group_t groups);
-
-#ifdef __cplusplus
-}
-#endif // __cplusplus
-
-#endif // FDR_H

src/fdr/fdr_compile.cpp

@@ -1,917 +0,0 @@
-/*
- * Copyright (c) 2015-2019, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief FDR literal matcher: build API.
- */
-
-#include "fdr_compile.h"
-
-#include "fdr_internal.h"
-#include "fdr_confirm.h"
-#include "fdr_compile_internal.h"
-#include "fdr_engine_description.h"
-#include "teddy_compile.h"
-#include "teddy_engine_description.h"
-#include "grey.h"
-#include "ue2common.h"
-#include "hwlm/hwlm_build.h"
-#include "util/compare.h"
-#include "util/container.h"
-#include "util/dump_mask.h"
-#include "util/math.h"
-#include "util/noncopyable.h"
-#include "util/target_info.h"
-#include "util/ue2string.h"
-#include "util/verify_types.h"
-
-#include <algorithm>
-#include <array>
-#include <cassert>
-#include <cctype>
-#include <cstdio>
-#include <cstdlib>
-#include <cstring>
-#include <limits>
-#include <map>
-#include <memory>
-#include <numeric>
-#include <set>
-#include <string>
-#include <unordered_map>
-#include <unordered_set>
-#include <vector>
-
-#include <boost/multi_array.hpp>
-
-using namespace std;
-
-namespace ue2 {
-
-namespace {
-
-class FDRCompiler : noncopyable {
-private:
-    const FDREngineDescription &eng;
-    const Grey &grey;
-    vector<u8> tab;
-    vector<hwlmLiteral> lits;
-    map<BucketIndex, std::vector<LiteralIndex> > bucketToLits;
-    bool make_small;
-
-    u8 *tabIndexToMask(u32 indexInTable);
-#ifdef DEBUG
-    void dumpMasks(const u8 *defaultMask);
-#endif
-    void setupTab();
-    bytecode_ptr<FDR> setupFDR();
-    void createInitialState(FDR *fdr);
-
-public:
-    FDRCompiler(vector<hwlmLiteral> lits_in,
-                map<BucketIndex, std::vector<LiteralIndex>> bucketToLits_in,
-                const FDREngineDescription &eng_in,
-                bool make_small_in, const Grey &grey_in)
-        : eng(eng_in), grey(grey_in), tab(eng_in.getTabSizeBytes()),
-          lits(move(lits_in)), bucketToLits(move(bucketToLits_in)),
-          make_small(make_small_in) {}
-
-    bytecode_ptr<FDR> build();
-};
-
-u8 *FDRCompiler::tabIndexToMask(u32 indexInTable) {
-    assert(indexInTable < tab.size());
-    return &tab[0] + (indexInTable * (eng.getSchemeWidth() / 8));
-}
-
-static
-void setbit(u8 *msk, u32 bit) {
-    msk[bit / 8] |= 1U << (bit % 8);
-}
-
-static
-void clearbit(u8 *msk, u32 bit) {
-    msk[bit / 8] &= ~(1U << (bit % 8));
-}
-
-static
-void andMask(u8 *dest, const u8 *a, const u8 *b, u32 num_bytes) {
-    for (u32 i = 0; i < num_bytes; i++) {
-        dest[i] = a[i] & b[i];
-    }
-}
-
-void FDRCompiler::createInitialState(FDR *fdr) {
-    u8 *start = (u8 *)&fdr->start;
-
-    /* initial state should to be 1 in each slot in the bucket up to bucket
-     * minlen - 1, and 0 thereafter */
-    for (BucketIndex b = 0; b < eng.getNumBuckets(); b++) {
-        // Find the minimum length for the literals in this bucket.
-        const vector<LiteralIndex> &bucket_lits = bucketToLits[b];
-        u32 min_len = ~0U;
-        for (const LiteralIndex &lit_idx : bucket_lits) {
-            min_len = min(min_len, verify_u32(lits[lit_idx].s.length()));
-        }
-
-        DEBUG_PRINTF("bucket %u has min_len=%u\n", b, min_len);
-        assert(min_len);
-
-        for (PositionInBucket i = 0; i < eng.getBucketWidth(b); i++) {
-            if (i < min_len - 1) {
-                setbit(start, eng.getSchemeBit(b, i));
-            }
-        }
-    }
-}
-
-/**
- * \brief Lay out FDR structures in bytecode.
- *
- * Note that each major structure (header, table, confirm, flood control) is
- * cacheline-aligned.
- */
-bytecode_ptr<FDR> FDRCompiler::setupFDR() {
-    auto floodTable = setupFDRFloodControl(lits, eng, grey);
-    auto confirmTable = setupFullConfs(lits, eng, bucketToLits, make_small);
-
-    size_t headerSize = sizeof(FDR);
-    size_t tabSize = eng.getTabSizeBytes();
-
-    // Note: we place each major structure here on a cacheline boundary.
-    size_t size = ROUNDUP_CL(headerSize) + ROUNDUP_CL(tabSize) +
-                  ROUNDUP_CL(confirmTable.size()) + floodTable.size();
-
-    DEBUG_PRINTF("sizes base=%zu tabSize=%zu confirm=%zu floodControl=%zu "
-                 "total=%zu\n",
-                 headerSize, tabSize, confirmTable.size(), floodTable.size(),
-                 size);
-
-    auto fdr = make_zeroed_bytecode_ptr<FDR>(size, 64);
-    assert(fdr); // otherwise would have thrown std::bad_alloc
-
-    u8 *fdr_base = (u8 *)fdr.get();
-
-    // Write header.
-    fdr->size = size;
-    fdr->engineID = eng.getID();
-    fdr->maxStringLen = verify_u32(maxLen(lits));
-    fdr->numStrings = verify_u32(lits.size());
-    assert(eng.bits > 8 && eng.bits < 16); // we allow domains 9 to 15 only
-    fdr->domain = eng.bits;
-    fdr->domainMask = (1 << eng.bits) - 1;
-    fdr->tabSize = tabSize;
-    fdr->stride = eng.stride;
-    createInitialState(fdr.get());
-
-    // Write table.
-    u8 *ptr = fdr_base + ROUNDUP_CL(sizeof(FDR));
-    assert(ISALIGNED_CL(ptr));
-    copy(tab.begin(), tab.end(), ptr);
-    ptr += ROUNDUP_CL(tabSize);
-
-    // Write confirm structures.
-    assert(ISALIGNED_CL(ptr));
-    fdr->confOffset = verify_u32(ptr - fdr_base);
-    memcpy(ptr, confirmTable.get(), confirmTable.size());
-    ptr += ROUNDUP_CL(confirmTable.size());
-
-    // Write flood control structures.
-    assert(ISALIGNED_CL(ptr));
-    fdr->floodOffset = verify_u32(ptr - fdr_base);
-    memcpy(ptr, floodTable.get(), floodTable.size());
-    ptr += floodTable.size(); // last write, no need to round up
-
-    return fdr;
-}
-
-//#define DEBUG_ASSIGNMENT
-
-/**
- * Utility class for computing:
- *
- *    score(count, len) = pow(count, 1.05) * pow(len, -3)
- *
- * Calling pow() is expensive. This is mitigated by using pre-computed LUTs for
- * small inputs and a cache for larger ones.
- */
-class Scorer {
-    unordered_map<u32, double> count_factor_cache;
-
-    // LUT: pow(count, 1.05) for small values of count.
-    static const array<double, 100> count_lut;
-
-    double count_factor(u32 count) {
-        if (count < count_lut.size()) {
-            return count_lut[count];
-        }
-
-        auto it = count_factor_cache.find(count);
-        if (it != count_factor_cache.end()) {
-            return it->second;
-        }
-        double r = our_pow(count, 1.05);
-        count_factor_cache.emplace(count, r);
-        return r;
-    }
-
-    // LUT: pow(len, -3) for len in range [0,8].
-    static const array<double, 9> len_lut;
-
-    double len_factor(u32 len) {
-        assert(len <= len_lut.size());
-        return len_lut[len];
-    }
-
-public:
-    double operator()(u32 len, u32 count) {
-        if (len == 0) {
-            return numeric_limits<double>::max();
-        }
-        return count_factor(count) * len_factor(len);
-    }
-};
-
-const array<double, 100> Scorer::count_lut{{
-    pow(0, 1.05),  pow(1, 1.05),  pow(2, 1.05),  pow(3, 1.05),  pow(4, 1.05),
-    pow(5, 1.05),  pow(6, 1.05),  pow(7, 1.05),  pow(8, 1.05),  pow(9, 1.05),
-    pow(10, 1.05), pow(11, 1.05), pow(12, 1.05), pow(13, 1.05), pow(14, 1.05),
-    pow(15, 1.05), pow(16, 1.05), pow(17, 1.05), pow(18, 1.05), pow(19, 1.05),
-    pow(20, 1.05), pow(21, 1.05), pow(22, 1.05), pow(23, 1.05), pow(24, 1.05),
-    pow(25, 1.05), pow(26, 1.05), pow(27, 1.05), pow(28, 1.05), pow(29, 1.05),
-    pow(30, 1.05), pow(31, 1.05), pow(32, 1.05), pow(33, 1.05), pow(34, 1.05),
-    pow(35, 1.05), pow(36, 1.05), pow(37, 1.05), pow(38, 1.05), pow(39, 1.05),
-    pow(40, 1.05), pow(41, 1.05), pow(42, 1.05), pow(43, 1.05), pow(44, 1.05),
-    pow(45, 1.05), pow(46, 1.05), pow(47, 1.05), pow(48, 1.05), pow(49, 1.05),
-    pow(50, 1.05), pow(51, 1.05), pow(52, 1.05), pow(53, 1.05), pow(54, 1.05),
-    pow(55, 1.05), pow(56, 1.05), pow(57, 1.05), pow(58, 1.05), pow(59, 1.05),
-    pow(60, 1.05), pow(61, 1.05), pow(62, 1.05), pow(63, 1.05), pow(64, 1.05),
-    pow(65, 1.05), pow(66, 1.05), pow(67, 1.05), pow(68, 1.05), pow(69, 1.05),
-    pow(70, 1.05), pow(71, 1.05), pow(72, 1.05), pow(73, 1.05), pow(74, 1.05),
-    pow(75, 1.05), pow(76, 1.05), pow(77, 1.05), pow(78, 1.05), pow(79, 1.05),
-    pow(80, 1.05), pow(81, 1.05), pow(82, 1.05), pow(83, 1.05), pow(84, 1.05),
-    pow(85, 1.05), pow(86, 1.05), pow(87, 1.05), pow(88, 1.05), pow(89, 1.05),
-    pow(90, 1.05), pow(91, 1.05), pow(92, 1.05), pow(93, 1.05), pow(94, 1.05),
-    pow(95, 1.05), pow(96, 1.05), pow(97, 1.05), pow(98, 1.05), pow(99, 1.05),
-}};
-
-const array<double, 9> Scorer::len_lut{{
-    0, pow(1, -3.0), pow(2, -3.0), pow(3, -3.0), pow(4, -3.0),
-       pow(5, -3.0), pow(6, -3.0), pow(7, -3.0), pow(8, -3.0)}};
-
-/**
- * Returns true if the two given literals should be placed in the same chunk as
- * they are identical except for a difference in caselessness.
- */
-static
-bool isEquivLit(const hwlmLiteral &a, const hwlmLiteral &b,
-                const hwlmLiteral *last_nocase_lit) {
-    const size_t a_len = a.s.size();
-    const size_t b_len = b.s.size();
-
-    if (a_len != b_len) {
-        return false;
-    }
-
-    bool nocase = last_nocase_lit && a_len == last_nocase_lit->s.size() &&
-                  !cmp(a.s.c_str(), last_nocase_lit->s.c_str(), a_len, true);
-    return !cmp(a.s.c_str(), b.s.c_str(), a.s.size(), nocase);
-}
-
-struct Chunk {
-    Chunk(u32 first_id_in, u32 count_in, u32 length_in)
-        : first_id(first_id_in), count(count_in), length(length_in) {}
-    u32 first_id; //!< first id in this chunk
-    u32 count;    //!< how many are in this chunk
-    u32 length;   //!< how long things in the chunk are
-};
-
-static
-vector<Chunk> assignChunks(const vector<hwlmLiteral> &lits,
-                           const map<u32, u32> &lenCounts) {
-    const u32 CHUNK_MAX = 512;
-    const u32 MAX_CONSIDERED_LENGTH = 16;
-
-    // TODO: detailed early stage literal analysis for v. small cases (actually
-    // look at lits) yes - after we factor this out and merge in the Teddy
-    // style of building we can look at this, although the teddy merge
-    // modelling is quite different. It's still probably adaptable to some
-    // extent for this class of problem.
-
-    vector<Chunk> chunks;
-    chunks.reserve(CHUNK_MAX);
-
-    const u32 maxPerChunk = lits.size() /
-            (CHUNK_MAX - MIN(MAX_CONSIDERED_LENGTH, lenCounts.size())) + 1;
-
-    u32 currentSize = 0;
-    u32 chunkStartID = 0;
-    const hwlmLiteral *last_nocase_lit = nullptr;
-
-    for (u32 i = 0; i < lits.size() && chunks.size() < CHUNK_MAX - 1; i++) {
-        const auto &lit = lits[i];
-
-        DEBUG_PRINTF("i=%u, lit=%s%s\n", i, escapeString(lit.s).c_str(),
-                      lit.nocase ? " (nocase)" : "");
-
-        // If this literal is identical to the last one (aside from differences
-        // in caselessness), keep going even if we will "overfill" a chunk; we
-        // don't want to split identical literals into different buckets.
-        if (i != 0 && isEquivLit(lit, lits[i - 1], last_nocase_lit)) {
-            DEBUG_PRINTF("identical lit\n");
-            goto next_literal;
-        }
-
-        if ((currentSize < MAX_CONSIDERED_LENGTH &&
-             (lit.s.size() != currentSize)) ||
-            (currentSize != 1 && ((i - chunkStartID) >= maxPerChunk))) {
-            currentSize = lit.s.size();
-            if (!chunks.empty()) {
-                chunks.back().count = i - chunkStartID;
-            }
-            chunkStartID = i;
-            chunks.emplace_back(i, 0, currentSize);
-        }
-next_literal:
-        if (lit.nocase) {
-            last_nocase_lit = &lit;
-        }
-    }
-
-    assert(!chunks.empty());
-    chunks.back().count = lits.size() - chunkStartID;
-    // close off chunks with an empty row
-    chunks.emplace_back(lits.size(), 0, 0);
-
-#ifdef DEBUG_ASSIGNMENT
-    for (size_t j = 0; j < chunks.size(); j++) {
-        const auto &chunk = chunks[j];
-        printf("chunk %zu first_id=%u count=%u length=%u\n", j, chunk.first_id,
-               chunk.count, chunk.length);
-    }
-#endif
-
-    DEBUG_PRINTF("built %zu chunks (%zu lits)\n", chunks.size(), lits.size());
-    assert(chunks.size() <= CHUNK_MAX);
-    return chunks;
-}
-
-static
-map<BucketIndex, vector<LiteralIndex>> assignStringsToBuckets(
-                                    vector<hwlmLiteral> &lits,
-                                    const FDREngineDescription &eng) {
-    const double MAX_SCORE = numeric_limits<double>::max();
-
-    assert(!lits.empty()); // Shouldn't be called with no literals.
-
-    // Count the number of literals for each length.
-    map<u32, u32> lenCounts;
-    for (const auto &lit : lits) {
-        lenCounts[lit.s.size()]++;
-    }
-
-#ifdef DEBUG_ASSIGNMENT
-    for (const auto &m : lenCounts) {
-        printf("l<%u>:%u ", m.first, m.second);
-    }
-    printf("\n");
-#endif
-
-    // Sort literals by literal length. If tied on length, use lexicographic
-    // ordering (of the reversed literals).
-    stable_sort(lits.begin(), lits.end(),
-                [](const hwlmLiteral &a, const hwlmLiteral &b) {
-                    if (a.s.size() != b.s.size()) {
-                        return a.s.size() < b.s.size();
-                    }
-                    auto p = mismatch(a.s.rbegin(), a.s.rend(), b.s.rbegin());
-                    if (p.first != a.s.rend()) {
-                        return *p.first < *p.second;
-                    }
-                    // Sort caseless variants first.
-                    return a.nocase > b.nocase;
-                });
-
-    vector<Chunk> chunks = assignChunks(lits, lenCounts);
-
-    const u32 numChunks = chunks.size();
-    const u32 numBuckets = eng.getNumBuckets();
-
-    // 2D array of (score, chunk index) pairs, indexed by
-    // [chunk_index][bucket_index].
-    boost::multi_array<pair<double, u32>, 2> t(
-        boost::extents[numChunks][numBuckets]);
-
-    Scorer scorer;
-
-    for (u32 j = 0; j < numChunks; j++) {
-        u32 cnt = 0;
-        for (u32 k = j; k < numChunks; ++k) {
-            cnt += chunks[k].count;
-        }
-        t[j][0] = {scorer(chunks[j].length, cnt), 0};
-    }
-
-    for (u32 i = 1; i < numBuckets; i++) {
-        for (u32 j = 0; j < numChunks - 1; j++) { // don't do last, empty row
-            pair<double, u32> best = {MAX_SCORE, 0};
-            u32 cnt = chunks[j].count;
-            for (u32 k = j + 1; k < numChunks - 1; k++) {
-                auto score = scorer(chunks[j].length, cnt);
-                if (score > best.first) {
-                    break; // now worse locally than our best score, give up
-                }
-                score += t[k][i-1].first;
-                if (score < best.first) {
-                    best = {score, k};
-                }
-                cnt += chunks[k].count;
-            }
-            t[j][i] = best;
-        }
-        t[numChunks - 1][i] = {0,0}; // fill in empty final row for next iter
-    }
-
-#ifdef DEBUG_ASSIGNMENT
-    for (u32 j = 0; j < numChunks; j++) {
-        printf("%03u: ", j);
-        for (u32 i = 0; i < numBuckets; i++) {
-            const auto &v = t[j][i];
-            printf("<%0.3f,%3d> ", v.first, v.second);
-        }
-        printf("\n");
-    }
-#endif
-
-    // our best score is in t[0][N_BUCKETS-1] and we can follow the links
-    // to find where our buckets should start and what goes into them
-    vector<vector<LiteralIndex>> buckets;
-    for (u32 i = 0, n = numBuckets; n && (i != numChunks - 1); n--) {
-        u32 j = t[i][n - 1].second;
-        if (j == 0) {
-            j = numChunks - 1;
-        }
-
-        // put chunks between i - j into bucket (numBuckets - n).
-        u32 first_id = chunks[i].first_id;
-        u32 last_id = chunks[j].first_id;
-        assert(first_id < last_id);
-        UNUSED const auto &first_lit = lits[first_id];
-        UNUSED const auto &last_lit = lits[last_id - 1];
-        DEBUG_PRINTF("placing [%u-%u) in one bucket (%u lits, len %zu-%zu, "
-                     "score %0.4f)\n",
-                     first_id, last_id, last_id - first_id,
-                     first_lit.s.length(), last_lit.s.length(),
-                     scorer(first_lit.s.length(), last_id - first_id));
-
-        vector<LiteralIndex> litIds;
-        u32 cnt = last_id - first_id;
-        // long literals first for included literals checking
-        for (u32 k = 0; k < cnt; k++) {
-            litIds.emplace_back(last_id - k - 1);
-        }
-
-        i = j;
-        buckets.emplace_back(litIds);
-    }
-
-    // reverse bucket id, longer literals come first
-    map<BucketIndex, vector<LiteralIndex>> bucketToLits;
-    size_t bucketCnt = buckets.size();
-    for (size_t i = 0; i < bucketCnt; i++) {
-        bucketToLits.emplace(bucketCnt - i - 1, move(buckets[i]));
-    }
-
-    return bucketToLits;
-}
-
-#ifdef DEBUG
-void FDRCompiler::dumpMasks(const u8 *defaultMask) {
-    const size_t width = eng.getSchemeWidth();
-    printf("default mask: %s\n", dumpMask(defaultMask, width).c_str());
-    for (u32 i = 0; i < eng.getNumTableEntries(); i++) {
-        u8 *m = tabIndexToMask(i);
-        if (memcmp(m, defaultMask, width / 8)) {
-            printf("tab %04x: %s\n", i, dumpMask(m, width).c_str());
-        }
-    }
-}
-#endif
-
-static
-bool getMultiEntriesAtPosition(const FDREngineDescription &eng,
-                               const vector<LiteralIndex> &vl,
-                               const vector<hwlmLiteral> &lits,
-                               SuffixPositionInString pos,
-                               map<u32, unordered_set<u32>> &m2) {
-    assert(eng.bits < 32);
-
-    u32 distance = 0;
-    if (eng.bits <= 8) {
-        distance = 1;
-    } else if (eng.bits <= 16) {
-        distance = 2;
-    } else {
-        distance = 4;
-    }
-
-    for (auto i = vl.begin(), e = vl.end(); i != e; ++i) {
-        if (e - i > 5) {
-            __builtin_prefetch(&lits[*(i + 5)]);
-        }
-        const hwlmLiteral &lit = lits[*i];
-        const size_t sz = lit.s.size();
-        u32 mask = 0;
-        u32 dontCares = 0;
-        for (u32 cnt = 0; cnt < distance; cnt++) {
-            int newPos = pos - cnt;
-            u8 dontCareByte = 0x0;
-            u8 maskByte = 0x0;
-            if (newPos < 0 || ((u32)newPos >= sz)) {
-                dontCareByte = 0xff;
-            } else {
-                u8 c = lit.s[sz - newPos - 1];
-                maskByte = c;
-                u32 remainder = eng.bits - cnt * 8;
-                assert(remainder != 0);
-                if (remainder < 8) {
-                    u8 cmask = (1U << remainder) - 1;
-                    maskByte &= cmask;
-                    dontCareByte |= ~cmask;
-                }
-                if (lit.nocase && ourisalpha(c)) {
-                    maskByte &= 0xdf;
-                    dontCareByte |= 0x20;
-                }
-            }
-            u32 loc =  cnt * 8;
-            mask |= maskByte << loc;
-            dontCares |= dontCareByte << loc;
-        }
-
-        // truncate m and dc down to nBits
-        mask &= (1U << eng.bits) - 1;
-        dontCares &= (1U << eng.bits) - 1;
-        if (dontCares == ((1U << eng.bits) - 1)) {
-            return true;
-        }
-        m2[dontCares].insert(mask);
-    }
-    return false;
-}
-
-void FDRCompiler::setupTab() {
-    const size_t mask_size = eng.getSchemeWidth() / 8;
-    assert(mask_size);
-
-    vector<u8> defaultMask(mask_size, 0xff);
-    for (u32 i = 0; i < eng.getNumTableEntries(); i++) {
-        memcpy(tabIndexToMask(i), &defaultMask[0], mask_size);
-    }
-
-    for (BucketIndex b = 0; b < eng.getNumBuckets(); b++) {
-        const vector<LiteralIndex> &vl = bucketToLits[b];
-        SuffixPositionInString pLimit = eng.getBucketWidth(b);
-        for (SuffixPositionInString pos = 0; pos < pLimit; pos++) {
-            u32 bit = eng.getSchemeBit(b, pos);
-            map<u32, unordered_set<u32>> m2;
-            bool done = getMultiEntriesAtPosition(eng, vl, lits, pos, m2);
-            if (done) {
-                clearbit(&defaultMask[0], bit);
-                continue;
-            }
-            for (const auto &elem : m2) {
-                u32 dc = elem.first;
-                const unordered_set<u32> &mskSet = elem.second;
-                u32 v = ~dc;
-                do {
-                    u32 b2 = v & dc;
-                    for (const u32 &mskVal : mskSet) {
-                        u32 val = (mskVal & ~dc) | b2;
-                        clearbit(tabIndexToMask(val), bit);
-                    }
-                    v = (v + (dc & -dc)) | ~dc;
-                } while (v != ~dc);
-            }
-        }
-    }
-
-    for (u32 i = 0; i < eng.getNumTableEntries(); i++) {
-        u8 *m = tabIndexToMask(i);
-        andMask(m, m, &defaultMask[0], mask_size);
-    }
-#ifdef DEBUG
-    dumpMasks(&defaultMask[0]);
-#endif
-}
-
-bytecode_ptr<FDR> FDRCompiler::build() {
-    setupTab();
-    return setupFDR();
-}
-
-static
-bool isSuffix(const hwlmLiteral &lit1, const hwlmLiteral &lit2) {
-    const auto &s1 = lit1.s;
-    const auto &s2 = lit2.s;
-    size_t len1 = s1.length();
-    size_t len2 = s2.length();
-    assert(len1 >= len2);
-
-    if (lit1.nocase || lit2.nocase) {
-        return equal(s2.begin(), s2.end(), s1.begin() + len1 - len2,
-            [](char a, char b) { return mytoupper(a) == mytoupper(b); });
-    } else {
-        return equal(s2.begin(), s2.end(), s1.begin() + len1 - len2);
-    }
-}
-
-/*
- * if lit2 is a suffix of lit1 but the case sensitivity, groups or mask info
- * of lit2 is a subset of lit1, then lit1 can't squash lit2 and lit2 can
- * possibly match when lit1 matches. In this case, we can't do bucket
- * squashing. e.g. AAA(no case) in bucket 0, AA(no case) and aa in bucket 1,
- * we can't squash bucket 1 if we have input like "aaa" as aa can also match.
- */
-static
-bool includedCheck(const hwlmLiteral &lit1, const hwlmLiteral &lit2) {
-    /* lit1 is caseless and lit2 is case sensitive */
-    if ((lit1.nocase && !lit2.nocase)) {
-        return true;
-    }
-
-    /* lit2's group is a subset of lit1 */
-    if (lit1.groups != lit2.groups &&
-        (lit2.groups == (lit1.groups & lit2.groups))) {
-        return true;
-    }
-
-    /* TODO: narrow down cases for mask check */
-    if (lit1.cmp != lit2.cmp || lit1.msk != lit2.msk) {
-        return true;
-    }
-
-    return false;
-}
-
-/*
- * if lit2 is an included literal of both lit0 and lit1, then lit0 and lit1
- * shouldn't match at the same offset, otherwise we give up squashing for lit1.
- * e.g. lit0:AAA(no case), lit1:aa, lit2:A(no case). We can have duplicate
- * matches for input "aaa" if lit0 and lit1 both squash lit2.
- */
-static
-bool checkParentLit(
-            const vector<hwlmLiteral> &lits, u32 pos1,
-            const unordered_set<u32> &parent_map,
-            const unordered_map<u32, unordered_set<u32>> &exception_map) {
-    assert(pos1 < lits.size());
-    const auto &lit1 = lits[pos1];
-    for (const auto pos2 : parent_map) {
-        if (contains(exception_map, pos2)) {
-            const auto &exception_pos = exception_map.at(pos2);
-            if (contains(exception_pos, pos1)) {
-                return false;
-            }
-        }
-
-        /* if lit1 isn't an exception of lit2, then we have to do further
-         * exclusive check.
-         * TODO: More mask checks. Note if two literals are group exclusive,
-         * it is possible that they match at the same offset. */
-        assert(pos2 < lits.size());
-        const auto &lit2 = lits[pos2];
-        if (isSuffix(lit2, lit1)) {
-            return false;
-        }
-    }
-
-    return true;
-}
-
-static
-void buildSquashMask(vector<hwlmLiteral> &lits, u32 id1, u32 bucket1,
-                     size_t start, const vector<pair<u32, u32>> &group,
-                     unordered_map<u32, unordered_set<u32>> &parent_map,
-                     unordered_map<u32, unordered_set<u32>> &exception_map) {
-    auto &lit1 = lits[id1];
-    DEBUG_PRINTF("b:%u len:%zu\n", bucket1, lit1.s.length());
-
-    size_t cnt = group.size();
-    bool included = false;
-    bool exception = false;
-    u32 child_id = ~0U;
-    for (size_t i = start; i < cnt; i++) {
-        u32 bucket2 = group[i].first;
-        assert(bucket2 >= bucket1);
-
-        u32 id2 = group[i].second;
-        auto &lit2 = lits[id2];
-        // check if lit2 is a suffix of lit1
-        if (isSuffix(lit1, lit2)) {
-            /* if we have a included literal in the same bucket,
-             * quit and let the included literal to do possible squashing */
-            if (bucket1 == bucket2) {
-                DEBUG_PRINTF("same bucket\n");
-                return;
-            }
-            /* if lit2 is a suffix but doesn't pass included checks for
-             * extra info, we give up sqaushing */
-            if (includedCheck(lit1, lit2)) {
-                DEBUG_PRINTF("find exceptional suffix %u\n", lit2.id);
-                exception_map[id1].insert(id2);
-                exception = true;
-            } else if (checkParentLit(lits, id1, parent_map[id2],
-                       exception_map)) {
-                if (lit1.included_id == INVALID_LIT_ID) {
-                    DEBUG_PRINTF("find suffix lit1 %u lit2 %u\n",
-                                 lit1.id, lit2.id);
-                    lit1.included_id = lit2.id;
-                } else {
-                    /* if we have multiple included literals in one bucket,
-                     * give up squashing. */
-                    DEBUG_PRINTF("multiple included literals\n");
-                    lit1.included_id = INVALID_LIT_ID;
-                    return;
-                }
-                child_id = id2;
-                included = true;
-            }
-        }
-
-        size_t next = i + 1;
-        u32 nextBucket = next < cnt ? group[next].first : ~0U;
-        if (bucket2 != nextBucket) {
-            if (included) {
-                if (exception) {
-                    /* give up if we have exception literals
-                     * in the same bucket as the included literal. */
-                    lit1.included_id = INVALID_LIT_ID;
-                } else {
-                    parent_map[child_id].insert(id1);
-
-                    lit1.squash |= 1U << bucket2;
-                    DEBUG_PRINTF("build squash mask %2x for %u\n",
-                                 lit1.squash, lit1.id);
-                }
-                return;
-            }
-            exception = false;
-        }
-    }
-}
-
-static constexpr u32 INCLUDED_LIMIT = 1000;
-
-static
-void findIncludedLits(vector<hwlmLiteral> &lits,
-                      const vector<vector<pair<u32, u32>>> &lastCharMap) {
-    /* Map for finding the positions of literal which includes a literal
-     * in FDR hwlm literal vector. */
-    unordered_map<u32, unordered_set<u32>> parent_map;
-
-    /* Map for finding the positions of exception literals which could
-     * sometimes match if a literal matches in FDR hwlm literal vector. */
-    unordered_map<u32, unordered_set<u32>> exception_map;
-    for (const auto &group : lastCharMap) {
-        size_t cnt = group.size();
-        if (cnt > INCLUDED_LIMIT) {
-            continue;
-        }
-        for (size_t i = 0; i < cnt; i++) {
-            u32 bucket1 = group[i].first;
-            u32 id1 = group[i].second;
-            buildSquashMask(lits, id1, bucket1, i + 1, group, parent_map,
-                            exception_map);
-        }
-    }
-}
-
-static
-void addIncludedInfo(
-               vector<hwlmLiteral> &lits, u32 nBuckets,
-               map<BucketIndex, vector<LiteralIndex>> &bucketToLits) {
-    vector<vector<pair<u32, u32>>> lastCharMap(256);
-
-    for (BucketIndex b = 0; b < nBuckets; b++) {
-        if (!bucketToLits[b].empty()) {
-            for (const LiteralIndex &lit_idx : bucketToLits[b]) {
-                const auto &lit = lits[lit_idx];
-                u8 c = mytoupper(lit.s.back());
-                lastCharMap[c].emplace_back(b, lit_idx);
-            }
-        }
-    }
-
-    findIncludedLits(lits, lastCharMap);
-}
-
-} // namespace
-
-static
-unique_ptr<HWLMProto> fdrBuildProtoInternal(u8 engType,
-                                            vector<hwlmLiteral> &lits,
-                                            bool make_small,
-                                            const target_t &target,
-                                            const Grey &grey, u32 hint) {
-    DEBUG_PRINTF("cpu has %s\n", target.has_avx2() ? "avx2" : "no-avx2");
-
-    if (grey.fdrAllowTeddy) {
-        auto proto = teddyBuildProtoHinted(engType, lits, make_small, hint,
-                                           target);
-        if (proto) {
-            DEBUG_PRINTF("build with teddy succeeded\n");
-            return proto;
-        } else {
-            DEBUG_PRINTF("build with teddy failed, will try with FDR\n");
-        }
-    }
-
-    auto des = (hint == HINT_INVALID) ? chooseEngine(target, lits, make_small)
-                                      : getFdrDescription(hint);
-    if (!des) {
-        return nullptr;
-    }
-
-    // temporary hack for unit testing
-    if (hint != HINT_INVALID) {
-        des->bits = 9;
-        des->stride = 1;
-    }
-
-    auto bucketToLits = assignStringsToBuckets(lits, *des);
-    addIncludedInfo(lits, des->getNumBuckets(), bucketToLits);
-    auto proto =
-        std::make_unique<HWLMProto>(engType, move(des), lits, bucketToLits,
-                                    make_small);
-    return proto;
-}
-
-unique_ptr<HWLMProto> fdrBuildProto(u8 engType, vector<hwlmLiteral> lits,
-                                    bool make_small, const target_t &target,
-                                    const Grey &grey) {
-    return fdrBuildProtoInternal(engType, lits, make_small, target, grey,
-                                 HINT_INVALID);
-}
-
-static
-bytecode_ptr<FDR> fdrBuildTableInternal(const HWLMProto &proto,
-                                        const Grey &grey) {
-
-    if (proto.teddyEng) {
-        return teddyBuildTable(proto, grey);
-    }
-
-    FDRCompiler fc(proto.lits, proto.bucketToLits, *(proto.fdrEng),
-                   proto.make_small, grey);
-    return fc.build();
-}
-
-bytecode_ptr<FDR> fdrBuildTable(const HWLMProto &proto, const Grey &grey) {
-    return fdrBuildTableInternal(proto, grey);
-}
-
-#if !defined(RELEASE_BUILD)
-
-unique_ptr<HWLMProto> fdrBuildProtoHinted(u8 engType,
-                                          vector<hwlmLiteral> lits,
-                                          bool make_small, u32 hint,
-                                          const target_t &target,
-                                          const Grey &grey) {
-    return fdrBuildProtoInternal(engType, lits, make_small, target, grey,
-                                 hint);
-}
-
-#endif
-
-size_t fdrSize(const FDR *fdr) {
-    assert(fdr);
-    return fdr->size;
-}
-
-} // namespace ue2

src/fdr/fdr_compile.h

@@ -1,72 +0,0 @@
-/*
- * Copyright (c) 2015-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief FDR literal matcher: build API.
- */
-
-#ifndef FDR_COMPILE_H
-#define FDR_COMPILE_H
-
-#include "ue2common.h"
-#include "hwlm/hwlm_build.h"
-#include "util/bytecode_ptr.h"
-
-#include <vector>
-
-struct FDR;
-
-namespace ue2 {
-
-struct hwlmLiteral;
-struct Grey;
-struct target_t;
-
-bytecode_ptr<FDR> fdrBuildTable(const HWLMProto &proto, const Grey &grey);
-
-#if !defined(RELEASE_BUILD)
-std::unique_ptr<HWLMProto> fdrBuildProtoHinted(
-                                          u8 engType,
-                                          std::vector<hwlmLiteral> lits,
-                                          bool make_small, u32 hint,
-                                          const target_t &target,
-                                          const Grey &grey);
-#endif
-
-std::unique_ptr<HWLMProto> fdrBuildProto(
-                                     u8 engType,
-                                     std::vector<hwlmLiteral> lits,
-                                     bool make_small, const target_t &target,
-                                     const Grey &grey);
-
-/** \brief Returns size in bytes of the given FDR engine. */
-size_t fdrSize(const struct FDR *fdr);
-
-} // namespace ue2
-
-#endif

src/fdr/fdr_compile_internal.h

@@ -1,87 +0,0 @@
-/*
- * Copyright (c) 2015-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef FDR_COMPILE_INTERNAL_H
-#define FDR_COMPILE_INTERNAL_H
-
-#include "ue2common.h"
-#include "hwlm/hwlm_literal.h"
-#include "util/bytecode_ptr.h"
-
-#include <map>
-#include <utility>
-#include <vector>
-
-struct FDRConfirm;
-struct LitInfo;
-
-namespace ue2 {
-
-// a pile of decorative typedefs
-// good for documentation purposes more than anything else
-typedef u32 LiteralIndex;
-typedef u32 SuffixPositionInString; // zero is last byte, counting back
-                                    // into the string
-typedef u32 BucketIndex;
-typedef u32 SchemeBitIndex;
-typedef u32 PositionInBucket;  // zero is 'we are matching right now!",
-                               // counting towards future matches
-
-class EngineDescription;
-class FDREngineDescription;
-struct hwlmStreamingControl;
-struct Grey;
-
-bytecode_ptr<u8> setupFullConfs(
-      const std::vector<hwlmLiteral> &lits,
-      const EngineDescription &eng,
-      const std::map<BucketIndex, std::vector<LiteralIndex>> &bucketToLits,
-      bool make_small);
-
-// all suffixes include an implicit max_bucket_width suffix to ensure that
-// we always read a full-scale flood "behind" us in terms of what's in our
-// state; if we don't have a flood that's long enough we won't be in the
-// right state yet to allow blindly advancing
-bytecode_ptr<u8> setupFDRFloodControl(const std::vector<hwlmLiteral> &lits,
-                                      const EngineDescription &eng,
-                                      const Grey &grey);
-
-bytecode_ptr<u8>
-fdrBuildTableStreaming(const std::vector<hwlmLiteral> &lits,
-                       hwlmStreamingControl &stream_control);
-
-static constexpr u32 HINT_INVALID = 0xffffffff;
-
-// fdr_compile_util.cpp utilities
-size_t maxLen(const std::vector<hwlmLiteral> &lits);
-size_t minLenCount(const std::vector<hwlmLiteral> &lits, size_t *count);
-u32 absdiff(u32 i, u32 j);
-
-} // namespace ue2
-
-#endif

src/fdr/fdr_compile_util.cpp

@@ -1,65 +0,0 @@
-/*
- * Copyright (c) 2015, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include "fdr_compile_internal.h"
-#include "hwlm/hwlm_literal.h"
-
-#include <algorithm>
-#include <vector>
-
-using namespace std;
-
-namespace ue2 {
-
-size_t maxLen(const vector<hwlmLiteral> &lits) {
-    size_t rv = 0;
-    for (const auto &lit : lits) {
-        rv = max(rv, lit.s.size());
-    }
-    return rv;
-}
-
-size_t minLenCount(const vector<hwlmLiteral> &lits, size_t *count) {
-    size_t rv = (size_t)-1;
-    *count = 0;
-    for (const auto &lit : lits) {
-        if (lit.s.size() < rv) {
-            rv = lit.s.size();
-            *count = 1;
-        } else if (lit.s.size() == rv) {
-            (*count)++;
-        }
-    }
-    return rv;
-}
-
-u32 absdiff(u32 i, u32 j) {
-    return (i > j) ? (i - j) : (j - i);
-}
-
-} // namespace ue2

src/fdr/fdr_confirm.h

@@ -1,94 +0,0 @@
-/*
- * Copyright (c) 2015-2019, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef FDR_CONFIRM_H
-#define FDR_CONFIRM_H
-
-#include "ue2common.h"
-#include "hwlm/hwlm.h"
-
-static really_inline
-u32 mul_hash_64(u64a lv, u64a andmsk, u64a mult, u32 nBits) {
-    return ((lv & andmsk) * mult) >> (sizeof(u64a)*8 - nBits);
-}
-
-// data structures
-// TODO: fix this hard-coding
-#define CONF_TYPE u64a
-#define CONF_HASH_CALL mul_hash_64
-
-/**
- * \brief Flag indicating this literal doesn't need to be delivered more than
- * once, used in LitInfo::flags.
- */
-#define FDR_LIT_FLAG_NOREPEAT   1
-
-/**
- * \brief Structure describing a literal, linked to by FDRConfirm.
- *
- * This structure is followed in memory by a variable-sized string prefix, for
- * strings that are longer than CONF_TYPE.
- */
-struct LitInfo {
-    CONF_TYPE v;
-    CONF_TYPE msk;
-    hwlm_group_t groups;
-    u32 id; // literal ID as passed in
-    u8 size;
-    u8 flags; //!< bitfield of flags from FDR_LIT_FLAG_* above.
-    u8 next;
-};
-
-#define FDRC_FLAG_NO_CONFIRM 1
-#define FDRC_FLAG_NOREPEAT   2
-
-/**
- * \brief FDR confirm header.
- *
- * This structure is followed in memory by:
- *
- * -# lit index mapping (array of u32)
- * -# list of LitInfo structures
- */
-struct FDRConfirm {
-    CONF_TYPE andmsk;
-    CONF_TYPE mult;
-    u32 nBits;
-    hwlm_group_t groups;
-};
-
-static really_inline
-const u32 *getConfirmLitIndex(const struct FDRConfirm *fdrc) {
-    const u8 *base = (const u8 *)fdrc;
-    const u32 *litIndex =
-        (const u32 *)(base + ROUNDUP_N(sizeof(*fdrc), alignof(u32)));
-    assert(ISALIGNED(litIndex));
-    return litIndex;
-}
-
-#endif // FDR_CONFIRM_H

src/fdr/fdr_confirm_compile.cpp

@@ -1,340 +0,0 @@
-/*
- * Copyright (c) 2015-2019, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include "fdr_internal.h"
-#include "fdr_compile_internal.h"
-#include "fdr_confirm.h"
-#include "engine_description.h"
-#include "teddy_engine_description.h"
-#include "ue2common.h"
-#include "util/alloc.h"
-#include "util/bitutils.h"
-#include "util/compare.h"
-#include "util/container.h"
-#include "util/verify_types.h"
-
-#include <algorithm>
-#include <cstring>
-#include <set>
-
-using namespace std;
-
-namespace ue2 {
-
-using BC2CONF = map<BucketIndex, bytecode_ptr<FDRConfirm>>;
-
-static
-u64a make_u64a_mask(const vector<u8> &v) {
-    assert(v.size() <= sizeof(u64a));
-    if (v.size() > sizeof(u64a)) {
-        throw std::exception();
-    }
-
-    u64a mask = 0;
-    size_t vlen = v.size();
-    size_t len = std::min(vlen, sizeof(mask));
-    unsigned char *m = (unsigned char *)&mask;
-    memcpy(m + sizeof(mask) - len, &v[vlen - len], len);
-    return mask;
-}
-
-/**
- * Build a temporary vector of LitInfo structures (without the corresponding
- * pointers to the actual strings; these cannot be laid out yet). These
- * stay in 1:1 correspondence with the lits[] vector as that's the only
- * place we have to obtain our full strings.
- */
-static
-void fillLitInfo(const vector<hwlmLiteral> &lits, vector<LitInfo> &tmpLitInfo,
-                 CONF_TYPE &andmsk) {
-    const CONF_TYPE all_ones = ~(u64a)0;
-    andmsk = all_ones; // fill in with 'and' of all literal masks
-
-    for (LiteralIndex i = 0; i < lits.size(); i++) {
-        const hwlmLiteral &lit = lits[i];
-        LitInfo &info = tmpLitInfo[i];
-        memset(&info, 0, sizeof(info));
-        info.id = lit.id;
-        u8 flags = 0;
-        if (lit.noruns) {
-            flags |= FDR_LIT_FLAG_NOREPEAT;
-        }
-        info.flags = flags;
-        info.size = verify_u8(max(lit.msk.size(), lit.s.size()));
-        info.groups = lit.groups;
-
-        // these are built up assuming a LE machine
-        CONF_TYPE msk = all_ones;
-        CONF_TYPE val = 0;
-        for (u32 j = 0; j < sizeof(CONF_TYPE); j++) {
-            u32 shiftLoc = (sizeof(CONF_TYPE) - j - 1) * 8;
-            if (j >= lit.s.size()) {
-                msk &= ~((CONF_TYPE)0xff << shiftLoc);
-            } else {
-                u8 c = lit.s[lit.s.size() - j - 1];
-                if (lit.nocase && ourisalpha(c)) {
-                    msk &= ~((CONF_TYPE)CASE_BIT << shiftLoc);
-                    val |= (CONF_TYPE)(c & CASE_CLEAR) << shiftLoc;
-                } else {
-                    val |= (CONF_TYPE)c << shiftLoc;
-                }
-            }
-        }
-
-        info.v = val;
-        info.msk = msk;
-        if (!lit.msk.empty()) {
-            u64a l_msk = make_u64a_mask(lit.msk);
-            u64a l_cmp = make_u64a_mask(lit.cmp);
-
-            // test for consistency - if there's intersection, then v and msk
-            // values must line up
-            UNUSED u64a intersection = l_msk & info.msk;
-            assert((info.v & intersection) == (l_cmp & intersection));
-
-            // incorporate lit.msk, lit.cmp into v and msk
-            info.msk |= l_msk;
-            info.v |= l_cmp;
-        }
-
-        andmsk &= info.msk;
-    }
-}
-
-//#define FDR_CONFIRM_DUMP 1
-
-static
-bytecode_ptr<FDRConfirm> getFDRConfirm(const vector<hwlmLiteral> &lits,
-                                       bool make_small) {
-    // Every literal must fit within CONF_TYPE.
-    assert(all_of_in(lits, [](const hwlmLiteral &lit) {
-        return lit.s.size() <= sizeof(CONF_TYPE);
-    }));
-
-    vector<LitInfo> tmpLitInfo(lits.size());
-    CONF_TYPE andmsk;
-    fillLitInfo(lits, tmpLitInfo, andmsk);
-
-#ifdef FDR_CONFIRM_DUMP
-    printf("-------------------\n");
-#endif
-
-    // just magic numbers and crude measures for now
-    u32 nBits;
-    if (make_small) {
-        nBits = min(10U, lg2(lits.size()) + 1);
-    } else {
-        nBits = lg2(lits.size()) + 4;
-    }
-
-    CONF_TYPE mult = (CONF_TYPE)0x0b4e0ef37bc32127ULL;
-
-    // we can walk the vector and assign elements from the vectors to a
-    // map by hash value
-    map<u32, vector<LiteralIndex> > res2lits;
-    hwlm_group_t gm = 0;
-    for (LiteralIndex i = 0; i < lits.size(); i++) {
-        LitInfo & li = tmpLitInfo[i];
-        u32 hash = CONF_HASH_CALL(li.v, andmsk, mult, nBits);
-        DEBUG_PRINTF("%016llx --> %u\n", li.v, hash);
-        res2lits[hash].emplace_back(i);
-        gm |= li.groups;
-    }
-
-#ifdef FDR_CONFIRM_DUMP
-    // print out the literals reversed - makes it easier to line up analyses
-    // that are end-offset based
-    for (const auto &m : res2lits) {
-        const u32 &hash = m.first;
-        const vector<LiteralIndex> &vlidx = m.second;
-        if (vlidx.size() <= 1) {
-            continue;
-        }
-        printf("%x -> %zu literals\n", hash, vlidx.size());
-        size_t min_len = lits[vlidx.front()].s.size();
-
-        vector<set<u8>> vsl; // contains the set of chars at each location
-                             // reversed from the end
-
-        for (const auto &litIdx : vlidx) {
-            const auto &lit = lits[litIdx];
-            if (lit.s.size() > vsl.size()) {
-                vsl.resize(lit.s.size());
-            }
-            for (size_t j = lit.s.size(); j != 0; j--) {
-                vsl[lit.s.size() - j].insert(lit.s[j - 1]);
-            }
-            min_len = min(min_len, lit.s.size());
-        }
-        printf("common     ");
-        for (size_t j = 0; j < min_len; j++) {
-            if (vsl[j].size() == 1) {
-                printf("%02x", *vsl[j].begin());
-            } else {
-                printf("__");
-            }
-        }
-        printf("\n");
-        for (const auto &litIdx : vlidx) {
-            const auto &lit = lits[litIdx];
-            printf("%8x  %c", lit.id, lit.nocase ? '!' : ' ');
-            for (size_t j = lit.s.size(); j != 0; j--) {
-                size_t dist_from_end = lit.s.size() - j;
-                if (dist_from_end < min_len && vsl[dist_from_end].size() == 1) {
-                    printf("__");
-                } else {
-                    printf("%02x", lit.s[j - 1]);
-                }
-            }
-            printf("\n");
-        }
-        size_t total_compares = 0;
-        for (const auto &v : vsl) {
-            total_compares += v.size();
-        }
-        size_t total_string_size = 0;
-        for (const auto &litIdx : vlidx) {
-            const auto &lit = lits[litIdx];
-            total_string_size += lit.s.size();
-        }
-        printf("Total compare load: %zu Total string size: %zu\n\n",
-               total_compares, total_string_size);
-    }
-#endif
-
-    const size_t bitsToLitIndexSize = (1U << nBits) * sizeof(u32);
-
-    // this size can now be a worst-case as we can always be a bit smaller
-    size_t size = ROUNDUP_N(sizeof(FDRConfirm), alignof(u32)) +
-                  ROUNDUP_N(bitsToLitIndexSize, alignof(LitInfo)) +
-                  sizeof(LitInfo) * lits.size();
-    size = ROUNDUP_N(size, alignof(FDRConfirm));
-
-    auto fdrc = make_zeroed_bytecode_ptr<FDRConfirm>(size);
-    assert(fdrc); // otherwise would have thrown std::bad_alloc
-
-    fdrc->andmsk = andmsk;
-    fdrc->mult = mult;
-    fdrc->nBits = nBits;
-
-    fdrc->groups = gm;
-
-    // After the FDRConfirm, we have the lit index array.
-    u8 *fdrc_base = (u8 *)fdrc.get();
-    u8 *ptr = fdrc_base + sizeof(*fdrc);
-    ptr = ROUNDUP_PTR(ptr, alignof(u32));
-    u32 *bitsToLitIndex = (u32 *)ptr;
-    ptr += bitsToLitIndexSize;
-
-    // After the lit index array, we have the LitInfo structures themselves,
-    // which vary in size (as each may have a variable-length string after it).
-    ptr = ROUNDUP_PTR(ptr, alignof(LitInfo));
-
-    // Walk the map by hash value assigning indexes and laying out the
-    // elements (and their associated string confirm material) in memory.
-    for (const auto &m : res2lits) {
-        const u32 hash = m.first;
-        const vector<LiteralIndex> &vlidx = m.second;
-        bitsToLitIndex[hash] = verify_u32(ptr - fdrc_base);
-        for (auto i = vlidx.begin(), e = vlidx.end(); i != e; ++i) {
-            LiteralIndex litIdx = *i;
-
-            // Write LitInfo header.
-            LitInfo &finalLI = *(LitInfo *)ptr;
-            finalLI = tmpLitInfo[litIdx];
-
-            ptr += sizeof(LitInfo); // String starts directly after LitInfo.
-            assert(lits[litIdx].s.size() <= sizeof(CONF_TYPE));
-            if (next(i) == e) {
-                finalLI.next = 0;
-            } else {
-                finalLI.next = 1;
-            }
-        }
-        assert((size_t)(ptr - fdrc_base) <= size);
-    }
-
-    // Return actual used size, not worst-case size. Must be rounded up to
-    // FDRConfirm alignment so that the caller can lay out a sequence of these.
-    size_t actual_size = ROUNDUP_N((size_t)(ptr - fdrc_base),
-                                   alignof(FDRConfirm));
-    assert(actual_size <= size);
-    fdrc.shrink(actual_size);
-
-    return fdrc;
-}
-
-bytecode_ptr<u8>
-setupFullConfs(const vector<hwlmLiteral> &lits,
-               const EngineDescription &eng,
-               const map<BucketIndex, vector<LiteralIndex>> &bucketToLits,
-               bool make_small) {
-    unique_ptr<TeddyEngineDescription> teddyDescr =
-        getTeddyDescription(eng.getID());
-
-    BC2CONF bc2Conf;
-    u32 totalConfirmSize = 0;
-    for (BucketIndex b = 0; b < eng.getNumBuckets(); b++) {
-        if (contains(bucketToLits, b)) {
-            vector<hwlmLiteral> vl;
-            for (const LiteralIndex &lit_idx : bucketToLits.at(b)) {
-                vl.emplace_back(lits[lit_idx]);
-            }
-
-            DEBUG_PRINTF("b %d sz %zu\n", b, vl.size());
-            auto fc = getFDRConfirm(vl, make_small);
-            totalConfirmSize += fc.size();
-            bc2Conf.emplace(b, move(fc));
-        }
-    }
-
-    u32 nBuckets = eng.getNumBuckets();
-    u32 totalConfSwitchSize = ROUNDUP_CL(nBuckets * sizeof(u32));
-    u32 totalSize = totalConfSwitchSize + totalConfirmSize;
-
-    auto buf = make_zeroed_bytecode_ptr<u8>(totalSize, 64);
-    assert(buf); // otherwise would have thrown std::bad_alloc
-
-    u32 *confBase = (u32 *)buf.get();
-    u8 *ptr = buf.get() + totalConfSwitchSize;
-    assert(ISALIGNED_CL(ptr));
-
-    for (const auto &m : bc2Conf) {
-        const BucketIndex &idx = m.first;
-        const bytecode_ptr<FDRConfirm> &p = m.second;
-        // confirm offset is relative to the base of this structure, now
-        u32 confirm_offset = verify_u32(ptr - buf.get());
-        memcpy(ptr, p.get(), p.size());
-        ptr += p.size();
-        confBase[idx] = confirm_offset;
-    }
-
-    return buf;
-}
-
-} // namespace ue2

src/fdr/fdr_confirm_runtime.h

@@ -1,104 +0,0 @@
-/*
- * Copyright (c) 2015-2019, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef FDR_CONFIRM_RUNTIME_H
-#define FDR_CONFIRM_RUNTIME_H
-
-#include "scratch.h"
-#include "fdr_internal.h"
-#include "fdr_loadval.h"
-#include "hwlm/hwlm.h"
-#include "ue2common.h"
-#include "util/bitutils.h"
-#include "util/compare.h"
-
-// this is ordinary confirmation function which runs through
-// the whole confirmation procedure
-static really_inline
-void confWithBit(const struct FDRConfirm *fdrc, const struct FDR_Runtime_Args *a,
-                 size_t i, hwlmcb_rv_t *control, u32 *last_match,
-                 u64a conf_key, u64a *conf, u8 bit) {
-    assert(i < a->len);
-    assert(i >= a->start_offset);
-    assert(ISALIGNED(fdrc));
-
-    const u8 * buf = a->buf;
-    u32 c = CONF_HASH_CALL(conf_key, fdrc->andmsk, fdrc->mult,
-                           fdrc->nBits);
-    u32 start = getConfirmLitIndex(fdrc)[c];
-    if (likely(!start)) {
-        return;
-    }
-
-    const struct LitInfo *li
-        = (const struct LitInfo *)((const u8 *)fdrc + start);
-
-    struct hs_scratch *scratch = a->scratch;
-    assert(!scratch->fdr_conf);
-    scratch->fdr_conf = conf;
-    scratch->fdr_conf_offset = bit;
-    u8 oldNext; // initialized in loop
-    do {
-        assert(ISALIGNED(li));
-
-        if (unlikely((conf_key & li->msk) != li->v)) {
-            goto out;
-        }
-
-        if ((*last_match == li->id) && (li->flags & FDR_LIT_FLAG_NOREPEAT)) {
-            goto out;
-        }
-
-        const u8 *loc = buf + i - li->size + 1;
-
-        if (loc < buf) {
-            u32 full_overhang = buf - loc;
-            size_t len_history = a->len_history;
-
-            // can't do a vectored confirm either if we don't have
-            // the bytes
-            if (full_overhang > len_history) {
-                goto out;
-            }
-        }
-        assert(li->size <= sizeof(CONF_TYPE));
-
-        if (unlikely(!(li->groups & *control))) {
-            goto out;
-        }
-
-        *last_match = li->id;
-        *control = a->cb(i, li->id, scratch);
-    out:
-        oldNext = li->next; // oldNext is either 0 or an 'adjust' value
-        li++;
-    } while (oldNext);
-    scratch->fdr_conf = NULL;
-}
-
-#endif

src/fdr/fdr_dump.cpp

@@ -1,210 +0,0 @@
-/*
- * Copyright (c) 2015-2020, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include "config.h"
-
-#include "fdr_compile.h"
-#include "fdr_compile_internal.h"
-#include "fdr_confirm.h"
-#include "fdr_dump.h"
-#include "fdr_engine_description.h"
-#include "fdr_internal.h"
-#include "teddy_engine_description.h"
-#include "teddy_internal.h"
-#include "ue2common.h"
-
-#include <cstdio>
-#include <memory>
-
-#ifndef DUMP_SUPPORT
-#error No dump support!
-#endif
-
-using namespace std;
-
-namespace ue2 {
-
-static
-bool fdrIsTeddy(const FDR *fdr) {
-    assert(fdr);
-    u32 engine = fdr->engineID;
-
-    /* teddys don't have an fdr engine description (which is why the dump code
-     * is so broken). */
-
-    return !getFdrDescription(engine);
-}
-
-static
-void dumpLitIndex(const FDRConfirm *fdrc, FILE *f) {
-    const u32 *lit_index = getConfirmLitIndex(fdrc);
-    u32 num_lits = 1U << fdrc->nBits;
-    u32 lits_used = count_if(lit_index, lit_index + num_lits,
-                             [](u32 idx) { return idx != 0; });
-
-    fprintf(f, "      load    %u/%u (%0.2f%%)\n", lits_used, num_lits,
-            (double)lits_used / (double)(num_lits)*100);
-}
-
-static
-void dumpConfirms(const void *fdr_base, u32 conf_offset, u32 num_confirms,
-                  FILE *f) {
-    const u32 *conf = (const u32 *)((const char *)fdr_base + conf_offset);
-    for (u32 i = 0; i < num_confirms; i++) {
-        const auto *fdrc = (const FDRConfirm *)((const char *)conf + conf[i]);
-        fprintf(f, "    confirm %u\n", i);
-        fprintf(f, "      andmsk  0x%016llx\n", fdrc->andmsk);
-        fprintf(f, "      mult    0x%016llx\n", fdrc->mult);
-        fprintf(f, "      nbits   %u\n", fdrc->nBits);
-        fprintf(f, "      groups  0x%016llx\n", fdrc->groups);
-        dumpLitIndex(fdrc, f);
-    }
-}
-
-static
-void dumpTeddyReinforced(const u8 *rmsk, const u32 num_tables, FILE *f) {
-    // dump reinforcement masks
-    for (u32 b = 0; b < num_tables; b++) {
-        fprintf(f, "    reinforcement table for bucket %u..%u:\n",
-                   b * 8, b * 8 + 7);
-        for (u32 i = 0; i <= N_CHARS; i++) {
-            fprintf(f, "      0x%02x: ", i);
-            for (u32 j = 0; j < 8; j++) {
-                u8 val = rmsk[b * ((N_CHARS + 1) * 8) + i * 8 + j];
-                for (u32 k = 0; k < 8; k++) {
-                    fprintf(f, "%s", ((val >> k) & 0x1) ? "1" : "0");
-                }
-                fprintf(f, " ");
-            }
-            fprintf(f, "\n");
-        }
-        fprintf(f, "\n");
-    }
-}
-
-static
-void dumpTeddyDupMasks(const u8 *dmsk, u32 numMasks, FILE *f) {
-    // dump nibble masks
-    u32 maskWidth = 2;
-    fprintf(f, "    dup nibble masks:\n");
-    for (u32 i = 0; i < numMasks * 2; i++) {
-        fprintf(f, "      -%d%s: ", 1 + i / 2, (i % 2) ? "hi" : "lo");
-        for (u32 j = 0; j < 16 * maskWidth * 2; j++) {
-            u8 val = dmsk[i * 16 * maskWidth * 2 + j];
-            for (u32 k = 0; k < 8; k++) {
-                fprintf(f, "%s", ((val >> k) & 0x1) ? "1" : "0");
-            }
-            fprintf(f, " ");
-        }
-        fprintf(f, "\n");
-    }
-    fprintf(f, "\n");
-}
-
-static
-void dumpTeddyMasks(const u8 *baseMsk, u32 numMasks, u32 maskWidth, FILE *f) {
-    // dump nibble masks
-    fprintf(f, "    nibble masks:\n");
-    for (u32 i = 0; i < numMasks * 2; i++) {
-        fprintf(f, "      -%d%s: ", 1 + i / 2, (i % 2) ? "hi" : "lo");
-        for (u32 j = 0; j < 16 * maskWidth; j++) {
-            u8 val = baseMsk[i * 16 * maskWidth + j];
-            for (u32 k = 0; k < 8; k++) {
-                fprintf(f, "%s", ((val >> k) & 0x1) ? "1" : "0");
-            }
-            fprintf(f, " ");
-        }
-        fprintf(f, "\n");
-    }
-    fprintf(f, "\n");
-}
-
-static
-void dumpTeddy(const Teddy *teddy, FILE *f) {
-    fprintf(f, "TEDDY:         %u\n", teddy->engineID);
-    auto des = getTeddyDescription(teddy->engineID);
-    if (!des) {
-        fprintf(f, "   <unknown engine>\n");
-        return;
-    }
-
-    fprintf(f, "    masks      %u\n", des->numMasks);
-    fprintf(f, "    buckets    %u\n", des->getNumBuckets());
-    fprintf(f, "    packed     %s\n", des->packed ? "true" : "false");
-    fprintf(f, "    strings    %u\n", teddy->numStrings);
-    fprintf(f, "    size       %zu bytes\n", fdrSize((const FDR *)teddy));
-    fprintf(f, "    max length %u\n", teddy->maxStringLen);
-    fprintf(f, "    floodoff   %u (%x)\n", teddy->floodOffset,
-            teddy->floodOffset);
-    fprintf(f, "\n");
-
-    u32 maskWidth = des->getNumBuckets() / 8;
-    size_t headerSize = sizeof(Teddy);
-    const u8 *teddy_base = (const u8 *)teddy;
-    const u8 *baseMsk = teddy_base + ROUNDUP_CL(headerSize);
-    dumpTeddyMasks(baseMsk, des->numMasks, maskWidth, f);
-    size_t maskLen = des->numMasks * 16 * 2 * maskWidth;
-    const u8 *rdmsk = baseMsk + ROUNDUP_CL(maskLen);
-    if (maskWidth == 1) { // reinforcement table in Teddy
-        dumpTeddyReinforced(rdmsk, maskWidth, f);
-    } else { // dup nibble mask table in Fat Teddy
-        assert(maskWidth == 2);
-        dumpTeddyDupMasks(rdmsk, des->numMasks, f);
-    }
-    dumpConfirms(teddy, teddy->confOffset, des->getNumBuckets(), f);
-}
-
-static
-void dumpFDR(const FDR *fdr, FILE *f) {
-    fprintf(f, "FDR:           %u\n", fdr->engineID);
-    auto des = getFdrDescription(fdr->engineID);
-    if (!des) {
-        fprintf(f, "   <unknown engine>\n");
-        return;
-    }
-
-    fprintf(f, "    domain     %u\n", fdr->domain);
-    fprintf(f, "    stride     %u\n", fdr->stride);
-    fprintf(f, "    strings    %u\n", fdr->numStrings);
-    fprintf(f, "    size       %zu bytes\n", fdrSize(fdr));
-    fprintf(f, "    max length %u\n", fdr->maxStringLen);
-    fprintf(f, "    floodoff   %u (%x)\n", fdr->floodOffset, fdr->floodOffset);
-    fprintf(f, "\n");
-
-    dumpConfirms(fdr, fdr->confOffset, des->getNumBuckets(), f);
-}
-
-void fdrPrintStats(const FDR *fdr, FILE *f) {
-    if (fdrIsTeddy(fdr)) {
-        dumpTeddy((const Teddy *)fdr, f);
-    } else {
-        dumpFDR(fdr, f);
-    }
-}
-
-} // namespace ue2

src/fdr/fdr_dump.h

@@ -1,49 +0,0 @@
-/*
- * Copyright (c) 2015, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief FDR literal matcher: dump API.
- */
-
-#ifndef FDR_DUMP_H
-#define FDR_DUMP_H
-
-#if defined(DUMP_SUPPORT)
-
-#include <cstdio>
-
-struct FDR;
-
-namespace ue2 {
-
-void fdrPrintStats(const struct FDR *fdr, FILE *f);
-
-} // namespace ue2
-
-#endif // DUMP_SUPPORT
-#endif // FDR_DUMP_H

src/fdr/fdr_engine_description.cpp

@@ -1,227 +0,0 @@
-/*
- * Copyright (c) 2015-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include "fdr_compile_internal.h"
-#include "fdr_engine_description.h"
-#include "hs_compile.h"
-#include "util/target_info.h"
-#include "util/compare.h" // for ourisalpha()
-
-#include <cassert>
-#include <cstdlib>
-#include <map>
-#include <string>
-
-using namespace std;
-
-namespace ue2 {
-
-FDREngineDescription::FDREngineDescription(const FDREngineDef &def)
-    : EngineDescription(def.id, targetByArchFeatures(def.cpu_features),
-                        def.numBuckets),
-      schemeWidth(def.schemeWidth), stride(0), bits(0) {}
-
-u32 FDREngineDescription::getDefaultFloodSuffixLength() const {
-    // rounding up, so that scheme width 32 and 6 buckets is 6 not 5!
-    // the +1 avoids pain due to various reach choices
-    return ((getSchemeWidth() + getNumBuckets() - 1) / getNumBuckets()) + 1;
-}
-
-void getFdrDescriptions(vector<FDREngineDescription> *out) {
-    static const FDREngineDef def = {0, 64, 8, 0};
-    out->clear();
-    out->emplace_back(def);
-}
-
-static
-u32 findDesiredStride(size_t num_lits, size_t min_len, size_t min_len_count) {
-    u32 desiredStride = 1; // always our safe fallback
-    if (min_len > 1) {
-        if (num_lits < 250) {
-            // small cases we just go for it
-            desiredStride = min_len;
-        } else if (num_lits < 800) {
-            // intermediate cases
-            desiredStride = min_len - 1;
-        } else if (num_lits < 5000) {
-            // for larger but not huge sizes, go to stride 2 only if we have at
-            // least minlen 3
-            desiredStride = MIN(min_len - 1, 2);
-        }
-    }
-
-    // patch if count is quite large - a ton of length 2 literals can
-    // break things
-#ifdef TRY_THIS_LATER
-    if ((min_len == 2) && (desiredStride == 2) && (min_len_count > 20)) {
-        desiredStride = 1;
-    }
-#endif
-
-    // patch stuff just for the stride 4 case; don't let min_len=4,
-    // desiredStride=4 through as even a few length 4 literals can break things
-    // (far more fragile)
-    if ((min_len == 4) && (desiredStride == 4) && (min_len_count > 2)) {
-        desiredStride = 2;
-    }
-
-    return desiredStride;
-}
-
-unique_ptr<FDREngineDescription> chooseEngine(const target_t &target,
-                                              const vector<hwlmLiteral> &vl,
-                                              bool make_small) {
-    vector<FDREngineDescription> allDescs;
-    getFdrDescriptions(&allDescs);
-
-    // find desired stride
-    size_t count;
-    size_t msl = minLenCount(vl, &count);
-    u32 desiredStride = findDesiredStride(vl.size(), msl, count);
-
-    DEBUG_PRINTF("%zu lits, msl=%zu, desiredStride=%u\n", vl.size(), msl,
-                 desiredStride);
-
-    FDREngineDescription *best = nullptr;
-    u32 best_score = 0;
-
-    FDREngineDescription &eng = allDescs[0];
-
-    for (u32 domain = 9; domain <= 15; domain++) {
-        for (size_t stride = 1; stride <= 4; stride *= 2) {
-            // to make sure that domains >=14 have stride 1 according to origin
-            if (domain > 13 && stride > 1) {
-                continue;
-            }
-            if (!eng.isValidOnTarget(target)) {
-                continue;
-            }
-            if (msl < stride) {
-                continue;
-            }
-
-            u32 score = 100;
-
-            score -= absdiff(desiredStride, stride);
-
-            if (stride <= desiredStride) {
-                score += stride;
-            }
-
-            u32 effLits = vl.size(); /* * desiredStride;*/
-            u32 ideal;
-            if (effLits < eng.getNumBuckets()) {
-                if (stride == 1) {
-                    ideal = 8;
-                } else {
-                    ideal = 10;
-                }
-            } else if (effLits < 20) {
-                ideal = 10;
-            } else if (effLits < 100) {
-                ideal = 11;
-            } else if (effLits < 1000) {
-                ideal = 12;
-            } else if (effLits < 10000) {
-                ideal = 13;
-            } else {
-                ideal = 15;
-            }
-
-            if (ideal != 8 && eng.schemeWidth == 32) {
-                ideal += 1;
-            }
-
-            if (make_small) {
-                ideal -= 2;
-            }
-
-            if (stride > 1) {
-                ideal++;
-            }
-
-            DEBUG_PRINTF("effLits %u\n", effLits);
-
-            if (target.is_atom_class() && !make_small && effLits < 4000) {
-                /* Unless it is a very heavy case, we want to build smaller
-                 * tables on lightweight machines due to their small caches. */
-                ideal -= 2;
-            }
-
-            score -= absdiff(ideal, domain);
-
-            DEBUG_PRINTF("fdr %u: width=%u, domain=%u, buckets=%u, stride=%zu "
-                         "-> score=%u\n",
-                         eng.getID(), eng.schemeWidth, domain,
-                         eng.getNumBuckets(), stride, score);
-
-            if (!best || score > best_score) {
-                eng.bits = domain;
-                eng.stride = stride;
-                best = &eng;
-                best_score = score;
-            }
-        }
-    }
-
-    if (!best) {
-        DEBUG_PRINTF("failed to find engine\n");
-        return nullptr;
-    }
-
-    DEBUG_PRINTF("using engine %u\n", best->getID());
-    return std::make_unique<FDREngineDescription>(*best);
-}
-
-SchemeBitIndex FDREngineDescription::getSchemeBit(BucketIndex b,
-                                                  PositionInBucket p) const {
-    assert(p < getBucketWidth(b));
-    SchemeBitIndex sbi = p * getNumBuckets() + b;
-    assert(sbi < getSchemeWidth());
-    return sbi;
-}
-
-u32 FDREngineDescription::getBucketWidth(BucketIndex) const {
-    u32 sw = getSchemeWidth();
-    u32 nm = getNumBuckets();
-    assert(sw % nm == 0);
-    return sw/nm;
-}
-
-unique_ptr<FDREngineDescription> getFdrDescription(u32 engineID) {
-    vector<FDREngineDescription> allDescs;
-    getFdrDescriptions(&allDescs);
-
-    if (engineID >= allDescs.size()) {
-        return nullptr;
-    }
-
-    return std::make_unique<FDREngineDescription>(allDescs[engineID]);
-}
-
-} // namespace ue2

src/fdr/fdr_engine_description.h

@@ -1,73 +0,0 @@
-/*
- * Copyright (c) 2015-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef FDR_ENGINE_DESCRIPTION_H
-#define FDR_ENGINE_DESCRIPTION_H
-
-#include "engine_description.h"
-
-#include <map>
-#include <memory>
-#include <vector>
-
-namespace ue2 {
-
-struct FDREngineDef {
-    u32 id;
-    u32 schemeWidth;
-    u32 numBuckets;
-    u64a cpu_features;
-};
-
-class FDREngineDescription : public EngineDescription {
-public:
-    u32 schemeWidth;
-    u32 stride;
-    u32 bits;
-
-    u32 getSchemeWidth() const { return schemeWidth; }
-    u32 getBucketWidth(BucketIndex b) const;
-    SchemeBitIndex getSchemeBit(BucketIndex b, PositionInBucket p) const;
-    u32 getNumTableEntries() const { return 1 << bits; }
-    u32 getTabSizeBytes() const {
-        return schemeWidth / 8 * getNumTableEntries();
-    }
-
-    explicit FDREngineDescription(const FDREngineDef &def);
-
-    u32 getDefaultFloodSuffixLength() const override;
-};
-
-std::unique_ptr<FDREngineDescription>
-chooseEngine(const target_t &target, const std::vector<hwlmLiteral> &vl,
-             bool make_small);
-std::unique_ptr<FDREngineDescription> getFdrDescription(u32 engineID);
-void getFdrDescriptions(std::vector<FDREngineDescription> *out);
-} // namespace ue2
-
-#endif

src/fdr/fdr_internal.h

@@ -1,105 +0,0 @@
-/*
- * Copyright (c) 2015-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief FDR literal matcher: data structures.
- */
-
-#ifndef FDR_INTERNAL_H
-#define FDR_INTERNAL_H
-
-#include "ue2common.h"
-#include "hwlm/hwlm.h" // for hwlm_group_t, HWLMCallback
-
-struct hs_scratch;
-
-typedef enum {
-    NOT_CAUTIOUS, //!< not near a boundary (quantify?)
-    VECTORING     //!< potentially vectoring
-} CautionReason;
-
-/** \brief number of different ids that can be triggered by floods of any given
- * character. */
-#define FDR_FLOOD_MAX_IDS 16
-
-struct FDRFlood {
-    hwlm_group_t allGroups; //!< all the groups or'd together
-    u32 suffix;
-
-    /** \brief 0 to FDR_FLOOD_MAX_IDS-1 ids that are generated once per char on
-     * a flood.
-     * If larger we won't handle this through the flood path at all. */
-    u16 idCount;
-
-    u32 ids[FDR_FLOOD_MAX_IDS]; //!< the ids
-    hwlm_group_t groups[FDR_FLOOD_MAX_IDS]; //!< group ids to go with string ids
-};
-
-/** \brief FDR structure.
- *
- * 1. struct as-is
- * 2. primary matching table
- * 3. confirm stuff
- */
-struct FDR {
-    u32 engineID;
-    u32 size;
-    u32 maxStringLen;
-    u32 numStrings;
-    u32 confOffset;
-    u32 floodOffset;
-    u8 stride; /* stride - how frequently the data is consulted by the first
-                * stage matcher */
-    u8 domain; /* number of bits used to index into main FDR table. This value
-                * is used only of debugging/asserts. */
-    u16 domainMask; /* pre-computed domain mask */
-    u32 tabSize; /* pre-computed hashtable size in bytes */
-    m128 start; /* initial start state to use at offset 0. The state has been
-                 * set up based on the min length of buckets to reduce the need
-                 * for pointless confirms. */
-};
-
-/** \brief FDR runtime arguments.
- *
- * This structure handles read-only things that are passed extensively around
- * the FDR run-time functions. They are set by the API, passed by value into
- * the main function, then a pointer is passed around to all the various
- * sub-functions (confirm & flood). */
-struct FDR_Runtime_Args {
-    const u8 *buf;
-    size_t len;
-    const u8 *buf_history;
-    size_t len_history;
-    size_t start_offset;
-    HWLMCallback cb;
-    struct hs_scratch *scratch;
-    const u8 *firstFloodDetect;
-    const u64a histBytes;
-};
-
-#endif

src/fdr/fdr_loadval.h

@@ -1,71 +0,0 @@
-/*
- * Copyright (c) 2015-2016, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef FDR_LOADVAL_H
-#define FDR_LOADVAL_H
-
-#include "ue2common.h"
-#include "util/unaligned.h"
-
-#define MAKE_LOADVAL(type, name)                \
-    static really_inline                                                \
-    type name(const u8 *ptr, UNUSED const u8 *lo, UNUSED const u8 *hi)
-
-#define NORMAL_SAFE(type)                                               \
-    do {                                                                \
-        assert(ptr >= lo);                                              \
-        assert(ptr + sizeof(type) - 1 < hi);                            \
-    } while(0)
-
-#define MAKE_LOOP_CE(TYPE)                                              \
-    TYPE v = 0;                                                         \
-    for (TYPE i = 0; i < sizeof(TYPE); i++) {                           \
-        if ((lo <= ptr + i) && (ptr + i < hi)) {                        \
-            v += (TYPE)ptr[i] << (i*8);                                 \
-        }                                                               \
-    }                                                                   \
-    return v;
-
-// no suffix = normal (unaligned)
-// _ce       = cautious everywhere (in both directions); test against hi and lo
-
-MAKE_LOADVAL(u16, lv_u16) {
-    NORMAL_SAFE(u16);
-    return unaligned_load_u16(ptr);
-}
-
-MAKE_LOADVAL(u64a, lv_u64a) {
-    NORMAL_SAFE(u32);
-    return unaligned_load_u64a(ptr);
-}
-
-MAKE_LOADVAL(u16, lv_u16_ce) { MAKE_LOOP_CE(u16); }
-
-MAKE_LOADVAL(u64a, lv_u64a_ce) { MAKE_LOOP_CE(u64a); }
-
-#endif

src/fdr/flood_compile.cpp

@@ -1,231 +0,0 @@
-/*
- * Copyright (c) 2015-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include "fdr_internal.h"
-#include "fdr_confirm.h"
-#include "fdr_compile_internal.h"
-#include "fdr_engine_description.h"
-#include "grey.h"
-#include "ue2common.h"
-#include "util/alloc.h"
-#include "util/bitutils.h"
-#include "util/charreach.h"
-#include "util/compare.h"
-#include "util/ue2string.h"
-#include "util/verify_types.h"
-
-#include <cstring>
-#include <map>
-#include <memory>
-#include <string>
-#include <vector>
-
-using namespace std;
-
-namespace ue2 {
-
-namespace {
-struct FloodComparator {
-    bool operator()(const FDRFlood &f1, const FDRFlood &f2) const {
-        return std::memcmp(&f1, &f2, sizeof(f1)) < 0;
-    }
-};
-}
-
-static
-bool isDifferent(u8 oldC, u8 c, bool caseless) {
-    if (caseless) {
-        return mytolower(oldC) != mytolower(c);
-    } else {
-        return oldC != c;
-    }
-}
-
-static
-void updateFloodSuffix(vector<FDRFlood> &tmpFlood, u8 c, u32 suffix) {
-    FDRFlood &fl = tmpFlood[c];
-    fl.suffix = MAX(fl.suffix, suffix + 1);
-    DEBUG_PRINTF("Updated Flood Suffix for char 0x%02x to %u\n", c, fl.suffix);
-}
-
-static
-void addFlood(vector<FDRFlood> &tmpFlood, u8 c, const hwlmLiteral &lit,
-              u32 suffix) {
-    FDRFlood &fl = tmpFlood[c];
-    fl.suffix = MAX(fl.suffix, suffix + 1);
-    if (fl.idCount < FDR_FLOOD_MAX_IDS) {
-        fl.ids[fl.idCount] = lit.id;
-        fl.allGroups |= lit.groups;
-        fl.groups[fl.idCount] = lit.groups;
-        // when idCount gets to max_ids this flood no longer happens
-        // only incremented one more time to avoid arithmetic overflow
-        DEBUG_PRINTF("Added Flood for char '%c' suffix=%u len[%hu]=%u\n",
-                     c, fl.suffix, fl.idCount, suffix);
-        fl.idCount++;
-   }
-}
-
-bytecode_ptr<u8> setupFDRFloodControl(const vector<hwlmLiteral> &lits,
-                                      const EngineDescription &eng,
-                                      const Grey &grey) {
-    vector<FDRFlood> tmpFlood(N_CHARS);
-    u32 default_suffix = eng.getDefaultFloodSuffixLength();
-
-    // zero everything to avoid spurious distinctions in the compares
-    memset(&tmpFlood[0], 0, N_CHARS * sizeof(FDRFlood));
-
-    for (u32 c = 0; c < N_CHARS; c++) {
-        tmpFlood[c].suffix = default_suffix;
-    }
-
-    for (const auto &lit : lits) {
-        DEBUG_PRINTF("lit: '%s'%s\n", escapeString(lit.s).c_str(),
-                     lit.nocase ? " (nocase)" : "");
-        u32 litSize = verify_u32(lit.s.size());
-        u32 maskSize = (u32)lit.msk.size();
-        u8 c = lit.s[litSize - 1];
-        bool nocase = ourisalpha(c) ? lit.nocase : false;
-
-        if (nocase && maskSize && (lit.msk[maskSize - 1] & CASE_BIT)) {
-            c = (lit.cmp[maskSize - 1] & CASE_BIT) ? mytolower(c) : mytoupper(c);
-            nocase = false;
-        }
-
-        u32 iEnd = MAX(litSize, maskSize);
-        u32 upSuffix = iEnd; // upSuffix is used as an upper case suffix length
-                             // for case-less, or as a suffix length for case-sensitive;
-        u32 loSuffix = iEnd; // loSuffix used only for case-less as a lower case suffix
-                             // length;
-
-        for (u32 i = 0; i < iEnd; i++) {
-            if (i < litSize) {
-                if (isDifferent(c, lit.s[litSize - i - 1], lit.nocase)) {
-                    DEBUG_PRINTF("non-flood char in literal[%u]: "
-                                 "0x%02x != 0x%02x\n",
-                                 i, c, lit.s[litSize - i - 1]);
-                    upSuffix = MIN(upSuffix, i);
-                    loSuffix = MIN(loSuffix, i); // makes sense only for case-less
-                    break;
-                }
-            }
-            if (i < maskSize) {
-                u8 m = lit.msk[maskSize - i - 1];
-                u8 cm = lit.cmp[maskSize - i - 1] & m;
-                if(nocase) {
-                    if ((mytoupper(c) & m) != cm) {
-                        DEBUG_PRINTF("non-flood char in mask[%u] %c != %c\n",
-                                                            i, mytoupper(c), cm);
-                        upSuffix = MIN(upSuffix, i);
-                    }
-                    if ((mytolower(c) & m) != cm) {
-                        DEBUG_PRINTF("non-flood char in mask[%u] %c != %c\n",
-                                                            i, mytolower(c), cm);
-                        loSuffix = MIN(loSuffix, i);
-                    }
-                    if (loSuffix != iEnd && upSuffix != iEnd) {
-                        break;
-                    }
-                } else if ((c & m) != cm) {
-                    DEBUG_PRINTF("non-flood char in mask[%u] %c != %c\n", i, c, cm);
-                    upSuffix = MIN(upSuffix, i);
-                    break;
-                }
-            }
-        }
-        if(upSuffix != iEnd) {
-            updateFloodSuffix(tmpFlood, nocase ? mytoupper(c) : c, upSuffix);
-        } else {
-            addFlood(tmpFlood, nocase ? mytoupper(c) : c, lit, upSuffix);
-        }
-        if (nocase) {
-            if(loSuffix != iEnd) {
-                updateFloodSuffix(tmpFlood, mytolower(c), loSuffix);
-            } else {
-                addFlood(tmpFlood, mytolower(c), lit, loSuffix);
-            }
-        }
-    }
-
-#ifdef DEBUG
-    for (u32 i = 0; i < N_CHARS; i++) {
-        FDRFlood &fl = tmpFlood[i];
-        if (!fl.idCount) {
-            continue;
-        }
-
-        printf("i is %02x fl->idCount is %hd fl->suffix is %d fl->allGroups is "
-               "%016llx\n", i, fl.idCount, fl.suffix, fl.allGroups);
-        for (u32 j = 0; j < fl.idCount; j++) {
-            printf("j is %d fl.groups[j] %016llx\n", j, fl.groups[j]);
-        }
-    }
-#endif
-
-    // If flood detection has been switched off in the grey box, we comply by
-    // setting idCount too high for all floods.
-    if (!grey.fdrAllowFlood) {
-        for (auto &fl : tmpFlood) {
-            fl.idCount = FDR_FLOOD_MAX_IDS;
-        }
-    }
-
-    map<FDRFlood, CharReach, FloodComparator> flood2chars;
-    for (u32 i = 0; i < N_CHARS; i++) {
-        FDRFlood fl = tmpFlood[i];
-        flood2chars[fl].set(i);
-    }
-
-    u32 nDistinctFloods = flood2chars.size();
-    size_t floodHeaderSize = sizeof(u32) * N_CHARS;
-    size_t floodStructSize = sizeof(FDRFlood) * nDistinctFloods;
-    size_t totalSize = ROUNDUP_16(floodHeaderSize + floodStructSize);
-
-    auto buf = make_zeroed_bytecode_ptr<u8>(totalSize, 16);
-    assert(buf); // otherwise would have thrown std::bad_alloc
-
-    u32 *floodHeader = (u32 *)buf.get();
-    FDRFlood *layoutFlood = (FDRFlood *)(buf.get() + floodHeaderSize);
-
-    u32 currentFloodIndex = 0;
-    for (const auto &m : flood2chars) {
-        const FDRFlood &fl = m.first;
-        const CharReach &cr = m.second;
-        layoutFlood[currentFloodIndex] = fl;
-        for (size_t c = cr.find_first(); c != cr.npos; c = cr.find_next(c)) {
-            floodHeader[c] = currentFloodIndex;
-        }
-        currentFloodIndex++;
-    }
-
-    DEBUG_PRINTF("made a flood structure with %zu + %zu = %zu\n",
-                 floodHeaderSize, floodStructSize, totalSize);
-
-    return buf;
-}
-
-} // namespace ue2

src/fdr/flood_runtime.h

@@ -1,337 +0,0 @@
-/*
- * Copyright (c) 2015-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef FLOOD_RUNTIME
-#define FLOOD_RUNTIME
-
-#if defined(ARCH_64_BIT)
-#define FLOOD_64
-#else
-#define FLOOD_32
-#endif
-#define FLOOD_MINIMUM_SIZE 256
-#define FLOOD_BACKOFF_START 32
-
-static really_inline
-const u8 * nextFloodDetect(const u8 * buf, size_t len, u32 floodBackoff) {
-    // if we don't have a flood at either the start or end,
-    // or have a very small buffer, don't bother with flood detection
-    if (len < FLOOD_MINIMUM_SIZE) {
-        return buf + len;
-    }
-
-    /* entry points in runtime.c prefetch relevant data */
-#ifndef FLOOD_32
-    u64a x11 = *(const u64a *)ROUNDUP_PTR(buf, 8);
-    u64a x12 = *(const u64a *)ROUNDUP_PTR(buf+8, 8);
-    if (x11 == x12) {
-        return buf + floodBackoff;
-    }
-    u64a x21 = *(const u64a *)ROUNDUP_PTR(buf + len/2, 8);
-    u64a x22 = *(const u64a *)ROUNDUP_PTR(buf + len/2 + 8, 8);
-    if (x21 == x22) {
-        return buf + floodBackoff;
-    }
-    u64a x31 = *(const u64a *)ROUNDUP_PTR(buf + len - 24, 8);
-    u64a x32 = *(const u64a *)ROUNDUP_PTR(buf + len - 16, 8);
-    if (x31 == x32) {
-        return buf + floodBackoff;
-    }
-#else
-    u32 x11 = *(const u32 *)ROUNDUP_PTR(buf, 4);
-    u32 x12 = *(const u32 *)ROUNDUP_PTR(buf+4, 4);
-    if (x11 == x12) {
-        return buf + floodBackoff;
-    }
-    u32 x21 = *(const u32 *)ROUNDUP_PTR(buf + len/2, 4);
-    u32 x22 = *(const u32 *)ROUNDUP_PTR(buf + len/2 + 4, 4);
-    if (x21 == x22) {
-        return buf + floodBackoff;
-    }
-    u32 x31 = *(const u32 *)ROUNDUP_PTR(buf + len - 12, 4);
-    u32 x32 = *(const u32 *)ROUNDUP_PTR(buf + len - 8, 4);
-    if (x31 == x32) {
-        return buf + floodBackoff;
-    }
-#endif
-    return buf + len;
-}
-
-static really_inline
-const u8 * floodDetect(const struct FDR * fdr,
-                       const struct FDR_Runtime_Args * a,
-                       const u8 ** ptrPtr,
-                       const u8 * tryFloodDetect,
-                       u32 * floodBackoffPtr,
-                       hwlmcb_rv_t * control,
-                       u32 iterBytes) {
-    DEBUG_PRINTF("attempting flood detection at %p\n", tryFloodDetect);
-    const u8 * buf = a->buf;
-    const size_t len = a->len;
-    HWLMCallback cb = a->cb;
-    struct hs_scratch *scratch = a->scratch;
-
-    const u8 * ptr = *ptrPtr;
-    // tryFloodDetect is never put in places where unconditional
-    // reads a short distance forward or backward here
-    // TODO: rationale for this line needs to be rediscovered!!
-    size_t mainLoopLen = len > 2 * iterBytes ? len - 2 * iterBytes : 0;
-    const u32 i = ptr - buf;
-    u32 j = i;
-
-    // go from c to our FDRFlood structure
-    u8 c = buf[i];
-    const u8 * fBase = ((const u8 *)fdr) + fdr->floodOffset;
-    u32 fIdx = ((const u32 *)fBase)[c];
-    const struct FDRFlood * fsb = (const struct FDRFlood *)(fBase + sizeof(u32) * 256);
-    const struct FDRFlood * fl = &fsb[fIdx];
-
-#ifndef FLOOD_32
-    u64a cmpVal = c;
-    cmpVal |= cmpVal << 8;
-    cmpVal |= cmpVal << 16;
-    cmpVal |= cmpVal << 32;
-    u64a probe = *(const u64a *)ROUNDUP_PTR(buf+i, 8);
-#else
-    u32 cmpVal = c;
-    cmpVal |= cmpVal << 8;
-    cmpVal |= cmpVal << 16;
-    u32 probe = *(const u32 *)ROUNDUP_PTR(buf+i, 4);
-#endif
-
-    if ((probe != cmpVal) || (fl->idCount >= FDR_FLOOD_MAX_IDS)) {
-        *floodBackoffPtr *= 2;
-        goto floodout;
-    }
-
-    if (i < fl->suffix + 7) {
-        *floodBackoffPtr *= 2;
-        goto floodout;
-    }
-
-    j = i - fl->suffix;
-
-#ifndef FLOOD_32
-    j -= (u32)((uintptr_t)buf + j) & 0x7; // push j back to yield 8-aligned addrs
-    for (; j + 32 < mainLoopLen; j += 32) {
-        u64a v = *(const u64a *)(buf + j);
-        u64a v2 = *(const u64a *)(buf + j + 8);
-        u64a v3 = *(const u64a *)(buf + j + 16);
-        u64a v4 = *(const u64a *)(buf + j + 24);
-        if ((v4 != cmpVal) || (v3 != cmpVal) || (v2 != cmpVal) || (v != cmpVal)) {
-            break;
-        }
-    }
-    for (; j + 8 < mainLoopLen; j += 8) {
-        u64a v = *(const u64a *)(buf + j);
-        if (v != cmpVal) {
-            break;
-        }
-    }
-#else
-    j -= (u32)((size_t)buf + j) & 0x3; // push j back to yield 4-aligned addrs
-    for (; j + 16 < mainLoopLen; j += 16) {
-        u32 v = *(const u32 *)(buf + j);
-        u32 v2 = *(const u32 *)(buf + j + 4);
-        u32 v3 = *(const u32 *)(buf + j + 8);
-        u32 v4 = *(const u32 *)(buf + j + 12);
-        if ((v4 != cmpVal) || (v3 != cmpVal) || (v2 != cmpVal) || (v != cmpVal)) {
-            break;
-        }
-    }
-    for (; j + 4 < mainLoopLen; j += 4) {
-        u32 v = *(const u32 *)(buf + j);
-        if (v != cmpVal) {
-            break;
-        }
-    }
-#endif
-    for (; j < mainLoopLen; j++) {
-        u8 v = *(const u8 *)(buf + j);
-        if (v != c) {
-            break;
-        }
-    }
-    if (j > i ) {
-        j--; // needed for some reaches
-        u32 itersAhead = (j-i)/iterBytes;
-        u32 floodSize = itersAhead*iterBytes;
-
-        DEBUG_PRINTF("flooding %u size j %u i %u fl->idCount %hu "
-                     "*control %016llx fl->allGroups %016llx\n",
-                     floodSize, j, i, fl->idCount, *control, fl->allGroups);
-        DEBUG_PRINTF("mainloopLen %zu mainStart ??? mainEnd ??? len %zu\n",
-                     mainLoopLen, len);
-
-        if (fl->idCount && (*control & fl->allGroups)) {
-            switch (fl->idCount) {
-#if !defined(FLOOD_DEBUG)
-            // Carefully unrolled code
-            case 1:
-                for (u32 t = 0; t < floodSize && (*control & fl->allGroups);
-                     t += 4) {
-                    DEBUG_PRINTF("aaa %u %llx\n", t, fl->groups[0]);
-                    if (*control & fl->groups[0]) {
-                        *control = cb(i + t + 0, fl->ids[0], scratch);
-                    }
-                    if (*control & fl->groups[0]) {
-                        *control = cb(i + t + 1, fl->ids[0], scratch);
-                    }
-                    if (*control & fl->groups[0]) {
-                        *control = cb(i + t + 2, fl->ids[0], scratch);
-                    }
-                    if (*control & fl->groups[0]) {
-                        *control = cb(i + t + 3, fl->ids[0], scratch);
-                    }
-                }
-                break;
-            case 2:
-                for (u32 t = 0; t < floodSize && (*control & fl->allGroups); t += 4) {
-                    if (*control & fl->groups[0]) {
-                        *control = cb(i + t, fl->ids[0], scratch);
-                    }
-                    if (*control & fl->groups[1]) {
-                        *control = cb(i + t, fl->ids[1], scratch);
-                    }
-                    if (*control & fl->groups[0]) {
-                        *control =
-                            cb(i + t + 1, fl->ids[0], scratch);
-                    }
-                    if (*control & fl->groups[1]) {
-                        *control = cb(i + t + 1, fl->ids[1], scratch);
-                    }
-                    if (*control & fl->groups[0]) {
-                        *control = cb(i + t + 2, fl->ids[0], scratch);
-                    }
-                    if (*control & fl->groups[1]) {
-                        *control = cb(i + t + 2, fl->ids[1], scratch);
-                    }
-                    if (*control & fl->groups[0]) {
-                        *control = cb(i + t + 3, fl->ids[0], scratch);
-                    }
-                    if (*control & fl->groups[1]) {
-                        *control = cb(i + t + 3, fl->ids[1], scratch);
-                    }
-                }
-                break;
-            case 3:
-                for (u32 t = 0; t < floodSize && (*control & fl->allGroups); t += 2) {
-                    if (*control & fl->groups[0]) {
-                        *control = cb(i + t, fl->ids[0], scratch);
-                    }
-                    if (*control & fl->groups[1]) {
-                        *control = cb(i + t, fl->ids[1], scratch);
-                    }
-                    if (*control & fl->groups[2]) {
-                        *control = cb(i + t, fl->ids[2], scratch);
-                    }
-                    if (*control & fl->groups[0]) {
-                        *control = cb(i + t + 1, fl->ids[0], scratch);
-                    }
-                    if (*control & fl->groups[1]) {
-                        *control = cb(i + t + 1, fl->ids[1], scratch);
-                    }
-                    if (*control & fl->groups[2]) {
-                        *control = cb(i + t + 1, fl->ids[2], scratch);
-                    }
-                }
-                break;
-            default:
-                // slow generalized loop
-                for (u32 t = 0; t < floodSize && (*control & fl->allGroups); t += 2) {
-
-                    if (*control & fl->groups[0]) {
-                        *control = cb(i + t, fl->ids[0], scratch);
-                    }
-                    if (*control & fl->groups[1]) {
-                        *control = cb(i + t, fl->ids[1], scratch);
-                    }
-                    if (*control & fl->groups[2]) {
-                        *control = cb(i + t, fl->ids[2], scratch);
-                    }
-                    if (*control & fl->groups[3]) {
-                        *control = cb(i + t, fl->ids[3], scratch);
-                    }
-
-                    for (u32 t2 = 4; t2 < fl->idCount; t2++) {
-                        if (*control & fl->groups[t2]) {
-                            *control = cb(i + t, fl->ids[t2], scratch);
-                        }
-                    }
-
-                    if (*control & fl->groups[0]) {
-                        *control = cb(i + t + 1, fl->ids[0], scratch);
-                    }
-                    if (*control & fl->groups[1]) {
-                        *control = cb(i + t + 1, fl->ids[1], scratch);
-                    }
-                    if (*control & fl->groups[2]) {
-                        *control = cb(i + t + 1, fl->ids[2], scratch);
-                    }
-                    if (*control & fl->groups[3]) {
-                        *control = cb(i + t + 1, fl->ids[3], scratch);
-                    }
-
-                    for (u32 t2 = 4; t2 < fl->idCount; t2++) {
-                        if (*control & fl->groups[t2]) {
-                            *control = cb(i + t + 1, fl->ids[t2], scratch);
-                        }
-                    }
-                }
-                break;
-#else
-            // Fallback for debugging
-            default:
-                for (u32 t = 0; t < floodSize && (*control & fl->allGroups); t++) {
-                    for (u32 t2 = 0; t2 < fl->idCount; t2++) {
-                        if (*control & fl->groups[t2]) {
-                            *control = cb(i + t, fl->ids[t2], scratch);
-                        }
-                    }
-                }
-#endif
-            }
-        }
-        ptr += floodSize;
-    } else {
-        *floodBackoffPtr *= 2;
-    }
-
-floodout:
-    if (j + *floodBackoffPtr < mainLoopLen - 128) {
-        tryFloodDetect = buf + MAX(i,j) + *floodBackoffPtr;
-    } else {
-        tryFloodDetect = buf + mainLoopLen; // set so we never do another flood detect
-    }
-    *ptrPtr = ptr;
-    DEBUG_PRINTF("finished flood detection at %p (next check %p)\n",
-                 ptr, tryFloodDetect);
-    return tryFloodDetect;
-}
-
-#endif

src/fdr/teddy.h

@@ -1,110 +0,0 @@
-/*
- * Copyright (c) 2016-2017, Intel Corporation
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- *
- *  * Redistributions of source code must retain the above copyright notice,
- *    this list of conditions and the following disclaimer.
- *  * Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *  * Neither the name of Intel Corporation nor the names of its contributors
- *    may be used to endorse or promote products derived from this software
- *    without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/** \file
- * \brief Teddy literal matcher: function declarations.
- */
-
-#ifndef TEDDY_H_
-#define TEDDY_H_
-
-#include "hwlm/hwlm.h" // for hwlm_group_t
-#include "util/arch.h"
-
-struct FDR; // forward declaration from fdr_internal.h
-struct FDR_Runtime_Args;
-
-hwlm_error_t fdr_exec_teddy_msks1(const struct FDR *fdr,
-                                  const struct FDR_Runtime_Args *a,
-                                  hwlm_group_t control);
-
-hwlm_error_t fdr_exec_teddy_msks1_pck(const struct FDR *fdr,
-                                      const struct FDR_Runtime_Args *a,
-                                      hwlm_group_t control);
-
-hwlm_error_t fdr_exec_teddy_msks2(const struct FDR *fdr,
-                                  const struct FDR_Runtime_Args *a,
-                                  hwlm_group_t control);
-
-hwlm_error_t fdr_exec_teddy_msks2_pck(const struct FDR *fdr,
-                                      const struct FDR_Runtime_Args *a,
-                                      hwlm_group_t control);
-
-hwlm_error_t fdr_exec_teddy_msks3(const struct FDR *fdr,
-                                  const struct FDR_Runtime_Args *a,
-                                  hwlm_group_t control);
-
-hwlm_error_t fdr_exec_teddy_msks3_pck(const struct FDR *fdr,
-                                      const struct FDR_Runtime_Args *a,
-                                      hwlm_group_t control);
-
-hwlm_error_t fdr_exec_teddy_msks4(const struct FDR *fdr,
-                                  const struct FDR_Runtime_Args *a,
-                                  hwlm_group_t control);
-
-hwlm_error_t fdr_exec_teddy_msks4_pck(const struct FDR *fdr,
-                                      const struct FDR_Runtime_Args *a,
-                                      hwlm_group_t control);
-
-#if defined(HAVE_AVX2)
-
-hwlm_error_t fdr_exec_fat_teddy_msks1(const struct FDR *fdr,
-                                      const struct FDR_Runtime_Args *a,
-                                      hwlm_group_t control);
-
-hwlm_error_t fdr_exec_fat_teddy_msks1_pck(const struct FDR *fdr,
-                                          const struct FDR_Runtime_Args *a,
-                                          hwlm_group_t control);
-
-hwlm_error_t fdr_exec_fat_teddy_msks2(const struct FDR *fdr,
-                                      const struct FDR_Runtime_Args *a,
-                                      hwlm_group_t control);
-
-hwlm_error_t fdr_exec_fat_teddy_msks2_pck(const struct FDR *fdr,
-                                          const struct FDR_Runtime_Args *a,
-                                          hwlm_group_t control);
-
-hwlm_error_t fdr_exec_fat_teddy_msks3(const struct FDR *fdr,
-                                      const struct FDR_Runtime_Args *a,
-                                      hwlm_group_t control);
-
-hwlm_error_t fdr_exec_fat_teddy_msks3_pck(const struct FDR *fdr,
-                                          const struct FDR_Runtime_Args *a,
-                                          hwlm_group_t control);
-
-hwlm_error_t fdr_exec_fat_teddy_msks4(const struct FDR *fdr,
-                                      const struct FDR_Runtime_Args *a,
-                                      hwlm_group_t control);
-
-hwlm_error_t fdr_exec_fat_teddy_msks4_pck(const struct FDR *fdr,
-                                          const struct FDR_Runtime_Args *a,
-                                          hwlm_group_t control);
-
-#endif /* HAVE_AVX2 */
-
-#endif /* TEDDY_H_ */